Gesture recognition method based on AR / MR technology

Abstract

The invention discloses a gesture recognition method based on an AR / MR technology. The gesture recognition method comprises the following steps of S1, collecting gesture data of a user in real time; s2, performing feature extraction on the gesture data by using a deep learning technology; s3, combining a voice input function; s4, using an artificial intelligence feedback voice input function; s5, virtual elements are combined with the real world through the AR / MR technology; s7, adaptively adjusting the sensitivity and response speed of gesture recognition according to the hand size and action speed of the user; s8, regularly training and verifying model parameters, performing feature extraction by applying a CNN (convolutional neural network) to improve the accuracy and robustness of gesture recognition, predicting possible positions of sheltered fingers by applying a machine learning model, and performing virtual reconstruction through a three-dimensional modeling technology to solve the influence of finger sheltering on gesture recognition, so that the gesture recognition accuracy and robustness are improved. The threshold value can be dynamically adjusted according to the hand size and the action speed of the user through a self-adaptive threshold value adjustment algorithm so as to adapt to the specific characteristics of the user.

Description

Technical Field

[0001] The present invention relates to the technical field of augmented reality and mixed reality, and in particular to a gesture recognition method based on AR / MR technology. Background Art

[0002] The gesture recognition method based on AR / MR technology is leading the new trend of human-computer interaction, and is committed to achieving a more natural, intuitive and immersive user experience. With the integration and innovation of technologies, especially the rapid development of advanced technologies such as computer vision and deep learning, gesture recognition technology has been able to achieve significant improvements in accuracy and efficiency. This change has not only promoted the intelligent process in many fields such as smart home, game entertainment, education and training, but also brought unprecedented interactive possibilities to AR / MR applications, allowing users to freely shuttle between virtual and reality through simple gesture operations, and enjoy unprecedented interactive fun and convenience.

[0003] After searching, the invention patent with Chinese patent number CN115798054A discloses a gesture recognition method and electronic device based on AR / MR technology, in the field of spatial data analysis and recognition technology, setting and storing relevant variables in the gesture recognition process; setting constant information as the basis for gesture recognition judgment, executing the gesture recognition engine, and performing recognition judgment on each frame: judging whether the SDK tracks the thumb tip and the ring finger tip, if any finger tip is not tracked, the recognition process of this frame ends; if the thumb tip and the ring finger tip are recognized at the same time, the position of the thumb tip in the current frame, i.e., the three-dimensional spatial coordinate, is recorded; the present invention can summarize the relevant variables, constant information and gesture recognition engine for gesture recognition, which is convenient for users to refer to and use, and is convenient for the system to execute the corresponding gesture recognition process according to the relevant variables, constant information and gesture recognition engine. Effectively improve the accuracy and precision of gesture recognition.

[0004] However, the above device only relies on thumb and ring fingertip tracking during actual use: This method mainly relies on tracking the positions of thumb and ring fingertips to recognize gestures, which may lead to inaccurate recognition in some cases, such as finger occlusion or posture changes. Lack of personalized adjustment: This method does not mention how to adjust the parameters of gesture recognition according to the user's personal habits or preferences, which may cause inconvenience or discomfort to some users. Fixed time and distance thresholds: This method uses fixed thresholds to judge the effectiveness of gestures, which may not be suitable for all users and usage scenarios, because different people may have different hand sizes and movement speeds. Therefore, a gesture recognition method based on AR / MR technology is proposed. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the prior art that gesture recognition only relies on tracking the positions of the thumb and ring fingertips, which may lead to inaccurate recognition in some cases, lack of personalized adjustment: this may cause inconvenience or discomfort to some users, and fixed time and distance thresholds: using fixed thresholds to judge the validity of gestures is not suitable for all users and usage scenarios, and proposes a gesture recognition method based on AR / MR technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A gesture recognition method based on AR / MR technology includes the following steps:

[0008] S1: collect user gesture data in real time and obtain the user's hand movements through a camera or sensor;

[0009] S2: Use deep learning technology to extract features from gesture data to identify specific gestures;

[0010] S3: Combined with voice input function, allowing users to enhance the interactive experience through voice commands;

[0011] S4: Use artificial intelligence feedback voice input function to provide feedback information to users through speech synthesis technology;

[0012] S5: Combine virtual elements with the real world through AR / MR technology to provide an immersive interactive experience;

[0013] S6: Adaptively adjust the sensitivity and response speed of gesture recognition according to the user's hand size and movement speed;

[0014] S7: Regularly train and verify model parameters to adapt to users' personalized needs and evaluate the performance of the model in different application scenarios.

[0015] The above technical solution further includes:

[0016] Preferably, the S1: data collection is further refined as follows:

[0017] S1.1: Use a high-resolution camera to capture the user's gestures;

[0018] S1.2: Collect physical data of gestures through sensors;

[0019] S1.3: Integrate the data from the camera and sensors to form a complete gesture dataset, including:

[0020]

[0021] Among them, HandGestureData represents the collected gesture data, and Camera and Sensor represent the camera and sensor used to collect gesture data respectively.

[0022] Preferably, the S2: feature extraction is further refined as follows:

[0023] S2.1: Apply convolutional neural network (CNN) to extract features from gesture images captured by the camera;

[0024] S2.2: Extract key features from gesture data through deep learning model.

[0025] Specifically include:

[0026] FeatureExtraction(HandGestureData)=CNN(HandGestureData)

[0027] Among them, FeatureExtraction represents the feature extraction function, and DeepLearningModel represents the deep learning model, which is used to extract key features from gesture data.

[0028] Preferably, the S2: occlusion processing is further refined as follows:

[0029] S3.1: When finger occlusion is detected, use the machine learning model to predict the possible position of the occluding finger;

[0030] S3.2: Virtual reconstruction using 3D modeling technology, including:

[0031] OcclusionHandling(FeatureExtraction(HandGestureData))

[0032] =MLModel(FeatureExtraction(HandGestureData))

[0033] Among them, OcclusionHandling represents the occlusion processing function, and MLModel represents the machine learning model used to predict the position of the occluded finger.

[0034] Preferably, S2: gesture recognition is further refined as follows:

[0035] S4.1: Combining the results of feature extraction and occlusion processing, recognizing the user's gesture;

[0036] S4.2: Mapping the recognized gesture to a specific operation of the virtual object, including:

[0037] GestureRecognition(FeatureExtraction(HandGestureData),

[0038] OcclusionHandling(FeatureExtraction(HandGestureData)))

[0039] =GestureClass

[0040] Among them, GestureRecognition represents the gesture recognition function, and GestureClass represents the recognized gesture category.

[0041] Preferably, the S3: voice input is further refined as follows:

[0042] S5.1: Use multi-microphone array technology to capture user voice commands;

[0043] S5.2: Perform noise suppression and echo cancellation;

[0044] S5.3: Parsing voice commands through natural language processing, including:

[0045] VoiceInput(UserCommand)=MicArray(UserCommand)

[0046] Among them, VoiceInput represents the voice input function, UserCommand represents the voice command input by the user, and NLP represents the natural language processing function, which is used to parse the voice command.

[0047] Preferably, the S4: voice feedback is further refined as follows:

[0048] S6.1: Synthesize feedback speech using a text-to-speech (TTS) engine based on the user's interactive instructions and context;

[0049] S6.2: Play feedback audio through the speaker of the AR / MR device, including:

[0050] VoiceFeedback(UserCommand)=TTS(FeedbackContent)

[0051] Among them, VoiceFeedback represents a voice feedback function, and TTS represents a text-to-speech synthesis function, which is used to convert feedback content into voice form.

[0052] Preferably, the S5: virtual element interaction is further refined as follows:

[0053] S7.1: Implement gesture and voice-driven interactions with virtual elements;

[0054] S7.2: Allow users to manipulate virtual objects through gestures, including:

[0055] Interaction(GestureRecognition(FeatureExtraction(HandGestureData),

[0056] OcclusionHandling(FeatureExtraction(HandGestureData)))),VirtualObject)

[0057] =UpdatedVirtualObject

[0058] Among them, Interaction represents the virtual element interaction function, GestureRecognition represents the gesture recognition function, FeatureExtraction represents the feature extraction function, OcclusionHandling represents the occlusion processing function, HandGestureData represents the collected gesture data, VirtualObject represents the virtual object, and UpdatedVirtualObject represents the virtual object state updated after the interaction.

[0059] Preferably, the step S6: adaptive adjustment is further refined as follows:

[0060] S8.1: Based on the user's hand size and movement speed data, apply online learning technology to update the threshold model in real time;

[0061] S8.2: Adjust the sensitivity and response speed of gesture recognition, including:

[0062] AdaptiveThreshold(UserHandSize,UserActionSpeed)=ThresholdValue

[0063] Among them, AdaptiveThreshold represents the adaptive threshold function, UserHandSize and UserActionSpeed ​​represent the user's hand size and action speed respectively, and ThresholdValue represents the threshold adjusted according to the user's characteristics.

[0064] Preferably, the S7: model training and verification is further refined as follows:

[0065] S9.1: Train the model using the collected dataset;

[0066] S9.2: Evaluate the performance of the model using validation data, including:

[0067] ModelTraining(TrainingData)=TrainedModelModel

[0068] Validation(TrainedModel,ValidationData)=PerformanceMetrics

[0069] Among them, ModelTraining represents the model training function, TrainingData represents the data used to train the model, TrainedModel represents the trained model, ModelValidation represents the model validation function, ValidationData represents the data used to validate the model, and PerformanceMetrics represents the performance indicators of the model on the validation data.

[0070] Preferably, a convolutional neural network (CNN) is used to extract features from the gesture image captured by the camera. Pre-trained models such as ResNet, VGG, etc. can be used as basic models and fine-tuned for gesture recognition tasks.

[0071] Preferably, when finger occlusion is detected, a deep learning model (such as a Transformer-based model) can be used to predict the position of the occluding finger. Virtual reconstruction is performed through three-dimensional modeling technology to ensure the integrity of gesture recognition.

[0072] Preferably, the user's gesture is recognized by combining the results of feature extraction and occlusion processing. A classification algorithm such as a support vector machine (SVM) or a decision tree may be used to recognize the gesture category.

[0073] Preferably, a multi-microphone array technology is used to capture the user's voice commands, and noise suppression and echo cancellation are performed. The voice commands are analyzed by natural language processing (NLP) technology.

[0074] Preferably, the feedback voice is synthesized using text-to-speech (TTS) technology according to the user's interactive instructions and contextual environment, and played through the speaker of the AR / MR device.

[0075] Preferably, gesture and voice-driven virtual element interaction is implemented, allowing users to operate virtual objects through gestures.

[0076] Preferably, based on the user's hand size and movement speed data, online learning technology is applied to update the threshold model in real time to adjust the sensitivity and response speed of gesture recognition.

[0077] Preferably, the model is trained using the collected data set, and the performance of the model is evaluated by the validation data. Methods such as cross-validation can be used to ensure the generalization ability of the model.

[0078] Preferably, multiple languages ​​are supported, including English, Chinese, Spanish, etc. Users can choose a language that suits them for voice input.

[0079] Preferably, in order to improve the accuracy of speech recognition and user experience, the method automatically adjusts the language setting according to the user's language habits and environmental background.

[0080] Preferably, voice input and gesture input are complementary interaction modes. The user can choose to use voice input or gesture input alone, or use both input modes at the same time.

[0081] Preferably, the order of voice input and gesture input can be adjusted according to the user's needs and scenarios. For example, the user can first input a command through voice, and then operate the virtual object through gestures; or first perform gesture recognition, and then further confirm or modify the command through voice input.

[0082] Preferably, the method dynamically adjusts the combination of voice input and gesture input according to the user's interaction history and preferences to provide a more personalized and natural interaction experience.

[0083] The present invention has the following beneficial effects:

[0084] 1. In the present invention, the accuracy and robustness of gesture recognition are improved by using convolutional neural network (CNN) for feature extraction. CNN can automatically learn the deep features of gesture images, thereby improving the accuracy and generalization ability of recognition. In addition, by enhancing the extraction of key gesture features through the attention mechanism, the present invention further improves the accuracy and robustness of gesture recognition.

[0085] 2. In the present invention, the possible position of the blocked finger is predicted by using a machine learning model, and virtual reconstruction is performed through three-dimensional modeling technology, thereby solving the impact of finger occlusion on gesture recognition. In practical applications, finger occlusion is a common phenomenon, and traditional CNN-based gesture recognition methods may fail. Through the above method, the present invention can accurately recognize gestures even if part of the fingers are blocked, thereby improving the accuracy and robustness of gesture recognition. The threshold can be dynamically adjusted according to the user's hand size and movement speed through an adaptive threshold adjustment algorithm to adapt to the user's specific characteristics. This personalized adjustment method can improve the adaptability and user experience of the gesture recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a structural schematic diagram of a gesture recognition method based on AR / MR technology proposed by the present invention. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Embodiment 1

[0089] like Figure 1 As shown, the present invention proposes a gesture recognition method based on AR / MR technology, comprising the following steps:

[0090] S1: collect user gesture data in real time and obtain the user's hand movements through a camera or sensor;

[0091] S2: Use deep learning technology to extract features from gesture data;

[0092] S3: Combined with voice input function, allowing users to enhance the interactive experience through voice commands;

[0093] S4: Use artificial intelligence feedback voice input function to provide feedback information to users through speech synthesis technology;

[0094] S5: Combine virtual elements with the real world through AR / MR technology to provide an immersive interactive experience;

[0095] S6: Adaptively adjust the sensitivity and response speed of gesture recognition according to the user's hand size and movement speed;

[0096] S7: Regularly train and verify model parameters to adapt to users' personalized needs and evaluate the performance of the model in different application scenarios.

[0097] In one embodiment, S1: data collection is further refined as follows:

[0098] S1.1: Use a high-resolution camera to capture the user's gestures;

[0099] S1.2: Collect physical data of gestures through sensors;

[0100] S1.3: Integrate the data from the camera and sensors to form a complete gesture dataset, including:

[0101] HandGestureData=Camera°Sensor

[0102] Among them, HandGestureData represents the collected gesture data, and Camera and Sensor represent the camera and sensor used to collect gesture data respectively.

[0103] In one embodiment, S2: feature extraction is further refined as follows:

[0104] S2.1: Apply convolutional neural network (CNN) to extract features from gesture images captured by the camera;

[0105] S2.2: Extract key features from gesture data through deep learning model.

[0106] Specifically include:

[0107] FeatureExtraction(HandGestureData)=CNN(HandGestureData)

[0108] Among them, FeatureExtraction represents the feature extraction function, and DeepLearningModel represents the deep learning model, which is used to extract key features from gesture data.

[0109] In one embodiment, S2: occlusion processing is further refined as follows:

[0110] S3.1: When finger occlusion is detected, use the machine learning model to predict the possible position of the occluding finger;

[0111] S3.2: Virtual reconstruction using 3D modeling technology, including:

[0112] OcclusionHandling(FeatureExtraction(HandGestureData))

[0113] =MLModel(FeatureExtraction(HandGestureData))

[0114] Among them, OcclusionHandling represents the occlusion processing function, and MLModel represents the machine learning model used to predict the position of the occluded finger.

[0115] In one embodiment, S2: gesture recognition is further refined as follows:

[0116] S4.1: Combining the results of feature extraction and occlusion processing, recognizing the user's gesture;

[0117] S4.2: Mapping the recognized gesture to a specific operation of the virtual object, including:

[0118] GestureRecognition(FeatureExtraction(HandGestureData),

[0119] OcclusionHandling(FeatureExtraction(HandGestureData)))

[0120] =GestureClass

[0121] Among them, GestureRecognition represents the gesture recognition function, and GestureClass represents the recognized gesture category.

[0122] In one embodiment, S3: voice input is further refined as follows:

[0123] S5.1: Use multi-microphone array technology to capture user voice commands;

[0124] S5.2: Perform noise suppression and echo cancellation;

[0125] S5.3: Parsing voice commands through natural language processing, including:

[0126] VoiceInput(UserCommand)=MicArray(UserCommand)

[0127] Among them, VoiceInput represents the voice input function, UserCommand represents the voice command input by the user, and NLP represents the natural language processing function, which is used to parse the voice command.

[0128] In one embodiment, S4: the voice feedback step is further refined as follows:

[0129] S6.1: Synthesize feedback speech using a text-to-speech (TTS) engine based on the user's interactive instructions and context;

[0130] S6.2: Play feedback audio through the speaker of the AR / MR device, including:

[0131] VoiceFeedback(UserCommand)=TTS(FeedbackContent)

[0132] Among them, VoiceFeedback represents a voice feedback function, and TTS represents a text-to-speech synthesis function, which is used to convert feedback content into voice form.

[0133] In one embodiment, S5: the virtual element interaction step is further refined as follows:

[0134] S7.1: Implement gesture and voice-driven interactions with virtual elements;

[0135] S7.2: Allow users to manipulate virtual objects through gestures, including:

[0136] Interaction(GestureRecognition(FeatureExtraction(HandGestureData),

[0137] OcclusionHandling(FeatureExtraction(HandGestureData)))),VirtualObject)

[0138] =UpdatedVirtualObject

[0139] Among them, Interaction represents the virtual element interaction function, GestureRecognition represents the gesture recognition function, FeatureExtraction represents the feature extraction function, OcclusionHandling represents the occlusion processing function, HandGestureData represents the collected gesture data, VirtualObject represents the virtual object, and UpdatedVirtualObject represents the virtual object state updated after the interaction.

[0140] In one embodiment, S6: the adaptive adjustment step is further refined as follows:

[0141] S8.1: Based on the user's hand size and movement speed data, apply online learning technology to update the threshold model in real time;

[0142] S8.2: Adjust the sensitivity and response speed of gesture recognition, including:

[0143] AdaptiveThreshold(UserHandSize,UserActionSpeed)=ThresholdValue

[0144] Among them, AdaptiveThreshold represents the adaptive threshold function, UserHandSize and UserActionSpeed ​​represent the user's hand size and action speed respectively, and ThresholdValue represents the threshold adjusted according to the user's characteristics.

[0145] In one embodiment, S7: the model training and verification step is further refined as follows:

[0146] S9.1: Train the model using the collected dataset;

[0147] S9.2: Evaluate the performance of the model using validation data, including:

[0148] ModelTraining(TrainingData)=TrainedModelModel

[0149] Validation(TrainedModel,ValidationData)=PerformanceMetrics

[0150] Among them, ModelTraining represents the model training function, TrainingData represents the data used to train the model, TrainedModel represents the trained model, ModelValidation represents the model validation function, ValidationData represents the data used to validate the model, and PerformanceMetrics represents the performance indicators of the model on the validation data.

[0151] In one embodiment, the user's gesture data is collected in real time. When the user starts the AR / MR application, the high-resolution camera starts to capture the user's gesture movements, and the sensor starts to collect physical data of the gesture, such as angle changes, acceleration, etc.

[0152] In one embodiment, deep learning technology is used for feature extraction, and the captured gesture image is subjected to feature extraction by a convolutional neural network (CNN). After feature extraction is completed, key features such as hand contours, finger positions, etc. are extracted.

[0153] In one embodiment, combined with the voice input function, the user can enhance the interactive experience through voice commands. The voice commands are captured and transmitted to the natural language processing module.

[0154] In one embodiment, using the artificial intelligence feedback voice input function, the voice command issued by the user is parsed and combined with the gesture data. The parsed command and gesture data are used together to determine the user's intention. Based on the user's interactive instructions, the system uses TTS technology to synthesize feedback speech.

[0155] In one embodiment, an immersive interactive experience is provided through AR / MR technology. The system combines virtual elements with the real world to provide users with an immersive interactive experience.

[0156] In one embodiment, the interaction between gestures and virtual objects is implemented, and the user operates the virtual object through gestures. The gestures are recognized and mapped to specific operations of the virtual object.

[0157] In one embodiment, the sensitivity of gesture recognition is adjusted according to the size of the user's hand and the speed of movement. The system adjusts the sensitivity and response speed of gesture recognition in real time according to the size of the user's hand and the speed of movement. For example, if it is detected that the user's hands are large and the movements are fast, the gesture recognition threshold is adjusted to respond to these gestures more quickly.

[0158] In one embodiment, the model parameters are trained and verified regularly, and the system regularly uses the collected data sets to train the model and evaluate the performance of the model through the verification data.

[0159] In one embodiment, the user starts the AR / MR application and the camera and sensors are ready. The camera starts capturing the user's gesture movements. The sensors collect physical data such as the speed, direction, and angle of the hand movement. From the gesture images captured by the camera, feature extraction is performed using a pre-trained CNN. The CNN model extracts key features from the gesture images, such as hand contours, finger positions, etc. The user issues a voice command: "Open the menu." The voice command is captured and passed to the natural language processing module for parsing. The parsed voice command is combined with the gesture data. The system recognizes that the user wants to open the menu.

[0160] The user makes a gesture, such as waving his hand. The gesture is recognized and mapped to the opening operation of the virtual menu. The system detects that the user has a large hand and waves his hand quickly. The system adjusts the sensitivity and response speed of gesture recognition in real time to adapt to the user's characteristics. The system uses TTS technology to synthesize feedback speech: "The menu is open". The feedback speech is played through the speaker of the AR / MR device. The system regularly trains the model using the collected data set. The performance of the model is evaluated through the validation data.

[0161] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A gesture recognition method based on AR / MR technology, characterized in that: The following steps are involved: S1: collect user gesture data in real time and obtain the user's hand movements through a camera or sensor; S2: Use deep learning technology to extract features and process occlusion of gesture data; S3: Combined with voice input function, allowing users to enhance the interactive experience through voice commands; S4: Use artificial intelligence feedback voice input function to provide feedback information to users through speech synthesis technology; S5: Combine virtual elements with the real world through AR / MR technology to provide an immersive interactive experience; S6: Adaptively adjust the sensitivity and response speed of gesture recognition according to the user's hand size and movement speed; S7: Regularly train and verify model parameters to adapt to users' personalized needs and evaluate the performance of the model in different application scenarios.

2. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S1: data collection is further refined as follows: S1.1: Use a high-resolution camera to capture the user's gestures; S1.2: Collect physical data of gestures through sensors; S1.3: Integrate the data from the camera and sensors to form a complete gesture dataset, including: HandGestureData=Camera°Sensor Among them, HandGestureData represents the collected gesture data, and Camera and Sensor represent the camera and sensor used to collect gesture data respectively.

3. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S2: feature extraction is further refined as follows: S2.1: Apply convolutional neural network (CNN) to extract features from gesture images captured by the camera; S2.2: Extract key features from gesture data through deep learning model. Specifically include: FeatureExtraction(HandGestureData)=CNN(HandGestureData) Among them, FeatureExtraction represents the feature extraction function, and DeepLearningModel represents the deep learning model, which is used to extract key features from gesture data.

4. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S2: occlusion processing is further refined as follows: S3.1: When finger occlusion is detected, use the machine learning model to predict the possible position of the occluding finger; S3.2: Virtual reconstruction using 3D modeling technology, including: OcclusionHandling(FeatureExtraction(HandGestureData)) =MLModel(FeatureExtraction(HandGestureData)) Among them, OcclusionHandling represents the occlusion processing function, and MLModel represents the machine learning model used to predict the position of the occluded finger.

5. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S2: gesture recognition is further refined as follows: S4.1: Combining the results of feature extraction and occlusion processing, recognizing the user's gesture; S4.2: Mapping the recognized gesture to a specific operation of the virtual object, including: GestureRecognition(FeatureExtraction(HandGestureData), OcclusionHandling(FeatureExtraction(HandGestureData))) =GestureClass Among them, GestureRecognition represents the gesture recognition function, and GestureClass represents the recognized gesture category.

6. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S3: voice input is further refined as follows: S5.1: Use multi-microphone array technology to capture user voice commands; S5.2: Perform noise suppression and echo cancellation; S5.3: Parsing voice commands through natural language processing, including: VoiceInput(UserCommand)=MicArray(UserCommand) Among them, VoiceInput represents the voice input function, UserCommand represents the voice command input by the user, and NLP represents the natural language processing function, which is used to parse the voice command.

7. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S4: voice feedback is further refined as follows: S6.1: Synthesize feedback speech using a text-to-speech (TTS) engine based on the user's interactive instructions and context; S6.2: Play feedback audio through the speaker of the AR / MR device, including: VoiceFeedback(UserCommand)=TTS(FeedbackContent) Among them, VoiceFeedback represents a voice feedback function, and TTS represents a text-to-speech synthesis function, which is used to convert feedback content into voice form.

8. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S5: Virtual Element Interaction is further refined as follows: S7.1: Implement gesture and voice-driven interactions with virtual elements; S7.2: Allow users to manipulate virtual objects through gestures, including: Interaction(GestureRecognition(FeatureExtraction(HandGest ureData), OcclusionHandling(FeatureExtraction(HandGestureData)))),V irtualObject) =UpdatedVirtualObject Among them, Interaction represents the virtual element interaction function, GestureRecognition represents the gesture recognition function, FeatureExtraction represents the feature extraction function, OcclusionHandling represents the occlusion processing function, HandGestureData represents the collected gesture data, VirtualObject represents the virtual object, and UpdatedVirtualObject represents the virtual object state updated after the interaction.

9. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S6: Adaptive Adjustment: is further refined as follows: S8.1: Based on the user's hand size and movement speed data, apply online learning technology to update the threshold model in real time; S8.2: Adjust the sensitivity and response speed of gesture recognition, including: AdaptiveThreshold(UserHandSize,UserActionSpeed)=ThresholdValue Among them, AdaptiveThreshold represents the adaptive threshold function, UserHandSize and UserActionSpeed ​​represent the user's hand size and action speed respectively, and ThresholdValue represents the threshold adjusted according to the user's characteristics.

10. The gesture recognition method based on AR / MR technology according to claim 1, characterized in that: The S7: model training and validation is further refined as follows: S9.1: Train the model using the collected dataset; S9.2: Evaluate the performance of the model using validation data, including: ModelTraining(TrainingData)=TrainedModelModel Validation(TrainedModel,ValidationData)=PerformanceMetrics Among them, ModelTraining represents the model training function, TrainingData represents the data used to train the model, TrainedModel represents the trained model, ModelValidation represents the model validation function, ValidationData represents the data used to validate the model, and PerformanceMetrics represents the performance indicators of the model on the validation data.

Images