Multi-modal instruction interaction method and system, computing device and storage medium

Abstract

The invention discloses a multi-mode instruction interaction method and system based on a virtual character, computing equipment and a storage medium, and the method comprises the steps: obtaining and recognizing user posture data, and obtaining a first operation instruction corresponding to the posture data; voice data of the user are obtained and recognized to obtain intention data of the user, and a corresponding second operation instruction is obtained according to the intention data; the first operation instruction and the second operation instruction are subjected to timestamp alignment, the similarity of the first operation instruction and the second operation instruction in the semantic space is calculated, the first operation instruction and the second operation instruction are fused and dynamically adjusted through a weighted average method, and an instruction with the highest priority is obtained to serve as a final operation instruction to control an object in the virtual scene to move. According to the method, the gesture instruction and the voice instruction are deeply fused, the interaction space between the user and the virtual environment is greatly expanded, the limitation of a traditional single interaction mode is broken through, and more natural, efficient and intelligent immersive experience is provided for the user.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-modal instruction interaction method, system, computing device and storage medium based on a virtual character. Background Art

[0002] In recent years, with the rapid development of artificial intelligence such as deep learning and knowledge graphs, human-computer interaction technology in the XR field is ushering in new breakthrough opportunities. Some companies have laid out new XR interaction solutions, which integrate advanced AI technologies such as computer vision, natural language understanding, and knowledge reasoning, and have initially realized multi-channel and multi-modal interaction fusion, bringing users a more immersive and intelligent experience. Although independent voice or gesture control systems are relatively mature, these solutions often ignore the synergy of non-verbal gestures and voice commands in human communication. In multimodal interaction environments, especially in applications such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), a single interaction method is difficult to provide a natural and efficient user experience. Therefore, a command interaction system that can understand and process voice and gestures is needed. In virtual and virtual plus reality environments, natural and efficient human-computer interaction is crucial to user experience. Users need to be able to interact with virtual objects and interfaces intuitively and quickly in these immersive environments to achieve true immersion and ease of operation.

[0003] The limitation of the existing single interaction mode is that it only supports single voice or gesture control, which has many limitations. A single interaction mode often cannot meet the diverse needs in complex scenarios. For example, existing intelligent assistants use NLP and technology to achieve voice interaction with users, but they are still limited in complex tasks and specific field applications. They have limited contextual understanding and accurate command parsing capabilities for multi-round conversations. The voice interaction function of existing smart home systems usually relies on predefined instruction sets, which lack flexibility and intelligence and are difficult to handle complex and diverse user needs. The accuracy of voice and gesture recognition of existing VR interaction systems needs to be improved, especially in noisy environments and complex instructions. At the same time, the system's command parsing and execution logic is relatively simple, which makes it difficult to meet users' complex operation needs.

[0004] In addition, existing technologies lack a unified framework and efficient fusion algorithm when processing multimodal input (voice, text, gesture), resulting in inconsistent performance of the system under various input methods, affecting the user experience. Existing intelligent systems have difficulty accurately understanding user intentions and performing corresponding operations when processing complex and multi-step tasks. The ability to manage multiple rounds of dialogue and context is limited, resulting in users having to repeatedly input and confirm. Summary of the invention

[0005] In response to the above technical problems, the present invention provides a multimodal command interaction method, system, computing device and storage medium, which breaks through the limitations of the traditional single interaction mode and provides users with a more natural, efficient and intelligent immersive experience.

[0006] The first embodiment of the present invention provides a multimodal instruction interaction method, the method comprising: obtaining user posture data, identifying the posture data, and obtaining a first operation instruction corresponding to the posture data; obtaining user voice data, identifying the voice data to obtain user intention data, and obtaining a corresponding second operation instruction according to the intention data; aligning the first operation instruction and the second operation instruction by timestamp, and calculating the similarity of the first operation instruction and the second operation instruction in the semantic space, fusing the first operation instruction and the second operation instruction by a weighted average method and dynamically adjusting them, and obtaining the instruction with the highest priority as the final operation instruction; and controlling the movement of objects in a virtual scene according to the final operation instruction.

[0007] Optionally, the steps of obtaining user posture data, identifying the posture data, and obtaining a first operation instruction corresponding to the posture data include: entering a virtual environment through an XR device, and using a depth camera of the XR device to collect the user's hand image and joint position data in real time; semantically classifying the user's gestures through a deep learning algorithm to identify corresponding gesture instructions; and converting the gesture instructions into corresponding first operation instructions according to a predefined mapping relationship.

[0008] Optionally, the step of obtaining the user's voice data, identifying the voice data to obtain the user's intention data, and obtaining the corresponding second operation instruction based on the intention data includes: obtaining the user's voice data and transcribing it into text information through an acoustic model; performing semantic analysis on the text information to obtain the user's intention data; and converting the intention data into a corresponding second operation instruction based on a predefined mapping relationship.

[0009] Optionally, the steps of obtaining user posture data, identifying the posture data, and obtaining a first operation instruction corresponding to the posture data include: entering a virtual environment through an XR device, and using a posture sensor and a gaze tracker of the XR device to capture the azimuth angle and gaze focus of the user's head; semantically classifying the azimuth angle and gaze focus of the user's head through a deep learning algorithm, and identifying corresponding head instructions; and converting the head instructions into corresponding third operation instructions according to a predefined mapping relationship.

[0010] Optionally, the step of identifying the corresponding gesture instruction also includes: the user senses the motor vibration through a tactile glove or a handle controller, and the motor vibration prompts the user that the gesture has been recognized, and different vibration modes generate different feedback signals through the display of the user's XR device to distinguish different gestures.

[0011] Optionally, the method also includes: the feedback signal generated by the motor vibration of the display of the user's XR device is used to prompt the user to start or stop acquiring the user's voice data; the feedback signal generated by the motor vibration of the display of the user's XR device is also used to prompt the user that the second operation instruction has been executed.

[0012] A second embodiment of the present invention provides a multimodal instruction interaction system, the system comprising: a data acquisition module, used to acquire user posture data and voice data; a data recognition module, used to recognize the posture data and acquire a first operation instruction corresponding to the posture data; recognize the voice data to acquire the user's intention data, and acquire the corresponding second operation instruction according to the intention data; a strategy determination module, used to align the timestamps of the first operation instruction and the second operation instruction, and calculate the similarity of the first operation instruction and the second operation instruction in the semantic space, and use the weighted average method to fuse the first operation instruction and the second operation instruction and dynamically adjust them to obtain the instruction with the highest priority as the final operation instruction; an execution module, used to control the movement of objects in the virtual scene according to the final operation instruction.

[0013] Optionally, the data acquisition module includes an XR device, through which the virtual environment is entered, and a depth camera of the XR device is used to collect hand images and joint position data of the user in real time, or a posture sensor and a sight tracker of the XR device are used to capture the orientation angle and sight focus of the user's head;

[0014] The data recognition module includes a deep learning algorithm model, which semantically classifies or categorizes the user's gestures through the deep learning algorithm model, identifies corresponding gesture instructions or head instructions, and converts the gesture instructions or head instructions into corresponding first operation instructions or third operation instructions according to a predefined mapping relationship.

[0015] Optionally, the data acquisition module further includes an audio acquisition device, and the audio acquisition device is used to acquire voice data of the user.

[0016] Optionally, the system further comprises a tactile feedback device, wherein the tactile feedback device comprises a motor or a force feedback controller.

[0017] A third embodiment of the present invention provides a computing device, comprising a memory and a processor, wherein the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of any one of the multimodal instruction interaction methods described above are implemented.

[0018] A fourth embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of any of the multimodal instruction interaction methods described above.

[0019] In the technical solution provided by the embodiment of the present invention, the user's posture data and voice data are respectively obtained for identification, the corresponding operation instructions are obtained, the different operation instructions are timestamp aligned, and fused and dynamically adjusted to obtain the instruction with the highest priority as the final operation instruction, and the movement of objects in the virtual scene is controlled according to the final operation instruction. Compared with the prior art, the present invention deeply integrates the two commonly used interaction channels of gesture instructions and voice instructions, and optimizes the algorithms and strategies according to the characteristics of the XR scene, which greatly expands the interaction space between users and virtual environments, breaks through the limitations of the traditional single interaction mode, and provides users with a more natural, efficient and intelligent immersive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a flow chart of one embodiment of a multi-modal instruction interaction method of the present invention;

[0021] Figure 2 A schematic diagram of a flow chart of another embodiment of a multi-modal instruction interaction method of the present invention;

[0022] Figure 3 A schematic diagram of a flow chart of gesture recognition of a multi-modal instruction interaction method of the present invention;

[0023] Figure 4 A schematic diagram of a head recognition process of a multi-modal instruction interaction method of the present invention;

[0024] Figure 5 A schematic diagram of a flow chart of speech recognition of a multi-modal instruction interaction method of the present invention;

[0025] Figure 6 The figure is a schematic diagram of the structure of a multi-modal instruction interaction system of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] First, the terms involved in one or more embodiments of this specification are explained.

[0029] Multimodal interaction: Users can communicate with digital humans through voice, text, expressions, actions and gestures, and digital humans can also respond to users through voice, text, expressions, actions and gestures.

[0030] Immersion: refers to the subjective feeling of being in a virtual environment that users get, which is usually related to factors such as the realism of the display device and the naturalness of the interaction method.

[0031] XR: is a general term covering technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR), which aims to extend and expand human perception and interactive experience through digital technology.

[0032] Semantic understanding: It is a branch of natural language processing, which aims to enable machines to understand the true meaning of language like humans, including tasks such as word sense disambiguation, coreference resolution, and semantic role labeling.

[0033] In the current multimodal command interaction methods, multimodal input lacks a unified framework and efficient fusion algorithm, resulting in inconsistent performance of the system under various input methods, affecting the user experience. Existing technologies have difficulty accurately understanding user intent and performing corresponding operations when processing complex and multi-step tasks. The ability to manage multiple rounds of dialogue and context is limited, resulting in users having to repeatedly input and confirm.

[0034] Based on this, the present invention provides a multi-modal instruction interaction method, please refer to Figure 1 , Figure 2 As shown, the method comprises the following steps:

[0035] Step S10, obtaining user posture data, identifying the posture data, and obtaining a first operation instruction corresponding to the posture data.

[0036] In one embodiment of the present invention, please refer to Figure 3 As shown, the step S10 specifically includes:

[0037] Step S11, enter the virtual environment through the XR device, and use the depth camera of the XR device to collect the user's hand image and joint point position data in real time.

[0038] Step S12: semantically classify the user's gestures through a deep learning algorithm to identify corresponding gesture instructions.

[0039] Step S13: converting the gesture instruction into a corresponding first operation instruction according to a predefined mapping relationship.

[0040] In one embodiment of the present invention, please refer to Figure 4 As shown, the method further comprises the following steps:

[0041] Step S14, enter the virtual environment through the XR device, and use the posture sensor and gaze tracker of the XR device to capture the orientation angle and gaze focus of the user's head.

[0042] Step S15: semantically classify the orientation angle and sight focus of the user's head through a deep learning algorithm to identify the corresponding head command.

[0043] Step S16: converting the header instruction into a corresponding third operation instruction according to a predefined mapping relationship.

[0044] In addition to hand gestures, the user's head posture and eye movement also contain rich interaction information. Through the posture sensors (such as accelerometers, gyroscopes) and gaze tracking devices on the head-mounted device, the orientation angle of the user's head and the focus of sight can be captured to determine the user's attention distribution and interest preferences. Combined with this information, the system can automatically adjust the display content and interaction method, such as recommending relevant operation instructions based on the virtual object that the user's sight is focused on; or pausing gesture interaction based on the user's head-down action to avoid misoperation. This implementation requires semantic understanding of the head and eye movement data and the formulation of a reasonable interaction adjustment strategy.

[0045] Step S20, obtaining the user's voice data, and identifying the voice data to obtain the user's intention data, and obtaining the corresponding second operation instruction according to the intention data.

[0046] In one embodiment of the present invention, please refer to Figure 5 As shown, the step S20 specifically includes:

[0047] Step S21, obtaining the user's voice data and transcribing it into text information through an acoustic model.

[0048] Step S22, performing semantic analysis on the text information to obtain the user's intention data.

[0049] Step S23: converting the intention data into a corresponding second operation instruction according to a predefined mapping relationship.

[0050] Step S30, align the timestamps of the first operation instruction and the second operation instruction, calculate the similarity between the first operation instruction and the second operation instruction in the semantic space, fuse the first operation instruction and the second operation instruction using a weighted average method and dynamically adjust them, and obtain the instruction with the highest priority as the final operation instruction.

[0051] Step S40: Control the movement of objects in the virtual scene according to the final operation instruction.

[0052] The present invention proposes a multimodal command interaction method, which breaks through the limitations of the traditional single interaction mode and provides users with a more natural, efficient and intelligent immersive experience. This method deeply integrates two commonly used interaction channels, gesture commands and voice commands, and optimizes the algorithm and strategy according to the characteristics of XR scenes, greatly expanding the interaction space between users and virtual environments.

[0053] Specifically, the user of the present invention enters the virtual environment by wearing an XR device (such as AR glasses, VR helmets, etc.). The user's hand image and joint point position data are collected in real time using the device's built-in depth camera. Then, the gesture is semantically classified through deep learning algorithms such as convolutional neural networks to identify various predefined gesture instructions (such as grabbing, scaling, rotating, etc.). The identified gesture instructions are passed to the application or operating system running on the XR device. This process is usually implemented through the device's API (application programming interface) or middleware. After receiving the gesture instruction, the application or operating system converts it into a first operation instruction according to a predefined mapping relationship. For example, the "grab" gesture may correspond to an operation instruction to select and move a virtual object. According to the recognized operation instruction, the application will update the state and behavior of the object in the virtual scene, taking the control of object movement as an example: a. Select the target: determine the virtual object to be operated according to the gesture instruction, usually the object pointed to by the gesture or the pre-selected object. b. Calculate the displacement: calculate the direction and distance the object needs to move according to the movement trajectory and speed of the gesture. c. Update the position: apply the calculated displacement to the target object to update its position coordinates in the virtual scene. d. Rendering update: Re-render the virtual scene according to the change of the object's position, and present the effect of the object's movement in real time. The present invention also supports users to customize new gestures, dynamically expand the gesture library through mechanisms such as few-sample learning and online incremental learning, and improve the flexibility of interaction.

[0054] When the user makes a gesture command (such as a grab gesture), the virtual object in the field of view is selected, given controllable properties by the physics engine, and locked to the user's virtual palm. Subsequently, the user can control the movement of the object through gestures, such as adjusting the distance with both hands to scale the object, and rotating the wrist to change the direction of the object. During the movement of the object, the application can provide visual, auditory and other forms of feedback, such as highlighting the selected object, playing moving sound effects, etc., to enhance the user's interactive experience. The user can perform multiple gesture commands in succession to achieve continuous interaction with the virtual scene. The system will repeat the above process until the user ends the interaction or the application exits. Wave your arm to change the position of the object. The changes in the state of the object are transmitted to the graphics rendering engine in real time and presented in the user's field of view with high fidelity, creating an immersive feeling.

[0055] In order to break the expression bottleneck of gesture interaction, the present invention introduces a voice control channel. Once the user starts gesture interaction, the multimodal command interaction system adopted by the present invention automatically starts voice monitoring. The user's voice signal is transcribed into text information by the audio acquisition device through the acoustic model. Then, the natural language understanding module performs semantic analysis on the text information, extracts keywords and semantic slots, and the semantic slot is an important concept in natural language understanding (NLU), which plays a key role in voice command parsing and recognition. It refers to the keywords or phrases that carry specific semantic information in the sentence. By identifying the semantic slot, the system can understand the specific intention of the user's expression. For example, in the voice command "rotate the model 90 degrees to the left", "left" and "90 degrees" are two semantic slots, which respectively indicate the direction and angle of rotation. By extracting these two semantic slots, the system can accurately understand the user's rotation intention and map it to predefined operation instructions, obtain candidate voice control instructions, namely the second operation instructions (such as "zoom in", "zoom out", "rotate 90 degrees left", etc.), and transmit the voice instructions to the application or operating system running on the XR device, thereby achieving precise control of the XR device.

[0056] The semantics of the instructions output in parallel by gesture recognition and voice recognition may overlap, conflict or complement each other. To solve this problem, the present invention first aligns the timestamps of instructions from different sources. Then, the similarity in the semantic space is calculated, and the instructions are fused using the weighted average method, and the weights are dynamically adjusted based on contextual information such as the current scene and user intent. Finally, the fused instructions are sorted and screened, and one or more instructions with the highest execution priority are output as the final instructions.

[0057] The present invention adopts speaker adaptation and environment adaptation strategies in speech recognition, dynamically adjusts the acoustic model by online learning of the user's timbre characteristics and background noise characteristics, and suppresses noise interference. In gesture recognition, adaptive thresholds, confidence weighting and other strategies are adopted to improve the accuracy of recognition. In the gesture recognition algorithm, the recognition is considered successful; otherwise, the recognition is considered failed. Different users may have different gesture habits, hand sizes and environmental conditions. Fixed thresholds are difficult to adapt to such diversity, resulting in reduced recognition accuracy. By collecting user gesture data, the threshold is dynamically adjusted to adapt to the user's personal characteristics. Specifically, the threshold can be automatically optimized based on statistical information such as the success rate and failure rate of the user's historical gestures, so that the recognition result is closer to the user's actual intention. Confidence is an indicator to measure the credibility of gesture recognition results, and the value range is usually [0,1]. The higher the confidence, the stronger the certainty of the recognition algorithm for the result. In the scenario of continuous gesture recognition or multi-gesture fusion, the confidence of different gestures may be different. In order to obtain a more accurate final result, it is necessary to comprehensively consider the confidence of each gesture and give a greater weight to gestures with high confidence. Common weighting strategies include linear weighting and nonlinear weighting. Linear weighting directly uses the confidence of the gesture as the weight and performs a weighted average on the recognition results of each gesture. Nonlinear weighting uses some complex functions, such as exponential functions and logarithmic functions, to transform the confidence and then weight it to highlight the impact of high-confidence gestures. At the same time, explicit interaction backup channels (such as physical buttons) are preset for users as an alternative means of manual intervention.

[0058] The present invention combines multimodal input processing of speech recognition and gesture recognition to more accurately obtain user intentions; uses deep learning and other technologies to jointly model and fuse speech data and gesture data to improve the convenience of multimodal input; the present invention supports dynamic adaptive fusion of multimodal input, and intelligently adjusts the weights and priorities of speech and gesture input according to different scenarios and user preferences; uses deep semantic understanding and reasoning technology to process complex semantic structures and implicit logic, and accurately parses users' complex intentions; the present invention provides a customized training mechanism to support model fine-tuning and optimization for specific industries or user scenarios, and improves the adaptability of the interactive system; uses lightweight and efficient algorithms and model deployment architectures to ensure a low-latency, high-responsive real-time interactive experience.

[0059] In one embodiment of the present invention, the method further comprises:

[0060] The user senses the motor vibration through the tactile gloves or handle controller, and the motor vibration prompts the user that the gesture has been recognized. Different vibration patterns generate different feedback signals through the display of the user's XR device to distinguish different gestures.

[0061] The feedback signal generated by the motor vibration of the display of the user's XR device is used to prompt the user to start or stop acquiring the user's voice data;

[0062] The feedback signal generated by the motor vibration of the display of the user's XR device is also used to prompt the user that the second operation instruction has been executed.

[0063] Specifically, the present invention also includes a tactile feedback device (such as a vibration motor, a force feedback controller, etc.) to provide users with real-time tactile prompts and enhance the realism of the interaction. When the system successfully recognizes the user's gesture command, a short vibration can be generated by the vibration motor to prompt the user that the gesture has been recognized. Different gestures can be mapped to different vibration patterns to help users distinguish recognition results. During voice interaction, motor vibration can be used to prompt key events, such as the start and end of voice recognition, or the successful execution of voice commands. Virtual object interaction: When the user's gesture controls a virtual object, the force feedback device can provide the user with a realistic touch based on the material, weight and other properties of the object. For example, when the user grabs a virtual wooden block, the force feedback device can simulate the hardness and texture of the wood; when the user moves a virtual heavy object, the force feedback device can generate resistance to simulate the sense of weight. Virtual buttons: For some commonly used operations, such as confirmation, cancellation, etc., corresponding buttons can be set in the virtual environment. When the user's gesture triggers these buttons, the force feedback device can generate a tactile sensation similar to that of physical buttons, such as the feeling of pressing and bouncing, to enhance the realism of the operation. Tactile information can be synchronized with voice and gesture information, allowing users to obtain a similar texture to the real object when operating virtual objects, or to alert users when specific events occur, enriching the user experience from a multi-sensory dimension.

[0064] The present invention can also customize personalized interaction strategies based on the different preferences of different users for voice and gesture interactions. In addition, the present invention not only supports the interaction between a single user and a virtual environment, but also connects multiple users to form a shared virtual space. For example, user A can say "pass the ball to me", and after the system recognizes the command, it transfers the control of the ball from user B to user A; or when user A spreads out five fingers, other users can be released from controlling the current object and lock it into a public state. In collaborative interaction, differentiated permissions and task mechanisms should also be introduced to avoid operational conflicts between users. This requires setting more complex semantic rules by comprehensively considering the scene status and user identity when parsing instructions.

[0065] The present invention also provides a multi-modal instruction interaction system, please refer to Figure 6As shown, the system includes a data acquisition module 100, a data identification module 200, a strategy determination module 300 and an execution module 400. The data acquisition module 100 is used to obtain user posture data and voice data; the data identification module 200 is used to identify the posture data and obtain the first operation instruction corresponding to the posture data; the voice data is identified to obtain the user's intention data, and the corresponding second operation instruction is obtained according to the intention data; the strategy determination module 300 is used to align the timestamps of the first operation instruction and the second operation instruction, and calculate the similarity of the first operation instruction and the second operation instruction in the semantic space, and use the weighted average method to fuse the first operation instruction and the second operation instruction and dynamically adjust them to obtain the instruction with the highest priority as the final instruction; the execution module 400 is used to control the movement of objects in the virtual scene according to the final operation instruction.

[0066] In one embodiment of the present invention, the data acquisition module 100 includes an XR device, through which the virtual environment is entered, and a depth camera of the XR device is used to collect hand images and joint position data of the user in real time, or a posture sensor and a sight tracker of the XR device are used to capture the orientation angle and sight focus of the user's head;

[0067] The data recognition module 200 includes a deep learning algorithm model, which semantically classifies or categorizes the user's gestures through the deep learning algorithm model, identifies corresponding gesture instructions or head instructions, and converts the gesture instructions or head instructions into corresponding first operation instructions or third operation instructions according to a predefined mapping relationship.

[0068] In one embodiment of the present invention, the data acquisition module 100 further includes an audio acquisition device, which is used to acquire the user's voice data. The system further includes a tactile feedback device, which includes a motor or a force feedback controller, and the motor vibrates to prompt the user that the gesture has been recognized, wherein different gestures are mapped to different vibration patterns.

[0069] The above is a schematic scheme of a multimodal instruction interaction system of the present invention. It should be noted that the technical scheme of the multimodal instruction interaction system and the technical scheme of the multimodal instruction interaction method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the multimodal instruction interaction system can be found in the description of the technical scheme of the multimodal instruction interaction method mentioned above.

[0070] The multimodal command interaction system of the present invention adopts an improved deep learning model in gesture recognition and voice recognition solutions. After training and testing on a large-scale multi-scene data set, the gesture recognition accuracy rate reaches more than 95%, realizing real-time human-computer interaction. The introduced multi-channel heterogeneous sensor fusion technology effectively improves the environmental adaptability, shows good robustness in complex light and noisy environments, reduces the average failure time by 50%, and the personalized and adaptive strategy optimization significantly improves the interactive experience.

[0071] The present invention also provides a computing device, including but not limited to a memory and a processor, the processor and the memory are connected via a bus, the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multimodal instruction interaction method described in any of the above embodiments are implemented.

[0072] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned multimodal instruction interaction method.

[0073] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multimodal instruction interaction method described above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the multimodal instruction interaction method described above.

[0074] An embodiment of the present invention further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned multimodal instruction interaction method.

[0075] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the multimodal instruction interaction method described above are of the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the multimodal instruction interaction method described above.

[0076] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal instruction interaction method, characterized in that: The method comprises: Acquire user posture data, identify the posture data, and acquire a first operation instruction corresponding to the posture data; Acquire the user's voice data, identify the voice data to acquire the user's intention data, and acquire the corresponding second operation instruction according to the intention data; The first operation instruction and the second operation instruction are timestamped and calculated, and the similarity between the first operation instruction and the second operation instruction in the semantic space is calculated. The first operation instruction and the second operation instruction are merged and dynamically adjusted according to the weighted average method, and the instruction with the highest priority is obtained as the final operation instruction. Control the movement of objects in the virtual scene according to the final operation instructions.

2. The multimodal instruction interaction method according to claim 1, characterized in that: The steps of obtaining user posture data, identifying the posture data, and obtaining a first operation instruction corresponding to the posture data include: Enter the virtual environment through the XR device, and use the depth camera of the XR device to collect the user's hand image and joint position data in real time; Use deep learning algorithms to semantically classify user gestures and identify corresponding gesture commands; The gesture instruction is converted into a corresponding first operation instruction according to a predefined mapping relationship.

3. The multimodal instruction interaction method according to claim 1, characterized in that: The step of acquiring the user's voice data, identifying the voice data to acquire the user's intention data, and acquiring the corresponding second operation instruction according to the intention data comprises: Obtain the user's voice data and transcribe it into text information through the acoustic model; Perform semantic analysis on text information to obtain user intention data; The intention data is converted into a corresponding second operation instruction according to a predefined mapping relationship.

4. The multimodal instruction interaction method according to claim 1, characterized in that: The steps of obtaining user posture data, identifying the posture data, and obtaining a first operation instruction corresponding to the posture data include: Enter the virtual environment through the XR device, and use the XR device's gesture sensor and gaze tracker to capture the user's head orientation angle and gaze focus; The deep learning algorithm is used to semantically classify the user's head orientation angle and sight focus, and identify the corresponding head commands; The header instruction is converted into a corresponding third operation instruction according to a predefined mapping relationship.

5. The multimodal instruction interaction method according to claim 2, characterized in that: After the step of identifying the corresponding gesture instruction, the following step further includes: The user senses the motor vibration through the tactile gloves or handle controller, and the motor vibration prompts the user that the gesture has been recognized. Different vibration patterns generate different feedback signals through the display of the user's XR device to distinguish different gestures.

6. The multimodal instruction interaction method according to claim 5, characterized in that: The method further comprises: The feedback signal generated by the motor vibration of the display of the user's XR device is used to prompt the user to start or stop acquiring the user's voice data; The feedback signal generated by the motor vibration of the display of the user's XR device is used to prompt the user that the second operation instruction has been executed.

7. A multimodal instruction interaction system, characterized in that: The system comprises: A data acquisition module, used to acquire user posture data and voice data; A data recognition module is used to recognize the posture data and obtain a first operation instruction corresponding to the posture data; recognize the voice data to obtain the user's intention data, and obtain a corresponding second operation instruction according to the intention data; A strategy determination module is used to align the timestamps of the first operation instruction and the second operation instruction, calculate the similarity of the first operation instruction and the second operation instruction in the semantic space, fuse the first operation instruction and the second operation instruction by weighted average method and dynamically adjust them, and obtain the instruction with the highest priority as the final operation instruction; The execution module is used to control the movement of objects in the virtual scene according to the final operation instructions.

8. The multimodal instruction interaction system according to claim 7, characterized in that: The data acquisition module includes an XR device, through which the virtual environment is entered, and the user's hand image and joint point position data are collected in real time by using the depth camera of the XR device, or the orientation angle and sight focus of the user's head are captured by using the posture sensor and sight tracker of the XR device; The data recognition module includes a deep learning algorithm model, which semantically classifies or categorizes the user's gestures through the deep learning algorithm model, identifies corresponding gesture instructions or head instructions, and converts the gesture instructions or head instructions into corresponding first operation instructions or third operation instructions according to a predefined mapping relationship.

9. The multimodal instruction interaction system according to claim 8, characterized in that: The data acquisition module also includes an audio acquisition device, and the audio acquisition device is used to acquire the user's voice data.

10. The multimodal instruction interaction system according to claim 7, characterized in that: The system also includes a haptic feedback device including a motor or a force feedback controller.

11. A computing device comprising a memory and a processor, wherein the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the multimodal instruction interaction method according to any one of claims 1 to 6 are implemented.

12. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal instruction interaction method according to any one of claims 1 to 6.

Images