A mixed reality space interaction instruction parsing and distribution method

Abstract

The application discloses a mixed reality space interaction instruction analysis and distribution method, comprising the following steps: S1, collecting gesture joint posture data sequence, eye movement fixation point data sequence and voice data stream of a user; S2, constructing a multi-modal space-time graph; S3, inputting the multi-modal space-time graph into a space-time graph convolution network model, and outputting a predicted region of an intention target point in a three-dimensional space; S4, performing real-time voice recognition and semantic analysis on the voice data stream, obtaining an auxiliary semantic instruction, and generating an intention probability distribution value fused with semantics; S5, determining a unique target interaction object, and generating a space interaction instruction; and S6, calculating dynamic priority weight values of instructions in a queue. According to the application, the multi-modal space-time graph is constructed, so that the category probability value of an interaction intention and the predicted region of an intention target point can be stably output in an early stage of a user's hand stretching process, and intention skipping and false triggering caused by missing time sequence information are reduced.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically to a method for parsing and distributing interactive instructions in mixed reality space. Background Technology

[0002] In mixed reality (MR) environments, interpreting the user's interactive intent expressed through multimodal channels such as gestures, eye movements, and voice is a core prerequisite for generating correct interactive commands and ensuring smooth interaction. Interactive intent is not only determined by instantaneous actions but is also implicit in the continuous spatiotemporal process of hand and eye coordination.

[0003] For example, Chinese publication CN113253851B discloses an immersive flow field visualization human-computer interaction method based on eye tracking. This method can quickly and continuously optimize eye tracking calibration as it is used; it can also achieve eye tracking de-shaking in different states to improve the immersiveness of the interaction; and it can complete the interaction confirmation operation in human-computer interaction by relying only on eye and head rotation, thus reducing the error rate.

[0004] However, the above-mentioned interaction methods rely solely on eye-tracking channel data. When a user gazes at a virtual object but their true intention is to point at another object, the lack of gesture pointing information can easily lead to the wrong triggering of commands related to the gazed object, resulting in false triggering that goes against the user's intention. At the same time, the recognition results are easily distributed in a first-in-first-out manner, without establishing a distribution mechanism adapted to the characteristics of the intention. High-urgency commands are also blocked in the queue, making it difficult to meet the real-time and stability requirements of mixed reality interaction. Summary of the Invention

[0005] To address this, the present invention provides a method for parsing and distributing interactive commands in mixed reality space, which solves the problems in the prior art where the lack of gesture pointing information easily leads to the erroneous triggering of commands related to the gazed object; at the same time, the lack of a distribution mechanism adapted to the intent characteristics and the lack of priority control are also problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for parsing and distributing interactive instructions in mixed reality space includes the following steps:

[0008] S1: Continuously collect user gesture joint posture data sequences, eye gaze point data sequences, and voice data streams through a mixed reality head-mounted device;

[0009] S2: Based on the hand gesture joint posture data sequence and eye gaze point data sequence within each time window, construct a multimodal spatiotemporal graph including spatial hand skeleton edges, cross-modal association edges, and temporal edges;

[0010] S3: Input the multimodal spatiotemporal graph into the spatiotemporal graph convolutional network model. The spatiotemporal graph convolutional network outputs the predicted region of the intended target point in three-dimensional space through multi-layer graph convolution and temporal convolution operations.

[0011] S4: Perform real-time speech recognition and semantic parsing on the speech data stream in S1 to obtain auxiliary semantic instructions including action semantics and target object semantics; map the auxiliary semantic instructions to semantic prior probability values, and use the semantic prior probability values ​​to generate the intent probability distribution value of fused semantics based on the category probability values;

[0012] S5: Based on the intent probability distribution value and prediction region of the fused semantics, in the scene model associated with the mixed reality head-mounted device, a unique target interaction object is determined and spatial interaction instructions are generated.

[0013] S6: Place the spatial interaction command into the dynamic priority distribution queue. At each distribution time, calculate the dynamic priority weight of each command in the queue, and distribute the command to the target application in descending order of dynamic priority weight through the system interface.

[0014] Furthermore, the specific steps of S2 are as follows:

[0015] S2.1: First, collect the three-dimensional coordinates of the hand joints in a continuous T-frame hand joint posture data sequence within each time window from S1, and select the three-dimensional coordinates of T gaze points within the same time period from the eye-tracking gaze point data sequence, and define each hand joint in each frame as a first-class node.

[0016] Then, the eye-tracking fixation point within each frame is defined as a second type of node;

[0017] Finally, the first type of nodes and the second type of nodes are combined within the time window to form a node set, which in turn forms the edge of the spatial hand skeleton.

[0018] S2.2: Calculate the spatial Euclidean distance between the second type node and all first type nodes at the same time, select the K first type nodes with the smallest distance, and establish undirected edges between these K first type nodes and the second type node to obtain cross-modal association edges;

[0019] S2.3: Connect instances of the same joint node at different times along the time axis, and also connect instances of gaze points at different times to form a temporal edge;

[0020] S2.4: Combine the spatial hand skeleton edges, cross-modal association edges, and temporal edges to form a multimodal spatiotemporal graph.

[0021] Furthermore, the specific steps of S4 are as follows:

[0022] S4.1: The voice data stream acquired by S1 is used to perform real-time speech recognition using a streaming speech recognition engine, converting continuous audio signals into text sequences; and semantic parsing is performed on the text sequences to extract auxiliary semantic instructions, including action semantic slots and target object semantic slots.

[0023] S4.2: Map auxiliary semantic instructions to semantic prior probability values ;

[0024] S4.3: Calculate the intent probability distribution based on the category probability values ​​and semantic prior probability values ​​in S3. The calculation formula is as follows:

[0025]

[0026] in, Let c be the category probability value of the intent. The total number of predefined interaction intent categories; For the first The semantic prior probability value of the class intent; This is a preset semantic confidence adjustment factor; This represents the class probability distribution output by the spatiotemporal graph convolutional network in S3 after passing through a fully connected classification layer and Softmax normalization. The probability value of each intent category.

[0027] Furthermore, in step S4.2, if the action semantic slot is empty, then all intent categories are assigned a uniform distribution, and the semantic prior probability value is... =1 / .

[0028] Furthermore, the specific steps of S5 are as follows:

[0029] S5.1: Based on the predicted area The spatial filtering range is used to filter virtual objects in the scene model whose 3D center coordinates or any bounding box vertices fall within the prediction area, thus obtaining a set of candidate target objects.

[0030] S5.2: Calculate the overall matching score for each candidate object in the candidate target object set. The candidate object with the highest overall matching score is then identified as the target interaction object.

[0031] S5.3: After determining the target interaction object, generate a spatial interaction instruction including the object identifier and action type based on the current action type and the scene space information of the target interaction object.

[0032] Furthermore, the aforementioned comprehensive matching score The calculation formula is as follows:

[0033]

[0034] in, For the first The three-dimensional center coordinates of each candidate target object; To predict the center of the region With the center of the candidate object The spatial Euclidean distance between them; The intent category with the highest probability value in the semantic intent probability distribution; Candidate objects With Intent Category Normalized score of historical interaction frequency between them; The preset spatial proximity weighting coefficient; Match weight coefficients to preset intent categories.

[0035] Furthermore, the spatial interaction instruction in S5 includes an object identifier and an action type; the object identifier is a unique identifier of the target interaction object in the scene model; the action type is the operation type corresponding to the intent category with the highest probability value in the intent probability distribution value of the fused semantics.

[0036] Furthermore, the dynamic priority weight The calculation formula is as follows:

[0037]

[0038] in, The intent confidence score is taken from the maximum probability value in the intent probability distribution of the fused semantics generated by S4. This indicates the urgency of the operation type; a higher value indicates a more urgent operation. For contextual matching degree; , and These are the weighting coefficients for intent confidence, urgency of operation type, and context matching, respectively.

[0039] This invention has the following advantages: By constructing a multimodal spatiotemporal graph including spatial hand skeleton edges, cross-modal association edges, and temporal edges, and inputting the multimodal spatiotemporal graph into a spatiotemporal graph convolutional network model, the spatial graph convolutional layer aggregates the neighbor features of the first type of nodes and the second type of nodes, so that the spatial configuration of the gesture joints and the spatial position of the gaze point mutually enhance each other. The temporal convolutional layer captures the dynamic evolution pattern of the hand joint movement trajectory and the gaze point transfer process along the temporal edges, so that the whole can stably output the category probability value of the interaction intention and the prediction area of ​​the intention target point in the early stage of the user's hand reaching process, reducing the intention jump and false triggering caused by the lack of temporal information.

[0040] Meanwhile, by parsing auxiliary semantic instructions from the speech data stream, action semantic slots are mapped to semantic prior probability values, and an intent probability distribution value of fused semantics is generated. When gestures and gaze points become ambiguous due to unconscious actions, the speech instructions enhance the probability of the corresponding intent category through a semantic confidence adjustment factor. The comprehensive matching score is then calculated by combining the intent probability distribution value of fused semantics, which facilitates the determination of a unique target interaction object through dual constraints and the generation of spatial interaction instructions, ensuring the mapping from intent prediction results to specific execution objects.

[0041] In addition, the use of a dynamic priority distribution queue instead of a traditional first-in-first-out queue enables high-urgency instructions to bypass low-priority queued instructions and be distributed first, effectively solving the problem of critical instructions being blocked at the end of the queue in existing technologies.

[0042] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0043] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0044] Figure 1 This is a flowchart illustrating the implementation of a mixed reality spatial interaction command parsing and distribution method according to the present invention. Detailed Implementation

[0045] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these embodiments are merely for further explanation of the present invention and should not be construed as limiting the scope of protection of the present invention. Technical engineers in the field can make some non-essential improvements and adjustments to the present invention based on the above-described content. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figure 1 A method for parsing and distributing interactive instructions in mixed reality space, comprising the following steps:

[0047] S1: By continuously collecting user gesture joint posture data sequences, eye movement gaze point data sequences, and voice data streams through a mixed reality head-mounted device, a data foundation is provided for constructing a multimodal spatiotemporal map and extracting spatiotemporal features, ensuring the accuracy of intent parsing and the real-time performance of the system.

[0048] The mixed reality headset includes a depth camera, eye-tracking sensors, and a microphone array. The camera projects infrared structured light or time-of-flight light pulses onto the user's hand area at a preset frame rate (e.g., 60Hz) and receives reflected signals, generating depth and infrared images of the hand area. Using a built-in hand keypoint detection algorithm, the headset identifies and locates the 3D spatial positions of 21 hand joints from each frame of the depth image, outputting the 3D coordinates of the hand joints at each moment, forming a sequence of hand gesture posture data. This records the spatial configuration and movement trajectory of the user's hand over time.

[0049] The aforementioned hand keypoint detection algorithm refers to any known computer vision algorithm in this field capable of extracting the 3D coordinates of hand skeletal joints from depth or infrared images. It is primarily used to process each frame of depth image captured by a depth camera, extracting features and performing regression calculations to output the 3D spatial positions of 21 hand joints, thus forming a sequence of hand gesture pose data. This algorithm can be a direct coordinate regression model based on convolutional neural networks, such as the hand keypoint detection model in the MediaPipe Hands framework; or it can be a skeletal tracking method based on random decision forests and mean shift.

[0050] The eye-tracking sensor includes a near-infrared light source and a high-speed camera. Based on the pupil-corneal reflection method, it continuously captures the pupil center and corneal reflection spots of both eyes. It calculates the gaze direction of the left and right eyes using a gaze estimation model, and then uses the principle of binocular parallax to obtain the user's three-dimensional gaze point coordinates, thus outputting an eye-tracking gaze point data sequence. This sequence reflects the spatial location and transfer process of the user's visual attention. The aforementioned pupil-corneal reflection method is a well-known and mature eye-tracking technology in this field; its specific optical principles and image processing details will not be elaborated upon here.

[0051] A microphone array acquires multi-channel audio signals to form a speech data stream. The three data streams mentioned above are marked with a unified system timestamp at the moment of acquisition. The timestamp synchronization mechanism ensures that the gesture joint posture data, eye movement gaze point data and speech data are aligned in the time dimension.

[0052] S2: Based on the hand gesture joint pose data sequence and eye gaze point data sequence within each time window, a multimodal spatiotemporal graph is constructed, including spatial hand skeleton edges, cross-modal association edges, and temporal edges. This facilitates the representation of the spatiotemporal coordination relationship between gestures and eye movements as a graph topology, enabling the spatiotemporal graph convolutional network to simultaneously learn hand-eye spatial coupling and temporal evolution patterns.

[0053] The specific steps of S2 are as follows:

[0054] S2.1: First, collect the three-dimensional coordinates of the hand joints from a continuous T-frame hand joint pose data sequence within each time window from S1, and select the three-dimensional coordinates of T gaze points within the same time period from the eye-tracking gaze point data sequence. Define each hand joint in each frame as a first-class node, and the feature vector of this node is the three-dimensional coordinates of the joint in the current frame.

[0055] The eye-tracking gaze point within each frame is then defined as a second type of node, and its feature vector is the three-dimensional coordinates of that gaze point.

[0056] Finally, the first and second types of nodes are combined within the time window to form a node set consisting of N×T first-type nodes and 1×T second-type nodes, where N is the number of hand joints. Then, based on the skeletal connections of the human hand, two first-type nodes with skeletal connections can be connected to form spatial hand skeleton edges. This set of edges remains fixed within each frame, representing the natural topological constraints of the hand.

[0057] S2.2: For any second-type node at any given time, calculate its spatial Euclidean distance to all first-type nodes at the same time t. Select the K first-type nodes with the smallest distances (K is a preset positive integer, e.g., K=2), and establish undirected edges between these K first-type nodes and the second-type node to obtain cross-modal association edges. Cross-modal association edges are used to bind the user's current gaze point to the spatially nearest hand joint.

[0058] S2.3: Connect instances of the same joint node at different times t along the time axis, and also connect instances of gaze point nodes at different times to form temporal edges. Temporal edges preserve the motion trajectory change information of each node.

[0059] S2.4: Combine spatial hand skeleton edges, cross-modal association edges, and temporal edges to construct a multimodal spatiotemporal graph. This graph structure is used to uniformly express the spatial configuration of hand gestures, the spatial location of gaze points, the dynamic spatial coupling relationship between the two, and their respective temporal evolution laws as a computable graph data structure.

[0060] S3: Input the multimodal spatiotemporal graph into the spatiotemporal graph convolutional network model. The spatiotemporal graph convolutional network extracts gesture spatial-temporal features and eye-tracking spatial-temporal features through multi-layer graph convolution and temporal convolution operations. The fully connected layer outputs the category probability value of the user's interaction intention at the current moment, and the regression branch outputs the predicted region of the intention target point in three-dimensional space.

[0061] The spatiotemporal graph convolutional network model is composed of The spatiotemporal graph convolutional blocks are stacked sequentially, with each block consisting of a spatial graph convolutional layer and a temporal convolutional layer. After inputting the multimodal spatiotemporal graph constructed by S2, at each time step... The spatial graph convolutional layer operates on the intra-frame spatial graph composed of the first type of nodes, the second type of nodes, the spatial hand skeleton edges, and the cross-modal association edges. It updates the representation of each node by aggregating the features of its neighboring nodes, so that the spatial configuration of the gesture joints and the spatial position of the gaze point influence and reinforce each other. The temporal convolutional layer convolves the feature sequence of the same node at consecutive time points along the temporal edge, capturing the dynamic evolution pattern of the hand joint movement trajectory and the gaze point transfer process, thereby encoding the temporal rules of hand-eye coordination into the node features.

[0062] The node feature update process in the spatial graph convolutional layer described above uses an aggregation formula based on node degree normalization to update the features, resulting in the updated features. This facilitates the weighted averaging and transformation of neighboring node features, ensuring effective fusion of cross-modal information. The calculation formula is as follows:

[0063]

[0064] in, It is a non-linear activation function, such as the ReLU function; For the first Layer Time node The input feature vector; For a moment node The set of neighboring nodes includes both other first-class nodes connected by spatial hand skeleton edges and second-class nodes connected by cross-modal association edges, so that gaze point features directly participate in the update of hand joint nodes, and hand features also directly participate in the update of gaze point nodes. and node and nodes At any moment The number of neighbors of each in the intra-frame spatial graph; For the first Linear transformation weight matrix for layer learning.

[0065] After processing through multiple spatiotemporal graph convolutional blocks, global average pooling is performed on all node features output from the last layer in both the temporal and node dimensions to obtain a fixed-length spatiotemporal fusion feature vector. This feature vector is then fed into two parallel branches: the first branch is a fully connected classification layer, whose output is normalized using the Softmax function to obtain the category probability value of the user's interaction intent at the current time step. ,in The first branch is a regression branch, which outputs the predicted region parameters of the intent target point in 3D space, i.e., the coordinates of the region center, from the fully connected layer, given the predefined total number of intent categories. and region radius The predicted region is obtained. , which represents the three-dimensional spatial range in which the user's interaction intent object is most likely to fall.

[0066] S4: Perform real-time speech recognition and semantic parsing on the speech data stream in S1 to obtain auxiliary semantic commands including action semantics and target object semantics; map the auxiliary semantic commands to semantic prior probability values, and use the semantic prior probability values ​​to generate the intent probability distribution value of the fused semantics based on the category probability values. The specific steps of S4 are as follows:

[0067] S4.1: The voice data stream acquired by S1 is processed in real-time using a streaming speech recognition engine, converting continuous audio signals into text sequences. Semantic parsing is then performed on the text sequences to extract auxiliary semantic instructions, including action semantic slots and target object semantic slots. Action semantic slots indicate the type of operation the user expects to perform, such as open, move, close, delete, etc.; target object semantic slots indicate the name or category of the virtual object to which the operation is directed, such as browser, file A, window, etc.

[0068] When no valid speech is detected in the speech data stream, the action semantic slot and target object semantic slot of the auxiliary semantic instruction are both set to null.

[0069] S4.2: Map auxiliary semantic instructions to semantic prior probability values Assign a higher preset probability value (e.g., 0.9) to the intent category corresponding to the action semantic slot, and distribute the remaining probability equally among the other intent categories. (Semantic prior probability value) =(1-preset probability value); If the action semantic slot is empty, then all intent categories are assigned a uniform distribution, i.e., the semantic prior probability value. =1 / , This represents the total number of predefined interaction intent categories. It facilitates the conversion of explicit command information from the voice channel into a computable probabilistic form.

[0070] S4.3: Calculate the intent probability distribution based on the category probability values ​​and semantic prior probability values ​​in S3. The calculation formula is as follows:

[0071]

[0072] in, Let c be the category probability value of the intent. , The total number of predefined interaction intent categories; Let be the semantic prior probability value of the c-th type of intent; The preset semantic confidence adjustment factor is used when the speech signal is clear and the recognition confidence is high. Take a larger value to enhance the correction strength; when the ambient noise is high or the user is silent. Approaching 0; This represents the class probability distribution output by the spatiotemporal graph convolutional network in S3 after passing through a fully connected classification layer and Softmax normalization. The probability value of each intent category.

[0073] S5: Based on the intent probability distribution value and predicted region of the fused semantics, a unique target interaction object is determined in the scene model associated with the mixed reality head-mounted device through probability weighting and region matching, and a spatial interaction command including object identification and action type is generated. By performing dual constraint matching of the intent probability distribution value and predicted region of the fused semantics, a unique target interaction object can be accurately located and a complete command can be generated in multiple candidate scenes.

[0074] The specific steps for S5 are as follows:

[0075] S5.1: Predicting the region The spatial filtering range selects virtual objects whose 3D center coordinates or any bounding box vertices fall within the prediction region, resulting in a set of candidate target objects. If the set of candidate target objects is empty, the radius of the prediction region is gradually expanded (e.g., by 1.2 times, 1.5 times) until the candidate set is not empty or reaches a preset expansion limit.

[0076] S5.2: Calculate the overall matching score for each candidate object in the candidate target object set. The higher the interaction frequency, the higher the score. The candidate object with the highest overall matching score is then selected as the target interaction object. The calculation formula is as follows:

[0077]

[0078] in, For the first The three-dimensional center coordinates of each candidate target object; To predict the center of the region With the center of the candidate object The spatial Euclidean distance between them; The intent category with the highest probability value in the semantic intent probability distribution; Candidate objects With Intent Category The normalized score of historical interaction frequency between users is obtained by statistically analyzing the user's historical operation records, and its value ranges from [value range missing]. The more frequent the interaction, the closer the value is to 1; The preset spatial proximity weighting coefficient; Match weight coefficients to preset intent categories. .

[0079] S5.3: After determining the target interaction object, a spatial interaction instruction is generated based on the current action type and the scene space information of the target interaction object. This instruction includes an object identifier and an action type. The object identifier is a unique identifier for the target interaction object in the scene model; the action type is the operation type corresponding to the intent category with the highest probability value in the semantic intent probability distribution. This spatial interaction instruction fully describes the object, operation method, and spatial constraints of the user's intent operation, providing a precise instruction carrier for subsequent distribution and execution.

[0080] S6: Place the spatial interaction command into the dynamic priority dispatch queue. At each dispatch time, calculate the dynamic priority weight of each command in the queue. The system then distributes instructions to the target application via the system interface, prioritizing them according to their dynamic priority values ​​from highest to lowest. (Dynamic priority values) The calculation formula is as follows:

[0081]

[0082] in, The intent confidence score is taken from the maximum probability value in the intent probability distribution of the fused semantics generated by S4, and its value range is [0,1]. The urgency level is determined by looking up the action type of the current instruction in a preset action type-urgency mapping table, and its value range is [value range missing]. A larger value indicates a more urgent operation; Context matching degree represents the degree of coherence between the current instruction and the user's historical operation sequence and the current scene state. The larger the value, the better the match with the current context. , and These are the weighting coefficients for intent confidence, urgency of operation type, and context matching, respectively.

[0083] The above action type-urgency mapping table is a preset static mapping table, which is constructed based on the security risk level and reversibility of interactive operations.

[0084] For example, for irreversible or security-related actions such as deletion and stopping, the default operation type urgency is close to 1.0; for regular reversible operations such as moving and selecting, the default operation type urgency is 0.3-0.5. Entering the action type during the query will directly return the corresponding result. value.

[0085] Context matching The calculation formula is as follows:

[0086]

[0087] in, This is the consistency weighting coefficient for historical behavior; This represents the scene state compatibility weight coefficient. ; The action type carried by the current spatial interaction command is the intent category with the highest probability value in the intent probability distribution of the fused semantics in S5. The action type of the most recently executed spatial interaction command in the user's historical operation sequence; The historical action transition consistency score, representing the reasonableness of transitioning from the previous action type to the current action type, is obtained by looking up a table from the action transition probability matrix pre-calculated based on user operation logs, and its value range is [value range missing]. ; The object identifier of the target interaction object pointed to by the current spatial interaction command, that is, the identifier of the target interaction object determined in S5; The scene state compatibility score represents the appropriateness of the current action type acting on the current target interaction object. It is obtained by looking up a preset action-object compatibility mapping table, and its value range is [value range missing]. .

[0088] The above action transition probability matrix was obtained by offline statistical analysis of action sequence logs from multiple users in standard operation tasks. Matrix elements represent actions. Then an action occurred The empirical probability. The action type of the spatial interaction command during the query. The row index and the action type carried by the current space interaction command. The historical action transfer consistency score is obtained by looking up the column index in the table; if the current instruction is the first one, it is directly set to 1.

[0089] The action-object compatibility mapping table is based on preset rules for matching object semantic tags with action functions. For example, the compatibility preset for "open" with the "interactive document" tag is 1.0, and with "static background" it is 0.0; the preset for "move" with "movable object" is 0.9. When querying, input the action type and the semantic tag of the target object, and the table will be consulted to obtain the scene state compatibility score.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for parsing and distributing interactive instructions in mixed reality space, characterized in that, Includes the following steps: S1: Continuously collect user gesture joint posture data sequences, eye gaze point data sequences, and voice data streams through a mixed reality head-mounted device; S2: Based on the hand gesture joint posture data sequence and eye gaze point data sequence within each time window, construct a multimodal spatiotemporal graph including spatial hand skeleton edges, cross-modal association edges, and temporal edges; S3: Input the multimodal spatiotemporal graph into the spatiotemporal graph convolutional network model. The spatiotemporal graph convolutional network outputs the predicted region of the intended target point in three-dimensional space through multi-layer graph convolution and temporal convolution operations. S4: Perform real-time speech recognition and semantic parsing on the speech data stream in S1 to obtain auxiliary semantic instructions including action semantics and target object semantics; map the auxiliary semantic instructions to semantic prior probability values, and use the semantic prior probability values ​​to generate the intent probability distribution value of fused semantics based on the category probability values; S5: Based on the intent probability distribution value and prediction region of the fused semantics, in the scene model associated with the mixed reality head-mounted device, a unique target interaction object is determined and spatial interaction instructions are generated. S6: Place the spatial interaction command into the dynamic priority distribution queue. At each distribution time, calculate the dynamic priority weight of each command in the queue, and distribute the command to the target application in descending order of dynamic priority weight through the system interface.

2. The method for parsing and distributing interactive instructions in mixed reality space according to claim 1, characterized in that, The specific steps of S2 are as follows: S2.1: First, collect the three-dimensional coordinates of the hand joints in a continuous T-frame hand joint posture data sequence within each time window from S1, and select the three-dimensional coordinates of T gaze points within the same time period from the eye-tracking gaze point data sequence. Define each hand joint in each frame as a first-class node. Then, the eye-tracking fixation point within each frame is defined as a second type of node; Finally, the first type of nodes and the second type of nodes are combined within the time window to form a node set, which in turn forms the edge of the spatial hand skeleton. S2.2: Calculate the spatial Euclidean distance between the second type node and all first type nodes at the same time, select the K first type nodes with the smallest distance, and establish undirected edges between these K first type nodes and the second type node to obtain cross-modal association edges; S2.3: Connect instances of the same joint node at different times along the time axis, and also connect instances of gaze points at different times to form a temporal edge; S2.4: Combine the spatial hand skeleton edges, cross-modal association edges, and temporal edges to form a multimodal spatiotemporal graph.

3. The method for parsing and distributing interactive instructions in mixed reality space according to claim 1, characterized in that, The specific steps of S4 are as follows: S4.1: The voice data stream acquired by S1 is used to perform real-time speech recognition using a streaming speech recognition engine, converting continuous audio signals into text sequences; and semantic parsing is performed on the text sequences to extract auxiliary semantic instructions, including action semantic slots and target object semantic slots. S4.2: Map auxiliary semantic instructions to semantic prior probability values ; S4.3: Calculate the intent probability distribution based on the category probability values ​​and semantic prior probability values ​​in S3. The calculation formula is as follows: in, Let c be the category probability value of the intent. The total number of predefined interaction intent categories; For the first The semantic prior probability value of the class intent; This is a preset semantic confidence adjustment factor; This represents the class probability distribution output by the spatiotemporal graph convolutional network in S3 after passing through a fully connected classification layer and Softmax normalization. The probability value of each intent category.

4. The method for parsing and distributing interactive instructions in mixed reality space according to claim 3, characterized in that, If the action semantic slot is empty in S4.2, then all intent categories are assigned a uniform distribution, and the semantic prior probability value is... =1 / .

5. The method for parsing and distributing interactive instructions in mixed reality space according to claim 1, characterized in that, The specific steps of S5 are as follows: S5.1: Based on the predicted area The spatial filtering range is used to filter virtual objects in the scene model whose 3D center coordinates or any bounding box vertices fall within the prediction area, thus obtaining a set of candidate target objects. S5.2: Calculate the overall matching score for each candidate object in the candidate target object set. The candidate object with the highest overall matching score is then identified as the target interaction object. S5.3: After determining the target interaction object, generate a spatial interaction instruction including the object identifier and action type based on the current action type and the scene space information of the target interaction object.

6. The method for parsing and distributing interactive instructions in mixed reality space according to claim 5, characterized in that, The comprehensive matching score The calculation formula is as follows: in, For the first The three-dimensional center coordinates of each candidate target object; To predict the center of the region With the center of the candidate object The spatial Euclidean distance between them; The intent category with the highest probability value in the semantic intent probability distribution; Candidate objects With Intent Category Normalized score of historical interaction frequency between them; The preset spatial proximity weighting coefficient; Match weight coefficients to preset intent categories.

7. The method for parsing and distributing mixed reality spatial interaction commands according to claim 1, characterized in that, The spatial interaction instructions in S5 include an object identifier and an action type; the object identifier is a unique identifier of the target interaction object in the scene model; the action type is the operation type corresponding to the intent category with the highest probability value in the fused semantic intent probability distribution value.

8. The method for parsing and distributing interactive instructions in mixed reality space according to claim 1, characterized in that, The dynamic priority weight The calculation formula is as follows: in, The intent confidence score is taken from the maximum probability value in the intent probability distribution of the fused semantics generated by S4. This indicates the urgency of the operation type; a higher value indicates a more urgent operation. For contextual matching degree; , and These are the weighting coefficients for intent confidence, urgency of operation type, and context matching, respectively.

Images