XR interaction method, device and equipment based on voice control

Abstract

The invention provides an XR interaction method, device and equipment based on voice control, and belongs to the technical field of human-computer interaction. The method comprises the following steps: receiving a multi-modal interaction data stream acquired by a multi-modal sensor; recognizing and analyzing a voice signal in the multi-modal interaction data stream to determine a corresponding voice control instruction; wherein the voice control instruction comprises an operation intention and space description; the spatial description is spatial constraint information of the user on the target operation object; determining a corresponding target interaction space area based on the voice control instruction and one or more non-voice behavior information synchronized with the voice signal in the multi-mode interaction data stream; and matching a target operation object corresponding to the voice control instruction according to the information of each operable object in the target interaction space region, and executing an XR interaction operation corresponding to the operation intention according to the target operation object. Therefore, high-precision, high-dynamic-adaptability and high-reliability voice control interaction is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to an XR interaction method and device based on voice control and equipment. BACKGROUND

[0002] At present, there are still significant limitations in the human-computer interaction mode of extended reality (XR) equipment: the mainstream interaction relies on gestures, eye movement tracking or physical controllers, and in the case of user movement or complex environment, the accuracy, convenience and robustness of these modes are greatly reduced. Traditional voice control technology has more prominent problems: on the one hand, only relying on voice signal analysis to determine the operation intent lacks deep perception of user behavior and environmental context, making it difficult to accurately identify specific interface elements of user intent operation, resulting in frequent misoperations. On the other hand, when the user moves in the XR environment, the user's own coordinate system (a space reference system centered on the user) and the XR device interface coordinate system (a global space reference system defined by the device) change dynamically, so that spatial descriptions such as "front" and "right side" in the voice instruction cannot be accurately mapped to the device display space, causing coordinate drift, which seriously affects the stability and continuity of control. SUMMARY

[0003] The embodiments of the present application provide an XR interaction method, device and equipment based on voice control, to solve the technical problem of how to realize high-precision, dynamically adaptive and high-reliability voice control interaction in an XR environment.

[0004] In one aspect, the embodiments of the present application provide an XR interaction method based on voice control, which comprises: receiving a multi-modal interaction data stream obtained by a multi-modal sensor; identifying and analyzing a voice signal in the multi-modal interaction data stream to determine a corresponding voice control instruction; wherein the voice control instruction includes an operation intent and a spatial description; the spatial description is spatial constraint information of a target operation object by a user; determining a corresponding target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multi-modal interaction data stream synchronized with the voice signal; According to the information of each operable object in the target interaction space region, matching the target operation object corresponding to the voice control instruction, and performing an XR interaction operation corresponding to the operation intent according to the target operation object.

[0005] In an implementation form of the application, the target interaction space region is determined based on the voice control instruction and one or more non-voice behavior information in the multi-modal interaction data stream synchronized with the voice signal, specifically comprising: The non-voice behavior information is converted into a behavior feature vector in a three-dimensional space; wherein the non-voice behavior information at least includes one or more of the following: head orientation information, line-of-sight direction information, hand pointing information; The space description in the voice control instruction is converted into a space constraint condition; the space constraint condition includes a constraint rule parameter group representing the space constraint information; Based on the fusion calculation result of the behavior feature vector and the space constraint condition, the target interaction space region boundary in the three-dimensional space coordinate system is determined to obtain the target interaction space region according to the target interaction space region boundary.

[0006] In an implementation form of the application, based on the fusion calculation result of the behavior feature vector and the space constraint condition, the target interaction space region boundary in the three-dimensional space coordinate system is determined, specifically comprising: The preset fusion weight group, the behavior feature vector and the reference vector corresponding to the space constraint condition are weighted calculated to generate a target direction vector after fusion calculation; The target direction vector corresponding to the direction is taken as a reference to generate the target interaction space region boundary according to the space constraint condition; wherein the space constraint condition at least includes: distance constraint parameter, range constraint parameter.

[0007] In an implementation form of the application, before the preset fusion weight group, the behavior feature vector and the reference vector corresponding to the space constraint condition are weighted calculated to generate a target direction vector after fusion calculation, the method further comprises: Based on the motion state detection information from the multi-modal sensor, the motion state of the user is determined; the motion state includes a static state and a moving state; When the motion state is a static state, a first preset fusion weight group is determined according to the information type of the non-voice behavior information, and weighted calculation is performed according to the first preset fusion weight group; When the motion state is a moving state, a user motion intensity value is determined according to the motion state detection information, and the user motion intensity value is matched with a plurality of preset intensity threshold intervals; According to the preset intensity threshold interval where the user motion intensity value is located, a corresponding dynamic weight adjustment factor is determined, and a reference preset fusion weight group is loaded; According to the dynamic weight adjustment factor, the base preset fusion weight set is corrected to obtain a corresponding second preset fusion weight set, and a corresponding weighted calculation is performed.

[0008] In an implementation manner of the present application, after determining the motion state of the user, the method further comprises: In a case where the motion state is a moving state, a predicted user pose is determined based on a preset pose prediction model and the current motion state detection information; According to the pose information and the position information corresponding to the predicted user pose, a dynamic mapping matrix from the user's own coordinate system to the XR device interface coordinate system is constructed; the pose information includes rotation parameters, and the position information includes translation parameters; According to the dynamic mapping matrix, the space description is converted into coordinates in the XR device interface coordinate system corresponding to the predicted user pose, and the converted coordinates are updated to the space constraint condition.

[0009] In an implementation manner of the present application, the target operation object corresponding to the voice control instruction is matched according to the information of each operable object in the target interaction space region, specifically comprising: According to the information of each operable object, the spatial coordinates and the functional attributes of each operable object are determined; the operable object is an XR device interface element; According to the spatial coordinates, the functional attributes and the voice control instruction, the matching degrees of each operable object and the voice control instruction are calculated; the matching degrees at least include spatial coordinate matching degrees and functional attribute matching degrees; Each matching degree is compared with a preset matching degree threshold value, so as to determine the target operation object according to the comparison result.

[0010] In an implementation manner of the present application, the target operation object is determined according to the comparison result, specifically comprising: In a case where there are multiple undetermined operable objects according to the comparison result, each undetermined operable object is added to an XR interface display list and displayed to the user; Voice prompt information is played so that the user selects a display label of each undetermined operable object in the XR interface display list; The target operation object is determined based on a secondary voice instruction of the user.

[0011] In an implementation manner of the present application, before the XR interaction operation corresponding to the operation intention is performed according to the target operation object, the method further comprises: generating a visual marking signal of the target operation object; wherein the visual marking signal at least includes one or more of the following: a highlighted border, dynamic flashing, color gradient; after receiving the confirmation instruction of the user, performing the XR interaction operation; wherein the confirmation instruction at least includes: a voice confirmation instruction, a preset gesture instruction or a preset action instruction.

[0012] In a second aspect, the embodiments of the present application further provide an XR interaction device based on voice control, which can execute the XR interaction method based on voice control described above; the device comprises: a receiving module, configured to receive a multi-modal interaction data stream acquired by a multi-modal sensor; a first determining module, configured to identify and analyze a voice signal in the multi-modal interaction data stream to determine a corresponding voice control instruction; wherein the voice control instruction includes an operation intention and a space description; the space description is space constraint information of a target operation object by a user; a second determining module, configured to determine a corresponding target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multi-modal interaction data stream synchronized with the voice signal; a matching and executing module, configured to match the target operation object corresponding to the voice control instruction according to each operation object information in the target interaction space region, and execute an XR interaction operation corresponding to the operation intention according to the target operation object.

[0013] In a third aspect, the embodiments of the present application further provide an XR interaction device based on voice control, which comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the XR interaction method based on voice control described above.

[0014] Compared with the prior art, the present application has the following significant effects: 1) Through the technical scheme, the XR interaction control scheme based on voice control is provided, multi-modal interaction data streams are fused and processed, a target interaction space region determined in combination with voice signals and non-voice behavior information is obtained, the limitation that traditional voice control only relies on semantic analysis is broken, misoperation caused by ambiguous space description is greatly reduced, and accurate identification of a target operation object is realized. Meanwhile, the target interaction space region obtained by fusing user behavior information and space description in the voice control instruction is dynamically adapted to the relative change of the user's own coordinate system and the XR device interface coordinate system, the coordinate drift problem when the user moves is effectively avoided, and it is ensured that even in dynamic scenes such as walking and turning the head, the voice instruction can still be stably mapped to the target position of the device display space, and the continuity and reliability of the control are significantly improved.

[0015] 2) Through the cooperative processing of multi-modal information, the user can interact with the XR device through the intuitive way of "voice + natural behavior", without relying on an additional controller, the application range of the XR device in a mobile scene is expanded, and the smoothness of the user experience is improved. SUMMARY

[0016] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and a description thereof serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a schematic diagram of an application scenario of an XR interaction method based on voice control in an embodiment of the present application; Figure 2 It is a flowchart of an XR interaction method based on voice control in an embodiment of the present application; Figure 3 It is a flowchart of a method for determining a target interaction space region in an embodiment of the present application; Figure 4 It is a flowchart of a method for determining a dynamic weight adjustment factor and performing weight correction in an embodiment of the present application; Figure 5 It is a flowchart of a method for updating a space constraint condition in an embodiment of the present application; Figure 6 It is a work flowchart of performing an XR device operation in an embodiment of the present application; Figure 7 It is a structural schematic diagram of an XR interaction device based on voice control in an embodiment of the present application; Figure 8 It is a structural schematic diagram of an XR interaction device based on voice control in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0018] The human-computer interaction mode of the XR device mainly relies on gestures, eye movement tracking or physical controllers. These interaction means perform well in specific scenarios, but their accuracy, convenience and robustness still face challenges in user movement or complex environments. In particular, traditional voice control technology often fails to accurately identify specific interface elements of user intent operations due to the lack of deep perception of user behavior and environmental context, which easily leads to misoperation. In addition, when the user moves in the XR environment, the dynamic changes between the device coordinate system and the user's own coordinate system make it difficult to accurately map the spatial description contained in the voice instruction, thereby affecting the stability and continuity of control.

[0019] Based on this, the embodiments of the present application provide a voice control based XR interaction method, device and equipment to solve the technical problem of how to realize high-precision, dynamically adaptive and high-reliability voice control interaction in the XR environment.

[0020] The various embodiments of the present application will be described in detail below in connection with the drawings.

[0021] The embodiments of the present application provide a voice control based XR interaction method, which can be applied in an application environment as shown in Figure 1 The XR device of the present application can be set as a virtual reality (VR) glasses, augmented reality (AR) glasses, mixed reality (MR) glasses and the like, for the convenience of description, in the embodiments of the present application, the XR device is taken as an XR smart glasses for example to explain and describe. As shown in Figure 1As shown, the application environment can include: smart glasses 101 worn by the user 107, a communication network 102, a server 103, a voice semantic processing server 104, a multi-modal behavior data processing server 105, and a database 106. The communication network 102 can serve as a channel for data transmission, providing a communication link for communication between the smart glasses 101 and the server 103, so that the smart glasses 101 can transmit the multi-modal interaction data stream collected by the smart glasses 101 to the server 103 based on the communication network 102, and also can feed back the information processed by the server 103 to the smart glasses 101 for rendering display. The server 103 is a server that can provide various services, and in order to improve processing efficiency, the server 103 is connected with the voice semantic processing server 104 and the multi-modal behavior data processing server 105, so that after receiving the information collected by the smart glasses 101 based on the communication network 102, the voice signal recognized can be sent to the voice semantic processing server 104 for processing; the non-voice behavior information can be sent to the multi-modal behavior data processing server 105 for processing. After processing, the server 103 determines the target operation object in combination with the voice signal detected by the smart glasses 101, and performs rendering display related processing on the target operation object, and finally feeds back to the smart glasses 101 based on the communication network 102 to present to the user.

[0022] The database 106 is connected with the voice semantic processing server 104 and the multi-modal behavior data processing server 105, respectively, and can use, store and manage the voice signal data of the voice semantic processing server 104 and the behavior data of the multi-modal behavior data processing server 105. The database 106 can be integrated on the server, or can be placed on the cloud or other network servers.

[0023] The following describes in detail a voice control based XR interaction method provided by an embodiment of the present application. Figure 2 A flowchart of a voice control based XR interaction method provided by an embodiment of the present application is shown in Figure 2 As shown, the method can include steps S201-S204: S201, receiving multi-modal interaction data stream acquired by a multi-modal sensor.

[0024] A cloud server is set, which can process corresponding multi-modal data, and store voice control instructions, target interaction space regions, operable object information, preset fusion weight sets, preset pose prediction models and other related data through a database, and can interact with XR glasses to realize corresponding data transmission and processing functions.

[0025] The multi-modal sensor is a combination of hardware capable of synchronously collecting multiple types of interaction information, and the multi-modal interaction data stream is a time-synchronized data set containing dimensions of voice, behavior, motion, etc. The multi-modal sensor can be set inside the smart glasses or establish an external electrical connection with the smart glasses, including but not limited to an accelerometer, a gyroscope, a visual odometry, a visual sensor, an eye tracking sensor, a sound sensor, etc. The specific sensor type and quantity can be set during actual use, and are not specifically limited here.

[0026] The multi-modal sensor collects the user's voice, vision (head orientation, line of sight direction), motion (displacement, acceleration, angular velocity), and gesture information in real time and in parallel, continuously, obtains multiple interaction data streams, and performs time synchronization processing on all data streams to ensure the accuracy of information fusion.

[0027] S202, recognizing and analyzing the voice signal in the multi-modal interaction data stream to determine the corresponding voice control instruction.

[0028] The voice control instruction includes an operation intent and a spatial description. The spatial description is the user's spatial constraint information for the target operation object.

[0029] The voice signal in the multi-modal interaction data stream can be recognized and analyzed by a voice processing module composed of an automatic speech recognition (ASR) unit and a natural language understanding (NLU). The ASR converts the voice into text, and the NLU extracts the core operation intent (such as "open", "select", "close", "adjust size") and any explicit spatial description (such as "front", "right side", "this", "that", "third") from the text. For example, when the user says "open the second application on the right", the system will identify the operation intent as "open" and the spatial description as "the second on the right", and the target type as "application".

[0030] The above space description is the space constraint information of the target operation object directly expressed by the user through the voice instruction, which is the explicit limitation of the user on the spatial attributes such as the position and range of the target object based on the self-cognition. For example, “1 meter on the right side” is the space limitation directly expressed by the user through the voice, which includes the direction constraint “right side” (limiting the target to the right side of the user) and the distance constraint “1 meter (m)” (limiting the distance range of the target from the user). Meanwhile, the application can also quantify the space description and convert the space constraint information directly expressed by the user into machine-identifiable parameters, for example, “right side” corresponds to the positive direction of the Y-axis of the user coordinate system (perpendicular to the right of the head orientation), and “1 meter” corresponds to the distance constraint parameter d = 1.0 m (allowing an error of ± 0.2 m, which reflects the user's tolerance range for distance accuracy).

[0031] In addition, the application also sets an instruction classifier, which can preliminarily classify the above obtained voice control instruction to quickly execute the corresponding control operation.

[0032] S203, determining a corresponding target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multi-modal interaction data stream synchronized with the voice signal.

[0033] The application also analyzes the non-voice behavior information synchronized with the voice signal while determining the voice control instruction, and determines the target interaction space region in combination with the voice control instruction. Thus, in combination with the voice control instruction pointing to the XR device interface coordinate system and the non-voice behavior information directly associated with the state of the user's own coordinate system, the voice control instruction and the non-voice control instruction are synchronously fused to construct the target interaction space region, thereby realizing the dynamic association of the user's own coordinate system and the XR device interface coordinate system to realize the stability and continuity of the interaction control.

[0034] In the embodiment of the application, the above determination of the corresponding target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multi-modal interaction data stream synchronized with the voice signal includes the following steps: Figure 3 as shown, specifically comprising the following steps: S301, converting the non-voice behavior information into a behavior feature vector in a three-dimensional space. The non-voice behavior information includes at least one or more of the following: head orientation information, line-of-sight direction information, and hand pointing information.

[0035] S302, converting the space description in the voice control instruction into a space constraint condition. The space constraint condition includes a constraint rule parameter group representing the space constraint information.

[0036] S303, based on the fusion calculation result of the behavior feature vector and the space constraint condition, determine the target interaction space region boundary in the three-dimensional space coordinate system, to obtain the target interaction space region according to the target interaction space region boundary.

[0037] Specifically, the application can obtain the head posture data of the user in real time through the inertial measurement unit (IMU) and the visual sensor built in the intelligent glasses, obtain the head orientation information, and convert it into a head orientation vector in the XR device interface coordinate system. Using the eye tracking sensor, the eye movement of the user is accurately captured to obtain the gaze direction information, and the gaze direction vector of the user in the XR device interface coordinate system is calculated. This usually involves analysis and geometric mapping of eye images. Through the depth sensor or gesture recognition algorithm in the server, the key points of the user's hand are detected in real time, and the vector of the user's hand pointing is calculated according to the gesture (such as pointing with the index finger). For non-directional gestures (such as grabbing, pinching), they are used as auxiliary confirmation or operation instructions. After obtaining the vector corresponding to one or more non-speech behavior information, the application constructs a behavior feature vector. That is, the non-speech behavior information is obtained based on the user's body movement, eye movement and body posture features.

[0038] Further, the space description is converted into a space constraint condition to generate a constraint rule parameter group, which directly corresponds to the space constraint information in the user's voice and can include: direction constraint: based on the user's "right side" to determine the reference direction vector S=(0, 1, 0); Distance constraint: based on the user's "1 meter" to set d=1.0m (allowing an error of ±0.2m to match the user's habit of ambiguous expression of distance); Range constraint: combine common interaction scenarios to quantify the user's implicit "nearby area" as a cubic region (length x width x height=0.5m x 0.5m x 0.5m) centered on the target point.

[0039] Subsequently, the behavior feature vector and the space constraint condition are fused to obtain a fusion calculation result. The fusion calculation can use weighted average, Bayesian network, or attention mechanism based on deep learning, and the present application does not make specific limitations. After fusion calculation, one or more target space interaction regions can be defined in the XR virtual space. For example, if the voice instruction is "open this", the system will give priority to the area where the user's visual focus is located; if the instruction is "right", a fan-shaped or conical region is determined in the user's right field of view in combination with the head orientation and the visual direction. If the voice instruction contains a mention of a specific type of element (such as "open application" or "select file"), the range of the region is further narrowed down to only consider the type of interactive elements. This multi-modal fusion mechanism significantly improves the context understanding ability of voice instructions and the accuracy of target positioning, effectively avoiding misrecognition caused by semantic ambiguity or complex environment in traditional voice control.

[0040] Based on the above scheme, the present application fuses voice and non-voice behavior information to calculate, constructs a target interaction space region relying on data constructed in two different dimensions, and the two complement each other to avoid relying on voice or behavior information alone during interaction.

[0041] Further, in some embodiments of the present application, the fusion calculation result based on the behavior feature vector and the space constraint condition determines the boundary of the target interaction space region in the three-dimensional space coordinate system, specifically including: The preset fusion weight set, the behavior feature vector, and the reference vector corresponding to the space constraint condition are weighted to generate a target direction vector after fusion calculation (i.e., the fusion calculation result). The direction corresponding to the target direction vector is taken as the reference, and the target interaction space region boundary is generated according to the space constraint condition. The space constraint condition at least includes a distance constraint parameter and a range constraint parameter.

[0042] In other words, the present application presets the fusion weight set, that is, the initial weights can be assigned to different modalities, such as a speech semantic weight 0.4, a line-of-sight direction weight 0.3, a head orientation weight 0.2, and a hand pointing weight 0.1. The head orientation vector H (x1, y1, z1) in the behavior feature vector, the line-of-sight direction vector E (x2, y2, z2), and G (x3, y3, z3) are respectively multiplied by the corresponding line-of-sight direction weight 0.3, the head orientation weight 0.2, and the hand pointing weight 0.1 to obtain a first parameter; a reference vector is extracted according to the spatial constraint condition, which can include its {direction angle range, distance threshold, region shape parameter}, and the product value of the reference vector and the speech semantic weight 0.4 is calculated to obtain a second parameter; the sum of the first parameter and the second parameter is taken as the target direction vector after fusion calculation. Subsequently, the target direction vector is taken as a reference to determine the target interaction space region boundary in combination with the direction constraint, distance constraint, and range constraint in the spatial constraint condition. For example, a conical region with the target direction vector corresponding direction as the center, an angle range of ±20°, and a distance of 20-50 cm from the user is taken as the target interaction space region boundary.

[0043] Further, the behavior feature vector and the corresponding spatial constraint condition of the speech are flexibly fused to demarcate the target interaction space region boundary, so as to realize accurate positioning in the three-dimensional space.

[0044] In addition, in some embodiments of the present application, before the preset fusion weight set, the behavior feature vector, and the reference vector corresponding to the spatial constraint condition are weighted and calculated to generate the target direction vector after fusion calculation, as shown in Figure 4 The present application also provides the following embodiments, and the specific steps are as follows: S401, determining the motion state of the user based on the motion state detection information from the multi-modal sensor. The motion state includes a static state and a moving state; S402, when the motion state is the static state, determining a first preset fusion weight set according to the information type of the non-speech behavior information, and performing weighted calculation according to the first preset fusion weight set; S403, when the motion state is the moving state, determining a user motion intensity value according to the motion state detection information, and matching the user motion intensity value with a plurality of preset intensity threshold intervals; S404, determining a corresponding dynamic weight adjustment factor according to the preset intensity threshold interval in which the user motion intensity value is located, and loading a reference preset fusion weight set; S405, correcting the reference preset fusion weight set according to the dynamic weight adjustment factor to obtain a corresponding second preset fusion weight set, and performing corresponding weighted calculation.

[0045] In other words, the application will further determine whether the user is in the process of motion, which can specifically continuously obtain the motion state detection information of the user through the IMU data, and determine whether the user is in a stationary, uniform motion, accelerated motion, rotation, etc. state by analyzing the amplitude and rate of change of acceleration and angular velocity. For example, when the acceleration and angular velocity are both lower than a preset threshold value, it is determined to be a stationary state; otherwise, it is a moving state. The preset threshold value can be set by the user or the developer according to expert experience, which is not specifically limited here. When it is detected that the user is in a stationary state, the application can use the above-mentioned default initial values as the first preset fusion weight set, i.e., the voice semantic weight 0.4, the line of sight direction weight 0.3, the head orientation weight 0.2, and the hand pointing weight 0.1, to execute the above-mentioned S303. When it is detected that the user is in a moving state, the application will start the adaptive attention adjustment mechanism, intelligently dynamically adjust the weights of each modality according to the user motion intensity, and effectively avoid misoperations caused by unstable sensor data or user unconscious actions. The modality weights dynamically adjusted according to the user motion intensity are located in the baseline preset fusion weight set, and the weights in the baseline preset fusion weight set can be the same as the weight values in the above-mentioned first preset fusion weight set, or can be set by the user or the developer based on expert experience in actual scenarios, which is not specifically limited here.

[0046] Specifically, the application can use real-time acceleration amplitude as the user motion intensity value, or use other data as the user motion intensity value, such as combining real-time acceleration amplitude, speed, heart rate, etc. to calculate the user motion intensity value by weighting, which can be specifically set by the user or the developer according to expert experience, which is not specifically limited here. Taking real-time acceleration amplitude as the user motion intensity value as an example, the real-time acceleration amplitude is defined as accel_magnitude, and multiple motion intensity threshold values are set, such as: Threshold_low_accel (light motion threshold value, such as 1.5 m / s 2 ): below this value, the user is considered to be in a relatively stable state. Threshold_high_accel (intense motion threshold value, such as 4.0 m / s 2 ): above this value, the user is considered to be in an intense motion state. The preset intensity threshold value interval includes: (0, Threshold_low_accel), [Threshold_low_accel, Threshold_high_accel], (Threshold_high_accel, Threshold_high_accel2). Threshold_high_accel2 is an upper limit of the intensity threshold value set by the user or the developer, which can be obtained from expert experience, which is not specifically limited here.

[0047] In the determination of the dynamic weight adjustment factor adjust ment_factor, the specific strategy can be: 1. When accel_magnitude is in the first interval (0, Threshold_low_accel), adjust ment_factor is 1.0, indicating no adjustment, and the weight of the non-speech behavior mode remains the original value; 2. When accel_magnitude is in the second interval [Threshold_low_accel, Threshold_high_accel], adjust ment_factor is linearly interpolated, and decreases linearly with the increase of the user motion intensity value. For example, adjust ment_factor = 1.0 - 0.8 * (accel_magnitude - Threshold_low_accel) / (Threshold_high_accel - Threshold_low_accel); 3. When accel_magnitude is in the third interval (Threshold_high_accel, Threshold_high_accel2), adjust ment_factor is 0.2 (or lower), indicating a significant reduction in the non-speech behavior mode, at which point more reliance will be placed on speech semantic information.

[0048] For the adjustment strategy of the dynamic weight adjustment factor, the above is only exemplary, and the specific adjustment strategy can be set by the user or the developer according to expert experience or the scene, and is not specifically limited here.

[0049] After obtaining the dynamic weight adjustment factor, the behavior-related weights in the baseline preset fusion weight set can be multiplied by the dynamic weight adjustment factor to obtain the adjusted behavior weights, and at the same time, the weight of speech semantics will be correspondingly increased to compensate for the part of the reduced behavior weight, ensuring that the total weight of all modes remains unchanged (for example, ensuring that the sum of the weights in the preset fusion weight set is 1), obtaining a second preset fusion weight set. Thus, the part of the reduced behavior weight is allocated to the speech semantic weight, so that it occupies a dominant position when the user is in vigorous motion. The adjusted weight is applied to the above S303 step in real time.

[0050] In addition, when the accel_magnitude is greater than the Threshold_high_accel2, at this time the user can be in an extreme state of vigorous exercise, the dependence on accurate spatial positioning can be temporarily reduced, or the user can be prompted visually / audibly that "your current exercise amplitude is large, please try voice control after stabilizing a little", or the voice control function can be temporarily suspended to avoid misoperation caused by sensor data jitter or user unconscious action. Through the above adaptive attention adjustment mechanism, the application ensures that the system can maintain high robustness and stability in various complex motion scenarios, and provides a safer and more reliable interactive experience for the user.

[0051] Through the above scheme, the weighted fusion algorithm can dynamically adjust the weight of the behavior feature vector (such as reducing the hand pointing weight when exercising vigorously), solve the problem of user motion state interference with sensor data (such as pointing deviation caused by hand shaking when running), and dynamically adapt the weight to ensure that the system can maintain stable performance in static, moving, and vigorous motion scenarios, avoid misoperation caused by environmental interference, and enhance system robustness.

[0052] In some embodiments of the application, the user is in a relatively vigorous exercise process, which can make the coordinate drift more serious, so after determining the motion state of the user, as shown in Figure 5 The application also provides the following embodiments, specifically including steps S501-S503: S501, in the case that the motion state is a moving state, determining a predicted user pose based on a preset pose prediction model and current motion state detection information; S502, constructing a dynamic mapping matrix from the user's own coordinate system to the XR device interface coordinate system according to the attitude information and the position information corresponding to the predicted user pose; the attitude information includes rotation parameters, and the position information includes translation parameters; S503, converting the space description into coordinates in the XR device interface coordinate system corresponding to the predicted user pose according to the dynamic mapping matrix, and updating the converted coordinates to the spatial constraint condition.

[0053] That is, when the user moves, the application uses a preset pose prediction model to predict the user's pose, and constructs a dynamic mapping matrix that can map coordinates, so as to ensure that the spatial description expressed by the user is accurately mapped in the XR device interface coordinate system, and avoid coordinate drift.

[0054] In a voice instruction processing process, the application can continuously detect the motion state of the user in real time, including displacement speed, acceleration and other parameters. Based on these motion data, when it is detected that the user is in a moving state, the system starts a preset pose prediction model to predict the spatial pose (position and attitude) of the user in a very short future time based on the accurate pose of the user at the current time (provided by Simultaneous Localization and Mapping (SLAM) or Visual-Inertial Odometry (VIO)), real-time speed, acceleration and angular velocity and other kinematic parameters, to obtain the predicted user pose. When the voice instruction contains spatial description words such as "front", "right side" and "upper side", the system will map these relative spatial descriptions to the XR device interface coordinate system corresponding to the predicted user pose. This means that even if the user is walking, turning his head or performing other body movements, the system can accurately understand and execute the instruction according to his predicted future position and orientation, ensuring the stability and continuity of voice control and greatly improving the interaction experience in a moving scenario.

[0055] The above-mentioned preset pose prediction model can adopt, for example, Kalman Filter or Extended Kalman Filter (EKF): suitable for linear or weakly nonlinear systems, and can optimally estimate and predict the user's pose by fusing IMU data and visual odometry data. For example, Unscented Kalman Filter (UKF) or Particle Filter: suitable for strongly nonlinear systems, which can more accurately handle pose prediction under complex user motion patterns. For example, a deep learning-based sequence prediction model: such as a recurrent neural network (RNN), a long short-term memory network or a Transformer model, which can predict the future pose sequence by learning a large amount of user motion data, and is particularly suitable for predicting user intentional motion. The specific model can be set by the developer based on expert experience or use scenarios, and is not limited here.

[0056] Subsequently, based on the attitude information and position information corresponding to the predicted user pose, a dynamic mapping matrix from the user's own coordinate system to the XR device interface coordinate system is constructed, the attitude information includes rotation parameters, and the position information includes translation parameters. The dynamic mapping matrix is used to realize real-time conversion of three-dimensional coordinates from the user's own coordinate system to the XR device interface coordinate system.

[0057] When the spatial description in the voice control instruction contains directional words relative to the user's own direction, the dynamic mapping matrix is used to convert the spatial description into coordinates in the XR device display space corresponding to the predicted user pose. The directional words include "front", "right", "up", and "left front", and the coordinates include absolute coordinates or relative coordinates. This means that even if the user issues a "front" instruction while walking, turning his head, or making other body movements, the system can accurately point to the user's predicted future front, rather than the front at the time of instruction issuance, thereby completely eliminating coordinate drift caused by user movement and ensuring the stability and continuity of voice control.

[0058] It should be noted that the above-mentioned user's own coordinate system is a three-dimensional coordinate system established with a specific part of the user's body as the origin, including the head or torso, and its coordinate axis direction corresponds to the user's own front-back, left-right, and up-down directions. The XR device interface coordinate system is a three-dimensional global coordinate system established by the XR device (smart glasses) for positioning interface elements, and its origin and coordinate axis direction correspond to the physical space or virtual space perceived by the XR device, and is used to uniformly describe the spatial positions of all interface elements. In the user's own coordinate system, the directional words relative to the user's own direction correspond to specific coordinate axis directions. "Front" corresponds to the front coordinate axis of the user's own coordinate system, "right" corresponds to the right coordinate axis of the user's own coordinate system, and "up" corresponds to the up coordinate axis of the user's own coordinate system.

[0059] Through the above-mentioned scheme, the dynamic mapping matrix constructed in the user's moving state accurately converts the relative position in the user's own coordinate system into the absolute position in the XR device interface coordinate system, ensuring that even in dynamic scenarios such as walking and turning the head, the spatial description in the voice instruction can be accurately mapped to the corresponding virtual interface element, avoiding operation deviation caused by coordinate drift.

[0060] S204, according to the information of each operable object in the target interaction space region, matching the target operation object corresponding to the voice control instruction, and performing the XR interaction operation corresponding to the operation intention according to the target operation object.

[0061] In this application, the operable object can be understood as an XR interface element in the display interface corresponding to the XR device interface coordinate system, such as buttons, menus, information cards, 3D models, etc., and the operable object information is structured data for describing the relevant attributes of these XR interface elements in the XR device interface coordinate system, such as spatial attribute information, functional attribute information, state attribute information, and identification attribute information. The user can operate the XR interface element through voice or gesture operations. The XR interaction operation is, for example, a specific operation such as "opening an application" or "selecting a file".

[0062] In the embodiments of the present application, the matching of the target operation object corresponding to the voice control instruction according to the information of each operable object in the target interaction space region comprises the following steps: According to the information of each operable object, the spatial coordinates and the function attribute of each operable object are determined. The operable object is an XR device interface element. According to the spatial coordinates, the function attribute and the voice control instruction, the matching degree of each operable object and the voice control instruction is calculated. The matching degree at least includes the spatial coordinate matching degree and the function attribute matching degree. The matching degrees are compared with the preset matching degree threshold, so as to determine the target operation object according to the comparison result.

[0063] That is, the present application obtains the spatial coordinates and the function attribute of each operable object from the information of the operable object, and then calculates the matching degree for each operable object and the operation intention and the spatial description of the voice control instruction. The matching degree can include the spatial coordinate matching degree, which can be calculated by calculating the cosine similarity, the Euclidean distance, etc., to calculate whether the spatial coordinates of the operable object are in the target interaction space region; and the function attribute matching degree, which can be obtained by calculating the cosine similarity of the text semantics of the operation intention and the function attribute, so as to quantify the degree of coincidence between the function attribute of the element and the operation intention of the voice instruction. Among them, the spatial coordinate matching degree can further include the geometric matching degree and the spatial matching degree. The geometric matching degree: the alignment degree of the position and size of the element and the fused attention vector (head, line of sight, hand pointing direction), for example, the smaller the angle between the element center and the line of sight direction vector, the higher the matching degree; the spatial matching degree: whether the element meets the spatial description (such as “front”, “second from the right”) in the voice instruction.

[0064] The matching degree is obtained by comprehensively considering the above-mentioned spatial coordinate matching degree and function attribute matching degree, and then the matching degree is compared with the preset matching degree threshold. The operable object with a matching degree greater than the preset matching degree threshold is taken as the target operation object. The preset matching degree threshold can be set by the user, which is not limited in the present application. In addition, if there are multiple operable objects with a matching degree greater than the preset matching degree threshold, the operable object corresponding to the maximum value in the matching degree can be taken as the target operation object.

[0065] When all the matching degrees are less than the preset matching degree threshold, it can be judged as a recognition failure, at this time, a voice prompt such as “Sorry, I didn’t understand your instruction, please say it again” or a selection list is displayed on the XR interface to guide the user to reissue the instruction or make a selection. If there are multiple operable objects with a matching degree greater than the preset matching degree threshold, and the multiple matching degrees are similar, at this time, a multi-target conflict processing is entered.

[0066] Through the multi-dimensional matching degree calculation of the spatial coordinates and the functional attributes, multi-dimensional verification of the target operation object can be realized, a matching degree threshold is set, and the optimal object is selected, so as to avoid erroneous selection caused by single-dimensional matching and improve the accuracy of target matching.

[0067] In some embodiments of the present application, in the case of multi-target conflict processing, the present application determines the target operation object according to the comparison result, specifically including: In the case where it is determined according to the comparison result that there are multiple pending operable objects, each pending operable object is added to the XR interface display list and displayed to the user. Voice prompt information is played to allow the user to select the display label of each pending operable object in the XR interface display list. Based on the secondary voice instruction of the user, the target operation object is determined.

[0068] In other words, the present application provides a secondary selection function for the user to explicitly select the target operation object. For example, the XR interface display list displays the display labels of multiple pending operable objects, and the display label can be a number assigned to each pending operable object. The secondary voice instruction of the user can include the display label, such as "number 1", "number 2", etc. The present application can also present the user in the form of a thumbnail, and the voice prompt information can be "there are multiple matching items, please say the number you want to select".

[0069] The present application aims at the multi-target conflict scene, and guides the user to make a secondary selection through list display + voice prompt, solves the pain point that the traditional multi-target conflict cannot be decided; through the secondary voice instruction to explicitly determine the target, avoids random selection errors caused by close matching degree, improves the processing capability of the system for complex interface scenes, and guarantees the certainty of interaction.

[0070] In an embodiment of the present application, if the target operation object is matched, a direct interaction operation is performed, which may cause misoperation, especially in a motion state. Therefore, before performing the XR interaction operation corresponding to the operation intention according to the target operation object, the following embodiments are provided, including: A visual marker signal of the target operation object is generated. The visual marker signal includes at least one or more of the following: a highlight frame, dynamic flashing, and color gradient. After receiving a confirmation instruction of the user, the XR interaction operation is performed. The confirmation instruction includes at least one of the following: a voice confirmation instruction, a preset gesture instruction, or a preset action instruction.

[0071] In other words, after the multimodal target positioning technology successfully determines the unique or most likely XR interface element controlled by the user's intention, the system does not immediately perform the operation. The intelligent feedback generation unit of the feedback control module immediately generates and displays a clear visual feedback signal to intuitively inform the user of the target recognized by the system and provide the user with a confirmation opportunity. These visual feedback methods include but are not limited to: highlight display: significantly increasing the color, brightness or saturation of the target interface element to make it stand out in the XR environment; border flashing / outline: adding a dynamic flashing or continuously outlined border around the target interface element; color / transparency change: changing the color or periodically changing the transparency of the target element; zooming / floating effect: slightly enlarging or slightly floating the target element in space to attract the user's attention.

[0072] The duration of the visual feedback can be set to 1.5 to 2.5 seconds to ensure that the user has enough time to observe and understand the feedback information. During the display of the visual feedback, the system waits for the user to confirm. The user can confirm in various ways, increasing the flexibility of the interaction: such as issuing a short voice command again: for example, the user can say "confirm", "yes", "execute", etc. For example, a specific non-voice action: the user can perform a preset nodding action (identified by a head pose sensor) or a specific gesture (such as the "OK" gesture).

[0073] In addition, in some scenarios (such as when the user has explicitly expressed a preference), if the user does not perform any other operation or issue a cancellation instruction within a preset short period of time (for example, 3 seconds) after the visual feedback is displayed, the system can automatically consider the operation as a confirmation and execute it. Once the user's confirmation signal is received, the instruction execution unit sends the final control instruction to the XR device, triggering the corresponding interface operation (such as opening an application, selecting a menu item, adjusting a parameter, submitting a form, etc.).

[0074] If, after the visual feedback is displayed, the user finds that the system has made a mistake and is not the target operation object that the user wants to operate, the user can immediately issue a cancellation instruction (such as the voice control instructions "cancel", "no", "return") or perform a specific cancellation gesture. After receiving the cancellation instruction, the system will cancel the current operation to be performed and may prompt the user to re-enter or provide alternative solutions.

[0075] The present application provides an intuitive and effective voice control feedback mechanism by using visual marker signals for the user to intuitively confirm the target operation object, solving the problem that the user cannot immediately confirm whether the system correctly understands the voice instruction; adding a confirmation link allows the user to correct errors in time, significantly reducing the risk of misoperation, enhancing the user's sense of trust and operation safety for the system, and improving the reliability of the interaction and user experience.

[0076] In addition, the present application can also continuously monitor the entire response time from the issuance of voice instructions to the execution of operations. If the response time exceeds the preset time threshold, which can be preset by the user or the developer and is not specifically limited here, the system will judge it as a delay. At this time, the system will provide the user with a prompt that "the system is processing, please wait" and try to optimize the processing flow or prompt the user to check the network connection so as to execute the XR interaction operation after the processing system delay.

[0077] Based on the above-mentioned XR interaction method based on voice control, the present application in the working process of executing XR device operation, as shown in the figure, includes three parts of initialization stage, real-time processing stage and abnormal processing mechanism, which realizes the XR device operation process through information interaction. Figure 6

[0078] The initialization stage specifically includes: sensor module startup and self-check, user personalized profile loading / establishment, calibration of head tracking and eye tracking system, initialization of voice recognition engine and natural language processing module, establishment of three-dimensional space coordinate system of current environment.

[0079] More specifically, when the smart glasses are started for the first time or the user switches, the system will perform a series of initialization operations to ensure the accuracy and personalization of subsequent voice control. Sensor module startup and self-check: activate all built-in sensors (microphone array, camera, eye tracker, IMU, etc.) and perform self-check to ensure normal operation of the hardware. User personalized profile loading / establishment: load the user's personalized voice model, interaction preference settings, and historical interaction data. For new users, guide them to complete voice model training and calibration. Calibration of head tracking and eye tracking system: through a specific calibration program (such as the user staring at a point on the screen), accurately calibrate the mapping relationship between head and eye movement and XR display space, ensuring the accuracy of gaze direction and head orientation data. Initialization of voice recognition engine and natural language processing module: load pre-trained ASR and NLU models and adapt them to the user's language and accent. Establishment of three-dimensional space coordinate system of current environment: use SLAM technology to construct a three-dimensional map of the environment where the user is located and establish a stable global coordinate system for positioning XR interface elements and user pose.

[0080] The real-time processing stage specifically includes: input collection, voice processing, motion detection and pose prediction, target positioning, feedback generation, user confirmation and instruction execution, XR device operation.

[0081] ​More specifically, the real-time processing stage is the core stage of the system's continuous operation, responsible for responding to the user's voice instructions in real time and conducting corresponding XR interactions. Input acquisition: The multi-modal input acquisition module acquires the user's voice, vision (head orientation, gaze direction), motion (displacement, acceleration, angular velocity), and gesture information in parallel and continuously. All data streams are time-synchronized to ensure the accuracy of information fusion. Voice processing: The collected voice signal is sent to the voice processing module in real time. The ASR engine converts it into text, and the NLU unit parses the operation intent and spatial description. At the same time, the instruction classifier preliminarily classifies the instructions. Motion detection and pose prediction: The motion perception unit continuously monitors the user's motion state. If it detects that the user is in a moving state, the pose prediction module will predict the user's pose in the future very short time according to the current motion data, and generate a dynamic coordinate mapping matrix. Target positioning: The target positioning module fuses the semantic information of the voice instruction, the user's behavior (head orientation, gaze direction, hand pointing) under the predicted pose, and the dynamic coordinate mapping matrix, accurately calculates the "target space region" controlled by the user's intent, and matches the XR interface element that best matches the intent in the region. The adaptive attention adjustment algorithm dynamically adjusts the weights of each behavior signal in this process. Feedback generation: Once the target interface element is determined, the feedback control module will immediately generate visual feedback signals (such as highlighting, flashing, color change), intuitively prompting the user on the XR display interface that the system has identified the target. This step is crucial, as it provides the user with an opportunity to confirm. User confirmation and instruction execution: The system waits for the user to confirm the visual feedback. The user can complete the confirmation by issuing a brief "confirmation" voice instruction again, a specific nodding action, or not having other operations within a preset short time (automatic confirmation). Once confirmed, the execution confirmation unit sends the final control instruction to the XR device, triggering the corresponding interface operation (such as opening an application, selecting a menu item, adjusting a parameter, etc.).

[0082] The abnormal processing mechanism specifically includes: failure recognition processing, motion interference processing, multi-target conflict processing, system delay processing.

[0083] To improve the robustness and user experience of the system, the application designs a perfect abnormality processing mechanism. Specifically, the recognition failure processing: when the highest confidence calculated by the target positioning confidence assessment unit is lower than the preset threshold, the system will judge as recognition failure. At this time, the system will guide the user to reissue the instruction or make a selection through voice prompt (such as "I'm sorry, I didn't understand your instruction, please say it again") or display the alternative list on the XR interface. Motion interference processing: when the motion sensing unit detects that the user's acceleration or angular velocity exceeds the threshold of violent motion, the system will temporarily reduce the attention weight of the user's behavior (head, line of sight, gesture), and even can temporarily suspend the voice control function, and prompt the user through vision or voice "your current motion amplitude is large, please try voice control after stabilizing", in order to avoid misoperation caused by violent motion. Multi-target conflict processing: in some cases, the target positioning module may identify multiple candidate target interface elements with similar confidence. At this time, the system will not immediately execute the operation, but will present these candidate elements in the form of number or thumbnail to the user on the XR interface, and prompt the user through voice "there are multiple matching items, please say the number you want to select", and wait for the user to further specify the selection. System delay processing: the system will continuously monitor the entire response time from the issuance of voice instruction to the execution of operation. If the response time exceeds the preset threshold, the system will judge as delay. At this time, the system will provide the user with the prompt "the system is processing, please wait", and try to optimize the processing flow or prompt the user to check the network connection.

[0084] Through the above technical solutions, the application provides an XR interaction control scheme based on voice control, which fuses multi-modal interaction data streams to obtain a target interaction space region determined in combination with voice signals and non-voice behavior information, breaks through the limitation of traditional voice control relying only on semantic analysis, greatly reduces the misoperation caused by ambiguous space description, and realizes accurate identification of the target operation object. At the same time, the target interaction space region obtained by fusing the user behavior information and the space description in the voice control instruction dynamically adapts to the relative change of the user's own coordinate system and the XR device interface coordinate system, effectively avoiding the problem of coordinate drift when the user moves, ensuring that even in dynamic scenes such as walking and turning the head, the voice instruction can still be stably mapped to the target position of the device display space, significantly improving the continuity and reliability of the control.

[0085] In addition, through the cooperative processing of multi-modal information, the user can interact with the XR device through the intuitive way of "voice + natural behavior" without relying on additional controllers, which expands the application range of XR devices in mobile scenarios and improves the smoothness of user experience. A high-precision, dynamically adaptive and reliable voice control interaction is realized.

[0086] Based on the same inventive concept, the embodiment of the present application also provides an XR interaction device based on voice control, which can execute the XR interaction method based on voice control described above. As shown in Figure 7 The XR interaction device based on voice control 700 includes: A receiving module 701 is configured to receive a multi-modal interaction data stream acquired by a multi-modal sensor. A first determining module 702 is configured to identify and analyze a voice signal in the multi-modal interaction data stream to determine a corresponding voice control instruction. The voice control instruction includes an operation intention and a spatial description. The spatial description is spatial constraint information of a target operation object by a user. A second determining module 703 is configured to determine a corresponding target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multi-modal interaction data stream synchronized with the voice signal. A matching execution module 704 is configured to match a target operation object corresponding to the voice control instruction according to operation object information in the target interaction space region, and execute an XR interaction operation corresponding to the operation intention according to the target operation object.

[0087] The second determining module 703 is specifically configured to: convert the non-voice behavior information into a behavior feature vector in a three-dimensional space. The non-voice behavior information includes at least one or more of head orientation information, line-of-sight direction information, and hand pointing information. The spatial description in the voice control instruction is converted into a spatial constraint condition. The spatial constraint condition includes a constraint rule parameter group representing the spatial constraint information. Based on a fusion calculation result of the behavior feature vector and the spatial constraint condition, a target interaction space region boundary in a three-dimensional space coordinate system is determined to obtain the target interaction space region according to the target interaction space region boundary.

[0088] The second determining module 703 is specifically configured to: perform weighted calculation on a preset fusion weight group, the behavior feature vector, and a reference vector corresponding to the spatial constraint condition to generate a target direction vector after fusion calculation. A direction corresponding to the target direction vector is taken as a reference to generate a target interaction space region boundary according to the spatial constraint condition. The spatial constraint condition includes at least a distance constraint parameter and a range constraint parameter.

[0089] Before performing the weighted calculation on the preset fusion weight group, the behavior feature vector, and the reference vector corresponding to the spatial constraint condition to generate the target direction vector after fusion calculation, the system can also: The motion state of the user is determined based on motion state detection information from the multi-modal sensor. The motion state includes a stationary state and a moving state. When the motion state is the stationary state, a first preset fusion weight set is determined according to the information type of the non-speech behavior information, and weighted calculation is performed according to the first preset fusion weight set. When the motion state is the moving state, a user motion intensity value is determined according to the motion state detection information, and the user motion intensity value is matched with a plurality of preset intensity threshold intervals. According to the preset intensity threshold interval in which the user motion intensity value is located, a corresponding dynamic weight adjustment factor is determined, and a baseline preset fusion weight set is loaded. The baseline preset fusion weight set is corrected according to the dynamic weight adjustment factor to obtain a corresponding second preset fusion weight set, and corresponding weighted calculation is performed.

[0090] After determining the motion state of the user, the system can also: When the motion state is the moving state, a predicted user pose is determined based on a preset pose prediction model and current motion state detection information. A dynamic mapping matrix from the user's own coordinate system to the XR device interface coordinate system is constructed according to the attitude information and position information corresponding to the predicted user pose. The attitude information includes rotation parameters, and the position information includes translation parameters. The spatial description is converted into coordinates in the XR device interface coordinate system corresponding to the predicted user pose according to the dynamic mapping matrix, and the converted coordinates are updated to the spatial constraint condition.

[0091] The matching execution module 704 is specifically configured to: According to the information of each operable object, the spatial coordinates and functional attributes of each operable object are determined. The operable object is an XR device interface element. According to the spatial coordinates, the functional attributes and the voice control instruction, the matching degrees of each operable object and the voice control instruction are calculated. The matching degrees include at least spatial coordinate matching degrees and functional attribute matching degrees. Each matching degree is compared with a preset matching degree threshold to determine a target operation object according to the comparison result.

[0092] The matching execution module 704 is specifically further configured to: In a case where there are a plurality of pending operable objects determined according to the comparison result, each pending operable object is added to an XR interface display list and displayed to the user. Voice prompt information is played to allow the user to select a display label of each pending operable object in the XR interface display list. The target operation object is determined based on a secondary voice instruction of the user.

[0093] Before performing the XR interaction operation corresponding to the operation intention according to the target operation object, the system can also: A visual marking signal of the target operation object is generated. The visual marking signal includes at least one or more of the following: a highlighted border, dynamic flashing, and color gradient. After receiving a confirmation instruction of the user, an XR interaction operation is performed. The confirmation instruction includes at least one of the following: a voice confirmation instruction, a preset gesture instruction, or a preset action instruction.

[0094] Based on the same inventive concept, the present application also provides an XR interaction device based on voice control. Referring to Figure 8 , Figure 8 A structural schematic diagram of an XR interaction device based on voice control is provided for the embodiments of the present application. The device includes: at least one processor; and a memory connected to the at least one processor in communication. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: receive a multi-modal interaction data stream acquired by a multi-modal sensor. The voice signal in the multi-modal interaction data stream is identified and analyzed to determine a corresponding voice control instruction. The voice control instruction includes an operation intent and a spatial description. The spatial description is spatial constraint information of the target operation object by the user. Based on the voice control instruction and one or more non-voice behavior information in the multi-modal interaction data stream synchronized with the voice signal, a corresponding target interaction space region is determined. According to the information of each operable object in the target interaction space region, the target operation object corresponding to the voice control instruction is matched, and the XR interaction operation corresponding to the operation intent of the target operation object is performed.

[0095] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device and equipment embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0096] The device and equipment provided by the embodiments of the present application correspond to the method, so the device and equipment also have similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and equipment will not be described here.

[0097] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0098] The above description is merely illustrative of the application, and not restrictive. Various modifications and changes can become apparent to those skilled in the art. Incorporating any modification, equivalent substitution, improvement, etc. within the spirit and principle of the application, shall be included in the scope of the claims of the application.

Claims

1. A voice control based XR interaction method, characterized in that, The method comprises: receiving a multimodal interaction data stream acquired by a multimodal sensor; recognizing and analyzing a voice signal in the multimodal interaction data stream to determine a corresponding voice control instruction; wherein the voice control instruction comprises an operation intent and a spatial description; the spatial description is spatial constraint information of a target operation object by a user; determining a corresponding target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multimodal interaction data stream synchronized with the voice signal; matching the target operation object corresponding to the voice control instruction according to the information of each operable object in the target interaction space region, and performing an XR interaction operation corresponding to the operation intent according to the target operation object.

2. The method of claim 1, wherein, determining a corresponding target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multimodal interaction data stream synchronized with the voice signal, specifically comprising: converting the non-voice behavior information into a behavior feature vector in a three-dimensional space; wherein the non-voice behavior information at least includes one or more of the following: head orientation information, gaze direction information, hand pointing information; convert the spatial description in the voice control instruction into a spatial constraint condition; the spatial constraint condition includes a constraint rule parameter group representing the spatial constraint information; determining the target interaction space region boundary in the three-dimensional space coordinate system based on the fusion calculation result of the behavior feature vector and the spatial constraint condition, to obtain the target interaction space region according to the target interaction space region boundary.

3. The method of claim 2, wherein, determining the target interaction space region boundary in the three-dimensional space coordinate system based on the fusion calculation result of the behavior feature vector and the spatial constraint condition, specifically comprising: performing weighted calculation on a preset fusion weight group, the behavior feature vector, and a reference vector corresponding to the spatial constraint condition to generate a target direction vector after fusion calculation; generating the target interaction space region boundary according to the spatial constraint condition with the direction corresponding to the target direction vector as the reference; wherein the spatial constraint condition at least includes: distance constraint parameter, range constraint parameter.

4. The method of claim 3, wherein, Before performing weighted calculation on a preset fusion weight group, the behavior feature vector, and a reference vector corresponding to the spatial constraint condition to generate a target direction vector after fusion calculation, the method further comprises: determining the motion state of the user based on the motion state detection information from the multimodal sensor; the motion state includes a stationary state and a moving state; when the motion state is a stationary state, determining a first preset fusion weight group according to the information type of the non-voice behavior information to perform weighted calculation according to the first preset fusion weight group; when the motion state is a moving state, determining a user motion intensity value according to the motion state detection information, and matching the user motion intensity value with a plurality of preset intensity threshold intervals; determining a corresponding dynamic weight adjustment factor according to the preset intensity threshold interval where the user motion intensity value is located, and loading a reference preset fusion weight group; According to the dynamic weight adjustment factor, the reference preset fusion weight set is corrected to obtain a corresponding second preset fusion weight set, and a corresponding weighted calculation is performed.

5. The method of claim 4, wherein, After determining the motion state of the user, the method further comprises: In the case where the motion state is a moving state, a predicted user pose is determined based on a preset pose prediction model and the current motion state detection information; According to the attitude information and position information corresponding to the predicted user pose, a dynamic mapping matrix from the user's own coordinate system to the XR device interface coordinate system is constructed; the attitude information includes rotation parameters, and the position information includes translation parameters; According to the dynamic mapping matrix, the spatial description is converted into coordinates in the XR device interface coordinate system corresponding to the predicted user pose, and the converted coordinates are updated to the spatial constraint condition.

6. The method of claim 1, wherein, According to the information of each operable object in the target interaction space region, the target operation object corresponding to the voice control instruction is matched, specifically including: According to the information of each operable object, the spatial coordinates and functional attributes of each operable object are determined; the operable object is an XR device interface element; According to the spatial coordinates, the functional attributes and the voice control instruction, the matching degree of each operable object and the voice control instruction is calculated; the matching degree at least includes spatial coordinate matching degree and functional attribute matching degree; Each matching degree is compared with a preset matching degree threshold to determine the target operation object according to the comparison result.

7. The method of claim 6, wherein, According to the comparison result, the target operation object is determined, specifically including: In the case where there are multiple pending operable objects according to the comparison result, each of the pending operable objects is added to an XR interface display list and displayed to the user; Play voice prompt information to allow the user to select the display label of each of the pending operable objects in the XR interface display list; Based on the secondary voice instruction of the user, the target operation object is determined.

8. The method of claim 1, wherein, Before performing the XR interaction operation corresponding to the operation intention according to the target operation object, the method further comprises: Generating a visual marker signal of the target operation object; wherein the visual marker signal at least includes one or more of the following: highlight frame, dynamic flashing, color gradient; After receiving the user's confirmation instruction, the XR interaction operation is performed; wherein the confirmation instruction at least includes: voice confirmation instruction, preset gesture instruction or preset action instruction.

9. A voice control based XR interaction device, characterized in that, The device can perform the XR interaction method based on voice control according to any one of claims 1-8; the device comprises: A receiving module for receiving multi-modal interaction data streams obtained by a multi-modal sensor; A first determining module for identifying and analyzing voice signals in the multi-modal interaction data streams to determine corresponding voice control instructions; wherein the voice control instruction includes operation intention and spatial description; the spatial description is the spatial constraint information of the user to the target operation object; A second determining module for determining the target operation object corresponding to the voice control instruction according to the information of each operable object in the target interaction space region; A second determination module is configured to determine a target interaction space region based on the voice control instruction and one or more non-voice behavior information in the multimodal interaction data stream that is synchronized with the voice signal; A matching execution module is configured to match the target operation object corresponding to the voice control instruction according to the operation object information in the target interaction space region, and execute the XR interaction operation corresponding to the operation intention according to the target operation object.

10. A voice control based XR interaction device, characterized by, The device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the voice control-based XR interaction method of any one of claims 1-8.

Images