Virtual reality data interaction method and device based on AR / VR / MR glasses and medium

Abstract

The invention relates to a virtual reality data interaction method and device based on AR / VR / MR glasses and a medium, and the method comprises the steps: obtaining multi-modal interaction data collected by wearable equipment in a use process based on a data collection device corresponding to the wearable equipment, carrying out the recognition of the multi-modal interaction data, and carrying out the recognition of the multi-modal interaction data; according to the probability data distribution information, the interaction mode weight information and the interaction feature function, performing data fusion on the multi-modal interaction data, generating interaction instruction data of the wearable device, according to the multi-modal interaction data, constructing a virtual reality space, and generating a corresponding virtual display interface; generating an interface adjustment instruction under the virtual display interface according to the interaction instruction data, adjusting the virtual display interface based on the interface adjustment instruction, generating a target virtual display interface, and displaying the target virtual display interface through the wearable device. Therefore, data interaction based on the wearable device is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a virtual reality data interaction method, device and medium based on AR / VR / MR glasses. Background Art

[0002] With the continuous development of social productivity and science and technology, all walks of life have attached increasing importance to the research of virtual reality (VR) technology. VR technology has also made great progress and gradually become a new field of science and technology. As a simulation system for experiencing the virtual world, VR technology uses interactive three-dimensional dynamic vision and system simulation of entity behavior to immerse the operator in the simulation environment. At present, VR technology can be used to derive various human-computer interaction products. The operator can use a wireless controller or a handheld wired controller for interactive operations. In this operation mode, the operator sees the virtual screen, but still operates the wireless controller or handheld wired controller in reality, which reduces the sense of presence of the VR experience. Summary of the invention

[0003] In view of this, the present application provides a virtual reality data interaction method, device and medium based on AR / VR / MR glasses.

[0004] According to a first aspect of an embodiment of the present application, a virtual reality data interaction method based on AR / VR / MR glasses is provided, which is applied to a wearable device, wherein the wearable device includes: augmented reality AR glasses, virtual reality VR glasses or mixed reality MR glasses, and the method includes: Based on the data collection device corresponding to the wearable device, obtain multimodal interaction data collected by the wearable device during use, wherein the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data; Identify the multimodal interaction data to determine probability data distribution information, interaction mode weight information, and interaction feature function of each modality interaction data; According to the probability data distribution information, the interaction mode weight information and the interaction feature function, data fusion is performed on the multimodal interaction data to generate interaction instruction data of the wearable device; According to the multimodal interaction data, a virtual reality space is constructed to generate a corresponding virtual display interface; according to the interaction instruction data, an interface adjustment instruction under the virtual display interface is generated, and the virtual display interface is adjusted based on the interface adjustment instruction to generate a target virtual display interface, and the target virtual display interface is displayed through the wearable device.

[0005] Optionally, performing data fusion on the multimodal interaction data according to the probability data distribution information, the interaction mode weight information and the interaction feature function to generate interaction instruction data for the wearable device includes: Preprocessing the multimodal interaction data to generate target interaction data, wherein the preprocessing includes standardization and normalization; According to the data type corresponding to the multimodal interaction data and the interaction feature function, feature extraction is performed on the multimodal interaction data to generate feature data corresponding to the data type; Determining data weight information corresponding to the data type according to the probabilistic data distribution information and the interaction mode weight information; The feature data is fused according to the data weight information to generate the interaction instruction data.

[0006] Optionally, performing data fusion on the feature data according to the data weight information to generate the interaction instruction data includes: According to the data weight information, the feature data are spliced ​​to generate a target feature vector; identifying user intention information corresponding to the multimodal interaction data according to the target feature vector; The interaction instruction data is generated according to the user intention information and the current virtual display interface of the wearable device.

[0007] Optionally, the identifying, according to the target feature vector, user intention information corresponding to the multimodal interaction data includes: Get multiple function icons under the current virtual display interface; Determining, according to the target feature vector, a target function icon currently selected by the user based on the corresponding multiple function icons; The user intention information is determined according to the target function icon.

[0008] Optionally, constructing a virtual reality space according to the multimodal interaction data to generate a corresponding virtual display interface includes: Determining image data information and corresponding depth data information captured by a front camera device of the wearable device according to the multimodal interaction data; Generate three-dimensional space data according to the image data information and the depth data information; The three-dimensional space constructed by the three-dimensional space data is rendered according to a preset virtual display interface to generate the virtual display interface.

[0009] Optionally, generating three-dimensional space data according to the image data information and the depth data information includes: According to the image data information, distinguish the types of objects in the image acquisition area corresponding to the wearable device to obtain multiple local areas; Determine a plurality of centroid positions corresponding to the plurality of local areas one by one; According to the plurality of centroid positions and the depth data information, adjacent points in the corresponding plurality of local areas are grouped to generate a plurality of image encoding data; The plurality of image coded data are encoded to generate the three-dimensional space data.

[0010] According to a second aspect of an embodiment of the present application, a virtual reality data interaction device based on AR / VR / MR glasses is provided, which is applied to a wearable device, wherein the wearable device includes: augmented reality AR glasses, virtual reality VR glasses or mixed reality MR glasses, and the device includes: An acquisition module is configured to acquire multimodal interaction data collected by the wearable device during use based on a data acquisition device corresponding to the wearable device, wherein the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data; A determination module is configured to identify the multimodal interaction data and determine probability data distribution information, interaction mode weight information and interaction feature function of each modal interaction data; A generating module is configured to perform data fusion on the multimodal interaction data according to the probability data distribution information, the interaction mode weight information and the interaction characteristic function, so as to generate interaction instruction data for the wearable device; The execution module is configured to construct a virtual reality space according to the multimodal interaction data to generate a corresponding virtual display interface; generate an interface adjustment instruction under the virtual display interface according to the interaction instruction data, adjust the virtual display interface based on the interface adjustment instruction, generate a target virtual display interface, and display the target virtual display interface through the wearable device.

[0011] Optionally, the generating module includes: A first generating submodule is configured to preprocess the multimodal interaction data to generate target interaction data, wherein the preprocessing includes standardization processing and normalization processing; A second generating submodule is configured to extract features from the multimodal interaction data according to the data type corresponding to the multimodal interaction data and the interaction feature function, and generate feature data corresponding to the data type; A determination submodule, configured to determine data weight information corresponding to the data type according to the probabilistic data distribution information and the interaction mode weight information; The third generating submodule is configured to perform data fusion on the feature data according to the data weight information to generate the interaction instruction data.

[0012] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0013] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including: processor; a memory for storing processor-executable instructions; The processor is configured to execute executable instructions in the memory to implement the method described in any one of the first aspects.

[0014] The technical solution provided by the embodiments of the present application may have the following beneficial effects: Through the above method, based on the data collection device corresponding to the wearable device, the multimodal interaction data collected by the wearable device during use is obtained, and the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data. The multimodal interaction data is identified to determine the probability data distribution information, interaction mode weight information, and interaction feature function of each modal interaction data. According to the probability data distribution information, the interaction mode weight information, and the interaction feature function, the multimodal interaction data is data fused to generate the interaction instruction data of the wearable device. According to the multimodal interaction data, a virtual reality space is constructed to generate a corresponding virtual display interface; according to the interaction instruction data, an interface adjustment instruction under the virtual display interface is generated, and the virtual display interface is adjusted based on the interface adjustment instruction to generate a target virtual display interface, and the target virtual display interface is displayed through the wearable device. Thus, by identifying the multimodal data, the interaction data input by the user to the wearable device based on the multimodality is determined, and the virtual interface is displayed according to the interaction data, so as to realize data interaction based on the wearable device.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] Figure 1It is a flowchart of a virtual reality data interaction method based on AR / VR / MR glasses according to an exemplary embodiment.

[0018] Figure 2 It is a block diagram of a virtual reality data interaction device based on AR / VR / MR glasses according to an exemplary embodiment. DETAILED DESCRIPTION

[0019] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0020] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0021] Figure 1 is a flow chart of a virtual reality data interaction method based on AR / VR / MR glasses according to an exemplary embodiment. Figure 1 As shown, the method is applied to a wearable device, the wearable device includes: AR glasses, VR glasses or MR glasses, and the method includes: In step S11, based on a data acquisition device corresponding to the wearable device, multimodal interaction data collected during use of the wearable device is acquired.

[0022] For example, in this embodiment, the wearable device is a virtual reality glasses, which is equipped with multiple sensing devices, multiple image acquisition devices and multiple distance recognition devices, which are used to collect relevant interaction data of the user in the process of using the wearable device, so as to identify the control intention of the user in the process of using the wearable device. For example, through the built-in viewpoint position recognition device of the wearable device, the viewpoint position of the user in the process of using the wearable device is identified, so as to identify the control instruction input by the user based on the viewpoint. Among them, the multimodal interaction data may include at least one of the following: gesture data, voice data, visual data, and posture data. The wearable device recognizes the multimodal interaction data to determine the user's interaction intention. That is, through the fusion of multimodal interaction data, it is determined that the user's current interaction intention is to control, select, slide or view. For example, the wearable device can determine the display icon selected by the user based on the wearable device according to the visual data, and then match the gesture data with the preset gesture posture. If the gesture data currently captured by the wearable device matches the determined gesture posture, it is determined based on the visual point position and the determined gesture of the wearable device to determine the icon currently displayed by the wearable device. Therefore, through the interaction between multimodal data and display data, the input of wearable device control data and the output of display data can be realized, so as to realize contactless control of wearable devices and enhance the user experience.

[0023] In this embodiment, the control data input of the wearable device is a common input of multimodal interaction data. The wearable device identifies the user's control intention according to the instructions input by the user through multiple instruction input methods during the use of the wearable device, and responds accordingly based on the control intention. In this embodiment, the form of multimodal interaction data is not limited, and it can be gestures, voice, visual points, the position of the wearable device, fixed actions, acceleration, etc. Different data types of multimodal interaction data correspond to different data acquisition devices configured in the wearable device. For example, by configuring an image acquisition device in the wearable device to not perform user gestures within the N-meter range of the wearable device's front sight, the user's voice data during the use of the wearable device is obtained through an audio acquisition device, and the user's eyeball position during the use of the wearable device is identified through the inward camera of the wearable device, thereby determining the user's viewpoint position, and configuring a level in the wearable device to determine the posture data of the user during the use of the wearable device.

[0024] In step S12, the multimodal interaction data is identified to determine the probability data distribution information, interaction mode weight information and interaction feature function of each modality interaction data.

[0025] It should be understood that in the process of multimodal interaction data identification, it is necessary to determine the credibility of each modal interaction data based on the probability data distribution information of each modal interaction data, and perform data fusion on multiple modal interaction data according to the credibility, interaction mode weight and interaction feature function. When a user uses a wearable device at the same time, he may issue interaction instructions to the wearable device based on multiple modal interaction data to realize data interaction between the user and the wearable device. However, in the data determination and input fusion stage, it is necessary to determine how to perform data fusion on the multimodal interaction data based on the probability data distribution, interaction mode weight and interaction feature function of each modal interaction data. For example, in this embodiment, the mapping relationship between different multimodal interaction data and probability data distribution information, interaction mode weight information and interaction feature function can be configured in the wearable device, and the corresponding probability data distribution information, interaction mode weight information and interaction feature function can be determined according to the data type of the multimodal interaction data.

[0026] Optionally, in this embodiment, the credibility of multimodal interaction data can also be identified. The multimodal interaction data can be compared with the standard data under the same type of modal data to determine the similarity between the multimodal interaction data collected by the current wearable device and the standard data, and the corresponding credibility can be determined based on the similarity, and the probability data distribution information of each modal interaction data can be determined based on the credibility. The interaction mode weight information is used to indicate the interaction weight of each multimodal interaction data. In this embodiment, the data fusion weight of the corresponding multimodal interaction data can be determined based on the interaction mode weight, and the corresponding multimodal interaction data can be fused based on the data fusion weight.

[0027] In step S13, the multimodal interaction data is fused according to the probability data distribution information, the interaction mode weight information and the interaction characteristic function to generate interaction instruction data of the wearable device.

[0028] For example, in this embodiment, multimodal interaction data is fused according to the probability data distribution information of the multimodal interaction data, the interaction mode weight information and the interaction feature function, to generate interaction instruction data sent by the current user to the wearable device based on the multimodal interaction mode during the process of using the wearable device.

[0029] Optionally, in some embodiments, the above step S13 includes: Preprocessing the multimodal interaction data to generate target interaction data, wherein the preprocessing includes standardization and normalization; According to the data type corresponding to the multimodal interaction data and the interaction feature function, feature extraction is performed on the multimodal interaction data to generate feature data corresponding to the data type; Determining data weight information corresponding to the data type according to the probabilistic data distribution information and the interaction mode weight information; The feature data is fused according to the data weight information to generate the interaction instruction data.

[0030] For example, in this embodiment, the multimodal interaction data is preprocessed, and it is necessary to perform data cleaning on the multimodal interaction data based on multiple preprocessing methods to generate target interaction data corresponding to the multimodal interaction data. According to the data type corresponding to the multimodal interaction data and the interaction feature function, feature extraction is performed on the multimodal interaction data to generate feature data corresponding to the data type, and data weight information corresponding to the data type is determined according to the probability data distribution information and the interaction mode weight information, and data fusion is performed on the feature data according to the data weight information to generate the interaction instruction data.

[0031] Optionally, in some embodiments, the above step of performing data fusion on the feature data according to the data weight information to generate the interaction instruction data includes: According to the data weight information, the feature data are spliced ​​to generate a target feature vector; identifying user intention information corresponding to the multimodal interaction data according to the target feature vector; The interaction instruction data is generated according to the user intention information and the current virtual display interface of the wearable device.

[0032] Optionally, in some embodiments, identifying user intention information corresponding to the multimodal interaction data according to the target feature vector includes: Get multiple function icons under the current virtual display interface; Determining, according to the target feature vector, a target function icon currently selected by the user based on the corresponding multiple function icons; The user intention information is determined according to the target function icon.

[0033] In step S14, a virtual reality space is constructed according to the multimodal interaction data to generate a corresponding virtual display interface; an interface adjustment instruction under the virtual display interface is generated according to the interaction instruction data, and the virtual display interface is adjusted based on the interface adjustment instruction to generate a target virtual display interface, and the target virtual display interface is displayed through the wearable device.

[0034] For example, in this embodiment, based on the above multimodal interaction data, a virtual reality space is constructed to realize the construction of a virtual reality scene based on a wearable device. A virtual display interface is generated in the virtual reality scene, and multiple function icons can be configured on the virtual display interface. The user can issue interaction instruction data to the wearable device based on the multiple function icons and the multimodal interaction data, generate an interface adjustment instruction under the virtual display interface, adjust the virtual display interface based on the interface adjustment instruction, generate a target virtual display interface, and display the virtual display interface through the wearable device. In this way, data interaction between the user and the wearable device is realized, so that the user can realize the corresponding virtual reality function based on the wearable device.

[0035] Through the above method, based on the data collection device corresponding to the wearable device, the multimodal interaction data collected by the wearable device during use is obtained, and the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data. The multimodal interaction data is identified to determine the probability data distribution information, interaction mode weight information, and interaction feature function of each modal interaction data. According to the probability data distribution information, the interaction mode weight information, and the interaction feature function, the multimodal interaction data is data fused to generate the interaction instruction data of the wearable device. According to the multimodal interaction data, a virtual reality space is constructed to generate a corresponding virtual display interface; according to the interaction instruction data, an interface adjustment instruction under the virtual display interface is generated, and the virtual display interface is adjusted based on the interface adjustment instruction to generate a target virtual display interface, and the target virtual display interface is displayed through the wearable device. Thus, by identifying the multimodal data, the interaction data input by the user to the wearable device based on the multimodality is determined, and the virtual interface is displayed according to the interaction data, so as to realize data interaction based on the wearable device.

[0036] Optionally, in some embodiments, constructing a virtual reality space according to the multimodal interaction data to generate a corresponding virtual display interface includes: Determining image data information and corresponding depth data information captured by a front camera device of the wearable device according to the multimodal interaction data; Generate three-dimensional space data according to the image data information and the depth data information; The three-dimensional space constructed by the three-dimensional space data is rendered according to a preset virtual display interface to generate the virtual display interface.

[0037] Optionally, generating three-dimensional space data according to the image data information and the depth data information includes: According to the image data information, distinguish the types of objects in the image acquisition area corresponding to the wearable device to obtain multiple local areas; Determine a plurality of centroid positions corresponding to the plurality of local areas one by one; According to the plurality of centroid positions and the depth data information, adjacent points in the corresponding plurality of local areas are grouped to generate a plurality of image encoding data; The plurality of image coded data are encoded to generate the three-dimensional space data.

[0038] Figure 2 is a block diagram of a virtual reality data interaction device based on AR / VR / MR glasses according to an exemplary embodiment. Figure 2 As shown, the device 100 is applied to a wearable device, the wearable device includes: augmented reality AR glasses, virtual reality VR glasses or mixed reality MR glasses, and the device 100 includes: The acquisition module 110 is configured to acquire multimodal interaction data collected by the wearable device during use based on the data acquisition device corresponding to the wearable device, wherein the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data; The determination module 120 is configured to identify the multimodal interaction data and determine the probability data distribution information, interaction mode weight information and interaction feature function of each modal interaction data; A generating module 130 is configured to perform data fusion on the multimodal interaction data according to the probability data distribution information, the interaction mode weight information and the interaction characteristic function, and generate interaction instruction data for the wearable device; The execution module 140 is configured to construct a virtual reality space according to the multimodal interaction data to generate a corresponding virtual display interface; generate an interface adjustment instruction under the virtual display interface according to the interaction instruction data, adjust the virtual display interface based on the interface adjustment instruction, generate a target virtual display interface, and display the target virtual display interface through the wearable device.

[0039] Optionally, the generating module 130 includes: A first generating submodule is configured to preprocess the multimodal interaction data to generate target interaction data, wherein the preprocessing includes standardization processing and normalization processing; A second generating submodule is configured to extract features from the multimodal interaction data according to the data type corresponding to the multimodal interaction data and the interaction feature function, and generate feature data corresponding to the data type; A determination submodule, configured to determine data weight information corresponding to the data type according to the probabilistic data distribution information and the interaction mode weight information; The third generating submodule is configured to perform data fusion on the feature data according to the data weight information to generate the interaction instruction data.

[0040] Optionally, the third generation submodule includes: A first generating unit, configured to concatenate the feature data according to the data weight information to generate a target feature vector; an identification unit, configured to identify user intention information corresponding to the multimodal interaction data according to the target feature vector; The second generating unit is used to generate the interaction instruction data according to the user intention information and the current virtual display interface of the wearable device.

[0041] Optionally, the identification unit is used to: Get multiple function icons under the current virtual display interface; Determining, according to the target feature vector, a target function icon currently selected by the user based on the corresponding multiple function icons; The user intention information is determined according to the target function icon.

[0042] Optionally, the execution module 140 includes: A determination submodule, configured to determine image data information and corresponding depth data information captured by a front camera device of the wearable device according to the multimodal interaction data; A third generating submodule, used to generate three-dimensional space data according to the image data information and the depth data information; The fourth generating submodule is used to render the three-dimensional space constructed by the three-dimensional space data according to a preset virtual display interface to generate the virtual display interface.

[0043] Optionally, the third generation submodule is used to: According to the image data information, distinguish the types of objects in the image acquisition area corresponding to the wearable device to obtain multiple local areas; Determine a plurality of centroid positions corresponding to the plurality of local areas one by one; According to the plurality of centroid positions and the depth data information, adjacent points in the corresponding plurality of local areas are grouped to generate a plurality of image encoding data; The plurality of image coded data are encoded to generate the three-dimensional space data.

[0044] Through the above method, based on the data collection device corresponding to the wearable device, the multimodal interaction data collected by the wearable device during use is obtained, and the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data. The multimodal interaction data is identified to determine the probability data distribution information, interaction mode weight information, and interaction feature function of each modal interaction data. According to the probability data distribution information, the interaction mode weight information, and the interaction feature function, the multimodal interaction data is data fused to generate the interaction instruction data of the wearable device. According to the multimodal interaction data, a virtual reality space is constructed to generate a corresponding virtual display interface; according to the interaction instruction data, an interface adjustment instruction under the virtual display interface is generated, and the virtual display interface is adjusted based on the interface adjustment instruction to generate a target virtual display interface, and the target virtual display interface is displayed through the wearable device. Thus, by identifying the multimodal data, the interaction data input by the user to the wearable device based on the multimodality is determined, and the virtual interface is displayed according to the interaction data, so as to realize data interaction based on the wearable device.

[0045] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the aforementioned embodiments.

[0046] The present application also provides an electronic device, including: processor; a memory for storing processor-executable instructions; The processor is configured to execute executable instructions in the memory to implement the method described in any one of the aforementioned embodiments.

[0047] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the present application. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0048] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A virtual reality data interaction method based on AR / VR / MR glasses, characterized in that: Applied to a wearable device, the wearable device includes: augmented reality (AR) glasses, virtual reality (VR) glasses, or mixed reality (MR) glasses, the method includes: Based on the data collection device corresponding to the wearable device, obtain multimodal interaction data collected by the wearable device during use, wherein the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data; Identify the multimodal interaction data to determine probability data distribution information, interaction mode weight information, and interaction feature function of each modality interaction data; According to the probability data distribution information, the interaction mode weight information and the interaction feature function, data fusion is performed on the multimodal interaction data to generate interaction instruction data of the wearable device; According to the multimodal interaction data, a virtual reality space is constructed to generate a corresponding virtual display interface; according to the interaction instruction data, an interface adjustment instruction under the virtual display interface is generated, and the virtual display interface is adjusted based on the interface adjustment instruction to generate a target virtual display interface, and the target virtual display interface is displayed through the wearable device.

2. The virtual reality data interaction method based on AR / VR / MR glasses according to claim 1, characterized in that: The step of fusing the multimodal interaction data according to the probability data distribution information, the interaction mode weight information and the interaction feature function to generate the interaction instruction data of the wearable device includes: Preprocessing the multimodal interaction data to generate target interaction data, wherein the preprocessing includes standardization and normalization; According to the data type corresponding to the multimodal interaction data and the interaction feature function, feature extraction is performed on the multimodal interaction data to generate feature data corresponding to the data type; Determining data weight information corresponding to the data type according to the probabilistic data distribution information and the interaction mode weight information; The feature data is fused according to the data weight information to generate the interaction instruction data.

3. The virtual reality data interaction method based on AR / VR / MR glasses according to claim 2, characterized in that: The performing data fusion on the feature data according to the data weight information to generate the interaction instruction data includes: According to the data weight information, the feature data are spliced ​​to generate a target feature vector; identifying user intention information corresponding to the multimodal interaction data according to the target feature vector; The interaction instruction data is generated according to the user intention information and the current virtual display interface of the wearable device.

4. The virtual reality data interaction method based on AR / VR / MR glasses according to claim 3 is characterized in that: The identifying, according to the target feature vector, user intention information corresponding to the multimodal interaction data includes: Get multiple function icons under the current virtual display interface; Determining, according to the target feature vector, a target function icon currently selected by the user based on the corresponding multiple function icons; The user intention information is determined according to the target function icon.

5. The virtual reality data interaction method based on AR / VR / MR glasses according to any one of claims 1 to 4, characterized in that: The step of constructing a virtual reality space according to the multimodal interaction data to generate a corresponding virtual display interface includes: Determining image data information and corresponding depth data information captured by a front camera device of the wearable device according to the multimodal interaction data; Generate three-dimensional space data according to the image data information and the depth data information; The three-dimensional space constructed by the three-dimensional space data is rendered according to a preset virtual display interface to generate the virtual display interface.

6. The virtual reality data interaction method based on AR / VR / MR glasses according to claim 5, characterized in that: Generating three-dimensional space data according to the image data information and the depth data information includes: According to the image data information, distinguish the types of objects in the image acquisition area corresponding to the wearable device to obtain multiple local areas; Determine a plurality of centroid positions corresponding to the plurality of local areas one by one; According to the plurality of centroid positions and the depth data information, adjacent points in the corresponding plurality of local areas are grouped to generate a plurality of image encoding data; The plurality of image coded data are encoded to generate the three-dimensional space data.

7. A virtual reality data interaction device based on AR / VR / MR glasses, characterized in that: Applied to a wearable device, the wearable device includes: augmented reality AR glasses, virtual reality VR glasses or mixed reality MR glasses, the device includes: An acquisition module is configured to acquire multimodal interaction data collected by the wearable device during use based on a data acquisition device corresponding to the wearable device, wherein the multimodal interaction data includes at least one of the following: gesture data, voice data, visual data, and posture data; A determination module is configured to identify the multimodal interaction data and determine probability data distribution information, interaction mode weight information and interaction feature function of each modal interaction data; A generating module is configured to perform data fusion on the multimodal interaction data according to the probability data distribution information, the interaction mode weight information and the interaction characteristic function, so as to generate interaction instruction data for the wearable device; The execution module is configured to construct a virtual reality space according to the multimodal interaction data to generate a corresponding virtual display interface; generate an interface adjustment instruction under the virtual display interface according to the interaction instruction data, adjust the virtual display interface based on the interface adjustment instruction, generate a target virtual display interface, and display the target virtual display interface through the wearable device.

8. The virtual reality data interaction device based on AR / VR / MR glasses according to claim 7, characterized in that: The generation module comprises: A first generating submodule is configured to preprocess the multimodal interaction data to generate target interaction data, wherein the preprocessing includes standardization processing and normalization processing; A second generating submodule is configured to extract features from the multimodal interaction data according to the data type corresponding to the multimodal interaction data and the interaction feature function, and generate feature data corresponding to the data type; A determination submodule, configured to determine data weight information corresponding to the data type according to the probabilistic data distribution information and the interaction mode weight information; The third generating submodule is configured to perform data fusion on the feature data according to the data weight information to generate the interaction instruction data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute executable instructions in the memory to implement the method described in any one of claims 1 to 6.

Images