Human-computer Interaction Acquisition Method, Device and System for Wearable Extended Reality Devices

Abstract

The present invention provides a human-computer interaction acquisition method, device and system for a wearable extended reality device. By analyzing the collected human perception data, human state information is obtained. By modeling the collected environmental perception data, a corresponding three-dimensional environmental scene is obtained, and the extended reality device is used for centralized display, so that an individual can simultaneously obtain his own human state information, the three-dimensional environmental scene of the location where he is located, and the extended reality scene, and can always understand his own state and the changes in the surrounding environment. The obtained human state information and three-dimensional environmental scene are transmitted to the cloud center, so that the human state information and three-dimensional environmental scene of partners can be always understood through data sharing, which is convenient for realizing human-machine collaborative work.

Description

Technical Field

[0001] The present invention relates to the technical field of wearable extended reality devices, and in particular, to a human-computer interaction acquisition method, device, and system for a wearable extended reality device. Background Art

[0002] With the development of the times, the methods and efficiency of information acquisition often determine the success or failure of a thing. In some occasions where it is necessary to always keep abreast of the latest information changes and personal changes. In the prior art, through wearable intelligent devices, including physiological sensors such as eye trackers, electroencephalographs, heart rate / respiration monitors, and behavior observation systems, etc., rely on multi-modal data such as personnel physiology, eye movement, and electroencephalogram to detect the state of personnel; through devices such as cameras, lidar, etc. These devices are used to sense the object or spatial information of the environment in order to obtain the environmental information where the personnel are located in real time.

[0003] In addition, the existing extended reality technologies include AR (augmented reality technology), VR (virtual reality technology), MR (mixed reality technology). These technologies can enable users to obtain more information and interact with others, the environment, etc. on the basis of reality, or enable users to obtain information and interact with others, the environment, etc. during the process of combining the real and virtual spaces. Taking the AR augmented reality technology as an example, some products adopting the AR augmented reality technology adopt a wearable design. Users can see additional virtual information on the basis of the real world through wearable AR glasses and other products. For example, users can see the superimposed scene of the real world and virtual information through AR glasses. At present, there are methods and technologies for digitally modeling the real world. Relevant technologies and researchers can digitize the real world through modeling, and others can view the scenes around the world through the digital achievements.

[0004] The prior art can meet people's single-sided needs, including digitally modeling the physical world, real-time collecting data information such as personnel physiology and eye movement, HUD (head-up display information), etc. However, the existing AR / VR devices cannot simultaneously sense the human body and the real environment and model the real environment, so that users cannot interact with the environment model based on the real environment in real time. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a human-computer interaction acquisition method, device, and system for a wearable extended reality device to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a human-computer interaction acquisition method for a wearable extended reality device, the method including the following steps:

[0007] Obtain the data synchronously collected by an extended reality device integrated with a human parameter sensing device and an environmental parameter sensing device. The synchronously collected data includes the virtual reality basic signal collected by the extended reality device, the human sensing data collected by the human parameter sensing device, and the environmental sensing data collected by the environmental parameter sensing device. The human sensing data includes at least one of eye movement data, physiological parameters, and electroencephalogram data. Provide an extended reality scene based on the virtual reality basic signal collected by the extended reality device, obtain the human state information of an individual based on the human sensing data collected by the human parameter sensing device, perform three-dimensional modeling on the environmental sensing data collected by the environmental parameter sensing device to obtain a three-dimensional environmental scene of the location where the individual is located. Display the extended reality scene, the human state information, and the three-dimensional environmental scene on the display unit of the extended reality device. Upload the human state information of the individual and the three-dimensional environmental scene to the cloud server.

[0008] In some embodiments of the present invention, the method further includes: obtaining the motion intention of an individual based on the human sensing data collected by the human parameter sensing device, and converting the human sensing data displaying the motion intention into a control command for a collaborative device; interacting with the collaborative device through changes in the human sensing data.

[0009] In some embodiments of the present invention, the method further includes: generating relevant evaluation information based on the human sensing data and the environmental sensing data, where the relevant evaluation information includes three-dimensional environmental scene evaluation information and individual behavior decision information.

[0010] In some embodiments of the present invention, the method further includes: the cloud server generating team behavior decision information based on the body state information and environmental sensing data of multiple individuals in a team, and generating team behavior suggestions based on the team behavior decision information.

[0011] In some embodiments of the present invention, the environmental sensing data includes positioning data and image data; the environmental parameter sensing device includes a camera and a radar; performing three-dimensional modeling on the environmental sensing data collected by the environmental parameter sensing device includes digitally three-dimensionally modeling the environmental data using simultaneous localization and mapping technology or three-dimensional reconstruction technology.

[0012] In some embodiments of the present invention, the method further includes: obtaining a training data set based on the historically collected data to train a machine model, and inputting the currently collected data into the trained machine model to output the relevant evaluation information via the trained machine model.

[0013] In some embodiments of the present invention, the three-dimensional modeling is performed in a manner of edge computing.

[0014] Another aspect of the present invention provides a human-computer interaction acquisition device for a wearable extended reality device, comprising:

[0015] An extended reality device integrated with a human parameter perception device and an environmental parameter perception device, configured to collect virtual reality basic signals; the human parameter perception device is used to sense and collect human perception data; the environmental parameter perception device is used to collect environmental perception data; a data processing module, configured to convert the virtual reality basic signals into an extended reality scene corresponding to the real scene; convert the collected human perception data into individual human state information; convert the collected environmental perception data into a three-dimensional environmental scene of the individual's location; a display module, configured to display the extended reality scene, human state information, and three-dimensional environmental scene obtained by the data processing module; a data transmission module, configured to upload the individual's human state information and the three-dimensional environmental scene of the individual's location to a cloud server.

[0016] Another aspect of the present invention provides a human-computer interaction acquisition system based on an extended reality device, comprising a processor and a memory, wherein computer instructions are stored in the memory, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the above-mentioned human-computer interaction acquisition method based on an extended reality device.

[0017] Another aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above-mentioned human-computer interaction acquisition method based on an extended reality device are implemented.

[0018] A human-computer interaction acquisition method, device, and system based on an extended reality device of the present invention integrate a human parameter perception device and an environmental parameter perception device into the extended display device, synchronously collect human perception data of the wearer of the extended display device, environmental perception data of the location, and virtual reality basic data, and synchronously present the obtained human state information, three-dimensional environmental scene, and extended reality scene on the display unit. The wearer of the extended display device can simultaneously obtain their own human state information, three-dimensional environmental scene of the location, extended reality scene, and real environment, and can always understand their own state and changes in the surrounding environment; in addition, the obtained human state information and three-dimensional environmental scene are transmitted to the cloud center, so as to always understand the human state information and three-dimensional environmental scene of partners through data sharing, facilitating the realization of human-machine collaborative work.

[0019] Additional advantages, objects, and features of the present invention will be partly set forth in the description which follows, and will partly become obvious to those of ordinary skill in the art after study of the following part, or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structure particularly pointed out in the specification and the drawings.

[0020] Those skilled in the art will understand that the objects and advantages that can be achieved by the present invention are not limited to those specifically described above, and the above and other objects that the present invention can achieve will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention, and form a part of this application, and do not limit the present invention. The components in the drawings are not drawn to scale, but are only for showing the principles of the present invention. For the convenience of showing and describing some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0022] Figure 1 It is a flowchart for human-computer interaction acquisition of an extended reality device.

[0023] Figure 2 It is a signal transmission flowchart of a human-computer interaction acquisition device for an extended reality device.

[0024] Figure 3 It is a structural diagram of information transmission and interaction for an extended reality device of a cloud server. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To make the objects, technical solutions, and advantages of the present invention more clear and understandable, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not limit the present invention.

[0026] Here, it should also be noted that in order to avoid obscuring the present invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the present invention are shown in the drawings, and other details less related to the present invention are omitted.

[0027] It should be emphasized that the term "comprising / including" when used herein refers to the presence of features, elements, steps, or components, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0028] Here, it should also be noted that if not otherwise specified, the term "connection" in this article can not only refer to direct connection, but also represent indirect connection with an intermediate.

[0029] In the following, embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0030] A human-computer interaction acquisition method for an extended reality device, as Figure 1 shown, includes steps S110-S140; the corresponding signal transmission process is as Figure 2 shown.

[0031] In step S110, data synchronously collected by an extended reality device integrated with a human parameter perception device and an environmental parameter perception device is obtained. The synchronously collected data includes virtual reality basic signals collected by the extended reality device, human perception data collected by the human parameter perception device, and environmental perception data collected by the environmental parameter perception device. The human perception data includes at least one of eye movement data, physiological parameters, and electroencephalogram data;

[0032] In the above step S110, based on the NTP (simultaneous localization and mapping) clock synchronization technology, an augmented reality device integrated with a human parameter perception device and an environmental parameter perception device is used to simultaneously collect virtual reality basic data, human perception data, and environmental perception data, and the virtual reality basic data, human perception data, and environmental perception data are collected on the same time axis, so as to understand in real time the human perception data of an individual over time, as well as the virtual reality basic data and environmental parameter perception data of the location where the individual is located. Among them, the virtual reality basic data includes information such as the positions and appearances of various things in the real environment collected by acquisition devices such as cameras and radars; the human perception data includes physiological signals such as eye movement, physiology, and electroencephalogram collected by human physiological signal acquisition devices such as sensors and radars; the environmental perception data includes data such as the positions, appearances, structures, and sizes of feature points in the real environment collected by environmental acquisition devices such as cameras and radars.

[0033] In one embodiment, the eye movement data is collected by the pupil-corneal reflection method to collect data on the actions and changes of the eyes such as the pupil diameter, blink count, blink frequency, and saccade behavior of an individual; the physiological data is collected by collecting relevant electrical signals or by photoplethysmography technology to collect physiological signals such as HR heart rate, EDA skin conductance response, and SKT skin temperature, and the electroencephalogram data is collected by the electroencephalogram dry electrode method to collect electroencephalogram signals such as the brain. The environmental perception data includes positioning data and image data; the environmental parameter perception device includes a camera and a radar.

[0034] In step S120, an extended reality scene is provided based on the virtual reality basic signals collected by the extended reality device, the human body state information of an individual is obtained based on the human body perception data collected by the human body parameter perception device, and 3D modeling is performed based on the environmental perception data collected by the environmental parameter perception device to obtain the 3D environmental scene of the location where the individual is located.

[0035] In the above step S120, providing the extended reality scene based on the virtual reality basic signals collected by the extended reality device includes: adding virtual extended information to the real environment according to actual needs on the basis of the collected virtual reality basic signals to form an extended reality scene. For example, information such as attention marks and additional items is added to the real environment.

[0036] Obtaining the human body state information of an individual based on the human body perception data collected by the human body parameter perception device includes: using the eye movement data such as the pupil diameter, blink count, blink frequency, and saccade behavior of the collected individual to evaluate the physical state information such as the fatigue degree and cognitive load of the individual; using the eye movement data such as the fixation point, fixation duration, and fixation trajectory of the collected individual to evaluate the mental state information such as the attention distribution and attention state of the individual; using the changes in indicators such as SDNN (standard deviation of all sinus cardiac cycle intervals) and LF / HF (sympathetic and vagus nerve dynamic balance index) of the collected HRV (heart rate variability), and the changes in indicators such as SC (skin conductance) of the skin electrical response of EDA (skin electroactivity) to evaluate the emotional state information such as excitement, calmness, tension, and agitation of the individual; the collected electroencephalogram signals are analyzed through frequency domain analysis, time domain analysis, PSD (power spectral density) electroencephalogram signal power spectrum analysis, etc. to evaluate the physical state information, mental state information, and emotional state information such as fatigue, excitement, and cognitive load of the individual.

[0037] Physiological signals are related to the body's peripheral autonomic activities and can reflect changes in a person's physical, mental, and emotional states, etc. The steps of obtaining an individual's human body state information based on the human perception data collected by the human parameter perception device described above include: training a machine model based on the training data set obtained from the historically collected data, and inputting the currently collected data into the trained machine model to output the human body state information corresponding to the currently collected data through the trained machine model. Among them, the machine model obtains a training data set based on the historical collection data of the human perception data collected by the wearable extended reality device, and the obtained machine model uses the human perception data collected in real time by the wearable extended reality device as the input value, and correspondingly outputs the human body state information at each moment. Among them, the steps of identifying human body state information by using human perception parameters such as eye movement, physiology, and electroencephalogram include: collecting the human perception data required to identify human body state information; preprocessing the collected human perception data and extracting feature data; using machine learning algorithms for data training; using methods such as data classifiers to predict the trained data to obtain the human body state information of the current individual. For example, when a person is in a state of fatigue, tension, stress, or excitement, their HRV signal will change significantly compared to the resting state. The steps of obtaining an individual's human body state information based on the HRV signal include: collecting the PPG (pulse) signal of the individual; processing the HRV signal through the collected PPG signal, and performing signal denoising processing by means such as wavelet denoising, high / low-pass filtering, band-stop filtering, ectopic interval detection, or ectopic interval correction; analyzing the HRV signal by means such as time-domain analysis, frequency-domain analysis, or nonlinear analysis, and extracting HRV time-domain, frequency-domain, and nonlinear characteristic values; classifying the data through algorithms such as SVM and random forest to obtain the human body state information of the corresponding individual, such as the stress level (low, medium, high), etc.

[0038] In one embodiment, a variety of human parameter perception devices are integrated in the wearable extended reality glasses, which can monitor the changes in the user's human perception data in real time, and evaluate the wearer's human body state information through built-in algorithms and threshold criteria, and display the evaluation result of the human body state information on the glasses in real time, and allow the person to evaluate and change their actions according to their actual situation. For example, the fatigue state, tension state, calm state, angry state, etc. of the wearer are displayed on the glasses.

[0039] The three-dimensional modeling based on the environmental perception data collected by the environmental parameter perception device to obtain the three-dimensional environmental scene where the individual is located includes: using simultaneous localization and mapping technology or three-dimensional reconstruction technology to perform digital three-dimensional modeling on the environmental perception data to obtain a three-dimensional environmental scene, where the three-dimensional modeling is carried out by means of edge computing, so as to ensure that phenomena such as slow data transmission and data transmission congestion do not occur due to excessive data volume.

[0040] Step S120 further includes: generating relevant evaluation information based on the human perception data and environmental perception data, where the relevant evaluation information includes three-dimensional environmental scene evaluation information and individual behavior decision information.

[0041] The step of generating relevant evaluation information based on the human perception data and environmental perception data includes: training a machine model using a training data set obtained from historical collected data, and inputting the currently collected data into the trained machine model, so as to output the relevant evaluation information via the trained machine model. Among them, the machine model obtains a training data set based on the historical collected data of human perception data and environmental perception data collected by the wearable extended reality device, and the obtained machine model uses the human perception data and environmental perception data collected in real time by the wearable extended reality device as input values, and correspondingly outputs the relevant evaluation information at each moment. The relevant evaluation information includes: three-dimensional environmental scene evaluation information and individual behavior decision information.

[0042] In one embodiment, when the human perception data includes eye movement data, the relevant evaluation information is three-dimensional environmental scene evaluation information. The generating relevant evaluation information based on the human perception data and environmental perception data includes: determining the fixation point of the eyes in the three-dimensional environmental scene based on the eye movement data and environmental perception data, and generating three-dimensional environmental scene evaluation information based on the determined fixation point according to a predetermined evaluation strategy. For example, the extended reality device synchronously collects the eye movement data of the fixation point of an individual in the environment and the environmental perception data of the position where the individual is located, and then simultaneously obtains the three-dimensional environmental scene where the individual is located and the change trajectory of the fixation point of the eyes in the three-dimensional environmental scene, forming the three-dimensional fixation point of the individual. By analyzing the three-dimensional fixation point of the individual, the change of the individual's focus of attention in the environment is analyzed, whether it is the effect desired by the environmental designer, so as to evaluate and analyze the design of the real scene.

[0043] In another embodiment, when the human perception data includes physiological parameters, the generating relevant evaluation information based on the human perception data and environmental perception data includes: generating individual behavior decision information by the cloud center server or the augmented reality device based on the individual's human state information and environmental perception data, generating individual behavior suggestions based on the individual behavior decision information, and outputting the individual behavior suggestions through the augmented reality device. For example, the extended reality device simultaneously collects the physiological parameters of an individual in the environment and the environmental perception data of the position where the individual is located, and then simultaneously obtains the individual's human state information and the three-dimensional environmental scene where the individual is located, to assist the user in judging their own situation and the environmental situation, so as to make a more rational decision. For example, display the user's own physical energy consumption, assist the user in judging their own mobility, so that the user can more rationally determine their own action range and rest time;

[0044] In another embodiment, when the human body perception data includes electroencephalogram data, generating relevant evaluation information based on the human body perception data and environmental perception data includes: generating individual behavior decision information by the cloud center server or the augmented reality device based on the individual's human body state information and environmental perception data, generating individual behavior suggestions based on the individual behavior decision information, and outputting the individual behavior suggestions through the augmented reality device. For example, the extended reality device simultaneously collects the electroencephalogram data of an individual and the environmental perception data of the location where the individual is located, and thus simultaneously obtains the individual's human body state information and the three-dimensional environmental scene of the location where the individual is located, determines whether the individual can perform the corresponding human-computer interaction task under the current electroencephalogram data, and determines whether the task execution device should accept the current task.

[0045] The above step S120 further includes: obtaining the movement intention of an individual based on the human body perception data collected by the human body parameter perception device, and converting the human body perception data showing the movement intention into a control command for the collaborative device; interacting with the collaborative device through the change of the human body perception data.

[0046] The human body parameter perception device collects human body parameters such as eye movement data, physiological parameters, and electroencephalogram data. Combining with the artificial intelligence algorithm of the data processing and analysis module, the human body perception data is converted into command signals for specific collaborative devices, and then the communication with specific collaborative devices is completed. Interact with specific collaborative devices through the changes of multi-modal human body perception data such as eye movement behavior, physiological state changes, and electroencephalogram changes, and realize the human-machine collaborative operation between personnel and machine systems such as unmanned aerial vehicles and the collaborative operation between personnel.

[0047] The steps of converting human perception data into command signals for specific collaborative devices include: collecting human perception data required to generate command signals; preprocessing the human perception data and extracting feature data; classifying the extracted feature data, where different classified feature data point to different motion intentions for the collaborative device; and converting each type of classified feature data into a control instruction for the collaborative device. For example, when a person imagines limb movement or muscle movement without actual movement output, specific brain regions of the person will still be activated. By analyzing electroencephalogram (EEG) signals, detecting and identifying the activation effects of different brain regions to judge the user's intention, and thus realizing direct communication and control between the human brain and external devices. Among them, the steps of converting EEG signals into control commands for collaborative devices include: collecting EEG data using a method of EEG dry electrodes; processing the collected EEG data and extracting feature data using algorithms such as power spectrum analysis, wavelet transform, autoregressive model (AR), sample entropy (SampEn), or common spatial pattern (CSP); classifying the extracted feature data using algorithms such as Linear Discriminant Analysis (LDA), Support Vector Machines (SVM), Artificial Neural Network (ANN), or Bayes Classifier to obtain multiple types of classified feature data; and converting the classified feature data into control commands for the collaborative device based on a motor imagery (MI) brain-computer interface system, a steady-state visual evoked potential (SSVEP) brain-computer interface system, or a P300-based brain-computer interface system, thereby completing human-machine interaction.

[0048] In one embodiment, a wearable extended reality device collects an individual's physiological parameters, and through analyzing and processing changes in the user's physiological state, realizes the interaction between the person and the auxiliary operation device to complete collaborative operations; for example, by collecting and analyzing the individual's heart rate changes, identifying the individual's current emotional state, and judging whether the auxiliary operation device needs to take over the current task by determining whether the individual is currently suitable for performing the operation.

[0049] In another embodiment, a wearable extended reality device collects an individual's eye movement data, and through analyzing and processing the changes in the eye movement data, communicates with a specific collaborative device to realize the human-machine collaborative operation between the person and the auxiliary operation device; for example, identifying the eye fixation duration and fixation position data and performing related operations on this point.

[0050] In another embodiment, the wearable extended reality device collects the electroencephalogram data of an individual, and through analyzing and processing the changes in brain waves, realizes the interaction between the personnel and the auxiliary operation device to complete collaborative operations. For example, by collecting and analyzing the changes in the brain waves of the personnel, the auxiliary operation device can execute corresponding tasks according to the electroencephalogram information of the personnel, including controlling the direction of the operation device, executing the task status, etc.

[0051] In step S130, the extended reality scene, the human body state information, and the three-dimensional environment scene are displayed on the display unit of the extended reality device.

[0052] The display unit of the extended reality device uses a light-transmitting picture presentation medium. The wearer of the extended reality device can observe the real environment through the reality unit and can move freely at a specific location. One or more of the extended reality scene, the human body state information, and the three-dimensional environment scene are selectively presented on the display unit, where the extended display scene and the three-dimensional environment scene are overlapped according to the positions and fixed forms of the corresponding fiducial points. At the same time, based on the positions and fixed forms of the fiducial points in the real environment, the display picture in the reality unit is overlapped with the real environment. The wearer of the extended reality device can simultaneously see one or more of the extended reality scene, the three-dimensional environment scene, and the real scene, so as to help the wearer understand the surrounding environment more clearly and deeply. The human body state information is presented on the reality unit, so as to help the wearer understand their own state more clearly and timely and reasonably adjust their states such as movement, spirit, and emotion, and adjust their action plans. For example, relying on the multi-modal data of the wearer of the extended reality device collected, personnel state recognition is carried out, such as fatigue, high load, excitement, anger, etc., and the state recognition results are reflected in the extended reality device in real time, including imaging display on the display unit, voice reminder, vibration reminder, etc.

[0053] In the above step S130, the human body state information and the three-dimensional environment scene of other extended reality devices from the cloud server can also be displayed on the display unit of the extended reality device.

[0054] In step S140, the human body state information of the individual and the three-dimensional environment scene are uploaded to the cloud server.

[0055] The human body state information of the wearer of the extended reality device and the three-dimensional environment scene are uploaded to the cloud server. All collaborative operation personnel and commanders can obtain the human body state information and three-dimensional environment scene of each other through the cloud server, so as to facilitate work such as team message collaboration and sharing, real-time update of strategies, and adjustment of team personnel division of labor; remote control of devices can also be realized through human perception data to achieve remote human-computer interaction. Among them, for application scenarios with a large number of operation personnel, the information transmission and interaction structure between the cloud server and each extended reality device is asFigure 3 as shown

[0056] In one embodiment, the extended reality device can obtain the three-dimensional data coordinates of the physical world in the wearer's line of sight in real time, perform real-time digital modeling on the physical world, and upload it to the cloud server in real time, so that remote commanders and other personnel using the device can share this geographical information.

[0057] The above step S140 further includes: generating team behavior decision information by the cloud server based on the physical state information and environmental perception data of multiple individuals in the team, and generating team behavior suggestions based on the team behavior decision information.

[0058] In one embodiment, the human body state data of the user wearing the extended reality device is obtained and sent to the team leader to enable the team leader to grasp the physical conditions of the team members in real time, thereby assisting the team leader to make more scientific and rational decisions. For example, enabling the team leader to grasp the physical energy consumption of all team members in real time. If the physical energy consumption of most team members has reached 80%, the team is instructed to stop and find a place to rest.

[0059] Corresponding to the above method, a human-computer interaction acquisition device for a wearable extended reality device is an integrated device, mainly in the form of glasses, and includes:

[0060] An extended reality device integrated with a human body parameter perception device and an environmental parameter perception device, which is used to collect virtual reality basic signals; the human body parameter perception device is used to sense and collect human perception data; the environmental parameter perception device is used to collect environmental perception data; the human perception data includes at least one of eye movement data, physiological parameters, and electroencephalogram data. Among them, the infrared light source and camera required for collecting eye movement data, the light source and electrodes required for collecting physiological parameters, the electroencephalogram dry electrodes for collecting electroencephalogram data, and the camera and radar for collecting environmental perception data can all be integrated in the wearable extended display device, and data collection is not restricted by the environment and conditions.

[0061] A data processing module, which is used to convert the virtual reality basic signal into an extended reality scene corresponding to the real scene; convert the collected human perception data into individual human body state information; convert the collected environmental perception data into a three-dimensional environmental scene of the individual's location; and is also used to convert the collected human perception data into a control command for a collaborative device; generate relevant evaluation information according to the human perception data and environmental perception data, and the relevant evaluation information includes three-dimensional environmental scene evaluation information and individual behavior decision information.

[0062] A display module, configured to display the extended reality scene, human body state information, and three-dimensional environment scene obtained by the data processing module; and is further configured to display the human body state information and three-dimensional environment scene of other extended reality devices from the cloud server.

[0063] A data transmission module, configured to upload the human body state information of an individual and the three-dimensional environment scene of the location where the individual is located to the cloud server.

[0064] Corresponding to the above method, the present invention further provides a human-computer interaction acquisition system for a wearable extended reality device. The system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the human-computer interaction acquisition method of the wearable extended reality device as described above.

[0065] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the human-computer interaction acquisition method of the wearable extended reality device as described above are implemented. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0066] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment may be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link.

[0067] It should be clear that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0068] In the present invention, the features described and / or illustrated for one embodiment can be used in the same or similar manner in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0069] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A human-computer interaction acquisition method for a wearable extended reality device, characterized in that It includes the following steps: Obtain the data synchronously collected by an extended reality device integrated with a human parameter perception device and an environmental parameter perception device. The synchronously collected data includes the virtual reality basic signal collected by the extended reality device, the human perception data collected by the human parameter perception device, and the environmental perception data collected by the environmental parameter perception device. The human perception data includes eye movement data, physiological parameters, and electroencephalogram data; Provide an extended reality scene based on the virtual reality basic signal collected by the extended reality device, obtain the human state information of an individual based on the human perception data collected by the human parameter perception device, perform three-dimensional modeling based on the environmental perception data collected by the environmental parameter perception device to obtain the three-dimensional environmental scene of the location where the individual is located; generate relevant evaluation information based on the human perception data and the environmental perception data. The relevant evaluation information includes three-dimensional environmental scene evaluation information and individual behavior decision information; Obtain the motion intention of an individual based on the human perception data collected by the human parameter perception device, and convert the human perception data showing the motion intention into a control command for a collaborative device, so as to interact with the collaborative device through the change of the human perception data; among them, the human state information includes physical state information, mental state information, and emotional state information; Display the extended reality scene, the human state information, and the three-dimensional environmental scene on the display unit of the extended reality device; Upload the human state information of an individual and the three-dimensional environmental scene to a cloud server, so that the cloud server generates team behavior decision information based on the human state information and environmental perception data of multiple individuals in the team, and generates team behavior suggestions based on the team behavior decision information.

2. The method according to claim 1, characterized in that, The environmental perception data includes positioning data and image data; the environmental parameter perception device includes a camera and a radar; the three-dimensional modeling based on the environmental perception data collected by the environmental parameter perception device includes digitally three-dimensionally modeling the environmental data by using simultaneous localization and mapping technology or three-dimensional reconstruction technology.

3. The method according to claim 1, characterized in that, The method further includes: obtaining a training data set based on the historically collected data to train a machine model, and inputting the currently collected data into the trained machine model to output the relevant evaluation information via the trained machine model.

4. The method according to claim 1, wherein The three-dimensional modeling is performed in a manner of edge computing.

5. A human-computer interaction acquisition device for a wearable extended reality device, characterized in that, It includes: An extended reality device integrated with a human parameter perception device and an environmental parameter perception device, which is used to collect virtual reality basic signals; The human parameter perception device is used to sense and collect human perception data; the environmental parameter perception device is used to collect environmental perception data; A data processing module, which is used to convert the virtual reality basic signal into an extended reality scene corresponding to the real scene; convert the collected human perception data into the human state information of an individual; convert the collected environmental perception data into the three-dimensional environmental scene of the location where the individual is located; generate relevant evaluation information based on the human perception data and the environmental perception data. The relevant evaluation information includes three-dimensional environmental scene evaluation information and individual behavior decision information; Obtain the motion intention of an individual based on the human perception data collected by the human parameter perception device, and convert the human perception data showing the motion intention into a control command for the collaborative device, so as to interact with the collaborative device through the change of the human perception data; wherein, the human state information includes physical state information, mental state information and emotional state information; A display module, configured to display the extended reality scene, human state information and three-dimensional environment scene obtained by the reality data processing module; A data transmission module, configured to upload the human state information of an individual and the three-dimensional environment scene of the location where the individual is located to the cloud server, so that the cloud server generates team behavior decision information based on the human state information and environmental perception data of multiple individuals in the team, and generates team behavior suggestions based on the team behavior decision information.

6. A human-computer interaction acquisition system for a wearable extended reality device, characterized in that, Comprising a processor and a memory, characterized in that computer instructions are stored in the memory, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 4.

Images