Information interaction method, system, storage medium, processor and device

Abstract

Embodiments of the present application provide an information interaction method, system, storage medium, processor and device, and belong to the technical field of Internet of Things. The information interaction method comprises: acquiring an image frame corresponding to real information; determining a user interface corresponding to object information identified from the image frame according to the object information, and controlling the user interface to be overlaid and displayed on the image frame, wherein the user interface comprises a visual stimulation element; acquiring an electroencephalogram signal in response to the user interface, wherein the electroencephalogram signal is acquired through visual attention to the visual stimulation element; and analyzing the electroencephalogram signal, and controlling the user interface to respond to the electroencephalogram signal according to an analysis result. Through the information interaction method of the present application, the user interface can be controlled through visual attention in the case that both hands are occupied.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things, in particular to an information interaction method and system, a storage medium, a processor and a device. BACKGROUND

[0002] Augmented Reality (AR) technology aims to enhance human ability and provide various auxiliary information for human, and becomes an important medium for communication between human individuals and information world. At present, it has played a role in medical treatment, transportation, aerospace, communication, industrial maintenance and other fields. Based on computer real-time calculation and multi-sensor fusion, it combines the real world with virtual information. Computer vision technology (CV) is based on the perception of images to make useful decisions about objective objects and scenes, and visual recognition is a key component of computer vision, such as image classification, positioning and detection. Some augmented reality head-mounted systems have promoted augmented reality to a mixed reality experience (Mixed Reality, MR) through the visual perception ability of CV technology to the real world.

[0003] However, the AR scene interaction in the prior art depends on traditional interaction means, such as gestures, voice, controllers, etc. In industrial scenes, it is often difficult to trigger AR interface interaction when both hands are occupied for performing double-hand operation, or it is difficult to complete AR interface interaction due to image recognition / voice recognition difficulties caused by complex environment; the visual evoked potential VEP (Visual Evoked Potential) brain-computer interaction based on the AR interface completes the menu selection interaction on the AR interface, and cannot complete the mixed reality interaction based on environmental perception and analysis ability. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a method which realizes a more natural and efficient AR interaction form by combining VEP with computer vision and augmented reality technology.

[0005] In order to achieve the above-mentioned purpose, the embodiments of the present application provide an information interaction method, comprising:

[0006] obtaining an image frame corresponding to real information;

[0007] determining a user interface corresponding to the object information according to the object information identified from the image frame, and controlling the user interface to be overlaid and displayed on the image frame, wherein the user interface comprises a visual stimulation element;

[0008] obtaining an electroencephalogram signal in response to the user interface, wherein the electroencephalogram signal is obtained through visual attention to the visual stimulation element; and

[0009] analyze the electroencephalogram signal, and according to the analysis result, control the user interface to respond to the electroencephalogram signal.

[0010] The information interaction method combines VEP, CV and AR technology, extracts the electroencephalogram signal of the user in response to the visual stimulation element in the user interface from the electroencephalogram signal of the user wearing the brain-computer device, and analyzes the electroencephalogram signal to obtain the control intention of the user to the user interface, which corresponds to the disposal intention of the to-be-recognized object in the video stream collected in the real scene. The method realizes the control of the AR user interface through visual attention, provides a hands-free and little affected by complex environment form of realizing the interaction of real and virtual information, and provides a natural and convenient interactive experience for the control task in which both hands cannot participate.

[0011] Optionally, the visual stimulation element is defined based on a visual evoked potential (VEP) paradigm.

[0012] Preferably, the visual stimulation element can be selected and triggered through visual attention.

[0013] Optionally, the determination of the user interface corresponding to the object information and / or the analysis of the electroencephalogram signal are performed by the brain-computer device, a local server or a cloud server.

[0014] Optionally, the image frame is directly collected by an AR camera or is intercepted from the collected video stream.

[0015] Optionally, the user interface is an augmented reality interface.

[0016] Optionally, the object information is information of a mechanical device to be repaired; during the repair process of the mechanical device, the user interface changes according to the change of the repair state of the mechanical device.

[0017] Further, the user interface includes information of internal structures and parts of the mechanical device.

[0018] The types of the visual stimulation elements in the user interface include: confirmation of the parts, state marking of the user interface and jumping and switching of the user interface; and 、

[0019] The marking states of the parts include: normal, repaired and to be repaired.

[0020] In another aspect, the present application provides an information interaction system, comprising:

[0021] A first acquisition module is configured to acquire image frames corresponding to real information.

[0022] ​a user interface determining module configured to determine a user interface corresponding to the object information according to the object information identified from the image frame, and control the user interface to be overlaid and displayed on the image frame, wherein the user interface comprises a visual stimulation element;

[0023] a second obtaining module configured to obtain an electroencephalogram signal in response to the user interface, wherein the electroencephalogram signal is obtained through visual attention to the visual stimulation element; and

[0024] an analysis processing module configured to analyze the electroencephalogram signal, and control the user interface to respond to the electroencephalogram signal according to an analysis result.

[0025] Optionally, the visual stimulation element is defined based on a visual evoked potential (VEP) paradigm.

[0026] Preferably, the visual stimulation element can be selected and triggered through visual attention.

[0027] Optionally, the user interface determining module and / or the analysis processing module are located in a brain-computer device, a local server or a cloud server.

[0028] Optionally, the image frame is directly acquired through an AR camera, or is acquired by being intercepted from a video stream.

[0029] The user interface is an augmented reality interface.

[0030] Optionally, the object information is information of a mechanical device to be repaired; and during repair of the mechanical device, the user interface changes according to changes in a repair state of the mechanical device.

[0031] Further, the user interface comprises information of an internal structure and parts of the mechanical device.

[0032] Types of the visual stimulation element in the user interface comprise: confirmation of the parts 、 state marking and jump and switching of the user interface; and

[0033] The marking state of the parts comprises: normal, repaired and to be repaired.

[0034] In another aspect, the present application provides a machine readable storage medium, which stores instructions for causing a machine to execute the information interaction method of the present application.

[0035] In another aspect, the present application provides a processor for running a program, wherein the program is used to execute the information interaction method of the present application when being run.

[0036] In another aspect, the present application provides an apparatus comprising the machine readable storage medium of the present application, or the processor of the present application.

[0037] By the above technical solution, the brain electrical signals of the user responding to the visual stimulation elements in the user interface are extracted and analyzed from the brain electrical signals of the user of the brain-computer device in combination with VEP, CV and AR technologies, and the user interface is controlled to respond to the brain electrical signals according to the analysis result, the AR user interface can be controlled through the visual stimulation, a form of interaction of real and virtual information affected little by complex environment is provided, and the form of interaction provides a natural and convenient interactive experience for control tasks in which hands cannot be involved. The method is applied to scenes such as sports, industrial operation and medical surgery, can identify and simulate the state change of an object to be identified, and realizes mixed reality interaction based on environmental perception and analysis capability.

[0038] Other features and advantages of the embodiments of the present application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are included to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used together with the following specific implementation to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the drawings:

[0040] Figure 1 is a flowchart of an embodiment of the information interaction method of the present application;

[0041] Figure 2 is a constituent schematic diagram of an embodiment of the information interaction method of the present application;

[0042] Figure 3 is a constituent schematic diagram of another embodiment of the information interaction method of the present application;

[0043] Figure 4 is a user interface control schematic diagram of an embodiment of the information interaction system of the present application for mechanical equipment maintenance; and

[0044] Figure 5 and 6 is Figure 4 an embodiment of the information interaction system in DETAILED DESCRIPTION

[0045] The specific implementation of the embodiments of the present application will be described in detail below in combination with the drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiments of the present application, and is not used to limit the embodiments of the present application.

[0046] The specific implementation of the embodiments of the present application will be described in detail below in combination with the drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiments of the present application, and is not used to limit the embodiments of the present application. Figures 1-3The information interaction method of the present application is described. It should be noted that Figure 1 The information interaction method flowchart shown covers the process of data interaction of both the terminal and the server. In actual implementation, it can be selected to complete the user interface display, the electroencephalogram signal acquisition, and the electroencephalogram signal data analysis and processing in one integrated system, or to select the implementation mode in which the terminal display module, the electroencephalogram signal acquisition module, and the electroencephalogram signal data analysis processor are three independent modules of the user interface. Figure 2 In the implementation mode shown, the user interface display, the electroencephalogram signal acquisition, and the electroencephalogram signal data analysis and processing are completed in one integrated system. Figure 3 In the implementation mode of the user interface shown, the terminal display module, the electroencephalogram signal acquisition module, and the electroencephalogram signal data analysis processor are three independent modules.

[0047] In this embodiment, a visual trigger real and virtual information interaction method is implemented, and the implementation process is as shown in Figure 1 The specific implementation steps are as follows:

[0048] Step 1: Obtain an image frame corresponding to real information.

[0049] In this embodiment, a video stream corresponding to real information can also be acquired in real time, and an image frame can be obtained from the video stream.

[0050] In this embodiment, the collected real information is a scene of repairing a small car, and the terminal is an Android tablet computer. The video stream is acquired by an AR camera of the terminal.

[0051] In some implementation modes, the application scenario can also be a sports, industrial operation, medical surgery, construction engineering, etc. The terminal can also be an AR glasses, smart glasses, etc. terminal device, or an image / video acquisition module of an integrated system as shown in Figure 2

[0052] Step 2: According to the object information identified from the image frame, determine a user interface corresponding to the object information, and control the user interface to be overlaid and displayed on the image frame, wherein the user interface includes a visual stimulation element.

[0053] In this embodiment, the to-be-identified object in the image frame / video stream is the small car to be repaired and the parts and equipment on the small car. The tablet computer terminal can directly identify the to-be-identified object in the acquired video stream, or send the video stream to the server for identification, and determine the corresponding user interface according to the identification result. The user interface is an augmented reality interface / a user interface reflecting real information. The user interface is displayed through an Android tablet computer, and a video stream shot on site and a user interface defined and published in advance are loaded simultaneously. The user interface includes a pre-defined visual stimulation element, and the visual stimulation element can be selected and triggered through visual attention.

[0054] In some implementation modes, the visual stimulation element is defined based on a visual evoked potential VEP paradigm.​

[0055] Step 3: obtaining the brain electrical signals in response to the user interface, wherein the brain electrical signals are obtained through visual attention to the visual stimulus elements.

[0056] In this embodiment, the brain electrical signals are obtained through an on-site occipital brain-computer device, the user of the brain-computer device observes the visual stimulus elements in the user interface and actively selects through visual attention; the occipital brain-computer device collects the corresponding brain electrical signals and transmits the data to the program in the Android tablet through Bluetooth.

[0057] Step 4: analyzing the brain electrical signals, and controlling the displayed user interface in response to the brain electrical signals according to the analysis results.

[0058] In this embodiment, the Android program transmits the brain electrical signal data to the cloud server, and the VEP algorithm deployed on the cloud server analyzes and judges the brain electrical data, identifies the visual stimulus elements actively selected by the maintenance personnel through visual attention from the brain electrical signal data, judges the control intention of the maintenance personnel to the user interface, and controls the terminal to respond to the control intention, for example, the control intention can be to call up the corresponding page feedback or trigger the corresponding interface switching.

[0059] In some embodiments, the Android tablet can also execute the data analysis processing program, at this time, there is no need to transmit the data to the cloud server, but to execute the VEP algorithm to analyze and judge the brain electrical data locally.

[0060] Compared with the prior art, the advantages of this embodiment are:

[0061] (1) combining visual brain-computer interface technology, computer vision technology and AR interaction technology, the visual stimulus elements in the user interface selected by the user of the brain-computer device through visual attention are extracted and analyzed from the brain electrical signals of the user, and the user interface is controlled to respond to the brain electrical signals according to the analysis results, providing a natural, convenient, efficient, environment-perceiving and analyzing hybrid reality interaction technology framework;

[0062] (2) providing an interactive form of reality and virtual information that is less affected by complex environments, which provides a natural and convenient interactive experience for control tasks that cannot be performed by both hands;

[0063] (3) the method of this embodiment is applied to sports, industrial operations, medical operations and other scenes, can identify and simulate the state changes of the to-be-identified objects, and realize hybrid reality interaction based on environment perception and analysis capabilities.

[0064] The application also provides an embodiment of a visual trigger-based reality and virtual information interaction system, which will be described below in combination with Figures 4-6The embodiment is applied to a specific implementation of mechanical equipment maintenance.

[0065] The visual trigger real and virtual information interaction system of the embodiment has a structure as shown in Figure 5 , which includes a PAD, a cloud server and a brain-computer device. The brain-computer device is an external device using a visual evoked potential (VEP) technology and can be used to collect brain electrical signals of a wearer. The cloud server is pre-arranged with a video stream recognition processing service program, a computer vision algorithm service program and a VEP algorithm service program. The PAD is an Android tablet and is pre-configured with an AR camera, a Bluetooth transmission component and a user interface program package.

[0066] In the embodiment, the PAD is used to execute the information interaction method for a terminal of the application, and the cloud server is used to execute the information interaction method for a server of the application. In a maintenance process, a data stream of a user interface is as shown in Figure 5 . First, the PAD terminal collects a video stream corresponding to real information in real time through a camera and transmits the video stream to the cloud server. The cloud server identifies components and position coordinates in a scene to be maintained at present through a computer vision algorithm service program, controls the PAD terminal to call a corresponding user interface, and the user interface includes visual stimulation elements and the visual stimulation elements are displayed around the identified components.

[0067] The flickering stimulation in the user interface can be observed by a user equipped with the brain-computer device. A data stream diagram after the flickering stimulation is actively selected by the user is as shown in Figure 6 . First, the PAD terminal receives brain electrical signals collected by the brain-computer device in response to the user interface and transmits the brain electrical signals to the cloud server. The cloud server analyzes the brain electrical signals after receiving the brain electrical signals and controls the PAD terminal to respond to the brain electrical signals according to an analysis result. The PAD terminal controls a displayed user interface to respond according to the analysis result after receiving the algorithm analysis result from the server.

[0068] The user interface and a control process of the embodiment are as shown in Figure 4 , which includes a start page and process pages of a plurality of maintenance tasks. Jumping between different process pages is needed according to actual maintenance process requirements. Figure 4 According to a legend shown in the upper left corner of Figure 4 , a gray area of different shapes displayed on each page represents a VEP visual stimulation element, which can also be referred to as a visual stimulation area.

[0069] In the embodiment, the visual stimulation area on the page is also marked with a currently selectable maintenance task, a state identifier of the current maintenance task, and a selectable next maintenance task, a user of the brain-computer device actively selects the visual stimulation area on the page through visual attention, after the brain electrical signal stimulated by the visual stimulation area is received by the VEP algorithm program, the brain electrical signal is analyzed and judged, and according to the algorithm result, a corresponding feedback interface on the user interface is called up or a corresponding page jump is triggered.

[0070] The core implementation step of the embodiment includes: calling the camera of the tablet computer, and performing part identification and state marking of the trolley through the tablet computer screen interface. The specific interaction task is to confirm the part to be maintained in the maintenance order specified by the system through the visual stimulation module and enter normal maintenance work, mark the state of the part after completing the maintenance of the part: normal, repaired, and to be repaired, and then sequentially confirm and mark the next part.

[0071] Specifically, the implementation steps are:

[0072] The camera of the tablet computer is called, and the video stream shot is transmitted to the cloud server;

[0073] The CV algorithm deployed on the cloud server analyzes the video information, identifies the parts, and sends the part name and position coordinate results to the Android program on the tablet computer;

[0074] The Android program calls up the corresponding UI interface and visual stimulation module according to the received results, and superimposes the corresponding UI interface on the tablet computer display interface (the presented content is the content shot by the camera) to realize the overlap of virtual information and the real world;

[0075] The user observes the visual stimulation module and actively selects through visual attention;

[0076] The occipital brain-computer device collects the corresponding brain electrical signal, and transmits the data to the Android program through Bluetooth, and then the Android program transmits the data to the cloud server;

[0077] The VEP algorithm deployed on the cloud server analyzes and judges the brain electrical data, and sends the algorithm result back to the Android program;

[0078] The Android program calls up the corresponding feedback UI or triggers the corresponding interface switching on the user interface.

[0079] The embodiment of the application provides a storage medium, which stores a program, and the program is executed by a processor to realize the information interaction method.

[0080] The embodiment of the present application provides a processor, which is used for running a program, wherein the information interaction method is executed when the program is running.

[0081] The embodiment of the present application provides a device, which comprises a processor, a memory, and a program stored in the memory and capable of running on the processor, and the processor executes the program to realize the information interaction method. The device herein can be a server, a PC, a PAD, a mobile phone, or the like.

[0082] The present application also provides a computer program product, which is suitable for executing the program initialized with any one of the processes according to the present application when executed on a data processing device. Figures 1-3

[0083] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0084] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices generate a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0085] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices, which realize the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0086] ​These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0087] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0088] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or other memory technologies, CD-ROM, digital versatile disc (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. In no case does the medium include a transitory signal.

[0089] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0090] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0091] ​​The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. An information interaction method, characterized in that, include: Acquire image frames that correspond to real-world information; Based on the object information identified from the image frame, a user interface corresponding to the object information is determined, and the user interface is controlled to be overlaid on the image frame. The object information is the information of the mechanical equipment to be repaired. During the repair of the mechanical equipment, the user interface changes according to the repair status of the mechanical equipment. The user interface includes visual stimuli and information on the internal structure and parts of the mechanical equipment. Acquire electroencephalogram (EEG) signals in response to the user interface, wherein the EEG signals are acquired through visual attention to the visual stimulus elements; and The electroencephalogram (EEG) signals are analyzed, and based on the analysis results, the user interface is manipulated to respond to the EEG signals, either by displaying the page feedback of the mechanical equipment to be repaired or by triggering a corresponding interface switch. The manipulation intent corresponds to the intention to handle the object to be identified in the video stream captured in the real-world scene.

2. The information interaction method according to claim 1, characterized in that, The visual stimulus elements are defined based on the visual evoked potential (VEP) paradigm.

3. The information interaction method according to claim 1, characterized in that, The visual stimuli can be selected and triggered through visual attention.

4. The information interaction method according to claim 1, characterized in that, The determination of the user interface corresponding to the object information and / or the analysis of the EEG signals are performed by a brain-computer interface device, a local server, or a cloud server.

5. The information interaction method according to claim 1, characterized in that, The image frames are obtained directly by an AR camera or by being extracted from a video stream. The user interface is an augmented reality interface.

6. The information interaction method according to claim 1, characterized in that, The types of visual stimuli in the user interface include: confirmation of the part, status markers, and navigation / switching of the user interface; and The marking status of the parts includes: normal, repaired, and pending repair.

7. An information interaction system, comprising: The first acquisition module is used to acquire image frames corresponding to real-world information; The user interface determination module is used to determine the user interface corresponding to the object information identified from the image frame, and control the user interface to be overlaid on the image frame, wherein the object information is the information of the mechanical equipment to be repaired; during the repair of the mechanical equipment, the user interface changes according to the repair status of the mechanical equipment, and the user interface includes visual stimulation elements and information on the internal structure and parts of the mechanical equipment; The second acquisition module is configured to acquire electroencephalogram (EEG) signals in response to the user interface, wherein the EEG signals are acquired through visual attention to the visual stimulus elements; and The analysis and processing module is used to analyze the electroencephalogram (EEG) signals and, based on the analysis results, obtain the control intentions for the user interface, control the user interface to respond to the EEG signals, and bring up the page feedback of the mechanical equipment to be repaired or trigger the corresponding interface switching. The manipulation intent corresponds to the intention to handle the object to be identified in the video stream captured in the real-world scene.

8. The information interaction system according to claim 7, characterized in that, The visual stimulus elements are defined based on the visual evoked potential (VEP) paradigm.

9. The information interaction system according to claim 7, characterized in that, The visual stimuli can be selected and triggered through visual attention.

10. The information interaction system according to claim 7, characterized in that, The user interface determination module and / or the analysis and processing module are located on the brain-computer interface device, a local server, or a cloud server.

11. The information interaction system according to claim 7, characterized in that, The image frames are obtained directly by an AR camera or by being extracted from a video stream. The user interface is an augmented reality interface.

12. The information interaction system according to claim 7, characterized in that, The types of visual stimuli in the user interface include: confirmation of the part, status markers, and navigation / switching of the user interface; and The marking status of the parts includes: normal, repaired, and pending repair.

13. A machine-readable storage medium storing instructions that cause a machine to perform: the information interaction method as described in any one of claims 1-6.

14. A processor, characterized in that, Used to run a program, wherein the program is run to perform: the information interaction method as described in any one of claims 1-6.

Images