Multi-modal man-machine interaction platform and electronic equipment

Abstract

The invention relates to a multi-modal man-machine interaction platform and electronic equipment. The multi-modal man-machine interaction platform is characterized in that a physical keyboard, a handwriting touch interaction screen, a handwriting pen, a microphone array, an SLAM sensor, a chip, an operating system and a docking station are integrated into the man-machine interaction platform; three high-frequency keyboards of a mobile phone, a computer and a scientific calculator are uniformly mapped to a physical keyboard; voice interaction, pen interaction, keyboard interaction, touch interaction and gesture interaction are integrated in a man-machine interaction platform, and the AR glasses and the SLAM technology are combined, so that multi-modal interaction is realized in a man-machine interaction environment in which a real space is combined with a virtual space.

Description

Technical Field

[0001] The present invention relates to the technical field of spatial computing and human-computer interaction, and in particular to a multimodal human-computer interaction platform and electronic equipment. Background Art

[0002] As virtual reality (VR), mixed reality (MR) and augmented reality (AR) technologies extend people's work, study and entertainment from real space to virtual space, the way of human-computer interaction has also changed. VR, MR and AR are collectively referred to as XR (extended reality). XR requires the use of handles and gestures (XR bare hand tracking + motion capture) to achieve human-computer interaction. The Vision Pro released by Apple in 2024 ushered in the era of spatial computing. Vision Pro is an MR (mixed reality) head-mounted all-in-one machine that uses VST (video see-through) technology to achieve virtual and real fusion. VST (video see-through) is the acronym for Video See Through. VST superimposes virtual images on real-time video streams in the real world, allowing users to view augmented reality content through head-mounted displays, thereby achieving virtual and real fusion. Unlike MR headsets, AR (augmented reality) glasses use the OST technology route to achieve augmented reality. OST is the acronym for Optical See Through. OST projects virtual images into the user's field of view through a transparent optical display module, allowing users to see both the real world and augmented reality content at the same time. At the experience level, the difference in perception brought by these two technical solutions is very obvious. In VST, users ultimately see a video that combines real and virtual content, while in OST, users see a real world mixed with virtual content. The Vision Pro head-mounted MR all-in-one integrates up to 23 sensors, high-computing chips and Vision OS operating system, including microphone arrays, inertial measurement units (IMUs), optical sensors, infrared sensors and lidars, etc. These sensors are used to implement VST, and the data collected by the sensors are used to implement SLAM (simultaneous localization and mapping) and spatial computing. SLAM (Simultaneous Localization and Mapping) is a technology that simultaneously realizes the positioning of the device itself and the construction of an environmental map. The principle of SLAM technology is to use sensors such as depth cameras, lidars, inertial measurement units (IMUs) to collect environmental information, and then use algorithms to fuse this information to determine the location of the device in an unknown environment and build an environmental map. SLAM technology is divided into lidar SLAM and visual SLAM. Lidar SLAM uses 2D or 3D laser radar, and visual SLAM uses optical and infrared sensors, which are currently mainly monocular cameras, binocular cameras, and RGBD cameras. SLAM technology gives Vision Pro a very good gesture interaction experience, but because Vision Pro integrates all components into the head-mounted all-in-one solution, its overall weight is as high as more than 600 grams, which is very uncomfortable to wear for a long time.AR glasses have chosen a split technology route, retaining only the optical machine (optical projection imaging unit), inertial measurement unit (IMU) and a small number of video and audio components; separating the sensors, chips and operating system from the head-mounted device, thereby greatly reducing the weight, generally not exceeding 100 grams, and can be worn for a long time. But at the same time, it brings another problem: the lack of sensors such as depth cameras to collect environmental information makes it difficult to achieve gesture interaction based on SLAM technology. XR handles play an important role in MR and AR interactions. Although various interactions can be completed in virtual space with the help of XR handles or gestures (XR bare hand tracking + motion capture), the input of text, numbers, and punctuation marks in the input method, if the ray emitted by the XR handle (presented as a beam in the virtual space) is used to click and select the XR virtual keyboard (mobile phone keyboard / computer keyboard / calculator keyboard) and symbol table in the virtual space (three-dimensional space), or the XR virtual keyboard is operated with gestures (XR bare hand tracking + motion capture), unlike the mobile phone touch screen virtual keyboard, the XR virtual keyboard has no tactile feedback, and its user experience is worse than the experience of using a mouse to click on the virtual keyboard and symbol table in the real space on the PC display to input characters, and the efficiency is lower. The three frequently used keyboards of mobile phones, computers, and scientific calculators should be uniformly mapped to the physical keyboard instead of using the XR virtual keyboard. The tactile feedback of the physical keyboard and the handwriting and hand drawing of the stylus on the plane interactive screen in the real space are irreplaceable by other XR interaction methods. For MR headsets and AR glasses to become productivity tools, they cannot do without physical keyboards + stylus + plane interactive screens. Today's mobile phones, tablets, and laptops are becoming increasingly thinner and more portable, but this leads to another problem: the general lack of expansion interfaces. The technical solution provided by the present invention integrates voice interaction, pen interaction, keyboard interaction, touch interaction, and gesture interaction into a human-computer interaction platform to achieve multimodal interaction; and integrates multiple sensors, chips, and expansion interfaces, relying on SLAM technology and spatial computing, and cooperating with AR glasses to achieve gesture interaction. In short, the technical solution provided by the present invention effectively solves the problems encountered by the above MR head displays and AR glasses in human-computer interaction. Summary of the invention

[0003] The purpose of the present invention is to provide a technical solution for multimodal human-computer interaction to solve the problems encountered in the above-mentioned XR human-computer interaction, uniformly map the three frequently used keyboards of mobile phones, computers, and scientific calculators to a physical keyboard, integrate voice interaction, pen interaction, keyboard interaction, touch interaction, and gesture interaction (XR bare hand tracking + motion capture) into a human-computer interaction platform to achieve multimodal interaction; and integrate multiple SLAM sensors, chips and operating systems, as well as expansion interfaces (expansion docks), relying on SLAM technology and spatial computing, and cooperating with AR glasses to achieve gesture interaction.

[0004] The technical solution adopted by the present invention includes the following contents:

[0005] Integrate the physical keyboard, handwriting touch interactive screen, stylus, microphone array, SLAM sensor, chip, operating system, and docking station into a human-computer interaction platform; uniformly map the three frequently used keyboards of mobile phones, computers, and scientific calculators to the physical keyboard, integrate voice interaction, pen interaction, keyboard interaction, touch interaction, and gesture interaction (XR bare hand tracking + motion capture) into a human-computer interaction platform, and cooperate with AR glasses to achieve multimodal interaction in a human-computer interaction environment that combines real space and virtual space.

[0006] On the other hand, the present invention also provides an electronic device, comprising:

[0007] Main device;

[0008] A processor, disposed in the main device;

[0009] The multimodal human-computer interaction platform is connected to the processor.

[0010] Compared with the prior art, the beneficial effects of the present invention are: three frequently used keyboards of mobile phones, computers, and scientific calculators are uniformly mapped to physical keyboards. Voice interaction, pen interaction, keyboard interaction, touch interaction, gesture interaction (XR bare hand tracking + motion capture) are all integrated into a human-computer interaction platform to achieve multimodal interaction. The integrated docking station makes it convenient for users to connect multiple devices and bring a better experience. The multimodal human-computer interaction platform + AR glasses combine the real space with the virtual space to achieve large screen + multiple screens + gesture interaction, one button controls multiple screens, and one screen controls multiple screens, greatly improving the interaction efficiency.

[0011] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings.

[0012] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A first embodiment of the present invention is shown.

[0014] Figure 2 A second embodiment of the present invention is shown.

[0015] Figure 3 A third embodiment of the present invention is shown.

[0016] Figure 4 A top view of a third embodiment is shown.

[0017] Figure 5Shown are bottom views of three embodiments of the present invention.

[0018] Figure 6 Shown is a schematic diagram of the working principle of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0020] Figure 1 The figure shows a schematic diagram of the first embodiment of the present invention, including: a handwriting touch interactive screen 1, an expansion dock 2, a microphone 3, a stylus 4, a monocular optical camera 5, a microphone 6 (together with the microphone 3 to form a microphone array), and a single-key area physical keyboard 7. The flat interactive screen adopts a 12-hour handwriting touch screen, which interacts with multiple facade screens in real space and virtual space, and one screen controls multiple screens. The microphone array has a far-field sound pickup and noise reduction function. The stylus adopts an active capacitive pen or an electromagnetic pen, and a lanyard is tied at the end of the stylus to prevent loss, and it is placed in the card slot when not in use. The monocular camera cooperates with the SLAM algorithm to realize gesture interaction. SLAM is a simultaneous localization and mapping technology (Simultaneous Localization and Mapping). As a visual SLAM sensor, the monocular optical camera has the advantages of being cheap and light, but the disadvantage is that it cannot obtain depth information. The monocular camera is fixed on the multimodal human-computer interaction platform by a bracket and can be rotated left and right to facilitate the selection of a suitable angle to realize gesture interaction. The multimodal human-computer interaction platform comes with its own CPU and operating system, which integrates keyboard interaction, pen interaction, voice interaction, touch screen interaction, and gesture interaction (XR bare hand tracking + motion capture) to achieve multimodal interaction with XR. AR glasses are different from MR headsets. To be lightweight, SLAM sensors, chips, and operating systems need to be integrated into the multimodal human-computer interaction platform. The multimodal human-computer interaction platform maps the three frequently used keyboards of mobile phones, computers, and scientific calculators to the physical keyboard. One key controls multiple screens, and one screen controls multiple screens, which can greatly improve the efficiency of human-computer interaction combining real space and virtual space.

[0021] Figure 2The second embodiment of the present invention is shown as a schematic diagram, including: a handwriting touch interactive screen 1, an expansion dock 2, a microphone 3, a stylus 4, a monocular optical camera 5, a microphone 6 (together with the microphone 3, forming a microphone array), a monocular optical camera 7 (together with the monocular optical camera 5, forming a binocular optical camera), and a single key area physical keyboard 8. The flat interactive screen adopts a 12-hour handwriting touch screen, which interacts with multiple facade screens in real space and virtual space, and one screen controls multiple screens. The microphone array has a far-field sound pickup and noise reduction function. The stylus adopts an active capacitive pen or an electromagnetic pen, and a lanyard is tied at the end of the stylus to prevent loss, and it is placed in the card slot when not in use. The binocular camera cooperates with the SLAM algorithm to realize gesture interaction. SLAM is a simultaneous localization and mapping technology (Simultaneous Localization and Mapping). The binocular optical camera is used as a visual SLAM sensor. The binocular SLAM uses the parallax of the left and right objects to calculate the distance of pixels, thereby realizing its own positioning. Stereoscopic vision can estimate depth both when moving and when stationary, eliminating the trouble of monocular vision being unable to obtain depth information. The camera is fixed on the multimodal human-computer interaction platform by a bracket and can be rotated left and right to facilitate the selection of a suitable angle for gesture interaction. The multimodal human-computer interaction platform comes with a CPU and operating system, which integrates keyboard interaction, pen interaction, voice interaction, touch screen interaction, and gesture interaction (XR bare hand tracking + motion capture) to achieve multimodal interaction with XR. AR glasses are different from MR headsets. To be lightweight, SLAM sensors, chips, and operating systems need to be integrated into the multimodal human-computer interaction platform.

[0022] Figure 3 The figure shows a schematic diagram of the third embodiment of the present invention, including: a handwriting touch interactive screen 1, an expansion dock 2, a microphone 3, a stylus 4, an optical plus infrared depth camera 5 (RGBD depth camera), a microphone 6 (together with the microphone 3 to form a microphone array), and a single-key area physical keyboard 7. The flat interactive screen adopts a 12-hour handwriting touch screen, which interacts with multiple facade screens in real space and virtual space, and one screen controls multiple screens. The microphone array has a far-field sound pickup and noise reduction function. The stylus adopts an active capacitive pen or an electromagnetic pen. A lanyard is attached to the end of the stylus to prevent loss, and it is placed in the card slot when not in use. The depth camera shown in the figure is in a horizontally lying position. The bracket of the depth camera can adjust the pitch angle within a range of 90 degrees. When used in conjunction with a PC, in order to avoid blocking the PC monitor and affecting the display effect, the bracket of the depth camera can be adjusted to a horizontally lying position. The RGBD camera uses an infrared camera to obtain depth information in a three-dimensional space. The characteristic is that it uses infrared structured light or Time - Based on the principle of-Fl ight, the distance between each pixel in the image and the camera is directly measured to achieve more accurate synchronous positioning and spatial calculation.

[0023] Figure 4 The top view of the third embodiment is shown, including: a handwriting touch interactive screen 1, a docking station 2, a microphone 3, a stylus 4, an optical plus infrared depth camera 5 (RGBD camera), a microphone 6 (together with the microphone 3 to form a microphone array), and a single-key area physical keyboard 7. The microphone array has a far-field sound pickup and noise reduction function. The depth camera shown in the figure is in a vertical standing position (the position state of the depth camera when working). When used in conjunction with AR glasses to realize gesture interaction, the bracket of the depth camera is adjusted to a vertical standing position. The depth camera at the top of the bracket can be rotated left and right to facilitate the selection of a suitable angle to realize gesture interaction. The XR virtual keyboard is different from the mobile phone touch screen virtual keyboard. There is no tactile feedback, the user experience is very poor, and it is even less efficient than the input of the physical keyboard. The multimodal human-computer interaction platform uniformly maps the three frequently used mobile phone, computer, and scientific calculator keyboards to the physical keyboard. One key controls multiple screens, and one screen controls multiple screens, which can greatly improve the efficiency of human-computer interaction combining real space and virtual space.

[0024] Figure 5 The bottom view of the first, second and third embodiments of the present invention is shown, and the specific position of the expansion dock 2 is marked with a dotted line, which corresponds to the expansion dock 2 in the above three embodiments (the effect seen from the rear view). Since the expansion dock 2 is located at the rear view position of the multimodal human-computer interaction platform, it cannot be seen from the perspective of the three-dimensional view in the first, second and third embodiments. Figure 4 The top view of the third embodiment is not visible. Combining the bottom view and the rear view (partial), it can be seen that the expansion dock contains multiple USB ports; as mobile phones, tablets and laptops become thinner and more portable, the lack of expansion ports has become a pain point for users. The multimodal human-computer interaction platform integrates the expansion dock to facilitate users to connect multiple devices and bring a better experience.

[0025] Figure 6 Schematic diagram of the working principle of three embodiments of the present invention. Figure 6As shown, it is presented in the form of a general-purpose computing device. Its components include: operating system 1, various applications 2, ROM (read-only memory) 3, storage unit 4, RAM (random access memory) 5, cache 6, processor unit 7, bus 8 connecting different components, I / O interface 9, display unit 10, network adapter 11. Among them, the storage unit stores the operating system and application code, and the application code is executed by the processing unit 7. The processing unit 7 can execute a variety of applications including SLAM algorithms to realize gesture interaction (XR bare hand tracking + motion capture) and multimodal interaction. Keyboard, pen, touch screen, sound sensor (microphone array), visual SLAM sensor (optical camera and infrared camera) are all external devices. The multimodal human-computer interaction platform comes with a CPU and operating system. Although it can be used as an independent terminal device alone (equivalent to a PC), its main task is to cooperate with AR glasses to realize multimodal human-computer interaction. AR glasses are different from MR head displays. To be lightweight, SLAM sensors, chips and operating systems need to be integrated into the multimodal human-computer interaction platform.

[0026] The three embodiments provided by the present invention all have a plural form of real part + imaginary part, the real part corresponds to a physical keyboard, and the imaginary part corresponds to a flat interactive screen. Combined with AR glasses and SLAM technology, multimodal interaction is realized in the human-computer interaction combining real space and virtual space, one key controls multiple screens, and one screen is empty for multiple screens, greatly improving the interaction efficiency. SLAM technology is divided into laser radar SLAM and visual SLAM. Laser radar SLAM (Lidar SLAM) uses 2D or 3D laser radar, and visual SLAM uses optical and infrared sensors, mainly monocular camera, binocular camera, RGBD camera. The above three embodiments use cheap and lightweight visual SLAM, omitting Lidar SLAM, but there are embodiments that use Lidar SLAM. The above embodiments all have the following technical effects: the three high-frequency keyboards of mobile phones, computers, and scientific calculators are uniformly mapped to physical keyboards. Voice interaction, pen interaction, keyboard interaction, touch interaction, and gesture interaction (XR bare hand tracking + motion capture) are all integrated into a human-computer interaction platform to achieve multimodal interaction. The integrated docking station facilitates users to connect multiple devices and bring a better experience. The multimodal human-computer interaction platform + AR glasses rely on SLAM to achieve gesture interaction, combining real space with virtual space, realizing large screen + multiple screens + gesture interaction, one button controls multiple screens, and one screen controls multiple screens, greatly improving interaction efficiency.

[0027] The above is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A multimodal human-computer interaction platform, characterized in that: include: Integrate the physical keyboard, handwriting touch interactive screen, stylus, microphone array, SLAM sensor, chip, operating system, and docking station into a human-computer interaction platform; uniformly map the three frequently used keyboards of mobile phones, computers, and scientific calculators to the physical keyboard; integrate voice interaction, pen interaction, keyboard interaction, touch interaction, and gesture interaction into a human-computer interaction platform, and combine AR glasses and SLAM technology to achieve multimodal interaction in a human-computer interaction environment that combines real space and virtual space.

2. The multimodal human-computer interaction platform according to claim 1, characterized in that: This multimodal human-computer interaction platform uses a monocular optical camera, combined with AR glasses and SLAM technology to achieve gesture interaction.

3. The multimodal human-computer interaction platform according to claim 1, characterized in that: This multimodal human-computer interaction platform uses binocular optical cameras, combined with AR glasses and SLAM technology to achieve gesture interaction.

4. The multimodal human-computer interaction platform according to claim 1, characterized in that: This multimodal human-computer interaction platform uses an RGBD depth camera, combined with AR glasses and SLAM technology to achieve gesture interaction.

5. The multimodal human-computer interaction platform according to claim 1, characterized in that: The multimodal human-computer interaction platform integrates an expansion dock to facilitate users to connect to multiple devices.

6. The multimodal human-computer interaction platform according to claim 1, characterized in that: This multimodal human-computer interaction platform has its own CPU and operating system, and combines AR glasses and SLAM technology to achieve multimodal interaction in a human-computer interaction environment that combines real space and virtual space.

7. The multimodal human-computer interaction platform according to claim 4, characterized in that: The RGBD depth camera on the multimodal human-computer interaction platform adopts a bracket with adjustable pitch angle. The RGBD depth camera is fixed on the top of the bracket and can be rotated left and right to facilitate the selection of a suitable angle to realize gesture interaction.

8. An electronic device, characterized in that include: Main device; A processor, disposed in the main device; A multimodal human-computer interaction platform as described in any one of claims 1 to 7, connected to the processor.

Images