Interaction method and system based on handheld intelligent device and head-mounted display device

Abstract

The invention provides an interaction method and system based on a handheld intelligent device and a head-mounted display device. The method comprises the steps that virtual content corresponding to the handheld intelligent device is displayed on a display screen of the head-mounted display device; operating the posture change of the virtual content through the head-mounted display device and / or operating the position change of the virtual content through the handheld intelligent device; and interacting data between the head-mounted display device and the handheld intelligent device to enable the posture change of the virtual content to be transmitted to the handheld intelligent device and / or the position change of the virtual content to be transmitted to the head-mounted display device.

Description

Technical Field

[0001] The present invention belongs to the field of information interaction, and in particular relates to an interaction method and system based on a handheld smart device and a head-mounted display device. Background Art

[0002] The statements in this section are merely intended to provide background information related to the present invention to help understanding the present invention. Such background information does not necessarily constitute prior art.

[0003] Augmented Reality (AR) is a technology that superimposes virtual information on the real world. In AR applications, the commonly used display interaction devices are handheld smart devices (such as mobile phones, tablets) and head-mounted display devices (such as AR glasses, smart glasses, AR helmets). Virtual content to be displayed to users can be presented on the display screen of a handheld smart device or a head-mounted display device, and users can interact with these virtual contents. Virtual content can be, for example, icons, pictures, text, emoticons, virtual people, virtual three-dimensional objects, three-dimensional models, an animation, a video, and so on. In AR applications, handheld smart devices and head-mounted display devices will determine their position and / or posture information (hereinafter collectively referred to as "posture information") in real time, and determine which virtual content to present and how to present these virtual contents based on the position and / or posture information.

[0004] Six degrees of freedom (6DOF) AR and three degrees of freedom (3DOF) AR are two different types of AR technology. 3DOF AR provides three degrees of freedom, which allows users to rotate in the horizontal plane (yaw, pitch) and in the vertical direction (roll), but does not allow users to translate in space. In 3DOF AR, the virtual content seems to be fixed in a certain position and cannot change position as the user moves, that is, 3DOF AR only tracks the user's posture changes. 6DOF AR provides six degrees of freedom, including three translational degrees of freedom (front and back, left and right, up and down) and three rotational degrees of freedom, which allows users to move freely around the virtual content, and the position and direction of the object will adjust accordingly according to the user's perspective. 6DOF AR is able to track the user's position and posture changes (collectively referred to as "posture changes").

[0005] Compared with 3DOF AR, 6DOF AR can provide a richer interactive experience and a stronger sense of immersion, and is more suitable for applications that require precise spatial positioning and high interactivity. At present, 6DOF AR has been applied and explored in many fields such as game entertainment, education and training, health care, industrial manufacturing, interior design, military defense, navigation and tourism, remote office and virtual office. However, 6DOF AR requires more advanced tracking technology (such as simultaneous localization and mapping (SLAM)), as well as more complex sensors and algorithms to achieve precise spatial positioning and mapping, so it requires higher performance hardware, such as displays with high refresh rates and high resolutions, powerful processors and advanced sensors, which will undoubtedly increase the complexity of the equipment, reduce the compatibility of the equipment and cause high costs. Summary of the invention

[0006] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an interaction method based on a handheld smart device and a head mounted display device, the method comprising:

[0007] Displaying virtual content corresponding to the handheld smart device on a display screen of the head mounted display device;

[0008] Operating the posture change of the virtual content through the head mounted display device and / or operating the position change of the virtual content through the handheld smart device; and

[0009] Data is exchanged between the head mounted display device and the handheld smart device so that a posture change of the virtual content is transmitted to the handheld smart device and / or a position change of the virtual content is transmitted to the head mounted display device.

[0010] The method according to the present invention preferably further comprises:

[0011] A first coordinate system is established for the head mounted display device, and a second coordinate system is established for the handheld smart device.

[0012] The posture change of the virtual content in the first coordinate system is operated by the head mounted display device and / or the position change of the virtual content in the second coordinate system is operated by the handheld smart device.

[0013] According to the method of the present invention, preferably, a first coordinate system is established with the head mounted display device as the origin, and a second coordinate system is established with the handheld smart device as the origin.

[0014] According to the method of the present invention, preferably, data is exchanged between the head-mounted display device and the handheld smart device so that the posture data of the virtual content in the first coordinate system is transmitted to the handheld smart device and / or the position data of the virtual content in the second coordinate system is transmitted to the head-mounted display device.

[0015] The method according to the present invention preferably further comprises:

[0016] The head mounted display device changes the position of its own virtual content based on the position change of the virtual content from the handheld smart device; and / or

[0017] The handheld smart device changes the posture of its own virtual content based on the posture change of the virtual content from the head mounted display device.

[0018] According to the method of the present invention, preferably, the handheld smart device is operated to reset the viewing angle so that the current direction angle of the head mounted display device is consistent with the direction angle of the handheld smart device.

[0019] The method according to the present invention preferably further comprises:

[0020] The user of the handheld smart device clicks a location in the virtual space on the screen of the handheld smart device;

[0021] The handheld smart device extracts the two-dimensional position data of the position and transmits it to the head mounted display device; and

[0022] The head mounted display device performs monocular rendering and displays virtual information at a corresponding two-dimensional position on its display screen based on the two-dimensional position data.

[0023] The method according to the present invention preferably further comprises:

[0024] Selecting a reference object on the screen of the handheld smart device;

[0025] The user of the head mounted display device looks at the reference object;

[0026] The handheld smart device transmits a calibration command to the head mounted display device;

[0027] The head mounted display device acquires its own posture data information and calculates the display position of the reference object in the head mounted display device based on the posture data information;

[0028] The head mounted display device calculates the position deviation between the display position and the physical center of the head mounted display device and transmits it to the handheld smart device; and

[0029] The handheld smart device corrects the actual gaze direction of the user of the head mounted display device based on the position deviation.

[0030] According to the method of the present invention, preferably, it also includes adjusting parameters of the head-mounted display device through the handheld smart device to enhance the visual effects of the virtual content of the head-mounted display device in one or more aspects.

[0031] According to the method of the present invention, preferably, the handheld smart device is operated to record the change process of the virtual content.

[0032] On the other hand, the present invention provides a storage medium storing a computer program, which, when executed by a processor, can be used to implement the interaction method based on a handheld smart device and a head-mounted display device according to the present invention.

[0033] On the other hand, the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it can be used to implement the interaction method based on a handheld smart device and a head-mounted display device according to the present invention.

[0034] On the other hand, the present invention provides an interactive system, including a handheld smart device and a head-mounted display device, wherein the handheld smart device and the head-mounted display device can be used to implement the interactive method based on the handheld smart device and the head-mounted display device according to the present invention.

[0035] Compared with the prior art, the present invention realizes 6DOF AR through the interaction between a tablet computer and AR glasses, does not require high-performance hardware equipment, and provides a richer interactive experience and a stronger sense of immersion at a lower cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0037] Figure 1 An interactive system based on a handheld smart device and a head mounted display device according to an embodiment of the present invention;

[0038] Figure 2 An interaction method based on a tablet computer and AR glasses according to an embodiment of the present invention;

[0039] Figure 3 A method for resetting viewing angle according to an embodiment of the present invention;

[0040] Figure 4 A method for visual calibration according to an embodiment of the present invention; and

[0041] Figure 5The present invention provides a multi-person interaction method based on a tablet computer and AR glasses. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] Figure 1 The interactive system based on a handheld smart device and a head mounted display device according to an embodiment of the present invention is shown, and the interactive system includes a tablet computer 103 of a user 101 and an AR glasses 104 of a user 102. The user 101 and the user 102 are located in a scene, the user 101 holds a tablet computer 103 in his hand, and the user 102 wears an AR glasses 104 on his head, and can experience various augmented reality or virtual reality applications through the tablet computer 103 and the AR glasses 104. In the system, the AR glasses 104 have a display screen, which is mainly used to present various virtual contents in augmented reality or virtual reality applications to the user 102 through the display screen. The tablet computer 103 includes at least two windows, an operation window W1 and a preview window W2. The operation window W1 is mainly used to implement interactive functions in various augmented reality or virtual reality applications, such as clicking an operation menu, clicking an operation button, selecting virtual content, placing virtual content, deleting virtual content, aiming at virtual content, clicking virtual content, moving virtual content, drawing virtual works, inputting information, etc.; the preview window W2 is used to show the virtual content presented in the AR glasses 104 to the user 101. The tablet computer 103 and the AR glasses 104 of the system establish communication in a wired or wireless manner known in the art. The virtual camera of the AR glasses 104 can generate and render virtual content in the field of vision of the user 102 according to the data transmitted by the tablet computer 103.

[0044] The position changes and / or posture changes of the tablet computer 103 and the AR glasses 104 can be measured or tracked by their respective sensors (for example, accelerometer, magnetic sensor, orientation sensor, gravity sensor, gyroscope, camera, etc.) through methods known in the art (for example, inertial navigation, visual odometer, SLAM, VSLAM, SFM, etc.), thereby determining the real-time position and / or posture of the tablet computer and the head-mounted display device in space.

[0045] In an embodiment of the present invention, the AR glasses 104 are 3DOF AR glasses, which can track the posture changes of the user 101 to manipulate the posture changes of the virtual content, and the tablet computer 103 can manipulate the position changes of the virtual content. The AR glasses 104 are combined with the tablet computer 103 to realize six-degree-of-freedom changes of the virtual content, including yaw, pitch and roll posture changes and position changes along the x-axis, y-axis and z-axis.

[0046] For example, Figure 1 As shown, the user 101 selects the virtual content letter “A” to be presented through the operation window W1 of the tablet computer 103 , and the letter “A” is synchronously presented in the preview window W2 of the tablet computer 103 and the display screen of the AR glasses 104 worn by the user 102 .

[0047] When the posture of the letter "A" needs to be changed, the AR glasses 104 realizes the posture change of the letter "A" through head tracking technology or eye tracking technology. For example, when the user turns his head, the letter "A" will perform yaw rotation accordingly; when the user tilts his head or changes his eye sight, the letter "A" will roll and rotate; when the user moves his head up and down, the letter "A" will pitch and rotate. In another embodiment, the user can also control the rotation of the virtual content through gestures, voice commands, buttons or other input devices. The AR glasses 104 transmits this rotation of the virtual content to the tablet computer 103 through the network, so that the virtual content in the preview window W2 of the tablet computer 103 also rotates synchronously. For example, the AR glasses 104 transmits the updated posture data of the virtual content to the tablet computer 103, and the tablet computer 103 changes the posture of its virtual content based on the updated posture data.

[0048] When the position of the letter "A" needs to be changed, the position of the virtual content in the virtual space along the x-axis, y-axis, and z-axis is changed through single-finger or multi-finger touch screen operation, gesture recognition, voice command, key operation, etc. of the tablet computer 103. The tablet computer 103 transmits this operation behavior to the AR glasses 104 through the network, so that the position of the virtual content in the display screen of the AR glasses 104 moves. For example, the tablet computer 103 transmits the updated position data of the virtual content to the AR glasses 104, and the AR glasses 104 changes the position of its virtual content based on the updated position data. The touch screen operation, gesture recognition, voice command operation, and key operation technology of the tablet computer 103 are well known in the art and will not be repeated here.

[0049] In addition, the size of the virtual content can also be changed through touch screen operation, gesture operation, voice operation or button operation of the tablet computer 103 or through voice, button, touch bar, gesture, eye and other operations on the AR glasses 104.

[0050] Figure 2 An interactive method based on a tablet computer and AR glasses according to an embodiment is shown. In this method, respective coordinate systems are established for the tablet computer and the AR glasses. The tablet computer is mainly used to control the position change of virtual content, and the AR glasses are mainly used to control the posture change of virtual content. The two cooperate to realize 6DOF tracking of virtual content. The method includes:

[0051] Step 201: Establish a first coordinate system for the AR glasses and a second coordinate system for the tablet computer.

[0052] Preferably, the first coordinate system is established with the AR glasses as the origin, and the second coordinate system is established with the tablet computer as the origin. After the coordinate origin is determined, the coordinate system is established with the horizontal direction as the x-axis, the vertical direction as the y-axis, and the depth direction as the z-axis.

[0053] In an embodiment of the present invention, taking the AR glasses as the origin includes taking the user's pupil center (i.e., the starting point of the user's line of sight) as the origin, taking the optical center of the AR glasses (i.e., the center point of the glasses lens) as the origin, taking the geometric center of the user's head as the origin, taking the user's current standing position as the origin, taking the environmental feature point as the origin, taking the user interaction point (e.g., the object touched or pointed by the user) as the origin, etc. In particular, the origin can be dynamically adjusted according to the user's perspective and interaction to provide the best user experience. In addition, taking the tablet computer as the origin includes taking the geometric center or screen center of the tablet computer as the origin, taking a corner of the tablet computer's screen (e.g., the upper left corner) as the origin, taking the touch point as the origin, taking the view center of the tablet computer's camera as the origin, taking the tablet computer's spatial position (e.g., the bottom center or the back center) as the origin, taking the environmental feature point as the origin, taking the center of the interactive interface as the origin, and taking the user-defined feature point as the origin. In particular, the origin can change dynamically according to the data of the inertial measurement unit (IMU) or other sensors.

[0054] In one embodiment, the coordinate system of the AR glasses is preferably aligned with the actual environment. Specifically, using, for example, SLAM technology, the camera and other sensors on the AR glasses capture environmental information in real time and build a 3D model of the environment, extract feature points from the 3D model of the environment for tracking and positioning, and combine IMU (inertial measurement unit), GPS and other sensor data to provide more accurate position and direction information, and ensure that the coordinate system of the AR glasses is aligned with the actual environment through initial calibration.

[0055] Step 202: Present the same virtual content on the AR glasses and the tablet computer.

[0056] The virtual content is selected by the operator of the tablet computer, for example, through a touch screen, buttons, voice, etc., and transmitted to the AR glasses through the network and presented on the display screen of the AR glasses. At the same time, the preview window of the tablet computer also displays the virtual content. The virtual content may include 2D images, 3D models, animations, videos or other visual elements, which are superimposed on the real world seen by the user through the AR glasses in the field of view of the AR glasses.

[0057] In one embodiment, the virtual camera of the AR glasses can generate and render virtual content in the user's field of view based on the data transmitted by the tablet computer. In some embodiments, when the AR glasses render or present virtual content on their display screens, it is necessary to know the position and posture information of the virtual content relative to the AR glasses in advance so that it can be correctly rendered or presented. Based on the present invention, by aligning the coordinate system of the AR glasses with the actual environment, when receiving data from the tablet computer, the virtual content can be correctly rendered or presented.

[0058] In one embodiment, the user of the AR glasses interacts with the virtual content through head movement, line of sight movement, gestures, voice commands, or a dedicated controller. In addition to visual elements, the AR experience can also include audio feedback and even tactile feedback to enhance immersion.

[0059] In one embodiment, the AR system of the AR glasses is able to simulate real-world lighting conditions, including shadows and reflections, to make virtual content appear more natural.

[0060] Step 203: Control the posture change of the virtual content through the AR glasses and / or control the position change of the virtual content through the tablet computer.

[0061] For the presented virtual content, when its posture needs to be changed, the user wearing the AR glasses interacts with the virtual content through head movement, line of sight movement, gestures, voice commands or a dedicated controller, and specifically realizes the posture change of the virtual content, such as yaw, roll or pitch, through head tracking technology or eye tracking technology.

[0062] In one embodiment, sensors in AR glasses are used to track the position and direction of the user's head; when the user's head moves to a specific position or reaches a certain angle, the rotation of the image is triggered; by analyzing the head tracking data, the rotation of the user's head is identified, for example, when the user turns his head to the right, the yaw rotation of the head around the vertical axis can be identified; the rotation of the head is mapped to the virtual content, which means that when the user's head rotates to the right, for example, the virtual content will also rotate accordingly; determine the angle of rotation of the virtual content, for example, if the user's head turns 30 degrees, the virtual content can be set to rotate 30 degrees; and according to the tracking data of the head rotation, the rendering of the virtual content is updated in real time so that it can correctly display the effect of the rotation in the user's field of view. In a further embodiment, visual or auditory feedback is provided to let the user know that their head movement has been recognized and triggered the image rotation. In addition, the user is allowed to control the rotation of the image through specific head movements (such as turning the head quickly) to increase interactivity.

[0063] In one embodiment, an eye tracking system is used to capture the user's pupil position and line of sight direction to determine the screen position or virtual content that the user is currently looking at; the user's line of sight is mapped into a virtual environment or interface to determine the point where the line of sight intersects or the virtual content being looked at; a rotation action is triggered when the line of sight moves to a specific area or reaches a specific angle, for example, when the user's line of sight moves from the center to the right edge of the screen, the image rotation is triggered; the movement of the user's line of sight is identified; based on the distance or angle of the line of sight movement, the angle that the virtual content needs to rotate is calculated, for example, if the user's line of sight moves from the center to the edge of the screen, it may correspond to a 30-degree rotation; the calculated rotation amount is applied to the virtual content to achieve the rotation effect by updating the rotation attribute of the image; and based on the eye tracking data and the rotation amount, the rendering of the virtual content is updated in real time to ensure that the rotation of the image is synchronized with the movement of the user's line of sight. In a further embodiment, visual or auditory feedback is provided to let the user know that their line of sight movement has been identified and triggered the image rotation. In addition, the user can also be allowed to adjust the sensitivity and rotation speed of the eye tracking control according to personal preferences.

[0064] In other embodiments, the rotation of AR virtual content is achieved by using voice, gestures, etc. These technologies are well known in the art and will not be described in detail here.

[0065] For the presented virtual content, when its position along the x-axis, y-axis and z-axis needs to be changed, the position of the virtual content in the virtual space can be changed by single-finger or multi-finger touch screen operation of the tablet computer (e.g., by dragging gestures on the touch screen). In another embodiment, the movement of the image is triggered by recognizing a specific gesture, such as a palm slide or a finger circle. In another embodiment, the image is moved by voice commands. In another embodiment, the image is moved by pressing a key.

[0066] In one embodiment, the size of the virtual content may be changed by operating the tablet computer, for example, by using a pinch gesture with multiple fingers to change the size of the virtual content.

[0067] Step 204: Exchange data between the AR glasses and the tablet computer.

[0068] The tablet computer and the AR glasses establish communication in a wired or wireless manner, for example, through a Wi-Fi network, Bluetooth, a cellular network, a Mesh network, a cloud service, etc. These communication methods are well known in the art and will not be described in detail here.

[0069] When the AR glasses change the posture of the virtual content, the posture change will be transmitted to the tablet, so the virtual content displayed on the tablet will also change accordingly. For example, when the virtual content is rotated 90° by moving the head, the virtual content presented by the AR glasses and the virtual content displayed on the tablet will both rotate 90°.

[0070] When the position of the virtual content operated by the tablet changes, this position change will also be transmitted to the AR glasses, so the virtual content presented by the AR glasses will also change accordingly. For example, when the virtual content displayed by the tablet is dragged by the touch screen to move 5 meters along the x-axis of the tablet's coordinate system, the virtual content presented by the AR glasses will also move 5 meters along the x-axis of the AR glasses' coordinate system.

[0071] Through the collaborative operation of AR glasses and tablet computers, both the virtual content presented in AR glasses and the virtual content displayed in tablet computers have six degrees of freedom and can achieve 6DOF tracking. This 6DOF tracking can be achieved with only 3DOF AR and tablet computers, without the need for advanced tracking technology and complex algorithms, which significantly reduces the complexity and cost of the equipment and improves the compatibility of the equipment.

[0072] In order to achieve realistic stereoscopic visual effects, binocular rendering technology is usually used, that is, slightly different images are generated and displayed for the left eye and the right eye respectively. In one embodiment, one of the binoculars of the AR glasses is used as the main eye (e.g., the right eye) and the other is used as the auxiliary eye (e.g., the left eye). This setting can be based on the user's personal habits or specific visual calibration requirements. The first coordinate system is established with the optical center of the main eye as the origin, and the optical center of the auxiliary eye is positioned in the first coordinate system, usually translated along the x-axis for a distance, which is related to the average distance between the user's pupils (IPD). The principle of binocular parallax is used, that is, the images observed by the two eyes due to different positions have slight differences, and this difference is interpreted by the brain as depth information to form stereoscopic vision. In particular, according to the different perspectives of the main eye and the auxiliary eye, the rendering engine generates two sets of images respectively. These images are adjusted according to the perspective of each eye to reflect the actual three-dimensional spatial relationship. The AR system accurately calculates the relative position between the virtual content and the user's eyes through spatial mapping and environmental understanding technology to ensure that the depth information of the rendered image matches the objects in the real world. In this embodiment, the user's brain fuses the images received by the left and right eyes to form a single stereoscopic vision with depth information, enhancing the realism and immersion of the virtual content.

[0073] Usually, the gravity sensor in AR glasses can only give the pitch angle and roll angle (also called roll angle in this article) of AR glasses relative to the gravity direction, but cannot give the yaw angle (also called direction angle in this article) of AR glasses. In addition, the gyroscope in AR glasses can only obtain angular velocity but not the initial direction angle, which will make it impossible for the wearer to start the AR glasses in the correct spatial orientation, thereby reducing the visual experience. For this reason, point clouds, image anchors or positioning base stations are usually used to provide the initial direction angle, but these methods have strong environmental dependence, poor accuracy, high requirements on infrastructure, poor security and high technical complexity.

[0074] In an embodiment of the present invention, in order to ensure that the user starts the AR glasses at the correct initial azimuth angle and obtains a consistent and coordinated visual experience when using augmented reality (AR) smart glasses and tablet computers, the system provides a function of resetting the viewing angle, which is used to adjust the coordinate system of the AR glasses and the coordinate system of the tablet computer to a consistent direction so that the viewing angles of the AR glasses and the tablet computer are the same. In one embodiment of the present invention, the initial orientation angle of the AR glasses is set to 0, and the angular velocity obtained by the gyroscope of the AR glasses is integrated to obtain the change amount of the attitude angle of the AR glasses, including the change amount of the pitch angle, roll angle and orientation angle. Based on the initial pitch angle and initial roll angle obtained by the gravity sensor of the AR glasses and the set initial orientation angle of 0, the attitude angle of the AR glasses can be continuously obtained. The attitude angle of the virtual camera of the AR glasses is consistent with the attitude angle of the AR glasses at the initial moment. When the wearer of the AR glasses is active, the attitude angle of the AR glasses is continuously and dynamically updated, and the attitude angle of the virtual camera of the AR glasses is also refreshed synchronously, which will cause the orientation angle of the virtual camera of the AR glasses in the first coordinate system to be inconsistent with the orientation angle of the virtual camera of the tablet computer in the second coordinate system, which makes the content in front of the AR glasses different from the scene content of the preview window of the tablet computer. Those skilled in the art can understand that the position and attitude of the virtual camera of the tablet computer are controllable and can be changed by the user operating on the tablet computer. In order to eliminate this inconsistency between the orientation angle of the virtual camera of the AR glasses and the orientation angle of the virtual camera of the tablet computer, an orientation angle correction is introduced to correct the orientation angle of the virtual camera of the AR glasses, that is, the orientation angle of the virtual camera of the AR glasses = the orientation angle of the AR glasses + the orientation angle correction. In addition, the orientation angle correction = the orientation angle of the virtual camera of the tablet computer - the orientation angle of the AR glasses.

[0075] In one embodiment, the tablet computer provides a function of resetting the viewing angle, and the specific method is as follows (see Figure 3 Flowchart of the method for resetting the viewing angle shown):

[0076] Step 301: the tablet computer transmits a "reset viewing angle" command to the AR glasses, and transmits the current direction angle data of the virtual camera of the tablet computer to the AR glasses;

[0077] In one embodiment, a "reset viewing angle" button is provided in the operation window of the tablet computer, and the "reset viewing angle" command is transmitted to the AR glasses by clicking the button. In another embodiment, the "reset viewing angle" command is transmitted to the AR glasses by voice.

[0078] Step 302: the AR glasses adjust their own direction angle based on the received direction angle data, and adjust the direction angle of the AR glasses to be consistent with the direction angle of the virtual camera of the tablet computer;

[0079] Step 303: The AR glasses obtain their pitch angle and roll angle through the gravity sensor;

[0080] In this way, the initial posture angle of the AR glasses is determined, and the direction angle in the posture angle is consistent with the direction angle of the virtual camera of the tablet computer. Therefore, the initial direction angle of the AR glasses is consistent with the direction angle of the virtual camera of the tablet computer.

[0081] Based on the initial state, the angular velocity obtained by the gyroscope of the AR glasses is integrated to obtain the change in the attitude angle of the AR glasses, thereby updating the attitude angle of the AR glasses in real time.

[0082] In one embodiment, during the use of the AR glasses, the tablet computer is operated as needed to reset the viewing angle to correct the orientation angle of the AR glasses.

[0083] In one embodiment, different windows of the tablet computer have different viewing angles. Figure 1 Taking the tablet computer 103 shown as an example, its operation window W1 has a wide angle to expand the field of view and facilitate operation, while the preview window W2 has the same viewing angle as the AR glasses 104 to help the tablet computer user 101 refer to the virtual content visible in the AR glasses 104 .

[0084] In an embodiment of the present invention, intuitive interaction between different users is allowed, so that different users can focus on the same position in the virtual space and realize the "pointing" function. In one embodiment, when the user of the AR glasses communicates with the user of the tablet computer (especially remote communication), a mark is made in the virtual space to be used as an indicator. Schematically but not restrictively, the user of the tablet computer clicks a certain position (point or area) in the virtual space on the touch screen; the tablet computer extracts the two-dimensional plane position data (such as x-axis and y-axis coordinates) of the position in the second coordinate system and transmits it to the AR glasses; after receiving the two-dimensional plane position data, the AR glasses display the point at the corresponding position of the coordinate system of the dominant eye. In fact, since the AR glasses render a stereoscopic image, a point on the two-dimensional plane will correspond to a line in the depth direction (such as the z direction). Therefore, when the AR glasses receive the two-dimensional plane position data provided by the tablet computer, a line will be rendered, which cannot accurately lock the specific position clicked by the tablet computer user. In order to solve this problem, in an embodiment of the present invention, when performing a pointing operation, the AR glasses are controlled to perform monocular rendering, that is, only the dominant eye of the AR glasses performs image rendering and its auxiliary eye does not perform image rendering. In this way, when the user of the tablet computer clicks on a certain position on the screen, the x-axis and y-axis coordinates of the position are transmitted to the AR glasses, and the dominant eye of the AR glasses is rendered at the same x-axis and y-axis coordinates in the plane, for example, the position is displayed with a cross, a circle, a black dot, or other mark. Since the first coordinate system with the dominant eye as the origin is in the same direction as the second coordinate system with the tablet computer as the origin, the indicated point seen by the dominant eye is exactly the same as the corresponding point displayed by the tablet computer. In addition, the posture data of the AR glasses is also fed back to the tablet computer in real time, and the tablet computer can mark the direction in which the AR glasses are looking in real time. This embodiment can realize communication between different users without calculating the intersection of the direction and the model. The positions seen by different devices are exactly the same, and the "pointing" function is simply and conveniently realized.

[0085] In one embodiment, the tablet computer provides a calibration function to eliminate the deviation between the visual center of the human eye and the center of the AR glasses screen, thereby eliminating individual differences. Individual differences generally include the pitch of the glasses, the pupil distance of the human eye, and the convergence angle of the human eye. The convergence angle refers to the angle between the visual axes of the two eyes. When the object is observed closer, the convergence angle is larger; when the object is observed farther, the convergence angle becomes smaller.

[0086] Specifically, see Figure 4 A flow chart of the visual calibration method is shown:

[0087] Step 401: Select a reference object on the screen of the tablet computer and let the wearer of the AR glasses look at the reference object;

[0088] In one embodiment, the aforementioned “pointing” function is used to select a reference object on the screen of the tablet computer and lock the reference object on the AR glasses;

[0089] In one embodiment, the reference object is clicked in the preview window on the screen of the tablet computer.

[0090] Step 402: The user of the tablet computer triggers a calibration operation and transmits a calibration command to the AR glasses;

[0091] In one embodiment, the calibration command is transmitted to the AR glasses through the network by clicking a calibration button in the operation window of the tablet computer or triggering the calibration operation by voice.

[0092] Step 403: The AR glasses obtain their own posture data information;

[0093] In one embodiment, the attitude data information is attitude angle (three Euler angles, including pitch angle, roll angle and azimuth angle), based on Figure 3 The method shown obtains the initial attitude angle of the AR glasses and the attitude angle of the AR glasses by obtaining the transformation amount of the attitude angle of the AR glasses obtained by the gyroscope.

[0094] Step 404: the AR glasses calculate the display position of the reference object in the AR glasses according to its posture data information;

[0095] Step 405: the AR glasses calculate the position deviation between the reference object and the physical center of the AR glasses (screen center) and transmit it to the tablet computer;

[0096] Step 406: The tablet computer corrects the gaze direction of the AR glasses according to the position deviation to obtain the user's actual gaze direction.

[0097] When observing an object (including virtual content), the human eye can perceive aspects such as scale, volume, hardness, color, and brightness. In an embodiment of the present invention, the user can adjust the parameters of the AR glasses through a tablet computer to enhance the visual effects of one or more aspects according to the needs. In one embodiment, the parameters of the AR glasses can be adjusted through a tablet computer to enhance the visual effects of one or more aspects while ensuring the sense of reality in other aspects. For example, by adjusting the parameters of the AR glasses through a tablet computer, the stereoscopic effect can be further enhanced in addition to ensuring the sense of reality of the scale (scale 1:1 restoration), and the scale visual effect can also be enhanced under the premise of ensuring the real stereoscopic effect (the virtual content is larger than the actual object). For another example, the pupil distance of a normal person is about 65mm, and the tablet computer performs image processing according to the pupil distance of 200mm, which will enhance the stereoscopic sense of the person, which enhances the stereoscopic sense on the basis of sacrificing the sense of reality of the pupil distance. An important application scenario of AR glasses is to allow multiple people to wear AR glasses, and these people can see the same AR content and operate the AR content. At present, AR glasses support multi-person interaction by combining point cloud positioning, 6DOF tracking technology (VSLAM), and gesture recognition. Point cloud positioning, the camera on the AR glasses takes a picture of the scene, compares it with the visual features of the scene previously collected, and calculates the position of the AR glasses in space. Because the calculation process of point cloud technology positioning is slow, AR glasses cannot be positioned in real time when they are moving, so it is necessary to add an IMU sensor to the AR glasses. The IMU can provide the instantaneous speed, acceleration, and angular velocity of the glasses. Combined with the previous point cloud positioning data, the current position of the glasses can be quickly calculated. If the user operates the content presented in the glasses, he can make corresponding gestures in the air with his hands. The smart glasses capture the hand movements through the camera and obtain the user's instructions. However, this requires installing multiple cameras on the AR glasses to capture the scene and process multiple video data at the same time, which is costly and has a short battery life.

[0098] Another embodiment of the present invention provides a multi-person interaction method based on a tablet computer and AR glasses, wherein at least two people wear AR glasses and one person holds a tablet computer, and all the people can see the same AR content and operate the AR content. Figure 5 The interactive method of the embodiment shown includes the following steps:

[0099] Step 501: Establish a first coordinate system for the first AR glasses, establish a second coordinate system for the second AR glasses, and establish a third coordinate system for the tablet computer;

[0100] Step 502: presenting the same virtual content on the first and second AR glasses and the tablet computer;

[0101] Step 503: operating a posture change of the virtual content through the first AR glasses or the second AR glasses and / or operating a position change of the virtual content through the tablet computer; and

[0102] Step 504: Synchronize data between the AR glasses and the tablet computer.

[0103] Based on this embodiment, the first AR glasses or the second AR glasses operate the posture change of the virtual content, and the tablet computer operates the position change of the virtual content. Taking the posture change of the first AR glasses operating the virtual content as an example, the first AR glasses, the second AR glasses and the tablet computer display the same virtual content. When the posture of the virtual content needs to be changed, the first AR glasses realize the posture change of the virtual content through, for example, head tracking technology or eye tracking technology, such as changing the three Euler angles of the virtual content. The first AR glasses obtain the three Euler angle data in real time and transmit it to the tablet computer. The tablet computer changes the Euler angle of the virtual content displayed on its display screen based on the real-time Euler angle data. At the same time, the tablet computer also transmits the real-time Euler angle data to the second AR glasses, and the second AR glasses also change the Euler angle of the virtual content displayed by itself based on the real-time Euler angle data; when the position of the virtual content needs to be changed, for example, the coordinates of the virtual content along the x-axis, y-axis and z-axis are changed, such as moving the virtual content through touch screen operation, gesture operation or voice operation, and at the same time, the real-time position data of the virtual content is also transmitted to the first AR glasses and the second AR glasses, and the first AR glasses and the second AR glasses change the position of the virtual content displayed by themselves based on the real-time position data. In this multi-person interactive embodiment, the perspective resetting, pointing and calibration functions in the aforementioned embodiments are also applicable and will not be described in detail here.

[0104] Those skilled in the art will appreciate that multi-person interaction is not limited to two people wearing AR glasses, and more than two people wearing AR glasses can also achieve a similar interaction process.

[0105] The multi-person interaction method based on a tablet computer and AR glasses of the present invention has special advantages in remote teaching, telemedicine and the like. Taking dentistry as an example, as people pay more attention to oral health and aesthetics, the orthodontic market continues to expand, especially in China, where the orthodontic industry shows a trend of rapid growth. AR interactive technology in the orthodontic industry is an emerging application that enhances the perception of the real world by superimposing virtual information in the patient's or doctor's field of vision. It can be used to educate patients, and to display the process and expected effects of tooth correction in a visual way to help patients better understand the treatment plan. In an embodiment of the present invention, the doctor controls the tablet computer, and the patient and the patient's parents or friends wear AR glasses. The doctor operates the tablet computer to display the AR image of the current state of the teeth and the expected AR image, and can also display the changes of the teeth over time. The corresponding virtual content can be presented in the AR glasses. When it is necessary to rotate the tooth AR image to view its various sides, the wearer of the AR glasses can realize the posture change of the AR image through his head movement, and the real-time posture data of the AR image is transmitted to the tablet computer. The tablet computer changes the posture of its AR image based on the real-time posture data, so that the doctor can also know the posture of the AR image currently seen by the patient and make corresponding explanations. In addition, when the position of the AR image of the teeth needs to be changed, the doctor performs corresponding operations on the tablet computer to shift the AR image up and down, left and right, or front and back, and transmits the real-time position data to the AR glasses, and the AR glasses change the position of their AR images according to the real-time position data. During the doctor-patient communication process, when the AR images seen by the two are inconsistent, the doctor can operate the tablet computer to reset the perspective. In addition, when the doctor emphasizes a special position, he will click on the tablet computer, and the tablet computer will transmit the position data of the position to the AR glasses. The AR glasses will be able to display the corresponding position, such as displaying a cross or other symbols at the position with the corresponding position data, and the patient will be able to look at the special position. In addition, the doctor can also obtain the patient's true gaze direction based on the calibration operation mentioned in the aforementioned embodiment to achieve accurate interaction.

[0106] The embodiment of the present invention realizes 6DOF AR through the interaction between a tablet computer and AR glasses, without the need for high-performance hardware devices, and provides a richer interactive experience and a stronger sense of immersion at a lower cost. In one embodiment of the present invention, the size, posture and position of virtual content are changed by operating the tablet computer.

[0107] According to the embodiments of the present invention, during the orthodontic treatment planning stage, AR technology can help doctors simulate tooth movement and predict changes during the treatment process. The tablet computer can also collect, record and reproduce relevant data. After the orthodontic treatment is completed, relevant data of the orthodontic process can be extracted at any time when needed, which plays an important role in treatment effect evaluation, quality control improvement, research and development, and long-term health management of patients.

[0108] A tablet computer is used as an example in the above description, but it can be understood that the present application is not limited to a tablet computer, but can be applied to other handheld smart devices, such as mobile phones, smart handles, controllers, etc.

[0109] The head mounted display device in the present invention may be AR glasses, smart glasses, AR helmets, or any other glasses that can be used to present information to users. The head mounted display device in the present application also includes glasses formed by adding components or plug-ins to ordinary optical glasses, for example, glasses formed by adding a display device to ordinary optical glasses.

[0110] According to other embodiments of the present invention, image rendering of AR glasses is completed through a tablet computer or the cloud, which reduces the hardware requirements for AR glasses, reduces the cost of AR glasses, and extends the standby time of AR glasses.

[0111] In one embodiment of the present invention, the present invention may be implemented in the form of a computer program. The computer program may be stored in various storage media (e.g., a hard disk, an optical disk, a flash memory, etc.), and when the computer program is executed by a processor, it can be used to implement the method of the present invention.

[0112] In another embodiment of the present invention, the present invention may be implemented in the form of an electronic device. The electronic device includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, the method of the present invention can be implemented.

[0113] References herein to "various embodiments", "some embodiments", "one embodiment", or "an embodiment" refer to specific features, structures, or properties described in conjunction with the embodiments being included in at least one embodiment. Therefore, the phrases "in various embodiments", "in some embodiments", "in one embodiment", or "in an embodiment" appearing throughout the present invention do not necessarily refer to the same embodiment. In addition, specific features, structures, or properties may be combined in any suitable manner in one or more embodiments. Therefore, specific features, structures, or properties shown or described in conjunction with one embodiment may be combined in whole or in part with features, structures, or properties of one or more other embodiments without restriction, as long as the combination is not illogical or inoperable. Expressions similar to "according to A", "based on A", "through A", or "using A" appearing herein are intended to be non-exclusive, that is, "according to A" may include "according to A only" or "according to A and B", unless it is specifically stated that its meaning is "according to A only". In this application, for the sake of clarity, some exemplary operating steps are described in a certain order, but those skilled in the art will understand that each of these operating steps is not essential, and some of the steps may be omitted or replaced by other steps. These operation steps do not have to be executed sequentially in the manner shown; on the contrary, some of these operation steps can be executed in different orders or in parallel according to actual needs, as long as the new execution manner is not illogical or inoperable.

[0114] Although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described herein but includes various changes and modifications that may be made without departing from the scope of the present invention.

Claims

1. A method for interaction between a handheld smart device and a head mounted display device, the method comprising: Displaying virtual content corresponding to the handheld smart device on a display screen of the head mounted display device; Operating the posture change of the virtual content through the head mounted display device and / or operating the position change of the virtual content through the handheld smart device; as well as Data is exchanged between the head mounted display device and the handheld smart device so that a posture change of the virtual content is transmitted to the handheld smart device and / or a position change of the virtual content is transmitted to the head mounted display device.

2. The method according to claim 1, further comprising: A first coordinate system is established for the head mounted display device, and a second coordinate system is established for the handheld smart device. The posture change of the virtual content in the first coordinate system is operated by the head mounted display device and / or the position change of the virtual content in the second coordinate system is operated by the handheld smart device.

3. The method according to claim 2, wherein: A first coordinate system is established with the head mounted display device as the origin, and a second coordinate system is established with the handheld smart device as the origin.

4. The method according to claim 2, wherein: Data is exchanged between the head-mounted display device and the handheld smart device so that posture data of the virtual content in the first coordinate system is transmitted to the handheld smart device and / or position data of the virtual content in the second coordinate system is transmitted to the head-mounted display device.

5. The method according to any one of claims 1 to 4, further comprising: The head mounted display device changes the position of its own virtual content based on the position change of the virtual content from the handheld smart device; and / or The handheld smart device changes the posture of its own virtual content based on the posture change of the virtual content from the head mounted display device.

6. The method according to claim 5, wherein: The handheld smart device is operated to reset the viewing angle so that the current direction angle of the head mounted display device is consistent with the direction angle of the handheld smart device.

7. The method according to any one of claims 1 to 4, further comprising: The user of the handheld smart device clicks a location in the virtual space on the screen of the handheld smart device; The handheld smart device extracts the two-dimensional position data of the position and transmits it to the head mounted display device; as well as The head mounted display device performs monocular rendering and displays virtual information at a corresponding two-dimensional position on its display screen based on the two-dimensional position data.

8. The method according to any one of claims 1 to 4, further comprising: Selecting a reference object on the screen of the handheld smart device; The user of the head mounted display device looks at the reference object; The handheld smart device transmits a calibration command to the head mounted display device; The head mounted display device acquires its own posture data information and calculates the display position of the reference object in the head mounted display device based on the posture data information; The head mounted display device calculates the position deviation between the display position and the physical center of the head mounted display device and transmits it to the handheld smart device; as well as The handheld smart device corrects the actual gaze direction of the user of the head mounted display device based on the position deviation.

9. The method according to any one of claims 1-4 further comprises adjusting parameters of the head mounted display device through the handheld smart device to enhance the visual effects of the virtual content of the head mounted display device in one or more aspects.

10. The method according to any one of claims 1 to 4, wherein: The handheld smart device is operated to record the change process of the virtual content.

11. A storage medium storing a computer program, wherein when the computer program is executed by a processor, the computer program can be used to implement the method according to any one of claims 1 to 10.

12. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it can be used to implement the method according to any one of claims 1 to 10.

13. An interactive system, comprising a handheld smart device and a head-mounted display device, wherein the handheld smart device and the head-mounted display device can be used to implement the method described in any one of claims 1-10.

Images