AR head-mounted device, and AR head-mounted device and terminal device combination system

Abstract

The embodiment of the application provides an AR head-mounted device and an AR head-mounted device and terminal device combined system, external data is collected through various modules, and distributed processing is performed on the data through a processor and multiple coprocessors, and a processing result is output. The combination of the AR head-mounted device and the terminal device improves the augmented reality effect, is higher in interactivity, and is stronger in user experience.

Description

[0001] This is a divisional application. The original application number is 201810859793.0, and the original application date is August 1, 2018. Technical Field

[0002] This invention relates to the field of augmented reality technology, and more specifically, to an AR headset and a combined system of the AR headset and terminal devices. Background Technology

[0003] AR (Augmented Reality), also known as mixed reality, works by using computer technology to apply virtual information to the real world, where the real environment and virtual objects are superimposed on the same scene or space in real time.

[0004] In existing technologies, AR headsets can only collect data through relatively simple sensors, and control the device based on the processing results after processing the collected data. This design lacks interactivity and results in a poor user experience. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides an AR head-mounted display device and terminal device combination system.

[0006] In a first aspect, embodiments of the present invention provide an AR head-mounted display device and terminal device combination system, wherein the AR head-mounted display device includes an AR power system, an AR communication module, and an AR processing module;

[0007] The terminal device includes a terminal power system, a terminal communication module, and a terminal processing module;

[0008] The AR headset and the terminal device establish a connection through the AR communication module and the terminal communication module.

[0009] Furthermore, the AR communication module is a wired communication module or a wireless communication module;

[0010] When the AR communication module is a wireless communication module, the AR power system includes a charging module and a power supply module;

[0011] When the AR communication module is a wired communication module, the AR power system includes at least a power supply module.

[0012] Furthermore, the AR headset also includes an AR coprocessing module, and the terminal device also includes a terminal coprocessing module;

[0013] The AR coprocessing module is used to perform collaborative processing with the AR processing module and / or the terminal processing module and / or the terminal coprocessing module;

[0014] The terminal coprocessing module is used to perform collaborative processing with the terminal processing module and / or the AR processing module or the AR coprocessing module.

[0015] Furthermore, the AR coprocessing module is an AR display processing module, the terminal coprocessing module is a terminal display processing module, the AR head-mounted display device also includes an AR display module, and the terminal device also includes a terminal display module;

[0016] At least one of the AR display processing module, the AR processing module, the terminal display processing module, and the terminal processing module processes the data of the AR display module and / or the terminal display module.

[0017] Furthermore, the AR coprocessing module is an AR spatial positioning coprocessing module, the terminal coprocessing module is a terminal spatial positioning coprocessing module, and the AR head-mounted display device also includes an AR spatial positioning data acquisition module;

[0018] At least one of the AR processing module, the AR spatial positioning coprocessing module, the terminal processing module, and the terminal spatial positioning coprocessing module processes the data collected by the AR spatial positioning data acquisition module and transmits it to at least one of the AR processing module, the AR spatial positioning coprocessing module, the terminal processing module, and the terminal spatial positioning coprocessing module.

[0019] Furthermore, the AR coprocessing module is an AR mixed reality coprocessing module, the terminal coprocessing module is a terminal mixed reality coprocessing module, and the AR head-mounted display device also includes an AR spatial positioning data acquisition module, an AR image acquisition module, an AR depth detection module, and an AR gaze tracking module;

[0020] The AR mixed reality coprocessing module and / or the AR processing module are used to receive data collected by the AR spatial positioning data acquisition module, AR image acquisition module, AR depth detection module, and AR gaze tracking module, and at least one of the AR mixed reality coprocessing module, the AR processing module, the terminal mixed reality coprocessing module, and the terminal processing module processes the data collected by the AR spatial positioning data acquisition module, AR image acquisition module, AR depth detection module, and AR gaze tracking module, and transmits it to at least one of the AR processing module, the AR mixed reality coprocessing module, the terminal processing module, and the terminal spatial positioning coprocessing module.

[0021] Furthermore, the AR coprocessing module is an AR collaborative mixed reality coprocessing module, the terminal coprocessing module is a terminal collaborative mixed reality coprocessing module, and the AR head-mounted display device also includes an AR spatial positioning data acquisition module, an AR image acquisition module, an AR depth detection module, and an AR gaze tracking module; the terminal device also includes a terminal spatial positioning data acquisition module, a terminal image acquisition module, and a terminal depth detection module.

[0022] At least one of the AR processing module, the AR collaborative mixed reality coprocessing module, the terminal processing module, and the terminal collaborative mixed reality coprocessing module processes the data collected by the AR spatial positioning data acquisition module, the AR image acquisition module, the AR depth detection module, and the AR gaze tracking module, as well as the data collected by the terminal spatial positioning data acquisition module, the terminal image acquisition module, and the terminal depth detection module, and transmits the data to at least one of the AR processing module, the AR collaborative mixed reality coprocessing module, the terminal processing module, and the terminal collaborative mixed reality coprocessing module.

[0023] Furthermore, the AR coprocessing module is an AR motion behavior perception coprocessing module, the terminal coprocessing module is a terminal motion behavior perception coprocessing module, and the AR head-mounted display device also includes an AR spatial positioning data acquisition module, an AR motion perception module, an AR depth detection module, an AR gaze tracking module, and an AR touch control module; the terminal device also includes a terminal spatial positioning data acquisition module, a terminal motion perception module, a terminal depth detection module, and a terminal touch control module.

[0024] At least one of the AR processing module, the AR motion behavior perception coprocessing module, the terminal processing module, and the terminal motion behavior perception coprocessing module processes the data collected by the AR spatial positioning data acquisition module, the AR motion perception module, the AR depth detection module, the AR gaze tracking module, and the AR touch control module, as well as the data collected by the terminal spatial positioning data acquisition module, the terminal motion perception module, the terminal depth detection module, and the terminal touch control module, and transmits the data to at least one of the AR processing module, the AR motion behavior perception coprocessing module, the terminal motion behavior perception coprocessing module, and the processing module.

[0025] Furthermore, the AR coprocessing module is an AR audio processing module, the terminal coprocessing module is a terminal audio processing module, the AR head-mounted display device also includes an AR audio module, and the terminal device also includes a terminal audio module;

[0026] At least one of the AR processing module, the AR audio processing module, the terminal processing module, and the terminal audio processing module processes the data collected by the AR audio module and the terminal audio module, and transmits the data to at least one of the AR processing module, the AR audio processing module, the terminal audio processing module, and the terminal processing module for further processing, and then outputs the data through the AR audio module and / or the terminal audio module.

[0027] Furthermore, at least two of the AR coprocessing module, the AR processing module, the terminal coprocessing module, and the terminal processing module work together to enable the AR head-mounted display device and the terminal device to work together.

[0028] The embodiments of the present invention bring the following beneficial effects:

[0029] This invention provides an AR headset and terminal device combination system. It collects external data through multiple modules and performs distributed data processing using a processor and multiple coprocessors, outputting the processing results. The combination of the AR headset and terminal device improves the augmented reality effect, enhances interactivity, and provides a stronger user experience.

[0030] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objectives and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description, claims, and drawings.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0032] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 This is a structural diagram of an AR head-mounted display device and terminal device combination system provided in an embodiment of the present invention;

[0034] Figure 2 This is a structural diagram of an AR head-mounted display device and a computer system provided in an embodiment of the present invention;

[0035] Figure 3This is a structural diagram of an AR head-mounted display device and a mobile phone system provided in an embodiment of the present invention;

[0036] Figure 4 This is a structural diagram of an AR head-mounted display device and a mobile phone system with synchronous display function provided in an embodiment of the present invention;

[0037] Figure 5 This is a structural diagram of an AR head-mounted display device and a mobile phone system with spatial positioning function provided in an embodiment of the present invention;

[0038] Figure 6 This is a structural diagram of an AR head-mounted display device and mobile phone system with mixed reality functionality provided in an embodiment of the present invention;

[0039] Figure 7 This is a structural diagram of an AR head-mounted display device and mobile phone system with collaborative mixed reality functionality provided in an embodiment of the present invention;

[0040] Figure 8 This is a structural diagram of an AR head-mounted display device and mobile phone system with comprehensive motion behavior perception and calculation function provided in an embodiment of the present invention;

[0041] Figure 9 This is a structural diagram of an AR head-mounted display device and mobile phone system with sound field function provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0043] Example 1

[0044] See Figure 1 The diagram shows a structural system combining an AR head-mounted display device and a terminal device. The system specifically includes an AR head-mounted display device and a terminal device, wherein the terminal device can be a computer or other mobile terminal.

[0045] The AR head-mounted display device includes an AR power system, an AR communication module, an AR processing module, and an AR co-processing module;

[0046] The terminal device includes a terminal power system, a terminal communication module, a terminal processing module, and a terminal coprocessing module;

[0047] The AR head-mounted display device and the terminal device establish a connection through the AR communication module and the terminal communication module.

[0048] In this embodiment of the invention, the AR communication module can be a wired communication module or a wireless communication module; when the AR communication module is a wireless communication module, the AR power system includes a charging module and a power supply module, but when the AR communication module is a wired communication module, the AR power system must at least include a power supply module.

[0049] AR headsets also include audio modules, image acquisition modules, spatial positioning data acquisition modules, motion sensing modules, depth detection modules, eye tracking modules, or touch control modules.

[0050] like Figure 2 As shown, the terminal device can be a computer, which can be configured with other modules such as a communication module, a display module, an image acquisition module, an audio module, a depth detection module, and a touch module.

[0051] like Figure 3 As shown, the terminal device can be a mobile terminal device such as a mobile phone. The mobile phone can be configured with other modules such as a communication module, a display module, an image acquisition module, an audio module, a spatial positioning data acquisition module, a motion sensing module, a depth detection module, and a touch module.

[0052] In another embodiment of the present invention, the terminal device may not have a display module and / or an audio module and / or a spatial positioning data acquisition module and / or a motion sensing module and / or a depth detection module and / or a touch module.

[0053] Furthermore, the AR processing module, the AR co-processing module, and at least one of the terminal processing module and the terminal co-processing module perform collaborative processing.

[0054] Specifically, other modules collect data and then process the collected data in the following ways, depending on requirements such as computational complexity: 1. Perform separate computation and processing only in the terminal processing module, terminal co-processing module, AR processing module, or AR co-processing module; 2. Perform distributed computation simultaneously on the AR headset and terminal device, including 9 methods: ① Simultaneous distributed processing by the AR processing module and terminal processing module; ② Simultaneous distributed processing by the AR co-processing module and terminal co-processing module; ③ Simultaneous distributed processing by the AR processing module and terminal co-processing module; ④ Simultaneous distributed processing by the AR co-processing module and terminal processing module; ⑤ Collaborative processing by the AR processing module, AR co-processing module, and terminal processing module; ⑥ Collaborative processing by the AR processing module, AR co-processing module, and terminal co-processing module; ⑦ Collaborative processing by the AR processing module, terminal processing module, and terminal co-processing module; ⑧ Collaborative processing by the AR co-processing module, terminal processing module, and terminal co-processing module; ⑨ Collaborative processing by the AR processing module, AR co-processing module, terminal processing module, and terminal co-processing module. There are a total of 10 data processing methods.

[0055] The following example illustrates how distributed computing can be performed simultaneously on both an AR headset and a terminal device:

[0056] Method 1: Each module of the AR headset transmits its data to the AR processing module, which then processes a portion of the data. Simultaneously, the AR processing module transmits the remaining data to the terminal processing module for further processing. This achieves simultaneous data processing distributed across both the AR and terminal processing modules. Because the two processing modules use processors common to both the AR glasses and the terminal (phone or PC), their data processing capabilities and speeds are relatively limited. However, using a collaborative processing approach between the AR and terminal processing modules results in faster processing times and higher efficiency (approximately 10%-30%) compared to using a single processing module on one end. Therefore, this co-processing method is suitable for applications with large computational demands and tight processing times. Furthermore, since the AR glasses incorporate an AR co-processing module, the performance requirements for the AR glasses' processor can be relaxed, facilitating miniaturization and portability of the glasses. Because the phone or PC's processor also shares the co-processing tasks, the power consumption and heat generation of the AR glasses are reduced, improving the wearing experience.

[0057] Method 2: After each module of the AR headset transmits its data to the AR processing module, the AR processing module sends a portion of the raw data to the terminal processing module, which then transmits it to the terminal co-processing module for processing via the terminal device bus. Simultaneously, the AR processing module transmits another portion of the data to the AR co-processing module for processing via the bus. This achieves simultaneous distributed data processing on both the AR co-processing module and the terminal co-processing module. Since AR glasses and mobile phones or PCs both use co-processing modules, this method is the fastest collaborative processing approach, suitable for computationally intensive, high-real-time processing of specific algorithms. The AR co-processing module of AR glasses can be a dedicated hardware IC outside the main processor chip, a dedicated DSP within the processing module, or implemented in a general-purpose processing module using dedicated software algorithms. Using a dedicated hardware IC increases the hardware area; software implementation in a general-purpose processing module may increase power consumption and processing time—both methods have their disadvantages. In this collaborative processing approach, the optimal implementation of the AR co-processing module is to have a dedicated DSP in the general processing module (such as the CPU) to co-process specific algorithms. This architecture has significant advantages in terms of computing power and speed, as well as in the miniaturization and power consumption of AR glasses.

[0058] Method 3 involves each module of the AR headset transmitting its data to the AR processing module of the AR glasses. The AR processing module then directly processes a portion of this data. Simultaneously, it transmits the remaining data to the terminal processing module, which in turn transmits it to the terminal co-processing module via the on-chip bus on the mobile phone (PC). This allows for simultaneous data processing distributed across the AR processing module and the terminal co-processing module. Because the AR glasses use a general-purpose processing module, and the mobile phone uses the terminal co-processing module for collaborative processing, the overall processing time is shorter compared to a single processing method. This approach also benefits the miniaturization and optimized heat dissipation of the AR glasses.

[0059] Method 4: Each module of the AR headset transmits its data to the AR processing module, which then sends a portion of the data to the terminal processing module for processing. Simultaneously, the AR processing module transmits another portion of the data via a bus to the AR co-processing module for processing. This allows for simultaneous data processing distributed across the AR co-processing module and the terminal processing module.

[0060] Example 2

[0061] like Figure 4As shown, synchronized display is one of the essential functional requirements of the AR headset and terminal device as a whole system. When the AR co-processing module is an AR display processing module, the terminal co-processing module is a terminal display processing module, the AR headset also includes an AR display module, and the terminal device also includes a terminal display module, there are two ways to process the data from the display module:

[0062] Method 1: The display content of terminal devices such as mobile phones, PCs, tablets, or laptops is transmitted to the terminal display processing module (terminal coprocessing module) through the display data interface for processing, and then transmitted to the AR glasses via wired or wireless means. After the AR glasses obtain the display content, the AR display processing module (AR coprocessing module) adapts and processes it according to its own display requirements and parameters, and then outputs it through the terminal display module of the terminal device such as mobile phone, PC, tablet, or laptop.

[0063] Method 2: To achieve higher resolution, higher bandwidth, and lower latency synchronous display, we can distribute the processing of display data simultaneously in the terminal display processing module and the AR display processing module. The process is as follows: The terminal device transmits a portion of the original display data to the AR device according to a certain allocation scheme, while the terminal device processes another portion of the original display data; After receiving a portion of the original display data, the AR device processes this portion of data using its own AR display processing module; The terminal display processing module processes the other portion of the original display data and transmits the processed portion of display data to the AR device; The AR device merges, adapts, and outputs its own processed portion of display data with the received processed portion of display data from the terminal device.

[0064] Example 3

[0065] like Figure 5 As shown, when displaying an image within a specific field of view of AR glasses, to ensure the image is displayed at a specific location, the spatial positioning data acquisition module first transmits the collected raw spatial positioning data to the AR processing module via a relevant data communication interface. Based on computational complexity and other requirements, the collected data is segmented according to a predetermined ratio. Except for the portion retained by the AR processing module, the other segmented data is transmitted to the terminal processing module and terminal co-processing module via the AR processing module or AR co-processing module. The AR processing module, AR co-processing module, terminal processing module, and terminal co-processing module collaboratively process the data collected by the spatial positioning data acquisition module. Terminal devices such as mobile phones, PCs, tablets, or laptops send video information to the AR glasses. The AR processing module or AR co-processing module performs mixed processing with the video information and spatial positioning data to obtain the final image, which is then sent to the display module of the AR glasses for image display. For example, a cartoon character can be placed on a table in a real environment; or video information or a virtual image can be displayed in the center of the field of view.

[0066] For example, to place a virtual portal model in the user's space: (1) At the initial moment, the AR processing module 2 and terminal processing module 5 of the device and glasses present the portal model image in front of the user through the display unit on the glasses. At the same time, the spatial positioning data acquisition module 1 of the glasses collects the feature point data of the user's space and the readings of sensors such as the gyroscope on the device in real time; (2) When the user turns his head or walks, the data collected by the spatial positioning data acquisition module 1 is processed according to one of the 10 data processing methods provided above to calculate the user's position movement and posture change in the real space; (3) The above position and posture change information is transmitted to the AR processing module 2 and terminal processing module 5. These two processing modules calculate the imaging state of the portal model in the user's field of vision at this time according to the changes in the user's position and posture, and present the new image in the user's field of vision through the display unit, thereby achieving the effect of fixing the virtual portal model in a certain position in space.

[0067] Example 4

[0068] Mixed reality is also a common functional requirement for augmented reality systems, such as... Figure 6 As shown. Using this architecture and module, we can add virtual objects to the real world seen by the user's eyes at the glasses. The virtual objects can change position in the field of view as the user's eyes move. The spatial positioning camera captures real images, and the algorithm analyzes the feature point information in the real images to calculate the user's position and posture changes in real space. This position and posture change information is then transmitted to the application layer so that the virtual object images seen by the user are also correctly imaged according to the above position and posture change information.

[0069] For example, to achieve a shopping experience in mixed reality: (1) At the initial moment, the spatial positioning data acquisition module of the glasses collects feature point data of the user's space in real time and sensor readings such as gyroscopes on the device; the depth detection module of the glasses detects and collects the structure of the user's environment, and completes point cloud stitching through algorithms to construct a three-dimensional map of the user's scene; the processing modules of the glasses and the device present virtual product images in front of the user through the display unit; (2) When the user's position and posture change, the device updates the virtual product image information according to the method described in Example 4; (3) The user constructs a three-dimensional model of the user's hand through the depth detection module or image acquisition module of the glasses, and assists the corresponding machine. (3) The eye-tracking module at the glasses monitors the position changes of the user's gaze points in real time; (4) When the user looks at a virtual product and intends to buy it, the gaze point information provided by the eye-tracking module is combined with the virtual product through one of the above 10 information processing methods to realize the selection. The user starts the next process operation through the specific gesture captured by the depth detection module; (5) If the user wants to search for a real object (such as a pair of shoes worn by a friend), when the user looks at the product and performs the operation described in (5), the image acquisition module on the glasses will collect the image information of the product. The image information is uploaded to the cloud server through two processing units to realize the product retrieval.

[0070] Example 5

[0071] Based on Example 4, by utilizing the image acquisition module on the mobile device, collaborative mixed reality functionality can be extended, such as... Figure 7 As shown. For example, a user can take a picture of a virtual object they see in their eyes using their mobile phone. This allows for the combination of spatial positioning data from the glasses, real-world image information from the glasses, and real-world image information captured by the mobile phone to construct holographic information of the virtual object, which can then be displayed to the user as needed. The processing methods for external world image data, spatial positioning data, and image data captured by the mobile phone can refer to the 10 data processing methods mentioned above. Because this function requires the mobile phone to be constantly moving, the other end of the system generally does not use a PC device.

[0072] This example can be used for applications such as third-person augmented reality shooting: User A wears glasses and interacts with virtual images in their field of vision to a certain extent (viewing, controller interaction, positional interaction, gesture interaction, voice interaction, etc.); a mobile phone in another perspective obtains the interaction information captured by User A's glasses (including changes in User A's position information, interaction information between User A and the virtual images, etc.) through a cloud server; a real image is captured through a camera connected to the mobile phone; on the mobile phone, the real image captured by the camera is combined with the virtual image to achieve third-person perspective capture of User A's interaction with the virtual image.

[0073] For example, in the shopping scenario of the above embodiment 4, a 6-DOF handle selection function is added, and a 3D model can be built for real objects to facilitate retrieval. Therefore, the following description is added after the above 5 items: (1) The spatial positioning data acquisition module on the mobile phone extracts the feature points of the user's space and the position and posture change information of the mobile phone itself, calculates the position movement of the mobile phone, and thus realizes the 6-DOF handle function; the user can hold the mobile phone and calculate which virtual product the user is currently pointing to through one of the above 10 information processing methods based on the location and direction of the mobile phone, thus realizing the selection; (2) The user's retrieval function for real objects can be realized through the image acquisition module of the glasses or the mobile phone. The two modules respectively collect image information of the object from different angles to improve the matching accuracy of the retrieval; (3) In order to further improve the accuracy of the retrieval, the user can scan around the object through the depth detection module of the mobile phone, and construct the 3D model of the object through a certain data processing method and a splicing algorithm, which is convenient for subsequent uploading to the corresponding server for subsequent product retrieval.

[0074] Example 6

[0075] like Figure 8 As shown, the data collected by various sensors on the glasses and mobile phone can be fused and processed to a large extent to recognize and perceive the user's behavior. Combined with the touch click information obtained by the touch module 7 / 13 and the image recognition gesture information, it covers almost all possibilities of recognizing action behavior.

[0076] The eye-tracking module at the glasses is used to track and recognize the movements of the human eye, such as blinking and eye movement.

[0077] The glasses-mounted spatial positioning data acquisition module and motion sensing module can provide users with spatial positioning information, head movement orientation, twisting speed, and spatial angle information.

[0078] The spatial positioning data acquisition module and motion sensing module on the mobile phone can provide corresponding information on the movement of the mobile phone (hand). The body's movement information can also be obtained by fusion calculation and modeling of IMU data from both the glasses and the mobile phone.

[0079] The spatial positioning data acquisition module and depth detection module at both ends of the glasses and mobile phone are used to sense and judge the distance and relative movement process of the user (head or hand) relative to the surrounding environment, people and objects.

[0080] The individual or fusion calculations involved in the above modules can be performed in one or more of processing modules 1 / 2 / 8 / 9. The above architecture can achieve at least two functional applications: 1. In conjunction with the touch module, it enables a rich, multi-dimensional control experience. For example, users can operate the system via a mobile phone, glasses touchpad, or touchscreen; users can also achieve motion control through their own actions; users can also give commands to the system through actions such as blinking; 2. With the popularization of head-mounted virtual reality products, it can collect large-scale user behavior and posture data, thereby statistically assisting in the analysis of the relationship between human behavior and health, and providing richer data samples for research on human behavior. Because this function requires the mobile device to be constantly mobile, the other end of the system generally will not use a PC device. Data processing methods can refer to the 10 data processing methods mentioned above.

[0081] This example can have many applications, such as:

[0082] 1) Augmented Reality Soccer Game – Users select the approximate area of ​​the virtual pitch by moving their head, and then precisely select the player they want to control by tracking their eye movements. The user then controls the selected player's actions through gestures and touch controls on their phone. This process utilizes at least the following hardware modules: the glasses' spatial positioning data acquisition module 3, motion sensing module 4, and eye-tracking module 6; and the phone's motion sensing module 11 and touch control module 13.

[0083] Many other augmented reality sports and driving games can be applied to this example, such as augmented reality table tennis games, racing games, flight games, etc., which will not be listed here.

[0084] 2) 3D Model Design – Designers or users adjust the viewing angle of the virtual model in front of them via head movements on the glasses, select specific parts of the model that the user wants to change by tracking eye movements, and stretch or shrink the virtual model in 3D space by making gestures in front of the glasses; on the mobile phone, planar details of a certain part of the virtual 3D model are drawn through touch. The above process uses at least the following hardware functional modules: spatial positioning data acquisition module 3, motion perception module 4, depth detection module 5, eye tracking module 6 on the glasses, and touch control module 13 on the mobile phone.

[0085] For example, when a mobile phone is connected to glasses, it can be used to draw in real space: (1) In the initial state, the depth detection module 5 on the glasses scans the user's spatial position and constructs a 3D spatial image; the spatial positioning data acquisition module 3 and the motion sensing module 4 on the glasses cooperate to collect feature points and gyroscope data of the user's space in real time and calculate the user's movement in real space; the above data is processed through one of the above 10 information processing methods; when the user's position and posture change, the virtual image information in the space is updated according to the method described in Example 3; (2) Spatial positioning data acquisition on the mobile phone Module 10, depth detection module 12 and motion sensing module 11 work together to calculate the position and posture information of the mobile phone through the algorithm according to the method described in (1), so that the mobile phone can realize the "6 degrees of freedom" handle function, that is, the user can move the mobile phone freely, and the position and posture information of the mobile phone can be transmitted to the processing module in real time; (3) the user can start drawing by clicking the touch module 7 or 13 on the glasses end or the short touch module of the mobile phone; the drawing trajectory will follow the movement trajectory of the mobile phone (collected by the spatial positioning data acquisition module 10, depth detection module 12 and motion sensing module 11 in the above manner), and cooperate to realize the data generation of tracking module 6.

[0086] Example 7

[0087] like Figure 9 As shown, excluding the display portion of the multimedia system, this system can construct a universally applicable sound environment. The sound input modules for both the glasses and mobile phones are generally microphones, while the sound output module for the mobile phone typically consists of 1-2 speakers plus 1 earpiece. The sound output module for the glasses can have 2 or even more speakers.

[0088] The architecture of this example can be used to implement many application scenarios.

[0089] 1) Sound Field Construction – Audio content can be shared and played between glasses and mobile phones (PCs). By using multiple sound output devices on both the glasses and the mobile phone (PC), the audio content can be played in a differentiated manner, allowing the entire system to achieve an immersive sound output effect for the user. For example, a sound environment resembling a live symphony orchestra can be created, with the sounds of different instruments played through sound output devices in different locations, giving the user the auditory experience of being at a live symphony orchestra.

[0090] 2) Noise Reduction – The audio input module can be used to reduce ambient noise. Noise reduction can be broadly divided into two scenarios: The first is when the augmented reality glasses user is making a voice call with someone else. The noise in the user's environment is eliminated, leaving almost only the valid voice signal of the speaker on the glasses. The principle is as follows: the valid call sound is picked up by the audio input device (microphone) closest to the speaker, while audio input devices at other locations (on the glasses or mobile phone) pick up noise from different parts of the environment. The noise picked up by the microphones at other locations is inverted and superimposed on the call sound before being transmitted to the other user. The second scenario is where the augmented reality glasses user enjoys a high-quality, valid sound environment while the noise in the user's environment is eliminated. The principle is roughly the same as the first scenario.

[0091] 3) Sound Virtual Reality System – Users wear highly noise-canceling earplugs (essentially eliminating any external sounds). The system collects ambient sounds through multiple sound input devices and, after selecting and discarding these sounds, superimposes additional sound sources from the device and plays them to the user. It can also achieve the effect of placing virtual objects at the sound source and presenting them in front of the user.

[0092] 4) Voice recognition – By combining the microphone arrays of the glasses and the mobile phone, multi-angle voice input is achieved, thereby realizing the voice recognition function.

[0093] In summary, the display module of an AR headset is generally a dual OLED display; the image acquisition module is usually a camera module similar to that of a mobile phone camera; the spatial positioning data acquisition module consists of a single or dual spatial positioning camera module and an IMU; the motion sensing module mainly refers to the IMU; the depth detection module can be a distance sensor, a structured light emitting and receiving module, a laser array emitting and receiving module, etc.; the eye tracking module is generally a set of eye tracking camera modules placed inside the eyes and facing the user's eyes; and the touch module is a touchpad or touch screen similar to that of a mobile phone.

[0094] This invention provides an AR headset and terminal device combination system. It collects external data through multiple modules and performs distributed data processing using a processor and multiple coprocessors, outputting the processing results. The combination of the AR headset and terminal device improves the augmented reality effect, enhances interactivity, and provides a stronger user experience.

[0095] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0096] In the description of this invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0097] Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0098] It should be noted that, in the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the destination of the solution in this embodiment according to actual needs.

[0099] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0100] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0102] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An AR head-mounted display device, comprising: An AR communication module, configured to enable the AR head-mounted display device to establish a connection with a terminal device; AR processing module; AR coprocessor module; and An AR spatial positioning data acquisition module is configured to transmit the acquired spatial positioning data to the AR processing module or the AR coprocessing module. The AR processing module is configured to segment the collected spatial positioning data according to a predetermined ratio based on the computational complexity of the collected spatial positioning data, retain a portion of the segmented data for direct processing or transmission to the AR co-processing module, and transmit the remaining segmented data to at least the terminal processing module or the terminal co-processing module of the terminal device; the collected spatial positioning data is simultaneously distributed and processed by at least one of the AR processing module and the AR co-processing module, and at least one of the terminal processing module and the terminal co-processing module. The AR processing module or the AR coprocessing module is further configured to perform mixed calculations on the video information sent by the terminal device and the processing result of the spatial positioning data to obtain a final image, and send it to the display module of the AR head-mounted display device for image display.

2. The AR head-mounted display device according to claim 1 further includes a module for collecting data, wherein the AR spatial positioning data acquisition module is configured to collect feature point data and sensor data of the user's location space; wherein, The module for collecting data includes: An AR gaze tracking module is configured to monitor changes in the position of the user's gaze point; An AR image acquisition module is configured to acquire images of real-world objects within the user's space. The AR depth detection module is configured to detect and collect the structure of the user's environment in order to construct a map of the three-dimensional scene in which the user is located, and to capture and recognize the user's gestures to initiate corresponding operations. The AR processing module and the terminal processing module of the terminal device are further configured to upload the image information collected by the AR image acquisition module to the cloud server for information retrieval of the real object when the AR gaze tracking module detects that the user's eyes are focused on a real object in the user's space, and to present the image of the corresponding virtual object in front of the user through the display unit.

3. The AR head-mounted display device according to claim 1 further includes an AR power supply system.

4. An AR head-mounted display device and terminal device combination system, characterized in that, The AR headset is configured as an AR headset as described in any one of claims 1 to 3; The terminal device includes a terminal power system, a terminal communication module, and a terminal processing module; The AR headset and the terminal device establish a connection through the AR communication module and the terminal communication module; The terminal device is configured to send video information to the AR headset device. The AR processing module or AR coprocessing module in the AR headset device performs mixed calculations on the video information and the processing results of spatial positioning data to obtain the final image, which is then sent to the display module in the AR headset device for image display.

5. The system according to claim 4, characterized in that, In response to the AR headset device including an AR power system, the AR communication module is a wired communication module or a wireless communication module; When the AR communication module is a wireless communication module, the AR power system includes a charging module and a power supply module; When the AR communication module is a wired communication module, the AR power system includes at least a power supply module.

6. The system according to claim 4, characterized in that, The terminal device also includes a terminal coprocessing module; The AR coprocessing module is used to perform collaborative processing with the AR processing module and / or the terminal processing module and / or the terminal coprocessing module; The terminal coprocessing module is used to perform collaborative processing with the terminal processing module and / or the AR processing module or the AR coprocessing module.

7. The system according to claim 6, characterized in that, The AR coprocessing module is an AR display processing module, the terminal coprocessing module is a terminal display processing module, the AR head-mounted display device also includes an AR display module, and the terminal device also includes a terminal display module; At least one of the AR display processing module, the AR processing module, the terminal display processing module, and the terminal processing module processes the data of the AR display module and / or the terminal display module.

8. The system according to claim 6, characterized in that, The AR coprocessing module is an AR spatial positioning coprocessing module, and the terminal coprocessing module is a terminal spatial positioning coprocessing module. At least one of the AR processing module, the AR spatial positioning coprocessing module, the terminal processing module, and the terminal spatial positioning coprocessing module processes the data collected by the AR spatial positioning data acquisition module and transmits it to at least one of the AR processing module, the AR spatial positioning coprocessing module, the terminal processing module, and the terminal spatial positioning coprocessing module.

9. The system according to claim 6, characterized in that, The AR coprocessing module is an AR mixed reality coprocessing module, and the terminal coprocessing module is a terminal mixed reality coprocessing module. The AR mixed reality coprocessing module and / or the AR processing module are used to receive data collected by the AR spatial positioning data acquisition module, AR image acquisition module, AR depth detection module, and AR gaze tracking module, and at least one of the AR mixed reality coprocessing module, the AR processing module, the terminal mixed reality coprocessing module, and the terminal processing module processes the data collected by the AR spatial positioning data acquisition module, AR image acquisition module, AR depth detection module, and AR gaze tracking module, and transmits it to at least one of the AR processing module, the AR mixed reality coprocessing module, the terminal processing module, and the terminal mixed reality coprocessing module.

10. The system according to claim 6, characterized in that, The AR coprocessing module is an AR-coordinated mixed reality coprocessing module, and the terminal coprocessing module is a terminal-coordinated mixed reality coprocessing module; the terminal device also includes a terminal spatial positioning data acquisition module, a terminal image acquisition module, and a terminal depth detection module. At least one of the AR processing module, the AR collaborative mixed reality coprocessing module, the terminal processing module, and the terminal collaborative mixed reality coprocessing module processes the data collected by the AR spatial positioning data acquisition module, the AR image acquisition module, the AR depth detection module, and the AR gaze tracking module, as well as the data collected by the terminal spatial positioning data acquisition module, the terminal image acquisition module, and the terminal depth detection module, and transmits the data to at least one of the AR processing module, the AR collaborative mixed reality coprocessing module, the terminal processing module, and the terminal collaborative mixed reality coprocessing module.

11. The system according to claim 6, characterized in that, The AR coprocessing module is an AR motion behavior perception coprocessing module, the terminal coprocessing module is a terminal motion behavior perception coprocessing module, and the module for collecting data also includes an AR motion perception module and an AR touch control module; the terminal device also includes a terminal spatial positioning data acquisition module, a terminal motion perception module, a terminal depth detection module, and a terminal touch control module; At least one of the AR processing module, the AR motion behavior perception coprocessing module, the terminal processing module, and the terminal motion behavior perception coprocessing module processes the data collected by the AR spatial positioning data acquisition module, the AR motion perception module, the AR depth detection module, the AR gaze tracking module, and the AR touch control module, as well as the data collected by the terminal spatial positioning data acquisition module, the terminal motion perception module, the terminal depth detection module, and the terminal touch control module, and transmits the data to at least one of the AR processing module, the AR motion behavior perception coprocessing module, the terminal motion behavior perception coprocessing module, and the processing module.

12. The system according to claim 6, characterized in that, The AR coprocessing module is an AR audio processing module, the terminal coprocessing module is a terminal audio processing module, the module for collecting data also includes an AR audio module, and the terminal device also includes a terminal audio module. At least one of the AR processing module, the AR audio processing module, the terminal processing module, and the terminal audio processing module processes the data collected by the AR audio module and the terminal audio module, and transmits the data to at least one of the AR processing module, the AR audio processing module, the terminal audio processing module, and the terminal processing module for further processing, and then outputs the data through the AR audio module and / or the terminal audio module.

13. The system according to claim 6, characterized in that, At least two of the AR coprocessing module, the AR processing module, the terminal coprocessing module, and the terminal processing module work together to enable the AR head-mounted display device and the terminal device to work together.

Images