Head-mounted display device and camera tracking method, device, system and storage medium thereof
By obtaining the location information of the tracking device and generating steering control instructions, controlling the camera's rotation lens steering, the problem of single content collection and lack of real-time performance of the VR headset passive camera tracking method is solved, achieving richer and more real-time collection of content, improving User experience.
Patent Information
- Application Number
- CN202111602737.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-24
AI Technical Summary
The passive camera tracking method of existing VR headsets collects content in a single way and lacks real-time quality, which cannot meet the current user needs. How to implement active camera tracking to improve the user experience.
By obtaining the position information of the tracking device, generating steering control instructions, controlling the rotating lens in the camera to turn, realizing active camera tracking of the tracking device, overcoming the shortcomings of the traditional passive tracking method that the collection content of the traditional passive tracking method is single and lacking real-time.
It improves the richness and real-timeness of the content collected by the camera, improves the user experience, and realizes active camera tracking of tracking devices.
Smart Images

Figure CN114363513B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of head-mounted display devices, and in particular to a camera tracking method, device, system, head-mounted display device and computer-readable storage medium for head-mounted display devices. Background Art
[0002] At present, head-mounted display devices such as VR (Virtual Reality) headsets can block people's vision and hearing to the outside world, guiding users to feel like they are in a virtual environment. In the existing technology, camera tracking solutions such as VR tracking systems all adopt passive tracking methods, that is, the camera's image acquisition angle is fixed, and the camera's image acquisition range is updated by the user turning his head; with the development of head-mounted display technology, passive tracking methods are increasingly unable to meet current user needs due to their limitations in acquisition content and lack of real-time performance.
[0003] Therefore, how to realize active camera tracking of head-mounted display devices, overcome the shortcomings of traditional passive tracking methods such as single collected content and lack of real-time performance, and improve user experience is an issue that needs to be solved urgently. Summary of the invention
[0004] The purpose of the present invention is to provide a camera tracking method, device, system, head-mounted display device and computer-readable storage medium for a head-mounted display device, so as to realize active camera tracking of the head-mounted display device, overcome the shortcomings of the traditional passive tracking method of single collection content and lack of real-time performance, and improve the user experience.
[0005] In order to solve the above technical problems, the present invention provides a camera tracking method for a head mounted display device, comprising:
[0006] Obtain tracking location information of tracking devices;
[0007] Obtaining a steering control instruction for a camera in a head mounted display device according to the tracking position information;
[0008] The steering control instruction is used to control the rotating lens in the camera to steer.
[0009] Optionally, obtaining the tracking location information of the tracking device includes:
[0010] Receive the position prediction information sent by the tracking device; wherein the position prediction information is the position information obtained by the tracking device through motion trend detection.
[0011] Optionally, obtaining a steering control instruction of a camera in a head mounted display device according to the tracking position information includes:
[0012] Determining whether the tracking location information is within a current detection range of the camera;
[0013] If not, the steering control instruction is generated according to the tracking position information.
[0014] Optionally, using the steering control instruction to control a rotating lens in the camera to steer includes:
[0015] The steering control instruction is sent to the camera to control the rotating mirror to rotate to an angle corresponding to the steering control instruction, so that the image data collected by the camera includes an image of the position corresponding to the tracking position information.
[0016] Optionally, after obtaining a steering control instruction of a camera in a head mounted display device according to the tracking position information, the method further includes:
[0017] generating a coordinate system configuration instruction for the tracking device according to the steering control instruction;
[0018] The coordinate system configuration instruction is sent to the tracking device, and the tracking device is controlled to switch to the coordinate system corresponding to the coordinate system configuration instruction.
[0019] Optionally, generating a coordinate system configuration instruction of the tracking device according to the steering control instruction includes:
[0020] Determining the field of view angle information of the camera according to the rotation direction information and the rotation angle information in the steering control instruction;
[0021] The coordinate system configuration instruction is generated according to the field of view angle information.
[0022] The present invention also provides a camera tracking device for a head mounted display device, comprising:
[0023] A location acquisition module is used to obtain the tracking location information of the tracking device;
[0024] An instruction acquisition module, used to acquire a steering control instruction of a camera in a head mounted display device according to the tracking position information;
[0025] The steering control module is used to control the rotating lens in the camera to steer using the steering control instruction.
[0026] The present invention also provides a head mounted display device, comprising:
[0027] Memory for storing computer programs;
[0028] A processor is used to implement the steps of the camera tracking method of the head-mounted display device as described above when executing the computer program.
[0029] The present invention also provides a camera tracking system for a head-mounted display device, comprising: the head-mounted display device and the tracking device as described above; wherein the head-mounted display device includes a camera provided with a rotating lens.
[0030] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the camera tracking method of the head-mounted display device as described above are implemented.
[0031] The present invention provides a camera tracking method for a head mounted display device, comprising: obtaining tracking position information of a tracking device; obtaining a steering control instruction of a camera in the head mounted display device according to the tracking position information; and controlling a rotating lens in the camera to turn using the steering control instruction;
[0032] It can be seen that the present invention can obtain the steering control instruction of the camera in the head-mounted display device according to the tracking position information, and can use the tracking position information of the tracking device to be tracked to obtain the steering control instruction of the rotating lens in the camera, thereby controlling the rotating lens in the camera to turn accordingly, and realizing active camera tracking of the tracking device, which can overcome the shortcomings of the traditional passive tracking method of single collection content and lack of real-time performance, improve the richness and real-time performance of the camera's collection content, and enhance the user experience. In addition, the present invention also provides a camera tracking device, system, head-mounted display device and computer-readable storage medium for a head-mounted display device, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0034] Figure 1 A flowchart of a camera tracking method for a head mounted display device provided by an embodiment of the present invention;
[0035] Figure 2 A schematic diagram of a framework of a camera tracking system for a head mounted display device provided by an embodiment of the present invention;
[0036] Figure 3 A schematic diagram of a usage scenario of a camera tracking system for a head mounted display device provided by an embodiment of the present invention;
[0037] Figure 4A schematic diagram of coordinate system adjustment of another camera tracking method for a head mounted display device provided by an embodiment of the present invention;
[0038] Figure 5 A structural block diagram of a camera tracking device for a head mounted display device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] Please refer to Figure 1 , Figure 1 A flowchart of a camera tracking method for a head mounted display device provided by an embodiment of the present invention. The method may include:
[0041] Step 101: Obtain tracking location information of a tracking device.
[0042] Among them, the tracking device in this embodiment can be a device that needs to be tracked by a camera, that is, a device that needs to be tracked by the camera in the head-mounted display device; the tracking device in this embodiment can be communicatively connected with the head-mounted display device, so that the processor of the head-mounted display device can utilize the communication with the tracking device to obtain the tracking position information of the tracking device.
[0043] Specifically, the specific device type of the tracking device in this embodiment can be set by the designer according to practical scenarios and user needs. For example, the tracking device can be a handle device, such as Figure 2 A VR controller that is connected to a VR headset (such as a wired connection or a wireless connection); the tracking device may also be a mobile terminal device that is connected to a VR headset (such as a smart watch, a bracelet, or a mobile phone).
[0044] It is understandable that the tracking position information in this step may be information about the position that needs to be tracked by the camera, that is, information about the position that the camera in the head mounted display device needs to track. The specific content of the tracking position information in this step may be set by the designer according to practical scenarios and user needs. For example, the tracking position information may be the real-time position information of the tracking device, that is, the real-time position information collected by the tracking device in real time; the tracking position information may also be the predicted position information of the tracking device, such as the position information obtained by motion trend detection. This embodiment does not impose any restrictions on this.
[0045] Specifically, the specific method for the processor of the head mounted display device to obtain the tracking position information of the tracking device in this step can be set by the designer according to practical scenarios and user needs. For example, when the tracking position information is the real-time position information of the tracking device, the processor can directly receive the real-time position information sent by the tracking device. When the tracking position information is the position prediction information of the tracking device, the processor can directly receive the position prediction information sent by the tracking device. For example, the position prediction information can be the position information obtained by the tracking device through motion trend detection; the processor can also use the received real-time position information sent by the tracking device to obtain the position prediction information of the tracking device through motion trend detection. This embodiment does not impose any restrictions on this.
[0046] Correspondingly, when the processor of the head-mounted display device receives the position prediction information sent by the tracking device, the method provided in this embodiment may further include a process in which the tracking device obtains and sends the position prediction information to the head-mounted display device before step 101. For example, the tracking device may obtain and send the position prediction information to the head-mounted display device by using the collected real-time position information through motion trend detection. The tracking device may also determine whether the rotating lens in the camera needs to turn based on the position prediction information obtained through motion trend detection; if so, the position prediction information is sent to the head-mounted display device; if not, the present process is terminated or the real-time position information continues to be collected and the position prediction information is obtained. For example, Figure 2 As shown, the VR handle (i.e., tracking device) can detect the movement trend of the VR handle through the internal IMU (inertial measurement unit) to obtain position prediction information; the field of view angle of the camera at the current angle does not exceed the camera detection range as a threshold, and when the threshold is reached, it is determined that the rotating lens needs to turn, and then the position prediction information is sent to the processor of the VR headset (i.e., head-mounted display device), so that the processor can use the received position prediction information (i.e., tracking position information) to control the rotating lens to effectively turn.
[0047] Step 102: Obtain a steering control instruction for a camera in a head mounted display device according to the tracking position information.
[0048] The steering control instruction in this step may be an instruction for controlling the steering of the rotating lens in the camera, that is, the camera may control the rotating lens to turn accordingly according to the received steering control instruction. The specific content of the steering control instruction in this step may be determined by the designer according to practical scenarios and user needs. For example, the steering control instruction may include the rotation angle information of the rotating lens, or the rotation angle information and the rotation direction information of the rotating lens. As long as the processor can use the steering control instruction to control the rotating lens in the camera to turn accordingly and increase the field of view (FOV) of the fixed-position camera, this embodiment does not impose any restrictions on this.
[0049] Specifically, the specific method for the processor of the head mounted display device to obtain the steering control instruction of the camera in the head mounted display device according to the tracking position information in this step can be set by the designer according to the practical scenario and user needs. For example, the processor can directly generate the steering control instruction according to the tracking position information. For example, the processor can generate the steering control instruction using the target rotation angle corresponding to the position prediction information and the current rotation angle of the rotating lens, so that the camera can control the rotating lens to rotate to the target rotation angle according to the steering control instruction. The processor can also determine whether the rotating lens in the camera needs to turn according to the tracking position information; if so, generate the steering control instruction according to the tracking position information; if not, end this process or return to step 101 to continue to obtain the tracking position information; for example, the processor can determine whether the tracking position information (such as the position prediction information) is within the current detection range of the camera; if not, it can be determined that the rotating lens needs to turn, and the steering control instruction can be generated according to the tracking position information; if so, it can be determined that the rotating lens does not need to turn, and end this process or return to step 101 to continue to obtain the tracking position information.
[0050] Step 103: Using the steering control instruction, control the rotating lens in the camera to steer.
[0051] It should be noted that in this step, the processor of the head-mounted display device can use the obtained steering control instruction to control the rotating lens in the camera to turn accordingly, so that the image data collected by the camera after the turning can include an image of the position corresponding to the tracking position information.
[0052] Specifically, this embodiment does not limit the specific manner in which the processor uses the steering control instruction to control the rotating lens in the camera to turn. For example, the processor can directly send the steering control instruction to the camera, that is, the processor sends the steering control instruction to the camera in this step to control the rotating lens to rotate to the angle corresponding to the steering control instruction, so that the image data collected by the camera includes an image of the position corresponding to the tracking position information, such as the center of the picture collected by the camera can be the position corresponding to the tracking position information.
[0053] It can be understood that the method provided in this embodiment can also include a process of adjusting and controlling the coordinate system of the tracking device. For example, in this embodiment, the processor can generate and send a coordinate system configuration instruction to the tracking device after step 102, so that the tracking device can adjust the coordinate system (such as the IMU plane coordinate system) to the coordinate system corresponding to the camera field of view after the rotating lens is turned according to the coordinate system configuration instruction. For example, the processor can generate a coordinate system configuration instruction for the tracking device according to the steering control instruction before or after step 103; send the coordinate system configuration instruction to the tracking device to control the tracking device to switch to the coordinate system corresponding to the coordinate system configuration instruction. For example, the processor can determine the field of view angle information of the camera according to the rotation direction information and the rotation angle information in the steering control instruction; generate a coordinate system configuration instruction according to the field of view angle information to control the tracking device to switch to the coordinate system corresponding to the field of view angle information; that is, the processor can use the coordinate system corresponding to each rotation angle stored in advance, and obtain the target rotation angle and determine the coordinate system corresponding to the target rotation angle according to the rotation direction information and the rotation angle information in the steering control instruction, so as to generate a coordinate system configuration instruction corresponding to the coordinate system, thereby controlling the tracking device to switch to the coordinate system.
[0054] Correspondingly, such as Figure 2 and Figure 3 As shown, in this embodiment, the processor can also generate a coordinate system configuration instruction for the tracking device after step 103 based on the image data (i.e., video information) collected by the camera after the rotating lens is turned; send the coordinate system configuration instruction to the tracking device, and control the tracking device to switch to the coordinate system corresponding to the coordinate system configuration instruction. Figure 4 As shown, since the motion trend detection of the VR handle requires a reference coordinate system, different plane coordinate systems can be used according to the different rotation angles of the rotating lens of the camera; by setting different plane coordinate systems (such as IMU plane coordinate systems) according to the field of view of different cameras, after the rotating lens rotates α, it can be converted from plane coordinate system 1 to plane coordinate system 2 according to the different field of view angles of the camera.
[0055] Furthermore, the method provided in this embodiment may also include a wearing detection process of a head-mounted display device. For example, in this embodiment, the processor may determine whether the powered-on head-mounted display device is in a wearing state before step 101; if so, enter step 101; if not, end this process; thereby reducing the power consumption of the head-mounted display device.
[0056] In this embodiment, the embodiment of the present invention obtains the steering control instruction of the camera in the head-mounted display device according to the tracking position information, and can use the tracking position information of the tracking device to be tracked to obtain the steering control instruction for controlling the rotating lens in the camera, thereby controlling the rotating lens in the camera to turn accordingly, thereby realizing active camera tracking of the tracking device, which can overcome the shortcomings of the traditional passive tracking method of single collection content and lack of real-time performance, improve the richness and real-time performance of the camera's collection content, and enhance the user experience.
[0057] Corresponding to the above method embodiment, an embodiment of the present invention also provides a camera tracking device for a head-mounted display device. The camera tracking device for a head-mounted display device described below and the camera tracking method for a head-mounted display device described above can be referenced to each other.
[0058] Please refer to Figure 5 , Figure 5 This is a structural block diagram of a camera tracking device for a head mounted display device provided by an embodiment of the present invention. The device may include:
[0059] A location acquisition module 10 is used to acquire tracking location information of a tracking device;
[0060] The instruction acquisition module 20 is used to acquire the steering control instruction of the camera in the head mounted display device according to the tracking position information;
[0061] The steering control module 30 is used to control the rotating lens in the camera to steer using a steering control instruction.
[0062] Optionally, the location acquisition module 10 may be specifically configured to receive location prediction information sent by a tracking device; wherein the location prediction information is location information obtained by the tracking device through motion trend detection.
[0063] Optionally, the instruction acquisition module 20 may include:
[0064] A judgment submodule is used to judge whether the tracking position information is within the current detection range of the camera;
[0065] The instruction generation submodule is used to generate a steering control instruction based on the tracking position information if the vehicle is not within the current detection range of the camera.
[0066] Optionally, the steering control module 30 may be specifically configured to send a steering control instruction to the camera to control the rotating lens to rotate to an angle corresponding to the steering control instruction, so that the image data collected by the camera includes an image of a position corresponding to the tracking position information.
[0067] Optionally, the device may also include:
[0068] An instruction generation module, used for generating a coordinate system configuration instruction of a tracking device according to a steering control instruction;
[0069] The coordinate configuration module is used to send the coordinate system configuration instruction to the tracking device and control the tracking device to switch to the coordinate system corresponding to the coordinate system configuration instruction.
[0070] Optionally, the instruction generation module may include:
[0071] The field of view angle determination submodule is used to determine the field of view angle information of the camera according to the rotation direction information and the rotation angle information in the steering control instruction;
[0072] The instruction generation submodule is used to generate coordinate system configuration instructions according to the field of view angle information.
[0073] In this embodiment, the embodiment of the present invention obtains the steering control instruction of the camera in the head-mounted display device according to the tracking position information through the instruction acquisition module 20, and can use the tracking position information of the tracking device to be tracked to obtain the steering control instruction for controlling the rotating lens in the camera, thereby controlling the rotating lens in the camera to turn accordingly, thereby realizing active camera tracking of the tracking device, which can overcome the shortcomings of the traditional passive tracking method of single collection content and lack of real-time performance, improve the richness and real-time performance of the camera's collection content, and enhance the user experience.
[0074] Corresponding to the above method embodiment, an embodiment of the present invention further provides a head-mounted display device. The head-mounted display device described below and the camera tracking method of the head-mounted display device described above can refer to each other.
[0075] An embodiment of the present invention provides a head mounted display device, comprising:
[0076] Memory for storing computer programs;
[0077] The processor is used to implement the steps of the camera tracking method of the head-mounted display device provided in the above embodiment when executing the computer program.
[0078] Among them, the head-mounted display device in this embodiment can be provided with a camera, and a rotating lens can be provided in the camera.
[0079] Specifically, the head mounted display device provided in this embodiment may be a HMD (Head Mount Display) device such as a VR device, an AR device or an MR device, such as Figure 2 VR headset in.
[0080] Corresponding to the above method embodiment, an embodiment of the present invention further provides a camera tracking system for a head-mounted display device. The camera tracking system for a head-mounted display device described below and the camera tracking method for a head-mounted display device described above can be referenced to each other.
[0081] An embodiment of the present invention provides a camera tracking system for a head-mounted display device, comprising: the head-mounted display device and a tracking device as described above; wherein the head-mounted display device includes a camera provided with a rotating lens.
[0082] Specifically, the tracking device in this embodiment can be connected to the head mounted display device for communication (such as wired connection or wireless connection). The tracking device can be specifically a handle device, such as Figure 2 The tracking device can also be a mobile terminal device connected to the head-mounted display device, such as a smart watch, a bracelet, and a mobile phone.
[0083] Corresponding to the above method embodiment, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium described below and the camera tracking method of a head-mounted display device described above can be referenced to each other.
[0084] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the camera tracking method for a head-mounted display device provided in the above method embodiment are implemented.
[0085] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device, system, head-mounted display device and computer-readable storage medium disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0086] The above is a detailed introduction to the camera tracking method, device, system, head-mounted display device and computer-readable storage medium of a head-mounted display device provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
Claims
1. A camera tracking method for a head mounted display device, characterized in that: include: Obtain tracking location information of tracking devices; The tracking device is communicatively connected to the head mounted display device, so that the processor of the head mounted display device obtains the tracking position information by communicating with the tracking device; Obtaining a steering control instruction for a camera in a head mounted display device according to the tracking position information; Using the steering control instruction, the rotating lens in the camera is controlled to turn; Wherein, obtaining a steering control instruction of a camera in a head mounted display device according to the tracking position information includes: Determining whether the tracking location information is within a current detection range of the camera; If not, generating the steering control instruction according to the tracking position information; The method of controlling the rotating lens in the camera to turn by using the turning control instruction includes: The steering control instruction is sent to the camera to control the rotating mirror to rotate to an angle corresponding to the steering control instruction, so that the image data collected by the camera includes an image of the position corresponding to the tracking position information.
2. The camera tracking method of a head mounted display device according to claim 1, characterized in that: The obtaining of tracking location information of the tracking device includes: Receive the position prediction information sent by the tracking device; wherein the position prediction information is the position information obtained by the tracking device through motion trend detection.
3. The camera tracking method of the head mounted display device according to claim 1 or 2, characterized in that: After obtaining the steering control instruction of the camera in the head mounted display device according to the tracking position information, the method further includes: generating a coordinate system configuration instruction for the tracking device according to the steering control instruction; The coordinate system configuration instruction is sent to the tracking device, and the tracking device is controlled to switch to the coordinate system corresponding to the coordinate system configuration instruction.
4. The camera tracking method of the head mounted display device according to claim 3, characterized in that: The step of generating a coordinate system configuration instruction for the tracking device according to the steering control instruction comprises: Determining the field of view angle information of the camera according to the rotation direction information and the rotation angle information in the steering control instruction; The coordinate system configuration instruction is generated according to the field of view angle information.
5. A camera tracking device for a head mounted display device, characterized in that: include: A location acquisition module is used to obtain the tracking location information of the tracking device; The tracking device is communicatively connected to the head mounted display device, so that the processor of the head mounted display device obtains the tracking position information by communicating with the tracking device; An instruction acquisition module, used to acquire a steering control instruction of a camera in a head mounted display device according to the tracking position information; A steering control module, used to control the rotating lens in the camera to steer using the steering control instruction; Wherein, the instruction acquisition module includes: A determination submodule, used to determine whether the tracking position information is within the current detection range of the camera; An instruction generation submodule, for generating the steering control instruction according to the tracking position information if the position is not within the current detection range of the camera; The steering control module is specifically used to send the steering control instruction to the camera, control the rotating lens to rotate to the angle corresponding to the steering control instruction, so that the image data collected by the camera includes the image of the position corresponding to the tracking position information.
6. A head mounted display device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the camera tracking method for a head mounted display device as described in any one of claims 1 to 4 when executing the computer program.
7. A camera tracking system for a head mounted display device, characterized in that: include: A tracking device and a head mounted display device as claimed in claim 6; wherein the head mounted display device comprises a camera provided with a rotating lens.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the camera tracking method of the head-mounted display device according to any one of claims 1 to 4 are implemented.
