Spatial positioning method for a split virtual system, virtual system

Abstract

The application provides a space positioning method of a separated virtual system, a controller thereof and a virtual system, and solves the technical problems that the virtual system cannot be set separately and cannot be accurately positioned in the prior art. The space positioning method of the separated virtual system provided by the application determines the translational degree of freedom from a first image captured by a camera arranged on a mobile terminal when the mobile terminal is installed on a virtual head-mounted device and enters a virtual mode, obtains the rotational degree of freedom based on a first inertia sensor arranged on the mobile terminal, and further determines the space positioning information of the virtual head-mounted device, thereby increasing the accuracy of the space positioning of the mobile terminal. In addition, the human-computer interaction based on the space positioning information improves the game experience, thereby improving the performance of the separated head-mounted reality device of the mobile terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to the virtual technical field, and particularly relates to a space positioning method of a separated virtual system and a virtual system. BACKGROUND

[0002] At present, a new type of virtual reality (VR) head-mounted device composed of a VR box and a mobile terminal has appeared in the market. The performance of the new type of VR head-mounted device is mainly determined by the performance of the mobile terminal, and the performance is far less than that of a traditional computer-VR head-mounted device or a VR all-in-one machine. The display of the computer-VR head-mounted device or the VR all-in-one machine is generally inseparable from the VR virtual head-mounted device, and can only be used in a matched manner, so that the convenience is limited.

[0003] However, after the mobile terminal is disassembled, the control handle and the head-mounted device need to be accurately positioned again, otherwise smooth human-computer interaction cannot be performed, and the game experience is affected. At present, there is no method for accurately positioning the separable VR head-mounted device of the mobile terminal. SUMMARY

[0004] Therefore, the present application provides a space positioning method of a separated virtual system and a virtual system, and solves the technical problems that the virtual system cannot be separated and accurately positioned in the prior art.

[0005] As a first aspect of the present application, the present application provides a space positioning method of a separated virtual system, the separated virtual system comprising: a virtual head-mounted device, a mobile terminal detachably installed on the virtual head-mounted device, and a control handle in communication connection with the mobile terminal, wherein a first inertial sensor is arranged on the mobile terminal, a camera is arranged on the mobile terminal and / or the virtual head-mounted device, and the first inertial sensor is used for detecting IMU data of the mobile terminal; wherein the space positioning method of the separated virtual system comprises: when it is determined that the mobile terminal is installed on the virtual head-mounted device, controlling the camera to take an image of an environment where the virtual head-mounted device is located, to form a first image; controlling the first inertial sensor to detect IMU data of the virtual head-mounted device; and calculating according to the IMU data of the virtual head-mounted device and the first image, to generate space positioning information of the virtual head-mounted device.

[0006] In an embodiment of the present application, the spatial positioning information of the virtual head-mounted device is generated by calculating based on the IMU data of the virtual head-mounted device and the first image, comprising: determining the translational freedom data of the virtual head-mounted device based on the first image; and determining the 6DOF data of the virtual head-mounted device based on the translational freedom data of the virtual head-mounted device and the IMU data of the virtual head-mounted device; wherein the spatial positioning information of the virtual head-mounted device comprises the 6DOF data of the virtual head-mounted device.

[0007] In an embodiment of the present application, the translational freedom data of the virtual head-mounted device is determined based on the first image, comprising: extracting image feature points in the first image, and extracting reference feature points in a reference image; matching the image feature points and the reference feature points to generate effective feature points; extracting key regions from the first image based on a sliding window method according to all effective feature points; generating map points based on the key regions, the effective feature points, and a map of a space where the virtual display device is located; and determining map information of the virtual display device based on the map points and the map, the map information comprising the translational freedom data.

[0008] In an embodiment of the present application, before extracting key regions from the first image based on a sliding window method; the translational freedom data of the virtual head-mounted device is determined based on the first image, further comprising: determining whether the first image is initialized; when it is determined that the first image is initialized, extracting key regions from the first image based on a sliding window method according to all effective feature points.

[0009] In an embodiment of the present application, the translational freedom data of the virtual head-mounted device is determined based on the first image, further comprising: when it is determined that the first image is not initialized, repositioning the first image according to the map to determine the positioning state of the first image; extracting key regions from the first image based on a sliding window method according to all effective feature points, comprising: extracting key regions from the first image based on a sliding window method according to all effective feature points based on the positioning state of the first image.

[0010] In an embodiment of the present application, when it is determined that the first image is initialized, extracting key regions from the first image based on a sliding window method according to all effective feature points, comprising: determining the sliding window state of the sliding window in the first image; determining the positioning state of the first image according to the sliding window state; sliding the sliding window across the first image according to the positioning state; and detecting the image region crossed by the sliding window, and saving the sliding window that detects effective feature points as the key region.

[0011] In an embodiment of the present application, the control handle is provided with an infrared sensor and a second inertial sensor, and the camera on the mobile terminal is further configured to capture a light spot image of the infrared sensor; wherein the spatial positioning method of the split virtual system further comprises: when it is determined that the mobile terminal is mounted on the virtual head-mounted device, controlling the camera to capture the infrared sensor on the control handle to form a light spot image; controlling the second inertial sensor to detect the IMU data of the control handle; and calculating the spatial positioning information of the control handle according to the spatial positioning information of the virtual head-mounted device, the IMU data of the control handle, and the light spot image.

[0012] In an embodiment of the present application, when it is determined that the mobile terminal is mounted on the virtual head-mounted device, the camera is controlled to capture the environment where the virtual head-mounted device is located to form a first image, comprising: after the connection plug of the mobile terminal is in communication with the connection plug of the virtual head-mounted device, the camera is controlled to capture the environment where the virtual head-mounted device is located to form a first image.

[0013] As a second aspect of the present application, the present application provides a split virtual system, comprising: a mobile terminal; a first inertial sensor arranged on the mobile terminal, the first inertial sensor being configured to detect the IMU data of the mobile terminal; a virtual head-mounted device, the smart terminal being detachably mounted on the virtual head-mounted device; a camera arranged on the mobile terminal and / or the virtual head-mounted device; a spatial positioning controller of the split virtual system, the spatial positioning controller being in communication connection with the mobile terminal, the first inertial sensor, and the camera respectively; wherein the spatial positioning controller comprises: a function switching unit configured to switch the function of the mobile terminal to a virtual head-mounted function when it is determined that the mobile terminal is mounted on the virtual head-mounted device; a camera control unit configured to control the camera to capture the environment of the virtual head-mounted device to form a first image; a sensor control unit configured to control the first inertial sensor to detect the IMU data of the virtual head-mounted device; and a positioning unit configured to calculate the spatial positioning information of the virtual head-mounted device according to the IMU data of the virtual head-mounted device and the first image.

[0014] In an embodiment of the present application, the positioning unit comprises: a translation freedom degree determining module configured to determine translation freedom degree data of the virtual head-mounted device based on the first image; and a six freedom degree determining module configured to determine 6DOF data of the virtual reality head-mounted device based on the translation freedom degree data of the virtual head-mounted device and the IMU data of the virtual head-mounted device; wherein the spatial positioning information of the virtual head-mounted device comprises the 6DOF data of the virtual head-mounted device.

[0015] In an embodiment of the present application, the translation freedom degree determining module comprises: a feature determining module configured to extract image feature points from the first image, extract reference feature points from a reference image, and match the image feature points and the reference feature points to generate effective feature points; a region extracting module configured to extract key regions from the first image based on a sliding window method according to all effective feature points; a map point generating module configured to generate map points based on the key regions, the effective feature points, and a map of a space where the virtual display device is located; and a map information determining module configured to determine map information of the virtual display device based on the map points and the map, wherein the map information comprises the translation freedom degree data.

[0016] In an embodiment of the present application, the separated virtual system further comprises: a control handle; and an infrared sensor and a second inertial sensor arranged on the control handle, wherein the second inertial sensor is configured to measure IMU data of the control handle; the camera on the mobile terminal is further configured to capture a light spot image of the infrared sensor; wherein the camera control unit is further configured to control the camera to capture the infrared sensor on the control handle to form a light spot image; the sensor control unit is further configured to control the second inertial sensor to detect the IMU data of the control handle; and the positioning unit is further configured to calculate and generate spatial positioning information of the control handle based on the spatial positioning information of the virtual head-mounted device, the IMU data of the control handle, and the light spot image.

[0017] The spatial positioning method of the separated virtual system provided in the present application can determine translation freedom degree from a first image captured by a camera arranged on a mobile terminal when the mobile terminal is installed on a virtual head-mounted device and the mobile terminal enters a virtual mode, and can obtain rotation freedom degree based on a first inertial sensor arranged on the mobile terminal, thereby determining spatial positioning information of the virtual head-mounted device, and increasing the accuracy of spatial positioning of the mobile terminal. In addition, human-computer interaction based on the spatial positioning information can improve the gaming experience, thereby improving the performance of the separated head-mounted reality device of the mobile terminal. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description thereof taken in conjunction with the accompanying drawings, in which: like reference numerals in the figures refer to identical or similar components. The drawings provided are for illustrative purposes only and, therefore, should not be considered to limit the present application. In the drawings:

[0019] Figure 1 Fig. 1 shows a flowchart of a method for spatial positioning of a split virtual system according to an embodiment of the present application;

[0020] Figure 2 Fig. 2 shows a flowchart of a method for spatial positioning of a split virtual system according to another embodiment of the present application;

[0021] Figure 3 Fig. 3 shows a flowchart of a method for spatial positioning of a split virtual system according to another embodiment of the present application;

[0022] Figure 4 Fig. 4 shows a flowchart of a method for spatial positioning of a split virtual system according to another embodiment of the present application;

[0023] Figure 5 Fig. 5 shows a flowchart of a method for spatial positioning of a split virtual system according to another embodiment of the present application;

[0024] Figure 6 Fig. 6 shows a flowchart of a method for spatial positioning of a split virtual system according to another embodiment of the present application;

[0025] Figure 7 Fig. 7 shows a schematic diagram of a split virtual system according to an embodiment of the present application;

[0026] Figure 8 Fig. 8 shows a schematic diagram of a split virtual system according to another embodiment of the present application;

[0027] Figure 9 Fig. 9 shows a schematic diagram of a split virtual system according to another embodiment of the present application;

[0028] Figure 10 Fig. 10 shows a schematic diagram of a split virtual system according to another embodiment of the present application;

[0029] Figure 11 Fig. 11 shows a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited. All directional indications (such as upper, lower, left, right, front, back, top, bottom, etc.) in the embodiments of the application are only used to explain the relative position relationship, movement condition, etc. between the components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications will also change accordingly. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0031] In addition, the reference to "embodiments" herein means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily mean that it refers to the same embodiment, nor is it independent or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0033] Figure 1 A flowchart of a spatial positioning method of a split virtual system provided by the application is shown, wherein the split system includes a virtual head-mounted device, a mobile terminal detachably installed on the virtual head-mounted device, and a control handle in communication connection with the mobile terminal, wherein the mobile terminal is provided with a first inertial sensor, and the mobile terminal and / or the virtual head-mounted device is provided with a camera device, and the first inertial sensor is used to detect the IMU data of the mobile terminal. Specifically, the number and installation position of the camera device can be the following three cases: (1) the number of camera devices is equal to 1, and the camera device can be installed on the mobile terminal or the virtual head-mounted device; (2) the number of camera devices is greater than 1, and the camera devices can be all installed on the mobile terminal or all installed on the virtual head-mounted device; (3) the number of camera devices is greater than 1, and part of the camera devices are installed on the mobile terminal and part of the camera devices are installed on the virtual head-mounted device.

[0034] Specifically, the inertial sensor (Inertial Measurement Unit, IMU for short) is a device for measuring the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three-axis coordinate system of the carrier, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system. Therefore, the inertial sensor can measure the angular velocity and acceleration of the object in three-dimensional space and calculate the attitude of the object, such as calculating the rotational degrees of freedom of the object. The rotational degrees of freedom refer to the three position-related degrees of freedom of up, down, front, back, and left and right. The IMU data is the result data of the inertial sensor detecting the object, that is, the angular velocity and acceleration data of the object in three-dimensional space detected by the inertial sensor. Therefore, the first inertial sensor arranged on the mobile terminal can detect the IMU data of the mobile terminal, and the IMU data of the mobile terminal can be used to calculate the attitude of the mobile terminal, such as the rotational degrees of freedom of the mobile terminal. The rotational degrees of freedom refer to the three position-related degrees of freedom of up, down, front, back, and left and right.

[0035] As shown in Figure 1 The spatial positioning method of the split virtual system includes the following steps:

[0036] S101: When it is determined that the mobile terminal is installed on the virtual head-mounted device, the camera is controlled to capture the environment of the virtual head-mounted device to form a first image;

[0037] Since the mobile terminal virtual head-mounted device is detachably connected, the mobile terminal not only has the virtual head-mounted function, but also has other functions. For example, when the mobile terminal is a smart phone, the mobile terminal can be used as a smart phone without being installed on the virtual head-mounted device, and has the functions of a smart phone, such as communication function, display function, etc. When the mobile terminal is installed on the virtual head-mounted device, the mobile terminal exercises the virtual head-mounted function, and together with the virtual head-mounted device and the control handle, forms a virtual system. For example, when the virtual display device is a VR helmet, that is, when the mobile terminal is installed on the VR helmet, the mobile terminal together with the VR helmet and the control handle forms a VR system.

[0038] Specifically, the specific detachable installation mode of the mobile terminal detachably installed on the virtual head-mounted device can be as follows: the virtual head-mounted device is provided with a reserved installation slot, and the mobile terminal is installed into the reserved installation slot of the virtual head-mounted device, so that the mobile terminal is fixed on the virtual head-mounted device.

[0039] Specifically, the communication connection mode of the mobile terminal and the control handle can be as follows: the mobile terminal is provided with a connection port at two ends, so as to be connected with the control handle through a connection line connected to the connection port. It can be understood that the mobile terminal and the control handle can also be connected through a wireless mode such as Bluetooth. When the mobile terminal and the control handle are successfully connected, it is confirmed to enter the virtual mode. After entering the virtual mode, the mobile terminal has the functions of game interface display and image processing.

[0040] When it is determined that the mobile terminal is installed on the virtual head-mounted device, that is, the mobile terminal exercises the virtual head-mounted function, at this time, the mobile terminal and the control handle will also be connected in communication, at this time, the mobile terminal exercises the virtual head-mounted function, that is, the mobile terminal, the control handle and the virtual head-mounted device together constitute a virtual system, and in the working process of the virtual system, the mobile terminal and the control handle need spatial positioning, at this time, the camera device located on the mobile terminal controls to shoot the environment of the virtual head-mounted device, the camera device is controlled to start working to shoot the environment where the virtual head-mounted device is located, to form a first image, for spatial positioning of the mobile terminal.

[0041] S102: control the first inertial sensor to detect the IMU data of the virtual head-mounted device;

[0042] When it is determined that the mobile terminal is connected with the virtual head-mounted device, the mobile terminal, the virtual head-mounted device and the control handle constitute a virtual system, and the user operates the virtual system to experience the virtual scene, at this time, the first inertial sensor is controlled to start working, and the first inertial sensor detects the IMU data of the virtual head-mounted device under control.

[0043] The first inertial sensor arranged on the mobile terminal is controlled to detect the IMU data of the mobile terminal.

[0044] S103: calculate the IMU data of the virtual head-mounted device and the first image to generate spatial positioning information of the virtual head-mounted device.

[0045] When the first image transmitted by the camera device and the IMU data transmitted by the first inertial sensor are acquired, the spatial positioning information of the mobile terminal, that is, the six-degree-of-freedom data (hereinafter referred to as 6DOF data) of the mobile terminal, can be determined based on the IMU data and the first image, that is, six angles of freedom can be obtained based on the translational freedom and the rotational freedom.

[0046] The application provides a space positioning method of a separated virtual system. When a mobile terminal is installed on a virtual head-mounted device, the mobile terminal enters a virtual mode, a first image captured by a camera arranged on the mobile terminal and a rotation degree of freedom obtained by a first inertial sensor arranged on the mobile terminal are used to determine the space positioning information of the virtual head-mounted device, thereby improving the accuracy of the space positioning of the mobile terminal. In addition, the space positioning information is used for human-computer interaction, thereby improving the game experience and the performance of the separated head-mounted reality device.

[0047] In an embodiment of the application, Figure 2 As shown in FIG. 6, the flowchart shows the space positioning method of the separated virtual system provided by another embodiment of the application. Figure 2 As shown in FIG. 6, step S103 (calculating the IMU data of the virtual head-mounted device and the first image to generate the space positioning information of the virtual head-mounted device) specifically includes the following steps:

[0048] Step S1031: determining the translation degree of freedom data of the virtual head-mounted device based on the first image.

[0049] The translation degree of freedom refers to the left-right, up-down and forward-backward movement data of the virtual head-mounted device in space.

[0050] Step S1032: determining the 6DOF data of the virtual reality head-mounted device based on the translation degree of freedom data of the virtual head-mounted device and the IMU data of the virtual head-mounted device; wherein the space positioning information of the virtual head-mounted device includes the 6DOF data of the virtual head-mounted device.

[0051] The IMU data of the virtual head-mounted device refers to the rotation IMU data of the virtual head-mounted device, and the rotation IMU data refers to the three rotation degree of freedom data of the virtual head-mounted device. The six degree of freedom data (6DOF data) of the virtual head-mounted device can be determined according to the translation IMU data and the rotation IMU data of the virtual head-mounted device. The accurate positioning of the virtual head-mounted device in space can be tracked according to the 6DOF data of the virtual head-mounted device, and the movement of the virtual head-mounted device in the real world can be accurately mapped in the virtual scene.

[0052] In an embodiment of the application, Figure 3 As shown in FIG. 6, the flowchart shows the space positioning method of the separated virtual system provided by another embodiment of the application. Figure 3 As shown in FIG. 6, step S1031 (determining the translation degree of freedom data of the virtual head-mounted device based on the first image) specifically includes the following steps:

[0053] Step S1: extracting image feature points in the first image and reference feature points in a reference image.

[0054] The reference image refers to a part of the image in the map of the space environment where the virtual head-mounted device is located, for example, when the space environment where the virtual head-mounted device is located is a living room, the map is a living room map, and the reference image is an image of a part of the space map, for example, an image of a corner of the living room.

[0055] Step S2: match the image feature points and the reference feature points to generate effective feature points;

[0056] The image feature points are matched with the reference feature points, and the image feature points that are successfully matched are effective feature points.

[0057] Step S3: according to all effective feature points, extract a key region from the first image based on a sliding window method;

[0058] The sliding window method can prune the search space by the part of the leftmost sliding window, thereby reducing repeated calculation, reducing time complexity, and avoiding brute-force search.

[0059] Step S4: based on the key region, the effective feature points, and the map of the space where the virtual display device is located, generate map points; and

[0060] Step S5: determine the map information of the virtual display device based on the map points and the map, and the map information includes translational freedom data.

[0061] The present application determines the effective feature points in the first image based on the map of the space and the reference image, and then extracts the key region in the first image based on the sliding window method according to the effective feature points, which not only improves the accuracy of the translational freedom data, but also reduces repeated calculation, reduces time complexity, and improves efficiency.

[0062] In an embodiment of the present application, Figure 4 As shown in the flowchart of the space positioning method of the split virtual system provided by another embodiment of the present application, Figure 4 As shown, before step S3 (extracting a key region from the first image based on a sliding window method according to all effective feature points), step S1031 (determining the translational freedom data of the virtual head-mounted device based on the first image) further includes the following steps:

[0063] Step S10: determining whether the first image is initialized;

[0064] When the determination result in step S10 is yes, that is, the first image has been initialized, step S3 is executed, that is, a key region is extracted from the first image based on a sliding window method according to effective feature points, wherein the first image is the initialized image.

[0065] The first image is initialized refers to that the brightness, gray scale and resolution of the first image meet preset requirements, and the first image is determined as an initialized image.

[0066] When the determination result in step S10 is no, that is, the first image is not initialized, step S6 is performed.

[0067] Step S6: repositioning the first image according to the map to determine the positioning state of the first image.

[0068] At this time, step S3 (extracting a key region from the first image based on the sliding window method according to all effective feature points) specifically includes: S31: based on the positioning state of the first image, extracting a key region from the first image based on the sliding window method according to all effective feature points.

[0069] In an embodiment of the present application, Figure 5 As shown in the figure, the flowchart of the spatial positioning method of the split virtual system provided by another embodiment of the present application is shown. Figure 5 As shown, step S3 (extracting a key region from the first image based on the sliding window method according to all effective feature points) specifically includes the following steps:

[0070] Step S32: determining the sliding window state of the sliding window in the first image.

[0071] Step S33: determining the positioning state of the first image according to the sliding window state.

[0072] Step S34: sliding the sliding window through the first image according to the positioning state; and

[0073] Step S35: detecting the image region through which the sliding window is slid, and saving the sliding window in which effective feature points are detected as a key region.

[0074] In an embodiment of the present application, an infrared sensor and a second inertial sensor are installed on the control handle, and the camera on the mobile terminal is further used to shoot a light spot image of the infrared sensor. Figure 6 As shown in the figure, the flowchart of the spatial positioning method of the split virtual system provided by another embodiment of the present application is shown. Figure 6 As shown, the spatial positioning method of the split virtual system further includes:

[0075] Step S104: when it is determined that the mobile terminal is installed on the virtual head-mounted device, controlling the camera to shoot the infrared sensor on the control handle to form a light spot image.

[0076] When it is confirmed that the mobile terminal is installed on the virtual head-mounted display device, the control handle establishes a communication connection with the mobile terminal. At this time, the camera device set on the mobile terminal takes a picture of the infrared sensor on the control handle to form a light spot image of the infrared sensor.

[0077] Step S105: Control the second inertial sensor to detect the IMU data of the control handle;

[0078] When it is confirmed that the mobile terminal is installed on the virtual head-mounted display device, the control handle establishes a communication connection with the mobile terminal. At this time, the control two inertial sensors are activated to detect the IMU data of the control handle.

[0079] Step S106: Calculate and generate the spatial positioning information of the control handle based on the spatial positioning information of the virtual head-mounted display device, the IMU data of the control handle, and the light spot image.

[0080] The spatial positioning of the control handle can be achieved through steps S104-S106. Specifically, the spatial positioning information of the control handle can be determined by capturing the light spot image of the infrared sensor on the control handle using a camera device on the mobile terminal, and by combining this image with the IMU data detected by the second inertial sensor on the palm of the hand.

[0081] In one embodiment of the present invention, step S101 (when it is determined that the mobile terminal is installed on the virtual head-mounted display device, controlling the camera device to capture images of the virtual head-mounted display device environment to form a first image) specifically includes the following steps:

[0082] Step S1011: Determine whether the connection plug of the mobile terminal is connected to the connection plug of the virtual head-mounted display device;

[0083] Specifically, the connection plugs for mobile terminals and virtual head-mounted displays can be either communication connectors or mechanical connectors.

[0084] When the judgment result in step S1011 is yes, that is, when the connection plug of the mobile terminal is connected to the connection plug of the virtual head-mounted display device, it means that the mobile terminal is executing the virtual head-mounted display mode, and then step S1012 is executed.

[0085] Step S1012: Control the camera device to capture images of the environment where the virtual head-mounted display device is located, forming a first image.

[0086] It should be noted that steps S1011-S1012 refer to the process of determining whether the mobile terminal is connected to the virtual head-up display device. When it is determined that the mobile terminal is connected to the virtual head-up display device, it indicates that the mobile terminal is executing virtual mode. As a second aspect of the invention, the invention also provides a spatial positioning controller for a separate virtual system. Figure 7A working principle diagram of a separated virtual system provided by an embodiment of the application is shown in the figure. Figure 7 As shown in the figure, the separated virtual system comprises:

[0087] A mobile terminal 2;

[0088] A first inertial sensor 20 and a camera 30 are arranged on the mobile terminal, and the first inertial sensor 20 is used to detect IMU data of the mobile terminal.

[0089] A virtual head-mounted device, and the smart terminal 2 is detachably installed on the virtual head-mounted device; and

[0090] A spatial positioning controller 10 of the separated virtual system, and the spatial positioning controller 10 is in communication connection with the mobile terminal 2, the first inertial sensor 20 and the camera 30 respectively.

[0091] The spatial positioning controller 10 comprises:

[0092] A function switching unit 100, configured to switch a function of the mobile terminal to a virtual head-mounted function when it is determined that the mobile terminal is installed on the virtual head-mounted device.

[0093] A camera control unit 200, in communication connection with the function switching unit 100, and when the function switching unit 100 determines that the function of the mobile terminal is the virtual head-mounted function, the function switching unit 100 transmits a switching success signal to the camera control unit 200, at this time, the mobile terminal, the virtual head-mounted device and the control handle jointly constitute a virtual system, and when a user uses the virtual system, the camera control unit 200 controls the camera 30 to take pictures of an environment of the virtual head-mounted device to form a first image, so as to prepare for spatial positioning information of the mobile terminal.

[0094] A sensor control unit 300, configured to control the first inertial sensor 20 to detect IMU data of the virtual head-mounted device.

[0095] The sensor control unit 300 is in communication connection with the function switching unit 100, and when the function switching unit 100 determines that the function of the mobile terminal is the virtual head-mounted function, the function switching unit 100 transmits the switching success signal to the camera control unit 200, at this time, the mobile terminal, the virtual head-mounted device and the control handle jointly constitute the virtual system, and when the user uses the virtual system, the sensor control unit 300 controls the first inertial sensor 20 to detect the IMU data of the virtual head-mounted device, i.e. to rotate the IMU data.

[0096] A positioning unit 400, configured to calculate and generate spatial positioning information of the virtual head-mounted device according to the IMU data of the virtual head-mounted device and the first image.

[0097] The positioning unit 400 is in communication connection with the function switching unit 100, the camera 30 and the first inertial sensor 20 respectively, the camera 30 is controlled by the camera control unit 200 to take a picture of the surrounding environment of the virtual head-mounted device, to form a first image, and the camera 30 sends the first image to the positioning unit 400; at the same time, the first inertial sensor 20 is controlled by the sensor control unit 300 to detect the IMU data of the virtual head-mounted device, and the IMU data of the virtual head-mounted device is transmitted to the positioning unit 400; after the positioning unit 400 receives the first image and the IMU data of the virtual head-mounted device, the first image and the IMU data are calculated to generate the air landing positioning information of the virtual head-mounted device.

[0098] The spatial positioning controller of the separated virtual system provided by the application can control the camera arranged on the mobile terminal to take a first image of the environment where the virtual head-mounted device is located when the mobile terminal is installed on the virtual head-mounted device and the mobile terminal enters the virtual mode, control the first inertial sensor arranged on the mobile terminal to detect the IMU data of the virtual head-mounted device, and then determine the spatial positioning information of the virtual head-mounted device, thereby increasing the accuracy of the spatial positioning of the mobile terminal. In addition, the spatial positioning information is used for human-computer interaction to improve the game experience, thereby improving the performance of the separated head-mounted reality device of the mobile terminal.

[0099] Optionally, the positioning controller 10 can be arranged on the mobile terminal, when the positioning controller is arranged on the mobile terminal, the positioning controller can not only realize the spatial positioning of the mobile terminal when the virtual head-mounted function is implemented, but also realize the spatial positioning when the mobile terminal implements other functions, for example, when the mobile terminal is a mobile phone, the positioning controller can still position the communication space of the mobile phone when the mobile phone implements the communication function. When the mobile terminal and the game handle jointly constitute a game console, the positioning controller can still position the spatial position of the mobile terminal, so that the user can better experience the game.

[0100] Optionally, the positioning controller 10 can also be arranged on the virtual head-mounted device or the control handle.

[0101] In an embodiment of the application, the working principle diagram of the separated virtual system provided by another embodiment of the application is shown in the figure; as shown in the figure, the positioning unit 400 comprises: Figure 8

[0102] The translation freedom degree determination module 401 is used for determining the translation freedom degree data of the virtual head-mounted device based on the first image;

[0103] ​The translation freedom degree determination module 401 is in communication connection with the camera 30, the camera 30 transmits the first image captured to the translation freedom degree determination module 401, and the translation freedom degree determination module 401 determines the translation freedom degree data of the virtual head-mounted device according to the first image;

[0104] The six freedom degree determination module 402 is configured to determine the 6DOF data of the virtual reality head-mounted device based on the translation freedom degree data of the virtual head-mounted device and the IMU data of the virtual head-mounted device, wherein the spatial positioning information of the virtual head-mounted device comprises the 6DOF data of the virtual head-mounted device.

[0105] The six freedom degree determination module 402 is in communication connection with the translation freedom degree module 401 and the first inertial sensor 20 respectively, receives the translation freedom degree data transmitted by the translation freedom degree determination module 401 and the IMU data transmitted by the first inertial sensor 20, and calculates the 6DOF data of the virtual head-mounted device according to the IMU data and the translation freedom degree data.

[0106] In an embodiment of the present application, a working principle diagram of a separated virtual system provided by another embodiment of the present application is shown in FIG. 2. Figure 9 As shown in FIG. 2, the translation freedom degree determination module 401 comprises:

[0107] The feature determination module 404 is configured to extract image feature points in the first image, extract reference feature points in the reference image, and match the image feature points and the reference feature points to generate effective feature points.

[0108] The region extraction module 405 is configured to extract key regions from the first image based on the sliding window method according to all effective feature points.

[0109] The map point generation module 406 is configured to generate map points based on the key regions, the effective feature points, and a map of a space where the virtual display device is located.

[0110] The map information determination module 407 is configured to determine map information of the virtual display device based on the map points and the map, and the map information comprises the translation freedom degree data.

[0111] In an embodiment of the present application, Figure 10 As shown in FIG. 2, a working principle diagram of a separated virtual system provided by another embodiment of the present application is shown in FIG. 2. Figure 10 As shown in FIG. 2, the separated virtual system further comprises a control handle 4, and an infrared sensor 50 and a second inertial sensor 40 arranged on the control handle 4, and the camera 30 on the mobile terminal 2 is further configured to capture a light spot image of the infrared sensor. Figure 10As shown, the camera control unit 200 is further configured to control the camera 30 to capture the infrared sensor 50 on the control handle to form a light spot image; the sensor control unit 300 is further configured to control the second inertial sensor 40 to detect the IMU data of the control handle 4; and the positioning unit 400 is further configured to calculate the spatial positioning information of the control handle 4 according to the spatial positioning information of the virtual head-mounted device, the IMU data of the control handle 4, and the light spot image. That is, the camera 30 can be controlled to capture the infrared sensor, and the second inertial sensor 40 can be controlled to detect the IMU data on the control handle 4, so as to determine the spatial positioning information of the control handle.

[0112] Hereinafter, an electronic device according to an embodiment of the present application will be described with reference to the accompanying drawings. Figure 11 Figure 11 As shown, the electronic device 900 includes one or more processors 901 and a memory 902.

[0113] As shown, the electronic device 900 includes one or more processors 901 and a memory 902. Figure 11

[0114] The processor 901 can be a central processing unit (CPU) or other form of processing unit that has data processing and / or information executing capabilities, and can control other components in the electronic device 900 to perform desired functions.

[0115] The memory 901 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program information can be stored on the computer-readable storage medium, and the processor 901 can run the program information to implement the spatial positioning method of the separate virtual system according to various embodiments of the present application described above or other desired functions.

[0116] In one example, the electronic device 900 can further include an input device 903 and an output device 904, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0117] The input device 903 can include, for example, a keyboard, a mouse, and / or the like.

[0118] The output device 904 can output various information to the outside. The output device 904 can include, for example, a display, a communication network and a remote output device connected thereto, and / or the like.

[0119] ​​Of course, in order to simplify, Figure 11 Only some of the components of the electronic device 900 related to the present application are shown in the figure, and components such as buses, input / output interfaces, and the like are omitted. In addition to this, the electronic device 900 can include any other appropriate components according to the specific application.

[0120] In addition to the above-mentioned method and device, an embodiment of the present application can also be a computer program product, which includes computer program information, which, when executed by a processor, causes the processor to perform the steps in the spatial positioning method of the separated virtual system according to various embodiments of the present application described in the specification.

[0121] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language, such as Java, C++, and the like, and a conventional procedural programming language, such as the "C" language or the like. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0122] In addition, an embodiment of the present application can also be a computer readable storage medium, which stores computer program information, which, when executed by a processor, causes the processor to perform the steps in the spatial positioning method of the separated virtual system according to various embodiments of the present application described in the specification.

[0123] The computer readable storage medium can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include a communication connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above.

[0124] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the above disclosed specific details are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to be necessarily implemented with the above specific details.

[0125] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0126] It should also be noted that in the devices, apparatuses and methods of the present application, each component or step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application.

[0127] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the present application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0128] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and the like made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for spatial localization of a split virtual system, characterized in that, The separated virtual system comprises a virtual head-mounted device, a mobile terminal detachably mounted on the virtual head-mounted device, and a control handle in communication connection with the mobile terminal, wherein the mobile terminal is provided with a first inertial sensor, the mobile terminal and / or the virtual head-mounted device is provided with a camera, and the first inertial sensor is used to detect IMU data of the mobile terminal; The spatial positioning method of the separated virtual system comprises: When it is determined that the mobile terminal is mounted on the virtual head-mounted device, the camera is controlled to capture an environment where the virtual head-mounted device is located to form a first image; The first inertial sensor is controlled to detect IMU data of the virtual head-mounted device; Image feature points are extracted from the first image, and reference feature points are extracted from a reference image; The image feature points and the reference feature points are matched to generate effective feature points; It is determined whether the first image is initialized; When it is determined that the first image is initialized, a sliding window state of the sliding window in the first image is determined; The positioning state of the first image is determined according to the sliding window state; The sliding window is drawn from the first image according to the positioning state; An image region through which the sliding window is drawn is detected, and the sliding window in which effective feature points are detected is saved as a key region; Map points are generated based on the key region, the effective feature points, and a map of a space where the virtual display device is located; Map information of the virtual display device is determined based on the map points and the map, and the map information comprises translational degree-of-freedom data; 6DOF data of the virtual head-mounted device is determined based on the translational degree-of-freedom data of the virtual head-mounted device and the IMU data of the virtual head-mounted device; wherein the spatial positioning information of the virtual head-mounted device comprises the 6DOF data of the virtual head-mounted device.

2. The method of spatial localization of a split virtual system of claim 1, wherein, Further comprising: When it is determined that the first image is not initialized, the first image is repositioned according to the map to determine a positioning state of the first image; The key region is extracted from the first image based on the sliding window method according to all effective feature points, comprising: The key region is extracted from the first image based on the sliding window method according to all effective feature points based on the positioning state of the first image.

3. The method of claim 1, wherein, An infrared sensor and a second inertial sensor are mounted on the control handle, and the camera on the mobile terminal is further used to capture a light spot image of the infrared sensor; The spatial positioning method of the separated virtual system further comprises: When it is determined that the mobile terminal is mounted on the virtual head-mounted device, the camera is controlled to capture the infrared sensor on the control handle to form a light spot image; The second inertial sensor is controlled to detect IMU data of the control handle; The spatial positioning information of the control handle is generated by calculation according to the spatial positioning information of the virtual head-mounted device, the IMU data of the control handle, and the light spot image.

4. The method of spatial localization of a split virtual system of claim 1, wherein, controlling the camera to capture an environment in which the virtual head-mounted device is located to form a first image, comprising: controlling the camera to capture an environment in which the virtual head-mounted device is located to form a first image when the connection plug of the mobile terminal communicates with the connection plug of the virtual head-mounted device.

5. A split virtual system, characterized by comprising: a mobile terminal; a first inertial sensor disposed on the mobile terminal, the first inertial sensor being configured to detect IMU data of the mobile terminal; a virtual head-mounted device, the mobile terminal being detachably mounted on the virtual head-mounted device; a camera disposed on the mobile terminal and / or the virtual head-mounted device; a spatial positioning controller of a split virtual system, the spatial positioning controller being communicatively connected to the mobile terminal, the first inertial sensor, and the camera; wherein the spatial positioning controller comprises: a function switching unit configured to switch a function of the mobile terminal to a virtual head-mounted function when it is determined that the mobile terminal is mounted on the virtual head-mounted device; a camera control unit configured to control the camera to capture an environment in which the virtual head-mounted device is located to form a first image; a sensor control unit configured to control the first inertial sensor to detect IMU data of the virtual head-mounted device; and a positioning unit configured to calculate spatial positioning information of the virtual head-mounted device based on the IMU data of the virtual head-mounted device and the first image; calculating spatial positioning information of the virtual head-mounted device based on the IMU data of the virtual head-mounted device and the first image, comprising: determining translational degree of freedom data of the virtual head-mounted device based on the first image; and determining 6DOF data of the virtual head-mounted device based on the translational degree of freedom data of the virtual head-mounted device and the IMU data of the virtual head-mounted device; wherein the spatial positioning information of the virtual head-mounted device comprises the 6DOF data of the virtual head-mounted device; determining translational degree of freedom data of the virtual head-mounted device based on the first image, comprising: extracting image feature points in the first image and reference feature points in a reference image; matching the image feature points and the reference feature points to generate effective feature points; extracting a key region from the first image based on a sliding window method according to all effective feature points; generating map points based on the key region, the effective feature points, and a map of a space in which the virtual display device is located; and determining map information of the virtual display device based on the map points and the map, the map information comprising the translational degree of freedom data; before extracting a key region from the first image based on a sliding window method; the determining of the translational degree of freedom data of the virtual head-mounted device based on the first image further comprises: determining whether the first image is initialized; when it is determined that the first image is initialized, extracting a key region from the first image based on a sliding window method according to all effective feature points; When it is determined that the first image is initialized, a key region is extracted from the first image based on a sliding window method according to all valid feature points, including: determining a sliding window state of the sliding window in the first image; determining a positioning state of the first image according to the sliding window state; sliding the sliding window through the first image according to the positioning state; and detecting the image region through which the sliding window slides, and saving the sliding window in which valid feature points are detected as the key region.

6. The split virtual system of claim 5, wherein, The positioning unit includes: a translation degree of freedom determination module configured to determine translation degree of freedom data of the virtual head-mounted device based on the first image; and a six degree of freedom determination module configured to determine 6DOF data of the virtual head-mounted device based on the translation degree of freedom data of the virtual head-mounted device and the IMU data of the virtual head-mounted device; wherein the spatial positioning information of the virtual head-mounted device includes the 6DOF data of the virtual head-mounted device.

7. The split virtual system of claim 6, wherein, The translation degree of freedom determination module includes: a feature determination module configured to extract image feature points in the first image, extract reference feature points in a reference image, and match the image feature points and the reference feature points to generate valid feature points; a region extraction module configured to extract a key region from the first image based on a sliding window method according to all valid feature points; a map point generation module configured to generate map points based on the key region, the valid feature points, and a map of a space in which the virtual display device is located; and a map information determination module configured to determine map information of the virtual display device based on the map points and the map, the map information including the translation degree of freedom data.

8. The split virtual system of claim 5, wherein, Further comprising: a control handle; and an infrared sensor and a second inertial sensor disposed on the control handle, the second inertial sensor being configured to measure IMU data of the control handle; the camera on the mobile terminal is further configured to capture a light spot image of the infrared sensor; wherein the camera control unit is further configured to control the camera to capture the infrared sensor on the control handle to form a light spot image; the sensor control unit is further configured to control the second inertial sensor to detect the IMU data of the control handle; the positioning unit is further configured to calculate and generate spatial positioning information of the control handle according to the spatial positioning information of the virtual head-mounted device, the IMU data of the control handle, and the light spot image.

Images