Method for spatially determining the location of a separated virtual system, Virtual system

Abstract

To solve the technical problem that a virtual system cannot be separately installed and cannot be accurately located in the prior art.SOLUTION: A spatial position identification method for a split-type virtual system according to an embodiment comprises: when a mobile terminal is mounted on a virtual head-mounted display device and the mobile terminal enters a virtual mode, determining a translational degree of freedom from a first image captured by an imaging device installed in the mobile terminal; acquiring a degree of freedom of rotation by a first inertial sensor disposed on the mobile terminal; and then determining spatial position identification information of the virtual head-mounted display device to improve the accuracy of spatial position identification of the mobile terminal. In addition, man-machine interaction is performed based on the spatial position identification information to improve game experience, thereby improving performance of the split-type head-mounted display device of the mobile terminal.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the field of virtual technology, and in particular to a method for spatial localization of a separated virtual system, a virtual system. [Background technology]

[0002] At present, a large number of new virtual reality (VR) head-mounted display devices consisting of a VR box and a mobile terminal have appeared on the market. The performance of such new VR head-mounted display devices is mainly determined by the performance of the mobile terminal, and its performance is much lower than that of conventional computer-VR head-mounted display devices or VR integrated devices. The display of the computer-VR head-mounted display device or VR integrated device and the VR virtual head-mounted display device are generally inseparable and can only be used in combination, which limits the convenience.

[0003] However, the mobile terminal needs to be detached and then the control handle and the head-mounted display device must be accurately located again, otherwise smooth man-machine interaction cannot be achieved and the game experience will be further affected. And there is still no method for accurately locating the detachable VR head-mounted display device of the mobile terminal. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of the above, the present invention provides a method for spatially locating a separate virtual system and a virtual system, thereby solving the technical problem in the prior art that a virtual system cannot be installed separately and accurately located. [Means for solving the problem]

[0005] As a first aspect of the present invention, the present invention provides a method for spatial localization of a separate virtual system, the separate virtual system comprising a virtual head mounted display device, a mobile terminal detachably attached to the virtual head mounted display device, and a control handle communicatively connected to the mobile terminal, the mobile terminal being provided with a first inertial sensor, the mobile terminal and / or the virtual head mounted display device being provided with an imaging device, the first inertial sensor being for detecting IMU data of the mobile terminal, the method for spatial localization of the separate virtual system comprising: when determining that the mobile terminal is attached to the virtual head mounted display device, controlling the imaging device to capture an environment in which the virtual head mounted display device is located to form a first image; controlling the first inertial sensor to detect IMU data of the virtual head mounted display device; and performing calculations based on the IMU data of the virtual head mounted display device and the first image to generate spatial localization information of the virtual head mounted display device.

[0006] In one embodiment of the invention, performing calculations based on IMU data of the virtual head mounted display device and the first image to generate spatial localization information of the virtual head mounted display device includes determining translational degree of freedom data of the virtual head mounted display device based on the first image, and determining 6DOF data of the virtual reality head mounted display device based on the translational degree of freedom data of the virtual head mounted display device and the IMU data of the virtual head mounted display device, wherein the spatial localization information of the virtual head mounted display device includes the 6DOF data of the virtual head mounted display device.

[0007] In one embodiment of the present invention, determining translational degree of freedom data of the virtual head mounted display device based on the first image includes extracting image feature points from the first image and extracting reference feature points from a reference image, matching the image feature points and the reference feature points to generate effective feature points, extracting key regions from the first image based on the effective feature points based on a sliding window method, generating map points based on the key regions, the effective feature points and a map of a space in which the virtual display device is located, and determining map information of the virtual display device including the translational degree of freedom data based on the map points and the map.

[0008] In one embodiment of the present invention, determining translational degree of freedom data of the virtual head mounted display device based on the first image before extracting a key region from the first image based on a sliding window method further includes determining whether the first image is initialized, and if it is determined that the first image is initialized, extracting a key region from the first image based on a sliding window method based on the valid feature points.

[0009] In one embodiment of the present invention, determining the translational degree of freedom data of the virtual head mounted display device based on the first image further includes, if it is determined that the first image is not initialized, re-locating the first image based on the map to determine a localization state of the first image, and extracting key regions from the first image based on the effective feature points based on a sliding window method includes extracting key regions from the first image based on the effective feature points based on a sliding window method in accordance with the localization state of the first image.

[0010] In one embodiment of the present invention, when it is determined that the first image is initialized, extracting a key region from the first image based on the valid feature points and based on a sliding window method includes: determining a sliding window state of a sliding window in the first image; determining a localization state of the first image according to the sliding window state; swiping the sliding window from the first image according to the localization state; detecting an image region where the sliding window is swiped, and saving the sliding window in which the valid feature points are detected as the key region.

[0011] In one embodiment of the present invention, an infrared sensor and a second inertial sensor are attached to a control handle, and the photographing device on the mobile terminal is further used to photograph a spot image of the infrared sensor, and the spatial localization method of the separated virtual system further includes: when it is determined that the mobile terminal is attached to the virtual head-mounted display device, controlling the photographing device to photograph the infrared sensor on the control handle to form a spot image; controlling the second inertial sensor to detect IMU data of the control handle; and performing calculations according to the spatial localization information of the virtual head-mounted display device, the IMU data of the control handle and the spot image to generate the spatial localization information of the control handle.

[0012] In one embodiment of the present invention, when it is determined that the mobile terminal is attached to the virtual head mounted display device, controlling the photographing device to photograph an environment in which the virtual head mounted display device is located to form a first image includes, when a connection plug-in of the mobile terminal contacts a connection plug-in of the virtual head mounted display device, controlling the photographing device to photograph an environment in which the virtual head mounted display device is located to form a first image.

[0013] As a second aspect of the present invention, the present invention provides a separated virtual system, comprising: a mobile terminal; a first inertial sensor for detecting IMU data of the mobile terminal, the first inertial sensor being installed on the mobile terminal; a virtual head mounted display device to which the smart terminal is detachably attached; a photographing device being installed on the mobile terminal and / or the virtual head mounted display device; and a spatial localization controller of the separated virtual system communicatively connected to the mobile terminal, the first inertial sensor and the photographing device, respectively, wherein the spatial localization controller comprises: a function switching unit for switching a function of the mobile terminal to a virtual head mounted display function when it is determined that the mobile terminal is attached to the virtual head mounted display device; an photographing control unit for controlling the photographing device to photograph an environment in which the virtual head mounted display device is located to form a first image; a sensor control unit for controlling the first inertial sensor to detect IMU data of the virtual head mounted display device; and a localization unit for performing calculations based on the IMU data of the virtual head mounted display device and the first image to generate spatial localization information of the virtual head mounted display device.

[0014] In one embodiment of the present invention, the localization unit comprises a translational degree of freedom determination module for determining translational degree of freedom data of the virtual head mounted display device based on the first image, and a 6 degree of freedom determination module for determining 6DOF data of the virtual reality head mounted display device based on the translational degree of freedom data of the virtual head mounted display device and the IMU data of the virtual head mounted display device, and the spatial localization information of the virtual head mounted display device includes the 6DOF data of the virtual head mounted display device.

[0015] In one embodiment of the present invention, the translational degree of freedom determination module includes a feature determination module for extracting image feature points from the first image, extracting reference feature points from a reference image, and matching the image feature points with the reference feature points to generate effective feature points; a region extraction module for extracting a key region from the first image based on the effective feature points based on a sliding window method; a map point generation module for generating map points based on the key region, the effective feature points and a map of a space in which the virtual display device is located; and a map information determination module for determining map information of the virtual display device including the translational degree of freedom data based on the map points and the map.

[0016] In one embodiment of the present invention, the separated virtual system further includes a control handle, an infrared sensor and a second inertial sensor installed on the control handle, where the second inertial sensor is used to measure IMU data of the control handle, and the photographing device on the mobile terminal is further used to photograph a spot image of the infrared sensor, the photographing control unit is further used to control the photographing device to photograph the infrared sensor on the control handle to form a spot image, the sensor control unit is further used to control the second inertial sensor to detect the IMU data of the control handle, and the positioning unit is further used to perform calculations based on the spatial positioning information of the virtual head-mounted display device, the IMU data of the control handle and the spot image, to generate spatial positioning information of the control handle. Effect of the Invention

[0017] In the spatial localization method of the separated virtual system according to the present invention, when a mobile terminal is attached to a virtual head-mounted display device and the mobile terminal enters a virtual mode, the method determines a translational degree of freedom from a first image captured by a photographing device installed on the mobile terminal, obtains a rotational degree of freedom by a first inertial sensor installed on the mobile terminal, and further determines spatial localization information of the virtual head-mounted display device, thereby improving the accuracy of spatial localization of the mobile terminal, and performs man-machine interaction based on the spatial localization information to improve the game experience, thereby improving the performance of the mobile terminal separated head-mounted display device. [Brief description of the drawings]

[0018] The above and other objects, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail with reference to the drawings. The drawings are intended to provide a further understanding of the embodiments of the present invention, and are a part of the specification for interpreting the present invention together with the embodiments of the present invention, and are not intended to limit the present invention. In the drawings, the same reference numerals generally represent the same elements or steps. [Figure 1] 4 is a flowchart of a method for spatial localization of a separated virtual system according to one embodiment of the present invention. [Diagram 2] 4 is a flowchart of a method for spatial localization of a separated virtual system according to another embodiment of the present invention. [Diagram 3] 4 is a flowchart of a method for spatial localization of a separated virtual system according to another embodiment of the present invention. [Figure 4] 4 is a flowchart of a method for spatial localization of a separated virtual system according to another embodiment of the present invention. [Diagram 5] 4 is a flowchart of a method for spatial localization of a separated virtual system according to another embodiment of the present invention. [Figure 6] 4 is a flowchart of a method for spatial localization of a separated virtual system according to another embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating the operation principle of a separated virtual system according to an embodiment of the present invention. [Figure 8] FIG. 13 is a diagram illustrating the operation principle of a separated virtual system according to another embodiment of the present invention. [Figure 9] FIG. 13 is a diagram illustrating the operation principle of a separated virtual system according to another embodiment of the present invention. [Figure 10] FIG. 13 is a diagram illustrating the operation principle of a separated virtual system according to another embodiment of the present invention. [Figure 11] FIG. 2 is a diagram illustrating the operation principle of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In the description of the present invention, unless otherwise clearly and specifically limited, the meaning of "plurality" is at least two, for example, two, three, etc. All directional indications (e.g., up, down, left, right, front, back, top, bottom, etc.) in the embodiments of the present invention are intended to interpret only the relative positional relationship, movement status, etc. of each member in a certain position (shown in the drawings), and when the specific position changes, the directional indications also change correspondingly. In addition, the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or equipment including a series of steps or units is not limited to the recited steps or units, but may further include steps or units not recited, or may further include other steps or units specific to these processes, methods, products, or equipment.

[0020] In addition, the term "embodiment" as referred to in this specification means that a particular feature, structure, or characteristic described with reference to the embodiment may be included in at least one embodiment of the present invention. The appearance of the term in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is exclusive of other embodiments. As will be understood by those skilled in the art, both explicitly and implicitly, the embodiment described in this specification may be combined with other embodiments.

[0021] Hereinafter, the technical solutions of the embodiments of the present invention will be described clearly and completely with reference to the drawings of the embodiments of the present invention, and obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, any other embodiments obtained by a person skilled in the art without inventive efforts are all within the protection scope of the present invention.

[0022] 1 is a flow chart of a spatial position determination method of a separate virtual system according to the present invention, the separate system includes a virtual head mounted display device, a mobile terminal detachably mounted on the virtual head mounted display device, and a control handle communicatively connected to the mobile terminal, the mobile terminal is provided with a first inertial sensor, the mobile terminal and / or the virtual head mounted display device are provided with a photographing device, and the first inertial sensor is for detecting the IMU data of the mobile terminal. Specifically, the number and mounting positions of the photographing devices may be in three situations: (1) the number of photographing devices is equal to 1, the photographing devices may be mounted on the mobile terminal and the virtual head mounted display device; (2) the number of photographing devices is greater than 1, both the photographing devices may be mounted on the mobile terminal and both the photographing devices may be mounted on the virtual head mounted display device; and (3) the number of photographing devices is greater than 1, some of the photographing devices are mounted on the mobile terminal, and the remaining photographing devices are mounted on the virtual head mounted display device.

[0023] Specifically, an inertial sensor (abbreviated as Inertial Measurement Unit, IMU) is a device that measures the three-axis attitude angle (or angular velocity) and acceleration of an object. Generally, one IMU has three single-axis accelerometers and three single-axis gyros, and the accelerometers detect independent three-axis acceleration signals in the medium coordinate system of the object, while the gyros detect angular velocity signals relative to the navigation coordinate system of the medium. Thus, the inertial sensor measures the angular velocity and acceleration of the object in three-dimensional space, and calculates the attitude of the object based on this, for example, the rotational degrees of freedom of the object, where the rotational degrees of freedom refer to the degrees of freedom related to the three positions of up, down, front, back, left, and right. IMU data is the result data of the inertial sensor detecting an object, that is, the angular velocity and acceleration data of an object in three-dimensional space detected by the inertial sensor. Thus, a first inertial sensor installed in a mobile terminal can detect the IMU data of the mobile terminal, and the attitude of the mobile terminal, for example, the rotational degrees of freedom of the mobile terminal can be calculated based on the IMU data of the mobile terminal, where the rotational degrees of freedom refer to the degrees of freedom related to the three positions of up, down, front, back, left, and right.

[0024] As shown in FIG. 1, the spatial localization method for a separated virtual system includes the following steps. S101, when it is determined that the mobile terminal is mounted on the virtual head-mounted display device, control the photographing device to photograph the environment of the virtual head-mounted display device to form a first image; Since the virtual head-mounted display device of the mobile terminal is detachably connected, the mobile terminal not only has the virtual head-mounted display function, but also has other functions, for example, when the mobile terminal is a smartphone, when the mobile terminal is not attached to the virtual head-mounted display device, the mobile terminal can be used as a smartphone and has the functions that a smartphone should have, such as a communication function, a display function, etc. When the mobile terminal is attached to the virtual head-mounted display device, the mobile terminal performs the virtual head-mounted display function and forms a virtual system together with the virtual head-mounted display device and the control handle, for example, when the virtual display device is a VR helmet, i.e., when the mobile terminal is attached to the VR helmet, the mobile terminal forms a VR system together with the VR helmet and the control handle.

[0025] Specifically, when the mobile terminal is removably attached to the virtual head mounted display device, a specific removable attachment method may be to reserve an attachment groove on the virtual head mounted display device and fix the mobile terminal to the virtual head mounted display device by attaching the mobile terminal to the attachment groove reserved on the virtual head mounted display device.

[0026] Specifically, the communication connection manner between the mobile terminal and the control handle may be such that the mobile terminal has connection ports at both ends, and the mobile terminal is connected to the control handle by a connection line connected to the connection ports. It is understood that the mobile terminal and the control handle may also be connected by a wireless manner such as Bluetooth. After the mobile terminal and the control handle are successfully connected, it is confirmed that the virtual mode is entered. In the virtual mode, the mobile terminal has the functions of game interface display, image processing, etc.

[0027] When it is determined that the mobile terminal is mounted on the virtual head-mounted display device, i.e. when the mobile terminal executes the virtual head-mounted display function, the mobile terminal and the control handle also realize a communication connection, at this time, the mobile terminal executes the virtual head-mounted display function, i.e. the mobile terminal, together with the control handle and the virtual head-mounted display device, constitute a virtual system, and during the operation process of the virtual system, both the mobile terminal and the control handle need to perform spatial localization, at this time, the photographing device located on the mobile terminal is controlled to photograph the environment of the virtual head-mounted display device, the photographing device starts to operate according to the control, photographs the environment where the virtual head-mounted display device is located, and forms a first image for spatial localization for the mobile terminal.

[0028] S102, controlling a first inertial sensor to detect IMU data of a virtual head mounted display device; After the mobile terminal determines to connect to the virtual head-mounted display device, the mobile terminal forms a virtual system together with the virtual head-mounted display device and the control handle, and the user operates the virtual system to experience a virtual scene, and at this time, the first inertial sensor is controlled to start an action, and the first inertial sensor is controlled to detect the IMU data of the virtual head-mounted display device; A first inertial sensor installed in the mobile terminal is controlled to detect IMU data of the mobile terminal.

[0029] S103, calculating IMU data and a first image of the virtual head mounted display device to generate spatial localization information of the virtual head mounted display device.

[0030] After obtaining the first image transmitted by the photographing device and the IMU data transmitted by the first inertial sensor, spatial position identification information of the mobile terminal, i.e., six degrees of freedom data (hereinafter abbreviated as 6DOF data) of the mobile terminal can be determined based on the IMU data and the first image, that is, six angular degrees of freedom can be obtained based on the translational degrees of freedom and the rotational degrees of freedom. In the spatial localization method of the separated virtual system according to the present invention, when a mobile terminal is attached to a virtual head-mounted display device and the mobile terminal enters a virtual mode, a first image captured by a photographing device installed on the mobile terminal and a rotational degree of freedom are obtained by a first inertial sensor installed on the mobile terminal, and spatial localization information of the virtual head-mounted display device is determined to improve the accuracy of spatial localization of the mobile terminal, and man-machine interaction is performed based on the spatial localization information to improve the game experience, thereby improving the performance of the mobile terminal separated head-mounted display device.

[0031] In one embodiment of the present invention, FIG. 2 is a flowchart of a spatial localization method for a separated virtual system according to another embodiment of the present invention. As shown in FIG. 2, step S103 (calculating IMU data and a first image of the virtual head-mounted display device to generate spatial localization information of the virtual head-mounted display device) specifically includes the following steps:

[0032] Step S1031, determining translational degree of freedom data for a virtual head mounted display device based on a first image; The translational degrees of freedom refer to left / right, up / down, and front / back movement data in the space of the virtual head mounted display device.

[0033] Step S1032: determine 6DOF data of the virtual reality head mounted display device based on the translational degree of freedom data of the virtual head mounted display device and the IMU data of the virtual head mounted display device, and the spatial localization information of the virtual head mounted display device includes the 6DOF data of the virtual head mounted display device.

[0034] The IMU data of the virtual head mounted display device refers to the rotational IMU data of the virtual head mounted display device, and the rotational IMU data refers to the three rotational degrees of freedom data of the virtual head mounted display device. According to the translational IMU data and the rotational IMU data of the virtual head mounted display device, the six degrees of freedom data, i.e., 6DOF data, of the virtual head mounted display device can be determined, and according to the 6DOF data of the virtual head mounted display device, the precise positioning of the virtual head mounted display device in space can be accurately tracked, and the movement of the virtual head mounted display device in the real world can be accurately mapped in the virtual scene.

[0035] In one embodiment of the present invention, FIG. 3 is a flowchart of a spatial localization method for a separated virtual system according to another embodiment of the present invention. As shown in FIG. 3, step S1031 (determining translational degree of freedom data of the virtual head-mounted display device based on the first image) specifically includes the following steps: Step S1: extracting image feature points from a first image and extracting reference feature points from a reference image; The reference image refers to an image of a portion of a map of the spatial environment in which the virtual head mounted display device is located; for example, if the spatial environment in which the virtual head mounted display device is located is a living room, the map is a living room map and the reference image is an image of a portion of the spatial map, for example an image of a corner of the living room.

[0036] Step S2: Match the image feature points with the reference feature points to generate valid feature points; Image feature points are matched with reference feature points, and image feature points that successfully match are regarded as valid feature points.

[0037] Step S3: extracting a key region from the first image based on the effective feature points using a sliding window method; The sliding window method can prune the search space by swiping from the leftmost part of the sliding window, thereby reducing the repetitive calculations, lowering the time complexity, and avoiding brute-force searches.

[0038] Step S4: Generate map points based on the key regions, the effective feature points and the map of the space in which the virtual display device is located.

[0039] Step S5, determining map information for the virtual display device including translational degree of freedom data based on the map points and the map.

[0040] The present invention determines effective feature points in a first image based on a spatial map and a reference image, and then extracts key regions in the first image based on the effective feature points based on a sliding window method, which not only can improve the accuracy of translational degree of freedom data, but also further reduce repeated calculations, reduce time complexity, and improve efficiency.

[0041] In one embodiment of the present invention, Figure 4 is a flowchart of a spatial localization method for a separated virtual system according to another embodiment of the present invention. As shown in Figure 4, before step S3 (extracting key regions from the first image based on the sliding window method based on the effective feature points), step S1031 (determining translational degree of freedom data of the virtual head-mounted display device based on the first image) further includes the following steps:

[0042] Step S10: determining whether the first image is initialized; If the judgment result in step S10 is YES, that is, the first image is initialized, execute step S3, that is, extract a key region from the first image based on the valid feature points and the sliding window method, and the first image is the initialized image.

[0043] The first image being initialized means that if the brightness, grayscale, and resolution of the first image satisfy predetermined requirements, the first image is determined to be the image to be initialized.

[0044] If the determination result in step S10 is NO, that is, if the first image is not initialized, step S6 is executed.

[0045] Step S6: relocating the first image based on the map to determine a location state of the first image; At this time, step S3 (extracting a key region from the first image based on the effective feature points and the sliding window method) specifically includes S31 of extracting a key region from the first image based on the effective feature points and the sliding window method according to the position identification state of the first image.

[0046] In one embodiment of the present invention, FIG. 5 is a flowchart of a spatial localization method for a separated virtual system according to another embodiment of the present invention. As shown in FIG. 5, step S3 (extracting key regions from a first image based on the effective feature points and based on a sliding window method) specifically includes: a step S32 of determining a sliding window state of the sliding window in the first image; a step S33 of determining a localization state of the first image according to a sliding window state; a step S34 of swiping the sliding window away from the first image in response to the location state; and step S35 of detecting an image region where the sliding window is swiped, and storing the sliding window in which the valid feature points are detected as a key region.

[0047] In one embodiment of the present invention, an infrared sensor and a second inertial sensor are installed on the control handle, and the photographing device on the mobile terminal is further used to photograph the spot image of the infrared sensor. Figure 6 is a flowchart of a spatial localization method for a separated virtual system according to another embodiment of the present invention. As shown in Figure 6, the spatial localization method for the separated virtual system further includes the following:

[0048] Step S104, when it is determined that the mobile terminal is mounted on the virtual head-mounted display device, control the photographing device to photograph the infrared sensor on the control handle to form a spot image; When it is determined that the mobile terminal is attached to the virtual head-mounted display device, the control handle and the mobile terminal establish a communication connection, and at this time, the photographing device installed on the mobile terminal is controlled to photograph the infrared sensor on the control handle, thereby forming a spot image of the infrared sensor.

[0049] Step S105, control the second inertial sensor to detect the IMU data of the control handle; When it is determined that the mobile terminal is attached to the virtual head-mounted display device, the control handle and the mobile terminal establish a communication connection, and at this time, the second inertial sensor is controlled to start operating and detect the IMU data of the control handle.

[0050] Step S106, perform calculations based on the spatial localization information of the virtual head-mounted display device, the IMU data of the control handle and the spot image to generate spatial localization information of the control handle.

[0051] The spatial position of the control handle can be identified by steps S104 to S106. That is, the image of the spot of the infrared sensor on the control handle is captured by the photographing device on the mobile terminal, and the spatial position identification information of the control handle can be determined by the IMU data of the control handle detected by the second inertial sensor.

[0052] In one embodiment of the present invention, step S101 (when it is determined that the mobile terminal is attached to the virtual head-mounted display device, controlling the photographing device to photograph the environment of the virtual head-mounted display device to form a first image) specifically includes the following steps:

[0053] Step S1011, determining whether a connection plug-in of the mobile terminal and a connection plug-in of the virtual head mounted display device are connected; Specifically, the connection plug-in of the mobile terminal and the connection plug-in of the virtual head-mounted display device may be a communication connection member or a mechanical connection member.

[0054] If the determination result in step S1011 is YES, that is, if the connection plug-in of the mobile terminal connects to the connection plug-in of the virtual head mounted display device, it is described that the mobile terminal executes the virtual head mounted display mode, and at this time, executes step S1012.

[0055] Step S1012: Control the photographing device to photograph the environment in which the virtual head-mounted display device is located, and form a first image.

[0056] As described, steps S1011 to S1012 are the process of determining whether the mobile terminal is in contact with the virtual head mounted display device. If it is determined that the mobile terminal is in contact with the virtual head mounted display device, it is described that the mobile terminal executes a virtual mode. As a second aspect of the present invention, the present invention further provides a spatial localization controller of a separate virtual system, and FIG. 7 is an operation principle diagram of a separate virtual system according to an embodiment of the present invention. As shown in FIG. 7, the separate virtual system: A mobile terminal 2; A first inertial sensor 20 and an imaging device 30 are installed in a mobile terminal, and the first inertial sensor 20 is used to detect IMU data of the mobile terminal; a virtual head mounted display device to which the smart terminal 2 is removably attached; a spatial localization controller 10 of a separate virtual system communicatively connected to a mobile terminal 2, a first inertial sensor 20 and an imaging device 30, The spatial position determination controller 10 includes a function switching unit 100, an imaging control unit 200, a sensor control unit 300, and a position determination unit 400. the function switching unit 100 is for switching the function of the mobile terminal to a virtual head-mounted display function when determining that the mobile terminal is mounted on the virtual head-mounted display device; The photographing control unit 200 is communicatively connected to the function switching unit 100. When the function switching unit 100 determines the function of the mobile terminal as a virtual head-mounted display function, the function switching unit 100 transmits a switching success signal to the photographing control unit 200. At this time, the mobile terminal forms a virtual system together with the virtual head-mounted display device and the control handle. When a user uses the virtual system, the photographing control unit 200 controls the photographing device 30 to photograph the environment of the virtual head-mounted display device to form a first image, thereby preparing for spatial location identification information of the mobile terminal.

[0057] The sensor control unit 300 is for controlling the first inertial sensor 20 to detect IMU data of the virtual head mounted display device; The sensor control unit 300 is communicatively connected to the function switching unit 100. When the function switching unit 100 determines the function of the mobile terminal as a virtual head-mounted display function, the function switching unit 100 transmits a switching success signal to the shooting control unit 200. At this time, the mobile terminal forms a virtual system together with the virtual head-mounted display device and the control handle. When the user uses the virtual system, the sensor control unit 300 controls the first inertial sensor 20 to detect the IMU data, i.e., the rotational IMU data, of the virtual head-mounted display device.

[0058] The localization unit 400 is for performing calculations based on the IMU data and the first image of the virtual head mounted display device to generate spatial localization information of the virtual head mounted display device.

[0059] The position determination unit 400 is communicatively connected to the function switching unit 100, the photographing device 30 and the first inertial sensor 20, respectively. The photographing device 30 photographs the surrounding environment of the virtual head mounted display device under the control of the photographing control unit 200 to form a first image. The photographing device 30 transmits the first image to the position determination unit 400. Meanwhile, the first inertial sensor 20 detects the IMU data of the virtual head mounted display device under the control of the sensor control unit 300 and transmits the IMU data of the virtual head mounted display device to the position determination unit 400. After the position determination unit 400 receives the first image and the IMU data of the virtual head mounted display device, it calculates the first image and the IMU data to generate spatial position determination information of the virtual head mounted display device.

[0060] In the spatial localization controller of the separated virtual system according to the present invention, when a mobile terminal is attached to the virtual head-mounted display device and the mobile terminal enters a virtual mode, the controller controls the photographing device installed on the mobile terminal to photograph a first image of the environment in which the virtual head-mounted display device is located, controls the first inertial sensor installed on the mobile terminal to detect the IMU data of the virtual head-mounted display device, and then determines spatial localization information of the virtual head-mounted display device, thereby improving the accuracy of spatial localization of the mobile terminal, and performs man-machine interaction based on the spatial localization information to improve the game experience, thereby improving the performance of the mobile terminal-separated head-mounted display device.

[0061] Alternatively, the positioning controller 10 may be installed in a mobile terminal, and when the positioning controller is installed in a mobile terminal, the positioning controller can not only realize the spatial positioning of the mobile terminal when performing the virtual head-mounted display function, but also realize the spatial positioning of the mobile terminal when performing other functions, for example, when the mobile terminal is a mobile phone, when the mobile phone performs the communication function, the positioning controller can still perform the communication spatial positioning of the mobile phone. When the mobile terminal and the game handle form a game mobile terminal, the positioning controller can still locate the spatial position of the mobile terminal, so that the user can better experience the game.

[0062] Optionally, the location controller 10 may also be mounted on a virtual head mounted display or on a control handle.

[0063] In one embodiment of the present invention, FIG. 8 is a diagram showing the operation principle of a separated virtual system according to another embodiment of the present invention. As shown in FIG. 8, the localization unit 400 includes a translational degree of freedom determination module 401 and a six-degree of freedom determination module 402; a translational degree of freedom determination module 401 for determining translational degree of freedom data of the virtual head mounted display device based on the first image; The translational degree of freedom determination module 401 is communicatively connected to the photographing device 30, and transmits a first image captured by the photographing device 30 to the translational degree of freedom determination module 401, and the translational degree of freedom determination module 401 determines translational degree of freedom data of the virtual head-mounted display device according to the first image; a 6 DOF determination module 402 for determining 6 DOF data of a virtual reality head mounted display device, including 6 DOF data of the virtual head mounted display device based on translational DOF data of the virtual head mounted display device and IMU data of the virtual head mounted display device; The 6-DOF determination module 402 is communicatively connected to the translational DOF module 401 and the first inertial sensor 20, respectively. The 6-DOF determination module 402 receives the translational DOF data transmitted by the translational DOF determination module 401 and the IMU data transmitted by the first inertial sensor 20, and performs calculations based on the IMU data and the translational DOF data to generate 6DOF data of the virtual head-mounted display device.

[0064] In one embodiment of the present invention, FIG. 9 is a diagram showing the operation principle of a separated virtual system according to another embodiment of the present invention. As shown in FIG. 9, the translational degree of freedom determination module 401: a feature determining module 404 for extracting image feature points from the first image, extracting reference feature points from the reference image, and matching the image feature points with the reference feature points to generate valid feature points; A region extraction module 405 for extracting a key region from the first image based on the effective feature points and based on a sliding window method; a map point generation module 406 for generating map points based on the key regions, the valid feature points and a map of the space in which the virtual display device is located; and a map information determination module 407 for determining map information for the virtual display device including translational degree of freedom data based on the map points and the map.

[0065] In one embodiment of the present invention, Fig. 10 is an operating principle diagram of a separate virtual system according to another embodiment of the present invention, as shown in Fig. 10, the separate virtual system further includes a control handle 4, an infrared sensor 50 and a second inertial sensor 40 installed on the control handle 4, the photographing device 30 on the mobile terminal 2 is further used to photograph the spot image of the infrared sensor, as shown in Fig. 10, the photographing control unit 200 is further used to control the photographing device 30 to photograph the infrared sensor 50 on the control handle to form the spot image, the sensor control unit 300 is further used to control the second inertial sensor 40 to detect the IMU data of the control handle 4, and the positioning unit 400 is further used to perform calculations according to the spatial positioning information of the virtual head-mounted display device, the IMU data of the control handle 4 and the spot image to generate the spatial positioning information of the control handle 4. That is, the photographing device 30 can be controlled to photograph the infrared sensor, and the second inertial sensor 40 can be controlled to detect the IMU data on the control handle 4, so that the spatial positioning information of the control handle 4 can be determined.

[0066] An electronic device according to an embodiment of the present invention will now be described with reference to Fig. 11. Fig. 11 is a structural schematic diagram of an electronic device according to an embodiment of the present invention.

[0067] As shown in FIG. 11, the electronic device 900 includes one or more processors 901 and a memory 902 .

[0068] The processor 901 may be a central processing unit (CPU) or other type of processing unit having data processing and / or information execution capabilities, and may control the other components of the electronic device 900 to perform desired functions.

[0069] The memory 902 may include one or more computer program products, which may include various types of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache). The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program information may be stored in the computer-readable storage medium, and the processor 901 may execute the program information to realize the spatial localization method or other desired functions of the separated virtual system of each embodiment of the present invention described above.

[0070] In one example, electronic device 900 may further include input devices 903 and output devices 904, these components being connected together by a bus system and / or other type of connection mechanism (not shown).

[0071] The input device 903 may include, for example, a keyboard, a mouse, and the like.

[0072] The output device 904 can output various information to the outside, and may include, for example, a display, a communication network, and a remote output device connected thereto.

[0073] Of course, for the sake of simplicity, Fig. 11 only shows some of the components related to the present invention in the electronic device 900, and omits components such as buses, input / output interfaces, etc. In addition, the electronic device 900 may further include any other suitable components according to specific application situations.

[0074] In addition to the methods and apparatus described above, an embodiment of the present invention may also be a computer program product including computer program information that, when executed by a processor, causes the processor to perform steps in the methods for spatial localization of a separated virtual system according to various embodiments of the present invention as described herein.

[0075] The computer program product may be written with program code for carrying out operations of embodiments of the present invention in any combination of one or more program design languages, including object oriented program design languages ​​such as Java, C++, etc., and further including conventional procedural program design languages ​​such as "C" or similar program design languages. The program code may run entirely on the user's computing device, partially on the user's device, as a separate software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or a server.

[0076] Additionally, an embodiment of the present invention may further be a computer readable storage medium having computer program information stored thereon, which when executed by a processor causes the processor to perform steps in the methods for spatial localization of a separated virtual system according to various embodiments of the present invention herein.

[0077] The computer readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include a communication connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0078] Although the basic principle of the present invention has been described above with reference to specific embodiments, it should be noted that the advantages, advantages, effects, etc. mentioned in the present invention are illustrative and not limiting, and these advantages, advantages, effects, etc. should not be considered as something that must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are merely illustrative and easy to understand, and are not limiting, and the details should not be construed as limiting the present invention so that it must be realized using the specific details.

[0079] Block diagrams of devices, apparatus, facilities, and systems mentioned in the present invention are merely illustrative examples, and are not intended to require or suggest that they must be connected, laid out, or arranged in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatus, facilities, and systems can be connected, laid out, or arranged in any manner. For example, words such as "include", "includes", and "having" are open class vocabulary terms and can be used interchangeably to mean "including, but not limited to, XXX". Unless otherwise clearly indicated by the context, the terms "or" and "and" used herein refer to, and can be used interchangeably with, the term "and / or". The term "for example" used herein refers to, and can be used interchangeably with, the collocation "including, but not limited to, XXX".

[0080] As further pointed out, in the apparatus, equipment and method of the present invention, each component or each step can be disassembled and / or recombined, and such disassembly and / or recombination should be considered as an equivalent of the present invention.

[0081] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

[0082] The above description is merely a preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent replacements, etc. made within the spirit and principle of the present invention should be included within the protection scope of the present invention.

Claims

1. 1. A method for spatial localization of a separated virtual system, comprising: A spatial localization controller, a virtual head-mounted display device, a mobile terminal detachably mounted on the virtual head-mounted display device, and a control handle communicatively connected to the mobile terminal, wherein a first inertial sensor is provided on the mobile terminal, and a photographing device is provided on the mobile terminal, the first inertial sensor is for detecting IMU data of the mobile terminal, an infrared sensor and a second inertial sensor are provided on the control handle, and the photographing device of the mobile terminal is also used to photograph a spot image of the infrared sensor; a method for spatial localization of the separated virtual system, the method being performed by the spatial localization controller; determining that the mobile terminal is attached to the virtual head mounted display device, and when determining that the mobile terminal has been attached to the virtual head mounted display device, controlling the image capture device to capture an image of an environment in which the virtual head mounted display device is located to form a first image; Controlling the first inertial sensor to detect IMU data of the virtual head mounted display device to which the mobile terminal is attached; performing calculations based on the IMU data of the virtual head mounted display and the first image to generate spatial localization information of the virtual head mounted display; Controlling the imaging device to image the infrared sensor on the control handle to form a spot image; Controlling the second inertial sensor to detect IMU data of the control handle; and performing a calculation based on the spatial localization information of the virtual head mounted display device, the IMU data of the control handle, and the spot image to generate spatial localization information of the control handle; performing a calculation based on the IMU data of the virtual head mounted display device and the first image to generate spatial localization information of the virtual head mounted display device, determining translational degree of freedom data for the virtual head mounted display device based on the first image; Determining translational degree of freedom data for the virtual head mounted display device based on the first image includes: extracting image features from the first image and extracting reference features from a reference image, the reference image being an image of a portion of a map of a space in which the virtual head mounted display device is located; matching the image feature points with the reference feature points to generate valid feature points; determining whether the first image is initialized; Whether the first image is initialized is determined by determining that the first image is initialized when the luminance, grayscale, and resolution of the first image satisfy a predetermined requirement; if determining that the first image is initialized, extracting a key region from the first image based on the effective feature points based on a sliding window method, and generating map points based on the key region, the effective feature points, and a map of a space in which the virtual head mounted display device is located; if determining that the first image is not initialized; Identifying a position of the first image based on a map of a space in which the virtual head mounted display device is located, and extracting a key region from the first image based on the effective feature points and a sliding window method in response to a state in which the position of the first image has been identified; and determining map information for said virtual head mounted display device, said map information including said translational degree of freedom data, based on said map points and said map.

2. performing a calculation based on the IMU data of the virtual head mounted display device and the first image to generate spatial localization information of the virtual head mounted display device, determining 6DOF data for the virtual head mounted display based on the translational degree of freedom data for the virtual head mounted display and the IMU data for the virtual head mounted display; 2. The method of claim 1, wherein said virtual head mounted display spatial localization information includes 6 DOF data for said virtual head mounted display.

3. When determining that the mobile terminal is attached to the virtual head mounted display device, controlling the image capturing device to capture an image of an environment in which the virtual head mounted display device is located to form a first image, The spatial position determination method for a separated virtual system as described in claim 1, characterized in that when the connection plug-in of the mobile terminal contacts the connection plug-in of the virtual head-mounted display device, it includes controlling the photographing device to photograph the environment in which the virtual head-mounted display device is located to form a first image.

4. A separate virtual system using the method for spatial localization of a separate virtual system according to any one of claims 1 to 3, A mobile device; A first inertial sensor for detecting IMU data of the mobile terminal, the first inertial sensor being installed on the mobile terminal; a virtual head mounted display device to which the mobile terminal is removably attached; A photographing device installed in the mobile terminal; a spatial localization controller of a separate virtual system communicatively coupled to the mobile terminal, the first inertial sensor, and the image capture device, respectively; A control handle; An infrared sensor and a second inertial sensor are installed on the control handle, the second inertial sensor being used to measure IMU data of the control handle; The spatial localization controller includes: a function switching unit for determining that the mobile terminal is attached to the virtual head mounted display device, and for switching a function of the mobile terminal to a virtual head mounted display function when determining that the mobile terminal has been attached to the virtual head mounted display device; an image capture control unit for controlling the image capture device to capture an image of an environment in which the virtual head mounted display device is located to form a first image; a sensor control unit for controlling the first inertial sensor to detect IMU data of the virtual head mounted display device to which the mobile terminal is attached; a localization unit for performing calculations based on the IMU data of the virtual head mounted display device and the first image to generate spatial localization information of the virtual head mounted display device; The photographing device of the mobile terminal is further used to photograph a spot image of the infrared sensor; The photographing control unit is further used for controlling the photographing device to photograph the infrared sensor on the control handle to form a spot image; The sensor control unit is further used for controlling the second inertial sensor to detect the IMU data of the control handle; The localization unit is further used for performing calculations according to the spatial localization information of the virtual head mounted display device, the IMU data of the control handle, and the spot image to generate spatial localization information of the control handle; The position determination unit includes: a translational degree of freedom determination module for determining translational degree of freedom data for the virtual head mounted display device based on the first image; The translational degree of freedom determination module includes: a feature determination module for extracting image feature points from the first image, extracting reference feature points from a reference image which is an image of a portion of a map of a space in which the virtual head mounted display device is located, and matching the image feature points with the reference feature points to generate valid feature points; a region extraction module for extracting a key region from the first image based on the effective feature points and based on a sliding window method; a map point generation module for generating map points based on the key region, the effective feature points, and a map of a space in which the virtual head mounted display device is located; and a map information determination module for determining map information for said virtual head mounted display including said translational degree of freedom data based on said map points and said map.

5. The position determination unit includes: and a six degree of freedom determination module for determining six-dof data of the virtual head mounted display device based on the translational degree of freedom data of the virtual head mounted display device and the IMU data of the virtual head mounted display device.

5. The separated virtual system of claim 4, wherein said virtual head mounted display spatial localization information includes 6 DOF data for said virtual head mounted display.

Images