Positioning method, positioning device and system

Abstract

The application provides a positioning method, a positioning device and a system. The method comprises: the positioning device tracks a virtual reality device; the positioning device projects a coded image through a projector; the positioning device receives a plurality of light intensity values from the virtual reality device; the positioning device determines a pose of the virtual reality device relative to the positioning device according to the projected coded image, the plurality of light intensity values and positions of a plurality of light intensity sensors in the virtual reality device; and the pose of the virtual reality device relative to the positioning device and a pose of the positioning device in an environment coordinate system are used to determine a pose of the virtual reality device in the environment coordinate system. According to the embodiment of the application, the positioning device can move along with the virtual reality device, and the pose of the virtual reality device is determined during the movement. The method can expand the activity range of the user and improve the use experience of the user.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of electronics, and in particular to a positioning method, a positioning device and a system. BACKGROUND

[0002] Virtual reality (VR) technology is a computer simulation technology that can create and experience a virtual scene. It uses a computer to generate a virtual scene, allowing users to immerse themselves in the virtual scene.

[0003] Virtual reality technology can use real-life data to generate electronic signals through computer technology, and combine them with various virtual reality devices (such as head-mounted display devices and handsets) to convert them into virtual scenes. Users can interact with the virtual scene through virtual reality devices. To achieve user interaction with the virtual scene, the virtual reality device needs to identify its position and posture in the real environment, so as to determine the virtual reality content corresponding to the position and posture of the virtual reality device.

[0004] Currently, due to the comfort and weight requirements of head-mounted display devices, too many positioning devices cannot be installed, and virtual reality devices are often positioned by fixed laser scanners or cameras in space. This method greatly limits the range of activities of users wearing virtual reality devices, and causes obstacles to improving the user experience.

[0005] How to expand the range of activities of users in the virtual scene and improve the immersive experience of users is the current and future research direction. SUMMARY

[0006] The present application provides a positioning method, a positioning device and a system. The positioning device can track the virtual reality device, and the positioning device determines the position and posture of the virtual reality device in the room by the position and posture of the positioning device in the room and the relative position and posture of the positioning device and the virtual display device. This method can expand the range of activities of users in the virtual scene and improve the immersive experience of users.

[0007] In a first aspect, embodiments of the present application provide a positioning method applied to a positioning device, the positioning device comprising a projector, and the method comprising:

[0008] The positioning device tracks the virtual reality device;

[0009] The positioning device projects a coded image through the projector;

[0010] The positioning device receives a plurality of light intensity values from the virtual reality device; the plurality of light intensity values comprise light intensities of the coded image received by a plurality of light intensity sensors of the virtual reality device;

[0011] The positioning device determines the pose of the virtual reality device relative to the positioning device according to the projected coded image, the plurality of light intensity values, and the positions of the plurality of light intensity sensors in the virtual reality device; and the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system are used to determine the pose of the virtual reality device in the environment coordinate system.

[0012] By implementing the embodiments of the present application, the positioning device can track the virtual reality device, and determine the pose of the virtual reality device in the indoor environment by using the pose of the positioning device in the indoor environment and the relative pose of the positioning device and the virtual display device. The positioning device can move along with the virtual reality device, and determine the pose of the virtual reality device during the movement. The method can expand the activity range of the user and improve the use experience of the user.

[0013] With reference to the first aspect, in a possible implementation manner, the method further includes:

[0014] The positioning device determines the pose of the virtual reality device in the environment coordinate system according to the pose of the positioning device in the environment coordinate system and the pose of the virtual reality device relative to the positioning device.

[0015] The positioning device sends the pose of the virtual reality device in the environment coordinate system to the virtual reality device.

[0016] With reference to the first aspect, in a possible implementation manner, the coded image includes a plurality of coded images; and the positioning device determines the pose of the virtual reality device relative to the positioning device according to the projected coded image, the plurality of light intensity values, and the positions of the plurality of light intensity sensors in the virtual reality device, including:

[0017] The positioning device generates the code of the light intensity sensor based on the light intensity value of each coded image received by the light intensity sensor.

[0018] The positioning device determines the pixel coordinates of the plurality of light intensity sensors based on the codes of the plurality of light intensity sensors.

[0019] The positioning device determines the pose of the virtual reality device relative to the projector based on the pixel coordinates of the plurality of light intensity sensors and the positions of the plurality of light intensity sensors in the virtual reality device.

[0020] The positioning device converts the pose of the virtual reality device relative to the projector into the pose of the virtual reality device relative to the positioning device based on the pose of the projector relative to the positioning device.

[0021] With reference to the first aspect, in a possible implementation manner, the plurality of coded images comprises M first images and N second images, M is a positive integer, N is a positive integer, the first image is a binary image of a pattern of a first-direction stripe, and the second image is a binary image of a pattern of a second-direction stripe; the positioning device generates a code of the light intensity sensor based on a light intensity value of each coded image received by the light intensity sensor, and the code of the light intensity sensor comprises:

[0022] The positioning device generates a first code of the light intensity sensor based on a light intensity value of the first image received by the light intensity sensor.

[0023] The positioning device generates a second code of the light intensity sensor based on a light intensity value of the second image received by the light intensity sensor.

[0024] The positioning device determines a pixel coordinate of the light intensity sensor based on the codes of the plurality of light intensity sensors, comprising: the positioning device determines a first coordinate of the light intensity sensor in the first direction based on the first code of the light intensity sensor; and the positioning device determines a second coordinate of the light intensity sensor in the second direction based on the second code of the light intensity sensor, and the pixel coordinate of the light intensity sensor comprises the first coordinate of the light intensity sensor and the second coordinate of the light intensity sensor.

[0025] The binary coded image is a binary image with a pattern distribution complying with a coding rule, and the coding rule of the coded pattern can be a binary code or a Gray code.

[0026] With reference to the first aspect, in a possible implementation manner, the method further comprises:

[0027] The positioning device determines the pose of the projector relative to the positioning device based on the pose of the projector relative to the camera and the pose of the positioning device relative to the camera.

[0028] With reference to the first aspect, in a possible implementation manner, the method further comprises:

[0029] The positioning device determines the display content based on the pose of the virtual reality device in the environment coordinate system, the three-dimensional map and the media resource.

[0030] The positioning device sends the display content to the virtual reality device, so that the virtual reality device displays the display content.

[0031] With reference to the first aspect, in a possible implementation manner, before the positioning device projects the coded images through the projector, the method comprises:

[0032] The positioning device sends indication information to the virtual reality device, and the indication information is used to instruct the virtual reality device to sample light intensities received by the plurality of light intensity sensors to obtain the plurality of light intensity values.

[0033] With reference to the first aspect, in a possible implementation manner, the positioning device tracks the virtual reality device, including:

[0034] The positioning device locates the position of the virtual reality device;

[0035] The positioning device moves to a position with a preset distance from the virtual display device;

[0036] The positioning device determines a relative position between a user wearing the virtual display device and the positioning device through the photographed image;

[0037] The positioning device moves to a direction in which the user's face is oriented based on the relative position.

[0038] The second aspect, the embodiments of the present application provide a positioning method applied to a virtual reality device, the virtual reality device including a plurality of light intensity sensors, the method including:

[0039] The plurality of light intensity sensors respectively receive light intensity of a coded image projected by a positioning device, to obtain a plurality of light intensity values, the positioning device tracking the virtual reality device;

[0040] The virtual reality device sends the plurality of light intensity values to the positioning device; the plurality of light intensity values, the coded image and positions of the plurality of light intensity sensors in the virtual reality device are used for the positioning device to determine a pose of the virtual reality device relative to the positioning device; the pose of the virtual reality device relative to the positioning device and a pose of the positioning device in an environment coordinate system are used for determining a pose of the virtual reality device in the environment coordinate system.

[0041] With reference to the second aspect, in a possible implementation manner, the method further includes:

[0042] The virtual reality device receives the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system from the positioning device;

[0043] The virtual reality device determines the pose of the virtual reality device in the environment coordinate system based on the pose of the positioning device in the environment coordinate system and the pose of the virtual reality device relative to the positioning device.

[0044] With reference to the second aspect, in a possible implementation manner, the method further includes:

[0045] The virtual reality device determines display content based on the pose of the virtual reality device in the environment coordinate system, a three-dimensional map and a media resource;

[0046] The virtual reality device displays the display content.

[0047] With reference to the second aspect, in a possible implementation manner, the method includes:

[0048] The virtual reality device samples the light intensities received by the plurality of light intensity sensors to obtain a plurality of light intensity values when receiving the indication information sent by the positioning device.

[0049] With reference to the second aspect, in a possible implementation, the second electronic device sends, to the first electronic device, the physical direction of the second electronic device including:

[0050] The second electronic device obtains the physical direction of the second electronic device through a sensor.

[0051] The second electronic device sends the physical direction to the first electronic device when the physical direction changes.

[0052] In a third aspect, the present application provides an electronic device. The electronic device can include a memory and a processor. The memory can be used to store a computer program. The processor can be used to invoke the computer program, so that the electronic device executes the first aspect or any possible implementation of the first aspect.

[0053] In a fourth aspect, the present application provides an electronic device. The electronic device can include a memory and a processor. The memory can be used to store a computer program. The processor can be used to invoke the computer program, so that the electronic device executes the second aspect or any possible implementation of the second aspect.

[0054] In a fifth aspect, the present application provides a computer program product including instructions, characterized by causing the electronic device to execute the first aspect or any possible implementation of the first aspect when the computer program product runs on the electronic device.

[0055] In a sixth aspect, the present application provides a computer program product including instructions, characterized by causing the electronic device to execute the second aspect or any possible implementation of the second aspect when the computer program product runs on the electronic device.

[0056] In a seventh aspect, the present application provides a computer-readable storage medium including instructions, characterized by causing the electronic device to execute the first aspect or any possible implementation of the first aspect when the instructions run on the electronic device.

[0057] In an eighth aspect, the present application provides a computer-readable storage medium including instructions, characterized by causing the electronic device to execute the second aspect or any possible implementation of the second aspect when the instructions run on the electronic device.

[0058] In a ninth aspect, an embodiment of the present application provides a positioning system, comprising a first electronic device and a second electronic device, wherein the first electronic device is the electronic device described in the third aspect, and the second electronic device is the electronic device described in the fourth aspect.

[0059] It can be understood that the electronic device provided in the third aspect and the fourth aspect, the computer program product provided in the fifth aspect and the sixth aspect, and the computer readable storage medium provided in the seventh aspect and the eighth aspect are all used to execute the method provided in the embodiments of the present application. Therefore, the beneficial effects achieved thereby can refer to the beneficial effects in the corresponding method, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a schematic diagram of a pixel coordinate system provided by an embodiment of the present application;

[0061] Figure 2A is a system architecture diagram of a virtual reality system provided by an embodiment of the present application;

[0062] Figure 2B is another system architecture diagram of a virtual reality system provided by an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of a head-mounted display device provided by an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of a robot provided by an embodiment of the present application;

[0065] Figure 5 is a schematic diagram of a relative position relationship between a head-mounted display device and a robot provided by an embodiment of the present application;

[0066] Figure 6 is a schematic diagram of a scene provided by an embodiment of the present application;

[0067] Figure 7 is a flowchart of a positioning method provided by an embodiment of the present application;

[0068] Figure 8 is a schematic diagram of an encoded image provided by an embodiment of the present application;

[0069] Figure 9 is a schematic diagram of two groups of encoded images provided by an embodiment of the present application;

[0070] Figure 10 is a schematic diagram of light intensity values received by a light intensity sensor provided by an embodiment of the present application;

[0071] Figure 11 is a structural schematic diagram of a head-mounted display device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; the "and / or" in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0073] Hereinafter, the terms "first" and "second" are only used for description purposes, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" is two or more than two.

[0074] The term "user interface (UI)" in the following embodiments of the present application is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java, extensible markup language (XML), etc. The interface source code is parsed, rendered, and finally presented as content that can be recognized by the user on the electronic device. The commonly used form of user interface is graphic user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, etc. displayed in the display screen of the electronic device.

[0075] First, the technical terms involved in the embodiments of the present application will be described first.

[0076] I. Structured light system

[0077] The structured light system refers to a system for measuring by using structured light. The structured light system can be composed of an optical projector, a camera and a computer processing system. The basic principle is that the optical projector projects a specific light signal (the specific light information can also be referred to as structured light) to the surface of an object and a background, the light signal projected by the optical projector is collected by the camera, and then the position and depth information of the object are calculated according to the change of the light signal caused by the object. The above position and depth information can be used to restore the entire three-dimensional space. The optical projector can be a projector and a laser, etc.

[0078] 1. Structured light

[0079] A set of projected light rays with known spatial directions is structured light.

[0080] Based on the beam mode, the structured light can be divided into point structured light, line structured light, multi-line structured light and surface structured light, etc. Furthermore, according to the beam mode of the structured light projected by the optical projector, the structured light system can be divided into point structured light mode, line structured light mode, multi-line structured light mode and surface structured light mode, etc.

[0081] The point structured light is a light beam projected onto an object to form a light spot. The point structured light mode refers to that a light beam emitted by a laser is projected onto an object to generate a light spot, the light spot is imaged on the image plane of the camera through the lens of the camera to form a two-dimensional point, the line of sight of the camera and the light beam intersect at the light spot in space to form a simple triangular geometric relationship. Through certain calibration, the triangular geometric constraint relationship can be obtained, and the spatial position of the light spot in a known world coordinate system can be uniquely determined therefrom. This method needs to scan the object point by point for measurement, and the image shooting and image processing need time, which increases sharply with the increase of the volume of the measured object.

[0082] The line structured light is a light beam projected onto an object to form a light strip, the line structured light mode is to project a light beam onto an object to form a light strip on the surface of the object; the multi-line structured light is to project multiple light strips; and the surface structured light mode projects a two-dimensional structured light image, and the structured light pattern can be a grating stripe.

[0083] 2. Surface structured light mode

[0084] The surface structured light is a light beam projected onto the surface of an object to form a two-dimensional image. The surface structured light mode is to project a two-dimensional structured light pattern onto the surface of an object by an optical projector. This method can realize three-dimensional profile measurement without scanning, and the measurement speed is very fast. The most commonly used method in the surface structured light is to project a grating stripe onto the surface of an object.

[0085] 3. Measurement method of coded structured light

[0086] The coded structured light is coded structured light, and a pattern formed by the coded structured light projected onto an object conforms to a coding rule corresponding to a coding method. The coding method includes Gray encoding, binary encoding, two-dimensional grid pattern encoding, random pattern encoding, color encoding, gray scale encoding, neighborhood encoding, phase encoding, and hybrid encoding, etc. For example, the pattern on the binary encoded image can be a stripe, and the arrangement of the stripe on the binary encoded image conforms to Gray encoding.

[0087] The correspondence between the object surface point and the pixel point of the two-dimensional image is determined by the coded structured light, which is called coded structured light measurement method. The coded structured light measurement method can be divided into spatial coding method and time coding method. Since the coded structured light is projected onto the object surface to form an image with a coded pattern, the coded structured light is referred to as a coded image for convenience of description.

[0088] The time domain coding method is that the optical projector projects a plurality of coded images in time sequence, wherein the pattern on each of the plurality of coded images is different, the pattern on each of the coded images is composed of a plurality of sub-patterns, and each sub-pattern of each coded image corresponds to a numerical value; the camera captures a projection image when the optical projector projects a coded image, and a plurality of projection images are obtained; the computer processing system can identify the sub-patterns on the plurality of projection images to obtain a plurality of sub-patterns corresponding to each pixel on the projection image, and then determine a digital sequence corresponding to each pixel based on the correspondence between the sub-patterns and the numerical values; by identifying the object on each pixel point, the digital sequence corresponding to the object on each pixel point is determined, and the digital sequence is used to indicate the position of the object.

[0089] 4. Binary encoded image

[0090] The binary encoded image refers to a binary image (Binary Image) whose pattern distribution conforms to the coding rule corresponding to the coding method. The binary image refers to an image in which each pixel has only two possible values or gray levels.

[0091] Before introducing the coding of the binary image, the pixel coordinate system is introduced.

[0092] The projector projects the binary encoded image onto the object, and the position of the object in the projector coordinate system can be determined by coding. The position of the object in the projector coordinate system can be represented by the pixel coordinate system. The pixel coordinate system is a two-dimensional coordinate system, and the unit of the pixel coordinate system is pixel. The coordinate origin O is in the upper left corner.

[0093] Figure 1 The pixel coordinate system of the first image is shown in the (A) example in FIG. 1. Figure 1As shown in (A), the rectangle represents the first image. With the top-left corner of the first image as the origin O, a pixel coordinate system can be established as shown in the figure. The units of the horizontal coordinate u and the vertical coordinate v in this pixel coordinate system are pixels. It can be understood that the horizontal coordinate u and the vertical coordinate v of a pixel in the image indicate the column number and row number, respectively.

[0094] like Figure 1 As shown in (B) above, vertical stripes represent a column of pixels in an image. The horizontal coordinate of the first column of pixels is 1, the horizontal coordinate of the second column is 2, the horizontal coordinate of the third column is 3, and so on. Figure 1 As shown in (C), vertical stripes represent a row of pixels in the image. The horizontal coordinate of the first row of pixels is 1, the horizontal coordinate of the second row of pixels is 2, the horizontal coordinate of the third row of pixels is 3, and so on.

[0095] For example, if a pixel has u=1 and v=2, then that pixel is located at... Figure 1 The first column in (B) is... Figure 1 The overlapping pixel positions in the second row of (C) in the diagram.

[0096] The following uses five-bit Gray code as an example to introduce the encoding and decoding process of images. Five-bit Gray code consists of 32 Gray codes, each composed of a five-bit code value.

[0097] In the process of encoding a binary image, the number of bits in the Gray code and the number of binary images can be determined first based on the size of the binary image. For example, if the width of a binary image is 32 pixels and the height is less than 32 pixels, then a five-bit Gray code can be used to encode the horizontal coordinate value of the binary image, and the number of binary images is 5. For example, if the five-bit Gray code corresponding to the first pixel position is 00011, then the encoding of the first binary image projected onto the first pixel position is 0; the encoding of the second binary image projected onto the first pixel position is 0; the encoding of the third binary image projected onto the first pixel position is 0; the encoding of the fourth binary image projected onto the first pixel position is 1; and the encoding of the fifth binary image projected onto the first pixel position is 1. Therefore, five binary images can constitute the five-bit Gray code for a pixel.

[0098] In the process of encoding a binary image, the two-dimensional coded image can be encoded according to the horizontal and vertical coordinates of the pixel position. Taking five-bit Gray code as an example, if the horizontal coordinate of a certain pixel position is 3, since the Gray code corresponding to the binary number 3 is 00011, the pixel position is encoded as 00011. Therefore, when the Gray code of a certain pixel position is determined to be 00011, the horizontal coordinate of that pixel position can be determined to be 3.

[0099] The embodiment of the present application is to replace the camera with a photosensitive sensor to cooperate with the projector to obtain the pixel coordinates of the object.

[0100] II. Pose

[0101] 1. Definition of pose

[0102] The pose of an object refers to the position and attitude of the object in a certain coordinate system, which can be described by the relative attitude of the coordinate system attached to the object.

[0103] For example, an object can be represented by a coordinate system B attached to the object, and the attitude of the object relative to the coordinate system A is equivalent to the attitude of the coordinate system B relative to the coordinate system A. For example, a robot coordinate system F2 is established with a fixed point as the origin, and the attitude of the robot relative to the environment coordinate system F0 is equivalent to the attitude of the robot relative to the robot coordinate system F2 relative to the environment coordinate system F0.

[0104] Wherein, the attitude of the coordinate system B relative to the coordinate system A can be represented by a rotation matrix R and a translation matrix T, and the attitude of the coordinate system B relative to the coordinate system A can be represented as The attitude of the object relative to the coordinate system A can be represented by It should be noted that when the coordinate system A is the environment coordinate system F0, the pose of the object can be represented by

[0105] In the embodiment of the present application, if the coordinate system in the room is represented by the environment coordinate system F0, the positioning of the head-mounted display device in the room is the pose of the head-mounted display device in the environment coordinate system F0. The head-mounted display device can be represented by the head-mounted display device coordinate system F3, and the pose of the head-mounted display device in the environment coordinate system F0 is

[0106] 2. Perspective-n-Point (PnP) algorithm

[0107] The PnP algorithm is a method for solving the motion of 3D to 2D point pairs, which is used to estimate the pose of an object based on n 3D space points and the projection positions of the n 3D space points. Wherein, the n 3D space points are points on the object, and n is a positive integer. Wherein, the projection positions of the n 3D space points can be obtained based on the structured light system, and the projection positions of the n 3D space points can be represented by the coordinates in the pixel coordinate system.

[0108] There are many methods to solve the PnP problem, such as P3P, direct linear transformation (DLT), and EPnP, which estimate the pose of three pairs of points. In addition, a nonlinear optimization method can be used to construct a least squares problem and iteratively solve it. ​

[0109] In the embodiments of the present application, the robot can determine the pose of the head-mounted display device in the projector coordinate system based on the PnP algorithm after obtaining the three-dimensional coordinates of the L optical sensors in the head-mounted display device coordinate system and the coordinates of the L optical sensors in the pixel coordinate system. Wherein, L is a positive integer.

[0110] III. Real-time positioning and simultaneous localization and mapping (SLAM) algorithm

[0111] 1. Classification of SLAM

[0112] Currently, the sensors used in SLAM are mainly divided into two categories: Lidar SLAM based on laser radar and VSLAM (Visual SLAM) based on vision. Among them, Lidar SLAM is based on the point cloud information returned by the laser radar, and VSLAM is based on the image information returned by the camera.

[0113] In the embodiments of the present application, the robot can use laser SLAM to realize map construction and positioning. It should be understood that laser SLAM is mainly applied in indoor, and the precision of the map constructed by laser SLAM is higher than that of the map constructed by visual SLAM. It should be noted that in other embodiments, visual SLAM can also be used to realize the indoor map construction and positioning function of the robot.

[0114] 2. Laser SLAM

[0115] Laser SLAM can use two-dimensional (2-dimension, 2D) laser radar or three-dimensional (3-dimension, 3D) laser radar. Among them, two-dimensional laser radar is generally used on indoor robots (such as sweeping robots), and two-dimensional laser radar is generally used in the field of unmanned driving.

[0116] The main functions of laser radar are two-fold: on the one hand, it can provide point cloud data for the mapping algorithm. When the mapping algorithm obtains sufficient point cloud data, it can construct a local map with the robot as the center and the radar range as the radius. On the other hand, it can correct the predicted pose of the Bayesian filter through the system observation model, and improve the accuracy of the filtered estimated robot pose. Under the condition of knowing the best pose, the local map can be updated to the global map.

[0117] The map constructed by two-dimensional laser SLAM is a two-dimensional grid map; the map constructed by three-dimensional laser SLAM is a three-dimensional grid map. Figure 1Generally, the three-dimensional point cloud map is a map composed of discrete three-dimensional space points. Currently, 2D-SLAM algorithms include the gmapping-SLAM algorithm and the Hector-SLAM algorithm, etc. For example, the gmapping-SLAM algorithm adopts a particle filtering method, which can finally convert the collected laser ranging data into a grid map. The core idea of particle filtering is to express the spatial distribution by randomly state particles extracted from the posterior probability (observation equation). In simple terms, particle filtering refers to a process of approximating a probability density function by finding a set of random samples propagating in the state space, replacing the state equation with the sample mean, so as to obtain the minimum variance distribution of the state. The above-mentioned sample refers to a particle.

[0118] In the embodiments of the present application, the robot can construct a two-dimensional map of the indoor space through two-dimensional laser SLAM, and combine the point cloud data detected by the structured light system to synthesize a three-dimensional map of the indoor space. The robot can also construct a three-dimensional map of the indoor space through three-dimensional laser SLAM.

[0119] In order to more clearly and in detail introduce the positioning method based on the virtual reality system provided in the embodiments of the present application, the virtual reality system provided in the embodiments of the present application will be introduced first.

[0120] Please refer to Figure 2A , Figure 2A is a system architecture diagram of a virtual reality system provided in the embodiments of the present application. As Figure 2A indicated, the virtual reality system includes a virtual reality device 100 and a positioning device 200. The virtual reality device 100 and the positioning device 200 can establish a communication connection. Among them:

[0121] The virtual reality device 100 can be a head-mounted display device and a handle, etc. The head-mounted display device can also be referred to as a head-mounted glasses device, a glasses device, a VR glasses, etc. The handle can also be referred to as an electromagnetic handle, a control handle, etc.

[0122] The positioning device 200 is used for positioning the virtual reality device 100. The positioning device is a device with a moving ability and a positioning ability, such as a mobile robot or a flying robot, etc.

[0123] In some embodiments, a user can wear a head-mounted display device to view a kind of virtual immersive scene, wherein the head-mounted display device determines the scene content displayed through its position in space. That is to say, the head-mounted display device will change the display content output based on the position change of the head-mounted display device. Here, the positioning problem of the head-mounted display device is involved, and the head-mounted display device needs to obtain its spatial positioning to determine the scene content displayed, so as to realize the immersive experience of the user and the obstacle avoidance function of the virtual reality system.

[0124] In the embodiment of the present application, the positioning device 200 can position the virtual reality device 100 to obtain the position information of the virtual display device, and then the positioning device 200 can send the position information to the virtual reality device 100, so that the virtual reality device 100 displays the virtual scene picture based on the position information. The specific implementation can be referred to in the following embodiments, which will not be described here.

[0125] The communication connection established between the virtual reality device 100 and the positioning device 200 can include but is not limited to: wireless fidelity direct (Wi-Fi direct) (also known as wireless fidelity peer-to-peer (Wi-Fi P2P)) communication connection, Bluetooth communication connection, near field communication (NFC) connection, etc.

[0126] Exemplarily, the embodiment of the present application provides another system architecture diagram of a virtual reality system.

[0127] Please refer to Figure 2B , Figure 2B is a virtual reality system provided by the embodiment of the present application. As shown in Figure 2B , the virtual reality system includes a head-mounted display device 300 and a robot 400. The head-mounted display device 300 is in communication connection with the robot 400 through Bluetooth, WiFi and other short-distance wireless communication methods. Wherein:

[0128] When the user wears the head-mounted display device 300 and walks in the room, the robot 400 can follow the user to move and position the head-mounted display device 300 in real time, and send the pose of the head-mounted display device 300 in the room to the head-mounted display device 300, so that the head-mounted display device 300 determines the display content based on its own pose.

[0129] The head-mounted display device 300 can be provided with a light intensity sensor, and the light intensity sensor is used to receive light intensity. For example, the robot 400 can be provided with a projector, and the projector projects to the head-mounted display device 300, and then the head-mounted display device 300 can send the light intensity value received by the light intensity sensor to the robot 400, so that the robot 400 determines the pose of the head-mounted display device 300 based on the above light intensity value.

[0130] The light intensity sensor is a sensitive device that has a response or conversion function to external light signals or light radiation. The light intensity sensor includes a phototube, a photomultiplier tube, a photoresistor, a photosensitive triode, a solar cell, an infrared sensor, an ultraviolet sensor, a fiber-optic photoelectric sensor, a color sensor, etc.

[0131] Please refer to Figure 3 , Figure 3 The possible distribution of light intensity sensors on the head-mounted display device 300 is shown. As Figure 3 shown in FIG. 3B, the black dots represent the light intensity sensors 310, Figure 3 The distribution of 20 light intensity sensors is shown.

[0132] Please refer to Figure 4 , Figure 4 A schematic diagram of a robot 400 provided by an embodiment of the present application is shown. As Figure 4 shown, the robot 400 includes a structured light system 410 and a moving system 420. Wherein:

[0133] The structured light system 410 includes a camera 411 and a projector 412. Wherein, the projector 412 is used to emit coded structured light, such as a binary coded image; the camera 411 is used to shoot images. As Figure 4 shown, the dashed line 101 and the dashed line 102 are used to indicate the shooting range of the camera 411, and the solid line 201 and the solid line 202 are used to indicate the projection range of the projector 412. It can be seen that the shooting range of the camera 411 and the projection range of the projector 412 have an overlapping area, so that the camera 411 can shoot the area where the projector 412 projects.

[0134] Wherein, the projector 412 can be a wide-angle infrared digital light projector or other projector, which is not limited here.

[0135] The moving system 420 can include a plurality of wheels and a driving device, and the driving device is used to drive the plurality of wheels to move the robot 400. The robot 400 can control the moving system 420 to keep itself at a preset distance and angle from the head-mounted display device 300, so that the head-mounted display device 300 is located within the working range of the camera 411 and the projector 412. For example, the relative position relationship between the head-mounted display device 300 and the robot 400 can be as Figure 5 shown, the user can wear the head-mounted display device 300, and the robot 400 keeps the head-mounted display device 300 within a preset distance by controlling the moving system 420, that is, when the user moves, the robot 400 moves following the movement of the user, so that the head-mounted display device 300 is located within the working range of the camera 411 and the projector 412.

[0136] In one implementation, the robot 400 maintains a preset distance and angle with the head-mounted display device 300 so that the head-mounted display device 300 is located in the working area of ​​the camera 411 and the projector 412; then, the robot 400 can control the projector 412 to emit coded structured light to the head-mounted display device 300; the robot 400 can take pictures of the head-mounted display device 300 through the camera 411 and decode the pictures taken by the camera 411 to determine the pose of the head-mounted display device 300.

[0137] In some embodiments, the camera 411 and the projector 412 are rotatable. The robot can control the rotation of the camera 411 and the projector 412, and calculate the transformation relationship between the rotated camera 411 and the projector 412 or the transformation relationship between the camera 411 and the robot 400, etc., through rotation parameters.

[0138] Optionally, the robot may also include a laser SLAM system. The laser SLAM system is used by the robot 400 to detect the indoor environment and generate an indoor map. Then, the robot 400 can determine its own position within the room based on this map, and thereby determine the pose of the head-mounted display device within the room based on its own position. Details regarding how the robot 400 determines the pose of the head-mounted display device within the room can be found in the following embodiments, and will not be repeated here.

[0139] by Figure 2B System architecture, Figure 3 Provided head-mounted display devices and Figure 4 The robot provided in this application embodiment offers a method such as... Figure 6 The scene shown.

[0140] Please see Figure 6 The user wears a head-mounted display device 300, and the user and the robot 400 are in the same indoor environment. At the same time, the indoor environment also includes several obstacles 500.

[0141] First, let's introduce several coordinate systems used indoors:

[0142] 1. Environmental coordinate system F0

[0143] The robot establishes a coordinate system with a point indoors as the origin. Figure 4 The environmental coordinate system F0 is shown. F0 is a three-dimensional coordinate system used to describe the three-dimensional coordinates of an object in the real world. Every location indoors can be represented by a coordinate point in this environmental coordinate system F0. In some embodiments, F0 may also be called the world coordinate system.

[0144] 2. Coordinate system of structured light system

[0145] The coordinate system of the structured light system comprises a camera coordinate system F11 and a projector coordinate system F12. The camera coordinate system F11 is a coordinate system with a camera optical center as an origin and an optical axis coinciding with the z-axis. The projector coordinate system F12 is a coordinate system with a projector optical center as an origin and an optical axis coinciding with the z-axis.

[0146] After the positions of the camera and the projector are determined, the camera parameters, the projector parameters and the transformation relationship between the camera coordinate system F11 and the projector coordinate system F12 can be obtained by calibrating the camera and the projector. The camera parameters comprise a camera focal length and a camera center point coordinate. The transformation relationship between the projector coordinate system F12 and the camera coordinate system F11 can be represented by a matrix , wherein R represents a rotation matrix and T represents a translation matrix. The calibration method can be a machine vision software (HALCON) or other calibration methods, which are not limited herein.

[0147] Specifically, the projector projects the coded structured light onto an object, and the camera captures the object. The coordinates of the object in the pixel coordinate system can be determined by decoding the image captured by the camera. The origin of the pixel coordinate system is at the top left corner of the image, and the basic unit is a pixel. It can be understood that the coordinates of the object in the pixel coordinate system can reflect the positional relationship between the object and the camera coordinate system F11 and the projector coordinate system F12, as the camera coordinate system F11, the projector coordinate system F12 and the position of the object are determined.

[0148] 3. Robot coordinate system F2

[0149] The robot coordinate system F2 is a coordinate system with a fixed point on the robot as an origin. The position of the fixed point is fixed relative to the structured light coordinate system. For example, the origin of the robot coordinate system F2 can be located at the position as shown in Figure 6 .

[0150] As shown in Figure 6 , the camera 411 is mounted on the robot 400. After the positions of the camera 411 and the robot 400 are determined, the rotation matrix and the translation matrix of the camera coordinate system F11 relative to the robot coordinate system F2 can be determined by the hand-eye calibration method, which can be represented by , which is used to indicate the transformation relationship of F11 relative to F2. It should be noted that the robot can also determine the rotation matrix and the translation matrix of the camera coordinate system F11 relative to the robot coordinate system F2 by other calibration methods, which are not limited herein.

[0151] 4. Head-mounted display device coordinate system F3

[0152] The head-mounted display device coordinate system F3 is a coordinate system with a fixed point on the head-mounted display device as an origin.​

[0153] In the embodiments of the present application, the head-mounted display device is provided with a plurality of light intensity sensors, each of which corresponds to a number. After the head-mounted display device coordinate system F3 is determined, the coordinates of each light intensity sensor in the head-mounted display device coordinate system F3 can be determined.

[0154] In the embodiments of the present application, the pose of the head-mounted display device, i.e., the pose of the head-mounted display device coordinate system F3 relative to the environment coordinate system F0, can be represented by .

[0155] The positioning method provided by the embodiments of the present application will be described in detail below based on the scenario shown in Figure 6 , in combination with the head-mounted display device provided by the embodiments of the present application and the robot provided by the embodiments of the present application. Figure 3 Figure 4 The positioning method provided by the embodiments of the present application will be described in detail below based on the scenario shown in Figure 6 , in combination with the head-mounted display device provided by the embodiments of the present application and the robot provided by the embodiments of the present application.

[0156] Figure 7 The positioning method provided by the embodiments of the present application is exemplarily shown. The positioning method can include the following steps or all of the steps:

[0157] S101, the robot and the head-mounted display device establish a communication connection.

[0158] The communication connection established between the head-mounted display device and the robot can include but is not limited to: Wi-Fi P2P communication connection, Bluetooth communication connection, NFC connection, etc.

[0159] S102, the robot determines the pose of the robot in the indoor environment based on the three-dimensional map of the indoor environment.

[0160] Wherein, the pose of the robot in the indoor environment can be represented by , which is used to indicate the transformation relationship of the robot coordinate system F2 relative to the environment coordinate system F0. It should be understood that the origin of the robot coordinate system F2 is located on the robot, and the pose of the robot coordinate system F2 relative to the environment coordinate system F0 is equivalent to the pose of the robot in the environment coordinate system F0, so that can represent the positioning of the robot in the indoor environment.

[0161] In some embodiments, the robot can first acquire the three-dimensional map of the indoor environment, then detect the surrounding environment in real time based on the laser SLAM system, and finally obtain the pose of the robot in the indoor environment based on the comparison between the surrounding environment and the three-dimensional map of the indoor environment

[0162] The method that the robot acquires the three-dimensional map of the indoor space can be generated by robot exploration. For example, the robot is equipped with a laser radar, and the robot can move slowly in the indoor space. A two-dimensional map of the indoor space is generated by a laser SLAM system. Then, based on the two-dimensional map and point cloud data of the indoor space acquired by a structured light system, a three-dimensional map of the indoor space is generated. For another example, the robot can acquire a three-dimensional map of the indoor space by a structured light system. Specifically, the robot projects a target area by a projector, acquires an image by a camera, and then analyzes the image to acquire a three-dimensional map of the indoor space.

[0163] It should be noted that the robot can also acquire and save a three-dimensional map of the indoor space from other devices. The robot can also acquire a three-dimensional map of the indoor space by other methods, which are not limited herein. It can be understood that the robot can determine a drivable area of the indoor space based on the three-dimensional map of the indoor space, so as to achieve self-obstacle avoidance of the robot.

[0164] S103, the robot projects a binary coded image to the head-mounted display device.

[0165] The binary coded image is a binary image with a pattern distribution complying with an encoding rule. The encoding rule of the encoding pattern can be a binary code and a Gray code, etc.

[0166] In some embodiments, the robot can project two groups of binary coded images to the head-mounted display device by the projector at a preset frequency at a preset time interval. The pattern on the first group of images is a stripe in a first direction, and the pattern on the second group of images is a stripe in a second direction. The first direction and the second direction are perpendicular directions. Each stripe corresponds to a pixel position. Taking the first direction as the vertical direction and the second direction as the horizontal direction as an example, each vertical stripe corresponds to each column of pixels. The first horizontal stripe corresponds to each row of pixels. It should be noted that the order of each group of binary coded images is determined based on the encoding rule. The robot projects the binary coded images in the order. Details can be seen in the following embodiments.

[0167] The number N of the binary coded images is determined by the size of the binary coded image and the encoding method. For example, the encoding method of the binary coded image is Gray coding, the length of the binary coded image is height, and the width of the binary coded image is width. The number N of the binary coded image in the first direction or the second direction is greater than M, M = max(log2(width), log2(height)), where N is a positive integer. For another example, the encoding method of the binary coded image is sine. The number of the binary coded image is at least three.

[0168] Taking a 5-bit Gray code as an example, a 5-bit Gray code can encode 32 (2 5 = 32) pixel positions. Please refer to Figure 8, Figure 8 An example is shown of a set of binary encoded images with a pattern of stripes and a coding rule of five-bit Gray code.

[0169] Please see Figure 8 , Figure 8 There are five binary coded images, each divided into multiple vertical bars. Each bar represents a pixel location. A bar with a diagonal line indicates a 0, while a blank bar indicates a 1. Each pixel location corresponds to a Gray code. The five binary coded images each correspond to one bit of the five-bit Gray code. Based on the encoding of the same pixel location in the five binary coded images, the corresponding five-bit Gray code can be determined. For example, the third position has an x-coordinate of 3, so its Gray code is 00011. Therefore, the third position can be encoded as 0 in the first, second, and third binary coded images, 1 in the fourth, and 1 in the fifth binary coded image. It should be noted that... Figure 1 Taking only 8 pixel positions as an example, the binary encoded image corresponding to a five-bit Gray code can include 32 pixel positions.

[0170] Please see Figure 9 , Figure 9 These are two sets of binary encoded images provided in the embodiments of this application. The first set of images is... Figure 8 The binary encoded images shown have vertical stripes in the first group and horizontal stripes in the second group. For details on the second group, please refer to [link to relevant documentation]. Figure 8 The relevant descriptions are not repeated here.

[0171] In some embodiments, the robot may first send instruction information to the head-mounted display device, and then transmit a binary coded image to the head-mounted display device. The instruction information is used to instruct the head-mounted display device to sample the light intensity received by the light intensity sensor. The robot may send the instruction information to the head-mounted display device in the following two ways:

[0172] In one implementation, the robot can first project an image with a preset brightness onto the head-mounted display device. This image is used to indicate the light intensity received by the head-mounted display device's sampling light intensity sensor. Then, it projects a binary-coded image onto the head-mounted display device at a preset projection frequency. It should be noted that because the preset projection frequency is high, the head-mounted display device can be located in real time even if the user moves while wearing it.

[0173] In another implementation, the robot can send a start message to the head-mounted display device through the communication connection established in step S101, the start message being used to instruct the head-mounted display device to sample the light intensity received by the light intensity sensor, and then project the binary coded image to the head-mounted display device at a preset projection frequency.

[0174] In some embodiments, the light intensity sensors are arranged around the head-mounted display device, and when the robot is kept at a preset distance from the head-mounted display device, the head-mounted display device is in the projection range of the robot, and the robot can project the binary coded image on the head-mounted display device. For example, the robot can determine the position of the head-mounted display device through Bluetooth positioning, and keep a preset distance from the head-mounted display device through a tracking algorithm, so that the robot can project the binary coded image on the head-mounted display device. For another example, in an initial state, the user wears the head-mounted display device, and the head-mounted display device is in the working range of the projector and the camera on the robot; when the user walks, the robot keeps a target distance from the head-mounted display device through a tracking algorithm, so that the head-mounted display device is in the working range of the projector and the camera on the robot. It should be noted that the head-mounted display device can be as shown in Figure 3 As long as the robot is located around the head-mounted display device, the head-mounted display device can be photographed.

[0175] In other embodiments, the light intensity sensors are arranged at part of the position of the head-mounted display device, and then the robot can move to a position in front of the part of the position of the head-mounted display device and keep a preset distance from the head-mounted display device, so that the head-mounted display device is in the projection range of the robot.

[0176] The robot can track the head-mounted display device or track the user wearing the head-mounted display device. For example, the robot can recognize the user wearing the head-mounted display device, and then track the user to keep a preset distance from the head-mounted display device, so that the robot can project on the head-mounted display device.

[0177] Optionally, the robot can also determine the positional relationship between the robot and the human body based on the trained neural network model, and then the robot can move to the front of the user to ensure that the user is in the working range of the projector. For example, the robot can capture an image through the camera, input the image into the trained neural network model, and obtain the features of the user. For example, if the robot obtains all facial features of the user based on the image, it can be determined that the robot is located in front of the user.

[0178] In one implementation, the robot can determine the position of the head-mounted display device via Bluetooth positioning; then, the robot moves to a position at a preset distance from the head-mounted display device; then, the robot captures an image and identifies the positional relationship between the robot and the head-mounted display device based on the image recognition; based on the positional relationship, the robot moves to a target position relative to the head-mounted display device, such as in front of the head-mounted display device.

[0179] S104. When the robot projects a binary encoded image onto the head-mounted display device, the head-mounted display device samples the light intensity values ​​received by the light intensity sensors to obtain a light intensity value sequence corresponding to each light intensity sensor.

[0180] The head-mounted display device is equipped with L light intensity sensors, where L is a positive integer greater than or equal to 4.

[0181] In some embodiments, when a robot projects a binary coded image onto a head-mounted display device, the head-mounted display device can receive light intensity signals through light intensity sensors to obtain the light intensity value received by each light intensity sensor. Furthermore, when the projector determines that projection has begun, the head-mounted display device can sample the light intensity received by each light intensity sensor at a preset sampling frequency through a data acquisition unit (also called a data acquisition card) to obtain a sequence of light intensity values ​​for each light intensity sensor.

[0182] The preset sampling frequency is greater than or equal to twice the preset projection frequency. For example, the robot sends the preset projection frequency to the head-mounted display device through the communication connection established in step S101. Correspondingly, the head-mounted display device receives the preset projection frequency and determines the preset sampling frequency based on twice the preset projection frequency.

[0183] For example, a robot projects onto a head-mounted display device. Figure 8 The binary coded image shown is assumed to have the first light intensity sensor located at the position corresponding to the third position. That is, the first light intensity sensor is located at the position where the light intensity of the third position in the binary coded image is received. Then the light intensity signal received by the first light intensity sensor can be as follows: Figure 10 As shown. Figure 10 The light intensity value of the light intensity signal received by the first light intensity sensor is shown, where the horizontal axis represents time and the vertical axis represents the light intensity value. Figure 10 The time when the first light intensity sensor receives the first image is represented by the origin time 0, and T is the projection period corresponding to the preset projection frequency. Within the first period, the robot projects the first binary coded image onto the head-mounted display device. The grayscale value at the third position in the first binary coded image is 0, indicating a weak light intensity. Figure 10As shown, the light intensity value received by the first light intensity sensor in the second period and the third period is consistent with the light intensity value received in the first period; the robot projects a fourth binary coded image in the fourth period to the head-mounted display device, and the light intensity received by the first light intensity sensor is the third position of the fourth binary coded image, the gray value of the third position is 1, and the light intensity value is strong, as shown in the figure. Figure 10 As shown, the light intensity value received by the first light intensity sensor in the fifth period is consistent with the light intensity value received in the fourth period. It should be noted that due to the influence of unstable factors such as other light in the real environment, the light intensity value may change slightly within a period, so Figure 10 The light intensity value in a period can be a curve.

[0184] In some embodiments, the head-mounted display device can determine the start of projection of the projector through the indication information sent by the robot to start sampling the light intensity, or can start sampling when the change of light intensity is detected, or can determine the start of projection of the projector through other methods, which is not limited here. For example, the indication information can be an image with a preset brightness, and the head-mounted display device can sample the light intensity received by each light intensity sensor at a preset sampling frequency when receiving the image with a preset brightness, to obtain the light intensity sequence value of each light intensity sensor.

[0185] The head-mounted display device can pre-store or receive the number of images or time of a projection process of the robot sent by the robot, and then stop sampling based on the above parameters and send the sampled light intensity value sequence to the robot. For example, the head-mounted display device pre-stores the projection time of the robot for one projection, and then the head-mounted display device can start sampling the light intensity value based on the indication information, and end sampling when the sampling time is the projection time, to obtain the light intensity value sequence and send the light intensity value sequence to the robot.

[0186] S105, the head-mounted display device sends the light intensity value sequence corresponding to each light intensity sensor and the three-dimensional coordinates of each light intensity sensor in the head-mounted display device coordinate system F3 to the robot based on the above communication connection.

[0187] In one implementation, the head-mounted display device generates data corresponding to each of the L light intensity sensors to obtain L sets of data; and sends the L sets of data to the robot, wherein any one of the L sets of data includes an identifier of the light intensity sensor, a light intensity value sequence corresponding to the light intensity sensor, and three-dimensional coordinates of the light intensity sensor in the head-mounted display device coordinate system F3; the identifier of the light intensity sensor is used to distinguish the L light intensity sensors, and the identifier of the light intensity sensor can be a number, etc.

[0188] For example, the coordinates of each light intensity sensor in the head-mounted display device coordinate system F3 can be represented as {id, x, y, z}, where id is the number of the light intensity sensor, and x, y, and z are the three-dimensional coordinate values of the light intensity sensor in the head-mounted display device coordinate system F3.

[0189] In S106, the robot determines the coordinates of each light intensity sensor in the pixel coordinate system based on the above light intensity value sequence.

[0190] Specifically, the robot can process the light intensity value sequence corresponding to each of the L light intensity sensors to generate the coordinates of the L light intensity sensors in the pixel coordinate system. The coordinates of each optical sensor in the pixel coordinate system can be represented as (u, v), where u is the horizontal coordinate of the optical sensor, and v is the vertical coordinate of the optical sensor.

[0191] In some embodiments, the robot can divide the light intensity value sequence of each group of light intensity sensors sent by the head-mounted display into two groups of light intensity value subsequences based on the projection period; the robot can process each of the two groups of light intensity value subsequences of each light intensity sensor to obtain two codes of each light intensity sensor; and the horizontal coordinate and the vertical coordinate of each light intensity sensor in the pixel coordinate system are obtained based on the two codes of each light intensity sensor.

[0192] For example, the projection period is T, the robot projects a binary coded image in each projection period, the robot projects a first group of images as shown in FIG. 6A in the first five periods, and projects a second group of images as shown in FIG. 6B in the last five periods. Figure 9 Figure 9 For each optical sensor, the robot can determine the light intensity values in the first five periods as a first subsequence and the light intensity values in the last five periods as a second subsequence; obtain a first Gray code based on the first subsequence and a second Gray code based on the second subsequence; determine the binary code corresponding to the first Gray code as the horizontal coordinate u of the light intensity sensor, and determine the binary code corresponding to the second Gray code as the vertical coordinate v of the light intensity sensor to obtain the coordinates (u, v) of the light intensity sensor in the pixel coordinate system.

[0193] For example, the robot can determine a target light intensity value from each period of the subsequence, such as the average of the light intensity values in the subsequence or any light intensity value in the subsequence, to obtain five target light intensity values; normalize each target light intensity value to obtain five code values; and finally, sort the five code values in chronological order to obtain the first Gray code.

[0194] For example, the robot can normalize the target light intensity value to obtain a code value by the following formula. ​

[0195]

[0196] wherein I is any one of the five light intensity values, I max is the light intensity value with the maximum value among the five light intensity values, I min is the light intensity value with the minimum value among the five light intensity values, is the normalized light intensity value. If the head-mounted display device can determine the corresponding encoding value is 1; otherwise, it is determined that the corresponding encoding value is 1.

[0197] In an implementation, after obtaining the two Gray codes of each light intensity sensor, the robot can convert the two Gray codes into binary codes respectively to obtain the horizontal coordinate and the vertical coordinate of the light intensity sensor in the pixel coordinate system. For example, the robot obtains the first Gray code as 00011 and the second Gray code as 00111 through the above processing, and can calculate that the binary code corresponding to the first Gray code is 3 and the binary code corresponding to the second Gray code is 6, so the coordinate of the optical sensor in the pixel coordinate system is (3, 6).

[0198] It should be noted that in some embodiments, the coordinates of each light intensity sensor can be calculated by the head-mounted display device, and after calculating the coordinates of each light intensity sensor, the head-mounted display device sends the coordinates of each light intensity sensor to the robot, and the following step S106 is performed.

[0199] S107, the robot determines the pose of the head-mounted display device in the projector coordinate system F12 based on the coordinates of each light intensity sensor in the pixel coordinate system and its three-dimensional coordinates in the head-mounted display device coordinate system F3

[0200] In some embodiments, the robot can obtain the three-dimensional coordinates of the L light intensity sensors in the head-mounted display device coordinate system F3 sent by the head-mounted display device in step S105, and can obtain the coordinates of the L light intensity sensors in the pixel coordinate system in S106, so the robot can determine the coordinates of each light intensity sensor in the pixel coordinate system and its three-dimensional coordinates in the head-mounted display device coordinate system F3 based on the identification of the light intensity sensor; and for the coordinates of each light intensity sensor in the pixel coordinate system and the three-dimensional coordinates in the head-mounted display device coordinate system F3, an equation is generated to obtain L equations; solving the L equations can obtain the pose of the head-mounted display device in the projector coordinate system F12 is used to indicate the transformation relationship between the head-mounted display device coordinate system F3 and the projector coordinate system F12.

[0201] For example, the robot can calculate the pose of the head-mounted display device in the projector coordinate system F12 based on the coordinates of the light intensity sensor in the pixel coordinate system, the intrinsic parameters of the projector, and the three-dimensional coordinates of the light intensity sensor in the head-mounted display device coordinate system F3 through a PnP algorithm The PnP algorithm can include a bundle adjustment (BA) algorithm and a direct linear transform (DLT) algorithm, and the like, which are not limited herein.

[0202] The robot generates an equation for each light intensity sensor to obtain an equation set. The pose of the head-mounted display device in the projector coordinate system F12 can be obtained by solving the equation set.

[0203] The robot can generate the equation based on the following formula:

[0204]

[0205] where (u, v) is the coordinate of the light intensity sensor in the pixel coordinate system, (x, y, z) is the three-dimensional coordinate value of the light intensity sensor in the head-mounted display device coordinate system F3, the intrinsic parameters of the projector include focal length and center point coordinates, the focal length of the projector is f x and f x , the center coordinates are (c x , c y ), and n is a proportional value.

[0206] S108, the robot converts the pose of the head-mounted display device in the projector coordinate system F12 into the pose of the head-mounted display device in the camera coordinate system F11 based on the transformation relationship between the projector coordinate system F12 and the camera coordinate system F11

[0207] In one implementation, the robot stores the transformation relationship between the projector coordinate system F12 and the camera coordinate system F11 The robot obtains the pose of the head-mounted display device in the camera coordinate system F11 based on the following formula:

[0208]

[0209] It should be understood that is a parameter of the structured light system, which can be determined through machine vision software or other calibration methods after the positions of the camera and the projector are determined.

[0210] S109, the robot converts the pose of the head-mounted display device in the projector coordinate system F12 into the pose of the head-mounted display device in the camera coordinate system F11 based on the transformation relationship between the robot coordinate system F2 and the camera coordinate system F11 the robot coordinate system F2 and the camera coordinate system F11​ pose of the head-mounted display device in the camera coordinate system F11 The pose of the head-mounted display device in the environment coordinate system F0 is calculated.

[0211] In an implementation, the robot can obtain the pose of the robot in the indoor environment through step S102 Further, the robot can obtain the pose of the head-mounted display device in the environment coordinate system F0 based on the following formula:

[0212]

[0213] wherein, is the pose of the robot in the environment coordinate system F0; is the calibration relationship between the robot coordinate system F2 and the camera coordinate system F11; is the pose of the head-mounted display device in the camera coordinate system F11. It should be noted that the calibration relationship between the robot coordinate system F2 and the camera coordinate system F11 can be determined by a hand-eye calibration method or other calibration methods. After the structured light system is installed and fixed on the robot, the transformation relationship between the robot coordinate system F2 and the camera coordinate system F11 can be calibrated.

[0214] S110, the robot sends the pose of the head-mounted display device in the environment coordinate system F0 to the head-mounted display device based on the above-mentioned communication connection.

[0215] In some embodiments, part of the steps S107 to S109 can be performed by the head-mounted display device. For example, the head-mounted display device receives the pose of the head-mounted display device relative to the robot and the pose of the robot in the environment coordinate system sent by the robot; and the head-mounted display device determines the pose of the head-mounted display device in the environment coordinate system according to the pose of the robot in the environment coordinate system and the pose of the head-mounted display device relative to the robot.

[0216] S111, the head-mounted display device generates display content based on the pose of the head-mounted display device in the environment coordinate system F0 and the three-dimensional map.

[0217] In an implementation, the head-mounted display device determines a target field of view range in the three-dimensional map based on the pose of the head-mounted display device in the environment coordinate system F0, and generates an image of the target field of view range in combination with existing media resources. The target field of view range is the range displayed when the user wears the head-mounted display device.

[0218] S112, the head-mounted display device displays the display content.

[0219] Users wearing the head-mounted display can see the aforementioned content. For example, based on the head-mounted display's position indoors, if it determines that there are no obstacles to the left of the user and obstacles to the right, the generated display can show the right-side obstacle as a virtual obstacle and the left-side obstacle as a virtual passage. After seeing this display, the user can avoid the obstacles and move within a safe area. The virtual obstacles and virtual passages are media resources stored in the head-mounted display's media library.

[0220] In other embodiments, the robot generates display content based on the pose of the head-mounted display device in the environmental coordinate system F0 and a 3D map, and then the robot sends the display content to the head-mounted display device.

[0221] In some embodiments, after step S107, the robot can determine the pose of the head-mounted display device in the robot coordinate system based on the pose of the head-mounted display device in the projector coordinate system and the transformation relationship between the robot coordinate system and the projector coordinate system; furthermore, based on the pose of the head-mounted display device in the robot coordinate system and the robot's pose indoors, the pose of the head-mounted display device in the environmental coordinate system F0 is determined. The transformation relationship between the robot coordinate system and the projector coordinate system can be pre-calibrated. For example, the transformation relationship between the robot coordinate system and the camera coordinate system, and the transformation relationship between the camera coordinate system and the projector coordinate system, can be calibrated using the camera in steps S108 and S109, thereby determining the transformation relationship between the robot coordinate system and the projector coordinate system. It is understood that after determining the transformation relationship between the robot coordinate system and the projector coordinate system, in some embodiments, the robot may not need to be equipped with a camera.

[0222] refer to Figure 11 , Figure 11 A schematic diagram of the structure of a head-mounted display device 300 provided in an embodiment of this application is shown. Figure 11 As shown, the head-mounted display device 300 may include: a processor 301, a memory 302, a communication module 303, a sensor module 304, a camera 305, a display device 306, and an audio device 307. These components can be coupled together and communicate with each other. Understandably, Figure 11 The structure shown does not constitute a specific limitation on the head-mounted display device 300.

[0223] In other embodiments of this application, the head-mounted display device 300 may include more or fewer components than illustrated. For example, the head-mounted display device 300 may also include physical buttons such as a power button, volume buttons, a USB port, etc.

[0224] The processor 301 can include one or more processing units, for example: the processor 301 can include an AP, a modem processor, a GPU, an ISP, a controller, a video codec, a DSP, a baseband processor, and / or an NPU, etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors. The controller can generate operation control signals according to instruction operation codes and timing signals, complete the control of fetching instructions and executing instructions, so that various components perform corresponding functions, such as human-computer interaction, motion tracking / prediction, rendering display, audio processing, etc.

[0225] The memory 302 stores executable program codes for executing the interaction method in the virtual reality scene provided by the embodiments of the present application, and the executable program codes include instructions. The memory 302 can include a program storage area and a data storage area. The communication module 303 can include a mobile communication module and a wireless communication module. Among them, the mobile communication module can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the head-mounted display device 300. The wireless communication module can provide a solution including WLAN, BT, GNSS, FM, IR, etc. Wireless communication solutions applied to the head-mounted display device 300. The communication module 303 can support the head-mounted display device 300 and the electronic device to communicate. The sensor module 304 is used to collect the motion state data of the user wearing the head-mounted display device 300. The sensor module 304 can include an accelerometer, a compass, a gyroscope, a magnetometer, or other sensors for detecting motion, etc.

[0226] In the embodiments of the present application, the memory 302 can store the three-dimensional coordinate values of each light intensity sensor on the head-mounted display device on the head-mounted display device.

[0227] In some embodiments, the sensor module 304 can be an inertial measurement unit (IMU) disposed in the head-mounted display device 300. The sensor module 304 can be used to obtain motion data of the user's head, such as head position information, displacement, speed, shaking, turning, etc. The sensor module 304 can also include an optical sensor for tracking the user's eye position in conjunction with the camera 305 and capturing eye movement data. For example, it can be used to determine the user's interpupillary distance, interocular distance, 3D position of each eye relative to the head-mounted display device 300, amplitude of each eye's twist and rotation (i.e. turning, pitching and shaking), and gaze direction, etc. The camera 305 can be used to capture still images or videos. The still images or videos can be images or videos facing outward of the user's surroundings, or can be images or videos facing inward.

[0228] In embodiments of the present application, the sensor module 304 can include at least four light intensity sensors.

[0229] The camera 305 can track the movement of the user's single or both eyes. The camera 305 includes, but is not limited to, a traditional color camera (RGB camera), a depth camera (RGB depth camera), a dynamic vision sensor (DVS) camera, etc.

[0230] The head-mounted display device 300 presents or displays the VR scene through a GPU, a display device 306, and an application processor, etc. The GPU is a microprocessor for image processing, which connects the display device 306 and the application processor. The processor 301 can include one or more GPUs, which execute program instructions to generate or change display information. The GPU is used to perform mathematical and geometric calculations according to the data obtained from the electronic device, to render a 3D virtual scene using computer graphics technology, computer simulation technology, etc., to provide content for display on the display device 306. The GPU is also used to add correction or pre-distortion to the rendering process of the virtual scene to compensate or correct the distortion caused by the optical device. The GPU can also adjust the content provided to the display device 306 based on the data from the sensor module 304. For example, the GPU can add depth-of-field information in the content provided to the display device 306 based on the 3D position of the user's eyes, the pupil distance, etc. In some embodiments of the present application, the display device 306 is used to receive the content provided by the GPU of the head-mounted display device 300, and to present or display the VR scene according to the content. In other embodiments of the present application, the display device 306 is used to receive the data or content processed by the electronic device (e.g., the data rendered by the electronic device), and to present the VR scene according to the data or content. In some embodiments, the display device 306 can present corresponding images for the user's left and right eyes, respectively, thereby simulating binocular vision.

[0231] In embodiments of the present application, the head-mounted display device 300 can receive the display content sent from the robot, and present the VR scene according to the data or content.

[0232] In some embodiments, the display device 306 can include a display screen and a cooperating optical device. The display screen can include a display panel that can be used to display virtual images to present a stereoscopic virtual scene to a user. The display panel can be an LCD, OLED, AMOLED, FLED, Miniled, Micro Led, Micro-oLed, QLED, etc. The number of display screens can be one or multiple. The optical device can include one or more optical elements, such as a Fresnel lens, a convex lens, a concave lens, a filter, etc. The optical device is used to direct light from the display screen to an exit pupil for a user to perceive. In some implementations, one or more optical elements in the optical device can have one or more coatings, such as an anti-reflective coating. Magnification of the image light by the optical device allows the display screen to be physically smaller, lighter, and consume less power. In addition, magnification of the image light can increase the field of view of the content displayed by the display screen. For example, the optical device can make the field of view of the content displayed by the display screen the full field of view of the user. The optical device can also be used to correct one or more optical errors. Examples of optical errors include barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, coma, field curvature, astigmatism, etc. In some implementations, the content provided to the display screen for display is pre-distorted, and the optical device corrects the distortion when receiving the image light generated based on the content from the display screen. In other embodiments, the display device 306 can include a projection device for projecting an optical signal (e.g., a light beam) directly onto a user’s retina. The projection device can be a projector. The projection device can receive content provided by a GPU, encode the content onto an optical signal, and project the encoded optical signal onto a user’s retina so that the user perceives a stereoscopic VR scene. The number of projection devices can be one or multiple. The audio device 307 is used to implement audio capture and output. The audio device 307 can include, but is not limited to, a microphone, a speaker, a headphone, etc.

[0233] In some embodiments, Figure 2BThe system shown can also include a handheld device. The handheld device can be wirelessly connected and communicate with the electronic device through BT, NFC, ZigBee, and other short-distance transmission technologies, and can also be wired connected and communicate through a USB interface, an HDMI interface, or a self-defined interface, etc. The handheld device can be implemented in the form of a handle, a mouse, a keyboard, a stylus, a bracelet, and the like. The handheld device can be configured with various sensors, such as an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, and the like. The pressure sensor can be arranged under a confirmation button of the handheld device. The confirmation button can be a physical button or a virtual button. The sensors of the handheld device are used to collect corresponding data, such as acceleration of the handheld device collected by the acceleration sensor, motion speed of the handheld device collected by the gyroscope sensor, and the like. The handheld device can send the data collected by the various sensors to the electronic device for analysis. The electronic device can determine the motion condition and the state of the handheld device according to the data collected by the various sensors in the handheld device. The motion condition of the handheld device can include, but is not limited to, whether to move, a moving direction, a moving speed, a moving distance, a moving track, and the like. The state of the handheld device can include whether the confirmation button of the handheld device is pressed. The electronic device can adjust the image displayed on the head-mounted display device 300 and / or start a corresponding function according to the motion condition and / or the state of the handheld device, such as moving a cursor in the image, and the moving track of the cursor is determined by the motion condition of the handheld device.

[0234] The embodiments of the present application also provide an electronic device, which comprises one or more processors and one or more memories; wherein the one or more memories are coupled with the one or more processors, and the one or more memories are configured to store computer program codes, the computer program codes comprising computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method described in the above embodiments.

[0235] The embodiments of the present application also provide a computer program product comprising instructions, which, when executed on an electronic device, cause the electronic device to perform the method described in the above embodiments.

[0236] The embodiments of the present application also provide a computer-readable storage medium comprising instructions, which, when executed on an electronic device, cause the electronic device to perform the method described in the above embodiments.

[0237] It can be understood that the embodiments of the present application can be combined in any manner to achieve different technical effects.

[0238] In the above embodiments, all or part of the processes can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the processes can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes described in the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc.

[0239] Those of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0240] In summary, the above only describes the embodiments of the technical scheme of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present application shall be included in the protection scope of the present application.

Claims

1. A positioning method, characterized by, The method is applied to a positioning device, the positioning device comprises a projector, and the method comprises: The positioning device tracks a virtual reality device, and the positioning device can follow the virtual reality device to move; The positioning device projects a coded image through the projector; The positioning device receives a plurality of light intensity values from the virtual reality device, wherein the plurality of light intensity values comprise light intensities of the coded image received by a plurality of light intensity sensors of the virtual reality device respectively; The positioning device determines a pose of the virtual reality device relative to the positioning device according to the projected coded image, the plurality of light intensity values, and positions of the plurality of light intensity sensors in the virtual reality device, wherein the pose of the virtual reality device relative to the positioning device and a pose of the positioning device in an environment coordinate system are used to determine a pose of the virtual reality device in the environment coordinate system, and the pose of the virtual reality device in the environment coordinate system is used to determine display content of the virtual reality device.

2. The method of claim 1, wherein, The method further comprises: The positioning device determines the pose of the virtual reality device in the environment coordinate system according to the pose of the positioning device in the environment coordinate system and the pose of the virtual reality device relative to the positioning device; The positioning device sends the pose of the virtual reality device in the environment coordinate system to the virtual reality device.

3. The method according to claim 1 or 2, characterized in that, The coded image comprises a plurality of coded images, and the positioning device determines the pose of the virtual reality device relative to the positioning device according to the projected coded image, the plurality of light intensity values, and the positions of the plurality of light intensity sensors in the virtual reality device, comprising: The positioning device generates a code of one light intensity sensor based on the light intensity value of each coded image received by the light intensity sensor; The positioning device determines pixel coordinates of the plurality of light intensity sensors based on the codes of the plurality of light intensity sensors; The positioning device determines the pose of the virtual reality device relative to the projector based on the pixel coordinates of the plurality of light intensity sensors and the positions of the plurality of light intensity sensors in the virtual reality device; The positioning device converts the pose of the virtual reality device relative to the projector into the pose of the virtual reality device relative to the positioning device based on the pose of the projector relative to the positioning device.

4. The method of claim 3, wherein, The plurality of coded images comprise M first images and N second images, M is a positive integer, N is a positive integer, the first image is a binary image with a first direction of stripes as a pattern, and the second image is a binary image with a second direction of stripes as a pattern; The positioning device generates a code of one light intensity sensor based on the light intensity value of each coded image received by the light intensity sensor, comprising: The positioning device generates a first code of the light intensity sensor based on the light intensity value of the first image received by the light intensity sensor; The positioning device generates a second code of the light intensity sensor based on the light intensity value of the second image received by the light intensity sensor; The positioning device determines the pixel coordinates of the plurality of light intensity sensors based on the codes of the plurality of light intensity sensors, including: the positioning device determines the first coordinates of the light intensity sensor in the first direction based on the first code of the light intensity sensor; the positioning device determines the second coordinates of the light intensity sensor in the second direction based on the second code of the light intensity sensor, and the pixel coordinates of the light intensity sensor include the first coordinates of the light intensity sensor and the second coordinates of the light intensity sensor.

5. The method of claim 4, wherein, The method further comprises: The positioning device determines the pose of the projector relative to the positioning device based on the pose of the projector relative to the camera and the pose of the positioning device relative to the camera.

6. The method of claim 1 or 2, wherein, The method further comprises: The positioning device determines the display content based on the pose of the virtual reality device in the environment coordinate system, the three-dimensional map and the media resource. The positioning device sends the display content to the virtual reality device to make the virtual reality device display the display content.

7. The method according to claim 1 or 2, characterized in that, Before the positioning device projects the coded image through the projector, the method comprises: The positioning device sends the indication information to the virtual reality device, and the indication information is used to instruct the virtual reality device to sample the light intensity received by the plurality of light intensity sensors to obtain the plurality of light intensity values.

8. The method of claim 1 or 2, wherein, The positioning device tracks the virtual reality device, including: The positioning device locates the position of the virtual reality device; The positioning device moves to a position with a preset distance from the virtual reality device; The positioning device determines the relative position of the user wearing the virtual reality device and the positioning device through the photographed image; The positioning device moves to the direction in which the face of the user faces based on the relative position.

9. A positioning method characterized by, Applied to a virtual reality device, the virtual reality device includes a plurality of light intensity sensors, and the method comprises: The plurality of light intensity sensors respectively receive the light intensity of the coded image projected by the positioning device to obtain a plurality of light intensity values, the positioning device tracks the virtual reality device, and the positioning device can move with the virtual reality device; The virtual reality device sends the plurality of light intensity values to the positioning device; the plurality of light intensity values, the coded image and the positions of the plurality of light intensity sensors in the virtual reality device are used for the positioning device to determine the pose of the virtual reality device relative to the positioning device; the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system are used to determine the pose of the virtual reality device in the environment coordinate system, and the pose of the virtual reality device in the environment coordinate system is used to determine the display content of the virtual reality device.

10. The method of claim 9, wherein, The method further comprises: The virtual reality device receives the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system from the positioning device; The virtual reality device determines the pose of the virtual reality device in the environment coordinate system based on the pose of the positioning device in the environment coordinate system and the pose of the virtual reality device relative to the positioning device.

11. The method according to claim 9 or 10, characterized in that, The method further comprises: The virtual reality device determines the display content based on the pose of the virtual reality device in the environment coordinate system, the three-dimensional map, and the media resource. The virtual reality device displays the display content.

12. The method of claim 9 or 10, wherein, The method comprises: When the virtual reality device receives the indication information sent by the positioning device, the virtual reality device samples the light intensity received by the plurality of light intensity sensors to obtain the plurality of light intensity values.

13. An electronic device, comprising: The electronic device comprises one or more processors and one or more memories; wherein the one or more memories are coupled with the one or more processors, and the one or more memories are configured to store computer program codes, the computer program codes comprising computer instructions, when the one or more processors execute the computer instructions, causing the electronic device to perform the method according to any one of claims 1-12.

14. A computer program product comprising instructions, characterized in that, When the computer program product runs on the electronic device, causing the electronic device to perform the method according to any one of claims 1-12.

15. A computer-readable storage medium comprising instructions, wherein: When the instructions run on the electronic device, causing the electronic device to perform the method according to any one of claims 1-12.

16. A positioning system, characterized by The positioning system comprises a first electronic device and a second electronic device, the first electronic device is configured to perform the method according to any one of claims 1-8, and the second electronic device is configured to perform the method according to any one of claims 9-12.

Images