Relocation method of security area and head-mounted display

Abstract

The invention relates to the technical field of relocation, provides a relocation method of a safety area and a head-mounted display, and aims to improve the number of feature points, adopt a plurality of cameras for relocation, improve the success rate of relocation and solve the problem of time consumption of relocation precision and feature matching in a relocation stage. According to the method, the historical hash table is queried to obtain at least one historical feature point with the same descriptor segment as the current feature point for matching, so that the feature point matching efficiency is effectively improved, and the time for repositioning the security region is shortened. According to the method, a plurality of matching point pairs are generated from the historical map points of the matched historical feature points in the historical map and the current feature points, and the relative pose of the current head-mounted display in the historical map can be calculated, so that the position of the safe area in the world coordinate system where the current head-mounted display is located is quickly obtained, and rendering display is performed; the waiting time of the user is shortened, and the user experience is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of repositioning, and provides a method for repositioning a safe area and a head mounted display. Background Art

[0002] When turning on AR / VR head-mounted displays, users are generally asked to mark a safe area to prevent them from colliding with other objects during movement. However, it is very cumbersome if users are required to re-mark the safe area every time they turn on the display. Therefore, if the user does not choose to re-mark the safe area, the marked safe area should be at the last marked position every time the head-mounted display is turned on, which requires the head-mounted display to have safe area relocation technology.

[0003] At present, the relevant technology mainly uses the relocation technology based on feature point matching to determine the safe area. This technology is very dependent on the number of feature points in the scene. The more the number, the higher the accuracy of relocation. Therefore, this technology usually expands the visual range by increasing the number of cameras on the head-mounted display, so as to obtain more feature points to improve the accuracy of relocation. However, as the number of cameras increases, the time used for feature matching will also increase, which will reduce the speed of relocation of the safe area and increase the waiting time of users.

[0004] Therefore, balancing the time and accuracy of safe area relocalization is an urgent problem to be solved in head-mounted displays. Summary of the invention

[0005] The embodiments of the present application provide a method for relocating a safe area and a head-mounted display, which are used to improve the accuracy and efficiency of relocation.

[0006] On the one hand, an embodiment of the present application provides a method for relocating a safe area, which is applied to a head mounted display, comprising:

[0007] When the machine is powered on, it loads the pre-generated historical map and historical hash table, and obtains the current environmental images collected by multiple cameras;

[0008] For each environment image, extract the current feature points in the environment image, and calculate the descriptor of each current feature point;

[0009] For each current feature point, segment the descriptor of the current feature point, and according to each segment, search the historical map point in the historical map from the historical hash table that has the same segment as the descriptor of the current feature point, to obtain a matching point pair;

[0010] Calculating the relative position of the current head mounted display in the historical map according to each pair of matching points;

[0011] According to the relative position and posture, a safety area corresponding to the current head mounted display is rendered and displayed.

[0012] On the other hand, an embodiment of the present application provides a head-mounted display, comprising a processor, a memory, a display, and a plurality of cameras, wherein the cameras, the display, the memory, and the processor are connected via a bus;

[0013] The camera is used to collect images;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Loading the pre-generated historical map and historical hash table when starting up, and obtaining the environment images currently collected by the multiple cameras;

[0016] For each environment image, extract the current feature points in the environment image, and calculate the descriptor of each current feature point;

[0017] For each current feature point, segment the descriptor of the current feature point, and according to each segment, search the historical map point in the historical map from the historical hash table that has the same segment as the descriptor of the current feature point, to obtain a matching point pair;

[0018] Calculating the relative position of the current head mounted display in the historical map according to each pair of matching points;

[0019] According to the relative posture, a safety area corresponding to the current head mounted display is rendered and displayed through the display.

[0020] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a method for relocating a security area provided in an embodiment of the present application.

[0021] In a method for relocating a safe area and a head-mounted display provided by an embodiment of the present application, multiple cameras are used for relocation, which increases the number of feature points and thus improves the success rate of relocation. At the same time, in order to balance the time-consuming problem of relocation accuracy and feature matching, in the relocation stage, by querying the historical hash table, at least one historical feature point with the same descriptor segment as the current feature point is obtained, and the Hamming distance between the two feature points is calculated for matching. Compared with simple brute force matching, the time consumption is reduced by about two-thirds, which effectively improves the efficiency of feature point matching, thereby reducing the time for relocating the safe area. By generating multiple matching point pairs from the historical map points in the historical map and the current feature points, the relative position of the current head-mounted display in the historical map can be calculated, so as to quickly obtain the position of the safe area in the world coordinate system where the current head-mounted display is located, and render and display it, which reduces the user's waiting time and effectively improves the user experience.

[0022] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A schematic diagram of a safety area provided in an embodiment of the present application;

[0025] Figure 2 A flow chart of the method for relocalization based on feature points and bag-of-words model;

[0026] Figure 3 A flowchart of a method for establishing a historical map provided in an embodiment of the present application;

[0027] Figure 4 A structural diagram of a head mounted display provided in an embodiment of the present application;

[0028] Figure 5 The historical hash representation provided for the embodiment of the present application is intended;

[0029] Figure 6 A schematic diagram of the triangulation principle provided in an embodiment of the present application;

[0030] Figure 7 A flow chart of a method for relocating a safe area provided in an embodiment of the present application;

[0031] Figure 8 A flow chart of a 2D-3D point matching method provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram of searching for historical feature points provided in an embodiment of the present application;

[0033] Fig.10 A structural diagram of a head-mounted display provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0035] When a head-mounted display such as AR / VR is started, if a safe area has been established the last time it was turned on, the historically marked safe area will usually be displayed if the user does not choose to re-mark the safe area. This requires the head-mounted display to have the function of saving historical maps (usually the map established the last time it was turned on). Only in this way can the current position of the head-mounted display on the historical map be accurately calculated through repositioning technology when it is turned on next time, so as to render the safe area on the head-mounted display.

[0036] like Figure 1 As shown, it is a schematic diagram of the safety zone. During the startup phase, the head-mounted display uses the repositioning technology to calculate the accurate position of the current head-mounted display in the historical map, thereby rendering a marked cylindrical safety zone on the head-mounted display. When the user moves within the safety zone while using the head-mounted display, collisions with real objects can be avoided.

[0037] In relocalization technology, the success rate and speed (i.e. time) of relocalization are very important indicators, because the speed and success rate of relocalization greatly affect the user experience. In the same scene, users always want to immediately display the safe area in the head-mounted display. Usually, the relocalization time does not exceed 2s.

[0038] At present, the commonly used safe area relocation method is based on feature points and bag-of-words models (such as the ORB-SLAM algorithm, the VINS-Mono algorithm, etc.), which converts the descriptor of each local feature point in the image captured by the camera into a word through a pre-loaded bag-of-words dictionary tree. Since the dictionary contains all words, a bag-of-words vector is obtained by counting the words of the entire image. The distance between the bag-of-words vectors represents the difference between the two images. In this way, during the relocation process, the inverted index method is used to find the historical frames with the same words as the current frame, and the similarity between the current frame and the historical frame is calculated based on the bag-of-words vectors. The historical frame with the highest similarity is matched with the current frame for feature points, and the current position of the head-mounted display in the historical map is calculated, so as to relocate the current safe area.

[0039] See also Figure 2 , is a flow chart of the method for relocalization based on feature points and bag-of-words model, which mainly includes the following steps:

[0040] S201: Obtain an image sample set, extract feature points from the image samples, and calculate descriptors of the feature points.

[0041] S202: Generate a bag-of-words dictionary based on the descriptors of the feature points.

[0042] S203: Determine the words contained in a frame of image according to the bag-of-words dictionary to obtain a bag-of-words vector.

[0043] The bag-of-words vector may be a multi-dimensional vector, where each dimension represents a word. When the dimension value is 1, it indicates that the image frame contains the word, and when the dimension value is 0, it indicates that the image frame does not contain the word.

[0044] S204: Find historical frames that have the same words as the current frame, and calculate the similarity between the current frame and the historical frame based on the bag-of-words vectors of the two.

[0045] S205: Using the RANSAC algorithm, brute force matching is performed between the descriptor of the feature point corresponding to the 3D point on the historical frame with the highest similarity and the descriptor of the 2D feature point in the current frame.

[0046] S206: Using the PnP algorithm, according to the matched 3D-2D point pairs, the posture of the head mounted display corresponding to the current frame in the historical map is calculated, and the safe area is rendered and displayed according to the posture.

[0047] It can be seen that the relocalization method based on feature matching and bag-of-words model relies on the preloaded bag-of-words dictionary. If the generated bag-of-words dictionary is not suitable for the current usage scenario, it is highly likely that no historical frames similar to the current frame will be found. In addition, the PnP algorithm generally only selects the current frame of a camera to calculate the pose of the camera. This will result in the 3D-2D point pairs matched by other cameras not being utilized. Too few matching point pairs may cause the PnP algorithm to fail or pose calculation errors, reducing the success rate of relocalization.

[0048] In view of this, an embodiment of the present application provides a method for relocating a safe area. The method does not rely on a bag-of-words model and does not require the calculation of similar historical frames. A historical hash table is pre-established through descriptor segmentation of historical feature points in historical frames. In this way, when relocating the safe area, by searching the historical hash table, historical feature points with the same segmentation as the descriptor of the current feature point are obtained, and the found historical feature points are matched with the current feature points. Compared with simple brute force matching, the efficiency of feature point matching is effectively improved, thereby reducing the time for relocating the safe area.

[0049] The method for relocating the safety zone provided in the embodiment of the present application is mainly divided into two stages: mapping and relocation. The mapping stage generally occurs when the head-mounted display is turned on for the first time. After the user customizes the safety zone, the head-mounted display uses historical images collected by multiple cameras over a period of time to build a map, and automatically saves the established historical map after shutdown. At the same time, in the process of building the historical map, the historical hash table is established based on the descriptors of the historical feature points in the historical images.

[0050] The process of creating a historical map Figure 3 As shown, it mainly includes the following steps:

[0051] S301: After the device is started, historical images collected continuously and synchronously by multiple cameras are obtained.

[0052] like Figure 4 As shown, it is a structural diagram of a head-mounted display, which includes four cameras facing in different directions to expand the field of view, so as to synchronously capture historical images containing different textures to increase the number of feature points.

[0053] S302: For each frame of the historical image, extract the pixel coordinates and descriptors of the historical feature points in the historical image.

[0054] In an example, historical feature points are extracted for each frame of historical images captured by multiple cameras, and the extraction methods include but are not limited to ORB feature extraction, SIFT feature extraction, and SURF feature extraction.

[0055] Among them, the feature point consists of two parts: pixel coordinates and descriptors. Pixel coordinates refer to the position of the feature point in the image. The descriptor is used to describe the information of the pixel points in the preset neighborhood of the feature point according to specific rules. It is usually represented by a multidimensional vector. In principle, feature points with similar appearance have similar descriptors. Therefore, when the descriptors of two feature points are close in the vector space (such as the Hamming distance), the two feature points may be the same feature points, that is, they correspond to the same 3D point in the three-dimensional space.

[0056] S303: Divide the descriptor of each historical feature point into several segments, generate a historical hash table and save it.

[0057] Since the descriptor is a multi-dimensional vector, it can be evenly divided into multiple segments, each containing the same bits. For example, each segment contains 3 bits. Figure 5 The following is an illustration of the historical hash representation.

[0058] S304: Match the historical feature points in the historical images captured by every two cameras, and use a triangulation method to determine the world coordinates of the historical feature points in the corresponding historical feature point pairs in the three-dimensional space based on the pixel coordinates of each matched historical feature point pair, and obtain the historical map points.

[0059] In one example, descriptors of historical feature points extracted from historical images synchronously acquired by every two cameras are matched to obtain multiple pairs of historical feature points, and the epipolar geometry is used to calculate the essential matrix based on the multiple pairs of historical feature points to obtain the relative pose (including rotation matrix and translation vector) between the two cameras. Then, a triangulation method is used to calculate the world coordinates of the corresponding 3D points in the three-dimensional space based on the pixel coordinates and relative pose of each matched pair of historical feature points to obtain multiple historical map points.

[0060] like Figure 6 As shown in FIG. 1 , it is a schematic diagram of the triangulation principle. The world coordinates of the same 3D point P in three-dimensional space can be solved through the pixel coordinates p1 and p2 in the historical images captured by two cameras, where O1 and O2 are the optical centers of the two cameras.

[0061] S305: Generate and save a historical map based on the world coordinates of each historical map point.

[0062] According to the world coordinates of each historical map point, a historical force map with marked safety areas is generated, and the historical force map is saved when the device is shut down. Each historical map point contains the world coordinates and the descriptor information of its corresponding historical feature point.

[0063] Based on the historical map and historical hash table of the marked security area, the security area is relocated at the next startup. The relocation process is as follows: Figure 7As shown, it mainly includes the following steps:

[0064] S701: Load the pre-generated historical map and historical hash table when starting up, and obtain the environment images currently collected by multiple cameras.

[0065] After the head mounted display device is powered on, the pre-generated historical map and historical hash table are loaded, the descriptor segments of the historical map points and their corresponding historical feature points are obtained, and the environment images currently collected by multiple cameras are obtained.

[0066] S702: For each environment image, extract the current feature points in the environment image, and calculate the descriptor of each current feature point.

[0067] For each environment image currently captured by multiple cameras, the feature point extraction method used in constructing the historical map is adopted to extract the current feature points in the environment image and obtain the descriptor of the current feature points.

[0068] S703: For each current feature point, segment the descriptor of the current feature point, and according to each segment, find the historical map point in the historical map that has the same segment as the descriptor of the current feature point from the historical hash table to obtain a matching point pair.

[0069] In the relocation phase, for each current feature point, if all historical feature points corresponding to the historical map points in the historical map are traversed for matching, it will consume a long matching time, thereby reducing the efficiency of creating the safe area and affecting the user experience.

[0070] In order to solve this problem, the embodiment of the present application uses a pre-generated historical hash table to perform a multi-index hash (MIH) search to accelerate the feature point matching process, thereby quickly and accurately re-determining the safe area. Specifically, the descriptor of each current feature point is divided into several segments, and then the historical hash table is searched according to each segment to match the feature points with the same descriptor segment. For a specific implementation method, see Figure 8 , mainly includes the following steps:

[0071] S7031: For each segment, search the historical hash table for at least one historical feature point that has the same segment as the descriptor of the current feature point.

[0072] In one example, after the descriptor of each current feature point is divided into several segments, when the number of segments reaches a certain value, the same descriptor segment must exist in the historical hash table, that is, at least one segment of the descriptors of two similar feature points is the same.

[0073] Take a current feature point as an example, Fig. 9As shown, it is assumed that the descriptor is divided into 4 segments, the first segment is 001, the second segment is 101, the third segment is 011, and the fourth segment is 000. Use the first segment 001 to search the historical hash table, obtain the historical feature points {A, B, C, D} of the 001 segment, and use the second segment 101 to search the historical hash table, obtain the historical feature points {A, E, F} of the 101 segment, and use the third segment 011 to search the historical hash table, obtain the historical feature points {B, C, G} of the 011 segment, and use the fourth segment 000 to search the historical hash table, obtain the historical feature points {B, E, F} of the 000 segment.

[0074] Among them, the historical feature points found in different segments may be the same. In order to avoid repeated matching, the repeated historical feature points can be eliminated.

[0075] For example, still taking the previous current feature as an example, at least one historical feature point with the same segment as it is found is {A, B, C, D, E, F, G}.

[0076] S7032: Calculate the Hamming distance between at least one historical feature point corresponding to each segment and the current feature point.

[0077] For each current feature point, after obtaining at least one historical feature point having the same segmentation as the current feature point based on multiple segments of the descriptor of the current feature point, a historical feature point matching the current feature point can be selected from the historical feature point descriptors. Specifically, since the descriptor of the feature point is usually represented by a vector, the Hamming distance between the descriptor of the current feature point and the descriptors of each historical feature point can be calculated in the vector space, so that a historical feature point matching the current feature point can be selected based on each Hamming distance.

[0078] S7033: Use the historical feature points corresponding to the Hamming distances less than the preset distance threshold as the historical feature points that match the current feature point.

[0079] For each current feature point, the Hamming distance between it and each historical feature point found is compared with the preset distance threshold. When the Hamming distance between two descriptors is less than the preset distance threshold, it indicates that the two feature points may be the same feature point, that is, the match is successful. Therefore, the historical feature point corresponding to the Hamming distance less than the preset distance threshold is used as the historical feature point matching the current feature point.

[0080] It should be noted that, when there are multiple Hamming distances that are smaller than the preset distance threshold, the historical feature point corresponding to the minimum Hamming distance is used as the historical feature point that matches the current feature point.

[0081] S7034: The historical map point corresponding to the matched historical feature point in the historical map and the current feature point are taken as a matching point pair.

[0082] After obtaining the historical feature point that matches the current feature point, obtain the historical map point corresponding to the matched historical feature point in the historical map, and combine it with the current feature point to obtain a matching point pair.

[0083] During the mapping phase, a set of 3D point clouds P = {P1, P2, P3, ...} of historical map points are obtained through the multiple cameras of the head-mounted display. After the computer is shut down and restarted, the world coordinate system after the restart may be different from the world coordinate system during the mapping, and a posture transformation is required, that is, the rotation matrix and translation vector of the current head-mounted display in the world coordinate system of the historical map are calculated. Therefore, during the relocalization phase, after a set of 2D point clouds Q = {Q1, Q2, Q3, ...} of current feature points are obtained through the multiple cameras of the head-mounted display, the points in the 2D point cloud Q are matched one by one with the points in the 3D point cloud P through matching.

[0084] S704: Calculate the relative position of the current head mounted display in the historical map according to each pair of matching points.

[0085] After traversing all the current feature points, multiple matching point pairs can be obtained. When the number of matching point pairs is greater than the preset threshold, the repositioning requirement is met, so that the relative position of the current head-mounted display in the historical map can be accurately calculated based on each matching point pair.

[0086] In one example, since each matching point pair includes a 3D historical map point and a 2D current feature point, a PnP algorithm may be used to calculate the relative position of the current head mounted display in the historical map.

[0087] Specifically, according to the world coordinates of the historical map points in each matching point pair and the pixel coordinates of the current feature point, combined with the pre-calibrated intrinsic parameters of the camera, the rotation matrix and translation vector of the current head-mounted display in the historical map are calculated, and the relative position and posture of the current head-mounted display in the historical map are obtained according to the rotation matrix and translation vector. The formula is as follows:

[0088]

[0089] Among them, [u, v] is the pixel coordinate of the current feature point, [X, Y, Z] represents the world coordinate of the historical map point, K represents the intrinsic parameters of the camera, and R, t are the rotation matrix and translation vector to be solved. Through multiple matching point pairs, equations containing multiple unknowns can be constructed to solve R and t.

[0090] S705: Render and display a safety area corresponding to the current head mounted display according to the relative posture.

[0091] By calculating the relative position of the current head mounted display in the history map, the position of the marked safe area in the world coordinate system where the current head mounted display is located can be obtained, so that the safe area is rendered and displayed at this position.

[0092] In actual applications, the real environment where the user is located may change significantly, and the marked safe area may not be able to adapt to the current environment, and the safe area may need to be re-marked. Therefore, in some embodiments, when the head mounted display is turned on, if the instruction information for re-creating the safe area is received, the newly created safe area is rendered and displayed according to the instruction information.

[0093] In the method for relocating the safety area provided in the embodiment of the present application, in order to increase the number of feature point matches, multiple cameras are used for relocation, thereby improving the success rate of relocation. As the number of cameras increases, the time for feature point matching will also increase. In order to balance the time-consuming problem between relocation accuracy and feature matching, when creating a historical map, the method generates a historical hash table according to the segmentation of the descriptors of the historical feature points. In this way, in the relocation stage, by querying the historical hash table, at least one historical feature point with the same descriptor segment as the current feature point is obtained, and matching is performed by calculating the Hamming distance between the two feature points. Compared with simple brute force matching, the time consumption is reduced by about two-thirds, effectively improving the efficiency of feature point matching, thereby reducing the time for relocating the safety area; by taking the matched historical feature point from the historical map as a 3D point matching the current feature point, the PnP algorithm can be used to calculate the relative position of the current head-mounted display in the historical map, thereby quickly obtaining the position of the safety area in the world coordinate system where the current head-mounted display is located, and rendering and displaying it, thereby reducing the user's waiting time and effectively improving the user experience.

[0094] Based on the same technical concept, an embodiment of the present application provides a head-mounted display, which includes but is not limited to VR devices, AR devices, MR devices, etc., and can implement the steps of the above-mentioned safe area repositioning method and achieve the same technical effect.

[0095] See also Fig.10 , the head mounted display comprises a processor 1001, a memory 1002, a display 1003 and a plurality of cameras 1004, the cameras 1004, the display 1003, the memory 1002 and the processor 1001 are connected via a bus 1005;

[0096] The camera 1004 is used to collect images;

[0097] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0098] When the computer is powered on, a pre-generated history map and history hash table are loaded, and the environment images currently collected by the multiple cameras 1004 are obtained;

[0099] For each environment image, extract the current feature points in the environment image and calculate the descriptor of each current feature point;

[0100] For each current feature point, the descriptor of the current feature point is segmented, and according to each segment, the historical map point with the same segment as the descriptor of the current feature point in the historical map is searched from the historical hash table to obtain a matching point pair;

[0101] According to each pair of matching points, the relative position of the current head mounted display in the historical map is calculated;

[0102] According to the relative posture, the safety area corresponding to the current head mounted display is rendered and displayed through the display 1003.

[0103] Optionally, the processor 1001 searches the historical map points in the historical map for the descriptor of the current feature point having the same segment as the segment from the historical hash table according to each segment, and obtains a matching point pair. The specific operation is:

[0104] For each segment, find at least one historical feature point with the same segment as the descriptor of the current feature point from the historical hash table;

[0105] Calculate the Hamming distance between at least one historical feature point corresponding to each segment and the current feature point;

[0106] The historical feature points corresponding to the Hamming distances less than the preset distance threshold are regarded as the historical feature points matching the current feature point;

[0107] The historical map point corresponding to the matched historical feature point in the historical map and the current feature point are taken as a matching point pair.

[0108] Optionally, the processor 1001 calculates the relative position of the current head mounted display in the historical map according to each matching point pair, and the specific operation is:

[0109] According to the world coordinates of the historical map points in each matching point pair and the pixel coordinates of the current feature point, combined with the pre-calibrated intrinsic parameters of the camera, the rotation matrix and translation vector of the current head mounted display in the historical map are calculated;

[0110] According to the rotation matrix and translation vector, the relative position of the current head mounted display in the historical map is obtained.

[0111] Optionally, after the head mounted display is turned on, the processor 1001 further executes:

[0112] If instruction information for re-creating the safety area is received, the newly created safety area is rendered and displayed according to the instruction information.

[0113] Optionally, the processor 1001 generates the historical map in the following manner:

[0114] Acquire historical images collected continuously and synchronously by multiple cameras;

[0115] For each frame of historical image, extract the pixel coordinates and descriptors of historical feature points in the historical image;

[0116] Divide the descriptor of each historical feature point into several segments, generate and save a historical hash table;

[0117] Match the historical feature points in the historical images collected by every two cameras, and use the triangulation method to determine the world coordinates of the historical feature points in the corresponding historical feature point pairs in the three-dimensional space according to the pixel coordinates of each matched historical feature point pair, and obtain the historical map points;

[0118] Generate and save historical maps based on the world coordinates of each historical map point.

[0119] The memory 1002 may be a volatile memory, such as a random-access memory (RAM); the memory 1002 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1002 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1002 may be a combination of the above memories;

[0120] The processor 1001 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0121] It should be noted that Fig.10It is only an example to provide the necessary hardware for the head mounted display to execute the steps of a method for relocating a safe area provided in an embodiment of the present application. If not shown, the head mounted display may also include conventional hardware such as a speaker, a pickup, a communication interface, and a power supply.

[0122] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a method for relocating a security area in the aforementioned embodiment.

[0123] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a method for relocating a security area in the aforementioned embodiment.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for relocating a safe area, characterized in that: Applied to a head mounted display, the method comprises: When the machine is powered on, it loads the pre-generated historical map and historical hash table, and obtains the current environmental images collected by multiple cameras; For each environment image, extract the current feature points in the environment image, and calculate the descriptor of each current feature point; For each current feature point, segment the descriptor of the current feature point, and according to each segment, search the historical map point in the historical map from the historical hash table that has the same segment as the descriptor of the current feature point, to obtain a matching point pair; Calculating the relative position of the current head mounted display in the historical map according to each pair of matching points; According to the relative position and posture, a safety area corresponding to the current head mounted display is rendered and displayed.

2. The method according to claim 1, characterized in that The step of searching, according to each segment, from the historical hash table for a historical map point in the historical map having the same segment as the descriptor of the current feature point, to obtain a matching point pair comprises: For each segment, searching from the historical hash table for at least one historical feature point having the same segment as the descriptor of the current feature point; Calculating the Hamming distance between at least one historical feature point corresponding to each of the segments and the current feature point; Taking the historical feature points corresponding to the Hamming distances less than the preset distance threshold as the historical feature points matching the current feature point; The historical map point corresponding to the matched historical feature point in the historical map and the current feature point are taken as a matching point pair.

3. The method according to claim 1, characterized in that The step of calculating the relative position of the current head mounted display in the historical map according to each pair of matching points includes: Calculate the rotation matrix and translation vector of the current head mounted display in the historical map according to the world coordinates of the historical map points in each matching point pair and the pixel coordinates of the current feature point in combination with the pre-calibrated intrinsic parameters of the camera; According to the rotation matrix and the translation vector, the relative position and posture of the current head mounted display in the historical map is obtained.

4. The method according to any one of claims 1 to 3, characterized in that The historical map is pre-generated in the following way: Acquire historical images collected continuously and synchronously by multiple cameras; For each frame of historical image, extract the pixel coordinates and descriptors of historical feature points in the historical image; Divide the descriptor of each historical feature point into several segments, generate and save a historical hash table; Match the historical feature points in the historical images collected by every two cameras, and use the triangulation method to determine the world coordinates of the historical feature points in the corresponding historical feature point pairs in the three-dimensional space according to the pixel coordinates of each matched historical feature point pair, and obtain the historical map points; Generate and save historical maps based on the world coordinates of each historical map point.

5. The method according to any one of claims 1 to 3, characterized in that After the head mounted display is turned on, the method further includes: If instruction information for re-creating the safety area is received, the newly created safety area is rendered and displayed according to the instruction information.

6. A head mounted display, characterized in that: It includes a processor, a memory, a display and a plurality of cameras, wherein the cameras, the display, the memory and the processor are connected via a bus; The camera is used to collect images; The memory stores a computer program, and the processor performs the following operations according to the computer program: Loading the pre-generated historical map and historical hash table when starting up, and obtaining the environment images currently collected by the multiple cameras; For each environment image, extract the current feature points in the environment image, and calculate the descriptor of each current feature point; For each current feature point, segment the descriptor of the current feature point, and according to each segment, search the historical map point in the historical map from the historical hash table that has the same segment as the descriptor of the current feature point, to obtain a matching point pair; Calculating the relative position of the current head mounted display in the historical map according to each pair of matching points; According to the relative posture, a safety area corresponding to the current head mounted display is rendered and displayed through the display.

7. The head mounted display according to claim 6, wherein: The processor searches the historical map point in the historical map that has the same segment as the descriptor of the current feature point from the historical hash table according to each segment, and obtains a matching point pair. The specific operation is: For each segment, searching from the historical hash table for at least one historical feature point having the same segment as the descriptor of the current feature point; Calculating the Hamming distance between at least one historical feature point corresponding to each of the segments and the current feature point; Taking the historical feature points corresponding to the Hamming distances less than the preset distance threshold as the historical feature points matching the current feature point; The historical map point corresponding to the matched historical feature point in the historical map and the current feature point are taken as a matching point pair.

8. The head mounted display according to claim 6, wherein: The processor calculates the relative position of the current head mounted display in the historical map according to each matching point pair, and the specific operation is: Calculate the rotation matrix and translation vector of the current head mounted display in the historical map according to the world coordinates of the historical map points in each matching point pair and the pixel coordinates of the current feature point in combination with the pre-calibrated intrinsic parameters of the camera; According to the rotation matrix and the translation vector, the relative position and posture of the current head mounted display in the historical map is obtained.

9. The head mounted display according to any one of claims 6 to 8, characterized in that: The processor generates a historical map by: Acquire historical images collected continuously and synchronously by multiple cameras; For each frame of historical image, extract the pixel coordinates and descriptors of historical feature points in the historical image; Divide the descriptor of each historical feature point into several segments, generate and save a historical hash table; Match the historical feature points in the historical images collected by every two cameras, and use the triangulation method to determine the world coordinates of the historical feature points in the corresponding historical feature point pairs in the three-dimensional space according to the pixel coordinates of each matched historical feature point pair, and obtain the historical map points; Generate and save historical maps based on the world coordinates of each historical map point.

10. The head mounted display according to any one of claims 6 to 8, characterized in that: After the head mounted display is turned on, the processor further executes: If instruction information for re-creating the safety area is received, the newly created safety area is rendered and displayed according to the instruction information.

Images