Large space positioning method and system based on feature extraction, head-mounted display device and medium
Feature extraction and matching are performed by the camera of the headset device, the problem of positioning error in large spaces is solved, precise positioning in large spaces is achieved, and the accuracy of displaying virtual content is improved.
Patent Information
- Application Number
- CN202411045545.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-01
AI Technical Summary
In large spaces, the spatial positioning technology errors of head-mounted display devices accumulate, resulting in large deviations in the display and design positions of virtual content, affecting the user experience.
The camera of the headset device obtains the shooting screen, extracts features and matches the preset spatial features, determines the target feature area and relative position, and calculates the coordinates of the headset device to achieve accurate positioning.
The positioning accuracy of the head-mounted display device in large space is improved, the virtual content display deviation caused by positioning errors is avoided, and the user experience is enhanced.
Smart Images

Figure CN118887293A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of head-mounted display devices, and in particular to a large-space positioning method, system, head-mounted display device and medium based on feature extraction. Background Art
[0002] The technologies used in head-mounted display devices (i.e., head-mounted display devices) mainly include AR (Augmented Reality) technology, VR (Virtual Reality) technology and MR (Mixed Reality) technology. Based on these technologies, when users wear head-mounted display devices, they can display virtual content in real scenes through various sensors, cameras or optical components in the device.
[0003] When displaying virtual content through a head-mounted display device in a real scene, the space needs to be positioned. When the user is in a small space (less than three meters), the spatial positioning technology of the head-mounted display device can usually achieve a good positioning effect. However, when the user is in a large space (greater than ten meters) such as a factory or exhibition hall, the error of the current spatial technology will accumulate to more than one meter. The display of virtual content will deviate greatly from the actual position intended to be displayed in the design, affecting the user experience.
[0004] In summary, how to improve the positioning accuracy of head-mounted display devices in a large space and avoid display deviations of virtual content caused by positioning errors in a large space has become a technical problem that urgently needs to be solved in this field.
[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0006] The main purpose of this application is to provide a large-space positioning method, system, head-mounted display device and medium based on feature extraction, aiming to solve the technical problem of improving the positioning accuracy of head-mounted display devices in large spaces.
[0007] To achieve the above purpose, the present application proposes a large space positioning method based on feature extraction, the method comprising:
[0008] Acquire a captured image through a camera of a head-mounted display device, and perform feature extraction on the captured image to obtain image features;
[0009] Searching for a target feature area matching the picture feature in the preset spatial features;
[0010] Acquire a first relative position of the image feature relative to the head mounted display device;
[0011] A first coordinate of the target feature area relative to a preset origin is obtained, and a second coordinate of the head-mounted display device relative to the preset origin is determined according to the first relative position and the first coordinate, so as to position the head-mounted display device.
[0012] In one embodiment, the step of extracting features from the captured image to obtain image features includes:
[0013] Converting the captured image into a grayscale image, and detecting corner points and edges of the captured image by comparing the grayscale values of each pixel in the grayscale image;
[0014] Extracting feature points in the captured image using a preset feature point extraction algorithm;
[0015] The corner points, the edges and the feature points are used as picture features of the captured picture.
[0016] In one embodiment, after the step of obtaining a first relative position of the picture feature relative to the head mounted display device, the method further includes:
[0017] Determining a preset second relative position of the virtual content relative to the target feature area;
[0018] Determining a third relative position of the virtual content relative to the head mounted display device according to the first relative position and the second relative position;
[0019] The virtual content is displayed in the head display device according to the third relative position.
[0020] In one embodiment, the step of obtaining a first relative position of the picture feature relative to the head mounted display device comprises:
[0021] Taking any feature point in the picture features as a target feature point;
[0022] Acquire a first relative position of the target feature point through an inertial sensor in the head display device;
[0023] Acquire a first relative distance of the target feature point by a distance measurement module of the head display device;
[0024] The target relative position of the target feature point relative to the head-mounted display device is determined according to the first relative orientation and the first relative distance, and the target relative position is used as the first relative position of the picture feature relative to the head-mounted display device.
[0025] In one embodiment, the first relative position includes: the first relative orientation and the first relative distance;
[0026] The step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate comprises:
[0027] Calculating the coordinate difference of the picture feature relative to the head display device in each orientation axis direction according to the first relative orientation and the first relative distance;
[0028] Subtracting the coordinate differences from the first coordinate obtains a second coordinate of the head-mounted display device relative to the preset origin.
[0029] In one embodiment, after the step of extracting features from the captured image to obtain image features, the method further includes:
[0030] When the head display device has a second coordinate, acquiring the camera orientation of the camera through an inertial sensor and an accelerometer of the head display device;
[0031] Determine a shooting area corresponding to the camera orientation in a preset space according to the second coordinates and the camera orientation;
[0032] The shooting area is used as a spatial feature for feature matching of the shooting picture.
[0033] In one embodiment, after the step of determining a second coordinate of the head mounted display device relative to the preset origin according to the first relative position and the first coordinate, the method further includes:
[0034] Acquire the motion posture of the head display device through the inertial sensor;
[0035] Acquire the movement direction and movement distance of the head display device through the accelerometer;
[0036] The second coordinate of the head display device is updated in real time according to the motion posture, the motion direction, the motion distance and the second coordinate to perform real-time positioning of the head display device.
[0037] In addition, to achieve the above purpose, the present application also proposes a large space positioning system based on feature extraction, the system comprising:
[0038] A feature extraction module is used to obtain a captured image through a camera of a head display device, and perform feature extraction on the captured image to obtain image features;
[0039] A feature matching module, used to search for a target feature area matching the picture feature in a preset spatial feature;
[0040] A relative position detection module, used to obtain a first relative position of the image feature relative to the head display device;
[0041] A positioning module is used to obtain a first coordinate of the target feature area relative to a preset origin, and determine a second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate, so as to position the head display device.
[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a head-mounted display device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large-space positioning method based on feature extraction as described above.
[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large-space positioning method based on feature extraction as described above are implemented.
[0044] The present application provides a large-space positioning method based on feature extraction. The present application obtains a captured image in real time through a camera of a head-mounted display device, extracts features from the captured image to obtain image features of the captured image, then compares the image features with preset spatial features to find a matching target feature area, and then obtains a first relative position of the image features relative to the head-mounted display device and a first coordinate of the target feature area relative to an origin. Since the image features match the target feature area, a second coordinate of the head-mounted display device relative to the origin can be obtained based on the first relative position and the first coordinate to position the head-mounted display device.
[0045] In summary, the present application stores the features of the space in which the head-mounted display device is located in advance and records the position of each feature in the space. After the head-mounted display device is turned on and captures the captured image through the camera, the features in the captured image are used to determine which area in the space the captured image corresponds to. Then, the image can be positioned according to the position of the area. The above method is not limited by the size of the space. It only needs to store the features in the space to enable the head-mounted display device to be accurately positioned in a space of any size, thereby improving the positioning accuracy of the head-mounted display device in a large space, thereby avoiding the display deviation of the virtual content caused by the positioning error in the large space. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0048] Figure 1 A schematic diagram of a flow chart provided for an embodiment of a large space positioning method based on feature extraction of the present application;
[0049] Figure 2 This is a schematic diagram of a scene involved in the first embodiment of the large space positioning method based on feature extraction of the present application;
[0050] Figure 3 A flowchart diagram of the third embodiment of the large space positioning method based on feature extraction provided in this application;
[0051] Figure 4 A schematic diagram of a flow chart provided for an embodiment of a large space positioning method based on feature extraction of the present application;
[0052] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the large space positioning method based on feature extraction in the embodiment of the present application.
[0053] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0055] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0056] The main solution of the embodiment of the present application is: acquiring a captured image through a camera of a head-mounted display device, and performing feature extraction on the captured image to obtain image features; searching for a target feature area matching the image features in preset spatial features; acquiring a first relative position of the image features relative to the head-mounted display device; acquiring a first coordinate of the target feature area relative to a preset origin, and determining a second coordinate of the head-mounted display device relative to the preset origin based on the first relative position and the first coordinate, so as to position the head-mounted display device.
[0057] In this embodiment, for the convenience of description, the following description is made with the head mounted display device as the execution subject.
[0058] The technologies used in head-mounted display devices mainly include AR (Augmented Reality) technology, VR (Virtual Reality) technology and MR (Mixed Reality) technology. Based on these technologies, when users wear head-mounted display devices, they can display virtual content in real scenes through various sensors, cameras or optical components in the device.
[0059] When displaying virtual content through a head-mounted display device in a real scene, the space needs to be positioned. When the user is in a small space (less than three meters), the spatial positioning technology of the head-mounted display device can usually achieve a good positioning effect. However, when the user is in a large space (greater than ten meters) such as a factory or exhibition hall, the error of the current spatial technology will accumulate to more than one meter. The display of virtual content will deviate greatly from the actual position intended to be displayed in the design, affecting the user experience.
[0060] In summary, how to improve the positioning accuracy of head-mounted display devices in a large space and avoid display deviations of virtual content caused by positioning errors in a large space has become a technical problem that urgently needs to be solved in this field.
[0061] In response to the above problems, the present application provides a large-space positioning method based on feature extraction. The present application obtains a captured picture in real time through the camera of a head-mounted display device, extracts features from the captured picture to obtain picture features of the captured picture, and then compares the picture features with preset spatial features to find a matching target feature area, and then obtains a first relative position of the picture features relative to the head-mounted display device and a first coordinate of the target feature area relative to the origin. Since the picture features match the target feature area, the second coordinate of the head-mounted display device relative to the origin can be obtained based on the first relative position and the first coordinate to position the head-mounted display device.
[0062] In summary, the present application stores the features of the space in which the head-mounted display device is located in advance and records the position of each feature in the space. After the head-mounted display device is turned on and captures the captured image through the camera, the features in the captured image are used to determine which area in the space the captured image corresponds to. Then, the image can be positioned according to the position of the area. The above method is not limited by the size of the space. It only needs to store the features in the space to enable the head-mounted display device to be accurately positioned in a space of any size, thereby improving the positioning accuracy of the head-mounted display device in a large space, thereby avoiding the display deviation of the virtual content caused by the positioning error in the large space.
[0063] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a head display device, etc. The head display device is taken as an example to illustrate this embodiment and the following embodiments.
[0064] Based on this, the embodiment of the present application provides a large space positioning method based on feature extraction, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the large space positioning method based on feature extraction of the present application.
[0065] In this embodiment, the large space positioning method based on feature extraction includes steps S10 to S40:
[0066] Step S10, obtaining a captured image through a camera of the head display device, and performing feature extraction on the captured image to obtain image features;
[0067] It should be noted that in this embodiment, the head display device is a wearable display technology device designed to project visual content directly in front of the user's eyes to create an immersive visual experience. This device is widely used in the fields of VR, AR and MR, and provides users with a unique interaction method and content display platform by simulating or enhancing the real world seen by users.
[0068] It should be noted that in this embodiment, all private data used is obtained with the user's knowledge and authorization.
[0069] In this embodiment, after the head display device is turned on, it obtains the captured images around the user, i.e., the wearer, through the camera in real time, and then applies computer vision technology, such as edge detection, corner detection, etc., to the captured images to identify unique visual features. These features can be the outline of an object, texture pattern, or specific markers (such as AR markers).
[0070] In addition, the camera does not need to collect additional images around the wearer, but can directly use the user's visual image collected by the camera as the shooting image for feature matching, which can reduce the operating frequency of the camera.
[0071] Further, in a feasible implementation manner, the step of extracting features from the captured image to obtain image features in the above step S10 may include steps S101 to S103:
[0072] Step S101, converting the captured image into a grayscale image, and detecting corner points and edges of the captured image by comparing the grayscale values of each pixel in the grayscale image;
[0073] In this embodiment, the captured image is first converted into a grayscale image. The purpose of converting a color image into a grayscale image is to simplify the processing process and reduce the amount of calculation. The grayscale image only contains brightness information, but not color information. Then, by comparing the grayscale values of each pixel in the grayscale image, the corners and edges in the image can be detected. Common corner detection algorithms include Harris corner detection, Shi-Tomasi corner detection, FAST feature point, etc. FAST feature point is a fast corner detection algorithm that detects corners by comparing the grayscale values of pixels; edge detection algorithms include Canny edge detection, Sobel operator, etc.
[0074] Grayscale images are easier to process than color images, thus reducing the computational complexity of subsequent processing steps. Corners and edges are usually relatively stable features in images, which can help the system recognize the same object or scene from different perspectives.
[0075] Step S102, extracting feature points in the captured image using a preset feature point extraction algorithm;
[0076] In this embodiment, feature points in the captured image are also extracted by a preset feature point extraction algorithm. These points usually have good repeatability and stability and can be accurately detected even under different viewing angles or lighting conditions. Common feature point extraction algorithms include but are not limited to SIFT (Scale Invariant Feature Transform), SURF (Speeded Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.
[0077] Feature point extraction algorithms usually have good noise resistance and robustness to changes in viewing angles, and the extracted feature points usually have a certain degree of uniqueness, which can help the system better identify specific objects or scenes.
[0078] Step S103: taking the corner points, the edges and the feature points as picture features of the captured picture.
[0079] In this embodiment, the corner points, edges and feature points extracted above can be combined to obtain a complete set of picture features. Combining multiple types of features can provide more comprehensive information, which helps to improve the accuracy of recognition. Different types of features may perform differently in different scenarios. Combining them can complement each other and enhance the robustness of the system.
[0080] Step S20, searching for a target feature area matching the picture feature in the preset spatial features;
[0081] It should be noted that, in this embodiment, the spatial feature refers to a pre-collected and stored environmental feature database.
[0082] The spatial feature may be a collection of all features in the space where the user wears the head-mounted display device, or may be a collection of features in a certain area, which is not limited here.
[0083] Before positioning, it is necessary to identify and store the various features of the area in the head-mounted display device. The space can be three-dimensionally modeled based on all the objects in the space, and then the three-dimensional features can be stored. When matching the features, they are converted into two-dimensional plane features for matching.
[0084] In this embodiment, the features in the current image are compared with the pre-stored spatial features to find the feature region with the highest matching degree.
[0085] Step S30, obtaining a first relative position of the image feature relative to the head mounted display device;
[0086] In this embodiment, the relative position relationship between the picture feature and the head-mounted display device is determined by a hardware device built into the head-mounted display device to obtain a first relative position of the picture feature relative to the head-mounted display device.
[0087] Furthermore, in a feasible implementation manner, after the above step S30, the method may further include steps A10 to A30:
[0088] Step A10, determining a preset second relative position of the virtual content relative to the target feature area;
[0089] In this embodiment, in augmented reality applications, virtual content usually needs to be placed at a preset position relative to a real-world reference object (here, the target feature area). This position is defined in advance and can vary depending on the application scenario.
[0090] For example, in an AR game, a user may want a virtual character to always appear at a certain distance directly above the target feature area. This positional relationship is the so-called "second relative position", which is pre-set by the developer and fixed in the program.
[0091] Step A20: determining a third relative position of the virtual content relative to the head display device according to the first relative position and the second relative position;
[0092] It should be noted that, in this embodiment, the first relative position refers to the positional relationship of the target feature area relative to the head-mounted display device, which has been determined in the above steps.
[0093] The second relative position refers to the position relationship of the virtual content relative to the target feature area.
[0094] In this embodiment, the system needs to combine these two relative positions to calculate the exact position of the virtual content relative to the head-mounted display device, for example, converting the position of the virtual content from the coordinate system of the target feature area to the coordinate system of the head-mounted display device. This final position is called the "third relative position."
[0095] Step A30: displaying the virtual content in the head display device according to the third relative position.
[0096] In this embodiment, once the third relative position of the virtual content with respect to the head display device is calculated, the virtual content can be displayed in the head display device. The system renders the virtual content into the user's field of view so that it appears to exist at a certain location in the real world.
[0097] Furthermore, in a feasible implementation manner, the above step S30 may further include steps S301 to S305:
[0098] Step S301, taking any feature point in the picture features as a target feature point;
[0099] In this embodiment, since the picture feature includes multiple feature points, it is easy to cause confusion during positioning. Therefore, any feature point in the picture feature can be used as a target feature point to assist positioning.
[0100] Step S302, acquiring a first relative position of the target feature point through an inertial sensor in the head display device;
[0101] It should be noted that, in this embodiment, the inertial sensor (Inertial Measurement Unit, IMU) is a type of sensor, usually including accelerometers and gyroscopes, etc., which are used to measure parameters such as speed, acceleration and rotation of an object. In a head-mounted display device, the IMU can help track the motion state of the device.
[0102] In this embodiment, the system obtains the orientation information of the target feature point relative to the head-mounted display device through the inertial sensor in the head-mounted display device, where the "first relative orientation" refers to the direction of the target feature point relative to the head-mounted display device, including pitch angle, yaw angle, roll angle, etc.
[0103] Step S303, obtaining a first relative distance of the target feature point through a distance measurement module of the head display device;
[0104] It should be noted that, in this embodiment, the ranging module is used to measure the first relative distance of the target feature point relative to the head-mounted display device. The ranging module generally includes one or more sensors, and the ranging module can communicate with the control chip and camera in the head-mounted display device.
[0105] The distance measurement module usually implements distance measurement based on different physical principles and technologies. Common technologies include:
[0106] Ultrasonic sensor: Utilizes the propagation characteristics of sound waves to calculate distance by sending sound wave pulses and receiving echoes; suitable for short-distance measurement, and can be used in smaller spaces as a ranging module for positioning head-mounted display devices.
[0107] Laser sensor: uses a laser beam for distance measurement and determines the distance by calculating the time difference between the laser emission and the reflected signal; it is used in situations requiring high precision and is suitable for positioning in large spaces, and can improve the positioning accuracy of head-mounted display devices.
[0108] Infrared sensor: measures distance by emitting infrared light and receiving reflected signals; suitable for short-distance and low-cost applications, and also suitable for smaller spaces, saving the positioning cost of head-mounted display devices.
[0109] Radar sensor: including microwave radar and millimeter wave radar, which measures distance by emitting electromagnetic waves and receiving reflected signals; suitable for the detection of long-distance and high-speed moving objects, suitable for scenarios where users are in a large space and are always on the move, and can measure the first relative distance of target feature points in real time.
[0110] Time of Flight (ToF) sensor: emits continuous or pulsed light signals and measures distance by calculating the time it takes for light to be emitted and returned; suitable for accurate measurement of short to medium distances, and can be used as a sensor for ranging modules in scenes of moderate size and where users do not move frequently.
[0111] Different types of sensors have certain advantages and limitations. In practical applications, one or more of the above sensors can be used in combination as a ranging module to measure the first relative position of the target feature point according to actual conditions.
[0112] In addition, in order to improve the measurement accuracy of the ranging module, the ranging module can be installed on a movable gimbal on the head-mounted display device, so that the measurement range of the ranging module can cover the shooting range of the camera.
[0113] In this embodiment, after the target feature point is recognized by the camera, the system controls the ranging module to measure the distance of the target feature point according to the image recognition result of the camera, and the measured distance is the first relative distance of the target feature point relative to the head display device.
[0114] Step S304: determine a target relative position of the target feature point relative to the head-mounted display device according to the first relative orientation and the first relative distance, and use the target relative position as a first relative position of the picture feature relative to the head-mounted display device.
[0115] In this embodiment, the spatial position of the target feature point relative to the head-mounted display device can be determined by combining the information of the first relative orientation and the first relative distance obtained previously.
[0116] Step S40: Acquire a first coordinate of the target feature area relative to a preset origin, and determine a second coordinate of the head-mounted display device relative to the preset origin according to the first relative position and the first coordinate, so as to position the head-mounted display device.
[0117] It should be noted that, in this embodiment, the preset origin refers to a fixed reference point, which is usually a known position set in a real environment. The origin can be used as a reference point of the positioning system.
[0118] In this embodiment, the system first determines the coordinates of the target feature area relative to the preset origin (i.e., the first coordinates), and then combines the first relative position to calculate the position of the head-mounted display device relative to the preset origin (i.e., the second coordinates).
[0119] In this embodiment, the present application stores the features of the space in which the head display device is located in advance and records the location of each feature in the space. After the head display device is turned on and captures the shooting picture through the camera, the features in the shooting picture are used to determine which area in the space the shooting picture corresponds to, and then the positioning can be performed according to the position of the area. The above method is not limited by the size of the space. It only needs to store the features in the space to enable the head display device to be accurately positioned in a space of any size, thereby improving the positioning accuracy of the head display device in a large space, thereby avoiding the display deviation of the virtual content caused by the positioning error in the large space.
[0120] Further, in a feasible implementation manner, the first relative position includes: the first relative orientation and the first relative distance; the step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate in the above step S40 may include steps S401 to S402:
[0121] Step S401, calculating the coordinate difference of the picture feature relative to the head display device in each orientation axis direction according to the first relative orientation and the first relative distance;
[0122] In this embodiment, the system needs to use the first relative orientation (including pitch angle, yaw angle and roll angle) and the first relative distance of the target feature point to calculate the coordinate difference of the picture feature relative to the head display device. The coordinate difference refers to the displacement of the picture feature relative to the head display device on three coordinate axes (usually X, Y, and Z axes).
[0123] Specifically, the system uses the first relative orientation and the first relative distance to calculate the displacement values of the image features in the directions of each coordinate axis through trigonometric functions and geometric relationships. These displacement values are coordinate differences.
[0124] Step S402: Subtract the coordinate differences from the first coordinate to obtain a second coordinate of the head display device relative to the preset origin.
[0125] In this embodiment, after calculating the coordinate differences, the system subtracts these coordinate differences from the first coordinates of the target feature point to obtain the second coordinates of the head-mounted display device relative to the preset origin.
[0126] Specifically, assuming that the coordinates of the image feature are (x1, y1, z1) and the coordinate difference is (Δx, Δy, Δz), then the coordinates of the head display device relative to the preset origin are (x h ,y h , z h ), the calculation formula is as follows:
[0127] x h =x1-Δx;
[0128] y h =y1-Δy;
[0129] z h =z1-Δz;
[0130] The second coordinate of the head display device can be obtained by calculating according to the above formula.
[0131] For example, in order to help understand the implementation process of the large space positioning method based on feature extraction obtained by combining this embodiment with the above-mentioned embodiment 1, an example is given in combination with an actual scenario. Specifically, please refer to Figure 2 , Figure 2 This is a schematic diagram of a scene involved in the first embodiment of the large space positioning method based on feature extraction of this application, such as Figure 2As shown, when the user wears the head-mounted display device in a large space, the camera of the head-mounted display device takes a picture to obtain a captured image. The dotted box in the figure is the content captured by the camera, and then the image features in the captured image are identified. For example, items such as cabinets placed on the wall in the figure will make the image features of this area different from other areas, so it can be identified, and the image features are matched with the pre-set spatial features. After matching to the corresponding target feature area, the position of the head-mounted display device can be calculated according to the positioning of the target feature area, and the position of the head-mounted display device can be located.
[0132] When the user wears the head-mounted display device and walks to other locations, and other picture features appear in the captured image of the head-mounted display device, positioning is performed again based on the other picture features to complete the positioning calibration.
[0133] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, after the above step S10, the method can also include steps B10 to B30:
[0134] Step B10, when the head display device has a second coordinate, obtaining the camera orientation of the camera through an inertial sensor and an accelerometer of the head display device;
[0135] Step B20, determining a shooting area corresponding to the camera orientation in a preset space according to the second coordinates and the camera orientation;
[0136] Step B30: Using the shooting area as a spatial feature for feature matching of the shooting picture.
[0137] In this embodiment, only when the head display device is turned on for the first time does it need to compare the image features with all the features in the space. If the head display device currently has a second coordinate, the inertial sensor and accelerometer of the head display device can be used to estimate the current orientation of the head display device, thereby estimating the area captured by the camera. It is only necessary to use the features of the area as spatial features for subsequent feature matching and positioning.
[0138] This method can greatly reduce the amount of positioning calculations and improve the efficiency of head-mounted display device positioning.
[0139] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and / or second embodiments can be referred to the above introduction, and will not be repeated in the following. Figure 3 , Figure 3 This is a schematic diagram of the implementation process involved in Example 3 of the large space positioning method based on feature extraction of this application, such as Figure 3As shown, after the above step S40, the method may further include steps C10 to C30:
[0140] Step C10, obtaining the motion posture of the head display device through the inertial sensor;
[0141] It should be noted that, in this embodiment, the inertial sensor generally includes components such as an accelerometer and a gyroscope, which can be used to measure the motion posture of the head-mounted display device.
[0142] In this embodiment, the motion posture of the head display device can be obtained through various parameters detected by the inertial sensor.
[0143] Step C20, obtaining the movement direction and movement distance of the head display device through the accelerometer;
[0144] It should be noted that, in this embodiment, the accelerometer can measure the linear acceleration of the head-mounted display device, including the acceleration caused by gravity. By continuously monitoring the acceleration data, the movement direction and moving distance of the head-mounted display device can be estimated.
[0145] In this embodiment, the system estimates the movement direction and movement distance of the head display device by analyzing the data of the accelerometer.
[0146] Step C30: updating the second coordinate of the head display device in real time according to the motion posture, the motion direction, the motion distance and the second coordinate, so as to perform real-time positioning of the head display device.
[0147] In this embodiment, the system combines the acquired motion posture, motion direction and motion distance information, as well as the previously determined second coordinates of the head-mounted display device relative to a preset origin, to update the position information of the head-mounted display device in real time.
[0148] Specifically, the system uses this data to calculate the new position of the headset to determine the new coordinates of the headset in three-dimensional space. This real-time update can be achieved using different algorithms, such as Kalman filters or complementary filters, which can help eliminate noise and improve the accuracy of position estimates.
[0149] In this embodiment, after the head-mounted display device is positioned through feature matching, the head-mounted display device can update the position of the head-mounted display device in subsequent movements through various built-in sensors.
[0150] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the large-space positioning method based on feature extraction of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0151] This application also provides a large space positioning system based on feature extraction, please refer to Figure 4 , the system comprising:
[0152] The feature extraction module 10 is used to obtain a captured image through a camera of the head display device, and extract features from the captured image to obtain image features;
[0153] A feature matching module 20 is used to search for a target feature area matching the picture feature in a preset spatial feature;
[0154] A relative position detection module 30, used to obtain a first relative position of the image feature relative to the head display device;
[0155] The positioning module 40 is used to obtain the first coordinates of the target feature area relative to the preset origin, and determine the second coordinates of the head display device relative to the preset origin according to the first relative position and the first coordinates, so as to position the head display device.
[0156] Optionally, the feature extraction module is also used to:
[0157] Converting the captured image into a grayscale image, and detecting corner points and edges of the captured image by comparing the grayscale values of each pixel in the grayscale image;
[0158] Extracting feature points in the captured image using a preset feature point extraction algorithm;
[0159] The corner points, edges and feature points are used as picture features of the captured picture.
[0160] Optionally, the large space positioning system based on feature extraction is also used for:
[0161] Determining a preset second relative position of the virtual content relative to the target feature area;
[0162] Determining a third relative position of the virtual content relative to the head mounted display device according to the first relative position and the second relative position;
[0163] The virtual content is displayed in the head display device according to the third relative position.
[0164] Optionally, the relative position detection module is also used for:
[0165] Taking any feature point in the picture features as a target feature point;
[0166] Acquire a first relative position of the target feature point through an inertial sensor in the head display device;
[0167] Acquire a first relative distance of the target feature point by a distance measurement module of the head display device;
[0168] The target relative position of the target feature point relative to the head-mounted display device is determined according to the first relative orientation and the first relative distance, and the target relative position is used as the first relative position of the picture feature relative to the head-mounted display device.
[0169] Optionally, the first relative position includes: the first relative orientation and the first relative distance; and the positioning module is further used for:
[0170] Calculating the coordinate difference of the picture feature relative to the head display device in each orientation axis direction according to the first relative orientation and the first relative distance;
[0171] Subtracting the coordinate differences from the first coordinate obtains a second coordinate of the head-mounted display device relative to the preset origin.
[0172] Optionally, the large space positioning system based on feature extraction is also used for:
[0173] When the head display device has a second coordinate, acquiring the camera orientation of the camera through an inertial sensor and an accelerometer of the head display device;
[0174] Determine a shooting area corresponding to the camera orientation in a preset space according to the second coordinates and the camera orientation;
[0175] The shooting area is used as a spatial feature for feature matching of the shooting picture.
[0176] Optionally, the large space positioning system based on feature extraction is also used for:
[0177] Acquire the motion posture of the head display device through the inertial sensor;
[0178] Acquire the movement direction and movement distance of the head display device through the accelerometer;
[0179] The second coordinate of the head display device is updated in real time according to the motion posture, the motion direction, the motion distance and the second coordinate to perform real-time positioning of the head display device.
[0180] The large space positioning system based on feature extraction provided by the present application adopts the large space positioning method based on feature extraction in the above-mentioned embodiment, which can solve the technical problem of improving the positioning accuracy of the head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the large space positioning system based on feature extraction provided by the present application are the same as the beneficial effects of the large space positioning method based on feature extraction provided by the above-mentioned embodiment, and the other technical features of the large space positioning system based on feature extraction are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.
[0181] The present application provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large-space positioning method based on feature extraction in the above-mentioned embodiment 1.
[0182] Reference below Figure 5 , which shows a schematic diagram of the structure of a head display device suitable for implementing the embodiments of the present application. The head display device includes, but is not limited to, mixed reality (MixedReality)-MR devices (such as MR glasses or MR helmets), augmented reality (AugmentedReality)-AR devices (such as AR glasses or AR helmets), virtual reality-(VirtualReality)-VR devices (such as VR glasses or VR helmets), extended reality (ExtendedReality)-XR devices or some combination thereof and other head-mounted display devices. Figure 5 The head display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0183] like Figure 5As shown, the head display device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the head display device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the head display device to communicate with other devices wirelessly or wired to exchange data. Although the head display device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0184] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0185] The head-mounted display device provided by the present application adopts the large space positioning method based on feature extraction in the above embodiment, which can solve the technical problem of improving the positioning accuracy of the head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the head-mounted display device provided by the present application are the same as the beneficial effects of the large space positioning method based on feature extraction provided in the above embodiment, and the other technical features in the head-mounted display device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0186] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0187] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0188] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the large-space positioning method based on feature extraction in the above-mentioned embodiment.
[0189] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0190] The computer-readable storage medium may be included in the head-mounted display device; or may exist independently without being assembled into the head-mounted display device.
[0191] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the head-mounted display device, the head-mounted display device: obtains a captured image through a camera of the head-mounted display device, and performs feature extraction on the captured image to obtain image features; searches for a target feature area that matches the image features in preset spatial features; obtains a first relative position of the image features relative to the head-mounted display device; obtains a first coordinate of the target feature area relative to a preset origin, and determines a second coordinate of the head-mounted display device relative to the preset origin based on the first relative position and the first coordinate, so as to position the head-mounted display device.
[0192] The computer program code for performing the operation of the present application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0193] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0194] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0195] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large-space positioning method based on feature extraction, and can solve the technical problem of improving the positioning accuracy of the head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the large-space positioning method based on feature extraction provided by the above-mentioned embodiment, and will not be repeated here.
[0196] The present application also provides a computer program product, including a computer program, which implements the steps of the large space positioning method based on feature extraction as described above when executed by a processor.
[0197] The computer program product provided by the present application can solve the technical problem of improving the positioning accuracy of a head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the large space positioning method based on feature extraction provided by the above embodiment, and will not be described in detail here.
[0198] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A large space positioning method based on feature extraction, characterized in that: The method comprises: Acquire a captured image through a camera of a head-mounted display device, and perform feature extraction on the captured image to obtain image features; Searching for a target feature area matching the picture feature in the preset spatial features; Acquire a first relative position of the image feature relative to the head mounted display device; A first coordinate of the target feature area relative to a preset origin is obtained, and a second coordinate of the head-mounted display device relative to the preset origin is determined according to the first relative position and the first coordinate, so as to position the head-mounted display device.
2. The large space positioning method based on feature extraction as claimed in claim 1, characterized in that: The step of extracting features from the captured image to obtain image features comprises: Converting the captured image into a grayscale image, and detecting corner points and edges of the captured image by comparing the grayscale values of each pixel in the grayscale image; Extracting feature points in the captured image using a preset feature point extraction algorithm; The corner points, the edges and the feature points are used as picture features of the captured picture.
3. The large space positioning method based on feature extraction as claimed in claim 1, characterized in that: After the step of obtaining a first relative position of the picture feature relative to the head mounted display device, the method further includes: Determining a preset second relative position of the virtual content relative to the target feature area; Determining a third relative position of the virtual content relative to the head mounted display device according to the first relative position and the second relative position; The virtual content is displayed in the head display device according to the third relative position.
4. The large space positioning method based on feature extraction as claimed in claim 1, characterized in that: The step of obtaining a first relative position of the picture feature relative to the head mounted display device comprises: Taking any feature point in the picture features as a target feature point; Acquire a first relative position of the target feature point through an inertial sensor in the head display device; Acquire a first relative distance of the target feature point by a distance measurement module of the head display device; The target relative position of the target feature point relative to the head-mounted display device is determined according to the first relative orientation and the first relative distance, and the target relative position is used as the first relative position of the picture feature relative to the head-mounted display device.
5. The large space positioning method based on feature extraction as claimed in claim 4, characterized in that: The first relative position includes: the first relative orientation and the first relative distance; The step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate comprises: Calculating the coordinate difference of the picture feature relative to the head display device in each orientation axis direction according to the first relative orientation and the first relative distance; Subtracting the coordinate differences from the first coordinate obtains a second coordinate of the head-mounted display device relative to the preset origin.
6. The large space positioning method based on feature extraction as claimed in claim 1, characterized in that: After the step of extracting features from the captured image to obtain image features, the method further includes: When the head display device has a second coordinate, acquiring the camera orientation of the camera through an inertial sensor and an accelerometer of the head display device; Determine a shooting area corresponding to the camera orientation in a preset space according to the second coordinates and the camera orientation; The shooting area is used as a spatial feature for feature matching of the shooting picture.
7. The large space positioning method based on feature extraction as claimed in claim 6, characterized in that: After the step of determining a second coordinate of the head mounted display device relative to the preset origin according to the first relative position and the first coordinate, the method further includes: Acquire the motion posture of the head display device through the inertial sensor; Acquire the movement direction and movement distance of the head display device through the accelerometer; The second coordinate of the head display device is updated in real time according to the motion posture, the motion direction, the motion distance and the second coordinate to perform real-time positioning of the head display device.
8. A large space positioning system based on feature extraction, characterized in that: The system comprises: A feature extraction module is used to obtain a captured image through a camera of a head display device, and extract features from the captured image to obtain image features; A feature matching module, used to search for a target feature area matching the picture feature in a preset spatial feature; A relative position detection module, used to obtain a first relative position of the image feature relative to the head display device; A positioning module is used to obtain a first coordinate of the target feature area relative to a preset origin, and determine a second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate, so as to position the head display device.
9. A head mounted display device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large space positioning method based on feature extraction as described in any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the large space positioning method based on feature extraction as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Method and apparatus for augmented virtual models and orienteering
CA3043686A1
Binocular camera-based space positioning method and service equipment
CN107562189A
Distance information-based anchor-free positioning method
CN109286977A
Camera calibration method and device
CN112819895A
Spatial position identification method and device, equipment and storage medium
CN114527880A
Cited By
AR equipment positioning system and method based on data fusion
CN120593736A
An ar device positioning system and method based on data fusion
CN120593736B
