Large space positioning method and system based on image identification, head-mounted display device and medium
By using a camera in a head-mounted display device to identify and match the target image identification in the image identification library, the problem of insufficient positioning accuracy in large spaces is solved, and more accurate virtual content display is achieved, improving user experience.
Patent Information
- Application Number
- CN202411045612.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-22
AI Technical Summary
The positioning accuracy of the head-mounted display device in large spaces is insufficient, resulting in a large deviation between the virtual content display and the actual position that is wanted in the design, affecting the user experience.
The camera of the headset device obtains the shooting screen, recognizes and matches the target image identification in the preset image identification database, obtains its position relative to the device, and locates it according to the coordinates of the preset origin, and updates the coordinates of the device in real time. Achieve accurate positioning.
It improves the positioning accuracy of head-mounted display devices in large spaces, reduces virtual content display deviations, and enhances user experience.
Smart Images

Figure CN119006584A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of head-mounted display devices, and in particular to a large-space positioning method, system, head-mounted display device and medium based on image identification. Background Art
[0002] The technologies used in head-mounted display devices (i.e., head-mounted display devices) mainly include AR (Augmented Reality) technology, VR (Virtual Reality) technology and MR (Mixed Reality) technology. Based on these technologies, when users wear head-mounted display devices, they can display virtual content in real scenes through various sensors, cameras or optical components in the device.
[0003] When displaying virtual content through a head-mounted display device in a real scene, the space needs to be positioned. When the user is in a small space (less than three meters), the spatial positioning technology of the head-mounted display device can usually achieve a good positioning effect. However, when the user is in a large space (greater than ten meters) such as a factory or exhibition hall, the error of the current spatial technology will accumulate to more than one meter. The display of virtual content will deviate greatly from the actual position intended to be displayed in the design, affecting the user experience.
[0004] In summary, how to improve the positioning accuracy of head-mounted display devices in a large space and avoid display deviations of virtual content caused by positioning errors in a large space has become a technical problem that urgently needs to be solved in this field.
[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0006] The main purpose of this application is to provide a large-space positioning method, system, head-mounted display device and medium based on image identification, aiming to solve the technical problem of improving the positioning accuracy of head-mounted display devices in large spaces.
[0007] To achieve the above purpose, the present application proposes a large space positioning method based on image identification, the method comprising:
[0008] Acquire the captured image through the camera of the head-mounted display device;
[0009] When there is a target image identifier in the captured image that is identical to any image identifier in a preset image identifier library, obtaining a first relative position of the target image identifier relative to the head mounted display device, wherein the image identifier library includes a plurality of different image identifiers;
[0010] A first coordinate of the target image identifier relative to a preset origin is obtained, and a second coordinate of the head-mounted display device relative to the preset origin is determined according to the first relative position and the first coordinate, so as to position the head-mounted display device.
[0011] In one embodiment, after the step of obtaining the first relative position of the target image identifier relative to the head mounted display device, the method further includes:
[0012] Determining a preset second relative position of the virtual content relative to the target image identifier;
[0013] Determining a third relative position of the virtual content relative to the head mounted display device according to the first relative position and the second relative position;
[0014] The virtual content is displayed in the head display device according to the third relative position.
[0015] In one embodiment, after the step of acquiring a captured image through a camera of a head display device, the method further includes:
[0016] Extracting a first key point feature from the captured image;
[0017] Obtaining a second key point feature of each image identifier in the image identifier library;
[0018] Calculating the matching degree between the key point feature and each of the second key point features;
[0019] The image identifier corresponding to the target second key point feature with the highest matching degree among the second key point features is used as the target image identifier.
[0020] In one embodiment, the step of obtaining a first relative position of the target image identifier relative to the head mounted display device comprises:
[0021] Acquire a first relative position of the target image marker through an inertial sensor in the head display device;
[0022] Acquire a first relative distance of the target image mark by a distance measurement module of the head display device;
[0023] A first relative position of the target image marker relative to the head mounted display device is determined according to the first relative orientation and the first relative distance.
[0024] In one embodiment, the first relative position includes: the first relative orientation and the first relative distance;
[0025] The step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate comprises:
[0026] Calculate the coordinate difference of the target image marker relative to the head-mounted display device in each orientation axis direction according to the first relative orientation and the first relative distance;
[0027] Subtracting the coordinate differences from the first coordinate obtains a second coordinate of the head-mounted display device relative to the preset origin.
[0028] In one embodiment, after the step of determining a second coordinate of the head mounted display device relative to the preset origin according to the first relative position and the first coordinate, the method further includes:
[0029] Acquiring a motion posture of the head-mounted display device through an inertial sensor of the head-mounted display device;
[0030] Acquire the movement direction and movement distance of the head display device through the accelerometer of the head display device;
[0031] The second coordinate of the head display device is updated in real time according to the motion posture, the motion direction, the motion distance and the second coordinate to perform real-time positioning of the head display device.
[0032] In one embodiment, when there are multiple target image identifiers, the step of determining the second coordinate of the head mounted display device relative to the preset origin according to the first relative position and the first coordinate includes:
[0033] Calculating the corresponding intermediate second coordinates of each of the target image markers according to each of the first relative positions and each of the first coordinates;
[0034] The second coordinate of the head display device relative to the preset origin is obtained by averaging the intermediate second coordinates.
[0035] In addition, to achieve the above purpose, the present application also proposes a large space positioning system based on image identification, the system comprising:
[0036] The image acquisition module is used to acquire the captured image through the camera of the head display device;
[0037] an identification detection module, configured to obtain a first relative position of the target image identification with respect to the head-mounted display device when there is a target image identification that is the same as any image identification in a preset image identification library in the captured image, wherein the image identification library includes a plurality of different image identifications;
[0038] A positioning module is used to obtain a first coordinate of the target image mark relative to a preset origin, and determine a second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate, so as to position the head display device.
[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a head-mounted display device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large-space positioning method based on image identification as described above.
[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large-space positioning method based on image identification as described above are implemented.
[0041] The present application provides a large-space positioning method based on image identification. The present application obtains a captured picture of the user's surroundings when the user wears the head-mounted display device through a camera of the head-mounted display device, and pre-sets an image identification library, in which a plurality of different image identifications are stored. By analyzing the captured picture, it is possible to analyze which image identification is included in the captured picture. When there is a target image identification in the captured picture that is the same as that in the image identification library, the first relative position of the target image identification relative to the head-mounted display device is obtained, and the second coordinate of the head-mounted display device can be positioned according to the first coordinate of the target image identification compared to a preset origin.
[0042] In summary, the present application establishes an image identification library in advance, presets the coordinates between each image identification and the origin, and then identifies the matching target image identification in the picture taken by the head-mounted display device, so as to position the head-mounted display device according to the relative position. The above method is not limited by the size of the space. It only needs to set the image identification in the space to enable the head-mounted display device to be accurately positioned in a space of any size, thereby improving the positioning accuracy of the head-mounted display device in a large space, thereby avoiding the display deviation of the virtual content caused by the positioning error in the large space. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0045] Figure 1 A schematic diagram of a flow chart provided for an embodiment of a large space positioning method based on image identification in the present application;
[0046] Figure 2 This is a schematic diagram of a scene involved in the first embodiment of the large space positioning method based on image identification of the present application;
[0047] Figure 3 This is a schematic diagram of a scene involved in the third embodiment of the large space positioning method based on image identification of the present application;
[0048] Figure 4 This is a schematic diagram of the module structure of a large space positioning system based on image identification according to an embodiment of the present application;
[0049] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the large space positioning method based on image identification in the embodiment of the present application.
[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0053] The main solution of the embodiment of the present application is: obtaining a captured image through a camera of a head-mounted display device; when there is a target image identifier that is identical to any image identifier in a preset image identifier library in the captured image, obtaining a first relative position of the target image identifier relative to the head-mounted display device, wherein the image identifier library includes a plurality of different image identifiers; obtaining a first coordinate of the target image identifier relative to a preset origin, and determining a second coordinate of the head-mounted display device relative to the preset origin based on the first relative position and the first coordinate, so as to position the head-mounted display device.
[0054] In this embodiment, for the convenience of description, the following description is made with the head mounted display device as the execution subject.
[0055] The technologies used in head-mounted display devices mainly include AR (Augmented Reality) technology, VR (Virtual Reality) technology and MR (Mixed Reality) technology. Based on these technologies, when users wear head-mounted display devices, they can display virtual content in real scenes through various sensors, cameras or optical components in the device.
[0056] When displaying virtual content through a head-mounted display device in a real scene, the space needs to be positioned. When the user is in a small space (less than three meters), the spatial positioning technology of the head-mounted display device can usually achieve a good positioning effect. However, when the user is in a large space (greater than ten meters) such as a factory or exhibition hall, the error of the current spatial technology will accumulate to more than one meter. The display of virtual content will deviate greatly from the actual position intended to be displayed in the design, affecting the user experience.
[0057] In summary, how to improve the positioning accuracy of head-mounted display devices in a large space and avoid display deviations of virtual content caused by positioning errors in a large space has become a technical problem that urgently needs to be solved in this field.
[0058] In response to the above problems, the present application provides a large-space positioning method based on image identification. The present application detects the captured picture around the user when the user wears the head-mounted display device through the camera of the head-mounted display device, and is provided with an image identification library. The image identification library stores multiple different image identifications. By analyzing the captured picture, it is possible to analyze which image identification is included in the captured picture. When there is a target image identification in the captured picture that is the same as that in the image identification library, the first relative position of the target image identification relative to the head-mounted display device is obtained, and the second coordinate of the head-mounted display device can be positioned according to the first coordinate of the target image identification compared to the preset origin.
[0059] In summary, the present application establishes an image identification library in advance, presets the coordinates between each image identification and the origin, and then identifies the matching target image identification in the picture taken by the head-mounted display device, so as to position the head-mounted display device according to the relative position. The above method is not limited by the size of the space. It only needs to set the image identification in the space to enable the head-mounted display device to be accurately positioned in a space of any size, thereby improving the positioning accuracy of the head-mounted display device in a large space, thereby avoiding the display deviation of the virtual content caused by the positioning error in the large space.
[0060] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a head display device, etc. The head display device is taken as an example to illustrate this embodiment and the following embodiments.
[0061] Based on this, the embodiment of the present application provides a large space positioning method based on image identification, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the large space positioning method based on image identification of the present application.
[0062] In this embodiment, the large space positioning method based on image identification includes steps S10 to S30:
[0063] Step S10, obtaining a captured image through a camera of a head display device;
[0064] It should be noted that in this embodiment, the head display device is a wearable display technology device designed to project visual content directly in front of the user's eyes to create an immersive visual experience. This device is widely used in the fields of VR, AR and MR, and provides users with a unique interaction method and content display platform by simulating or enhancing the real world seen by users.
[0065] It should be noted that in this embodiment, all private data used is obtained with the user's knowledge and authorization.
[0066] In this embodiment, after the head display device is turned on, it obtains the captured images around the user, i.e., the wearer, through the camera in real time, and performs positioning through the captured images of the user.
[0067] In addition, the camera does not need to collect additional images around the wearer, but can directly use the user's visual image collected by the camera as the shooting image for feature matching, which can reduce the operating frequency of the camera.
[0068] In a feasible implementation manner, after the above step S10, the method may further include steps A10 to A40:
[0069] Step A10, extracting the first key point feature in the captured image;
[0070] It should be noted that, in this embodiment, the key point feature refers to a local area in the image that is unique and repeatable, such as a corner point, an edge, a texture pattern, and the like.
[0071] In this embodiment, in order to identify whether there is a target image logo in the captured image, the system uses computer vision technology (such as feature detection algorithms such as SI FT, SURF, ORB, etc.) to extract key point features from the image captured by the head display device camera.
[0072] Step A20, obtaining the second key point feature of each image identification in the image identification library;
[0073] In this embodiment, the system can extract features from each image identifier in the image library in real time to obtain the second key point features, or can extract them in advance and store them in the image identifier library for direct call.
[0074] Step A30, calculating the matching degree between the key point feature and each of the second key point features;
[0075] In this embodiment, the matching degree between the first key point feature and each second key point feature can be calculated based on the extracted first key point feature and each second key point feature. The matching degree can be calculated in a variety of ways, such as calculating the distance or correlation between two feature vectors. The matching degree reflects the similarity between the two sets of key point features. If there are many similar key points between the two feature sets, the matching degree is high; otherwise, it is low.
[0076] Step A40: taking the image identifier corresponding to the target second key point feature with the highest matching degree among the second key point features as the target image identifier.
[0077] In this embodiment, the system will select the second key point feature with the highest matching degree, and its corresponding image identifier will be regarded as the target image identifier. This means that the image identifier found in the captured image is closest to a certain identifier in the image identifier library, so it can be considered that the correct identifier has been found.
[0078] In addition, it should be noted that in this embodiment, as a preferred implementation method, the image logo can be a picture that is asymmetric in the horizontal and vertical directions, and the size of the asymmetric part must be large enough (larger than 1 / 10 of the side length of the image), each image logo cannot be the same, and the size of the entire logo must meet the following requirements: when the user uses the head display device in daily use, when the camera takes pictures of the surroundings, the picture is clearly visible in the camera. For example, if a picture is pasted on the wall of a large factory, the user's daily walkway is 5 meters away from the wall, and the picture size is required to be large enough that the picture content is clearly visible 5 meters away.
[0079] In addition, when pasting positioning marks in a large space, it is required to be pasted near the virtual content that needs to be displayed (within 5m), that is, one positioning mark can ensure accurate positioning within 5m of the positioning mark. If the entire area needs to display accurately positioned content, the distance between two customized marks must be less than or equal to 10m.
[0080] In this embodiment, the entire system can not only identify the target image identifier, but also more accurately select the correct one from multiple possible image identifiers. This method improves the accuracy of recognition and reduces the risk of mismatching, especially in complex or dynamic environments.
[0081] In another preferred embodiment, each image identifier in the image identifier library can be set as an AR tag or a QR code image. Through the existing image recognition algorithm, the system can quickly identify whether there is an AR tag or a QR code image in the image identifier library in the captured picture, thereby using it as the target image identifier for subsequent positioning operations.
[0082] The above are only two feasible implementation methods for identifying the target image identifier in the captured image provided by this embodiment. This embodiment does not specifically limit the specific implementation method for identifying the target image identifier in the captured image.
[0083] Step S20, when there is a target image identifier in the captured image that is identical to any image identifier in a preset image identifier library, obtaining a first relative position of the target image identifier relative to the head mounted display device, wherein the image identifier library includes a plurality of different image identifiers;
[0084] It should be noted that, in this embodiment, the image marker library includes a series of predefined image markers (Image Markers), which are usually special patterns or marks placed in the real world. These markers can be quickly identified with the help of computer vision algorithms, thereby assisting the head-mounted display device in positioning.
[0085] In this embodiment, the system analyzes the image acquired by the camera of the head-mounted display device to find out whether there is a target image identifier that matches any image identifier in the image identifier library. Once a matching identifier is found, the system calculates the position of the identifier relative to the head-mounted display device, where the "first relative position" refers to the spatial position relationship of the target image identifier relative to the camera (i.e., the spatial position relationship of the target image identifier relative to the head-mounted display device), including distance, angle and other information.
[0086] In a feasible implementation manner, the step of obtaining the first relative position of the target image identifier relative to the head mounted display device in the above step S20 may include steps S201 to S204:
[0087] Step S201, acquiring a first relative position of the target image marker through an inertial sensor in the head display device;
[0088] It should be noted that, in this embodiment, the inertial measurement unit (IMU) is a type of sensor, usually including an accelerometer and a gyroscope, which is used to measure parameters such as the speed, acceleration and rotation of an object. In a head-mounted display device, the IMU can help track the motion state of the device.
[0089] In this embodiment, the system obtains the orientation information of the target image marker relative to the head-mounted display device through the inertial sensor in the head-mounted display device, where the "first relative orientation" refers to the direction of the target image marker relative to the head-mounted display device, including pitch angle, yaw angle, roll angle, etc.
[0090] Step S202, obtaining a first relative distance of the target image mark through a distance measurement module of the head display device;
[0091] It should be noted that, in this embodiment, the ranging module is used to measure the first relative distance of the target image mark relative to the head display device. The ranging module generally includes one or more sensors, and the ranging module can communicate with the control chip and camera in the head display device.
[0092] The distance measurement module usually implements distance measurement based on different physical principles and technologies. Common technologies include:
[0093] Ultrasonic sensor: Utilizes the propagation characteristics of sound waves to calculate distance by sending sound wave pulses and receiving echoes; suitable for short-distance measurement, and can be used in smaller spaces as a ranging module for positioning head-mounted display devices.
[0094] Laser sensor: uses a laser beam for distance measurement and determines the distance by calculating the time difference between the laser emission and the reflected signal; it is used in situations requiring high precision and is suitable for positioning in large spaces, and can improve the positioning accuracy of head-mounted display devices.
[0095] Infrared sensor: measures distance by emitting infrared light and receiving reflected signals; suitable for short-distance and low-cost applications, and also suitable for smaller spaces, saving the positioning cost of head-mounted display devices.
[0096] Radar sensor: including microwave radar and millimeter wave radar, which measures distance by emitting electromagnetic waves and receiving reflected signals; suitable for the detection of long-distance and high-speed moving objects, suitable for scenarios where users are in a large space and are always on the move, and can measure the first relative distance of the target image mark in real time.
[0097] Time of Flight (ToF) sensor: emits continuous or pulsed light signals and measures distance by calculating the time it takes for light to be emitted and returned; suitable for accurate measurement of short to medium distances, and can be used as a ranging module sensor in scenes of moderate size and where the user does not move frequently.
[0098] Different types of sensors have certain advantages and limitations. In practical applications, one or more of the above sensors can be used in combination as a distance measurement module to measure the first relative position of the target image mark according to actual conditions.
[0099] In addition, in order to improve the measurement accuracy of the ranging module, the ranging module can be installed on a movable gimbal on the head-mounted display device, so that the measurement range of the ranging module can cover the shooting range of the camera.
[0100] In this embodiment, after the target image logo is recognized by the camera, the system controls the ranging module to measure the distance of the target image logo according to the image recognition result of the camera, and the measured distance is the first relative distance of the target image logo relative to the head display device.
[0101] Step S203: Determine a first relative position of the target image marker relative to the head mounted display device according to the first relative orientation and the first relative distance.
[0102] In this embodiment, the spatial position of the target image marker relative to the head-mounted display device can be determined by combining the information of the first relative orientation and the first relative distance obtained previously.
[0103] In a feasible implementation manner, after the above step S20, the method may further include steps B10 to B30:
[0104] Step B10, determining a preset second relative position of the virtual content relative to the target image mark;
[0105] In this embodiment, in augmented reality applications, virtual content usually needs to be placed at a preset position relative to a real-world reference object (here, a target image marker), and this position is predefined and can vary depending on the application scenario.
[0106] For example, in an AR game, a user may want a virtual character to always appear at a certain distance directly above a target image logo. This positional relationship is the so-called "second relative position", which is pre-set by the developer and fixed in the program.
[0107] Step B20: determining a third relative position of the virtual content relative to the head display device according to the first relative position and the second relative position;
[0108] It should be noted that, in this embodiment, the first relative position refers to the positional relationship of the target image identifier relative to the head-mounted display device, which has been determined in the above steps.
[0109] The second relative position refers to the position relationship of the virtual content relative to the target image identifier.
[0110] In this embodiment, the system needs to combine these two relative positions to calculate the exact position of the virtual content relative to the head-mounted display device, for example, converting the position of the virtual content from the coordinate system of the target image identifier to the coordinate system of the head-mounted display device. This final position is called the "third relative position."
[0111] Step B30: displaying the virtual content in the head display device according to the third relative position.
[0112] In this embodiment, once the third relative position of the virtual content with respect to the head display device is calculated, the virtual content can be displayed in the head display device. The system renders the virtual content into the user's field of view so that it appears to exist at a certain location in the real world.
[0113] Step S30: Acquire a first coordinate of the target image mark relative to a preset origin, and determine a second coordinate of the head-mounted display device relative to the preset origin according to the first relative position and the first coordinate, so as to position the head-mounted display device.
[0114] It should be noted that, in this embodiment, the preset origin refers to a fixed reference point, which is usually a known position set in a real environment. The origin can be used as a reference point of the positioning system. Here, one of the image identifiers can be used as the origin.
[0115] In this embodiment, the system first determines the coordinates of the target image marker relative to the preset origin (i.e., the first coordinates), and then combines the first relative position to calculate the position of the head-mounted display device relative to the preset origin (i.e., the second coordinates).
[0116] In this embodiment, by pre-establishing an image identification library, the coordinates between each image identification and the origin are preset, and then by identifying the matched target image identification in the picture taken by the head-mounted display device, the head-mounted display device can be positioned according to the relative position. The above method is not limited by the size of the space. It only needs to set the image identification in the space to enable the head-mounted display device to be accurately positioned in a space of any size, thereby improving the positioning accuracy of the head-mounted display device in a large space, thereby avoiding the display deviation of the virtual content caused by the positioning error in the large space.
[0117] In a feasible implementation manner, the first relative position includes: the first relative orientation and the first relative distance; in the above step S30, the step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate may include steps S301 to S302:
[0118] Step S301, calculating the coordinate difference of the target image marker relative to the head display device in each orientation axis direction according to the first relative orientation and the first relative distance;
[0119] In this embodiment, the system needs to use the first relative orientation (including pitch angle, yaw angle and roll angle) and the first relative distance of the target image marker to calculate the coordinate difference of the target image marker relative to the head display device. The coordinate difference refers to the displacement of the target image marker relative to the head display device on three coordinate axes (usually X, Y, and Z axes).
[0120] Specifically, the system uses the first relative orientation and the first relative distance to calculate the displacement values of the target image mark in the directions of each coordinate axis through trigonometric functions and geometric relationships. These displacement values are coordinate differences.
[0121] Step S302: Subtract the coordinate differences from the first coordinate to obtain a second coordinate of the head display device relative to the preset origin.
[0122] In this embodiment, after calculating the coordinate differences, the system will subtract these coordinate differences from the first coordinates of the target image mark to obtain the second coordinates of the head display device relative to the preset origin. The "first coordinates" here refer to the coordinates of the target image mark relative to the preset origin.
[0123] Specifically, assuming that the coordinates of the target image are (x1, y1, z1) and the coordinate difference is (Δx, Δy, Δz), then the coordinates of the head display device relative to the preset origin are (x h ,y h , z h ), the calculation formula is as follows:
[0124] x h =x1-Δx;
[0125] y h =y1-Δy;
[0126] z h =z1-Δz;
[0127] The second coordinate of the head display device can be obtained by calculating according to the above formula.
[0128] For example, in order to help understand the implementation process of the large space positioning method based on image identification obtained by combining this embodiment with the above-mentioned embodiment 1, an example is given in combination with an actual scene. Specifically, please refer to Figure 2 , Figure 2 This is a schematic diagram of a scene involved in the first embodiment of the large space positioning method based on image identification of this application, such as Figure 2As shown, when the user wears the head-mounted display device in a large space, the camera of the head-mounted display device takes a picture to obtain a captured image. The dotted box in the figure is the content captured by the camera. Then, the image identifier in the captured image is identified. After identifying that the image identifier matches the image identifier A in the image identifier library, the coordinates of the image identifier A compared to the origin are called from the image identifier library, and then the relative position of the image identifier A compared to the head-mounted display device is calculated, so that the position of the head-mounted display device can be located.
[0129] When the user wears the head-mounted display device and walks to other locations, when other image markers appear in the shooting picture of the head-mounted display device, positioning is performed again based on the other image markers to complete the positioning calibration.
[0130] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, after the above step S30, the method can also include steps C10 to C30:
[0131] Step C10, acquiring a motion posture of the head display device through an inertial sensor of the head display device;
[0132] It should be noted that, in this embodiment, the inertial sensor generally includes components such as an accelerometer and a gyroscope, which can be used to measure the motion posture of the head-mounted display device.
[0133] In this embodiment, the motion posture of the head display device can be obtained through various parameters detected by the inertial sensor.
[0134] Step C20, obtaining the movement direction and movement distance of the head display device through the accelerometer of the head display device;
[0135] It should be noted that, in this embodiment, the accelerometer can measure the linear acceleration of the head-mounted display device, including the acceleration caused by gravity. By continuously monitoring the acceleration data, the movement direction and moving distance of the head-mounted display device can be estimated.
[0136] In this embodiment, the system estimates the movement direction and movement distance of the head display device by analyzing the data of the accelerometer.
[0137] Step C30: updating the second coordinate of the head display device in real time according to the motion posture, the motion direction, the motion distance and the second coordinate, so as to perform real-time positioning of the head display device.
[0138] In this embodiment, the system combines the acquired motion posture, motion direction and motion distance information, as well as the previously determined second coordinates of the head-mounted display device relative to a preset origin, to update the position information of the head-mounted display device in real time.
[0139] Specifically, the system uses this data to calculate the new position of the headset to determine the new coordinates of the headset in three-dimensional space. This real-time update can be achieved using different algorithms, such as Kalman filters or complementary filters, which can help eliminate noise and improve the accuracy of position estimates.
[0140] In this embodiment, after the head display device is positioned through image identification, the head display device can update the position of the head display device in subsequent movement through various built-in sensors.
[0141] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned embodiment 1 and / or embodiment 2 can be referred to the above description and will not be repeated later. On this basis, when there are multiple target image identifiers, the step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate in the above-mentioned step S30 includes steps D10 to D20:
[0142] Step D10, calculating the corresponding intermediate second coordinates of each of the target image markers according to each of the first relative positions and each of the first coordinates;
[0143] Step D20: averaging the intermediate second coordinates to obtain the second coordinate of the head display device relative to the preset origin.
[0144] In this embodiment, if the captured image of the camera includes multiple target image identifiers, the intermediate second coordinates of the head-mounted display device are calculated according to the first relative position and the first coordinates of each target image identifier, and then the average of the multiple intermediate second coordinates is calculated. The coordinates of each image identifier can be comprehensively combined to comprehensively locate the head-mounted display device, thereby improving the positioning accuracy.
[0145] For example, in order to help understand the implementation process of the large space positioning method based on image identification obtained by combining this embodiment with the above-mentioned embodiment 3, an example is given in combination with an actual scenario. Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of a scene involved in the third embodiment of the large space positioning method based on image identification of this application, such as Figure 3As shown, when there are two image identifiers in the picture captured by the user wearing the head-mounted display device, the two image identifiers are matched in the image identifier library, and the image identifiers A and B are matched respectively. The second coordinate A3 of the head-mounted display device can be obtained according to the first relative position A1 and the first coordinate A2 of the image identifier A, and the second coordinate B3 of the head-mounted display device can be obtained according to the first relative position B1 and the first coordinate B2 of the image identifier B. Then, the second coordinate A3 and the second coordinate B3 are averaged to obtain the second coordinate of the head-mounted display device.
[0146] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the large-space positioning method based on image identification of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0147] This application also provides a large space positioning system based on image identification, please refer to Figure 4 , the system comprising:
[0148] The image acquisition module 10 is used to acquire the captured image through the camera of the head display device;
[0149] an identification detection module 20, configured to obtain a first relative position of the target image identification relative to the head-mounted display device when there is a target image identification identical to any image identification in a preset image identification library in the captured image, wherein the image identification library includes a plurality of different image identifications;
[0150] The positioning module 30 is used to obtain the first coordinate of the target image mark relative to the preset origin, and determine the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate, so as to position the head display device.
[0151] Optionally, the large space positioning system based on image identification is also used for:
[0152] Determining a preset second relative position of the virtual content relative to the target image identifier;
[0153] Determining a third relative position of the virtual content relative to the head mounted display device according to the first relative position and the second relative position;
[0154] The virtual content is displayed in the head display device according to the third relative position.
[0155] Optionally, the large space positioning system based on image identification is also used for:
[0156] Extracting a first key point feature from the captured image;
[0157] Obtaining a second key point feature of each image identifier in the image identifier library;
[0158] Calculating the matching degree between the key point feature and each of the second key point features;
[0159] The image identifier corresponding to the target second key point feature with the highest matching degree among the second key point features is used as the target image identifier.
[0160] Optionally, the picture acquisition module is also used for:
[0161] Acquire a first relative position of the target image marker through an inertial sensor in the head display device;
[0162] Acquire a first relative distance of the target image mark by a distance measurement module of the head display device;
[0163] A first relative position of the target image marker relative to the head mounted display device is determined according to the first relative orientation and the first relative distance.
[0164] Optionally, the first relative position includes: the first relative orientation and the first relative distance; and the positioning module is further used for:
[0165] Calculate the coordinate difference of the target image marker relative to the head-mounted display device in each orientation axis direction according to the first relative orientation and the first relative distance;
[0166] Subtracting the coordinate differences from the first coordinate obtains a second coordinate of the head-mounted display device relative to the preset origin.
[0167] Optionally, the large space positioning system based on image identification is also used for:
[0168] Acquiring a motion posture of the head-mounted display device through an inertial sensor of the head-mounted display device;
[0169] Acquire the movement direction and movement distance of the head display device through the accelerometer of the head display device;
[0170] The second coordinate of the head display device is updated in real time according to the motion posture, the motion direction, the motion distance and the second coordinate to perform real-time positioning of the head display device.
[0171] Optionally, when there are multiple target image identifiers, the positioning module is further used to:
[0172] Calculating the corresponding intermediate second coordinates of each of the target image markers according to each of the first relative positions and each of the first coordinates;
[0173] The second coordinate of the head display device relative to the preset origin is obtained by averaging the intermediate second coordinates.
[0174] The large space positioning system based on image identification provided by the present application adopts the large space positioning method based on image identification in the above-mentioned embodiment, which can solve the technical problem of improving the positioning accuracy of the head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the large space positioning system based on image identification provided by the present application are the same as the beneficial effects of the large space positioning method based on image identification provided by the above-mentioned embodiment, and the other technical features of the large space positioning system based on image identification are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.
[0175] The present application provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large-space positioning method based on image identification in the above-mentioned embodiment 1.
[0176] Reference below Figure 5 , which shows a schematic diagram of the structure of a head display device suitable for implementing the embodiments of the present application. The head display device includes, but is not limited to, mixed reality (Mixed Reality)-MR devices (such as MR glasses or MR helmets), augmented reality (Augmented Reality)-AR devices (such as AR glasses or AR helmets), virtual reality-VR devices (such as VR glasses or VR helmets), extended reality (Extended Reality)-XR devices or some combination thereof and other head-mounted display devices. Figure 5 The head display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0177] like Figure 5As shown, the head display device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the head display device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the head display device to communicate with other devices wirelessly or wired to exchange data. Although the head display device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0178] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0179] The head-mounted display device provided by the present application adopts the large space positioning method based on image identification in the above embodiment, which can solve the technical problem of improving the positioning accuracy of the head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the head-mounted display device provided by the present application are the same as the beneficial effects of the large space positioning method based on image identification provided by the above embodiment, and the other technical features in the head-mounted display device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0180] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0181] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0182] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the large-space positioning method based on image identification in the above-mentioned embodiment.
[0183] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0184] The computer-readable storage medium may be included in the head-mounted display device; or may exist independently without being assembled into the head-mounted display device.
[0185] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a head-mounted display device, the head-mounted display device: obtains a captured image through a camera of the head-mounted display device; when there is a target image identifier that is the same as any image identifier in a preset image identifier library in the captured image, obtains a first relative position of the target image identifier relative to the head-mounted display device, wherein the image identifier library includes a plurality of different image identifiers; obtains a first coordinate of the target image identifier relative to a preset origin, and determines a second coordinate of the head-mounted display device relative to the preset origin based on the first relative position and the first coordinate, so as to position the head-mounted display device.
[0186] The computer program code for performing the operation of the present application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0187] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0188] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0189] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large-space positioning method based on image identification, and can solve the technical problem of improving the positioning accuracy of the head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the large-space positioning method based on image identification provided by the above-mentioned embodiment, and will not be repeated here.
[0190] The present application also provides a computer program product, including a computer program, which implements the steps of the large space positioning method based on image identification as described above when executed by a processor.
[0191] The computer program product provided by the present application can solve the technical problem of improving the positioning accuracy of a head-mounted display device in a large space. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the large space positioning method based on image identification provided by the above embodiment, and will not be repeated here.
[0192] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A large space positioning method based on image identification, characterized in that: The method comprises: Acquire the captured image through the camera of the head-mounted display device; When there is a target image identifier in the captured image that is identical to any image identifier in a preset image identifier library, obtaining a first relative position of the target image identifier relative to the head mounted display device, wherein the image identifier library includes a plurality of different image identifiers; A first coordinate of the target image identifier relative to a preset origin is obtained, and a second coordinate of the head-mounted display device relative to the preset origin is determined according to the first relative position and the first coordinate, so as to position the head-mounted display device.
2. The large space positioning method based on image identification as claimed in claim 1, characterized in that: After the step of obtaining a first relative position of the target image identifier relative to the head mounted display device, the method further includes: Determining a preset second relative position of the virtual content relative to the target image identifier; Determining a third relative position of the virtual content relative to the head mounted display device according to the first relative position and the second relative position; The virtual content is displayed in the head display device according to the third relative position.
3. The large space positioning method based on image identification as claimed in claim 1, characterized in that: After the step of acquiring the captured image through the camera of the head display device, the method further includes: Extracting a first key point feature from the captured image; Obtaining a second key point feature of each image identifier in the image identifier library; Calculating the matching degree between the key point feature and each of the second key point features; The image identifier corresponding to the target second key point feature with the highest matching degree among the second key point features is used as the target image identifier.
4. The large space positioning method based on image identification as claimed in claim 1, characterized in that: The step of obtaining a first relative position of the target image identifier relative to the head mounted display device comprises: Acquire a first relative position of the target image marker through an inertial sensor in the head display device; Acquire a first relative distance of the target image mark by a distance measurement module of the head display device; A first relative position of the target image marker relative to the head mounted display device is determined according to the first relative orientation and the first relative distance.
5. The large space positioning method based on image identification as claimed in claim 4, characterized in that: The first relative position includes: the first relative orientation and the first relative distance; The step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate comprises: Calculate the coordinate difference of the target image marker relative to the head-mounted display device in each orientation axis direction according to the first relative orientation and the first relative distance; Subtracting the coordinate differences from the first coordinate obtains a second coordinate of the head-mounted display device relative to the preset origin.
6. The large space positioning method based on image identification as claimed in claim 1, characterized in that: After the step of determining a second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate, the method further includes: Acquiring a motion posture of the head-mounted display device through an inertial sensor of the head-mounted display device; Acquire the movement direction and movement distance of the head display device through the accelerometer of the head display device; The second coordinate of the head display device is updated in real time according to the motion posture, the motion direction, the motion distance and the second coordinate to perform real-time positioning of the head display device.
7. The large space positioning method based on image identification as claimed in claim 1, characterized in that: When there are multiple target image identifiers, the step of determining the second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate includes: Calculating the corresponding intermediate second coordinates of each of the target image markers according to each of the first relative positions and each of the first coordinates; The second coordinate of the head display device relative to the preset origin is obtained by averaging the intermediate second coordinates.
8. A large space positioning system based on image identification, characterized in that: The system comprises: The image acquisition module is used to acquire the captured image through the camera of the head display device; an identification detection module, configured to obtain a first relative position of the target image identification with respect to the head-mounted display device when there is a target image identification that is the same as any image identification in a preset image identification library in the captured image, wherein the image identification library includes a plurality of different image identifications; A positioning module is used to obtain a first coordinate of the target image mark relative to a preset origin, and determine a second coordinate of the head display device relative to the preset origin according to the first relative position and the first coordinate, so as to position the head display device.
9. A head mounted display device, characterized in that: The head display device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large space positioning method based on image identification as described in any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the large-space positioning method based on image identification as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Indoor positioning method and device, computer equipment and storage medium
CN110132274A
Head-mounted display equipment, positioning method and positioning system
CN110579739A
Positioning tracking method and device, terminal equipment and computer readable storage medium
CN110794955A
Feature image recognition and positioning method and system
CN113409385A
Space-time calibration method and device and storage medium
CN115908586A
Cited By
Positioning method for smart glasses in large space, head-mounted display device, and storage medium
WO2026175151A1
