An angle automatic identification system of a head-mounted AR device and an identification method thereof

Abstract

The application discloses an angle automatic identification system of a head-mounted AR device, comprising a moving state monitoring module, which is used for monitoring the moving state of the head-mounted AR device in real time; an image recognition module, which is used for processing the real-time image collected by the head-mounted AR device and recognizing a reference image block; an angle change calculation module, which is used for calculating the current head-mounted AR device angle change amount according to the change of the reference image block in the real-time image; and a correction module, which is used for correcting the moving state monitoring module by using the head-mounted AR device angle change amount calculated by the angle change calculation module. The application can improve the defects of the prior art and realize the non-stop automatic calibration of the head-mounted AR device.

Description

Technical Field

[0001] This invention relates to the field of augmented reality imaging technology, and in particular to an automatic angle recognition system and method for head-mounted AR devices. Background Technology

[0002] Augmented reality (AR) imaging is a technology that combines virtual images with real-world images, widely used in multimedia displays, 3D modeling, and intelligent interaction. To ensure image realism, the virtual and real-world images must maintain precise relative positions. Current technologies typically involve initializing and calibrating the accelerometer and three-axis gyroscope before use, then calculating the angular position of the AR device relative to the real-world image based on real-time monitoring data from the accelerometer and gyroscope, and finally precisely combining the virtual and real-world images. However, this method suffers from cumulative deviations due to monitoring errors in the accelerometer and gyroscope, leading to significant discrepancies in the relative positions of the virtual and real-world images after prolonged continuous use. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an automatic angle recognition system and recognition method for head-mounted AR devices, which can overcome the shortcomings of the prior art and realize automatic calibration of head-mounted AR devices without stopping the machine.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows.

[0005] An automatic angle recognition system for head-mounted AR devices includes: The motion status monitoring module is used to monitor the motion status of the head-mounted AR device in real time. The image recognition module is used to process real-time images captured by the head-mounted AR device and identify reference image blocks; Angle change calculation module, used to calculate the current angle change of the head-mounted AR device based on the changes of the reference image block in the real-time image; The calibration module is used to calibrate the motion status monitoring module based on the angle change of the head-mounted AR device calculated by the angle change calculation module.

[0006] A recognition method for the aforementioned automatic angle recognition system for head-mounted AR devices includes the following steps: A. The motion status monitoring module monitors the motion status of the head-mounted AR device in real time; B. The image recognition module processes the real-time images captured by the head-mounted AR device and identifies reference image blocks; C. The angle change calculation module calculates the current angle change of the head-mounted AR device based on the changes of the reference image block in the real-time image. D. The correction module uses the angle change calculation module to calculate the angle change of the head-mounted AR device to correct the motion status monitoring module.

[0007] Preferably, the motion state monitoring module uses an accelerometer and a three-axis gyroscope to monitor the motion state of the head-mounted AR device in real time. The motion state includes the linear acceleration and angular acceleration of the head-mounted AR device in a three-dimensional coordinate system.

[0008] Preferably, step B, which processes the real-time image, includes the following steps: B11. Set the sampling time period. In each sampling time period, the image is sampled several times, and the time interval between each sampling is less than 1ms. B12. Sharpen the image for each sample; B13. Superimpose the sampled images from the same sampling time period to obtain the processed real-time image.

[0009] Preferably, step B13, which involves overlaying the sampled images, includes the following steps: B131. Perform a Discrete Fourier Transform on the sampled images to obtain a frequency domain dataset; group the frequency domain dataset according to frequency to obtain a high-frequency dataset, a mid-frequency dataset, and a low-frequency dataset, with the mid-frequency dataset accounting for more than 50% of the entire frequency range; B132. Establish a mapping relationship between high-frequency datasets, mid-frequency datasets, and low-frequency datasets. The mapping relationship consists of pixel coordinate relationships and gray-level similarity. B133. Perform bandpass filtering on the high-frequency dataset and the low-frequency dataset respectively. Use the mapping relationship to establish an associated data group that includes the filtered high-frequency data and the filtered low-frequency data. Delineate the filtering region on the sampled image according to the pixel coordinates corresponding to the associated data group. B134. Use the intermediate frequency dataset and the corresponding mapping relationship to enhance the information of the frequency domain data in the filtered region; B135. Merge the frequency domain data of different sampled images, and then obtain the superimposed sampled image through inverse Fourier transform.

[0010] Preferably, step B134, which enhances the information in the filtered region, includes the following steps: A first feature vector set is established for the high-frequency data of the filtered region and the intermediate-frequency dataset of the sampled image. A second feature vector set is established for the low-frequency data of the filtered region and the intermediate-frequency dataset of the sampled image. Linearly correlated feature vectors from the first and second feature vector sets are extracted to form a reference feature vector set. Intermediate-frequency data related to the reference feature vector set are used to perform up-frequency and down-frequency processing through a generative adversarial network to obtain high-frequency enhanced data and low-frequency enhanced data corresponding to the filtered region. The high-frequency enhanced data is merged with the filtered high-frequency dataset, and the low-frequency enhanced data is merged with the filtered low-frequency dataset.

[0011] Preferably, step B, identifying the reference image block, includes the following steps: B21. Mark the image regions in the real-time images obtained in each sampling time period that have not undergone positional changes; B22. Traverse the marked image regions, mark closed regions in each image region whose grayscale uniformity is greater than a preset threshold, and select the closed region with the largest area as the reference image block of the corresponding image region. B23. Select a reference image block that appears in at least two consecutive sampling time periods as the reference image block for the real-time image.

[0012] Preferably, step C, calculating the current angle change of the head-mounted AR device, includes the following steps. The angle change of the head-mounted AR device relative to the reference image block is calculated based on the position and orientation angle changes of the reference image block in two consecutive sampling time periods.

[0013] Preferably, step D, calibrating the motion status monitoring module, includes the following steps. Based on the motion status of the head-mounted AR device continuously monitored by the motion status monitoring module and the initial position of the head-mounted AR device, the current angle of the head-mounted AR device relative to the reference image block is calculated. If the calculation result deviates from the calculation result in step C, the calculation result in step C is used as the initial position of the head-mounted AR device to initialize and correct the motion status monitoring module.

[0014] The beneficial effects of adopting the above technical solution are as follows: By optimizing the preprocessing process of the reference image block, the present invention can quickly and accurately capture and identify the reference image block during the use of the head-mounted AR device, thereby using the relative position of the reference image block and the head-mounted AR device to calculate the position angle of the head-mounted AR device, and then using the calculation result as a calibration standard to correct the cumulative error of the motion state monitoring module of the head-mounted AR device. Attached Figure Description

[0015] Figure 1This is a structural diagram of a specific embodiment of the present invention. Detailed Implementation

[0016] Reference Figure 1 This invention adds an image recognition module to the existing head-mounted AR device for real-time image processing and recognition. Then, the cumulative error of the accelerometer and three-axis gyroscope is corrected by using the relative position of the head-mounted AR device and the reference image block.

[0017] Specifically, it includes: The motion status monitoring module 1 is used to monitor the motion status of the head-mounted AR device in real time. Image recognition module 2 is used to process real-time images captured by the head-mounted AR device and identify reference image blocks; Angle change calculation module 3 is used to calculate the current angle change of the head-mounted AR device based on the changes of the reference image block in the real-time image. The correction module 4 is used to correct the motion state monitoring module 1 using the angle change calculation module 3 to calculate the angle change of the head-mounted AR device.

[0018] A recognition method for an automatic angle recognition system for head-mounted AR devices includes the following steps: A. Motion status monitoring module 1 monitors the motion status of the head-mounted AR device in real time; B. Image recognition module 2 processes the real-time images captured by the head-mounted AR device and identifies reference image blocks; C. Angle Change Calculation Module 3 calculates the current angle change of the head-mounted AR device based on the changes of the reference image block in the real-time image. D. The correction module 4 uses the angle change calculation module 3 to calculate the angle change of the head-mounted AR device to correct the motion status monitoring module 1.

[0019] Among them, the motion state monitoring module 1 uses an accelerometer and a three-axis gyroscope to monitor the motion state of the head-mounted AR device in real time. The motion state includes the linear acceleration and angular acceleration of the head-mounted AR device in the three-dimensional coordinate system.

[0020] The challenge of the angle recognition method disclosed in this invention lies in how to quickly and accurately obtain the angular position of the head-mounted AR device relative to a reference object. Because the image clarity of a head-mounted AR device is insufficient due to interference factors such as changes in lighting and wearer movement when capturing real-world images, directly identifying a reference image block would result in an inaccurate calculated relative position angle between the head-mounted AR device and the reference image block, failing to effectively eliminate the accumulated errors of the head-mounted AR device. Therefore, this invention specifically processes the real-time images captured by the head-mounted AR device.

[0021] Processing real-time images includes the following steps: We eliminate interference from single sampling by superimposing images after multiple samplings. First, we set a sampling time period, sampling the image several times within each period, with each sampling interval less than 1ms. Then, after sharpening each sampled image, we perform a Discrete Fourier Transform (DFT) to obtain a frequency domain dataset. Because image features such as brightness, grayscale, and contrast are displayed in different frequency bands, we convert the image data into frequency domain data for processing. The frequency domain dataset is grouped according to frequency, resulting in high-frequency, mid-frequency, and low-frequency datasets, with the mid-frequency dataset covering more than 50% of the total frequency range. A mapping relationship is established between the high-frequency, mid-frequency, and low-frequency datasets, based on pixel coordinate relationships and grayscale similarity. Since image interference is mainly concentrated in the low-frequency and high-frequency bands, bandpass filtering is performed on the high-frequency and low-frequency datasets respectively. The mapping relationship is used to establish an associated data group including the filtered high-frequency and low-frequency data, and the filtering region is delineated on the sampled image based on the pixel coordinates corresponding to the associated data group. Intermediate frequency (IF) data contains most of the background information and contour features of the original image, so it can be used to enhance the filtered region. A first feature vector set is established between the high-frequency data of the filtered region and the IF dataset of the sampled image; a second feature vector set is established between the low-frequency data of the filtered region and the IF dataset of the sampled image. Linearly correlated feature vectors from the first and second feature vector sets are extracted to form a reference feature vector set. IF data associated with the reference feature vector set is then processed by a generative adversarial network (GAN) for up- and down-frequency enhancement to obtain the high-frequency and low-frequency enhanced data corresponding to the filtered region. The high-frequency enhanced data is then merged with the filtered high-frequency dataset, and the low-frequency enhanced data is merged with the filtered low-frequency dataset. Finally, the frequency domain data from different sampled images are merged, and then an inverse Fourier transform is used to obtain the superimposed sampled image.

[0022] The above preprocessing can effectively improve the clarity of the generated sampled image, thus facilitating the rapid and accurate identification of the reference image block.

[0023] The process of identifying reference image blocks is as follows: mark the image regions in the real-time image obtained in each sampling time period that have not changed position, traverse the marked image regions, mark the closed regions in each image region whose gray-scale uniformity is greater than a preset threshold, select the closed region with the largest area as the reference image block of the corresponding image region, and finally select the reference image block that appears in at least two consecutive sampling time periods as the reference image block of the real-time image.

[0024] Since the reference image block has been accurately identified, we can calculate the angle change of the head-mounted AR device relative to the reference image block based on the position and orientation angle changes of the reference image block in the real-time image over two consecutive sampling time periods. Then, based on the movement state of the head-mounted AR device continuously monitored by the motion state monitoring module 1 and the initial position of the head-mounted AR device, the current angle of the head-mounted AR device relative to the reference image block is calculated. If the calculated result deviates from the angle calculated directly using the reference image block, the calculation result directly using the reference image block is used as the initial position of the head-mounted AR device to initialize and correct the motion state monitoring module 1. Because the calculation process directly using the reference image block does not involve any accelerometer or three-axis gyroscope data, the interference of their accumulated errors can be completely avoided.

[0025] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0026] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A recognition method for an automatic angle recognition system of a head-mounted AR device, the automatic angle recognition system of the head-mounted AR device comprising: The motion status monitoring module (1) is used to monitor the motion status of the head-mounted AR device in real time. The image recognition module (2) is used to process the real-time images collected by the head-mounted AR device and identify reference image blocks; Angle change calculation module (3) is used to calculate the current angle change of the head-mounted AR device based on the change of the reference image block in the real-time image. The correction module (4) is used to correct the motion status monitoring module (1) using the angle change calculation module (3) to calculate the angle change of the head-mounted AR device. Its characteristics include the following steps: A. Motion status monitoring module (1) monitors the motion status of the head-mounted AR device in real time; the motion status monitoring module (1) uses an accelerometer and a three-axis gyroscope to monitor the motion status of the head-mounted AR device in real time, the motion status includes the linear acceleration and angular acceleration of the head-mounted AR device in the three-dimensional coordinate system; B. The image recognition module (2) processes the real-time images acquired by the head-mounted AR device and identifies reference image blocks; the processing of real-time images includes the following steps. B11. Set the sampling time period. In each sampling time period, the image is sampled several times, and the time interval between each sampling is less than 1ms. B12. Sharpen the image for each sample; B13. Superimpose sampled images from the same sampling time period to obtain a processed real-time image; superimposing sampled images includes the following steps. B131. Perform a Discrete Fourier Transform on the sampled images to obtain a frequency domain dataset; group the frequency domain dataset according to frequency to obtain a high-frequency dataset, a mid-frequency dataset, and a low-frequency dataset, with the mid-frequency dataset accounting for more than 50% of the entire frequency range; B132. Establish a mapping relationship between high-frequency datasets, mid-frequency datasets, and low-frequency datasets. The mapping relationship consists of pixel coordinate relationships and gray-level similarity. B133. Perform bandpass filtering on the high-frequency dataset and the low-frequency dataset respectively. Use the mapping relationship to establish an associated data group that includes the filtered high-frequency data and the filtered low-frequency data. Delineate the filtering region on the sampled image according to the pixel coordinates corresponding to the associated data group. B134. Use the intermediate frequency dataset and the corresponding mapping relationship to enhance the information of the frequency domain data in the filtered region; B135. Merge the frequency domain data of different sampled images, and then obtain the superimposed sampled image through inverse Fourier transform; C. Angle Change Calculation Module (3) Calculates the current angle change of the head-mounted AR device based on the change of the reference image block in the real-time image; D. The correction module (4) uses the angle change calculation module (3) to calculate the angle change of the head-mounted AR device to correct the motion status monitoring module (1).

2. The recognition method of the automatic angle recognition system for head-mounted AR devices according to claim 1, characterized in that: Step B134, which enhances the information in the filtered region, includes the following steps: A first feature vector set is established for the high-frequency data of the filtered region and the intermediate-frequency dataset of the sampled image. A second feature vector set is established for the low-frequency data of the filtered region and the intermediate-frequency dataset of the sampled image. Linearly correlated feature vectors from the first and second feature vector sets are extracted to form a reference feature vector set. Intermediate-frequency data related to the reference feature vector set are used to perform up-frequency and down-frequency processing through a generative adversarial network to obtain high-frequency enhanced data and low-frequency enhanced data corresponding to the filtered region. The high-frequency enhanced data is merged with the filtered high-frequency dataset, and the low-frequency enhanced data is merged with the filtered low-frequency dataset.

3. The recognition method of the automatic angle recognition system for head-mounted AR devices according to claim 2, characterized in that: Step B, identifying the reference image patch, includes the following steps: B21. Mark the image regions in the real-time images obtained in each sampling time period that have not undergone positional changes; B22. Traverse the marked image regions, mark closed regions in each image region whose grayscale uniformity is greater than a preset threshold, and select the closed region with the largest area as the reference image block of the corresponding image region. B23. Select a reference image block that appears in at least two consecutive sampling time periods as the reference image block for the real-time image.

4. The recognition method of the automatic angle recognition system for head-mounted AR devices according to claim 3, characterized in that: Step C, calculating the current angle change of the head-mounted AR device, includes the following steps: The angle change of the head-mounted AR device relative to the reference image block is calculated based on the position and orientation angle changes of the reference image block in two consecutive sampling time periods.

5. The recognition method of the automatic angle recognition system for head-mounted AR devices according to claim 4, characterized in that: Step D, calibrating the motion status monitoring module (1) includes the following steps: Based on the motion status of the head-mounted AR device continuously monitored by the motion status monitoring module (1) and the initial position of the head-mounted AR device, the angle of the current head-mounted AR device relative to the reference image block is calculated. If the calculation result deviates from the calculation result in step C, the calculation result in step C is used as the initial position of the head-mounted AR device to initialize and correct the motion status monitoring module (1).

Images