An ar device positioning system and method based on data fusion

Abstract

The application relates to the technical field of device positioning, and discloses an AR device positioning system and method based on data fusion, which comprises a collection module, a common positioning module and a small-space positioning module. The method corresponds to the system. The application realizes accurate positioning of an AR device in different scenes, the collection module collects various data to provide rich information for positioning, the common positioning module fuses sensing and positioning data to generate a common positioning result, and meets the positioning requirements of general scenes; the small-space positioning module fuses the common positioning result, infrared data and high-frequency data according to space data judgment, and improves the accuracy and reliability of small-space positioning; and the problems of limited accuracy of an existing AR positioning system and incapability of adapting to complex scenes are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of device positioning, and in particular to an AR device positioning system and method based on data fusion. BACKGROUND

[0002] In the application field of augmented reality (AR) technology, accurate device positioning is crucial. Currently, existing AR positioning systems have many problems. Mature AR positioning systems have limited accuracy, such as HoloLens 2, HTC VIVE XR, etc., whose absolute position error and absolute rotation error cannot meet the needs of high-precision application scenarios such as military industry, aerospace, and medical treatment. Domestic AR surgical navigation systems are still in the early stages, mainly developed by combining HoloLens with traditional systems, and have defects such as insufficient computing power, low positioning accuracy, and short battery life.

[0003] In the published related patent literature, Chinese patent application publication No. CN118887293A discloses a large-space positioning method, system, and head-mounted device based on feature extraction and a medium, which has certain achievements in large-space positioning, but still has obvious deficiencies. Specifically, the patent application relies only on image data collected by a camera and a preset space feature library for positioning, the data fusion dimension is single, and multi-source sensing data is not fully utilized, resulting in poor positioning reliability in complex environments. Moreover, it is only suitable for large-space scenarios and has poor adaptability to fine operation scenarios such as medical surgery, and does not consider special needs of different industries, lacking the ability to apply in multiple fields.

[0004] In view of the various problems of the prior art, there is an urgent need for a new AR device positioning technology that can fuse multiple data, improve positioning accuracy, and be suitable for multiple industries. SUMMARY

[0005] The purpose of the present application is to provide an AR device positioning system and method based on data fusion to solve the technical problems raised in the background art.

[0006] To achieve the above-mentioned purpose, the present application discloses the following technical solutions:

[0007] In a first aspect, the present application discloses an AR device positioning system based on data fusion, comprising:

[0008] The acquisition module is configured to acquire sensing data and positioning data used for conventional positioning, and to acquire space data used for space size judgment, as well as infrared data and high-frequency data used for small-space positioning. The infrared data is near-infrared band light information, and the high-frequency data is high-frame-rate digital image data.

[0009] The constant positioning module is connected with the acquisition module and is configured to fuse the sensing data and the positioning data to generate a constant positioning result of the AR device.

[0010] The small-space positioning module is connected with the acquisition module and the constant positioning module and is configured to determine whether to use small-space positioning based on the space data, fuse the constant positioning result and the infrared data to generate an infrared positioning result when it is determined to use the small-space positioning, and continuously correct the infrared positioning result by using the high-frequency data to generate a small-space positioning result of the AR device.

[0011] Preferably, the constant positioning result is generated by:

[0012] The sensing data includes data collected by a nine-axis sensor, a light sensor and a distance sensor, and the positioning data includes GPS and Beidou positioning data.

[0013] The sensing data and the positioning data collected at the acquisition time , the predicted sensing data at the prediction time is generated based on the sensing data , the predicted positioning data at the prediction time is generated based on the predicted sensing data , the sensing data and the positioning data collected at the acquisition time , when the position deviation between the predicted positioning data and the positioning data is greater than or equal to a preset position deviation threshold, the predicted sensing data , the sensing data , the predicted positioning data and the positioning data are used to generate and output the constant positioning result.

[0014] Preferably, the predicted sensing data , the sensing data , the predicted positioning data and the positioning data are used to generate and output the constant positioning result, including:

[0015] The constant positioning result is calculated by using a constant position fusion formula in combination with the predicted sensing data , the sensing data , the predicted positioning data and the positioning data , and the constant position fusion formula is:

[0016]

[0017] wherein, is the calculated predicted sensor data and the data deviation of sensor data is the calculated fused constant positioning result.

[0018] As preferred, the generation of the predicted sensor data includes:

[0019] predicting the predicted positioning data at time based on the sensor data at time using a preset device motion model, wherein the device motion model is fitted based on historical sensor data and historical device motion conditions, and the device motion model is updated based on real-time motion data of a wearer of the AR device, and the real-time motion data is synchronously collected using the collection module.

[0020] As preferred, the generation of the predicted positioning data includes:

[0021] determining the predicted positioning data at time based on the predicted sensor data and the real-time motion data using a preset position correlation model, wherein the position correlation model is fitted based on historical positioning data and corresponding historical sensor data and historical device motion conditions.

[0022] As preferred, the generation of the small-space positioning result includes:

[0023] collecting spatial data at time and extracting corresponding spatial features , determining whether is true, if yes, it is determined that small-space positioning is needed, otherwise, it is determined that small-space positioning is not needed, wherein, is a preset small-space feature set, and when small-space positioning is needed, the following steps are performed:

[0024] A1: generating the constant positioning result using the constant positioning module;

[0025] A2: collecting and using the infrared data to capture target objects in the small space, updating the constant positioning result based on the target objects, and generating an infrared positioning result;

[0026] ​A3: Collect and utilize the high-frequency data to continuously correct the infrared positioning result, and generate a small space positioning result.

[0027] As preferred, the collection time point of the spatial data is the time point when the target object is in the small space. , determine whether the following condition is met, including:

[0028] Collect the spatial data at the collection time point using the collection module, extract the distance information and contour information in the spatial data to obtain the spatial feature , calculate the change rate of the spatial feature corresponding to the time sequence , compare the spatial feature and the corresponding change rate with the features in the small space feature set one by one, calculate the similarity between the features, and when the preset similarity threshold is met, determine , otherwise .

[0029] As preferred, step A2 includes:

[0030] Collect infrared data in the small space at the collection time point using a long-focus low-distortion near-infrared lens and obtain the corresponding infrared image, and use a target detection algorithm to identify and capture the target object in the small space from the infrared image;

[0031] Determine the image position of the target object in the infrared image , compare and fuse the image position with the conventional positioning result to update the conventional positioning result, and generate an infrared positioning result , the comparison and fusion are:

[0032]

[0033] wherein, is the calculated conventional positioning result out-of-range update frequency;

[0034] The calculation of the conventional positioning result out-of-range update frequency includes:

[0035] Statistical update the number of times the conventional positioning result is updated using the image position ;

[0036] Statistical out-of-range times when the deviation between the updated conventional positioning result and the conventional positioning result before updating is greater than or equal to a preset deviation threshold ;

[0037] Computing a regular positioning result out-of-range update frequency .

[0038] As preferred, step A3 comprises:

[0039] Using a high-frame-rate high-integration imaging sensor to continuously collect time points at a set high frequency High-frequency data of small spaces within the corresponding collection period, to obtain a continuous image sequence;

[0040] Analyzing the motion trajectory changes of the target object in the image sequence And the attitude changes , and based on the motion trajectory changes And the attitude changes Continuously fuse with the infrared positioning result To generate a small space positioning result ; wherein the fusion is:

[0041]

[0042] Wherein, The number of frames of the image sequence.

[0043] In a second aspect, the application discloses an AR device positioning method based on data fusion, which is suitable for the AR device positioning system based on data fusion as described above, and comprises the following steps:

[0044] S1: Collecting sensing data and positioning data for regular positioning; collecting space data for space size judgment; collecting infrared data and high-frequency data for small space positioning; the infrared data is near-infrared band light information, and the high-frequency data is high-frame-rate digital image data;

[0045] S2: Fusing the sensing data and the positioning data to generate a regular positioning result of the AR device;

[0046] S3: Judging whether small space positioning is needed based on the space data, when it is judged that small space positioning is needed, fusing the regular positioning result and the infrared data to generate an infrared positioning result, and continuously correcting the infrared positioning result using the high-frequency data to generate a small space positioning result of the AR device.

[0047] Beneficial effects: the AR device positioning system and method based on data fusion provided in the application realizes accurate positioning of AR devices in different scenarios; the collection module collects various data to provide rich information for positioning; the common positioning module fuses sensing and positioning data to generate common positioning results, meeting the positioning needs in general scenarios; the small space positioning module fuses the common positioning results, infrared data and high frequency data according to space data to improve the accuracy and reliability of small space positioning; and the problems of limited accuracy of existing AR positioning systems and inability to adapt to complex scenarios are solved. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0049] Figure 1 The structure block diagram of the AR device positioning system based on data fusion provided in the embodiments of the application is shown in the figure.

[0050] Figure 2 The flowchart of the AR device positioning method based on data fusion provided in the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the application will be described clearly and completely below. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0052] In this document, the term "comprising" is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "comprising" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0053] The first aspect of the embodiments discloses an AR device positioning system based on data fusion as shown in the figure, which comprises: Figure 1 The AR device positioning system based on data fusion comprises:

[0054] The collection module is configured to collect sensing data and positioning data for normal positioning, collect spatial data for spatial size judgment, and collect infrared data and high-frequency data for small-space positioning. The infrared data is near-infrared band light information, and the high-frequency data is high-frame-frequency digital image data.

[0055] The normal positioning module is connected with the collection module and is configured to fuse the sensing data and the positioning data to generate a normal positioning result of the AR device.

[0056] The small-space positioning module is connected with the collection module and the normal positioning module and is configured to judge whether small-space positioning is needed based on the spatial data. When it is judged that small-space positioning is needed, the normal positioning result and the infrared data are fused to generate an infrared positioning result, and the high-frequency data is used to continuously correct the infrared positioning result to generate a small-space positioning result of the AR device.

[0057] By the above, the embodiment uses the collection module to collect multi-source data, the normal positioning module to fuse sensing and positioning data, and the small-space positioning module to judge and fuse multi-source data based on spatial data, so as to realize accurate positioning of the AR device in different scenes. The collection module collects various data to provide rich information for positioning. The normal positioning module fuses sensing and positioning data to generate a normal positioning result, which meets the positioning needs in general scenes. The small-space positioning module judges based on spatial data, fuses the normal positioning result, infrared data, and high-frequency data, and improves the accuracy and reliability of small-space positioning. The problems of limited accuracy and inability to adapt to complex scenes of existing AR positioning systems are solved.

[0058] Specifically, the generation of the normal positioning result includes:

[0059] The sensing data includes data collected by a nine-axis sensor, a light response sensor, and a distance sensor, and the positioning data includes GPS and Beidou positioning data.

[0060] The collection time The sensing data and the positioning data The predicted sensing data at the generation time is generated based on the sensing data , and the predicted positioning data at the generation time is generated based on the predicted sensing data , the sensing data and the positioning data collected at the collection time , when the position deviation between the predicted positioning data and the positioning data is greater than or equal to a preset position deviation threshold, the predicted sensing data Sensor data Predictive positioning data and location data Generate and output standard location results.

[0061] Based on the above, this embodiment utilizes sensor data and positioning data collected at different times, combined with predicted data and a preset deviation threshold, to achieve more accurate and reliable generation of conventional positioning results. This is achieved by collecting data at different times... Predicting sensor data and positioning data The system compares real-time data with predicted and actual positioning data. When the deviation reaches a threshold, it uses both predicted and actual data to generate a standard positioning result. This process fully considers the continuity of device movement and the real-time nature of data. Compared to positioning methods that rely solely on currently collected data, it better addresses changes in device movement, reduces positioning errors, improves positioning accuracy, and enhances the adaptability of the positioning system to complex motion scenarios. This provides strong support for the stable positioning of AR devices in conventional scenarios, ensuring accurate location information acquisition in various application scenarios.

[0062] Specifically, the aforementioned use of predictive sensing data Sensor data Predictive positioning data and location data Generate and output standard localization results, including:

[0063] Using conventional location fusion formulas, combined with predictive sensor data Sensor data Predictive positioning data and location data The conventional positioning result is calculated using the following formula:

[0064]

[0065] in, For the calculated predictive sensing data and sensor data Data deviation, This is the calculated, fused, conventional localization result.

[0066] In this embodiment, based on predictive sensing data and actual sensor data The deviation affects the predicted positioning data. and actual location data Perform weighted calculations. When predicting sensor data... With actual sensing data When approaching, If the value is small, then The value is relatively large, at which point the predicted positioning data... In the final result The predicted sensor data carries a relatively large weight. This means that when the predicted sensor data is reliable, the predicted positioning data is given more weight. Based on this, the goal is to avoid excessive deviations in positioning results due to anomalies in the predicted sensor data, making the final positioning result more closely reflect the actual situation.

[0067] In a simple example, in an AR device positioning scenario, a distance sensor collects data to assist in positioning. There is a fixed reference point within the scene, and the AR device uses the distance sensor to obtain distance information to this reference point. At time... Based on the previous motion trajectory and algorithm prediction, the system determined that the predicted sensor data for the AR device's distance from the reference point was 50 cm, while the actual distance measured by the distance sensor was 60 cm. Based on the predicted sensor data, the system inferred that the horizontal coordinate of the AR device's predicted positioning data in the two-dimensional plane coordinate system was 80 cm. Through other positioning methods (such as inertial measurement unit-assisted positioning), the system determined the actual horizontal coordinate of the AR device's positioning data to be 90 cm. Based on this, the calculated conventional positioning result was 81.7 cm. Therefore, it can be seen that by using data collected by the distance sensor and combining it with the formula, a more accurate positioning result can be obtained by integrating the predicted and actual sensor and positioning data.

[0068] Based on the above, this embodiment utilizes a conventional location fusion formula, combining predicted and actual sensing and positioning data, to achieve accurate calculation of conventional positioning results. This formula weights the predicted and actual positioning data according to the deviation between the predicted and actual sensing data. When the deviation is small, the predicted positioning data has a higher weight; when the deviation is large, the actual positioning data has a higher weight. This dynamic weighting calculation method can flexibly adjust the fusion result according to the reliability of the data, fully utilize the advantages of different data, effectively reduce the impact of abnormal data on the positioning result, improve the accuracy and stability of positioning, and enable conventional positioning results to more accurately reflect the actual location of the AR device, meeting the stringent requirements for AR device positioning accuracy in different scenarios.

[0069] Specifically, the predictive sensing data The generation includes:

[0070] Using a preset device motion model, based on time Sensor data For time Predictive positioning data The prediction is made by fitting the device motion model based on historical sensor data and historical device motion data, and the device motion model is updated based on the real-time motion data of the wearer of the AR device. The real-time motion data is collected synchronously using the acquisition module.

[0071] Based on the above, this embodiment utilizes existing deep learning technology to construct a device motion model based on historical data fitting, and combines this with real-time motion data updates to achieve effective generation of predictive sensor data. The device motion model, built upon historical sensor data and device motion patterns, reflects the device's motion characteristics. As real-time motion data from AR device wearers is collected, the model is continuously updated to ensure its timeliness and accuracy. Based on this, according to the time... Predicting time based on sensor data Predictive sensor data can better match the actual movement trend of the device, providing reliable preliminary data support for the prediction of subsequent positioning data and the generation of routine positioning results, improving the entire positioning system's ability to track changes in the device's movement state, and ensuring the timeliness and accuracy of positioning.

[0072] Specifically, the predicted positioning data The generation includes:

[0073] Using a pre-defined location association model, based on predictive sensor data The time is determined by the real-time motion data. Predictive positioning data The location association model is obtained by fitting historical positioning data, corresponding historical sensor data, and historical device movement data.

[0074] Based on the above, this embodiment utilizes existing deep learning technology to construct a location association model based on historical data fitting. By combining predicted sensor data and real-time motion data, it achieves accurate determination of predicted positioning data. The location association model is fitted based on historical positioning, sensing, and device motion data, revealing the intrinsic relationship between sensor data and positioning data. Through this model, predicted positioning data is calculated based on predicted sensor data and real-time motion data, fully considering the impact of device motion state and environmental factors on positioning. This approach can more accurately predict the device's position at the next moment, providing a more reliable reference for generating conventional positioning results, enhancing the foresight and accuracy of the positioning system, and enabling AR devices to achieve high-precision positioning even in complex environments and dynamic scenes.

[0075] Specifically, the generation of the small space positioning result includes:

[0076] Collection time Spatial data and extract corresponding spatial features ,judge whether the condition is met, if yes, determining that small space positioning needs to be used, otherwise, determining that small space positioning does not need to be used, wherein, the preset small space feature set, when the small space positioning needs to be used, the following steps are performed:

[0077] A1: generating the normal positioning result by using the normal positioning module;

[0078] A2: collecting and using the infrared data to capture the target object in the small space, updating the normal positioning result based on the target object, and generating an infrared positioning result;

[0079] A3: collecting and using the high-frequency data to continuously correct the infrared positioning result, and generating a small space positioning result.

[0080] By the above, the embodiment uses the collected space data, extracts features, and compares with the preset feature set, combines the fusion processing of the normal positioning result, the infrared data, and the high-frequency data, and realizes the accurate positioning of the AR device in the small space scene. By collecting the space data to determine whether to enter the small space scene, in the small space, the initial positioning is provided by using the normal positioning result, the positioning is updated by capturing the target object in combination with the infrared data, and the high-frequency data is continuously corrected, the advantages of multiple data sources are comprehensively utilized, and the accuracy and real-time performance of the small space positioning are effectively improved. This solves the deficiencies of the prior art in small space positioning, enables the AR device to accurately work in small space high-precision positioning scenes such as medical surgery and precision instrument operation, expands the application range of AR technology, and promotes the technical development in related fields.

[0081] Specifically, the collection time of the space data and the corresponding space features are extracted , whether the condition is met is determined, including:

[0082] the space data at the collection time is collected by using the collection module, the distance information and the contour information in the space data are extracted to obtain the space features , the change rate of the space features corresponding to the time sequence is calculated , the features in the small space feature set and the corresponding change rate are compared one by one based on the space features , the similarity between the features is calculated, when the preset similarity threshold is met, it is determined , otherwise .

[0083] By the above, the embodiment realizes accurate judgment on small space scenes by using the collected space data to extract distance and contour information, calculating the feature change rate and comparing the similarity with the preset feature set. After the acquisition module obtains the space data, the key features are extracted and the change rate is analyzed, and the similarity between these information and the preset small space feature set is calculated. Compared with simple space data judgment, this multi-dimensional analysis and judgment method can more accurately identify small space scenes and reduce misjudgment. Accurate scene judgment provides a reliable basis for the subsequent small space positioning module, ensuring that the AR device can switch the positioning mode in time when entering the small space scene, improving the intelligence and adaptability of the positioning system, and ensuring the stable positioning of the AR device in different space scenes.

[0084] Specifically, step A2 includes:

[0085] Using a long-focus low-distortion near-infrared lens to collect infrared data in a small space at a moment and obtain the corresponding infrared image, using a target detection algorithm to identify and capture target objects in the small space from the infrared image;

[0086] determining the image position of the target object in the infrared image , comparing and fusing the image position with the conventional positioning result to update the conventional positioning result, and generating an infrared positioning result , the comparison and fusion are:

[0087]

[0088] wherein, is the calculated conventional positioning result out-of-range update frequency;

[0089] The calculation of the conventional positioning result out-of-range update frequency includes:

[0090] counting the number of times of updating the conventional positioning result using the image position ;

[0091] counting the number of times of updating the conventional positioning result using the image position ;

[0092] calculating the conventional positioning result out-of-range update frequency .

[0093] It should be noted that the long-focal-length low-distortion near-infrared lens in this embodiment is a lens of the long-focal-length low-distortion near-infrared optical system independently developed and manufactured by the Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences. It is an optical system designed for fixed wavelength, size, and imaging range markers. It can simplify algorithms and improve the accuracy of back-end processing algorithms. At the optical end, it ensures a large field of view while having low distortion and reducing target deformation. It also utilizes a Fast glass broadband anti-reflective coating to improve light transmittance and ensure data accuracy.

[0094] In the specific application of this embodiment, weighted fusion is used, based on the reliability of the conventional positioning results (the frequency of updating conventional positioning results beyond the range). (This is reflected in the fact that) a weighted sum is performed on the infrared image location information and the conventional positioning results. When When the value approaches 1, it means that the conventional positioning results are frequently updated beyond the acceptable range, resulting in low reliability. At this point, the infrared image position information... It accounts for a larger proportion in the fusion results; when When the value approaches 0, it indicates that the conventional positioning results are relatively stable and reliable. It dominates the fusion result. This dynamic weighting method adaptively combines the advantages of both types of data, improving the accuracy of the positioning results. Furthermore, the number of updates... This refers to the number of times the image location is updated to reflect the regular positioning results. It reflects the activity level of the infrared image location information in updating the regular positioning results, including the number of times the data is out of range. This refers to the number of times the updated and unupdated conventional positioning results deviate from the preset deviation threshold, indicating significant fluctuations in the conventional positioning results after being updated by infrared image location information. The ratio of the two values ​​is calculated to obtain... This can measure the stability of conventional positioning results during the process of updating location information using infrared imagery. If the number of times the range is exceeded... Relative update count A large number of such cases indicates that the routine localization results change significantly after the update and have poor stability. A larger value indicates a greater reliance on infrared image location information during fusion; conversely, a smaller value indicates a lower value, indicating less out-of-range occurrences, and thus a greater reliance on conventional positioning results.

[0095] By the above, the embodiment uses an existing long-focus low-distortion near-infrared lens to collect infrared data, combines an existing target detection algorithm and a conventional positioning result, and fuses by calculating the out-of-range update frequency of the conventional positioning result to achieve accurate generation of an infrared positioning result. The long-focus low-distortion near-infrared lens acquires a clear infrared image, the target detection algorithm identifies the position of a target object, and the image position and the conventional positioning result are weighted and fused according to the out-of-range update frequency of the conventional positioning result. This way combines infrared image information and conventional positioning results, dynamically adjusts the fusion weight according to the stability of the positioning result, effectively improves the accuracy of infrared positioning, provides more accurate initial positioning data for small space positioning, enhances the performance of the small space positioning module, and meets the requirement of small space positioning for high precision.

[0096] Specifically, step A3 includes:

[0097] Using a high-frame-rate high-integration imaging sensor to continuously collect time points at a set high frequency Corresponding high-frequency data of the small space in the collection period, to obtain a continuous image sequence;

[0098] Analyzing the motion trajectory change and the attitude change of the target object in the image sequence and continuously fusing the motion trajectory change and the attitude change with the infrared positioning result to generate a small space positioning result ; wherein the fusion is:

[0099]

[0100] wherein, is the number of frames of the image sequence.

[0101] It should be noted that the high-frame-rate high-integration imaging sensor of the embodiment is a high-frame-rate high-integration CMOS imaging system built by Changguangchen, which is self-developed by the Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences. The signal strength is greater than the noise strength, ensuring that the effective signal can be clearly imaged, and it has the characteristics of high frame rate and high integration, ensuring data update rate and realizing miniaturization and light weight.

[0102] In the specific application of the embodiment, based on the analysis of high-frame-rate imaging data and the fusion with the infrared positioning result, the positioning accuracy of the AR device in a small space is improved, as follows: a high-frame-rate high-integration imaging sensor is used to collect high-frequency data in a small space at a high frequency to form a continuous image sequence. The high-frame-rate feature enables the image sequence to capture the dynamic changes of the target object in the small space more finely; by analyzing these image sequences, the motion trajectory changes and attitude changes of the target object are obtained, which reflect the real-time motion state of the target object in the small space. is the infrared positioning result obtained in the previous step, is the accumulation of the target object motion trajectory changes in all frames of the image sequence, is the accumulation of the target object attitude changes in all frames of the image sequence, is the frame number of the image sequence. The infrared positioning result is used as the basic positioning information, and on this basis, the target object motion trajectory changes and attitude changes obtained by analyzing the high-frame-rate image sequence are superimposed. Since the high-frame-rate image sequence can reflect the dynamics of the target object in real time and in detail, by accumulating these dynamic changes and combining them with the infrared positioning result, the positioning result can be continuously corrected and improved, so that the small space positioning result can more accurately reflect the actual position and state of the target object in the small space. Such a design fully utilizes the advantages of high-frame-rate data and effectively improves the accuracy and real-time performance of small space positioning.

[0103] Based on the above, the embodiment uses existing high-frame-rate high-integration imaging sensors to collect high-frequency data, analyzes the target object motion trajectory and attitude changes, and fuses them with the infrared positioning result, to achieve continuous optimization of the small space positioning result. The high-frame-rate high-integration imaging sensor collects image sequences in a small space at a high frequency, from which the motion and attitude changes of the target object are analyzed. These changes are fused with the infrared positioning result, and the positioning result is continuously corrected as the number of image frames increases. This enables the small space positioning result to reflect the dynamic changes of the target object in real time, greatly improving the accuracy and real-time performance of small space positioning, ensuring that the AR device can achieve accurate positioning in complex motion scenarios in a small space, and meeting the stringent requirements of medical and industrial fields for high-precision positioning in a small space.

[0104] The second aspect of the embodiment discloses an AR device positioning method based on data fusion as shown in Figure 2 , which is applicable to the AR device positioning system based on data fusion as described above, and includes the following steps:

[0105] S1: collect sensing data and positioning data for normal positioning; collect space data for space size judgment; collect infrared data and high-frequency data for small space positioning; the infrared data is near-infrared band light information, and the high-frequency data is high-frame-frequency digital image data;

[0106] S2: fuse the sensing data and the positioning data to generate a normal positioning result of the AR device;

[0107] S3: judge whether small space positioning needs to be used based on the space data, when it is judged that the small space positioning needs to be used, fuse the normal positioning result and the infrared data to generate an infrared positioning result, and continuously correct the infrared positioning result by using the high-frequency data to generate a small space positioning result of the AR device.

[0108] It should be noted that the AR device positioning method based on data fusion in the embodiment corresponds to the AR device positioning system based on data fusion described above, and therefore, the content not specifically described in the AR device positioning method based on data fusion in the embodiment can be, but is not limited to, function definition, working principle and technical effect, and can be referred to the description in the AR device positioning system based on data fusion described above, which will not be described herein.

[0109] In summary, the AR device positioning system and method based on data fusion in the embodiment realize accurate positioning of the AR device in different scenes; the collection module collects various data to provide rich information for positioning; the normal positioning module fuses sensing data and positioning data to generate a normal positioning result, meeting the positioning demand in general scenes; the small space positioning module judges according to space data, fuses the normal positioning result, infrared data and high-frequency data to improve the accuracy and reliability of small space positioning; and the problem of limited accuracy of the existing AR positioning system and the problem that the existing AR positioning system cannot adapt to complex scenes are solved.

[0110] In the embodiments provided by the present application, it should be understood that the embodiments described herein can be realized by hardware, software, firmware, middleware, codes or any proper combination thereof. For hardware implementation, the processor can be realized in one or more of the following components: an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, other electronic units designed to perform the functions described herein, or a combination thereof. For software implementation, the procedures described herein can be implemented with a computer program that directs relevant hardware to complete the procedures. When implemented, the above program can be stored in a computer readable storage medium or transmitted as one or more instructions or codes on a computer readable storage medium. The computer readable storage medium includes computer storage media and communication media including any medium that facilitates the transfer of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a computer. The computer readable storage medium can include but not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer.

[0111] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified or some technical features can be replaced by equivalent ones, and any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A data fusion based AR device positioning system, characterized in that, The application relates to an AR device positioning method and device. The application comprises the following steps: A collection module is used for collecting sensing data and positioning data for normal positioning, and is also used for collecting space data for space size judgment; Infrared data and high-frequency data are collected for small-space positioning; the infrared data are near-infrared waveband light information, and the high-frequency data are high-frame-frequency digital image data; A normal positioning module is connected with the collection module and is used for fusing the sensing data and the positioning data to generate a normal positioning result of an AR device; A small-space positioning module is connected with the collection module and the normal positioning module and is used for judging whether small-space positioning is needed based on the space data; when it is judged that small-space positioning is needed, infrared positioning results are generated by fusing the normal positioning result and the infrared data, and the infrared positioning results are continuously corrected by using the high-frequency data to generate small-space positioning results of the AR device; The generation of the normal positioning result comprises the following steps: collection time sensing data and positioning data , based on the sensing data generate predicted sensing data at the collection time , and based on the predicted sensing data generate predicted positioning data at the collection time , obtain the sensing data and the positioning data collected at the collection time , when a position deviation between the predicted positioning data and the positioning data is greater than or equal to a preset position deviation threshold, generate and output a normal positioning result by using the predicted sensing data , the sensing data , the predicted positioning data and the positioning data ​ The sensing data comprises data collected by a nine-axis sensor, a light response sensor and a distance sensor, and the positioning data comprises GPS and Beidou positioning data; Collection time Spatial data and extract corresponding spatial features ,judge If the condition is true, then it is determined that small-space positioning is needed; otherwise, it is determined that small-space positioning is not needed. Given a pre-defined small spatial feature set, when small spatial localization is required, the following steps are performed: The generation of the small-space positioning result comprises the following steps: A1: the normal positioning result is generated by using the normal positioning module; A2: target objects in a small space are captured by using the infrared data, the normal positioning result is updated based on the target objects, and infrared positioning results are generated; A3: the infrared positioning results are continuously corrected by using the high-frequency data, and small-space positioning results are generated; Capturing a time instant using a long focal low distortion near infrared lens Infrared data within the small space is collected and a corresponding infrared image is obtained, and a target detection algorithm is used to identify and capture target objects within the small space from the infrared image; Determining image position of target object in infrared image , generating infrared positioning result based on the image position and comparing and fusing the constant positioning result to update the constant positioning result, and generating infrared positioning result , the comparison and fusion are: wherein, is a calculated constant bit result out-of-range update frequency; The conventional register result out-of-range update frequency computation, including: The statistics update the number of times the conventional bit result is updated using the image position ; The number of times that the deviation of the updated normal positioning result from the normal positioning result before the update is greater than or equal to a preset deviation threshold ; Computing a conventional bit result out-of-range update frequency ; The collection time of spatial data and extract the corresponding spatial features , determine whether it is true, including: Time is collected using the acquisition module. The spatial features are obtained by extracting distance and contour information from the spatial data. Calculate the spatial features The corresponding rate of change in the time series Based on spatial features and the corresponding rate of change In the small space feature set The features in the dataset are compared one by one, and the similarity between features is calculated. When a preset similarity threshold is met, a decision is made. ,otherwise ; Step A2 comprises the following steps: Using high frame rate high integration imaging sensor to continuously collect time point at set high frequency Corresponding high frequency data of small space in collection period, get continuous image sequence; analyze a motion trajectory change of a target object in the image sequence and a pose change , and based on the motion trajectory change and the pose change continuously fuse with the infrared positioning result to generate a small space positioning result ; wherein the fusion is: wherein, is the number of frames of the image sequence.

2. The data fusion based AR device positioning system of claim 1, wherein, The described utilizing predicted sensor data , sensor data , predicted positioning data and positioning data , generating and outputting a normal positioning result, comprising: using a conventional position fusion formula, in combination with predicted sensor data , sensor data , predicted positioning data and positioning data , to calculate a conventional positioning result, said conventional position fusion formula being: wherein, is the computed predicted sensor data and the data bias of the sensor data is the computed fused constant bit result.​ 3. The data fusion based AR device positioning system of claim 1, wherein, the predicted sensor data generation, comprising: Using a preset device motion model, based on time Sensor data For time Predictive positioning data The prediction is made by fitting the device motion model based on historical sensor data and historical device motion data, and the device motion model is updated based on the real-time motion data of the wearer of the AR device. The real-time motion data is collected synchronously using the acquisition module.

4. The data fusion based AR device positioning system of claim 3, wherein, The predicted positioning data comprises: using a predetermined position correlation model based on predicted sensor data and the real-time motion data to determine predicted positioning data at the time instance wherein the position correlation model is fitted based on historical positioning data and corresponding historical sensor data and historical device motion.

5. The data fusion based AR device positioning method, applicable to the data fusion based AR device positioning system according to any one of claims 1-4, characterized in that, Step A3 comprises the following steps: The application comprises the following steps: S1: sensing data and positioning data for normal positioning are collected; space data for space size judgment is collected; infrared data and high-frequency data for small-space positioning are collected; the infrared data are near-infrared waveband light information, and the high-frequency data are high-frame-frequency digital image data; S2: the sensing data and the positioning data are fused to generate a normal positioning result of an AR device; S3: whether small-space positioning is needed is judged based on the space data; when it is judged that small-space positioning is needed, infrared positioning results are generated by fusing the normal positioning result and the infrared data, and the infrared positioning results are continuously corrected by using the high-frequency data to generate small-space positioning results of the AR device.

Images