3D positioning using 3D imaging, lidar, IMU, and 3D ultrasound

Abstract

The invention focuses on enhancing indoor positioning and navigation through advanced synchronous fusion methodologies that integrate various technologies, including Inertial Navigation Systems (INS), Inertial Measurement Units (IMU), LiDAR, stereo vision, and Sonar. This comprehensive system aims to achieve high-precision data collection with centimeter-level accuracy, crucial for creating detailed three-dimensional indoor maps. A significant aspect of this innovation is its ability to address privacy concerns by incorporating measures that abstract sensitive segments of the environment when necessary, ensuring personal privacy is maintained during data collection. Additionally, the system tackles challenges such as voids in stereo imaging and temperature variations in ultrasound and LiDAR measurements through sophisticated algorithms and corrections, including back propagation techniques that utilize stereo depth data. By employing particle swarm optimization and Neuro-Fuzzy algorithms, the framework enhances the accuracy and reliability of navigation in complex environments, making it particularly beneficial for emergency responders and personal navigation in unfamiliar settings.

Description

Title of the Invention:Enhancement Of The 3D Indoor Positioning by Augmenting a Multitude of 3DImaging, LiDAR Distance Corrections, 1ML Sensors and 3-D UltrasoundBackground:

[0001] The article (EL-SHEIMY ; LI, 2021) Examines the present condition of indoor Positioning, Localisation, and Navigation (PLAN) systems, emphasising requirements, key stakeholders, sensors, and methodologies. It underscores the significance of ambient perception sensors such as HD maps, LiDAR, cameras, 5G, and loT signals. The article forecasts that PLAN systems would enhance in intelligence and robustness owing to sophisticated sensors, information fusion across several platforms, devices, and sensors, self-learning capabilities, and integration with artificial intelligence, 5G, loT, and edge / fog computing. It evaluates contemporary sensors and methodologies, encompassing position-faulting, dead-reckoning, database matching, multi-sensor fusion, and motion restrictions. The authors report emerging themes encompass enhancements in sensors, integration of multi-platform, multi-device, and multi-sensor information fusion, selflearning algorithms, 5G / IoT / edge computing convergence, and high- definition maps for indoor planning.

[0002] Indoor positioning technology has emerged as an essential tool in expansive venues such as hospitals, airports, and train stations, significantly enhancing the navigation experience for users. Nevertheless, the inherent limitations of two-dimensional maps often lead to confusion among users. To mitigate this challenge, an augmented reality navigation system (HUANG;HSU; CHU; WU, 2020) known as ARBIN has been developed. This innovative system projects navigation instructions directly onto the user's actual environment, thereby removing the necessity to align a 2D map with the physical surroundings. Research conducted at the National Taiwan University Hospital YunLin Branch demonstrated that ARBIN can achieve an accuracy of 3 to 5 meters, effectively guiding users to their desired locations. Additionally, the system tackles the intricacies associated with diverse Lbeacons by employing four distinct RSSI models. Internal evaluations indicated that ARBIN provided accurate directional guidance in 92.5% of cases. Future enhancements are planned to integrate landmark objects into the real-world environment and to deliver advertising messages through a connected information system. The constraint of this study arises from the necessity to possess the deployment map prior to use, additionally the system excludes the recognition of landmarks.

[0003] The Level 5 Indoor Navigation by (L5IN) (SCHULDT; SHOUSHTARI; HELLWEG; STERNBERG, 2021) initiative seeks to enhance indoor navigation capabilities by incorporating 5G technology into current smartphone frameworks. This initiative is focused on developing a dedicated application that facilitates pedestrian navigation within buildings, utilizing open interfaces to streamline the navigation process. A significant challenge faced by the project is the absence of pre-existing indoor maps, which necessitates the exploration of innovative methods for map generation. To address this, Building Information Modelling (BIM) is suggested as a means to aggregate and manage data essential for accurate positioning and visualization. The project aspires to achieve a standard deviation of 2-3 meters and a 68% confidence interval, leveraging smartphone sensors in conjunctionwith 5G technology. The deployment of 5G will occur on the HafenCity University campus, which will serve as a testbed for the project's development and experimentation. The research underscores the critical need to integrate geometrical, topological, and semantic data into a unified BIM framework to support the navigation system effectively. Additionally, the project includes the creation of an information delivery manual to facilitate efficient data export. The outcomes of the L5IN project highlight the potential of 5G technology to facilitate real-time indoor positioning and navigation. By combining 5G with sensor fusion techniques and BIM, the project envisions the creation of a standardized mobile application that will be compatible with both iOS and Android platforms, ultimately providing users with a reliable and intuitive 2-D and 3-D navigation experience. The authors state the necessity of pre-processed data such as building plans, images, and point clouds.

[0004] A novel indoor navigation system tailored for pedestrian use has been developed by (ABDALLAH; JAO; KASSAS; SHKEL, 2022) through the application of signal sensors. This sophisticated system integrates a footmounted INS with a zero-velocity update (ZUPT) technique, therefore significantly minimising cumulative errors. Furthermore, it employs LTE data into a synthetic aperture navigation (SAN) architecture enhanced by deep neural networks, harnessing the pedestrian's movement to reduce mistakes caused by multipath interference. The suggested DNN-SAN-LTE-ZUPT-INS (DUALS) system amalgamates the benefits of these subsystems via two separate architectures: loosely linked and tightly connected. The tightly coupled DUALS configuration exhibited a three-dimensional position RMSE of 1.34 meters at 600 meters travelling, outperforming the loosely coupledDUALS, ZUPT-assisted INS, and LTE-DNN-SAN systems. This result suggests that the tightly coupled DUALS framework is the most effective among the evaluated systems, attaining a 3-D position RMSE of approximately 1.34 meters. Additionally, the performance of the proposed standalone ZUPT-aided INS framework significantly exceeds that of existing ZUPT methodologies, which typically demonstrate around 1% accumulated positional errors in relation to the distance travelled. While the combination of Long-Term Evolution (LTE) signals with the ZUPT-aided INS approach yielded only slight improvements, it is anticipated that the errors associated with the ZUPT-aided INS will gradually escalate over time. The authors illustrate the integration of INS, altimeter, IMU, and Bluetooth technology into footwear necessitates the use of specialized shoes, raising concerns regarding their accessibility and practicality for the general population. However, such advancements may prove particularly beneficial for firefighters and personnel engaged in indoor operations, where the unique functionalities of these technologically enhanced shoes could enhance performance and safety.

[0005] The rapid advancement of technology in smart city networks has rendered navigation within intricate buildings increasingly challenging. The indoor GPS operates at ultra-high radio frequencies, which can be significantly hindered by thick walls, leading to degraded signal quality. The absence of GPS signals and the inability to maintain line of sight with satellites complicate indoor navigation compared to outdoor settings. While radiofrequency (RF) signals, Assisted GPS, and sensor-based solutions are employed for indoor tracking, they present various limitations. The article (ALKADY; RIZK; ALSEKAIT; ALLUHAIDAN et al., 2024) introduces theSmart Indoor Navigation System Using Augmented Reality (SINS_AR) as a viable solution to these challenges, leveraging readily available resources. The system facilitates the identification of locations such as retail stores in shopping malls, hotel rooms, or university lecture halls, offering users accurate visual guidance via their smartphones to navigate to their chosen destinations. SINS_AR utilizes augmented reality and 2-D visual markers (QR codes) strategically placed throughout the building; when scanned with the application, these markers display directional arrows on the screen to guide users. Additionally, the application enables users to select and modify their intended destinations seamlessly. The authors demonstrate the incorporation of QR code readers for indoor navigation, highlighting the constraints associated with sensor-based solutions.

[0006] The article (KIM; JUN, 2008) introduces a location positioning system that leverages vision-based technology and augmented reality to facilitate indoor navigation. By analysing sequences of images captured within indoor settings, the system identifies specific locations and superimposes relevant location data onto the user's visual field. It operates by utilizing a pre- established database of images and a corresponding location model, while relying on remote computers for tasks such as marker detection, matching image sequences, and recognizing locations. When evaluated in an indoor environment, the system demonstrated a success rate of 89%. The potential applications of this system extend to a wide range of consumer-oriented uses.

[0007] The research (GUPTA; KEJRIWAL; PALLAV; HASSAN et al. , 2016) introduces an innovative method for indoor mapping and localization tailored for Augmented Reality (AR) devices, which integrates inertial sensors withvisual odometry techniques. The methodology is exemplified through the use of Google Glass (GG) and Google Cardboard (GC), in conjunction with an Android smartphone. The Extended Kalman Filter (EKF) is employed for the fusion of sensor data, utilizing Bag of Visual Words Pairs (BoVWP) for image matching to enhance the accuracy of the fused odometry through bundle adjustment. The effectiveness of this approach was empirically tested across three distinct indoor environments within an office setting. Additionally, the study evaluates the discrepancies between the generated map and the actual ground-truth map of area R, which measures 4.4 x 3.8 m2. The topological map derived from the path of R is adjusted in scale and position to align with the ideal trajectory, with participants instructed to complete a single loop of R. The integrated data produces a compelling map overlay. The wearable AR device demonstrates an average processing time of 0.2 seconds to update the map overlay with the new node location. The findings are promising, as the GG operates at 5 frames per second, and the generated topological maps closely resemble the expected user path. The proposed framework for localization and navigation is designed for wearable AR devices, incorporating EKF-based fusion of visual odometry and accelerometer data to facilitate the creation of robust and dynamic topological maps with bundle adjustments. This paper addresses the challenge posed by the limited processing capabilities of the GG, which has been noted in the context of realtime applications, by presenting a functional prototype. The suggested approach relies exclusively on Google Glasses as an imaging device, lacking altitude positioning capabilities, which results in a two- dimensional positioning system. Furthermore, there is no documentation regarding the positioning errors expressed in measurable distance units.

[0008] The study by (SINHA; LEE; RIM; HWANG, 2019) introduces data augmentation methods designed to improve deep learning models for identifying user locations within indoor settings through mobile phone tracking and electronic markers sourced from positioning and access points. The focus of the research is on indoor localization that achieves both rapid processing suitable for real-time applications and resilience to variations in environmental factors. To extract features from a variety of Wi-Fi signals and to create pertinent characteristics, an offline four-layer convolutional neural network (CNN) is employed. An offline four-layer CNN is utilized to extract features from diverse Wi-Fi signals and to develop relevant characteristics. The real-time evaluation of these strategies yields an accuracy rate of 89.73%. This suggests that the combination of Scheme and a CNN exhibits a heightened ability to handle the fluctuations and inconsistencies of RSSI within complex indoor settings, thus improving its efficacy for indoor positioning. Future investigations will aim to refine the algorithm for comprehensive application across entire buildings, enabling the system to learn detailed maps of individual structures and to develop a unified classification model that can accurately predict the user's location within various local areas. The results of the work indicate that the suggested augmentation techniques can achieve a test accuracy of 89.73%, accompanied by an average localization error of only 2.54 meters, provided that a predefined map is utilized. Furthermore, additional 2D work is necessary for comprehensive analysis of entire buildings.

[0009] The work by (XIA; LU; LU; ZHANG et al., 2024) introduces an innovative mobile production monitoring system, termed the "actively pushing" mobile production monitoring system (AIMPMs), which integratesaugmented reality (AR) and indoor positioning technologies. By employing a fusion of ultra- wideband (UWB) and inertial measurement unit (IMU) technologies, the system delivers precise indoor positioning data pertinent to various production components within a manufacturing environment. A streamlined indoor map is developed, and a nearest-neighbor decision algorithm enhanced with a double rejection decision (NN-DRD) is proposed to activate virtual monitoring information. The AIMPMs has been implemented in a factory setting to assess its efficacy alongside a human factors evaluation index system and model. The experimental findings indicate that the deployment of AIMPMs significantly reduces both physiological and mental fatigue among production personnel, thereby fostering a more intelligent and user-friendly human-computer interaction framework. The horizontal positioning trajectories in the X and Y dimensions, as determined by the UWB-IMU fusion positioning system, closely correspond to the intended routes, achieving an accuracy of 0.02 meters in areas with minimal obstruction. However, the accuracy of the elevation measurement in the Z direction exhibits considerable variability, influenced by the degree of occlusion present. While the approach suggested significantly reduces physiological and mental fatigue among personnel and achieves high accuracy in horizontal positioning, it faces drawbacks in elevation measurement accuracy, which varies considerably due to occlusion. This variability can hinder the system’s overall effectiveness in certain scenarios.

[0010] The research (JIANG; SUBAKTI, 2023) presents the Indoor Locationbased Augmented Reality Framework (ILARF), which aims to improve indoor augmented reality (AR) systems. This framework is composed of threeprimary elements: the Indoor Localisation Unit (ILU), the Secure Context- Aware Message Exchange Unit (SCAMEU), and the Augmented Reality Visualisation and Interaction Unit (ARVIU). The ILU employs a combination of markers and device sensors to ascertain the user's position, achieving localization precision with reported inaccuracies ranging from 0.4 to 1.75 meters. The SCAMEU ensures secure data transmission through MQTT and HTTP protocols, while the ARVIU oversees user interactions and presents pertinent information. To illustrate the functionalities of ILARF, a prototype application named Gym Augmented Reality (GAR) was created, enabling users to establish profiles and receive instructions on gym equipment usage. GAR surpasses comparable systems by offering improved user location accuracy and secure communication capabilities. Nonetheless, GAR remains in its prototype stage, facing constraints such as being evaluated in a controlled setting rather than an actual gym environment. Plans for future enhancements include increasing user engagement and incorporating features such as path planning and three-dimensional exercise demonstrations. In summary, although ILARF holds significant potential for advancing indoor AR systems, it encounters obstacles related to localization accuracy and necessitates further refinement of its components.

[0011] Recent advancements in computer vision and mobile sensor technology, such as mobile LiDAR and RGB-D cameras, are enhancing applications in Augmented Reality (AR), robotics, indoor GIS, and selfdriving. A novel camera localization workflow (LI; WANG; KANG; ZHAO, 2020) has been developed, utilizing a highly accurate 3D prior map optimized through RGB-D SLAM and deep learning techniques. This workflow integrates an AR registration method with a game engine, aligning real andvirtual cameras to improve AR application development. Experimental results indicate that the localization accuracy achieves an average error of 35 cm, with a fine-tuned 3D feature database providing 3 cm accuracy compared to a ground-truth 3D LiDAR map. The study also highlights how localization accuracy impacts the visual quality of AR overlays, facilitating effective AR applications such as fire emergency response in a Virtual Geographic Environment. The enhanced localization system will achieve greater robustness and efficiency through the incorporation of short-term relative positioning techniques such as radar or ultrasound alongside geometric characteristics obtained from LiDAR and RGB-D sensors, specifically for augmented reality applications and indoor navigation.

[0012] A comprehensive examination by (SINGH; TYAGI; SINGH; ALI et al., 2024) of vision, inertial sensors, magnetic, and radio frequency -based technologies is provided, offering an in-depth understanding of their intricate concepts, accuracy levels, applications, challenges, and potential for integration. This research addresses various facets of Indoor Positioning Systems (IPS), including wayfinding, map matching, and the navigation of mobile robots. It also discusses the obstacles encountered when implementing IPS algorithms and methodologies in practical scenarios. The investigation highlights the critical significance of location data in bridging the digital and physical environments, noting that the advancement of IPS has significantly enhanced the precision and scope of indoor positioning, evolving from traditional technologies like RFID to more sophisticated sensor fusion techniques. Additionally, the study systematically reviews existing literature on IPS, adhering to the PRISMA framework, employing stringent inclusion and exclusion criteria, and conducting thorough database searches to ensure adetailed analysis of indoor positioning methods, technologies, algorithms, and related dimensions. The findings reveal innovative algorithms, a variety of positioning techniques, practical applications, and emerging opportunities within the IPS domain. By synthesizing previous research outcomes and addressing existing knowledge gaps, this work contributes to a deeper understanding of IPS. The incorporation of technologies such as machine learning and Wi-Fi has revolutionized IPS applications, while future research should prioritize addressing data security, privacy issues, and adherence to industry standards to ensure the ethical and secure deployment of indoor positioning technologies.

[0013] Indoor positioning systems have gained significant traction, facilitating the accurate determination of a user's location within a designated area. The efficacy of these systems, particularly through the utilization of the geomagnetic field as a distinguishing feature, is remarkable, largely due to its ubiquitous natural presence, which eliminates the necessity for specialized infrastructure. To create a distinct relationship between the sensor data collected at a given location and the reference data utilized by the system, even in settings where the geomagnetic field shows minimal fluctuations, a Recurrent Neural Network has been implemented (VINCIGUERRA; RUSSO; PALESI; ASCIA etal., 2024) . This process entails feeding the user's trajectory into the network. To mitigate the expenses associated with dataset collection, a simulation-augmented methodology is adopted. Additionally, the output from the proposed algorithm is integrated with data from an inertial measurement unit. Consequently, the system achieves an average error margin of approximately 0.4 meters in simulated environments and 1.6 meters in practical applications. Drawbacks of the indoor positioning systems includethe reliance on the geomagnetic field, which may have minimal fluctuations in certain environments, potentially affecting accuracy. The implementation of a Recurrent Neural Network requires a well-defined relationship between sensor and reference data, which can be complex to establish. Although a simulation-augmented methodology helps reduce dataset collection costs, it may not fully replicate real-world conditions, leading to discrepancies in accuracy. Additionally, while the system achieves a low average error margin in simulations, the higher error margin of 1.6 meters in practical applications indicates limitations in real -world performance.

[0014] Location fingerprinting that employs Received Signal Strength Indicators (RSSIs) has emerged as a widely adopted technique for indoor localization, capitalizing on the pre-existing Wi-Fi infrastructure. The precision of this approach can be significantly improved through the application of artificial intelligence, particularly Deep Neural Networks (DNNs), which necessitate substantial volumes of pre-processed and annotated data for optimal training. However, the process of gathering such data can be labor-intensive and fraught with difficulties, particularly in contexts like the COVID-19 pandemic. To address these obstacles, the research (TANG; LI; KIM; SMITH, 2024) investigates the augmentation of multi-dimensional RSSI data through Multi-Output Gaussian Process (MOGP), which leverages the interdependencies among RSSIs from various access points situated across different floors or buildings. Experimental results indicate that MOGP-based data augmentation is effective within a hierarchical indoor localization framework that utilizes a Recurrent Neural Network (RNN) in conjunction with the UJIIndoorLoc database. The RNN model, which was trained on MOGP-augmented data sourced from a singlebuilding, achieved a three-dimensional localization error of 8.42 meters, surpassing the performance of alternative augmentation techniques. The drawbacks of using location fingerprinting with Received Signal Strength Indicators (RSSIs) for indoor localization include the labor-intensive and challenging process of collecting the large amounts of pre-processed and annotated data required for training Deep Neural Networks (DNNs), especially exacerbated by circumstances. Additionally, while the Multi - Output Gaussian Process (MOGP) method shows promise in augmenting RSSI data and improving localization accuracy, it may still face limitations in scalability and generalizability across different environments, which could hinder its effectiveness in diverse indoor settings. The resulting three-dimensional localization error of 8.42 meters, while better than other techniques, still indicates that there are inherent limitations in accuracy that need to be addressed for more precise indoor localization.

[0015] The Xavier Institute of Engineering (PARAB; DESHPANDE; THAKUR; WARKE et al., 2024) has developed an indoor navigation system using LiDAR sensors and QR code technology for precise location mapping. The system creates high-resolution 3D maps of indoor environments, providing detailed spatial information. QR codes are strategically placed throughout the campus, acting as reference points for precise localization. Users' smartphones scan these codes, collecting real-time LiDAR data to pinpoint their location. The system offers an intuitive and efficient way for students, faculty, and visitors to navigate complex college buildings. The LiDAR-based mapping ensures precise indoor positioning, while QR codes serve as reliable landmarks for seamless orientation. The system's user-friendly interface provides step-by-step directions, optimizing routes based on real-time data. This hybrid approach demonstrates the potential of transforming indoor navigation experiences, improving campus accessibility, and ensuring a positive user experience. Despite the promising features of the advanced indoor navigation system, there are notable drawbacks, particularly concerning potential errors in location accuracy. The reliance on QR codes for localization may lead to issues if users fail to scan them correctly or if they are obstructed, resulting in inaccurate positioning. Additionally, the effectiveness of LiDAR sensors can be compromised in cluttered environments or when faced with reflective surfaces, which may distort the 3D mapping. Furthermore, the system’s dependency on users’ smartphones for data collection could introduce variability in performance based on device capabilities, leading to inconsistent user experiences. These factors could hinder the overall reliability and effectiveness of the navigation system, potentially frustrating users rather than enhancing their navigation experience.

[0016] The invention ( £| o 2024) relates to an indirect positioning device with a first horizontal portion with a first GNSS antenna installed at one end (P 1), a second horizontal portion with a second GNSS antenna installed at one end (P 2), and a vertical body portion connected to a point (P d) where the other ends of the first horizontal portion and the second horizontal portion meet. A method for measuring a distance (L 1) is provided, allowing a laser measurement module to aim at a positioning target object using the measurement results of a first GNSS antenna and a second GNSS antenna. The laser measurement module measures a distance (L 1) by aiming at apositioning target object, calculates an azimuth (A t) at which the laser measurement module aims at the target object using the measurement results of the first and second GNSS antennas, and calculates the coordinates (P t) of the positioning target object using the distance (L I) and azimuth (A t) measured by the laser measurement module. The drawbacks of the described indirect positioning device include potential complexity in setup and alignment due to the need for precise positioning of multiple components (GNSS antennas and laser measurement module). Additionally, reliance on GNSS signals may lead to inaccuracies in environments with poor satellite visibility or interference. The device’s performance could be affected by environmental factors, such as obstructions or atmospheric conditions, which may hinder the effectiveness of both GNSS measurements and laser distance calculations. Furthermore, the mechanical structure may introduce stability issues, impacting measurement reliability.

[0017] The invention2024) pertains to an indoor navigation system based on AR equipment. It consists of a mobile terminal and multiple positioning identifiers in the navigation space. The mobile terminal stores a navigation map and identifies the target position when navigation is started. It then determines the current position in the map and a travel track based on the current position, the map, and the target position. The mobile terminal also acquires azimuth data from a gyroscope, generates a prompt direction based on the azimuth data and the traveling track, and displays the prompt direction. The benefits of this system include accurate planning of a navigation path and direction, avoiding interference frompositioning signals in complex environments, and achieving a better navigation effect. Drawbacks of the indoor navigation system based on AR equipment include potential limitations in accuracy due to reliance on positioning identifiers, which may not always be effectively detected in certain environments. Additionally, the system’s performance could be hindered by the quality of the mobile terminal’s gyroscope, as inaccuracies in azimuth data may lead to incorrect prompt directions. Furthermore, the need for a pre-established navigation map may limit flexibility in dynamic or changing environments, and the system may struggle in areas with insufficient positioning identifiers. Lastly, the complexity of setup and maintenance of the positioning identifiers could pose challenges in practical implementation.

[0018] The invention (Tr o" l ; 0| ol’’ ’;2024) discloses a method and device for determining the location of a terminal based on movement attributes. This method involves estimating the terminal's distance and direction from its previous location, identifying the distance and direction ranges, including an error for the distance and direction, estimating a potential terminal location, and determining the current terminal's location based on whether the candidate location is within the distance and direction ranges. The method and device for locating a terminal based on movement attributes present several drawbacks. Firstly, the reliance on sensor data and location history may lead to inaccuracies, especially in environments with poor signal reception or interference. Additionally, the estimation of movement distance and direction can be affected by various factors such as user behavior, which may not always be predictable. The inclusion oferror margins in the distance and direction ranges could result in ambiguity, making it challenging to pinpoint the terminal’s exact location. Furthermore, the method may struggle in dynamic settings where rapid changes in movement occur, potentially leading to delays or errors in location determination.

[0019] The disclosure (YU; ZHENG; ZHANG; CUI et al., 2024) presents techniques for ascertaining one or multiple states of an autonomous vehicle during its transition between various environments. The system comprises a positioning mechanism that produces positional data indicative of the vehicle's current location, along with measurement uncertainties linked to previous states and data regarding the acquisition time. A processor in communication with the positioning system is operable to receive positional data from each subsystem, pre-process it based on a common frame of reference, generate a plurality of estimated current states, determine weighting values for the subsystems based on the estimated current states, and apply a respective weighting value to a corresponding estimated current state generated for the positional data collected by the positioning subsystem to estimate the position of the autonomous vehicle when transitioning between the two or more different environments. The processor is configured to generate corresponding maps based on sensor data generated by each of the sensors and generate the positional data based on a comparison between a current map and a previously generated map. In some embodiments, the processor is operable to determine localization information for the autonomous vehicle based on the weighted positioning subsystems, including vehicle linear position, linear velocity, orientation, and angular velocity. The method further comprises operating the processor to normalize coordinates of the positional data to the common frame of reference, pre-process eachpositional data to remove noise, and estimate the current state of the autonomous vehicle based on a vehicle dynamic model and a current time stamp. The described system for determining the states of an autonomous vehicle during transitions between different environments has several drawbacks. Firstly, the reliance on multiple positioning subsystems, such as GNSS, cellular, and inertial measurement units, can lead to inconsistencies and inaccuracies due to varying performance levels and environmental conditions affecting each subsystem differently. Additionally, the complexity of integrating and processing data from diverse sources may introduce latency and computational challenges, potentially impacting real-time decision-making. The need for extensive pre-processing, including noise removal and normalization to a common frame of reference, adds further layers of complexity and may not always guarantee improved accuracy. Moreover, the system’s dependence on prior states and measurement uncertainties could lead to cumulative errors, especially in dynamic environments where rapid changes occur. Lastly, the requirement for generating and comparing maps may limit the system’s adaptability in unfamiliar or rapidly changing environments, potentially hindering the vehicle’s performance and safety.

[0020] The proposed inventionet al., 2023) addresses the challenges of seamless navigation positioning in both indoor and outdoor settings by introducing a method that integrates environmental information with a multi-source sensor elastic fusion approach. This innovative technique is particularly advantageous for navigation platformsequipped with various types of environmental sensing and positioning sensors, as it enhances the ability to navigate continuously and stably within complex environments. By leveraging environmental data, the method identifies the appropriate sensor type, assesses the impact of distance discrepancies, and evaluates the environmental influence factor based on the positioning errors experienced by the sensors. Consequently, it facilitates a robust multi-source sensor elastic fusion navigation positioning system that is informed by environmental context. The advantages of this invention are manifold, notably in enhancing the stability and continuity of seamless positioning across different environments. The method allows for the flexible selection of sensor information and fusion weighting coefficients in response to changing environmental conditions, thereby further improving navigation reliability. Implemented through an elastic fusion weighting mechanism that is supported by environmental data, this approach is characterized by its low computational demand and straightforward engineering application. The invention is comprehensively detailed through a flow chart and accompanying descriptions, illustrating its operational framework and benefits. Despite its advantages, the proposed invention may face drawbacks such as potential limitations in sensor compatibility, which could hinder its effectiveness in diverse environments. Additionally, reliance on environmental data may introduce variability in performance due to fluctuating conditions, potentially affecting navigation accuracy. The complexity of integrating multiple sensor types and managing their fusion weighting could also pose engineering challenges, potentially complicating implementation in real-world scenarios. Lastly, while the method claims low computational demand, the actual performance may vary depending on the specific application and environmental factors.

[0021] The disclosure2023) relates to a method, device, equipment, and storage medium for determining a coordinate system conversion relation. The method involves acquiring indoor pictures and sensor information, determining pose information, determining attitude alignment relationships, determining translation and scale alignment relationships, and determining posture and translation alignment relationships. The information is acquired through a sensor, reducing data and labour costs. The method also includes determining a pose alignment relationship, rotational alignment relationship, and scale alignment relationship. The method also considers the z-axis and x-axis rotation relationships based on pose information and accelerometer information. The method also considers the z-axis rotation relationship of the acquisition terminal relative to the geographic coordinate system. The method also determines a plurality of first three-dimensional position coordinates of the acquisition terminal in the geographic coordinate system when a plurality of indoor pictures are acquired based on the network positioning information. This method aims to improve the accuracy and efficiency of coordinate system conversion relations. The present invention provides a method for determining scale information between indoor visual map coordinate systems and geographic coordinate systems. It includes an indoor positioning method, an indoor path planning method, and a coordinate system transformation relationship determining apparatus. The apparatus includes an acquisition module that acquires indoor pictures and sensor information, determining pose information, posture alignment, translational alignment, scale alignment, gesture alignment, and translation alignment. The functions may beimplemented by hardware or software. The apparatus may also include a memory for storing computer instructions and a processor for execution. The method can be implemented by an electronic device, a computer readable storage medium, or a computer program product. The technical scheme provides benefits such as reducing manual participation and data costs, as the network positioning information, accelerometer information, and magnetometer information are acquired through the sensor of the acquisition terminal when the indoor picture is acquired. The method for determining coordinate system conversion relations has several drawbacks. Firstly, it relies heavily on the accuracy of sensor data, such as accelerometer and magnetometer information, which can be prone to errors and inaccuracies, potentially leading to misalignment between coordinate systems. Additionally, the process requires multiple indoor pictures and corresponding sensor data, which may not always be readily available, complicating the implementation in certain environments. Furthermore, while the method aims to reduce manual participation, it may still necessitate some level of human oversight to ensure data quality and system reliability, which could negate some of the intended cost and labor savings. Lastly, the complexity of the method may pose challenges in terms of computational requirements and the need for sophisticated hardware and software integration.

[0022] The inventionrelates to a mobile robot indoor positioning system and method based on an ArUco array (Augmented Reality OpenCV Library). The system comprises a driving module, inertial measurement module, visual information module, and datafusion module. The driving module calculates pose information in a world coordinate system, the inertial measurement module acquires pose information in the IMU coordinate system, and the visual information module obtains camera pose information by analysing an ArUco array image. The data fusion module combines these modules using a fusion positioning algorithm to obtain the current pose state of the robot. The method improves indoor positioning accuracy and reduces the influence of dynamic environments by using a dynamic variance fusion positioning algorithm. The system also includes a motor, inertial measurement unit, and a data fusion module. The invention presents a mobile robot indoor positioning method using an ArUco array. This method accurately acquires robot positioning data from various distances, improves indoor positioning accuracy and robustness, and reduces the influence of dynamic environments. The method uses a dynamic variance fusion positioning algorithm, an IMU, and a vision sensor to achieve this. The mobile robot indoor positioning system has several drawbacks, including potential limitations in performance due to reliance on the ArUco array, which may be affected by lighting conditions and occlusions. Additionally, the complexity of integrating multiple modules — such as the driving, inertial measurement, and visual information modules — can lead to challenges in synchronization and data fusion. The system may also struggle in highly dynamic environments where rapid changes could disrupt the accuracy of pose estimation.

[0023] The invention (^K;2024) relates to a method and system for vehicle navigation and a reverse vehicle searching method. The vehicle navigation method involves navigating the vehicle based on an indoorenvironment map, using the indoor environment map when entering the indoor environment, and determining its location in the indoor environment based on the indoor environment map, the initial location at which the vehicle was in when entering the indoor environment, and state-related data for the vehicle traveling in the indoor environment. This method provides accurate, continuous, and complete vehicle navigation services for users in both outdoor and indoor environments. The system for vehicle navigation includes a memory configured to store a navigation map with an indoor environment map, a navigation module coupled to the memory, a processor configured to determine the location of the vehicle in the indoor environment based on the indoor environment map, an indoor initial location at which the vehicle is located when entering the indoor environment, and state-related data for the vehicle traveling in the indoor environment. A display is also provided to display at least one of an indoor environment map, a travel route of the vehicle, and a position of the vehicle on the indoor environment map during travel within the indoor environment. A mobile terminal is provided with a communication module, memory, processor, and display to receive parking position information from a vehicle-mounted system, store at least map data of the indoor environment, match the parking position information with the indoor environment map, and display the parking position on the indoor environment map. The apparatus for vehicle navigation includes a memory configured to store computer readable instructions and a navigation map, a processor coupled to the memory, and a computer readable instruction storage medium. This allows users to efficiently and conveniently find their parking positions in the indoor environment without arranging special parking position monitoring or vehicle searching equipment. The drawbacks of the vehicle navigation and reverse searching method include potentialreliance on the accuracy of indoor environment maps, which may not always be up-to-date or comprehensive, leading to navigation errors. Additionally, the system’s performance could be hindered by limitations in state-related data accuracy, which may affect the determination of the vehicle’s location. Furthermore, the absence of special equipment in indoor environments could limit the effectiveness of the navigation system in complex or large spaces, potentially causing user frustration. Lastly, the technology may face challenges in environments with poor signal reception or interference, impacting overall reliability.

[0024] The inventionrelates to aUWB, IMU, and GNSS fusion-based multi-scene seamless positioning method for unmanned equipment. It focuses on the technical field of unmanned positioning navigation and aims to provide an accurate and continuous navigation scheme. The method involves adopting a combined positioning mode for outdoor environments and a combined positioning mode for indoor environments. The positioning data is corrected by introducing weight factors for the indoor and outdoor intermediate transition regions, allowing the equipment to switch the current positioning mode based on the correction result. The process of switching includes comparing the number of satellites, base stations, and signal strength with a preset threshold value. The unmanned equipment then starts a positioning mode decision procedure based on the number of input information received by the main filter. The method also includes calculating the position of the mobile tag, gesture, speed, and position, and obtaining actual indoor and outdoor position information. The invention improves outdoor and indoor positioning accuracy by integratingGNSS and IMU navigation systems. It solves information loss problems and compensates for single sensor errors. The system also adaptively switches positioning modes based on satellite and base station information, ensuring stable and accurate positioning for unmanned vehicles. The UWB, IMU, GNSS fusion-based multi-scene seamless positioning method for unmanned equipment has several drawbacks. Firstly, the reliance on multiple positioning modes may lead to complexity in the switching process, potentially causing delays or inaccuracies during transitions between indoor and outdoor environments. Additionally, the method’s effectiveness is contingent on the availability and quality of satellite and base station signals, which can be inconsistent, particularly in challenging environments. Furthermore, the introduction of weight factors for data correction may complicate the system’s calibration and could result in errors if not properly managed. Lastly, the dependency on various technologies raises concerns about the overall robustness and reliability of the positioning system under diverse operational conditions.

[0025] The utility model2023) focuses on a vertical distance measuring device for obstacle avoidance, which includes a base, an angle measuring sensor, and a linear distance measuring device. The base is flat plate-arranged, and the angle measuring sensor has a rotor and stator. The linear distance measuring device has a measuring shell, display screen, main control circuit board, and infrared laser distance measuring sensor. The rotor is fixedly connected to the base, and the infrared laser distance measuring sensor is arranged at the second end. The display screen is connected to the main control circuit board. The device is designed to solve the problem ofobstacles below the center vertical line of the target object and can conveniently measure the relative position of the target object from the ground or wall. The device also includes a key module connected to the main control circuit board. The base has an outer plane and an inner plane at two opposite sides, and the rotatable infrared laser ranging sensor can be used to aim at the target object. The horizontal vertical distance between the target object and the wall can be conveniently calculated through Pythagorean theorem. The device's measuring results can be displayed on a display screen, avoiding obstacles and providing convenient relative position measurements. The vertical distance measuring device for obstacle avoidance measurement has several drawbacks, including potential limitations in accuracy due to reliance on infrared laser technology, which can be affected by environmental factors such as lighting and surface reflectivity. Additionally, the device’s design may complicate its use in tight or obstructed spaces, and the need for a stable base connection to a vertical fixture could restrict its versatility in various measurement scenarios.

[0026] The inventionet al., 2023) presents an indoor navigation positioning method based on through-wall pseudo-range residual weighting constraint. This method aims to achieve high-precision continuous stable navigation positioning of cooperative signals in an indoor complex environment. It uses pseudo-range compensation and pseudo-range residual error elimination to calculate cooperative positioning signals, predict current positioning results, correct pseudo-range signals for indoor motion conditions, and improve navigation positioning stability. The method can be applied to indoor navigation positioning implementation modes usingpseudolites, UWB, and sound waves. The invention is beneficial for improving navigation positioning precision and stability of indoor cooperative signals, is simple to implement, and promotes the development of indoor low- cost navigation positioning technology. It can solve navigation positioning continuity and stability problems under shielding, non-direct, and few stars conditions without additional auxiliary means. The method has no increase in complexity, low cost, and a wide application prospect, making it a valuable tool for indoor navigation positioning. In conclusion, the invention offers a simple and effective indoor navigation positioning method that can be applied to various indoor environments, including corridors, passageways, and stairs.The indoor navigation positioning method based on through-wall pseudorange residual weighting constraint has several drawbacks. It may struggle with accuracy in highly dynamic environments or where signal interference is significant, potentially leading to errors in positioning. Additionally, while it aims to function without additional auxiliary means, its reliance on previous positioning results could limit its effectiveness in rapidly changing scenarios. The method’s performance may also be compromised in areas with extreme shielding or complex layouts, which could hinder its overall reliability and applicability in diverse indoor settings.

[0027] The invention (TJ , 2021) relates to an indoor and outdoor fusion positioning method using a special terminal or smart phone, suitable for people, robots, and other intelligent mobile devices. The method uses inertial navigation, particle filtering, fingerprint matching, and indoor map matching for precision of about 1 meter. The system automatically switches indoor and outdoor scenes, seamlessly combining indoor and outdoor positioning. Thespecial terminal consists of hardware modules such as GPS, inertial navigation sensors, Wi-Fi or Bluetooth modules, and communication modules. The implementation site is based on existing networks, reducing costs and providing a stable, reliable indoor and outdoor positioning service without special hardware. The invention focuses on a nine-axis positioning terminal with a three-axis magnetometer, gyroscope, and accelerometer module. The system measures the acceleration of the relative plate plane and converts it into velocity components in three directions in real space. The system analyses the mode and direction of movement every step the object takes. The process involves obtaining a building plan, acquiring fingerprint information, downloading the files, generating an indoor map, and analysing the user's movement mode, gait, step length, and step direction in real time. The system also analyses the user's walking track, shape, and speed. The invention offers advantages such as seamless indoor and outdoor positioning, low cost, standard hardware, reduced indoor map making, and high accuracy. The system can be easily transplanted to other platforms and can be combined with GPS positioning in outdoor areas. The drawbacks of the indoor and outdoor fusion positioning method include the reliance on existing Wi-Fi and Bluetooth networks, which may not be uniformly available in all locations, potentially limiting its effectiveness in areas with poor connectivity. Additionally, while the system aims for high precision, achieving consistent accuracy of about 1 meter can be challenging in complex indoor environments with obstacles. The need for a special terminal or smartphone with advanced sensors may also restrict accessibility for users with older devices. Furthermore, the complexity of accurately mapping indoor spaces and the potential for user error duringthe initial setup could hinder the system’s overall usability and effectiveness.

[0028] The disclosed explorer system (CREASY; KARR; KLARER;MEREDITH et al., 2023) embodies a groundbreaking technological development aimed at aiding individuals with visual impairments in their navigation of environments by utilizing voice-guided instructions. This system integrates various data types, including location, directional, and orientation data, to ascertain the user's desired exploration direction. The system may also utilize a network of strategically positioned wireless indicators within indoor settings to improve the precision of location identification related to areas and levels within structures. The explorer interface can manifest as a software application sourced from the explorer server, while the location server is responsible for storing and disseminating information about geographic locations, structures, and other points of interest. The system's versatility allows for different operational modes, accommodating various device orientations, whether in portrait or landscape format. Functioning as a navigation aid, the explorer system enables users to traverse both outdoor and indoor spaces, beginning from a defined starting point. It assesses the proximity of the user to designated locations equipped with location indicators, and in the absence of nearby indicators, the system may deactivate the indicator monitor or adjust its signal reception settings. The navigation experience can differ significantly between indoor and outdoor environments, influenced by factors such as hardware configuration, user interaction, output methods, and variations in available datasets. Furthermore, the explorer interface can selectively filter mapped objects based on the navigation mode, enhancing the user experience. It is also capable of refining search results and audio narration during directionalexploration, thereby empowering visually impaired users to engage with their environment more comprehensively. Notably, the interface may adapt its features to accommodate the transition from outdoor to indoor navigation, ensuring a seamless exploration experience. The explorer system, while beneficial for visually impaired individuals, has several drawbacks. It relies heavily on the availability of location indicators, which may not be present in all indoor environments, potentially limiting navigation capabilities. Additionally, the system’s performance can be affected by hardware configurations, such as power conservation measures that disable certain location services. The need for constant communication with location indicators may also hinder usability in areas with poor connectivity. Furthermore, the complexity of transitioning between indoor and outdoor modes could lead to confusion for users, especially if the interface does not provide clear guidance during these transitions.

[0029] The inventionStS / ?, 2020) presents a positioning system and method for a petrochemical plant inspection robot, consisting of five observation units. The system collects observation data and sends it to a combined decision module, which then performs a multilayer map and determines an available observation type. The inertial state estimation module constructs corresponding observation data and sends it to the available positioning observation / SINS combined module. The positioning information fusion module integrates the estimation results of all local filters to obtain a final combined pose estimation result. The system is designed to ensure the continuity of the positioning process and the stability of the positioning result, particularly in complex areas of a petrochemicalplant. The invention belongs to the field of special robot positioning. The inspection robot uses a combination of various positioning subsystems to accurately and stably apply it to various complex occasions, such as indoor and outdoor places, flat and rugged road surfaces, daytime and night places, sparse and obvious features, rainy and snowy days, and petrochemical plants. The combined decision module receives UWB wireless positioning information, image characteristic information, and environmental point cloud information from the UWB, visual, and laser observation units. The inertial state estimation module measures z-axis acceleration perpendicular to the robot platform and inputs the combined decision module. The corresponding local filters construct an observation equation based on the available observation types, and the combination information distribution coefficient is determined according to the principle of information conservation. The self- adaptive combined filtering algorithm ensures automatic judgment of available positioning observation types in different environments, resulting in a positioning effect smaller than the error of any one positioning method used independently. The drawbacks of the described positioning system for the petrochemical plant inspection robot include potential complexity in integrating multiple observation units, which may lead to challenges in real-time data processing and decision-making. Additionally, reliance on various environmental factors, such as weather conditions and terrain types, could affect the system’s accuracy and stability. The need for continuous adaptation to different observation types may also introduce uncertainties in the self-adaptive filtering algorithm, potentially compromising the overall reliability of the positioning results in highly dynamic environments.Challenges and future trendsThe Literature discusses the potential of indoor AR systems, including sensor enhancements, multi-platform integration, self-learning algorithms, 5G / IoT / edge computing convergence, and high-definition maps for indoor planning. They highlight the need for pre-processed data and the need for specialized shoes for indoor operations. QR code readers for indoor navigation are also discussed, but their limitations include reliance on Google Glasses as an imaging device and lack of altitude positioning capabilities. The Indoor Location -based Augmented Reality Framework (ILARF) holds significant potential for advancing indoor AR systems, but it faces challenges in localization accuracy. The system relies on short-term relative positioning techniques like radar or ultrasound, along with geometric characteristics from LiDAR and RGB-D sensors. Machine learning and Wi-Fi technologies have revolutionized indoor positioning applications, but future research should prioritize data security, privacy issues, and adherence to industry standards. Publications also discuss the challenges of using location fingerprinting with Received Signal Strength Indicators (RSSIs) for indoor localization, such as the labor-intensive process of collecting large amounts of pre-processed and annotated data required for training Deep Neural Networks (DNNs). The Multi-Output Gaussian Process (MOGP) method shows promise in augmenting RSSI data but may still face limitations in scalability and generalizability across different environments. The indirect positioning device has potential limitations due to the need for precise positioning of multiple components, environmental factors, and mechanical structure. The indoor navigation system based on AR equipment has limitations in accuracy due to reliance on positioning identifiers, poor signal reception, and user behavior. Additionally, the system for determining autonomous vehicle states during transitions between different environments has several drawbacks, including reliance on multiple positioning subsystems, latency, computational challenges, anddependence on prior states and measurement uncertainties. Additional drawbacks, including sensor compatibility issues, environmental data variability, and the complexity of integrating multiple sensor types. It also faces challenges in real-world scenarios due to the need for human oversight and computational requirements. The method for determining coordinate system conversion relations relies heavily on sensor data accuracy, which can lead to errors and inaccuracies. The process requires multiple indoor pictures and corresponding sensor data, which may not always be readily available. The mobile robot indoor positioning system has limitations due to reliance on the ArUco array, which may be affected by lighting conditions and occlusions. The vehicle navigation and reverse searching method relies on the accuracy of indoor environment maps, which may not always be up-to-date or comprehensive, leading to navigation errors. The absence of special equipment in indoor environments could limit the effectiveness of the navigation system in complex or large spaces. The UWB, IMU, and GNSS fusion-based multi-scene seamless positioning method for unmanned equipment has several drawbacks, including the reliance on multiple positioning modes, inconsistent satellite and base station signals, and the introduction of weight factors for data correction. The vertical distance measuring device for obstacle avoidance measurement has potential limitations in accuracy due to reliance on infrared laser technology, design complications, and increased maintenance requirements. The indoor and outdoor fusion positioning method rely on existing Wi-Fi and Bluetooth networks, which may not be uniformly available in all locations. The explorer system relies heavily on location indicators, hardware configurations, and the complexity of transitioning between indoor and outdoor modes. Lastly, the factory plant inspection robot's positioning system faces potential complexity in integrating multiple observation units, environmental factors, and uncertainties in the self-adaptive filtering algorithm.Discussed next are not limited to determining a point precise positioning system of algorithms, software and hardware fusion of GNSS-INS-IMU-LiDAR-STEREO- RGB -ULTRASOUND RADAR for autonomous vehicle positioning and navigation of land or airborne units or humans by means of wearable devices independent or paired to smart phones INDOORS.The description below provides an unrestricted explanation of the invention and its various forms. The actual construction of the invention is not confined to the specific embodiments depicted in the figures.In the extensive explanation of the embodiments, a multitude of specific details are included to ensure a thorough comprehension of the claimed invention. Nevertheless, a person with expertise in the field would comprehend that several versions can be implemented without including those particular features.While the current embodiments describe specific combinations of features and elements, each individual feature or element can be used independently without the other features and elements of the embodiments. Additionally, they can be combined in various ways with or without other features and elements mentioned in this document.The claims describe the patentable breadth of the subject matter and may encompass further examples that are conceived by experts in the field. Additional instances that fall within the boundaries of the claims are also included.Description:This Invention discloses the advancement of synchronous fusion methodologies that incorporate Inertial Navigation Systems (INS), Inertial Measurement Units (IMU), LiDAR, stereo vision, and sonar radar technologies, with a specific emphasis on their application in achieving high-precision indoor positioning and navigation. The primary aim is to enhance the effectiveness and versatility of these sophisticated technologies, thereby enabling the collection and analysis of data with centimetrelevel accuracy in various environments, ultimately contributing to the creation of a three-dimensional indoor map within a surrounding spherical range of 12 meters utilizing stereo and sonar inputs.The proposed innovation includes a comprehensive array of hardware and algorithms, focusing on the creation of accurate positioning systems that utilize advanced computational techniques, alongside the harmonious integration of both software and hardware elements. This amalgamation of INS, IMU, LiDAR, stereo, and sonar technologies is particularly significant for the indoor navigation of both humans and autonomous vehicles, whether they are ground-based or aerial, especially in scenarios where dead reckoning or where GPS signals are unreliable. Furthermore, the disclosure highlights the potential for wearable devices that can function autonomously or in tandem with smartphones, thereby enhancing navigation capabilities for individuals. By examining these complex systems, the disclosure aspires to advance the field of navigation technologies for both humans and autonomous entities. The integration of diverse sensing modalities not only aims to improve positioning accuracy but also paves the way for innovative applications across various sectors, including support for emergency responders in unfamiliar indoor environments, service personnel, and personal navigation in novel or uncharted locations. Ultimately, this work endeavours to establish a robust real-time or semi-real-time framework that optimizes the application of these technologies in real- world indoor settings.dware Sensor Fusion: The disclosed hardware system, optimized for a compact design, encompasses a diverse array of sensors functioning collectively as a sensor swarm, which includes an Inertial Navigation System (INS), an Inertial Measurement Unit (IMU), Light Detection and Ranging (LiDAR), stereo imaging technologies, and a variety of miniature ultrasound sonar 3D scanning devices, all mounted on two-axis servo motors that facilitate scanning of the three-dimensional environment with a latency of just 0.1 milliseconds. This advanced configuration of sensors is meticulously designed to produce three-dimensional coloured visualizations of various indoor settings. By utilizing multiple LiDAR units, RGB stereo imaging systems, and 3D sonar scanners operating in different phases, the system adeptly addresses gaps in imaging and mitigates the challenges associated with dead reckoning, thereby allowing for the precise reconstruction of the topographical characteristics of its environment, which is essential for comprehensive mapping and analytical purposes. orithms: The system is invented to generate detailed three-dimensional sensory information that includes critical components such as geo-location and the identification of points of interest (POIs). The system utilizes advanced image processing techniques, particularly focusing on edge detection and object recognition, to enhance situational awareness by identifyingfamiliar objects within a three-dimensional context. This approach markedly increases the accuracy and relevance of the data collected, allowing for a more nuanced understanding of the environment. This process facilitates sonar registration, which is essential for correcting positional drift over time through the extraction of features from the sonar's field of view. To achieve this, a pose graph approach to simultaneous localization and mapping (SLAM) is utilized. This capability is particularly advantageous for scenarios that demand high levels of spatial awareness and environmental detail comprehension, rendering it an invaluable asset across various domains, including support for emergency responders in unfamiliar indoor environments, service personnel, and personal navigation in novel or uncharted locations. The implementation of particle swarm optimization (PSO) plays a crucial role in navigating non-deterministic environments. This optimization technique is instrumental in identifying the necessary coupling fusion for the effective integration of various technologies. The integration can be classified into distinct categories: uncoupled integration, which functions independently of Global Navigation Satellite System (GNSS) signals; weakly coupled integration; tightly coupled integration; and ultra-tight or deep integration. Each of these integration strategies presents different degrees of accuracy and dependability, with uncoupled integration being particularly advantageous in situations where GNSS signals are compromised, yet the demand for high-precision navigation persists. The enhancement of the field of vision can be achieved through the application of a template matching co-occurrence matrix, which aligns the visual input with recalibrated ranges from stereo and sonar imaging. This methodology is vital for the registration of sonar data, which aids inmitigating positional drift over time by extracting features from the sonar's field of view, this methodology addresses the inherent challenges posed by low-light scenarios and voids in the stereo imaging's field of view by focusing on the extraction of relevant features. Such processes are essential for maintaining accurate navigation in challenging environments where traditional visual cues may be insufficient. On the other hand, the dependency of ultrasound time-of-flight measurements and lidar temperature variations can be effectively addressed through the application of back propagation corrections, which leverage stereo imaging depth data. This approach allows for a more accurate interpretation of the spatial and thermal dynamics involved in these technologies, enhancing the reliability of the measurements obtained. By integrating stereo imaging techniques, the depth information can be utilized to refine the corrections applied to both ultrasound and lidar systems, ultimately leading to improved performance in various environmental conditions. Furthermore, the integration of particle swarm optimization with various artificial intelligence methodologies, such as Neuro-Fuzzy algorithms, can significantly improve the efficacy of optimization algorithms designed for specific applications. The Neuro-Fuzzy PSO Al Positioning and Navigation System is adept at generating three-dimensional sensory data that offers comprehensive insights into the geographical layout of indoor navigation, particularly in scenarios involving dead reckoning. This system operates in real-time or near real-time, achieving positioning accuracy levels below 0.5 meters by utilizing inputs from a diverse array of sensory sources. The implementation of a pose graph approach to simultaneous localization and mapping (SLAM) enhances this capability,making it especially beneficial for edge detection and object recognition in contexts that require heightened spatial awareness and detailed environmental understanding. o Additionally, the framework incorporates privacy protection measures that facilitate the abstraction of segments within the field of view or the three- dimensional scene when a privacy concern arises. This capability ensures that sensitive information is effectively managed, thereby maintaining the integrity of personal privacy while still allowing for comprehensive environmental analysis.ABDALLAH, A. A.; JAO, C. S.; KASSAS, Z. M.; SHKEL, A. M. A Pedestrian Indoor Navigation System Using Deep-Learning-Aided Cellular Signals and ZUPT-Aided Foot-Mounted IMUs. IEEE Sensors Journal, 22, n. 6, p. 5188-5198, 2022.ALKADY, Y.; RIZK, R.; ALSEKAIT, D. M.; ALLUHAIDAN, A. S. etal. SINS_AR: An Efficient Smart Indoor Navigation System Based on Augmented Reality. IEEE Access, 12, p. 109171 -109183, 2024.CREASY, D. K.; KARR, J.; KLARER, M.; MEREDITH, R. etal. Selective information provision and indoor navigation assistance for the visually impaired. USPTO. 2023.EL-SHEIMY, N.; LI, Y. Indoor navigation: state of the art and future trends. Satellite Navigation, 2, n. 1 , p. 7, 2021 / 05 / 032021 .GUPTA, G.; KEJRIWAL, N.; PALLAV, P.; HASSAN, E. etal., 2016, Indoor Localisation and Navigation on Augmented Reality Devices. 107-1 12.HUANG, B.-C.; HSU, J.; CHU, E. T.-H.; WU, H.-M. ARBIN: Augmented Reality Based Indoor Navigation System. Sensors, 20, n. 20, p. 5890, 2020.JIANG, J.-R.; SUBAKTI, H. An Indoor Location-Based Augmented Reality Framework. Sensors, 23, n. 3, p. 1370, 2023.KIM, J.; JUN, H. Vision-based location positioning using augmented reality for indoor navigation. IEEE Transactions on Consumer Electronics, 54, n. 3, p. 954-962, 2008.LI, J.; WANG, C.; KANG, X.; ZHAO, Q. Camera localization for augmented reality and indoor positioning: a vision-based 3D feature database approach. International Journal of Digital Earth, 13, n. 6, p. 727-741, 2020 / 06 / 022020.PARAB, D. K.; DESHPANDE, P. P.; THAKUR, R. M.; WARKE, V. A. etal., 2024, Indoor Navigation System using Augmented Reality. 720-725.SCHULDT, C.; SHOUSHTARI, H.; HELLWEG, N.; STERNBERG, H. L5IN: Overview of an Indoor Navigation Pilot Project. Remote Sensing, 13, n. 4, p. 624, 2021.SINGH, J.; TYAGI, N.; SINGH, S.; ALI, F. etal. A Systematic Review of Contemporary Indoor Positioning Systems: Taxonomy, Techniques, and Algorithms. IEEE Internet of Things Journal, 11 , n. 21 , p. 34717-34733, 2024.SINHA, R. S.; LEE, S.-M.; RIM, M.; HWANG, S.-H. Data Augmentation Schemes for Deep Learning in an Indoor Positioning Application. Electronics, 8, n. 5, p. 554, 2019.TANG, Z.; LI, S.; KIM, K. S.; SMITH, J. S. Multi-Dimensional Wi-Fi Received Signal Strength Indicator Data Augmentation Based on Multi-Output Gaussian Process for Large-Scale Indoor Localization. Sensors, v.24, n. 3, DOI: 10.3390 / s24031026.VINCIGUERRA, E.; RUSSO, E.; PALESI, M.; ASCIA, G. etal., 2024, Improving LSTM-based Indoor Positioning via Simulation-Augmented Geomagnetic Field Dataset. 251-259.XIA, L.; LU, J.; LU, Y.; ZHANG, H. etal. Augmented reality and indoor positioning based mobile production monitoring system to support workers with human-in-the-loop.Robotics and Computer-Integrated Manufacturing, 86, p. 102664, 2024 / 04 / 01 / 2024.YU, C.; ZHENG, H.; ZHANG, J.; CUI, Y. etal. Systems and methods for estimating a state for positioning autonomous vehicles transitioning between different environments. USPTO. 2024.Method and system for vehicle navigation and reverse vehicle searching method. CNIPA. 2024.etal. UWB, IMU, GNSS fusion-based multi-scene seamless positioning method for unmanned equipment. CNIPA. 2023. fits??. Petrochemical plant inspection robot positioning system 2020.n indoor and outdoorfusion positioning system and method. CNIPA. 2021.etal. Indoor navigation positioning method based on through- wall pseudo-range residual weighting constraint. CNIPA. 2023. HU etal. Multi-source sensor elastic fusion navigation positioning ironment information assistance. CNIPA. 2023.oordinate system conversion relation determination method, device, equipment and storage medium. CNIPA. 2023.Indoor navigation system based on AR equipment. 2024.ance measuring device for obstacle avoidance measurement. CNIPA. 2023.etal. Mobile robot indoor positioning system and method based on Arllco array. CNIPA. 2023.°lfr. Device and Method for Positioning of Terminal based on MovementAttribute. KIPO. 2024. s . Indirect positioning device and method. KIPO. 2024.Title of the Invention:Enhancement Of The 3D Indoor Positioning by Augmenting a Multitude of 3DImaging, LiDAR Distance Corrections, IMU Sensors and 3-D UltrasoundClaims[Claim 1 ] A multi-sensory and an Al system or device having a sensor array and processing platform to scan and analyse three-dimensional spatial and sensory data indoors or in confined environmental geometry or urban canyons. Wherein, the advancement of synchronous fusion methodologies that incorporate Inertial Navigation Systems (INS), Inertial Measurement Units (IMU), LiDAR, stereo vision, and sonar radar technologies, with a specific emphasis on achieving fast-high-precision indoor positioning and navigation. Wherein, the collection and analysis of data with centimetre-level accuracy in various environments is enabled in real-time or semi-real-time framework, ultimately contributing to the creation of a three-dimensional indoor map within a surrounding spherical range of 12 meters utilizing stereo and sonar inputs.[Claim2] A multi-sensory and an Al system or device of claim 1, wherein the system includes multitudes of Light Detection and Ranging (LiDAR), multitude of stereo imaging technologies, and a variety of miniature ultrasound sonar 3D scanning devices, mounted on two-axis servo motors that facilitate scanning of the three-dimensional environment with a latency of 0.1 milliseconds. Wherein, the configuration of sensors is meticulously designed to produce three-dimensional coloured visualizations of various indoor settings.

Claims

[Claim 1 ] A multi-sensory and an Al system or device of claim 1 and 2, wherein the system generates detailed three-dimensional sensory information that includes critical components such as geo-location and the identification of points of interest (POIs). Wherein, the system utilizes advanced image processing techniques including but not limited to edge detection and object recognition, to enhance situational awareness by identifying familiar objects within a three-dimensional context. Wherein the relevance of the data collected allows for a more nuanced understanding of the environment.[Claim2] A multi-sensory and an Al system or device of claim 1, wherein the system of LiDAR and Stereo Imaging are assigned to a processing platform performing computations including RAFT- Stereo, IGEV- Stereo, DLNR and Croco- Stereo for building the three-dimensional extra- wide spatial recognition in Realtime or semi-Realtime.[Claim3] A multi-sensory and an Al system or device of claim 1 and 2, wherein the field of vision is improved through the application of a template matching co-occurrence matrix aligning the visual input with recalibrated ranges from synchronized stereo and sonar imaging. Wherein, registration of sonar data aids in mitigating positional drift over time by extracting features from the sonar's field of view. Wherein, low-light scenarios and voids in the stereo imaging's field of view are compensated on the extraction of relevant features from the sonar.[Claim4] A multi-sensory and an Al system or device of claim 1 and 2, wherein the dependency of ultrasound time-of-flight measurements and lidar temperature variations can be corrected through the application of back propagation corrections, which leverage stereo imaging depth data to correct time-of-flight. Wherein, more accurate interpretation of the spatial and thermal corrections applies. Wherein, the depth information can be utilized to refinethe corrections applied to both ultrasound and lidar systems for improved performance in various environmental conditions.[Claim5] A multi-sensory and an Al system or device of claim 1 and 2, wherein the integration of particle swarm optimization with various artificial intelligence methodologies, such as Neuro-Fuzzy algorithms, can significantly improve the efficacy of optimization algorithms designed for specific applications. The Neuro-Fuzzy PSO Al Positioning and Navigation System is adept at generating three-dimensional sensory data that offers comprehensive insights into the geographical layout of indoor navigation, Wherein, the system operates in real-time or near real-time, achieving positioning accuracy levels below 0.5 meters by utilizing inputs from a diverse array of sensory sources. Wherein, the implementation of a pose graph approach to simultaneous localization and mapping (SLAM) enhances this capability, making it especially beneficial for edge detection and object recognition in contexts that require heightened spatial awareness and detailed environmental understanding.[Claim6] A multi-sensory and an Al system or device of claim 1 and 2, wherein the framework incorporates privacy protection measures that facilitate the abstraction of segments within the field of view or the three-dimensional scene when a privacy concern arises. This capability ensures that sensitive information is effectively managed, thereby maintaining the integrity of personal privacy while still allowing for comprehensive environmental analysis.[Claim?] A multi-sensory and an Al system or device of claim 1 and 2, wherein the system can generate three-dimensional sensory data that provides detailed information about the indoor distribution of crowds, obstacles, patterns and areas of dead reckoning and overlays the data of interest (Doi) on maps forguidance and navigation and can function autonomously or paired with smart devices such as: smartphones, PDA, arm bracelet, eyewear, projection device or Tablets.[Claim8] A multi-sensory and an Al system or device of claim 1 and 2, wherein the integration of diverse sensing modalities not only aims to improve positioning accuracy but also paves the way for innovative applications across various sectors, including support for emergency responders in unfamiliar indoor environments, service personnel, and personal navigation in novel or uncharted locations.

Images