Indoor navigation method and system for unmanned aerial vehicle, mobile terminal and storage medium

Abstract

The application relates to the technical field of aircrafts, and discloses an indoor navigation method and system of an unmanned aerial vehicle, a mobile terminal and a storage medium. The indoor navigation method of the unmanned aerial vehicle constructs a base station coordinate system based on an AR real scene picture of a control program, so that professional surveying and mapping equipment is not needed, the deployment time of a UWB base station is shortened, the deployment efficiency is improved, and the accuracy of indoor navigation is improved through a dynamic obstacle avoidance strategy based on a UWB confidence field, so that the flight safety is improved.

Description

Technical Field

[0001] This application relates to the field of aircraft technology, and in particular to an indoor navigation method, system, mobile terminal and storage medium for unmanned aerial vehicles. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs), also known as drones, are increasingly widely used due to their advantages such as small size, light weight, maneuverability, rapid response, unmanned operation, and low operational requirements. The various actions (or attitudes) of an UAV are typically achieved by controlling the different speeds of multiple drive motors in its power system.

[0003] With the rapid development of drone technology, its application scenarios are gradually expanding from outdoor to indoor environments. In outdoor environments, drones typically rely on GNSS (such as GPS and BeiDou) for positioning and navigation. However, in indoor environments (such as large warehouses, factory workshops, and underground parking garages), GNSS signals are severely blocked, making it impossible to provide effective positioning information.

[0004] Currently, indoor drone positioning mainly relies on visual SLAM, LiDAR SLAM, or external motion capture systems. Visual and LiDAR SLAM are prone to failure in environments with sparse features or drastic lighting changes, and they also require a large amount of computation; external motion capture systems are expensive and complex to deploy, and are only suitable for small-scale laboratory environments. In recent years, ultra-wideband (UWB) technology has begun to be applied to indoor positioning due to its centimeter-level positioning accuracy, strong penetration, and low power consumption.

[0005] However, the accuracy of existing UWB drone indoor navigation systems is highly dependent on the accuracy of base station coordinates. This usually requires professionals to manually measure and input the coordinates using equipment such as total stations, resulting in low deployment efficiency. Furthermore, manual measurement is prone to errors, leading to insufficient accuracy in indoor navigation and low flight safety. Summary of the Invention

[0006] The embodiments of this application aim to provide an indoor navigation method for unmanned aerial vehicles, which solves the technical problems of low base station deployment efficiency and low flight safety in existing indoor navigation methods, so as to improve deployment efficiency, improve indoor navigation accuracy, and thus improve flight safety.

[0007] The embodiments of this application provide the following technical solutions: On the one hand, this application provides an indoor navigation method for an unmanned aerial vehicle, applied to a mobile terminal, wherein the mobile terminal is communicatively connected to multiple UWB base stations and has a control program installed on it; The methods include: In response to a touch operation on the AR real-world image of the control program, the location of each UWB base station in the AR real-world image is determined; Based on the location of each UWB base station in the AR real-world scene, and using the distance information between the mobile terminal and each UWB base station, combined with the visual inertial odometry of the mobile terminal, the three-dimensional coordinates of each UWB base station in the pre-built AR world coordinate system are calculated to construct the base station coordinate system. Align the base station coordinate system with the local coordinate system of the UAV's takeoff point, and send the three-dimensional coordinates of each UWB base station to each UWB base station and the UAV. During the flight of the unmanned aerial vehicle, a UWB confidence field of the indoor space is constructed; The navigation mode of the unmanned aerial vehicle is dynamically adjusted based on the UWB confidence field to control the flight of the unmanned aerial vehicle.

[0008] In some embodiments, Based on the UWB confidence field, the navigation mode of the unmanned aerial vehicle is dynamically adjusted, including: If the unmanned aerial vehicle enters a region where the confidence level is within the first confidence level range, it switches to UWB-dominated mode. The first confidence level range includes: UWB confidence level greater than or equal to the first confidence threshold. If the UAV enters an area where the confidence level is within the second confidence level range, it switches to the UWB and vision fusion mode. The second confidence level range includes: UWB confidence level less than the first confidence level threshold, and UWB confidence level greater than or equal to the second confidence level threshold. If the UAV enters a region where the confidence level is in the third confidence level range, it will switch to a vision and IMU-dominated mode. The third confidence level range includes areas where the UWB confidence level is less than the second confidence level threshold.

[0009] In some embodiments, Unmanned aerial vehicles include depth cameras; During the flight of the unmanned aerial vehicle, the method also includes: Acquire environmental point clouds in real time from a depth camera; The environmental point cloud is registered with the pre-imported indoor BIM model to obtain a three-dimensional digital twin scene in real time.

[0010] In some embodiments, The method also includes: Acquire the location information of dynamic obstacles detected by the depth camera; Based on the location information of dynamic obstacles, a dynamic no-fly zone is generated in real time in the AR real-world scene; Based on trajectory algorithms, detour trajectories are generated to control the flight of unmanned aerial vehicles.

[0011] In some embodiments, The method also includes: Construct a virtual unmanned aerial vehicle model within a 3D digital twin scenario; The UWB positioning data and IMU data acquired by the unmanned aerial vehicle are fused to obtain the real-time pose of the unmanned aerial vehicle; Based on the real-time pose of the unmanned aerial vehicle (UAV), the virtual UAV model is driven to move synchronously in a three-dimensional digital twin scene.

[0012] In some embodiments, The method also includes: Obtain the position command from the AR real-world image, where the position command corresponds to a two-dimensional position coordinate; Convert two-dimensional position coordinates into three-dimensional virtual coordinates, and then convert the three-dimensional virtual coordinates into the physical coordinates of the unmanned aerial vehicle; Based on physical coordinates, a collision-free trajectory is generated to control the unmanned aerial vehicle to move to the location of the physical coordinates.

[0013] In some embodiments, Unmanned aerial vehicles include scanning equipment, which may include RFID readers or barcode scanners; The method also includes: During the flight of the unmanned aerial vehicle, the scanning information obtained by the scanning device scanning the target is acquired; The scan information is bound to the three-dimensional coordinates of the scanned target.

[0014] On the other hand, embodiments of this application provide a mobile terminal, including: At least one processor; At least one memory for storing at least one program; When at least one program is executed by at least one processor, the at least one processor performs the method described above.

[0015] On the other hand, embodiments of this application provide an indoor navigation system for an unmanned aerial vehicle, comprising: The aforementioned mobile terminals; Multiple UWB base stations connect to mobile terminals.

[0016] On the other hand, embodiments of this application provide a non-volatile computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0017] The beneficial effects of this application's embodiments are as follows: Unlike existing technologies, this application provides an indoor navigation method for unmanned aerial vehicles (UAVs), applied to a mobile terminal. The mobile terminal is communicatively connected to multiple UWB base stations and has a control program installed. The method includes: in response to a touch operation on an AR real-world image displayed in the control program, determining the position of each UWB base station in the AR real-world image; based on the position of each UWB base station in the AR real-world image, and using distance information between the mobile terminal and each UWB base station, combined with the mobile terminal's visual inertial odometry, calculating the three-dimensional coordinates of each UWB base station in a pre-constructed AR world coordinate system to construct a base station coordinate system; aligning the base station coordinate system with the local coordinate system of the UAV's takeoff point, and sending the three-dimensional coordinates of each UWB base station to each UWB base station and the UAV; during the UAV's flight, constructing a UWB confidence field for the indoor space; and dynamically adjusting the UAV's navigation mode based on the UWB confidence field to control the UAV's flight.

[0018] By constructing a base station coordinate system using AR real-world images based on control programs, the deployment time of UWB base stations can be shortened and deployment efficiency improved without the need for specialized surveying equipment. Furthermore, by employing a dynamic obstacle avoidance strategy based on UWB confidence fields, the accuracy of indoor navigation can be improved, thereby enhancing flight safety. Attached Figure Description

[0019] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0020] Figure 1 This is a schematic diagram of the structure of an indoor navigation system for an unmanned aerial vehicle provided in an embodiment of this application; Figure 2 This is a flowchart illustrating an indoor navigation method for an unmanned aerial vehicle provided in an embodiment of this application; Figure 3 This is a schematic diagram of a base station calibration process provided in an embodiment of this application; Figure 4 This is provided by the embodiments of this application. Figure 2 A detailed flowchart of step S205 in the process; Figure 5 This is a schematic diagram of a process for rendering a three-dimensional digital twin scene provided in an embodiment of this application; Figure 6 This is a schematic diagram of a process for controlling the synchronous movement of a virtual unmanned aerial vehicle model in a three-dimensional digital twin scene, provided in an embodiment of this application. Figure 7 This is a schematic diagram of a process for controlling the flight of an unmanned aerial vehicle according to an embodiment of this application; Figure 8 This is a schematic diagram of a process for controlling an unmanned aerial vehicle to move to a physical coordinate position according to an embodiment of this application; Figure 9 This is a schematic diagram of a process for binding scanning information and scanning target according to an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an indoor navigation device for an unmanned aerial vehicle provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a mobile terminal provided in an embodiment of this application.

[0021] Explanation of icon numbers: Detailed Implementation

[0022] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0025] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0026] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0027] With the rapid development of drone technology, drones are being widely used in public safety, emergency rescue, border patrol, forest fire prevention, and maritime search and rescue.

[0028] Currently, indoor drone positioning mainly relies on visual SLAM, LiDAR SLAM, or external motion capture systems. Visual and LiDAR SLAM are prone to failure in environments with sparse features or drastic lighting changes, and they also require a large amount of computation; external motion capture systems are expensive and complex to deploy, and are only suitable for small-scale laboratory environments. In recent years, ultra-wideband (UWB) technology has begun to be applied to indoor positioning due to its centimeter-level positioning accuracy, strong penetration, and low power consumption.

[0029] However, existing UWB drone indoor navigation systems have the following technical shortcomings: (1) Base station deployment and calibration are cumbersome: The accuracy of the UWB system is highly dependent on the accuracy of the base station coordinates. Existing solutions usually require professionals to use equipment such as total stations for manual measurement and input, resulting in low deployment efficiency.

[0030] (2) Lack of intuitive mobile terminal interaction: Existing systems mostly use PC-based backend for monitoring, which can only display two-dimensional trajectories or simple three-dimensional coordinate points. They lack real-time three-dimensional digital twin visualization on mobile apps, making it difficult for operators to intuitively perceive the real spatial state of the drone.

[0031] (3) Poor business linkage: In practical applications such as warehouse inventory or security patrol, the flight control of drones and business data (such as cargo barcodes and abnormal images) are often separated, lacking a data binding mechanism based on high-precision spatial location.

[0032] Based on this, this application provides an indoor navigation method for unmanned aerial vehicles, which solves the technical problems of low base station deployment efficiency and low flight safety in existing indoor navigation methods, so as to improve deployment efficiency, improve indoor navigation accuracy, and thus improve flight safety.

[0033] It should be noted that the indoor navigation method for unmanned aerial vehicles provided in this application can be applied to various movable objects driven by motors or electric motors, including but not limited to aircraft and robots. The aircraft may include unmanned aerial vehicles (UAVs) and unmanned spacecraft. A UAV will be used as an example for illustration.

[0034] Before describing the embodiments of this application, let's first explain some of the terms and concepts used in this application: (1) Ultra Wideband (UWB) refers to a carrierless communication technology that uses non-sinusoidal narrow pulses in the nanosecond to microsecond range to transmit data. It has the characteristics of high positioning accuracy (centimeter level), strong anti-multipath interference capability, and low power consumption.

[0035] (2) Time Difference of Arrival (TDOA) refers to a positioning algorithm that uses the time difference of signals arriving at different base stations to calculate the spatial coordinates of a positioning tag.

[0036] (3) Digital twin refers to the real-time mapping of physical entities in virtual space using physical models, sensor data, etc. In this application, it refers to the real-time generation of a three-dimensional virtual model of an indoor environment and an unmanned aerial vehicle on the App.

[0037] (4) Simultaneous Localization and Mapping (SLAM) refers to the technology that allows a robot to estimate its own position and build an environmental map while moving in an unknown environment.

[0038] (5) Augmented Reality (AR) refers to the technology that integrates virtual information with the real world. In this application, it is used to assist in calibrating the coordinates of UWB base stations through the camera of a mobile terminal.

[0039] (6) Point-to-Fly refers to the interactive method in which the user directly clicks on the target point in the 3D map of the application (APP), and the system automatically calculates the physical coordinates and drives the unmanned aerial vehicle to fly to the target point.

[0040] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0041] Please see Figure 1 , Figure 1 This is a structural schematic diagram of an indoor navigation system for an unmanned aerial vehicle provided in an embodiment of this application.

[0042] like Figure 1As shown, the indoor navigation system 100 of the unmanned aerial vehicle (UAV) includes: an UAV 10, a mobile terminal 20, and multiple UWB base stations 30. The UAV 10 is communicatively connected to the mobile terminal 20 and the multiple UWB base stations 30. For example, the UAV 10 and the mobile terminal 20 are wirelessly connected, for example, via 4G, 5G, Bluetooth, or other wireless methods. It is understood that the pilot or user can operate the mobile terminal 20 to control the movement of the UAV 10 via the wireless network.

[0043] It should be noted that the indoor navigation system 100 of the unmanned aerial vehicle consists of the unmanned aerial vehicle 10 (the onboard terminal of the drone) and the mobile terminal 20 (the ground control terminal), and communicates bidirectionally through a wireless data link (supporting 4G / 5G cellular networks or dedicated digital image transmission links).

[0044] The unmanned aerial vehicle 10 (the onboard unit of the drone) is equipped with a high-definition camera (supporting both visible and infrared dual spectra), an onboard computing unit (such as an NVIDIA Jetson series or equivalent computing chip), a GNSS / RTK positioning module, and a flight controller, responsible for autonomous flight control, real-time target detection, and uploading of detection results. The mobile terminal 20 (the ground control unit) runs on a mobile terminal (tablet, smartphone, or dedicated ground station) or wearable device (smart headset) equipped with a microphone, responsible for voice command acquisition and parsing, map interaction, task distribution, real-time video display, and result statistics.

[0045] Specifically, the unmanned aerial vehicle 10 includes: a fuselage 11, an arm 12 connected to the fuselage 11, a power unit 13 disposed on the arm 12, a gimbal 14 connected to the bottom of the fuselage 11, a camera device 15 mounted on the gimbal 14, and a flight controller (not shown) disposed within the fuselage 11. The unmanned aerial vehicle 10 also includes a UWB positioning tag.

[0046] The flight controller is connected to the power unit 13, which is mounted on the fuselage 11 and provides flight power for the unmanned aerial vehicle 10. Specifically, the flight controller sends control commands to the electronic speed controller (ESC) of the power unit 13, which then controls the drive motor of the power unit 13.

[0047] The fuselage 11 includes a central housing and one or more arms connected to the central housing, with the one or more arms 12 extending radially from the central housing. The connection between the arms 12 and the central housing can be integral or fixed. A power unit 13 is mounted on the arms 12.

[0048] The power unit 13 includes an electronic speed controller (ESC), a drive motor, and a propeller. The ESC is located within the cavity formed by the arm or central housing. The ESC is connected to both the controller and the drive motor. Specifically, the ESC is electrically connected to the drive motor to control it. The drive motor is mounted on the arm, and its rotating shaft is connected to the propeller. Driven by the drive motor, the propeller generates a force that moves the unmanned aerial vehicle (UAV) 10, such as lift or thrust.

[0049] The unmanned aerial vehicle (UAV) 10 achieves its various specified speeds, maneuvers (or attitudes) by controlling its drive motors via an electronic speed controller (ESC). An ESC adjusts the speed of the UAV 10's drive motors based on control signals. The principle of ESC control of the drive motor is roughly as follows: the drive motor is an open-loop control element that converts electrical pulse signals into angular or linear displacement. Under non-overload conditions, the drive motor's speed and stopping position depend only on the frequency and number of pulse signals, and are unaffected by load changes. When the driver receives a pulse signal, it drives the drive motor of the power unit to rotate a fixed angle in a set direction; its rotation is at a fixed angle. Therefore, the ESC can control the angular displacement by controlling the number of pulses, thereby achieving accurate positioning; simultaneously, it can control the speed and acceleration of the drive motor by controlling the pulse frequency, thereby achieving speed regulation.

[0050] Understandably, the main functions of the unmanned aerial vehicle (UAV) 10 are aerial photography, real-time image transmission, and high-risk area detection. To achieve these functions, a camera module is connected to the UAV 10. Specifically, the UAV 10 and the camera module are connected via a connecting structure, such as a vibration-damping ball. This camera module is used to acquire footage during aerial photography operations conducted by the UAV 10.

[0051] Specifically, the camera assembly includes a gimbal 14 and a shooting device 15. The gimbal 14 is connected to the unmanned aerial vehicle (UAV) 10. The shooting device 15 is mounted on the gimbal 14 and can be an image acquisition device used to capture images. This shooting device 15 includes, but is not limited to, a camera (e.g., a downward / forward depth camera), a video camera, a webcam, a scanner, or a camera phone. The gimbal is used to mount the shooting device to fix it in place or to freely adjust its attitude (e.g., change its height, tilt, and / or direction) and to keep it stably in a set attitude. For example, when the UAV 10 is taking aerial photos, the gimbal 14 is mainly used to keep the shooting device stably in a set attitude, preventing camera shake and ensuring stable footage.

[0052] The gimbal 14 is connected to the flight controller to enable data interaction between the two. For example, the flight controller sends a yaw command to the gimbal 14, which receives and executes the yaw speed and direction commands, and sends the data generated after executing the yaw command to the flight controller so that the flight controller can detect the current yaw status.

[0053] The gimbal 14 includes a gimbal motor and a gimbal base. The gimbal motor is mounted on the gimbal base. The flight controller can also control the gimbal motor via the electronic speed controller (ESC) of the power unit 13. Specifically, the flight controller is connected to the ESC, and the ESC is electrically connected to the gimbal motor. The flight controller generates control commands for the gimbal motor, and the ESC controls the gimbal motor via these control commands.

[0054] The gimbal base is connected to the aircraft fuselage and is used to fix the camera components to the aircraft fuselage.

[0055] The gimbal motors are connected to both the gimbal base and the shooting device. This gimbal can be a multi-axis gimbal, and correspondingly, multiple gimbal motors are used, with one motor for each axis. The gimbal motors drive the rotation of the shooting device, allowing for horizontal rotation and pitch angle adjustments on the shooting axis. The rotation can be manually or remotely controlled, or programmed to rotate automatically, achieving omnidirectional scanning and monitoring. Furthermore, during aerial photography, the rotation of the gimbal motors counteracts disturbances to the shooting device in real time, preventing camera shake and ensuring stable footage.

[0056] The imaging device 15 is mounted on the gimbal 14. The imaging device 15 is equipped with an inertial measurement unit (IMU), which measures the three-axis attitude angles (or angular rates) and acceleration of an object. Typically, an IMU contains three-axis gyroscopes and three-directional accelerometers to measure the object's angular velocity and acceleration in three-dimensional space, and thereby calculate the object's attitude. To improve reliability, more sensors can be equipped for each axis. Generally, the IMU is mounted at the center of gravity of the aircraft.

[0057] In some embodiments, the unmanned aerial vehicle 10 includes unmanned aerial vehicles such as multi-rotor drones, fixed-wing drones, unmanned helicopters, and hybrid-wing drones. The unmanned aerial vehicle 10 can also be an unmanned aerial vehicle powered by any type of propulsion, including but not limited to rotary-wing drones, fixed-wing drones, paragliding drones, flapping-wing drones, and helicopter models.

[0058] In some embodiments, the mobile terminal 20 includes a smart terminal, which can be any type of smart device used to establish a communication connection with the unmanned aerial vehicle 10, such as a mobile phone, tablet computer, or smart remote control. The mobile terminal 20 may be equipped with one or more different user interaction devices for collecting user commands or displaying and providing feedback to the user. Alternatively, the mobile terminal 20 includes a terminal device, which includes a computer, PC, or other device that establishes a communication connection with the unmanned aerial vehicle 10. This terminal device may be equipped with one or more different user interaction devices for collecting user commands or displaying and providing feedback to the user.

[0059] The aforementioned user interaction devices include, but are not limited to, buttons, mice, keyboards, displays, touchscreens, microphones, speakers, and remote control joysticks. For example, the mobile terminal 20 may be equipped with a microphone to acquire the user's voice and control the unmanned aerial vehicle 10 by recognizing the user's voice; or, the mobile terminal 20 may be equipped with a touchscreen display to receive the user's remote control commands to the unmanned aerial vehicle 10 and display information to the user, such as displaying AR real-world images and aerial images (i.e., image transmission images). The user can also switch the currently displayed image information on the screen using the remote control touchscreen. The user can also control the movement of the unmanned aerial vehicle, or control the gimbal direction and the focal length of the gimbal camera, etc., by operating the mouse or pressing the keyboard.

[0060] In some embodiments, the unmanned aerial vehicle 10 and the mobile terminal 20 can also integrate existing image vision processing technologies to provide more intelligent services. For example, the unmanned aerial vehicle 10 can acquire images through dual-light cameras, and the mobile terminal 20 can analyze the images to enable users to control the unmanned aerial vehicle 10 with gestures.

[0061] In some embodiments, the wireless network can be a wireless communication network based on any type of data transmission principle used to establish a data transmission channel between two nodes, such as a Bluetooth network, a WiFi network, a wireless cellular network, or a combination thereof located in different signal frequency bands.

[0062] Specifically, multiple UWB base stations 30 are deployed around the indoor environment, forming a UWB positioning base station network. The multiple UWB base stations 30 use TDOA or TOF algorithms to measure distances with the UWB positioning tags on the unmanned aerial vehicle 10.

[0063] The indoor navigation method for unmanned aerial vehicles provided in this application is described below through specific embodiments.

[0064] Please see Figure 2 , Figure 2 This is a flowchart illustrating an indoor navigation method for an unmanned aerial vehicle provided in an embodiment of this application.

[0065] The indoor navigation method for this unmanned aerial vehicle (UAV) can be applied to a mobile terminal. Specifically, the execution entity of the indoor navigation method is one or more processors of a mobile terminal. The mobile terminal includes, but is not limited to, devices such as remote controllers, mobile terminals, and tablet computers.

[0066] like Figure 2 As shown, the indoor navigation method of this unmanned aerial vehicle includes the following steps S201-S205: Step S201: In response to a touch operation on the AR real-world image of the control program, determine the location of each UWB base station in the AR real-world image.

[0067] Specifically, the mobile terminal supports ARCore / ARKit and has a built-in UWB chip. The mobile terminal is equipped with a control program (APP) to display AR real-world images. Operators can establish an AR world coordinate system by scanning the ground in the indoor environment. Furthermore, operators can perform touch operations on the AR real-world image in the control program, such as clicking. This clicking operation is used to click on each UWB base station in the indoor environment to determine the location of each UWB base station in the AR real-world image.

[0068] Understandably, in an indoor environment, a UWB base station is a physically existing piece of hardware, while the AR system (such as ARCore / ARKit) of the mobile terminal's control program (APP) runs in an independent virtual coordinate system. By having the operator click on the location of the UWB base station in the AR real-world scene, the click location is obtained, thus locating the physical UWB base station in the AR real-world scene. This allows the control program to obtain the two-dimensional coordinates of the click location on the AR screen and combine it with depth information to convert it into three-dimensional coordinates in the AR world coordinate system.

[0069] Step S202: Based on the location of each UWB base station in the AR real-world image, and based on the distance information between the mobile terminal and each UWB base station, combined with the visual inertial odometry of the mobile terminal, calculate the three-dimensional coordinates of each UWB base station in the pre-built AR world coordinate system to construct the base station coordinate system. Specifically, the mobile terminal's control program automatically calculates the three-dimensional coordinates of each UWB base station in the AR world coordinate system using the mobile terminal's visual inertial odometry (VIO) and built-in UWB ranging data.

[0070] Understandably, visual inertial odometry (VIO) is used to record the continuous trajectory (relative pose change) of a mobile terminal moving in space, while the UWB chip provides the absolute distance constraint from the mobile terminal to the UWB base station. Solving for the 3D coordinates is essentially a multi-source sensor fusion optimization problem (typically using extended Kalman filter (EKF) or graph optimization), for example: Assuming there are 3 UWB base stations (A, B, C) in the indoor environment (room), the solution process is as follows: (1) The operator walks around the room with a mobile terminal in hand. VIO records the precise relative position of the mobile terminal at times t_1, t_2, and t_3 (e.g., it moves forward 2 meters and to the right 1 meter).

[0071] (2) At time t_1, the UWB chip measured the distance between the mobile terminal and UWB base station A as 3 meters; at time t_2, the distance between the mobile terminal and UWB base station A was measured as 2.5 meters.

[0072] (3) The system constructs an optimization equation: find a set of three-dimensional coordinates of base stations A, B, and C, so that the error between the calculated theoretical distance and the actual distance measured by the UWB chip on the trajectory recorded by VIO is minimized.

[0073] (4) By solving the equation using the least squares method, the precise relative three-dimensional coordinates of UWB base stations A, B, and C in the AR world coordinate system can be automatically calculated.

[0074] It should be noted that the built-in UWB ranging data refers to the actual physical distance (usually in meters) between the mobile terminal's UWB chip and each physical UWB base station. This built-in UWB ranging data is obtained as follows: When the mobile terminal with the built-in UWB chip is running its control program, the mobile terminal acts as a UWB tag. By sending ranging pulses and receiving responses from the UWB base stations, it uses Two-Way Ranging (TWR) or Time Difference of Arrival (TDOA) algorithms to directly obtain the precise distance from the mobile terminal's current location to each UWB base station from the underlying hardware API, thus obtaining the built-in UWB ranging data.

[0075] After obtaining the three-dimensional coordinates of each UWB base station in the pre-built AR world coordinate system, the base station coordinate system is constructed, including: After calculating the three-dimensional coordinates of all UWB base stations, one base station (such as base station A) is selected as the origin (0,0,0), the line connecting base station A and base station B is selected as the X-axis, the plane containing base stations A, B, and C is selected as the XY plane, and the Z-axis is determined according to the right-hand rule to construct a base station coordinate system based on the base station network.

[0076] Step S203: Align the base station coordinate system with the local coordinate system of the UAV's takeoff point, and send the three-dimensional coordinates of each UWB base station to each UWB base station and the UAV.

[0077] It is understandable that the essence of alignment is to find the rotation matrix (R) and translation vector (T) between two coordinate systems.

[0078] Specifically, the alignment process includes: (1) The unmanned aerial vehicle is placed at the takeoff point, and the control program APP measures the coordinates P_{base} of the takeoff point of the unmanned aerial vehicle in the base station coordinate system through the UWB network.

[0079] (2) The coordinates of the take-off point of the unmanned aerial vehicle in the local coordinate system are the origin P_{local}=(0,0,0).

[0080] (3) Determine the translation vector T, which is P_{base}.

[0081] (4) Before the UAV takes off, its heading angle is determined by the onboard compass or IMU, and the control program APP obtains the north reference of the AR world coordinate system through the compass. By comparing the heading difference between the two, the rotation matrix R is calculated.

[0082] (5) Combine the rotation matrix R and the translation vector T to establish the transformation matrix between the base station coordinate system and the local coordinate system.

[0083] Once the transformation matrix is ​​determined, any position of the unmanned aerial vehicle in the local coordinate system can be transformed to the base station coordinate system using this transformation matrix, and vice versa.

[0084] Please refer to the following: Figure 3 , Figure 3 This is a schematic diagram of a base station calibration process provided in an embodiment of this application.

[0085] like Figure 3 As shown, the calibration process for this base station includes the following steps S301-S308: Step S301: Enable the AR calibration function of the control program.

[0086] Specifically, the operator launches the control program APP on the mobile terminal equipped with the control program and manually enables the AR calibration function. After the function is activated, the program completes internal initialization: loading the AR engine (ARCore / ARKit), waking up the terminal's built-in UWB chip, activating the visual inertial odometry (VIO) module, and opening relevant interfaces such as touch interaction, data acquisition, and coordinate calculation. The system enters the calibration state, preparing the hardware and software for subsequent AR mapping, base station positioning, and distance measurement calculation.

[0087] Step S302: Scan the ground using a mobile terminal to establish an AR world coordinate system.

[0088] Specifically, the operator holds a mobile terminal and slowly moves within the indoor space to be navigated, scanning the ground and surrounding environment. The mobile terminal collects visual features of the environment through its camera, and combined with the built-in IMU inertial data, the VIO module tracks the terminal's pose and spatial trajectory in real time. Based on the extracted environmental feature points and planar information, the AR engine autonomously constructs a unified AR world coordinate system. This AR world coordinate system serves as the spatial reference for the entire calibration process; all subsequent position data from base stations, terminals, and aircraft are first uniformly mapped to this coordinate system. Once the AR world coordinate system is constructed, the program indicates that the scan is complete.

[0089] Step S303: In response to a touch operation on the AR real-world image of the control program, determine the location of each UWB base station in the AR real-world image.

[0090] Specifically, the AR real-world imagery is overlaid in real time to display the real indoor environment and virtual visual markers. Operators, referring to the actual placement of the physical UWB base stations, perform click operations at the corresponding locations on the terminal screen. The terminal captures the two-dimensional pixel coordinates of the touch point on the screen; combined with depth detection and spatial pose data from the AR camera, the two-dimensional screen coordinates are converted into initial three-dimensional points in the AR world coordinate system, thus marking the approximate location of the current physical UWB base station in virtual space; the click marking is performed sequentially on all deployed UWB base stations indoors, and the program records the initial AR position information for each base station.

[0091] Step S304: Calculate the three-dimensional coordinates of the UWB base station.

[0092] Step S305: Have all UWB base stations been calibrated?

[0093] Step S306: Align the base station coordinate system with the local coordinate system of the unmanned aerial vehicle's takeoff point.

[0094] Step S307: Send the three-dimensional coordinates of each UWB base station to each UWB base station and the unmanned aerial vehicle.

[0095] Specifically, after the coordinate system alignment and transformation matrix are generated, the mobile terminal performs data transmission operations through the UWB wireless communication network: broadcasting the three-dimensional coordinate data of each base station to all indoor UWB base stations, enabling each base station to store the location information of all base stations in the network and support UWB positioning network calculations; and transmitting the coordinates of all UWB base stations and the coordinate system transformation matrix to the unmanned aerial vehicle; the aircraft receives and stores the data, and can use this set of parameters to complete its own position calculation, trajectory planning and navigation control during subsequent flight.

[0096] Step S308: Calibration complete.

[0097] Specifically, once all data has been distributed, the program determines that the entire UWB base station calibration and coordinate system alignment process is complete. The mobile terminal exits the AR calibration function, and the unmanned aerial vehicle and UWB base station enter normal working mode, enabling the initiation of routine services such as indoor navigation and flight missions.

[0098] Step S204: During the flight of the unmanned aerial vehicle, construct the UWB confidence field of the indoor space.

[0099] It is understandable that the UWB confidence field refers to the distribution map of the UWB positioning reliability (confidence) at each location in an indoor three-dimensional space. The UWB confidence field includes each spatial coordinate (x, y, z) and its corresponding confidence, and the value of the confidence is usually normalized to [0, 1].

[0100] It should be noted that, in the embodiments of this application, the confidence level is a weighted fusion of multiple signal quality indicators, wherein the signal quality indicators include: (1) Received Signal Strength Indication (RSSI): The stronger the signal, the higher the confidence level.

[0101] (2) Ranging variance: The degree of fluctuation in the ranging of the same base station in multiple consecutive measurements. The greater the fluctuation (there may be multipath effect or blockage), the lower the confidence level.

[0102] (3) Non-line-of-sight (NLOS) identification factor: By analyzing the UWB pulse waveform (such as the time difference between the first path signal and the strongest path signal), it is determined whether there is occlusion. The higher the NLOS probability, the lower the confidence level.

[0103] (4) Geometric precision factor (GDOP): It is determined by the spatial geometry of the current location and each base station. The worse the geometric distribution, the lower the confidence level.

[0104] Specifically, constructing the UWB confidence field for an indoor space includes the following steps: (1) During the flight of the unmanned aerial vehicle, record the current position (x, y, z) and its corresponding signal quality indicators (RSSI, Variance, NLOS, GDOP, etc.) in real time.

[0105] (2) Calculate the confidence level C of the current position using the preset evaluation function, where the preset evaluation function C = f({RSSI},{Variance},{NLOS},{GDOP}).

[0106] (3) Using spatial interpolation algorithms (such as Gaussian process regression GPR or Kriging interpolation), the confidence of areas not yet flown is calculated and predicted based on the confidence data of areas that have been flown over, so as to construct a continuous UWB confidence field covering the entire indoor space.

[0107] Step S205: Dynamically adjust the navigation mode of the unmanned aerial vehicle (UAV) based on the UWB confidence field to control the flight of the UAV.

[0108] For details, please refer to [link / reference]. Figure 4 , Figure 4 This is provided by the embodiments of this application. Figure 2 A detailed flowchart of step S205 in the process.

[0109] like Figure 4 As shown, step S205: dynamically adjusts the navigation mode of the unmanned aerial vehicle (UAV) based on the UWB confidence field to control the flight of the UAV, including the following steps S251-S253: Step S251: If the UAV enters a region where the confidence level is within the first confidence level range, then switch to UWB dominant mode, wherein the first confidence level range includes: UWB confidence level greater than or equal to the first confidence level threshold.

[0110] Step S252: If the UAV enters an area where the confidence level is within the second confidence level range, then switch to the UWB and vision fusion mode. The second confidence level range includes: the UWB confidence level is less than the first confidence level threshold, and the UWB confidence level is greater than or equal to the second confidence level threshold.

[0111] Step S253: If the UAV enters a region where the confidence level is within the third confidence level range, switch to vision and IMU-dominated mode. The third confidence level range includes: UWB confidence level less than the second confidence level threshold.

[0112] Specifically, based on a pre-constructed UWB confidence field of the indoor space, which includes the confidence of each coordinate in the indoor space, the coordinates of the UAV are acquired in real time during the flight of the UAV, so that the confidence corresponding to the coordinate is obtained based on the real-time coordinates of the UAV and the UWB confidence field.

[0113] The confidence level C ranges from [0, 1]. The first confidence threshold and the second confidence threshold can be set according to specific needs. For example, the first confidence threshold and the second confidence threshold can range from (0, 1). The first confidence threshold is greater than the second confidence threshold. For example, if the first confidence threshold is 0.85 and the second confidence threshold is 0.5, then the first confidence level range is [0.85, 1], the second confidence level range is (0.5, 0.85], and the third confidence level range is [0, 0.5].

[0114] It should be noted that the UWB-dominated mode, the UWB and vision fusion mode, and the vision and IMU-dominated mode represent the adaptive strategies of multi-sensor fusion systems in different environments.

[0115] Specifically, the UWB-dominated mode includes: the system sets the weight of UWB to the highest level, the localization result relies on the absolute coordinates provided by UWB, and visual SLAM and IMU are only used to provide high-frequency attitude updates and short-term smooth interpolation. Understandably, the UWB-dominated mode is suitable for open, unobstructed central indoor areas with excellent UWB signal, offering the highest localization accuracy and no cumulative error.

[0116] Specifically, the UWB and vision fusion mode includes dynamically balancing the weights of UWB and visual SLAM. UWB provides global constraints to limit drift, while visual SLAM provides high-precision local relative motion estimation to smooth UWB fluctuations. Understandably, the UWB and vision fusion mode can be applied to areas with partial occlusion (such as next to shelves or behind pillars), where UWB signals exhibit fluctuations or multipath effects. Through deep coupling of UWB and vision, their complementary advantages maintain stable navigation performance.

[0117] Specifically, the vision and IMU-dominated mode includes: the system reduces the UWB weight to extremely low or even zero, and relies entirely on visual SLAM and IMU for local odometry (VIO).

[0118] Understandably, the vision- and IMU-dominated mode can be used in severe non-line-of-sight (NLOS) areas, such as enclosed corridors and corners with severe metal shielding, where UWB signals are severely attenuated or lost. While long-term operation in vision- and IMU-dominated mode may result in cumulative drift, it ensures the safe flight of the UAV for a short period until it flies out of the blind spot and reacquires a high-quality UWB signal.

[0119] It should be noted that UWB weight refers to the degree of confidence assigned to UWB observation data in multi-sensor fusion localization algorithms (such as Extended Kalman Filter (EKF) or Error State Kalman Filter (ESKF)). The role of UWB weight is that the final localization result of the unmanned aerial vehicle is obtained by weighted averaging of data from multiple sensors, including UWB, visual SLAM, and IMU.

[0120] In this embodiment of the application, the method further includes: Based on the confidence level of the UAV's location, the system adaptively adjusts the UWB weights to dynamically switch navigation modes. For example, when the UAV enters an area where the UWB confidence level is lower than the first confidence threshold, the system automatically reduces the UWB weights and smoothly switches to a local navigation mode dominated by vision / laser SLAM.

[0121] In this embodiment of the application, the dynamic adjustment of UWB weights ensures a smooth transition in the positioning of the unmanned aerial vehicle under different signal environments, preventing crashes caused by the failure of a single sensor.

[0122] In this embodiment of the application, the unmanned aerial vehicle includes a depth camera. During the flight of the unmanned aerial vehicle, this embodiment of the application also uses three-dimensional point cloud fusion rendering to obtain a three-dimensional digital twin scene.

[0123] For details, please refer to Figure 5 , Figure 5 This is a schematic diagram of a process for rendering a three-dimensional digital twin scene provided in an embodiment of this application.

[0124] like Figure 5 As shown, the process of rendering a 3D digital twin scene includes the following steps S501-S502: Step S501: Acquire the environmental point cloud collected in real time by the depth camera.

[0125] Specifically, the depth camera on the drone continuously collects raw data of the surrounding environment during flight, generating local 3D environmental point clouds. After noise reduction and simplification preprocessing, the drone transmits the point cloud data to the control program APP on the mobile terminal in real time via a wireless image transmission link.

[0126] For example, after the mobile terminal's control program (APP) receives the local point cloud transmitted back by the UAV, it first performs downsampling (such as voxel grid filter) and noise reduction (such as statistical filtering) to reduce the amount of computation.

[0127] Step S502: Register the environmental point cloud with the pre-imported indoor BIM model to obtain a three-dimensional digital twin scene in real time.

[0128] Specifically, the control program loads a preset indoor BIM model and uses a point cloud registration algorithm to achieve spatial alignment between the real-time point cloud and the BIM model. For example: (1) The control program APP directly uses the current three-dimensional coordinates and attitude angles of the unmanned aerial vehicle to place the local point cloud to the approximate corresponding position of the BIM model (or historical point cloud) for initial alignment (coarse registration).

[0129] (2) The Iterative Closest Point (ICP) algorithm or its variant (such as the NDT algorithm) is used to find the nearest point pair on the local point cloud and the surface of the BIM model, calculate and minimize the distance error between them, thereby fine-tuning the pose of the local point cloud for accurate registration (fine registration) so that it fits perfectly with the BIM model.

[0130] After registration is completed, the 3D rendering engine of the control program APP (such as Unity, Unreal or a self-developed engine based on OpenGL) uses the BIM model as a static background and overlays the real-time registered local point cloud (which may include RGB color information) on top of it to merge the static BIM model with the dynamic real point cloud data. Then, the 3D rendering engine draws the picture in real time and generates and displays the indoor 3D digital twin scene on the screen of the mobile terminal.

[0131] Understandably, as unmanned aerial vehicles continue to fly, point clouds are constantly accumulated and updated, forming a dynamic three-dimensional digital twin scene.

[0132] In this embodiment of the application, by combining the BIM static model with the real-time point cloud of the depth camera, it is possible to take into account both the overall indoor layout and dynamic environmental changes, thereby improving the fidelity of the digital twin scene.

[0133] Please refer to the following: Figure 6 , Figure 6 This is a schematic diagram of a process for controlling the synchronous movement of a virtual unmanned aerial vehicle model in a three-dimensional digital twin scene, provided by an embodiment of this application.

[0134] like Figure 6 As shown, the process of controlling the synchronous movement of the virtual unmanned aerial vehicle model in a 3D digital twin scene includes the following steps S601-S603: Step S601: Construct a virtual unmanned aerial vehicle model in a 3D digital twin scene; Step S602: Fuse the UWB positioning data and IMU data acquired by the unmanned aerial vehicle to obtain the real-time pose of the unmanned aerial vehicle.

[0135] Specifically, UWB positioning data (low frequency, high precision) and IMU data (high frequency, low precision) are fused using an extended Kalman filter (EKF) to obtain a smooth, high-frequency 6DOF pose. For example: a system state equation is established, with the aircraft's 6DOF pose as the state variable to be estimated; high-frequency IMU data is used to perform state prediction, calculating continuous pose changes of the aircraft within a short time to fill the update gaps in the UWB data; when low-frequency UWB positioning data arrives, it is used as an observation to correct the drift error caused by IMU integration; through iterative filtering, the state estimate is continuously optimized, ultimately outputting high-frequency, low-drift, smooth, and continuous real-time 6DOF pose data.

[0136] Step S603: Based on the real-time pose of the unmanned aerial vehicle, drive the virtual unmanned aerial vehicle model to perform synchronous motion in the three-dimensional digital twin scene.

[0137] Specifically, the real-time pose of the unmanned aerial vehicle (UAV) is synchronized to the control program on the mobile terminal to drive the synchronous movement of the virtual UAV model in the 3D scene. For example, the 3D rendering engine of the control program assigns the real-time pose to the virtual UAV model, updates the model's spatial position in the scene based on the 3D coordinates, adjusts the model's deflection and tilt angles based on the three-axis attitude angles, and refreshes the model's state frame by frame according to the data update frequency, replicating the flight trajectory, turning, ascent, descent, and attitude changes of the real aircraft.

[0138] In this embodiment, the real-time pose is a six-degree-of-freedom parameter, including three-dimensional position parameters and three-axis attitude angle parameters. The three-dimensional position parameters are the X, Y, and Z coordinates in a coordinate system, used to characterize the spatial position of the unmanned aerial vehicle (UAV). The three-axis attitude angles include roll, pitch, and yaw angles, used to characterize the UAV's orientation and tilt. The entire set of pose parameters is obtained by fusing UWB data and IMU data, and is used to drive the synchronous movement of the virtual UAV model within the three-dimensional digital twin scene.

[0139] Furthermore, embodiments of this application also utilize UWB confidence fields to achieve dynamic obstacle avoidance.

[0140] For details, please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of a process for controlling the flight of an unmanned aerial vehicle provided in an embodiment of this application.

[0141] like Figure 7 As shown, the process for controlling the flight of an unmanned aerial vehicle includes the following steps S701-S703: Step S701: Obtain the location information of dynamic obstacles detected by the depth camera.

[0142] Specifically, during the flight of the unmanned aerial vehicle (UAV), the depth camera continuously outputs real-time environmental point clouds and visual images. The onboard unit performs differential calculations on the inter-frame data, compares the positional changes of point clouds and image features across multiple frames, and distinguishes between static environments (walls, fixed equipment) and dynamic obstacles (pedestrians, mobile forklifts, temporary stacks of materials, etc.).

[0143] By combining depth data, camera intrinsic parameters, and the aircraft's real-time 6DOF pose, the coordinates of the obstacle in the camera coordinate system are transformed to the global base station coordinate system / digital twin scene coordinate system. The center coordinates, outline, occupied space, movement speed, and direction of movement of the obstacle are calculated to form complete obstacle position and movement information.

[0144] Step S702: Generate a dynamic no-fly zone in real time in the AR real-world scene based on the location information of the dynamic obstacles.

[0145] Specifically, based on the spatial outline of dynamic obstacles, a preset safety buffer distance is extended outward to determine the dynamic no-fly zone. The size of the dynamic no-fly zone can be adaptively adjusted according to the size of the aircraft and its flight speed.

[0146] Step S703: Generate a detour trajectory based on the trajectory algorithm to control the flight of the unmanned aerial vehicle.

[0147] Specifically, the trajectory algorithm includes the A* algorithm or the RRT algorithm. By combining dynamic obstacles (such as moving forklifts and people) detected by the depth camera, a dynamic no-fly zone (highlighted in red) is generated in real time in the AR real-world image of the control program APP. The A* algorithm or the RRT algorithm is then used to replan the detour trajectory in the local space.

[0148] In this embodiment, by adapting to two mainstream path planning algorithms, A* and RRT, the appropriate algorithm can be flexibly selected according to the indoor environment, and a detour trajectory can be quickly generated to enable autonomous dynamic obstacle avoidance of the unmanned aerial vehicle, thereby improving the safety of indoor flight operations.

[0149] Furthermore, embodiments of this application can also realize Point-to-Fly.

[0150] Please refer to the following: Figure 8 , Figure 8 This is a schematic diagram of a process for controlling an unmanned aerial vehicle to move to a physical coordinate position according to an embodiment of this application.

[0151] like Figure 8 As shown, the process of controlling the unmanned aerial vehicle to move to its physical coordinates includes the following steps S801-S803: Step S801: Obtain the position command in the AR real-world image, where the position command corresponds to a two-dimensional position coordinate.

[0152] Specifically, the mobile terminal continuously renders real-time AR scene images, simultaneously overlaying the indoor real environment, UWB base station, drone, and obstacle visual information. Operators perform touch operations such as clicking and long-pressing within the AR scene to issue position commands for pointing and flying. By acquiring the two-dimensional pixel coordinates of the touch point on the screen, these coordinates become the original position command corresponding to this pointing and flying action. The program completes coordinate caching and parsing, then proceeds to the next conversion step.

[0153] Step S802: Convert the two-dimensional position coordinates into three-dimensional virtual coordinates, and then convert the three-dimensional virtual coordinates into the physical coordinates of the unmanned aerial vehicle.

[0154] Specifically, the mobile terminal's 3D rendering engine inversely projects the screen's 2D coordinates into 3D virtual coordinates, and then maps them to UWB target coordinates in the physical world. For example: First, based on the AR camera's intrinsic parameters and the current camera's six-DOF pose, inverse projection calculations are performed on the two-dimensional pixels of the screen. Combining depth information and indoor floor / wall planar features, the three-dimensional virtual coordinates (X, Y, Z) of the clicked point in the AR world coordinate system are calculated. These three-dimensional virtual coordinates share the same virtual spatial reference as the AR real-world image and the digital twin scene.

[0155] Then, the three-dimensional virtual coordinates are converted into the physical coordinates of the unmanned aerial vehicle, including: The coordinate transformation matrix formed by the rotation matrix R and the translation vector T obtained from the base station calibration is called to perform calculations on the three-dimensional virtual coordinates in the AR world coordinate system, thereby completing the mapping of the virtual space to the base station coordinate system and obtaining the real physical three-dimensional coordinates corresponding to the target point. These physical coordinates are the target points that can be directly used for navigation and positioning of the unmanned aerial vehicle, serving as the flight endpoint.

[0156] Step S803: Generate a collision-free trajectory based on physical coordinates to control the unmanned aerial vehicle to move to the location of the physical coordinates.

[0157] Specifically, starting from the current real-time physical position of the UAV and ending at the target physical coordinates obtained from the above transformation, and combining the static walls, furniture, dynamic obstacles, and dynamic no-fly zones detected by the depth camera, the entire area of ​​impassable areas is marked. Path planning algorithms (A* algorithm, RRT algorithm, etc.) are used to search for continuous waypoints in the feasible space to generate a smooth, collision-free flight trajectory from the starting point to the ending point. At the same time, the flight altitude and flight speed are adaptively planned according to the flight environment to avoid all obstacles and no-fly zones.

[0158] Afterwards, the mobile terminal sends the planned trajectory waypoints and flight commands to the unmanned aerial vehicle (UAV) via a wireless link; after receiving the commands, the UAV combines its real-time pose obtained from its own UWB+IMU fusion positioning and tracks the flight along the preset trajectory waypoint by waypoint.

[0159] During flight, the UAV continuously compares its current physical coordinates with the target physical coordinates. When the deviation between the two is less than a preset distance threshold, it is determined that it has reached the designated location, and the UAV enters a hovering state, thus ending the current guided flight process.

[0160] Furthermore, the unmanned aerial vehicle includes scanning equipment, which may include RFID readers or barcode scanners.

[0161] Please refer to the following: Figure 9 , Figure 9 This is a schematic diagram of a process for binding scanning information and scanning target provided in an embodiment of this application.

[0162] like Figure 9 As shown, the process of binding scan information with the scan target includes the following steps S901-S902: Step S901: During the flight of the unmanned aerial vehicle, acquire the scanning information obtained by the scanning device scanning the target.

[0163] Specifically, during autonomous flight along a preset or directed flight path, the unmanned aerial vehicle continuously activates its onboard scanning and data acquisition equipment to scan and acquire targets in the environment in real time.

[0164] In warehouse inventory management scenarios, unmanned aerial vehicles (UAVs) equipped with RFID readers and barcode / QR code scanning modules can scan shelves, storage locations, and cargo labels at close range to read cargo information such as unique identifiers, batch information, product type, and entry time in real time.

[0165] In security patrol scenarios, unmanned aerial vehicles are equipped with visible light cameras, infrared cameras, and environmental sensors to collect image frames, temperature data, and abnormal features in real time during patrols. They also use image recognition algorithms to detect security anomalies such as personnel intrusion, open flames, abnormal high temperatures, and water leakage, and retain corresponding on-site images, videos, anomaly types, and anomaly levels.

[0166] Each time the scanning device successfully collects valid data, it generates a frame of structured scan information and uploads it to the mobile control program in real time, completing the scan data collection and caching.

[0167] Step S902: Bind the scanning information to the three-dimensional coordinates of the scanning target.

[0168] Specifically, while acquiring scanning information, the system simultaneously acquires the high-precision real-time three-dimensional physical coordinates (X, Y, Z) of the unmanned aerial vehicle at the current moment, calculated by UWB+IMU fusion. These coordinates represent the actual spatial position of the aircraft in the global base station coordinate system at the instant of scanning. The control program associates and binds the scanning information with the current high-precision three-dimensional spatial coordinates one by one, forming a pair of data records of "spatial coordinates-target information", thereby achieving precise anchoring of data and physical position.

[0169] For example, in warehouse inventory scenarios, the system binds the read product ID, batch, category, and other product information to the corresponding 3D coordinates of the shelf at the moment of scanning, generating spatial anchor points for the products in a 3D digital twin scenario. Each anchor point precisely corresponds to the specific shelf layer, location, and spatial position, solving the problem of traditional inventory counting having only product information but no precise spatial location, thus achieving digital, visual, and location-based management of goods.

[0170] For security patrol scenarios, the system identifies abnormal events such as intrusions, fires, leaks, and high temperatures, along with captured images, and links them to the 3D coordinates of the location where the anomaly occurred. This generates security alarm anchor points on the corresponding spatial locations within a 3D digital twin map. Security personnel can directly click on these anchor points in the 3D scene to view the anomaly's location, type, and precise spatial coordinates, enabling the location, traceability, and rapid response to abnormal events.

[0171] In this embodiment, all anchor point data after binding is stored and visualized in the three-dimensional digital twin scene in real time, realizing a one-to-one correspondence between the scanned target information and the real physical space, and completing the spatial association between the scanned information and the scanned target.

[0172] The following example of automated inventory counting in a large-scale automated warehouse illustrates the processing procedure of this application: (1) Environment initialization and calibration: The warehouse manager holds a smartphone with the control program APP installed and places UWB base stations in the four corners of the warehouse. The manager opens the "AR calibration" function of the control program APP, scans the ground, and then points the phone camera at the four UWB base stations in turn and taps the screen. The control program APP automatically calculates the relative three-dimensional coordinates of the four UWB base stations within 1 minute and establishes the local navigation coordinate system of the warehouse.

[0173] (2) Task issuance and takeoff: The administrator imports the warehouse BIM model into the control program APP, and the interface of the control program APP presents a three-dimensional digital twin scene of the warehouse. The administrator selects the shelf area to be inventoried on the three-dimensional map (e.g., the 3rd to 5th shelves of the A area), and the App automatically generates a "bow"-shaped three-dimensional scanning route covering the area and sends it to the unmanned aerial vehicle.

[0174] (3) High-precision navigation and data acquisition: After takeoff, the UWB base station network is used for centimeter-level positioning. During flight, the barcode scanner at the front of the UWB continuously reads the labels on the goods on the shelf.

[0175] (4) Dynamic obstacle avoidance: When the UAV flies to an area with dense metal shelves, the UWB signal fluctuates (confidence decreases). The UAV automatically increases the weight of visual SLAM to maintain stable hovering, while the depth camera detects a forklift in operation ahead. The UAV transmits obstacle information back, and the control program APP marks the forklift's position as a red dynamic obstacle on the 3D map and replans a local trajectory to bypass the forklift upwards, allowing the UAV to pass safely.

[0176] (5) Data binding and visualization: For each item barcode scanned, the control program APP immediately obtains the current UWB 3D coordinates of the UAV (accurate to the specific shelf height and column number), and generates a green "item anchor point" at the corresponding position on the 3D map. After the inventory is completed, the administrator can not only see the inventory list in the control program APP, but also intuitively see the precise spatial distribution of all items on the 3D map. If items are found to be misplaced (e.g., items in area A appear in area B), the location can be directly located on the map and personnel can be assigned to handle the issue.

[0177] In this embodiment, an indoor navigation method for an unmanned aerial vehicle (UAV) is provided, applied to a mobile terminal. The mobile terminal is communicatively connected to multiple UWB base stations and has a control program installed. The method includes: in response to a touch operation on an AR real-world image displayed in the control program, determining the position of each UWB base station in the AR real-world image; based on the position of each UWB base station in the AR real-world image, and using distance information between the mobile terminal and each UWB base station, combined with the visual inertial odometry of the mobile terminal, calculating the three-dimensional coordinates of each UWB base station in a pre-constructed AR world coordinate system to construct a base station coordinate system; aligning the base station coordinate system with the local coordinate system of the UAV's takeoff point, and sending the three-dimensional coordinates of each UWB base station to each UWB base station and the UAV; constructing a UWB confidence field for the indoor space during the flight of the UAV; and dynamically adjusting the navigation mode of the UAV based on the UWB confidence field to control the flight of the UAV.

[0178] By constructing a base station coordinate system using AR real-world images based on control programs, the deployment time of UWB base stations can be shortened and deployment efficiency improved without the need for specialized surveying equipment. Furthermore, by employing a dynamic obstacle avoidance strategy based on UWB confidence fields, the accuracy of indoor navigation can be improved, thereby enhancing flight safety.

[0179] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an indoor navigation device for an unmanned aerial vehicle provided in an embodiment of this application.

[0180] The indoor navigation device of the unmanned aerial vehicle is applied to a mobile terminal. Specifically, the indoor navigation device of the unmanned aerial vehicle is applied to one or at least two processors of the mobile terminal.

[0181] like Figure 10 As shown, the indoor navigation device 110 of the unmanned aerial vehicle is applied to a mobile terminal. The mobile terminal is connected to multiple UWB base stations and has a control program installed. The indoor navigation device 110 of the unmanned aerial vehicle includes: The location unit 111 is used to determine the location of each UWB base station in the AR real-world scene in response to a touch operation on the AR real-world scene of the control program. Coordinate unit 112 is used to calculate the three-dimensional coordinates of each UWB base station in the pre-built AR world coordinate system based on the location of each UWB base station in the AR real scene, the distance information between the mobile terminal and each UWB base station, and the visual inertial odometry of the mobile terminal, so as to construct the base station coordinate system. Alignment unit 113 is used to align the base station coordinate system with the local coordinate system of the unmanned aerial vehicle's takeoff point, and send the three-dimensional coordinates of each UWB base station to each UWB base station and the unmanned aerial vehicle. The building unit 114 is used to build the UWB confidence field of the indoor space during the flight of the unmanned aerial vehicle; The adjustment unit 115 is used to dynamically adjust the navigation mode of the unmanned aerial vehicle according to the UWB confidence field in order to control the flight of the unmanned aerial vehicle.

[0182] In the embodiments of this application, the indoor navigation device of the unmanned aerial vehicle (UAV) can also be constructed from hardware devices. For example, the indoor navigation device of the UAV can be constructed from one or more chips, and the chips can work together to complete the indoor navigation method of the UAV described in the above embodiments. As another example, the indoor navigation device of the UAV can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM processors (Advanced RISC Machines) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0183] The indoor navigation device for the unmanned aerial vehicle in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0184] The indoor navigation device for the unmanned aerial vehicle in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not impose specific limitations.

[0185] The indoor navigation device for unmanned aerial vehicles provided in this application embodiment can achieve… Figure 2 To avoid repetition, the various processes involved will not be described in detail here.

[0186] It should be noted that the indoor navigation device for the unmanned aerial vehicle (UAV) described above can execute the indoor navigation method for the UAV provided in the above embodiments, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the embodiments of the indoor navigation device for the UAV can be found in the indoor navigation method for the UAV provided in the above embodiments.

[0187] In this embodiment, an indoor navigation device for an unmanned aerial vehicle (UAV) is provided, comprising: a position unit for determining the position of each UWB base station in the AR real-world image in response to a touch operation on an AR real-world image in the control program; a coordinate unit for calculating the three-dimensional coordinates of each UWB base station in a pre-built AR world coordinate system based on the position of each UWB base station in the AR real-world image, the distance information between the mobile terminal and each UWB base station, and the visual inertial odometry of the mobile terminal, to construct a base station coordinate system; an alignment unit for aligning the base station coordinate system with the local coordinate system of the UAV's takeoff point, and sending the three-dimensional coordinates of each UWB base station to each UWB base station and the UAV; a construction unit for constructing a UWB confidence field of the indoor space during the flight of the UAV; and an adjustment unit for dynamically adjusting the navigation mode of the UAV based on the UWB confidence field to control the flight of the UAV.

[0188] By constructing a base station coordinate system using AR real-world images based on control programs, the deployment time of UWB base stations can be shortened and deployment efficiency improved without the need for specialized surveying equipment. Furthermore, by employing a dynamic obstacle avoidance strategy based on UWB confidence fields, the accuracy of indoor navigation can be improved, thereby enhancing flight safety.

[0189] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a mobile terminal provided in an embodiment of this application.

[0190] like Figure 11 As shown, the mobile terminal 20 includes one or more processors 201 and a memory 202. Wherein, Figure 11 Take a processor 201 as an example.

[0191] The processor 201 and the memory 202 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0192] Processor 201 provides computing and control capabilities to control mobile terminal 20 to perform corresponding tasks, such as controlling mobile terminal 20 to execute the indoor navigation method for unmanned aerial vehicles (UAVs) in any of the above method embodiments. The method is applied to the mobile terminal, which is communicatively connected to multiple UWB base stations and has a control program installed. The method includes: in response to a touch operation on an AR real-world image in the control program, determining the position of each UWB base station in the AR real-world image; calculating the three-dimensional coordinates of each UWB base station in a pre-constructed AR world coordinate system based on the position of each UWB base station in the AR real-world image, distance information between the mobile terminal and each UWB base station, and the mobile terminal's visual inertial odometry, to construct a base station coordinate system; aligning the base station coordinate system with the local coordinate system of the UAV's takeoff point, and sending the three-dimensional coordinates of each UWB base station to each UWB base station and the UAV; constructing a UWB confidence field for the indoor space during the UAV's flight; and dynamically adjusting the UAV's navigation mode based on the UWB confidence field to control the UAV's flight.

[0193] By constructing a base station coordinate system using AR real-world images based on control programs, the deployment time of UWB base stations can be shortened and deployment efficiency improved without the need for specialized surveying equipment. Furthermore, by employing a dynamic obstacle avoidance strategy based on UWB confidence fields, the accuracy of indoor navigation can be improved, thereby enhancing flight safety.

[0194] Processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0195] The memory 202, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the indoor navigation method of the unmanned aerial vehicle in the embodiments of this application. The processor 201 can implement the indoor navigation method of the unmanned aerial vehicle in any of the above method embodiments by running the non-transitory software programs, instructions, and modules stored in the memory 202. Specifically, the memory 202 may include volatile memory (VM), such as random access memory (RAM); the memory 202 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or other non-transitory solid-state storage devices; the memory 202 may also include combinations of the above types of memory.

[0196] Memory 202 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 202 may optionally include memory remotely located relative to processor 201, and these remote memories may be connected to processor 201 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0197] One or more modules are stored in memory 202. When executed by one or more processors 201, they perform the indoor navigation method for the unmanned aerial vehicle in any of the above method embodiments, for example, the method described above. Figure 2 The steps shown can also be implemented. Figure 10 The functions of each module or unit.

[0198] In this embodiment, the mobile terminal 20 may also have wired or wireless network interfaces, keyboards, and input / output interfaces for input and output. The mobile terminal 20 may also include other components for implementing device functions, which will not be described in detail here.

[0199] This application also provides a non-volatile computer-readable storage medium, such as a memory including program code, which can be executed by a processor to complete the indoor navigation method for the unmanned aerial vehicle in the above embodiments. For example, the non-volatile computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0200] This application also provides a computer program product comprising one or more lines of program code stored in a non-volatile computer-readable storage medium. The processor of a mobile terminal reads the program code from the non-volatile computer-readable storage medium and executes the program code to complete the method steps of the indoor navigation method for unmanned aerial vehicles provided in the above embodiments.

[0201] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program or program code related to hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The non-volatile computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations as described above in different aspects of this application, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An indoor navigation method for an unmanned aerial vehicle, characterized in that, Applied to a mobile terminal, the mobile terminal is communicatively connected to multiple UWB base stations, and the mobile terminal is equipped with a control program; The method includes: In response to a touch operation on the AR real-world image of the control program, the location of each of the UWB base stations in the AR real-world image is determined; Based on the location of each UWB base station in the AR real-world image, and based on the distance information between the mobile terminal and each UWB base station, combined with the visual inertial odometry of the mobile terminal, the three-dimensional coordinates of each UWB base station in the pre-built AR world coordinate system are calculated to construct the base station coordinate system. Align the base station coordinate system with the local coordinate system of the unmanned aerial vehicle's takeoff point, and send the three-dimensional coordinates of each UWB base station to each UWB base station and the unmanned aerial vehicle; During the flight of the unmanned aerial vehicle, a UWB confidence field of the indoor space is constructed; The navigation mode of the unmanned aerial vehicle is dynamically adjusted based on the UWB confidence field to control the flight of the unmanned aerial vehicle.

2. The method according to claim 1, characterized in that, The step of dynamically adjusting the navigation mode of the unmanned aerial vehicle based on the UWB confidence field includes: If the unmanned aerial vehicle enters a region where the confidence level is within the first confidence level range, it switches to UWB-dominated mode, wherein the first confidence level range includes: UWB confidence level greater than or equal to the first confidence threshold. If the unmanned aerial vehicle enters a region where the confidence level is within the second confidence level range, it switches to the UWB and vision fusion mode. The second confidence level range includes: the UWB confidence level is less than the first confidence level threshold, and the UWB confidence level is greater than or equal to the second confidence level threshold. If the unmanned aerial vehicle enters a region where the confidence level is within the third confidence level range, it switches to a vision and IMU-dominated mode. The third confidence level range includes: UWB confidence level less than the second confidence level threshold.

3. The method according to claim 1, characterized in that, The unmanned aerial vehicle includes a depth camera; During the flight of the unmanned aerial vehicle, the method further includes: Obtain the environmental point cloud acquired in real time by the depth camera; The environmental point cloud is registered with the pre-imported indoor BIM model to obtain a three-dimensional digital twin scene in real time.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the position information of the dynamic obstacles detected by the depth camera; Based on the location information of the dynamic obstacles, a dynamic no-fly zone is generated in real time in the AR real-world image; Based on the trajectory algorithm, a detour trajectory is generated to control the flight of the unmanned aerial vehicle.

5. The method according to claim 3, characterized in that, The method further includes: In the aforementioned three-dimensional digital twin scene, a virtual unmanned aerial vehicle model is constructed; The UWB positioning data and IMU data acquired by the unmanned aerial vehicle are fused to obtain the real-time pose of the unmanned aerial vehicle; Based on the real-time pose of the unmanned aerial vehicle, the virtual unmanned aerial vehicle model is driven to move synchronously in the three-dimensional digital twin scene.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the position command in the AR real-world image, wherein the position command corresponds to a two-dimensional position coordinate; The two-dimensional position coordinates are converted into three-dimensional virtual coordinates, and the three-dimensional virtual coordinates are then converted into the physical coordinates of the unmanned aerial vehicle. Based on the physical coordinates, a collision-free trajectory is generated to control the unmanned aerial vehicle to move to the location of the physical coordinates.

7. The method according to claim 1, characterized in that, The unmanned aerial vehicle includes a scanning device, which includes an RFID reader or a barcode scanner. The method further includes: During the flight of the unmanned aerial vehicle, the scanning information obtained by the scanning device scanning the target is acquired; The scanning information is bound to the three-dimensional coordinates of the scanning target.

8. A mobile terminal, characterized in that, include: At least one processor; At least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, such that at least one of the processors implements claim 1 7. Any one of the methods described.

9. An indoor navigation system for an unmanned aerial vehicle, characterized in that, include: The mobile terminal as described in claim 8; Multiple UWB base stations are connected to the mobile terminal.

10. A non-volatile computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform as claimed in claim 1.

7. Any one of the methods described.

Images