Lightweight indoor dangerous point checking method and device based on depth vision for airship

Abstract

The application provides a lightweight airship indoor dangerous point checking method and device based on depth vision. The lightweight airship indoor dangerous point checking method based on depth vision of the application embodiment realizes high-precision identification and positioning (error less than 50 mm within 3 meters) of indoor dangerous points under lightweight design (total load 120 g), and effectively improves the maneuverability and control stability of the airship through a vector propulsion system and a double-mode (Bluetooth manual and PID automatic) flight control algorithm, thereby meeting the unmanned dynamic inspection requirements in a complex indoor environment.

Description

Technical Field

[0001] This invention relates to the field of small airship technology for hazard detection, and more particularly to a lightweight airship indoor hazard detection method and device based on depth vision. Background Technology

[0002] With the continuous development of intelligent security and industrial automation technologies, indoor hazard detection systems based on unmanned platforms are playing an increasingly important role in scenarios such as sports stadiums, transportation hubs, and industrial facilities. Among related technologies, a dynamic monitoring and intelligent identification technology system has been constructed through the collaborative operation of multiple platforms, including drones, airships, and ground robots. Specifically, this technology covers the entire process from environmental perception, image acquisition, and data processing to hazard identification and location, including key aspects such as monocular / binocular vision, inertial navigation, and deep learning algorithms. In recent years, small airships have gradually become an important research direction in the field of indoor inspection due to their low noise, long endurance, and good safety, especially demonstrating unique advantages in scenarios requiring long-term, large-scale, and high-precision monitoring.

[0003] However, most existing small airship designs employ fixed propeller propulsion systems, failing to adequately consider the balance between lightweight design and multifunctional integration. This can lead to increased equipment weight, limited control freedom, or decreased flight stability, thus affecting their effectiveness in complex indoor environments. Specifically, traditional airships typically require 3-5 propellers for basic motion control, while also carrying multiple sensors for positioning and obstacle avoidance. While this design enhances functionality, it significantly increases system complexity and power consumption, limiting their deployment capabilities in low-load, high-maneuverability scenarios. Furthermore, existing technologies often combine monocular vision with traditional PID control, making it difficult to achieve high-precision autonomous positioning and real-time hazard identification, resulting in a significant technological gap in tasks requiring centimeter-level accuracy for inspection. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method for identifying indoor hazards in lightweight airships based on depth vision.

[0006] The second objective of this invention is to propose a lightweight airship indoor hazard detection device based on depth vision.

[0007] To achieve the above objectives, the first aspect of this invention proposes a method for identifying indoor hazards in a lightweight airship based on depth vision, comprising: S1, using a dual-propeller vector propulsion system connected by rotatable carbon fiber rods to achieve basic forward, backward, ascent, descent, and yaw movements of the airship in an indoor environment, with the airship's weight balanced by helium gasbags to reduce the coupling between motor thrust and servo angle; S2, based on image data acquired by a binocular camera, combined with ArUco marker recognition and four-point positioning algorithms, calculating the airship's real-time coordinates in three-dimensional space, and acquiring the airship's attitude angles through a nine-axis inertial sensor, fusing the attitude angle information with binocular visual ranging results to improve positioning accuracy; S3, transmitting the real-time images from the binocular camera to a host computer for processing, using a YOLOv5 deep learning model to train, identify, and measure the hazard points in the images, combining the identification results with the positioning data to generate spatial location information of the hazard points and feeding it back to a Raspberry Pi; S4 establishes a closed-loop data transmission mechanism through the Socket communication protocol between the Raspberry Pi and the host computer, enabling real-time feedback of image recognition and positioning information within 1 second, and supporting the airship to conduct dynamic hazard point investigation and autonomous navigation in indoor environments.

[0008] In one embodiment of the present invention, the dual-propeller vector propulsion system connected by a rotatable carbon fiber rod, which enables the airship to perform basic forward, backward, ascent, descent, and yaw movements in an indoor environment, further includes: S11, driving the carbon fiber rod to rotate via a combination of servo motors and gear sets, so that the orientation of the two propellers can be continuously adjusted 180° in the vertical plane; S12, controlling the forward and reverse rotation and speed of the two coreless motors respectively by outputting independent PWM signals from a Raspberry Pi, in coordination with the propeller orientation, thereby achieving dynamic adjustment of the thrust vector.

[0009] In one embodiment of the present invention, the step of calculating the real-time coordinates of the airship in three-dimensional space based on image data acquired by a binocular camera and combined with ArUco marker recognition and a four-point positioning algorithm further includes: S21, using MATLAB to calibrate the binocular camera and obtain the intrinsic and extrinsic parameters of the left and right cameras for subsequent depth map generation; S22, by identifying at least four ArUco markers with different IDs, combining the binocular visual ranging results, using a four-point positioning algorithm to calculate the three-dimensional coordinates of the airship, and using Kalman filtering to fuse and optimize the attitude angles and coordinates.

[0010] In one embodiment of the present invention, the step of transmitting real-time images from the binocular camera to a host computer for processing and using the YOLOv5 deep learning model to identify and measure the distance to dangerous points in the image further includes: S31, preprocessing the binocular camera images using OpenCV, including noise reduction, image alignment, size scaling, and normalization; S32, inputting the preprocessed images into the YOLOv5 model for frame-by-frame identification, outputting the bounding boxes, category labels, and confidence scores of dangerous points, and calculating the actual distance between them and the airship using the binocular ranging algorithm.

[0011] In one embodiment of the present invention, it further includes: S5, receiving user click input in the real-time video stream through a visual interactive interface, calling a mouse callback function to convert the click position into three-dimensional spatial coordinates, and combining the current positioning information of the airship to calculate the relative distance and direction between the target point and the airship, so as to assist manual control of the airship for accurate investigation.

[0012] To achieve the above objectives, a second aspect of the present invention proposes a lightweight airship indoor hazard detection device based on depth vision, comprising: a propulsion control module, used to realize the basic movements of the airship such as forward, backward, ascent, descent, and yaw in an indoor environment through a dual-propeller vector propulsion system connected by a rotatable carbon fiber rod, wherein the airship's weight is balanced by helium gasbags to reduce the coupling between motor thrust and servo angle; and a three-dimensional positioning fusion module, used to calculate the real-time coordinates of the airship in three-dimensional space based on image data acquired by binocular cameras, combined with ArUco marker recognition and four-point positioning algorithms, and to acquire the airship's attitude angles through a nine-axis inertial sensor, fusing the attitude angle information with binocular visual ranging results to improve positioning accuracy; The hazard identification and ranging module transmits real-time images from the binocular camera to the host computer for processing. It uses the YOLOv5 deep learning model to train, identify, and measure the hazard points in the images. The identification results are combined with the positioning data to generate the spatial location information of the hazard points and feed it back to the Raspberry Pi. The real-time communication feedback module is used to build a data transmission closed loop through the Socket communication protocol between the Raspberry Pi and the host computer, so as to realize the real-time feedback of image recognition and positioning information within 1 second, supporting the airship to conduct dynamic hazard point investigation and autonomous navigation in indoor environments.

[0013] In one embodiment of the present invention, the propulsion control module is further configured to: drive the carbon fiber rod to rotate by a combination of servo motor and gear set, so that the orientation of the two propellers can be continuously adjusted by 180° in the horizontal plane; and control the forward and reverse rotation and speed of the two coreless motors by outputting independent PWM signals from the Raspberry Pi, thereby realizing dynamic adjustment of the thrust vector.

[0014] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0016] Figure 1 This is a flowchart of a method for identifying indoor hazards in a lightweight airship based on depth vision, according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the principle of achieving basic airship motion using a vector propulsion method according to an embodiment of the present invention.

[0018] Figure 3 This is an assembly drawing of an airship pod according to an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of a PID automatic control algorithm based on vector propulsion according to an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of a low-latency data transmission loop between an airship and a host computer according to an embodiment of the present invention;

[0021] Figure 6 This is a flowchart of an airship autonomous positioning algorithm according to an embodiment of the present invention;

[0022] Figure 7 This is a flowchart of the visual interactive interface processing according to an embodiment of the present invention;

[0023] Figure 8 This is the overall flow of a hazard point identification and location algorithm according to an embodiment of the present invention;

[0024] Figure 9 This is a schematic diagram of the structure of the lightweight airship indoor hazard detection device based on depth vision provided in an embodiment of the present invention. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0026] Figure 1 This is a flowchart of a method for identifying indoor hazards in a lightweight airship based on depth vision, according to an embodiment of the present invention.

[0027] Reference Appendix Figure 1-8 The method for identifying indoor hazards in lightweight airships based on depth vision includes the following steps:

[0028] S1, through a dual-propeller vector propulsion system connected by a rotatable carbon fiber rod, enables the airship to perform basic movements such as forward, backward, ascent, descent, and yaw in an indoor environment. The airship's own weight is balanced by lift provided by a helium gasbag, thereby reducing the coupling between motor thrust and servo angle.

[0029] Specifically, the invention utilizes a dual-propeller vector propulsion system connected by rotatable carbon fiber rods to achieve basic movements of the airship, such as forward, backward, ascent, descent, and yaw, in an indoor environment. A schematic diagram illustrating the principle of vector propulsion for achieving these basic airship movements is shown below. Figure 2 As shown, the airship pod assembly diagram is as follows: Figure 3 As shown, the schematic diagram of the PID automatic control algorithm based on vector propulsion is as follows. Figure 4 As shown, the system uses two 8520 coreless motors with a rated speed of 56,000 r / min and a maximum thrust of 0.4 N per motor. The motors are independently controlled by a Raspberry Pi Zero 2 W via PWM signals for forward / reverse rotation and speed adjustment, thus achieving dynamic thrust vector control. The propeller is mounted at both ends of a long, straight carbon fiber rod, which rotates continuously within a 180° range via servos and gear sets, allowing the propeller's thrust direction to vary at any angle in the vertical plane. In some implementations, the servo uses the TS90 model, with its output shaft connected to the carbon fiber rod via reduction gears to increase the servo's output torque. The servo control signal is also output as a PWM signal via the GPIO port of the Raspberry Pi.

[0030] The airship's weight is balanced by helium gasbags, allowing it to hover stably without power, thus significantly reducing the coupling between motor thrust and servo angles. This design eliminates the need for additional vertical thrust during attitude adjustments, improving the decoupling performance and response efficiency of the control system. In practical applications, this vector propulsion system is suitable for inspection tasks in complex indoor spaces, such as stadiums, airports, and train stations, achieving low-noise, high-precision, and highly maneuverable flight control. This step, as one of the core hardware innovations of this invention, provides a stable and flexible motion platform foundation for subsequent PID automatic control, binocular vision positioning, and hazard identification, demonstrating significant engineering practical value and technological advancement.

[0031] Furthermore, the airship adopts a soft-hull design, with the gasbag portion using a non-customized aluminum foil balloon, streamlined in shape, measuring approximately 1680mm*460mm*460mm, and having a net payload of approximately 120g. The gondola is attached directly below the gasbag's buoyancy center. A forward-facing, downward-sloping binocular camera is fixed at the lower front of the gasbag, providing a complete and unobstructed view, and is connected to a Raspberry Pi via a data cable. Three lightweight foam tail fins distributed in a 120-degree circle are mounted at the tail of the gasbag to enhance pitch and yaw stability.

[0032] Furthermore, S1 includes:

[0033] The S11 uses a combination of servo motors and gear sets to drive the carbon fiber rod to rotate, allowing the orientation of the two propellers to be continuously adjusted 180° in the vertical plane.

[0034] Specifically, in this invention, the carbon fiber rod is driven to rotate via a combination of a servo motor and a gear set, enabling continuous 180° adjustment of the two propellers in the vertical plane. This is the core control mechanism of its vector propulsion system. Technically, the servo motor (model TS90) is connected to the carbon fiber rod via a gear set. The gear set typically employs a single or two-stage reduction structure to increase the servo motor's output torque and reduce its rotational speed, thereby ensuring the stability and precision of the carbon fiber rod's rotation. The servo motor receives PWM control signals from a Raspberry Pi Zero 2 W, with a control frequency typically set to 50Hz. The servo motor's rotation angle range is 0° to 180°. Precise control of the servo motor's rotation angle can be achieved by adjusting the duty cycle of the PWM signal (generally between 5% and 10%). The carbon fiber rod serves as a power linkage, with an 8520 coreless motor and a propeller fixed at each end. A single motor can generate a maximum thrust of 0.4N and has a rated speed of 56,000 r / min, offering advantages in high responsiveness and lightweight design. By driving the carbon fiber rod to rotate via a servo motor, the thrust direction of the two propellers can be continuously changed in the horizontal plane, thereby providing the airship with multi-directional vector thrust and enabling compound movements such as forward, backward, and turning.

[0035] In terms of specifications, the servo's control accuracy is ±1°, and its response time is less than 0.1s, meeting the real-time attitude adjustment requirements of the airship in complex indoor environments. The gear reduction ratio is typically set between 1:2 and 1:5 to ensure that the servo can still provide sufficient output torque under low voltage (3.8V) power supply conditions. The carbon fiber rod is approximately 260mm long and 3mm in diameter, with a total weight controlled within 5g, balancing structural strength and overall lightweight design.

[0036] In terms of application scenarios, this vector propulsion structure is particularly suitable for indoor low-altitude inspection tasks, such as hazard identification in enclosed spaces like stadiums, airports, and subway stations. Through continuous adjustment of the propeller orientation, the airship can flexibly turn, hover, or move at low speeds in confined spaces, avoiding the problems of large size, heavy weight, and complex control associated with traditional multi-propeller structures.

[0037] In terms of technical effectiveness, this design significantly improves the airship's maneuverability and control freedom while reducing the number of hardware components, thus lowering system complexity and energy consumption. Through the coordinated control of servos and gear sets, the airship can achieve multi-degree-of-freedom flight control without increasing the number of propellers, providing a reliable motion foundation for subsequent PID automatic control and depth vision positioning. This is a key innovation of this invention in terms of lightweight design and high flexibility.

[0038] S12 controls the forward and reverse rotation and speed of the two coreless motors by outputting independent PWM signals from the Raspberry Pi, thereby achieving dynamic adjustment of the thrust vector.

[0039] Specifically, in this invention, controlling the forward and reverse rotation and speed of two coreless motors separately by outputting independent PWM signals from the Raspberry Pi is the core control method for dynamically adjusting the airship's thrust vector. This step utilizes the GPIO interface of the Raspberry Pi Zero 2W, combined with a dual-channel DC motor control module (model 2.5A dual-channel forward and reverse brushed motor control module), to achieve independent control of the two 8520 coreless motors. The speed of each motor is adjusted by the duty cycle of the PWM signal, while forward and reverse rotation is achieved by switching the motor power supply direction through the H-bridge circuit of the control module.

[0040] Specifically, the Raspberry Pi generates PWM signals using programming environments such as Python (e.g., the RPi.GPIO or pigpio library), typically with a frequency set to 20kHz to avoid audible noise from the motors and improve response stability. The duty cycle ranges from 0% to 100%, corresponding to a linear change in motor speed from complete stop to maximum speed (rated 56000 r / min). In terms of control logic, the Raspberry Pi sends signals with different duty cycles to the PWM inputs of the two motors according to instructions from the flight control program, thereby enabling differential propulsion, forward / reverse switching, and other operations to control the airship's direction and speed.

[0041] This step plays a crucial role in the airship's vector propulsion system. By independently controlling the speed and direction of the two motors, combined with the rotation of the carbon fiber linkage driven by the servo motors, the airship can achieve complex movements such as forward, backward, ascent, descent, and yaw in three-dimensional space. Compared to traditional multi-propeller airships, this solution significantly reduces the number of motors, improves the system's lightweight design, and enhances the flexibility and response speed of flight control. In practical applications, this control method supports seamless switching between Bluetooth manual and PID automatic control modes. The dual-mode flight control algorithm provides a reliable power foundation for the airship's autonomous inspection and hazard identification in complex indoor environments.

[0042] S2 calculates the real-time coordinates of the airship in three-dimensional space based on image data collected by binocular cameras, combined with ArUco marker recognition and four-point positioning algorithms. It also obtains the airship's attitude angles through a nine-axis inertial sensor and fuses the attitude angle information with the binocular visual ranging results to improve positioning accuracy.

[0043] Specifically, this step calculates the airship's real-time coordinates in three-dimensional space based on image data acquired by a binocular camera, combined with ArUco marker recognition and four-point positioning algorithms. It also obtains the airship's attitude angles using a nine-axis inertial sensor and fuses this attitude angle information with binocular visual ranging results to improve positioning accuracy. This technical solution plays a crucial role in airship autonomous navigation and indoor hazard detection.

[0044] At the technical implementation level, the binocular cameras first need to be calibrated. Using MATLAB tools with a checkerboard calibration board, the intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (rotation matrix and translation vector) of the left and right cameras are obtained to construct the geometric model of the binocular vision system. Subsequently, multiple ArUco markers are placed in the airship's flight environment as reference points with known coordinates. Each marker has a unique ID number for easy identification and matching. Using the ArUco module in the OpenCV library, the system can detect ArUco markers in the image in real time and calculate their pixel coordinates. Combined with the disparity calculation method of binocular vision, the system can further calculate the three-dimensional spatial distance between the markers and the cameras.

[0045] In the four-point positioning algorithm, the system requires at least four identifiable ArUco markers within the field of view that are not coplanar. The real-time coordinates of the airship are calculated using the PnP algorithm and the least squares method. If the number of markers is insufficient or coplanarities exist, a nine-axis inertial sensor (WT9011G4K) is introduced to obtain the airship's pitch, yaw, and roll angles. These are then combined with binocular ranging results and the one-point positioning algorithm for attitude compensation, thereby achieving more accurate coordinate estimation.

[0046] In terms of specifications, the binocular camera has a baseline distance of approximately 100mm, a focal length of approximately 3.6mm, a field of view (FOV) of 90°, supports a maximum resolution of 1920×1080, and a frame rate of 30fps. The nine-axis sensor incorporates a Kalman filter, achieving an output angular velocity accuracy of ±0.1° / s and an attitude angle output frequency of 100Hz. Within a 3-meter ranging range, the coordinate error of the four-point positioning algorithm can be controlled within ±50mm, and after fusing the attitude angles, the error is further reduced to ±30mm.

[0047] In application scenarios, this step is suitable for airships to perform autonomous inspections and hazard location in complex indoor environments. By acquiring the airship's three-dimensional coordinates and attitude information in real time, the system can achieve path planning, obstacle avoidance control, and precise target point positioning, providing a high-precision spatial reference for subsequent deep learning recognition and hazard distance measurement.

[0048] The technical effect of this step is that by fusing multi-source information from binocular vision and inertial sensors, the robustness and accuracy of airship positioning in indoor environments are significantly improved. This solves the positioning drift problem of traditional monocular vision or pure inertial navigation in complex scenarios, and provides reliable technical support for achieving high-precision, low-latency autonomous airship inspection.

[0049] Furthermore, S2 includes:

[0050] S21. Use MATLAB to calibrate the stereo camera and obtain the intrinsic and extrinsic parameters of the left and right cameras for subsequent depth map generation.

[0051] Specifically, in this invention, calibrating the binocular cameras using MATLAB is a crucial step in achieving depth map generation and autonomous spatial localization. This step, by accurately acquiring the intrinsic parameters (including focal length, principal point coordinates, distortion coefficients, etc.) and extrinsic parameters (such as rotation matrix R and translation vector T) of the left and right cameras, provides the basic parameters for geometric correction and coordinate transformation for subsequent binocular vision-based ranging and localization algorithms.

[0052] In the specific implementation, the calibration process uses a standard black and white checkerboard calibration board, with a size of 8×6 squares and each square having a side length of 30mm. By placing the calibration board at different angles and distances, image sequences from the left and right cameras are acquired respectively. Each set of images needs to cover a different field of view to ensure calibration accuracy. In MATLAB, the Stereo CameraCalibrator toolbox is used to perform corner detection and matching on the images, automatically calculating the intrinsic parameter matrix (3×3) and distortion coefficients (5-parameter model, including radial distortion k1, k2, k3 and tangential distortion p1, p2) for each camera. The extrinsic parameters are calculated based on the position and orientation of the calibration board in space, determining the relative rotation and translation relationship between the left and right cameras, and finally outputting the fundamental matrix and the essential matrix.

[0053] After calibration, the parameters of the left and right cameras will be used in subsequent stereo matching algorithms, such as disparity map calculation based on block matching (BM) or semi-global block matching (SGBM). Using the known baseline distance (B=100mm) and focal length (f=3.6mm), combined with the disparity value d, the depth value of the target point Z=f·B / d can be calculated. This step ensures that the error of the binocular vision system within a 3-meter ranging range does not exceed 50mm, meeting the accuracy requirements for indoor hazard point detection. Simultaneously, by combining the attitude angle information provided by the inertial sensor, the deviation between the camera coordinate system and the airship's body coordinate system can be further corrected, improving the robustness and accuracy of the overall positioning system.

[0054] S22 uses a four-point positioning algorithm to calculate the three-dimensional coordinates of the airship by identifying at least four different IDs of ArUco markers and combining the results of binocular visual ranging. It then uses Kalman filtering to fuse and optimize the attitude angles and coordinates.

[0055] Specifically, this step involves identifying at least four ArUco tags with different IDs, combining the results of binocular visual ranging, using a four-point positioning algorithm to calculate the three-dimensional coordinates of the airship, and then using Kalman filtering to fuse and optimize the attitude angles and coordinates. This is the core technology for enabling the airship to achieve autonomous positioning and navigation in an indoor environment.

[0056] At the technical implementation level, firstly, the binocular cameras mounted on the airship need to be calibrated using a checkerboard calibration board. The intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (rotation matrix, translation vector) of the left and right cameras are obtained using MATLAB's Stereo Camera Calibrator tool to construct a 3D reconstruction model of the binocular vision system. Subsequently, the system acquires binocular images in real time and uses the ArUco module in the OpenCV library to detect and identify markers in the images. Each ArUco marker has a unique ID number, facilitating the differentiation of different reference points. After identifying at least four markers with different IDs in the field of view, the system uses the disparity calculation method of binocular vision to obtain the 3D distance information between the airship and each marker.

[0057] Furthermore, based on the four-point positioning algorithm, the system uses four ArUco markers with known spatial coordinates as reference points and employs the PnP algorithm. When necessary (e.g., when there are more than four markers within the field of view), it combines the least squares method or nonlinear optimization methods (such as the Levenberg-Marquardt algorithm) to solve for the airship's three-dimensional coordinates. This algorithm requires that the four markers be non-coplanar in space and reasonably distributed to ensure positioning accuracy. In practical applications, the airship needs to deploy multiple ArUco markers in an indoor environment to ensure that it can always identify at least four markers during flight, thus achieving continuous positioning. If the number of markers is less than four or there are coplanar markers, the airship attitude angles obtained by the IMU module are introduced, and attitude compensation is performed by combining the binocular ranging results and the one-point positioning algorithm to achieve more accurate coordinate estimation.

[0058] To improve positioning stability, the system incorporates a Kalman filter to fuse and optimize the airship's attitude angles (provided by the nine-axis inertial sensor WT9011G4K) with its three-dimensional coordinates. The Kalman filter, through a prediction-update mechanism, combines visual positioning measurements with inertial sensor state estimates to reduce noise interference and improve positioning accuracy. Within a 3-meter ranging range, the system's positioning error can be controlled within ±50mm, and after fusing the attitude angles, the error is further reduced to ±30mm, meeting the accuracy requirements for indoor inspections.

[0059] This step plays a crucial role in the overall system, providing high-precision spatial location information for the airship's automatic navigation, obstacle avoidance, and hazard identification, and is an important technical support for realizing intelligent and unmanned inspection.

[0060] S3 transmits real-time images from the binocular camera to the host computer for processing. The YOLOv5 deep learning model is used to identify and measure the distance to dangerous points in the images. The identification results are combined with the positioning data to generate the spatial location information of the dangerous points and feed it back to the airship control system.

[0061] Specifically, in some implementations, this invention transmits real-time images from a binocular camera to a host computer for processing, and combines this with a YOLOv5 deep learning model to identify and measure the distance to hazard points in the images. Finally, the identification results are fused with positioning data to generate spatial location information of the hazard points, which is then fed back to the airship control system, thereby achieving intelligent hazard detection in the indoor environment. This step is the core link in enabling the entire system to achieve autonomous perception and intelligent decision-making.

[0062] At the technical implementation level, the binocular cameras are connected to a Raspberry Pi via a Micro USB interface. The Raspberry Pi runs an OpenCV image acquisition program, acquiring images from the left and right cameras at a frequency of 30 frames per second, and publishes the video stream to a specific URL within the local area network using the Flask framework. The host computer (Windows PC) accesses this URL to obtain the real-time image stream and uses the YOLOv5 model for target detection. In this invention, the YOLOv5 model is customized for specific hazardous points (such as five-hole sockets, flammable materials, etc.). The training dataset contains approximately 2000 labeled images at different angles and distances. The input image size is 640×640 pixels, and the model output is the coordinates of the detection box, the class label, and the confidence score. Typically, the confidence score can reach above 0.9, meeting the requirements for high-precision recognition.

[0063] In terms of specifications, the binocular camera has a baseline distance of 100mm, a focal length of 3.6mm, supports a maximum resolution of 1920×1080, an image synchronization error of less than 1ms, a ranging range of 0.5m to 3m, and a ranging error within ±50mm. The YOLOv5 model runs on an NVIDIA Jetson or equivalent GPU on the host computer, with inference latency controlled within 50ms to ensure real-time performance. After combining the recognition results with the airship's attitude angles (provided by the WT9011G4K nine-axis sensor), the three-dimensional position of the hazard point in the world coordinate system is calculated through coordinate transformation algorithms (such as perspective transformation and 3D reconstruction).

[0064] In application scenarios, this step is suitable for remote inspection tasks in complex indoor environments, such as stadiums, airports, subway stations and other densely populated places. The airship can identify and locate potential danger points in real time, such as exposed wires, piles of flammable materials, equipment malfunctions, etc., through autonomous flight or manual remote control, providing security personnel with visualized and operable early warning information.

[0065] The technical effect of this step is that by integrating binocular vision and deep learning, high-precision identification and three-dimensional positioning of dangerous points are achieved, providing key spatial information input for the airship control system. This supports subsequent advanced functions such as path planning, obstacle avoidance, and automatic alarm, significantly improving the intelligence level and response efficiency of the airship in indoor inspections.

[0066] Furthermore, S3 includes:

[0067] S31 uses OpenCV to preprocess images from binocular cameras, including noise reduction, image alignment (stereo correction), size scaling, and normalization.

[0068] Specifically, in this invention, the step of "preprocessing the binocular camera images using OpenCV, including necessary denoising, image alignment (stereo correction), size scaling, and normalization" is a key image processing step for achieving autonomous spatial positioning and hazard identification of the airship. The technical implementation of this step is based on image preprocessing workflows in computer vision, aiming to improve the accuracy and stability of subsequent depth map generation and target recognition.

[0069] At the technical implementation level, firstly, the raw images captured by the binocular cameras typically contain noise, affecting subsequent feature extraction and matching. To suppress noise, Gaussian filtering (Gaussian Blur, 5×5 kernel, standard deviation σ=1.5) in OpenCV is used to primarily eliminate Gaussian noise, or median filtering is used to address salt-and-pepper noise. Secondly, image alignment (Rectification) is performed. The rotation matrix R and projection matrix P obtained from the binocular camera calibration are used to correct the left and right images, aligning the epipolar lines of the two images horizontally, thus facilitating disparity calculation. The calibration board used during alignment is a black and white checkerboard with a size of 8×6, and the calibration error is controlled within the sub-pixel range (approximately 0.1–0.3 pixels) to ensure correction accuracy. Finally, the image size is scaled to the YOLOv5 input standard (e.g., 640×640 pixels) and normalized to map pixel values ​​to the [0,1] interval to meet the model input requirements.

[0070] In terms of parameters, the image preprocessing frequency is 30 frames / second, and the disparity map calculation accuracy is sub-pixel level. The image alignment process uses MATLAB to calculate calibration parameters, which are then imported into OpenCV for real-time processing. The calibration parameters include camera intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (rotation matrix, translation vector).

[0071] In this application scenario, this step runs on a Raspberry Pi Zero 2 W mounted on the airship. A local area network video streaming service built with Flask and Gevent transmits the pre-processed images in real time to a host computer for depth map generation and target recognition. During indoor inspections, the pre-processed images significantly improve the robustness of ArUco marker recognition and the accuracy of the YOLOv5 model in detecting hazard points.

[0072] The technical effect of this step is that, through a systematic image preprocessing process, it effectively improves the ranging accuracy and stability of the binocular vision system, providing high-quality input data for subsequent autonomous positioning and hazard point identification. It is an indispensable preprocessing step for realizing intelligent airship inspection.

[0073] S32 inputs the preprocessed image into the YOLOv5 model for frame-by-frame recognition, outputs the bounding box, category label and confidence score of the danger point, and calculates the actual distance between it and the airship using a binocular ranging algorithm.

[0074] Specifically, in this invention, the step of "inputting the preprocessed image into the YOLOv5 model for frame-by-frame recognition, outputting the bounding box, category label, and confidence score of the hazard point, and calculating the actual distance between it and the airship using a binocular ranging algorithm" is the core step in achieving intelligent identification and precise positioning of indoor hazard points. This step integrates deep learning target detection and binocular visual ranging technology, providing crucial support for the airship's autonomous inspection in complex indoor environments.

[0075] At the technical implementation level, the YOLOv5 model first preprocesses the images captured by the binocular cameras, including image denoising, alignment correction, normalization, and resizing to the YOLOv5 input standard (e.g., 640×640 pixels). The preprocessed images are then input into the YOLOv5 model for real-time frame-by-frame detection. YOLOv5 employs a single-stage detection architecture, extracting image features through a backbone network (CSPDarknet53), and then performing multi-scale feature fusion through PANet (Path Aggregation Network). Finally, the detection head outputs the target's bounding box, class label, and confidence score. In this invention, the YOLOv5 model undergoes customized training for indoor hazardous points (such as five-hole sockets, exposed wires, etc.), using approximately 2000 labeled images for training. The model achieves an average confidence score of over 0.9 on the test set, meeting the requirements for high-precision recognition.

[0076] Furthermore, the identification results are combined with a binocular ranging algorithm to achieve three-dimensional spatial positioning between the hazard point and the airship. The binocular camera obtains intrinsic and extrinsic parameters through calibration (using MATLAB for checkerboard calibration), and a depth map is generated using a parallax calculation method. The pixel coordinates of the identified hazard point in the image are converted into three-dimensional spatial coordinates using camera parameters, and then aligned with the airship's attitude angle (provided by a nine-axis sensor) to finally calculate the actual distance between the hazard point and the airship. This ranging algorithm has a ranging error of less than 50mm within a 3-meter range, meeting the accuracy requirements for indoor inspections.

[0077] In practical applications, this step enables airships to identify and locate potential hazards in indoor environments such as airports, stadiums, and industrial plants through real-time video streams, providing intuitive and accurate visual feedback to remote operators. Its technological value lies in achieving low-latency, high-precision hazard identification and ranging closed-loop systems, filling a technological gap in intelligent visual inspection of existing airships.

[0078] S4 establishes a closed-loop data transmission mechanism through the Socket communication protocol between the Raspberry Pi and the host computer, enabling real-time feedback of image recognition and positioning information within 1 second, and supporting the airship to conduct dynamic hazard point investigation and autonomous navigation in indoor environments.

[0079] Specifically, the data transmission closed loop constructed through the Socket communication protocol between the Raspberry Pi and the host computer in this invention is a key technical step in achieving real-time feedback of image recognition and positioning information within one second. In some implementations, the Raspberry Pi acts as the embedded control terminal of the airship, responsible for collecting real-time image data from the binocular cameras and attitude angle information from the inertial sensors, and uploading the image stream to a web server on the local area network via a Wi-Fi module. Specifically, the image data is acquired and preprocessed using OpenCV, then encapsulated into a video stream using the Flask framework and transmitted with low latency via Gevent's WSGI server, enabling the host computer (Windows computer) to obtain real-time video footage by accessing a specific URL.

[0080] Meanwhile, a TCP / IP-based socket communication channel is established between the Raspberry Pi and the host computer. The Raspberry Pi acts as the server, listening on a specified port (e.g., 5000), while the host computer, as the client, establishes a connection by inputting the Raspberry Pi's IP address and port number. During communication, the Raspberry Pi sends attitude angle information (including heading, pitch, and roll angles) in JSON format to the host computer via the socket. After processing the YOLOv5 image recognition and four-point positioning algorithm, the host computer returns the recognition results and the airship's coordinates to the Raspberry Pi in the same manner for subsequent flight control decisions.

[0081] This communication mechanism boasts significant advantages in technical specifications: image transmission latency is controlled within 500ms, attitude angle data transmission latency is less than 50ms, and the overall closed-loop response time is no more than 1 second, meeting the real-time requirements for indoor dynamic hazard point investigation. In application scenarios, this closed-loop system supports autonomous navigation and hazard identification of airships in complex indoor environments, making it particularly suitable for unmanned inspection tasks in densely populated areas such as stadiums, airports, and train stations. Through this Socket communication protocol, the system achieves efficient data interaction between the airship and the host computer, providing reliable data support for subsequent positioning and control algorithms, significantly improving the system's real-time performance and stability. This is a key innovation of this invention in terms of hardware and software collaborative control.

[0082] The method for detecting indoor hazards in lightweight airships based on depth vision in this invention enables lightweight airships to autonomously locate themselves and identify and measure hazards in indoor environments, effectively improving the flexibility, endurance, and accuracy of unmanned inspections.

[0083] In addition, embodiments of the present invention also receive user click input in the real-time video stream through a visual interactive interface, call the mouse callback function, convert the click position into three-dimensional spatial coordinates, and combine it with the airship's current positioning information to calculate the relative distance and direction between the target point and the airship, which is used to assist manual control of the airship for accurate investigation.

[0084] Specifically, in some implementations, a visual interactive interface receives user click input from the real-time video stream and calls a mouse callback function to convert the click location into three-dimensional spatial coordinates. Combined with the airship's current positioning information, the relative distance and direction between the target point and the airship are calculated. This is the core interactive mechanism in this invention for enabling remotely controlled airships for precise inspections. The technical implementation of this step is based on the deep integration of a binocular vision system and an airship autonomous positioning algorithm, featuring high precision, low latency, and strong interactivity.

[0085] At the technical implementation level, the host computer (Windows PC) receives a real-time video stream from a Raspberry Pi via a local area network. The video stream is captured by a binocular camera and transmitted through a lightweight web server built with Flask and Gevent. When the user clicks on a target point in the video window on the host computer, the system calls the OpenCV mouse callback function to obtain the two-dimensional coordinates (u, v) of the click in the pixel coordinate system. Subsequently, the system uses the calibration parameters of the binocular vision (including camera intrinsics, extrinsics, and fundamental matrices) to convert these coordinates into a three-dimensional point (X, Y, Z) in the depth image, where the Z value represents the distance between the target point and the camera. Furthermore, combining the three-dimensional coordinates obtained by the airship using a four-point localization algorithm and an IMU attitude angle-assisted one-point localization algorithm, the system can calculate the relative distance and direction vector of the target point relative to the airship, and convert them into heading and pitch angles to guide the airship's movement direction.

[0086] At the parameter level, the calibration accuracy of the binocular camera directly affects the accuracy of 3D coordinate transformation. This invention uses MATLAB for calibration, with calibration errors controlled at the sub-pixel level (approximately 0.1–0.3 pixels), and depth map calculation errors not exceeding 50mm within a 3-meter range. The mouse callback function response time is controlled within 10ms to ensure real-time interaction. The airship positioning algorithm updates at a frequency of 30Hz, matching the video stream's frame rate (30fps), thus achieving synchronous processing.

[0087] At the application level, this step is widely used in indoor hazard point investigation tasks. For example, in enclosed spaces such as airports, train stations, and stadiums, operators can click on suspicious areas through remote video screens, and the system can quickly calculate the spatial location of the target point. Combined with the airship's motion control module, it can guide the airship to approach the target point, achieving accurate inspection and positioning.

[0088] In terms of technical effects, this step significantly improves the precision and response efficiency of manual operation of airships, enabling airships to have the ability to "control what you see". It effectively makes up for the problem of insufficient positioning and ranging capabilities of traditional airships in complex indoor environments, and provides a reliable human-computer interaction foundation for achieving high-precision, low-latency remote troubleshooting.

[0089] The lightweight airship indoor hazard point investigation method based on depth vision in this invention, based on the realization of autonomous positioning of the lightweight airship and hazard point identification and ranging, receives user click input in real time through a visual interactive interface and converts it into three-dimensional coordinates. Combined with the airship positioning information, it calculates the relative distance and direction, which significantly improves the accuracy and response efficiency of manual operation and further enhances the interactivity and operational flexibility of inspection tasks in complex indoor environments.

[0090] To achieve the above embodiments, the present invention also proposes a lightweight airship indoor hazard detection device based on depth vision. Figure 9 This is a schematic diagram of a lightweight airship indoor hazard detection device based on depth vision, provided as an embodiment of the present invention. Figure 9 As shown, the device includes:

[0091] The propulsion control module 100 is used to realize the basic movements of the airship such as forward, backward, ascent, descent and yaw in an indoor environment through a dual propeller vector propulsion system connected by a rotatable carbon fiber rod. The airship's own weight is balanced by the lift provided by the helium gasbag to reduce the coupling between the motor thrust and the servo angle.

[0092] The 3D positioning fusion module 200 is used to calculate the real-time coordinates of the airship in 3D space based on the image data collected by the binocular camera, combined with ArUco marker recognition and four-point positioning algorithm, and to obtain the airship attitude angle through the nine-axis inertial sensor. The attitude angle information is fused with the binocular visual ranging results to improve positioning accuracy.

[0093] The hazard point identification and ranging module 300 is used to transmit real-time images from the binocular camera to the host computer for processing. It uses the YOLOv5 deep learning model to train, identify, and measure the hazard points in the image. The identification results are combined with the positioning data to generate the spatial location information of the hazard points and feed it back to the Raspberry Pi.

[0094] The real-time communication feedback module 400 is used to build a data transmission closed loop through the Socket communication protocol between the Raspberry Pi and the host computer, so as to realize the real-time feedback of image recognition and positioning information within 1 second, and support the airship to conduct dynamic hazard point investigation and autonomous navigation in indoor environments.

[0095] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0096] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0097] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0098] This invention is intended to provide implementation schemes for users to selectively prevent the use or access to personal information data. That is, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0099] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0100] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0101] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0102] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0103] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for identifying indoor hazards in lightweight airships based on depth vision, characterized in that, include: S1, through a dual-propeller vector propulsion system connected by a rotatable carbon fiber rod, enables the airship to perform basic forward, backward, ascending, and yaw movements in an indoor environment. The airship's own weight is balanced by lift provided by a helium gasbag, thereby reducing the coupling between motor thrust and servo angle. S2, based on image data collected by binocular cameras, combined with ArUco marker recognition and four-point positioning algorithms, calculates the real-time coordinates of the airship in three-dimensional space, and obtains the airship's attitude angle through a nine-axis inertial sensor. The attitude angle information is then fused with the binocular visual ranging results to improve positioning accuracy. S3 transmits real-time images from the binocular camera to the host computer for processing. It uses the YOLOv5 deep learning model to train, identify, and measure the distance to dangerous points in the images. The identification results are combined with the positioning data to generate the spatial location information of the dangerous points and feed it back to the Raspberry Pi. S4 constructs a data transmission closed loop through the Socket communication protocol between Raspberry Pi and the host computer, enabling real-time feedback of image recognition and positioning information within 1 second, and supporting the airship to conduct dynamic hazard point investigation and autonomous navigation in indoor environments; The calculation of the airship's real-time coordinates in three-dimensional space based on image data acquired by the binocular camera, combined with ArUco marker recognition and four-point positioning algorithms, also includes: The binocular camera was calibrated using MATLAB. The intrinsic and extrinsic parameters of the left and right cameras were obtained using a black and white checkerboard calibration board. The intrinsic parameters included focal length, principal point coordinates, and distortion coefficients, while the extrinsic parameters included rotation matrix and translation vector. Multiple ArUco markers are deployed in the airship's flight environment as reference points with known coordinates, and each marker has a unique ID number; Using the ArUco module in the OpenCV library, at least four ArUco tags with different IDs are identified. Combined with the binocular vision ranging results, a four-point positioning algorithm is used to calculate the three-dimensional coordinates of the airship. Kalman filtering is then used to fuse and optimize the attitude angles and coordinates. If the number of tags is insufficient or there are coplanar cases, a nine-axis inertial sensor is introduced to obtain the pitch, yaw, and roll angles of the airship. Attitude compensation is then performed by combining the binocular ranging results with the one-point positioning algorithm.

2. The method as described in claim 1, characterized in that, The dual-propeller vector propulsion system, connected by a rotatable carbon fiber rod, enables the airship to perform basic forward, backward, ascending, and yaw movements in an indoor environment. It also includes: S11 drives the carbon fiber rod to rotate through a combination of servo motor and gear set, so that the orientation of the two propellers can be continuously adjusted 180° in the vertical plane; S12 controls the forward and reverse rotation and speed of the two coreless motors by outputting independent PWM signals from the Raspberry Pi, which are coordinated with the propeller orientation to achieve dynamic adjustment of the thrust vector.

3. The method as described in claim 1, characterized in that, The process of transmitting real-time images from the binocular camera to a host computer for processing, and using the YOLOv5 deep learning model to identify and measure the distance to dangerous points in the images, also includes: S31 uses OpenCV to preprocess images from the binocular camera, including noise reduction, image alignment, scaling and normalization. S32 inputs the preprocessed image into the YOLOv5 model for frame-by-frame recognition, outputs the bounding box, category label and confidence score of the danger point, and calculates the actual distance between it and the airship using a binocular ranging algorithm.

4. The method as described in claim 1, characterized in that, Also includes: The system receives user clicks in the real-time video stream through a visual interactive interface, calls a mouse callback function to convert the click location into three-dimensional spatial coordinates, and combines this with the airship's current positioning information to calculate the relative distance and direction between the target point and the airship, which is used to assist manual control of the airship for accurate inspection.

5. A lightweight indoor hazard detection device for airships based on depth vision, characterized in that, include: The propulsion control module is used to realize the basic forward, backward, ascent, descent, and yaw movements of the airship in an indoor environment through a dual-propeller vector propulsion system connected by a rotatable carbon fiber rod. The airship's own weight is balanced by lift provided by a helium gasbag to reduce the coupling between motor thrust and servo angle. The 3D positioning fusion module is used to calculate the real-time coordinates of the airship in 3D space based on the image data collected by the binocular camera, combined with ArUco marker recognition and four-point positioning algorithm. It also obtains the airship's attitude angle through a nine-axis inertial sensor and fuses the attitude angle information with the binocular visual ranging results to improve positioning accuracy. The hazard point identification and ranging module is used to transmit real-time images from the binocular camera to the host computer for processing. It uses the YOLOv5 deep learning model to train, identify, and measure the hazard points in the image. The identification results are combined with the positioning data to generate the spatial location information of the hazard points and feed it back to the Raspberry Pi. The real-time communication feedback module is used to build a data transmission closed loop through the Socket communication protocol between the Raspberry Pi and the host computer, so as to realize the real-time feedback of image recognition and positioning information within 1 second, and support the airship to conduct dynamic hazard point investigation and autonomous navigation in indoor environments. The calculation of the airship's real-time coordinates in three-dimensional space based on image data acquired by the binocular camera, combined with ArUco marker recognition and four-point positioning algorithms, also includes: The binocular camera was calibrated using MATLAB. The intrinsic and extrinsic parameters of the left and right cameras were obtained using a black and white checkerboard calibration board. The intrinsic parameters included focal length, principal point coordinates, and distortion coefficients, while the extrinsic parameters included rotation matrix and translation vector. Multiple ArUco markers are deployed in the airship's flight environment as reference points with known coordinates, and each marker has a unique ID number; Using the ArUco module in the OpenCV library, at least four ArUco tags with different IDs are identified. Combined with the binocular vision ranging results, a four-point positioning algorithm is used to calculate the three-dimensional coordinates of the airship. Kalman filtering is then used to fuse and optimize the attitude angles and coordinates. If the number of tags is insufficient or there are coplanar cases, a nine-axis inertial sensor is introduced to obtain the pitch, yaw, and roll angles of the airship. Attitude compensation is then performed by combining the binocular ranging results with the one-point positioning algorithm.

6. The apparatus as claimed in claim 5, characterized in that, The propulsion control module is also used for: The carbon fiber rod is driven to rotate by a combination of servo motor and gear set, so that the orientation of the two propellers can be continuously adjusted 180° in the vertical plane. By outputting independent PWM signals from the Raspberry Pi, the forward and reverse rotation and speed of the two coreless motors are controlled respectively, which are coordinated with the propeller orientation to achieve dynamic adjustment of the thrust vector.

Images