ROS-based unmanned aerial vehicle license plate recognition simulation platform and construction method thereof

Abstract

The invention discloses an unmanned aerial vehicle license plate recognition simulation platform based on ROS and a construction method of the unmanned aerial vehicle license plate recognition simulation platform. The method comprises the steps of 1, building a scene, an unmanned aerial vehicle model, a sensor model and flight control; step 2, releasing the sensor data of the virtual scene through ROS; 3, a PX4 flight control instruction is converted into physical motion of the unmanned aerial vehicle model through a PX4-Gazebo simulation system, flight state data is obtained, and deployment and verification of a flight control algorithm are achieved; 4, completing the development and integration of a license plate recognition function, and obtaining recognition result data; and 5, realizing the display of the multi-dimensional data. According to the invention, the flight state and the recognition effect of the unmanned aerial vehicle during execution of the license plate recognition task can be simulated efficiently.

Description

Technical Field

[0001] This invention relates to the field of drone simulation technology, specifically to a ROS-based drone license plate recognition simulation platform and its construction method. Background Technology

[0002] Most existing drone license plate recognition systems rely on real-world testing, which is not only costly but also unable to simulate diverse flight and recognition scenarios under different environmental conditions.

[0003] The specific defects and shortcomings in the existing technology are as follows: 1. Environmental limitations: Many simulation platforms cannot provide sufficient environmental interaction and realistic external disturbance conditions (such as wind speed, air pressure, etc.), which limits the accuracy of simulation tests; 2. Insufficient sensor simulation: The sensor simulation function of the existing platform is relatively simple and cannot realistically reproduce sensor noise, faults and environmental effects; 3. The flight control algorithms are limited; many simulation platforms only support basic flight control algorithms and lack the ability to simulate complex flight modes and abnormal situations. 4. Lack of visualization capabilities: The existing platform's visualization capabilities are relatively basic and cannot provide users with comprehensive and intuitive real-time flight status and sensor feedback.

[0004] Therefore, there is an urgent need for an efficient platform that can realistically and comprehensively simulate the flight process, sensor data, and environmental interaction of drones. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, this invention provides a ROS-based drone license plate recognition simulation platform and its construction method. The platform combines virtual environment creation, flight control, sensor simulation and license plate recognition technologies, and can efficiently simulate the flight status and recognition effect of drones when performing license plate recognition tasks.

[0006] A ROS-based drone license plate recognition simulation platform, with ROS as the core communication framework, integrates a virtual environment module, a sensor simulation module, a flight control module, a Socket protocol and data transmission module, a license plate recognition algorithm module, and a data visualization module; The virtual environment module constructs scalable drone flight scenarios and drone physical simulation models based on the simulation requirements of different drone types and scenarios, and provides dynamic environmental interference. This module is based on the Gazebo simulation engine and supports the customization of scenario types, environmental interference simulation, drone models and physical parameters. The sensor simulation module simulates various types of sensors carried by the UAV and outputs high-fidelity sensor data, including visual camera sensors, IMU sensors, LiDAR sensors, RTK sensors, and remote controllers.

[0007] Among them, the visual camera sensor is mounted on the front of the drone simulation model in the virtual environment to photograph ground vehicles and output RGB images (topic: / camera / image_raw); The IMU sensor is positioned at the center of gravity of the drone's fuselage to collect real-time attitude and speed data of the drone and output IMU data via the domain name / imu / data. The lidar sensor is located at the front of the drone's nose, scanning obstacles and ground terrain in front of it, and outputting point cloud data. The RTK sensor is located at the top center of the drone's back. It receives virtual satellite signals, provides a high-precision position reference for flight control, and outputs the positioning topic / rtk / fix. The remote controller establishes communication with the flight control module via ROS, simulating the remote control of the UAV by the ground station, and outputs control commands via the topic / remote / control. The five sensors form a collaborative system of "control-perception-execution": the data output from the IMU sensor and RTK sensor are fused to provide attitude and position data for flight; the lidar sensor scans obstacles ahead in real time to ensure flight safety; the vision camera performs license plate image acquisition tasks; and the remote controller serves as an emergency control measure to take over the drone when the flight control algorithm fails.

[0008] The flight control module adjusts the UAV's flight attitude, speed, and heading parameters based on sensor data from the sensor simulation module. The control algorithms used include different types of flight control algorithms such as PID control, LQR control, and robust control, supporting complex flight modes and fault simulation. The Socket protocol and data transmission module can ensure real-time data synchronization between the flight control module, sensor simulation module, license plate recognition algorithm module and data visualization module, and realize low-latency communication between the modules. The license plate recognition algorithm module uses the YOLOv8 neural network to detect license plates, recognize characters, and output results from license plate images captured by the sensor simulation module in a virtual scene. The data visualization module displays the drone's flight status, sensor data, and license plate recognition results in real time.

[0009] A method for constructing a ROS-based drone license plate recognition simulation platform includes the following steps; Step 1: Build the virtual scene: The simulation platform is built by constructing scenarios, drone models, sensor models, and flight control systems, which serve as the physical foundation for the entire simulation platform.

[0010] Step 2: Sensor simulation module configuration: Based on the sensor model built in step 1, the sensor data of the virtual scene is published through ROS; Step 3: Flight Control Module Configuration: Based on the integration of the UAV model constructed in step 1 with PX4 flight control, the PX4-Gazebo simulation system is used to convert the flight control commands of PX4 into the physical motion of the UAV model, obtain flight status data, and realize the deployment and verification of flight control algorithms. Step 4: License Plate Recognition Algorithm Implementation: Based on the visual camera sensor data in step 2, complete the development and integration of the license plate recognition function and obtain the recognition result data; Step 5: Data visualization module design: Based on the sensor data from step 2, the flight status data from step 3, and the identification result data from step 4, a multi-dimensional data summary and display is achieved.

[0011] Step 1 specifically involves: Step 1-1: Scene Setup: Construct a multi-scene simulation environment based on the Gazebo simulation engine. Input the scene road and car model, and bind the license plate texture to the specified position of the car model through the Gazebo material editing function; simulate the light intensity and light direction at different time periods through the Gazebo lighting plugin to accurately evaluate the performance and adaptability of the sensor under different environmental conditions. Steps 1-2: Drone Model Building: Establish a drone model library containing URDF files for different drone models, including quadcopters and hexacopter drones. Define core parameters such as fuselage mass and wheelbase to enable expansion and customization as needed. Then import the model files into the ROS package to meet the simulation requirements for different drone types. Steps 1-3: Sensor Model Setup: Mount sensor models to the drone model and connect the sensors to the Gazebo physics engine using the gazebo_ros plugin to ensure that the sensors move synchronously with the drone. Steps 1-4: Flight Control Setup: Integrate the PX4 firmware into the ROS workspace, establish communication between ROS and PX4 through the mavros package, and ensure that subsequent flight control commands can be transmitted to the drone model.

[0012] Step 2 specifically involves: The sensor models include visual camera sensors, IMU sensors, LiDAR sensors, RTK sensors, and remote controllers; Step 2-1: Camera Sensor Simulation: The camera parameters, including resolution, focal length, and distortion coefficient, are configured using the Gazebo camera plugin gazebo_ros_camera. The plugin acquires virtual scene images in real time and generates RGB image data that conforms to the characteristics of real images. The acquired RGB image data is encapsulated into sensor_msgs / Image message type and published as a topic to provide visual input for the UAV's perception, localization, and navigation tasks. Step 2-2: LiDAR sensor simulation: To detect potential obstacles, LiDAR simulation is implemented based on the Gazebo ray detection function LineTraceByChannel; First, define the LiDAR parameters, including scanning range, resolution (horizontal and vertical angle interval), and maximum detection distance. Then, perform ray detection through iterative loops: generate ray direction vectors for each horizontal and vertical angle within the scanning range, call LineTraceByChannel to detect the intersection of the ray and the obstacle, and calculate the intersection distance. If the intersection distance between the ray and the obstacle is less than the maximum detection distance, then retain the point cloud data. Finally, all valid point clouds are encapsulated into sensor_msgs / PointCloud2 messages and published as topics for use by subsequent obstacle avoidance algorithms. Steps 2-3: RTK Sensor and Remote Controller Simulation: RTK simulation uses the gazebo_ros_gps plugin to simulate RTK positioning. The plugin generates positioning data that meets RTK positioning accuracy by superimposing Gaussian noise based on the actual position of the UAV in the Gazebo world coordinate system. The positioning data format includes latitude, longitude, altitude, and positioning confidence, and is published in topic format. Remote controller simulation reads keyboard input through the keyboard_remote_nodeROS node and maps the keyboard input to UAV control variables. Control commands include pitch control, roll control, yaw control, throttle control, and emergency stop function, and are published through ROS topics. Step 3 specifically involves: Step 3-1: PX4 Firmware Development Environment Setup: Install PX4-Gazebo during environment setup, successfully compile PX4 code under Linux system, add environment variables in the underlying environment to ensure that ROS and PX4 can call each other's function packages normally; Step 3-2: Sensor State Simulation in the Simulator: The core function of the simulator is to simulate the sensor state during real flight. For RTK sensors, positioning data is generated through Step 2-3. For IMU sensors, angular velocity and linear acceleration are generated through the gazebo_ros_imu plugin. The simulator encapsulates this sensor data into MAVLink messages and sends them to the PX4 flight controller. The state estimation module in the PX4 flight controller then calculates the real-time state variables of the UAV's position, velocity, and attitude using quaternion update formulas. Step 3-3: Simulation Environment and Flight Control Communication: The PX4 and Gazebo emulator communicate via the UDP protocol, employing a multi-port task allocation mechanism: UDP port 14540 is used for offboard mode communication, through which the flight control node on the ROS side sends desired speed / position commands to the PX4; UDP port 14550 is used for ground station communication, through which the ground station software receives UAV status data sent by the PX4 and sends mission commands; port 55555 is used for direct communication between the emulator and the PX4, through which Gazebo sends sensor simulation data to the PX4, and the PX4 sends motor control commands to Gazebo.

[0013] Step 4 specifically involves: Step 4-1: License plate image data acquisition and preprocessing: Acquire license plate images under different scenes, covering different lighting and angles, and annotate and enhance the images; divide the image dataset into training set, validation set and test set in an 8:1:1 ratio, train the dataset using the YOLOv8 network, and adjust the hyperparameters through the validation set accuracy.

[0014] Step 4-2: Model Format Conversion and Deployment: Convert the training-generated pt weight file into an ONNX file (Open Neural Network Exchange Format). The converted ONNX file is compatible with the C++ inference framework. Then, deploy the ONNX model in the C++ program to achieve efficient model inference. Step 4-3: Model Inference and Prediction Output: Call the trained model and use the preprocessed license plate image as input. The model outputs a feature map through feature extraction, feature fusion, detection and recognition networks. Specifically, the detection branch outputs the pixel coordinates and position confidence of the license plate bounding box vertices, the identification branch outputs the character category probability distribution, and the category with the highest probability is taken as the character detection result; Step 4-4: Feature Map Post-processing and Filtering: First, based on the license plate localization algorithm, extract all candidate license plate bounding boxes from the feature map. Then, use the Non-Maximum Suppression (NMS) algorithm to filter the frames and remove redundant bounding boxes. The core of the NMS algorithm is to calculate the Intersection over Union (IoU) of two candidate bounding boxes. If the IoU of two boxes is higher than the set NMS threshold (e.g., 0.5), only the box with higher location confidence is retained. Steps 4-5: Result Return: Organize the license plate bounding box information filtered by NMS into structured data, including bounding box coordinates, location confidence, and character category information, and publish the data through the ROS topic.

[0015] Step 5 specifically involves: Step 5-1: Flight Status Visualization: Use ROS's RViz tool to display the drone's flight status in real time, including the drone's current position, flight path, speed, attitude (pitch, roll, yaw), etc.; add a "RobotModel" display item in RViz, load the drone URDF file from Step 1-2, and visually present the drone's attitude; add a "Path" display item, subscribe to the drone's position topic, and draw the flight path in real time; add an "Odometry" display item to show the drone's speed and coordinate axes. Step 5-2: Sensor Data Visualization: Add corresponding display items in RViz and display the data collected by drone sensors (camera, GPS, IMU, etc.) by subscribing to topics published by the sensors; for license plate images and their location information captured by the camera, draw rectangles on the image to mark the license plate location and recognition results based on the bounding box coordinate information published in the license plate recognition results topic; Step 5-3: Visualize the license plate recognition results: This includes the license plate bounding box, the recognized character information, the recognition confidence level, and the recognition time. The license plate recognition results are displayed through a visual interface, and warnings are set for low confidence levels to facilitate users' monitoring of the accuracy and stability of the license plate recognition task. Step 5-4: Multi-interface data summary and display: Design 4 independent visualization interfaces to display flight trajectory, sensor accuracy, recognition status, and control status respectively, so as to realize the classification and display of data, so that users can comprehensively monitor all data in the simulation process.

[0016] Step 5-5: Interactive Function Implementation: Based on Rviz and custom plugins, provide real-time interactive functions: Users can adjust flight parameters in real time and send them to the flight control node via ROS topics; adjust sensor parameter settings and update the ROS parameter server via the rosparamset command; change the license plate recognition algorithm and switch the inference model via ROS service / license_plate / switch_model; Steps 5-6: Generating and saving the visualization report: A visualization report generation module was designed for key data during flight, with core indicators including license plate recognition success rate, flight path accuracy, and sensor feedback quality, providing data support for subsequent analysis and optimization.

[0017] The beneficial effects of this invention are: 1. Highly realistic simulation reproduction ensures the effectiveness of algorithm verification: Through Gazebo's high-precision scene modeling and PX4 physical motion closed loop, the flight process of the UAV and its interaction with the environment can be realistically reproduced, solving the problems of data distortion and difficulty in algorithm transfer in traditional simulation platforms.

[0018] 2. High-precision real-time recognition to meet practical application needs. Real-time data transmission between modules is achieved through the ROS message mechanism. The YOLOv8 license plate recognition model is optimized using simulation and real data, achieving real-time recognition at a frame rate higher than that of drone cameras (30fps), suitable for scenarios such as traffic monitoring and parking management.

[0019] 3. Intuitive management of multi-dimensional data enhances platform usability. Through the RViz+Qt visualization toolchain, the system synchronously displays drone attitude, sensor data, and recognition results, allowing users to intuitively grasp the system status. It also provides parameter configuration, alarm prompts, and report export functions, improving user development efficiency.

[0020] 4. Modular design with scalability. Based on ROS standardization, each module is independently decoupled, facilitating the addition of sensors, upgrading of recognition algorithms, and expansion of scenarios, greatly reducing secondary development costs and enabling rapid adaptation to different application scenarios. Attached Figure Description

[0021] Figure 1 This is a diagram of the overall platform structure of the present invention.

[0022] Figure 2 This is a flowchart of the drone license plate recognition algorithm of the present invention.

[0023] Figure 3 This is a rendering of the virtual simulation environment of the present invention.

[0024] Figure 4 This is a visualization of the ROS license plate recognition simulation operation effect of the present invention. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings.

[0026] like Figure 1 , Figure 2 As shown, this invention provides a ROS-based drone license plate recognition simulation platform and its construction method, aiming to provide an efficient simulation testing environment for drone flight control, sensor simulation, and license plate recognition tasks. This platform combines multiple modules such as virtual environment creation, sensor simulation, flight control, and data visualization, and can accurately simulate drone flight processes and license plate recognition tasks under different environments.

[0027] The following is a detailed description of the technical solution: 1. Virtual Environment Module The virtual environment module is responsible for creating and simulating the environment required for drone flight. This module provides a simulated scenario for license plate recognition tasks, including elements such as roads, vehicles, buildings, and license plates. The core functions of the virtual environment module include: Environment Simulation: Based on the Ubuntu 20.04 operating system, a realistic ground scene was constructed using the ROS1-compatible simulation tool Gazebo, including roads, traffic signs, and buildings. ROS1 supports both Python and C++, with different modules written for each language. First, a new Gazebo environment was created, and the drone simulation model was imported into it. Then, the 3D models of the car and license plate generated by Blender were imported into the Gazebo environment. The license plate target could be dynamically changed according to simulation needs (e.g., vehicle speed, license plate angle, etc.).

[0028] Simulation of dynamic environmental changes: Lighting, rain and snow, pedestrians, etc. are added to the virtual environment to set different lighting conditions, weather conditions and dynamic environments. These dynamic changes can effectively affect the license plate recognition results.

[0029] Target object dynamic simulation: In a virtual environment, the license plate target is set as a dynamically changing target, simulating different appearances, sizes, colors and positions of the license plate. The simulation task can simulate license plate recognition tasks at different angles and distances.

[0030] After simulating the virtual environment, configure it uniformly in the world file in Gazebo, and add the world file to the launch file associated with ROS1. Start the virtual environment node through ROS1 communication, and you can observe the dynamic changes of the virtual environment in real time.

[0031] 2. Sensor Module The sensor module is responsible for simulating various sensor data from the drone. The accuracy of the sensor simulation directly affects the realism of the simulation results, especially in license plate recognition, where the simulation of camera sensors is crucial. This module includes: Camera Simulator: This sensor simulator is used to simulate a high-definition camera on a drone, capturing and preprocessing images of ground license plates (e.g., noise reduction, enhancement). Camera parameters such as resolution, field of view, and focal length can be adjusted as needed.

[0032] License plate image acquisition and processing: Simulates a camera acquiring license plate images in different flight modes (such as cruise, fixed point, fixed altitude, etc.) and simulates the image preprocessing process. The processed images will be provided to the license plate recognition algorithm for identification.

[0033] Dynamic adjustment: The camera simulator can dynamically adjust the image acquisition effect according to environmental conditions and flight path.

[0034] GPS simulator: Simulates the Global Positioning System (GPS) signal of the drone, providing real-time location data. The GPS simulator needs to provide accurate latitude and longitude information for path planning and navigation in the flight control system.

[0035] IMU Simulator: The Inertial Measurement Unit (IMU) simulator provides attitude data (such as pitch, roll, and yaw angles), acceleration information, and angular velocity data for the UAV. IMU data is a key input in flight control and stability algorithms.

[0036] 3. Flight Control Module The flight control module, based on ROS and PX4 firmware, simulates the drone's flight control system, ensuring the drone can perform license plate recognition tasks in complex environments. The main functions of the flight control module include: Flight Control Algorithm: An automatic flight control system (Autopilot) is implemented using ROS control toolkits (such as ros_control), adjusting the drone's flight attitude, speed, heading, and other parameters based on sensor data. The control algorithm adopts a hierarchical control architecture of "main algorithm + auxiliary algorithm," including different types of flight control algorithms such as PID control, LQR control, and robust control. Conventional stabilization control uses PID control, designed for typical scenarios without strong interference, and is used for basic control of the attitude loop (roll / pitch / yaw) and position loop (x / y / z axes). LQR control is suitable for scenarios where the drone's motion is nearly linear (such as constant speed cruise or small attitude angles), used to improve trajectory tracking accuracy. Joint optimization of position and speed is achieved through the lqr_controller module of PX4. Robust control is suitable for scenarios with strong environmental interference (such as rain and snow). When the sensor module detects strong environmental interference, it automatically switches to robust control mode via the ROS service / control / switch_robust to ensure drone attitude stability during shooting.

[0037] Flight Mode Switching: The flight control module allows switching between various flight modes via ROS topic / task / set_mode, such as altitude hold, stationary flight, and cruise flight, to meet different license plate recognition task requirements. In each flight mode, the flight control algorithm automatically adjusts the flight path and attitude based on sensor feedback. For low-altitude license plate acquisition scenarios (such as parking lots and residential areas), the drone needs to photograph static or low-speed vehicles at a fixed altitude, employing altitude hold mode. In altitude hold mode, PID control is prioritized, and the z-axis altitude loop control is strengthened. For static license plate tracking scenarios, after recognizing the license plate, the drone needs to hover above the target and take photos from multiple angles, employing stationary mode. In stationary mode, the control algorithm uses a fusion control of "PID position loop + LQR speed loop," with the position loop tracking the target position and the speed loop optimizing the smoothness of the x / y direction speed. For license plate search over a large area (such as highway scenarios), it needs to enter cruise mode according to a preset trajectory. In cruise mode, LQR control is prioritized, and it can automatically switch to robust control if strong interference is encountered.

[0038] Real-time trajectory tracking adjustment: The flight control system needs to track the predetermined flight trajectory in real time. During flight, based on the current state of the UAV (position, speed, attitude) fed back by the sensor module and the preset trajectory, the flight control system calculates the error and corrects the control quantity according to the error, so as to make real-time adjustments and ensure that the UAV can continuously and accurately capture license plate targets during flight.

[0039] 4. Socket Protocol and Data Transmission In this simulation platform, the modules need to exchange data in real time. To ensure smooth data transmission between different modules, the Socket protocol is used for data communication. This protocol has the following characteristics: Efficient data transmission: The Socket protocol ensures real-time data synchronization between flight control, sensor modules, license plate recognition modules, and visualization modules. Data is transmitted through ROS topics and services, ensuring that each module can respond to changes promptly.

[0040] Scalability: The Socket protocol provides a flexible framework for data transmission. More sensor types or flight modes can be added as needed in the future, and all data interaction between modules can be seamlessly connected via the Socket protocol.

[0041] 5. License Plate Recognition Module The license plate recognition module is one of the core modules of this invention. It is responsible for processing and recognizing the acquired license plate images, and ultimately outputting the recognition result. The functions of this module include: Image preprocessing: Receives image data from the sensor module and performs preprocessing operations such as noise reduction, enhancement, and image segmentation to provide a clear image for subsequent license plate recognition.

[0042] License Plate Localization and Recognition: Based on machine learning or deep learning models (such as convolutional neural networks, CNNs), the system locates the license plate region and recognizes characters. The model can extract the license plate region from the image, identify the license plate number, and verify it. Taking the YOLOv8 deep neural network model as an example, the model takes a pre-processed 640×640 image as input. It first feeds the image into a feature extraction network to extract feature maps at five scales from 80×80 to 5×5. For each feature map grid point, three anchor boxes of 12×24, 16×32, and 20×20 are preset to match the license plate aspect ratio. The detection network predicts the anchor box position offset, target confidence, and class probability. Candidate boxes with a target confidence ≥ 0.5 are retained to generate an initial set. The IoU (Intersection over Union) is calculated for any two boxes in the initial candidate set. After sorting them in descending order of target confidence, non-maximum suppression is performed, retaining 1-2 boxes. The optimal license plate bounding box is determined and its coordinates are corrected. The resulting license plate sub-image is then cropped and uniformly adjusted to 240×80 pixels. Edge detection is performed on the license plate sub-image using the Canny operator, and affine transformation matrix is ​​constructed using Hough transform to eliminate tilt. The corrected sub-image is then adaptively binarized, and the character image is input into a character detection network to stitch together the complete license plate number. Finally, the license plate location confidence (optimal bounding box target confidence) and character recognition confidence (average of the maximum probability of all characters) are calculated and weighted to obtain the comprehensive confidence. The license plate number, comprehensive confidence, and location coordinates are then published to the visualization module via the ROS topic / license_plate / final_result.

[0043] Recognition result feedback: The license plate recognition results will be displayed in real-time in the visualization module, including information such as the license plate number, recognition confidence, and whether the recognition is successful, for users to conduct real-time monitoring and decision-making.

[0044] 6. Visualization module The visualization module is responsible for displaying the flight status of the drone, sensor data, and real-time results of license plate recognition. Through ROS visualization tools (such as RViz), users can intuitively understand the progress of the simulation process. The functions of the visualization module include: Flight trajectory display: Real-time display of the flight trajectory of the drone, including the current position of the drone, flight path, speed, and attitude information.

[0045] Visualization of sensor data: Display the data fed back by sensors (such as cameras, IMUs, GPS, etc.) to help users understand the status of the drone and the performance of the sensors.

[0046] License plate recognition results: Display information such as the recognized license plate number, recognition confidence, and whether the recognition is successful. Through the graphical interface, users can see in real-time the effect of the drone when performing the license plate recognition task. The ROS-based drone license plate recognition simulation platform of the present invention constructs a highly restored virtual simulation environment through Gazebo, as Figure 3 shown. Multiple yellow vehicles with license plates are deployed in this environment to simulate a real road scene; at the same time, a drone model is incorporated to provide a physical space for its flight and license plate collection. During the simulation process, the visual camera sensor carried by the drone collects images of ground vehicles, and the collected image data is transmitted to the license plate recognition algorithm module through ROS topics. As Figure 4 shown, the license plate recognition algorithm module is based on the YOLOv8 neural network, detects license plates and recognizes characters in the collected images, and finally outputs the recognition results in real-time in the ROS visualization interface (such as the license plate number "New I3SR2F"), while displaying key information such as recognition confidence. The entire simulation process realizes a full-link closed-loop from virtual environment construction, sensor data collection to license plate recognition result visualization, effectively verifying the support ability of the platform for license plate recognition tasks in different scenarios, and providing an efficient simulation test environment for the research and optimization of drone flight control and license plate recognition algorithms.

Claims

1. A ROS-based unmanned vehicle license plate recognition simulation platform, characterized in that, ROS is the core communication framework, integrating virtual environment module, sensor simulation module, flight control module, Socket protocol and data transmission module, license plate recognition algorithm module and data visualization module; The virtual environment module is based on the simulation requirements of different types of unmanned aerial vehicles and scenes, and constructs an extensible unmanned aerial vehicle flight scene and unmanned aerial vehicle physical simulation model, and provides dynamic environmental interference; The sensor simulation module simulates multiple types of sensors carried by the unmanned aerial vehicle and outputs high-fidelity sensor data; The flight control module adjusts the flight attitude, speed and heading parameters of the unmanned aerial vehicle according to the sensor data of the sensor simulation module; The Socket protocol and data transmission module can ensure real-time data synchronization between the flight control module, the sensor simulation module, the license plate recognition algorithm module and the data visualization module, and realize low-latency communication between the modules; The license plate recognition algorithm module detects, recognizes characters and outputs results based on the YOLOv8 neural network for the license plate images captured by the sensor simulation module in the virtual scene; The data visualization module displays the flight state of the unmanned aerial vehicle, sensor data and real-time results of license plate recognition in real time.

2. The ROS-based unmanned aerial vehicle license plate recognition simulation platform according to claim 1, wherein, The sensor simulation module includes a visual camera sensor, an IMU sensor, a laser radar sensor, an RTK sensor and a remote controller; The visual camera sensor is mounted at the front end of the nose of the unmanned aerial vehicle simulation model in the virtual environment, and is used to shoot ground vehicles and output an RGB image topic / camera / image_raw; The IMU sensor is configured at the center of gravity of the unmanned aerial vehicle body, and is used to collect real-time attitude and speed data of the unmanned aerial vehicle and output an IMU data topic / imu / data; The laser radar sensor is located at the front end of the nose of the unmanned aerial vehicle, scans the front obstacles and ground terrain, and outputs a point cloud topic / lidar / points; The RTK sensor is located at the top center of the back of the unmanned aerial vehicle, receives virtual satellite signals, provides high-precision position reference for flight control, and outputs a positioning topic / rtk / fix; The remote controller communicates with the flight control module through ROS, simulates remote control of the unmanned aerial vehicle by the ground station, and outputs a control instruction topic / remote / control; The data fusion of the IMU sensor and the RTK sensor provides attitude and position data for flight; the laser radar sensor scans the front obstacles in real time to ensure flight safety; the visual camera sensor performs license plate image acquisition tasks; the remote controller serves as an emergency control means to take over the unmanned aerial vehicle when the flight control algorithm fails.

3. The method for constructing a ROS-based unmanned vehicle license plate recognition simulation platform according to claim 1 or 2, characterized in that, The method comprises the following steps: Step 1: build a scene, an unmanned aerial vehicle model, a sensor model and flight control as the physical basis of the entire simulation platform; Step 2: based on the sensor model constructed in step 1, realize the publication of sensor data in the virtual scene through ROS; Step 3: Based on the integration of the unmanned aerial vehicle model constructed in Step 1 and the PX4 flight control, the PX4 flight control instructions are converted into physical motion of the unmanned aerial vehicle model through the PX4-Gazebo simulation system, flight state data is obtained, and the deployment and verification of the flight control algorithm are realized; Step 4: Based on the visual camera sensor data in Step 2, the development and integration of the license plate recognition function are completed, and the recognition result data is obtained; Step 5: Based on the sensor data in Step 2, the flight state data in Step 3, and the recognition result data in Step 4, multi-dimensional data display is realized.

4. The method according to claim 3, wherein, The step 1 is specifically: Step 1-1: Scene building: a multi-scene simulation environment is constructed with Gazebo simulation engine as the core, scene road and car model are input, and license plate map is bound to the specified position of the car model through Gazebo material editing function; the light intensity and direction of different time periods are simulated through Gazebo light plug-in, and the performance and adaptability of the sensor under different environmental conditions are accurately evaluated; Step 1-2: Unmanned aerial vehicle model building: an unmanned aerial vehicle model library is established, including URDF files of different models of quadcopters and hexacopters, core parameters such as body mass and wheelbase are defined, and extension and customization according to requirements are realized; then the model file is imported into the ROS function package to meet the simulation needs of different unmanned aerial vehicles; Step 1-3: Sensor model building: sensor models are mounted in the unmanned aerial vehicle model, and the sensor is associated with the Gazebo physical engine through gazebo_ros plug-in to ensure that the sensor moves synchronously with the unmanned aerial vehicle; Step 1-4: Flight control building: PX4 firmware is integrated into the ROS workspace, and communication between ROS and PX4 is established through the mavros function package to ensure that subsequent flight control instructions can be transmitted to the unmanned aerial vehicle model.

5. The method according to claim 4, wherein, The step 2 is specifically: The sensor model includes visual camera sensor, IMU sensor, laser radar sensor, RTK sensor, and remote controller; Step 2-1: Camera sensor simulation: Gazebo camera plug-in gazebo_ros_camera is used to configure camera parameters such as resolution, focal length, and distortion coefficient, the plug-in collects virtual scene images in real time, and generates RGB image data conforming to the characteristics of real images; the obtained RGB image data is packaged as sensor_msgs / Image message type, and is published in the form of topic, providing visual input for the perception, positioning, and navigation tasks of the unmanned aerial vehicle; Step 2-2: Laser radar sensor simulation: laser radar simulation is realized based on Gazebo ray detection function LineTraceByChannel to detect possible obstacles; First, define the laser radar parameters, including the scanning range, resolution, and maximum detection distance; perform ray detection through iterative loops: generate a ray direction vector for each angle in the horizontal direction and each angle in the vertical direction within the scanning range, call LineTraceByChannel to detect the intersection of the ray and the obstacle, and calculate the intersection distance; if the intersection distance of the ray and the obstacle is less than the maximum detection distance, the point cloud data is retained; Finally, all valid point cloud data is encapsulated as a sensor_msgs / PointCloud2 message and published in the form of a topic for subsequent obstacle avoidance algorithms; Step 2-3: RTK sensor and remote controller simulation: RTK simulation uses the gazebo_ros_gps plugin to simulate RTK positioning, which is based on the real position of the drone in the Gazebo world coordinate system, and superimposes Gaussian noise to generate positioning data that meets the RTK positioning accuracy. The positioning data format includes latitude, longitude, altitude, and position accuracy, and is published in the form of a topic; remote controller simulation reads keyboard input through the keyboard_remote_node ROS node and maps keyboard input to drone control quantities, including pitch control, roll control, yaw control, throttle control, and emergency stop function, and publishes control instructions through ROS topics.

6. The method according to claim 5, wherein, The step 3 is specifically: Step 3-1: PX4 firmware development environment setup: install PX4-Gazebo when setting up the environment, successfully compile PX4 code in the Linux system, and add environment variables in the underlying environment to ensure that ROS and PX4 can normally call each other's function packages; Step 3-2: Sensor state simulation of simulation simulator: the core function of the simulation simulator is to simulate the sensor state in real flight; for the RTK sensor, generate positioning data through step 2-3; for the IMU sensor, generate angular velocity and linear acceleration through the gazebo_ros_imu plugin; the simulator encapsulates these sensor data into MAVLink messages and sends them to the PX4 flight control, and then the state estimation module in the PX4 flight control calculates the real-time state quantities of the drone position, velocity, and attitude through the quaternion update formula; Step 3-3: Communication between simulation environment and flight control: PX4 and Gazebo simulator realize message communication through UDP protocol, using a multi-port division mechanism: UDP port 14540 is used for offboard mode communication, and the flight control node on the ROS side sends the desired speed / position instructions to PX4 through this port; UDP port 14550 is used for ground station communication, and the ground station software receives the drone state data sent by PX4 through this port and sends task instructions; port 55555 is used for direct communication between the simulator and PX4, and Gazebo sends sensor simulation data to PX4 through this port, and PX4 sends motor control instructions to Gazebo through this port.

7. The method according to claim 6, wherein, The step 4 is specifically: Step 4-1: License plate image data collection and preprocessing: Collect license plate images under different scenarios, covering different lighting conditions and angles, and label and enhance the images; divide the image dataset into training, validation, and test sets, and use the yolov8 network to train the dataset, adjusting the hyperparameters based on the validation set accuracy; Step 4-2: Model format conversion and deployment: Convert the pt weight file generated by training into an ONNX file, which is compatible with the C++ inference framework; then deploy the ONNX model in the C++ program to achieve efficient inference of the model; Step 4-3: Model inference and prediction result output: Call the trained model and use the preprocessed license plate image as input, and the model outputs feature maps through feature extraction, feature fusion, detection, and recognition networks; The detection branch outputs the pixel coordinates and position confidence of the license plate bounding box, and the recognition branch outputs the character class probability distribution, with the class with the highest probability being the character detection result; Step 4-4: Feature map post-processing and screening: First, extract all candidate license plate bounding boxes from the feature map based on the license plate positioning algorithm, then use the non-maximum suppression (NMS) algorithm to filter the frames, and remove redundant bounding boxes; the NMS algorithm calculates the intersection over union (IoU) of two candidate bounding boxes, and if the IoU of two boxes is higher than the set NMS threshold, only the bounding box with higher position confidence is retained; Step 4-5: Result return: Organize the license plate bounding box information filtered by NMS into structured data, including bounding box coordinates, position confidence, and character class information, and publish this data through the ROS topic.

8. The method according to claim 7, wherein, The step 5 is specifically: Step 5-1: Flight state visualization: Use the RViz tool of ROS to display the flight state of the UAV in real time, including the current position, flight path, speed, and attitude information of the UAV; add a "RobotModel" display item in RViz to load the UAV URDF file from step 1-2 and visually present the UAV attitude; add a "Path" display item to subscribe to the UAV position topic and draw the flight path in real time; Add an "Odometry" display item to display the UAV speed and coordinate axes; Step 5-2: Sensor data visualization: Add corresponding display items in RViz and display the data collected by the UAV sensors by subscribing to the sensor published topics; for the license plate images captured by the camera and their position information, based on the bounding box coordinate information published in the license plate recognition result topic, draw a rectangular frame on the image to label the license plate position and recognition result; Step 5-3: Visualization of license plate recognition results: License plate recognition includes license plate bounding box, recognized character information, recognition confidence, and recognition time; display the license plate recognition results through the visualization interface and set a warning for low confidence to help users monitor the accuracy and stability of the license plate recognition task; Step 5-4: Multi-interface data aggregation display: Design 4 independent visualization interfaces to display flight trajectory, sensor accuracy, recognition state, and control state, respectively, to achieve classified display of data for users to comprehensively monitor all data during simulation; Step 5-5: Interaction function implementation: Based on Rviz and custom plugins, provide real-time interaction functions: users adjust flight parameters in real time and send them to the flight control node through ROS topics; adjust sensor parameter settings and update the ROS parameter server through the rosparam set instruction; change the license plate recognition algorithm and switch the inference model through the ROS service / license_plate / switch_model; Step 5-6: Visual report generation and saving: Design a visual report generation module for key data during flight to provide data support for subsequent analysis and optimization.

Images