PX4-based unmanned aerial vehicle sensor and visual information time alignment method

Abstract

The invention discloses a PX4-based unmanned aerial vehicle sensor and visual information time alignment method, and belongs to the technical field of unmanned aerial vehicle image information acquisition and processing, and the method comprises the steps: carrying out the MAVLink message communication of a flight controller and an end machine in an unmanned aerial vehicle through a serial port; after the flight control receives the message transmitted by the receiver, a trigger instruction is sent to the camera, so that the camera collects an image, and timing and recording current picture information are carried out; and transmitting the obtained image and picture information to a terminal machine to form a one-to-one correspondence relationship of terminal machine information-flight control instruction-picture information, thereby achieving alignment of a trigger instruction and a picture sequence, and aligning sensor data through a timestamp. According to the method, the MAVLink message is sent through the flight controller to trigger the camera to collect the image, and various information of the image, including the timestamp, the image serial number and the like, is recorded, so that the accurate correspondence between the trigger instruction and the image collection is ensured, and the accuracy of time alignment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle image information acquisition and processing, and in particular to a method for time alignment of unmanned aerial vehicle sensors and visual information based on PX4. Background Art

[0002] During the flight of a drone, when the visual module is used to collect images, the image data is often lost, resulting in errors in the time series and causing problems in the experimental image collection. It is impossible to determine whether the collected image information is the flight data of the drone at the same time.

[0003] The patent with application number 202110968225.6 discloses a method for aligning the timestamp of data. It organizes the different data collected by the sensor and the camera into two data groups, and makes a curve chart of the camera data changing over time based on the time change trend of the camera data. This curve is then used to calculate the camera data value corresponding to each sensor acquisition time, generate updated alignment data, and then obtain pictures of various sensors and camera data at the same time point. However, the alignment method does not take into account the situation of frame loss. In the case of image acquisition loss, the obtained curve is incomplete or distorted, and the result of the interpolation calculation will no longer be accurate. The lost image data will result in no corresponding values ​​at certain time points, affecting the matching effect.

[0004] In this regard, if you want to obtain complete data, you can only optimize the data through data prediction or multivariate correction. However, these are compensatory measures taken after the image is lost and belong to the post-data processing process. They cannot solve these problems more directly. Moreover, these calculations are usually analyzed using software, and the reliability cannot be guaranteed and lacks robustness. This patent is more suitable for scenarios where sensor data is relatively simple, the requirements for data processing accuracy are not high, and the use of interpolation and simple prediction will not have a big impact.

[0005] Therefore, in order to cope with the need to accurately control the timing of data acquisition and to more efficiently and quickly complete the time alignment between sensor data and image information while solving the impact of lost image frames on time alignment, a new method is needed. Summary of the invention

[0006] The purpose of the present invention is to provide a method for temporal alignment of drone sensors and visual information based on PX4, which solves the problems mentioned in the background technology.

[0007] To achieve the above object, the present invention provides a method for time alignment of drone sensors and visual information based on PX4, the steps comprising:

[0008] S1. Use the serial port to communicate MAVLink messages between the flight control and the terminal in the drone;

[0009] S2, after the flight control receives the message transmitted by the terminal, it sends a trigger command to the camera. After receiving the command, the camera collects images, counts and records the current image information;

[0010] S3. The image and picture information obtained in step S2 is transmitted to the terminal, forming a one-to-one correspondence between the terminal message, the flight control command, and the picture information, so as to align the trigger command with the picture sequence, and then align the sensor data through the timestamp.

[0011] Preferably, the MAVLink message communication in step S1 is sending a camera trigger message in MAVLink.

[0012] Preferably, the current picture information in step S2 includes the size, timestamp, and image sequence of the current picture.

[0013] Preferably, the current picture information in step S2 includes the size, timestamp, and image sequence of the current picture.

[0014] Preferably, in step S2, after receiving the instruction, the camera performs image acquisition, timing and records the current picture information, including:

[0015] S21, after receiving the command, the camera collects images, times and stores the images in a preset txt file;

[0016] S22, read the image in the txt file and obtain the current image information;

[0017] S23, save the image, record and save the picture information currently corresponding to the image.

[0018] Preferably, when storing the image in step S21, it is determined whether a frame is lost. If a frame is lost, the frame number and error information are saved. If no frame is lost, the frame number is saved and stored according to the frame number.

[0019] Preferably, the image and picture information obtained in step S3 is transmitted to the terminal via the camera host computer SDK or in the form of a self-written picture message.

[0020] Preferably, in step S3, the trigger instruction sequence obtained after collection is matched with the image information stored in the terminal through the flight control log, forming a one-to-one correspondence between the terminal message-flight control instruction-image information.

[0021] Preferably, in step S3, the sensor data is interpolated linearly or nonlinearly by an interpolation method to improve the time alignment accuracy.

[0022] Therefore, the present invention adopts the above-mentioned PX4-based drone sensor and visual information time alignment method, which has the following beneficial effects:

[0023] (1) The method of the present invention uses a hardware trigger instruction to connect the flight control with the terminal using a serial port, and sends a camera trigger instruction through the flight control to enable the camera to capture an image and record various types of image information, thereby ensuring the precise correspondence between the trigger instruction and the image acquisition and improving the accuracy of time alignment;

[0024] (2) The method of the present invention aligns the time series of the flight control log and the collected image sequence to ensure that each image has corresponding sensor data, and performs linear or nonlinear interpolation on the sensor data through interpolation method to further improve the accuracy of time alignment;

[0025] (3) By sending trigger commands through the flight control, the timing of image acquisition can be flexibly controlled to adapt to different application scenarios. This method not only improves the efficiency of data acquisition, but also enhances the adaptability and robustness of the system;

[0026] (4) During the image acquisition process, the acquisition time and related information of each frame of the image are recorded to ensure the integrity and reliability of the data. Even if the image is lost, corresponding explanations can be provided without affecting the time alignment of subsequent data.

[0027] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of an embodiment of the present invention;

[0029] Figure 2 It is a main program flow chart of an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of the image collection thread of an embodiment of the present invention;

[0031] Figure 4 This is a diagram of the frame loss process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] Example

[0034] The present invention provides a method for time alignment of drone sensors and visual information based on PX4, the steps comprising:

[0035] S1. Use the serial port to communicate MAVLink messages between the flight control (PX4) and the terminal (such as NX) in the drone, and send the camera trigger message in MAVLink.

[0036] After S2 and PX4 receive the camera trigger message from the receiving end, they send a trigger command to the camera. After receiving the command, the camera acquires images, times and records the current image information, including the size, timestamp and image sequence of the current image.

[0037] The camera that collects the data of the image information must have an external / hard trigger function.

[0038] S3. The image and picture information obtained in step S2 are transmitted to the terminal through the camera host computer SDK or a self-written picture message. After the terminal receives the image and image-related information, the trigger instruction sequence after acquisition is obtained through the flight control log and the image information stored in the terminal is matched one by one, forming a one-to-one correspondence between the terminal message-flight control instruction-picture information, thereby achieving the alignment of the trigger instruction and the picture sequence, and then aligning the sensor data through the timestamp. In order to further improve the time alignment accuracy, the sensor data is linearly or nonlinearly interpolated through the interpolation method.

[0039] like Figure 1As shown in the figure, it is a simplified flowchart of the working process of this embodiment. It can be seen from the simplified flowchart that in each task, the terminal sends a camera trigger message of MavLink (hereinafter referred to as MAVLink message), the flight control receives the message and issues a command, the camera receives the command to collect and store the image, and when the terminal receives the image and related information or receives feedback, it sends the next trigger instruction, so that the trigger instruction corresponds to the data and information collected by the camera one by one, which is convenient for subsequent time alignment.

[0040] The time alignment method provided by the present invention is designed using multithreading, such as Figure 2 As shown in the figure, the main program is responsible for initializing the sending of serial port messages, initializing the camera, and triggering the camera, acquiring images, and storing messages through thread functions. After receiving the exit command, the image currently being processed is saved and then the program is exited. The multi-threaded design improves the response speed and stability of the system and ensures the continuity and integrity of image acquisition.

[0041] In the thread function, first call the camera to start capturing images and saving relevant information. This can be written based on the SDK of the existing camera device or by finding an existing open source program. Then, capture the image, such as Figure 3 As shown, the image acquisition process is as follows:

[0042] S21, after receiving the command, the camera collects the image, counts and stores the image in a preset txt file. When saving the image, it is first necessary to determine whether the frame is lost, such as Figure 4 As shown, if a frame is lost, the frame number and error information are saved; if no frame is lost, the frame number is saved and stored according to the frame number.

[0043] S22, read the image in the txt file, and obtain the size, timestamp, image sequence and other information of the current image.

[0044] S23, save the image, record the size, timestamp, image sequence and other information of the current picture corresponding to the image and save it, and exit the thread after saving.

[0045] Through image acquisition, the relevant time is recorded, and the relevant image sequence number, timestamp and some picture information are recorded, so that each flight control camera trigger command can have feedback, thereby ensuring that the stored image information data is complete, with corresponding sensor data, and the lost pictures have corresponding output error information. At the same time, the frame number is recorded, so that the picture sequence number corresponding to each trigger instruction is responded to, and it does not affect the time alignment of the subsequent collected pictures. At the same time, the time point of the image acquisition process is recorded, and the corresponding image acquisition, image transfer and other process times are obtained, which is convenient for more accurate use of data and optimization of the image acquisition process in the future, and improves the image acquisition frame rate.

[0046] If the experiment requires higher conditions, the acquisition time of each frame of the image recorded in the acquisition program can be used for relevant calculations and derivations to obtain a more accurate image timestamp, and then the relevant correspondence can be performed through the sensor's own time series. For example, the latitude, longitude, altitude obtained by GPS fusion and the three-axis attitude angle obtained by the inertial sensor IMU are taken as examples. Since sensors and other devices record position and attitude at a higher frequency, there is no accurate timestamp that can correspond one-to-one with the trigger instruction. The interpolation method can be used to perform corresponding linear or nonlinear interpolation on the sensor data to match it with the image information, so that the sensor information at the corresponding moment of the picture can be obtained.

[0047] Therefore, the present invention adopts the above-mentioned PX4-based drone sensor and visual information time alignment method, sends MAVLink messages through the flight control to trigger the camera to capture images, and records various types of image information, including timestamps, image sequence numbers, etc., to ensure the precise correspondence between trigger instructions and image acquisition, and improve the accuracy of time alignment.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for temporal alignment of drone sensors and visual information based on PX4, characterized in that the steps include: S1. Use the serial port to communicate MAVLink messages between the flight control and the terminal in the drone; S2, after the flight control receives the message transmitted by the terminal, it sends a trigger command to the camera. After receiving the command, the camera collects images, counts and records the current image information; S3. The image and picture information obtained in step S2 is transmitted to the terminal, forming a one-to-one correspondence between the terminal message, the flight control command, and the picture information, so as to align the trigger command with the picture sequence, and then align the sensor data through the timestamp.

2. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 1, characterized in that: The MAVLink message communication in step S1 is to send a camera trigger message in MAVLink.

3. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 1, characterized in that: In step S2, the camera has an external / hard trigger function.

4. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 1, characterized in that: The current picture information in step S2 includes the size, timestamp, and image sequence of the current picture.

5. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 4, characterized in that: In step S2, after receiving the instruction, the camera performs image acquisition, timing and records the current image information, including: S21, after receiving the command, the camera collects images, times and stores the images in a preset txt file; S22, read the image in the txt file and obtain the current image information; S23, save the image, record and save the picture information currently corresponding to the image.

6. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 5, characterized in that: When storing the image in step S21, it is determined whether a frame is lost. If a frame is lost, the frame number and error information are saved. If no frame is lost, the frame number is saved and stored according to the frame number.

7. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 1, characterized in that: The image and picture information obtained in step S3 are transmitted to the terminal via the camera host computer SDK or in the form of a self-written picture message.

8. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 1, characterized in that: In step S3, the trigger instruction sequence obtained after collection is matched with the image information stored in the terminal through the flight control log, forming a one-to-one correspondence between the terminal message-flight control instruction-image information.

9. The method for temporal alignment of drone sensors and visual information based on PX4 according to claim 1, characterized in that: In step S3, the sensor data is interpolated linearly or nonlinearly by an interpolation method.

Images