A time synchronization device and method between unmanned aerial vehicles based on light flashing coding

Abstract

The present invention discloses a time synchronization device between unmanned aerial vehicles based on light flicker coding, which realizes time synchronization between multiple unmanned aerial vehicles based on visual technology, and can realize time synchronization between multiple machines without installing a GPS module, and has the advantages of low cost and easy implementation. It includes: a light flicker coding unit, which is loaded on the unmanned aerial vehicle host; and a camera and processing unit, which is loaded on each unmanned aerial vehicle slave; the unmanned aerial vehicle host controls the light to flicker at the corresponding time synchronously and in real time according to its system time and the light flicker coding rules set by the light flicker coding unit; other unmanned aerial vehicle slaves use the camera and processing unit to continuously shoot the flashing light of the unmanned aerial vehicle host, and process the captured video image to obtain the decoded time information, and after comparing it with the time of their own system, correct the deviation between their system time and the system time of the unmanned aerial vehicle host, and complete the time synchronization between unmanned aerial vehicles.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle control, and in particular to a time synchronization device between unmanned aerial vehicles based on light flashing coding. The present invention also provides a time synchronization method between unmanned aerial vehicles based on light flashing coding. Background Art

[0002] Drone technology has been rapidly developed and applied in various industries. With the overall planning of the low-altitude economy, more and more drones are working together. Time synchronization between drones is the basis for multi-machine collaboration and has become a key focus in the field of drone applications. Time synchronization maintains a consistent time reference between drones, ensures the precise relative position and flight behavior between drones, and enables unified and precise actions to be performed at a specific time point. At the same time, drones are usually equipped with multiple types of sensors to achieve specific tasks, such as cameras, temperature and humidity sensors, etc. Only when the various sensors on each drone are synchronized in time can data matching analysis be performed.

[0003] Since the internal clock of the drone may drift, the system usually needs to be synchronized periodically. Conventional drone time synchronization is achieved using the Global Positioning System (GPS). Each drone receives the GPS signal and adjusts the internal clock of the drone system according to the received time information. GPS time synchronization has high accuracy, but it is highly dependent on the GPS signal. If the GPS signal is affected by environmental factors in underground tunnels, between high-rise buildings in cities, and in mountainous canyons, the effect of this time synchronization method will be significantly reduced. Ground base stations or reference nodes (such as radio, Wi-Fi base stations or transmitters) can also be used to communicate to synchronize time, but base stations need to be deployed on the ground in advance, which increases the complexity and cost of the system, and the activity range of drones is also limited. For this reason, it is urgent to develop a device to achieve time synchronization between multiple drones. Summary of the invention

[0004] In view of the above problems, the present invention provides a time synchronization device between UAVs based on light flashing coding, which realizes time synchronization between multiple UAVs based on visual technology, adapts to various application environments, and is particularly suitable for UAVs with pure visual navigation and applications. It can realize time synchronization between multiple UAVs without the need to install a GPS module, and has the advantages of low cost and easy implementation.

[0005] A time synchronization device between unmanned aerial vehicles based on light flashing coding, characterized in that it includes:

[0006] A light flashing coding unit, which is installed on the drone host;

[0007] and a camera and processing unit, which is mounted on each slave drone;

[0008] The host drone controls the lights to flash at the corresponding time synchronously and in real time according to its system time and the light flashing coding rules set by the light flashing coding unit; other slave drones use the camera and processing unit to continuously shoot the flashing lights of the host drone, and process the captured video images to obtain the decoded time information. After comparing it with the time of their own system, they correct the deviation between their system time and the system time of the host drone to complete the time synchronization between drones.

[0009] It is further characterized by:

[0010] The light flashing coding unit is composed of a time coding module and a lamp module. The time coding module controls the lamp module to flash at the actual moment matching the time information to be reflected by the light flashing according to the system time of the drone host and the set light flashing coding method, so as to realize the light flashing time coding signal synchronized with the host system time;

[0011] The video and processing unit is composed of a camera and a time code processing module. The camera is used to continuously shoot the flashing signal of the lighting module of the drone host to form a sequence of video images, and record the exposure time of each frame of the image based on its own system time. The time code processing module processes the video image shot by the camera, decodes the time information represented by the flashing of the light in the image, and compares it with the image exposure time to obtain the deviation between the drone slave system time and the host system time, and corrects the slave system time according to the deviation.

[0012] It is further characterized by:

[0013] The drone host and the drone slave are combined into a flight platform using existing drones. The drone itself has a time system and obtains a relatively stable system time when working.

[0014] The time encoding module of the drone host accurately and with low latency obtains the current time of the drone system, with an accuracy one order of magnitude higher than the time synchronization accuracy. A hardware-level high-precision timer is used to directly access registers in kernel mode from FPGA to obtain high-precision time information from hardware. The time code processing module of the drone slave uses a hardware-level high-precision timer to directly access registers in kernel mode from FPGA to obtain high-precision time information from hardware.

[0015] The light flashing coding method of the light flashing coding unit uses the number of light flashes for coding, and the time period of each flash is fixed. The time coding module uses a high-precision switch circuit to control the flashing of the light emitting device of the lamp module. The lamp module includes a light emitting device flashing multiple times for coding or includes multiple independent light emitting devices flashing simultaneously, representing different bits. The flashing start time is set according to all the time required for the complete flashing of the coded time information once, and the flashing starts at this time;

[0016] The camera has a high acquisition frame rate, which is 10 to 20 times higher than the light flickering frequency, and the camera exposure is controlled by an external trigger signal generated by a time code processing module, accurately matching the system time of the drone slave, and the trigger signal uses a TTL level or a differential signal;

[0017] The time code processing module realizes light feature extraction and time encoding and decoding processing, and performs time synchronization correction after calculating the time deviation, and is implemented using a mature SoC chip or FPGA chip.

[0018] The method for time synchronization between unmanned aerial vehicles based on light flashing adopts a time synchronization device between unmanned aerial vehicles based on light flashing coding, and is characterized in that it includes the following steps:

[0019] Step 1: Initialize the system time of each drone;

[0020] Step 2: The drone starts to perform the mission;

[0021] Step 3: Determine whether to start time synchronization. If synchronization is started, go to step 4; otherwise, go to step 2.

[0022] Step 4: Obtain system time from the drone And synchronously generate a camera external trigger signal to trigger the camera to shoot;

[0023] Step 5: The camera of the drone slave is aimed at the direction of the drone host lighting module, and the video images are continuously captured according to the external trigger signal, and the images are transmitted to the time code processing module;

[0024] Step 6: The time encoding module of the drone host obtains the system time

[0025] Step 7: The time coding module of the drone host calculates the time when the lamp starts flashing arrive At this moment, the lamp module starts to flash according to the coding method;

[0026] Step 8: After the time coding module of the drone host controls the lamp to complete the flashing set in step 7, the drone slave camera ends the video image shooting;

[0027] Step 9: The time code processing module of the drone slave processes the video image and decodes the time information;

[0028] Step 10: The time code processing module of the drone slave calculates the deviation between its own system time in the image and the host system time obtained by decoding;

[0029] Step 11: The drone slave corrects its own system time according to the time deviation. The corrected time

[0030] After adopting the present invention, it realizes time synchronization among multiple UAVs based on visual technology, adapts to various application environments, and can be used as a powerful supplement to GPS time synchronization in places where GPS is not available. It is particularly suitable for UAVs with pure visual navigation and applications. It can realize time synchronization among multiple UAVs without adding GPS modules, and has the advantages of low cost and easy implementation. Moreover, the time synchronization among multiple UAVs realized by it does not rely on external communication infrastructure and complex wireless communications, and can reduce radio interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram showing the structure of the device of the present invention;

[0032] Figure 2 is a flow chart of the method of the present invention;

[0033] Figure 3 An implementation of the light flicker coding of the method of the present invention (multiple consecutive coding values);

[0034] Figure 4 A schematic diagram of the timing of light flashing and camera image shooting in the method of the present invention; DETAILED DESCRIPTION

[0035] A time synchronization device between drones based on light flashing coding, see Figure 1 , which includes: a light flashing coding unit, which is loaded on the drone host;

[0036] and a camera and processing unit, which is mounted on each slave drone.

[0037] The host drone controls the lights to flash at the corresponding time synchronously and in real time according to its system time and the light flashing coding rules set by the light flashing coding unit; other slave drones use the camera and processing unit to continuously shoot the flashing lights of the host drone, and process the captured video images to obtain the decoded time information. After comparing it with the time of their own system, they correct the deviation between their system time and the system time of the host drone to complete the time synchronization between drones.

[0038] Among them, the light flashing coding unit is composed of a time coding module and a lamp module. The time coding module controls the lamp module to flash at the actual moment matching the time information to be reflected by the light flashing according to the system time of the drone host and the set light flashing coding method, so as to realize the light flashing time coding signal synchronized with the host system time;

[0039] Among them, the video and processing unit is composed of a camera and a time code processing module. The camera is used to continuously shoot the flashing signal of the lighting module of the drone host to form a sequence of video images, and record the exposure time of each frame of the image based on its own system time. The time code processing module processes the video images taken by the camera, decodes the time information represented by the flashing of lights in the image, and compares it with the image exposure time to obtain the deviation between the drone slave system time and the host system time, and corrects the slave system time according to the deviation.

[0040] In specific implementation, the drone host and the drone slave are combined into a flight platform using existing drones. The drone itself has a time system and obtains a relatively stable system time when working.

[0041] The time encoding module of the drone host accurately and with low latency obtains the current time of the drone system, with an accuracy one order of magnitude higher than the time synchronization accuracy. It uses a hardware-level high-precision timer to directly access registers from the FPGA in kernel mode to obtain high-precision time information from the hardware. The time code processing module of the drone slave uses a hardware-level high-precision timer to directly access registers from the FPGA in kernel mode to obtain high-precision time information from the hardware.

[0042] The light flashing coding method of the light flashing coding unit uses the number of light flashes for coding, and the time period of each flash is fixed. The time coding module uses a high-precision switch circuit to control the flashing of the light emitting device of the lamp module. The lamp module includes a light emitting device flashing multiple times for coding or includes multiple independent light emitting devices flashing simultaneously, representing different bits. The flashing start time is set according to all the time required for the complete flashing of the coded time information once, and the flashing starts when the time is reached;

[0043] The camera has a high acquisition frame rate, which is 10 to 20 times higher than the flashing frequency of the light. The camera exposure is controlled by the external trigger signal generated by the time code processing module, which accurately matches the system time of the drone slave. The trigger signal uses TTL level or differential signal.

[0044] The time code processing module realizes light feature extraction and time encoding and decoding processing, and performs time synchronization correction after calculating the time deviation. It is implemented using mature SoC chips or FPGA chips.

[0045] The method for time synchronization between UAVs based on light flashing adopts a time synchronization device between UAVs based on light flashing coding, see Figure 2 , which comprises the following steps:

[0046] Step 1: Initialize the system time of each drone;

[0047] Before the drones work, all drones are powered on and the time of each drone is initialized to ensure that the system time of each drone is not much different, such as within seconds;

[0048] Step 2: The drone starts to perform the mission;

[0049] Step 3: Determine whether to start time synchronization. If synchronization is started, go to step 4; otherwise, go to step 2.

[0050] Step 4: Obtain system time from the drone And synchronously generate a camera external trigger signal to trigger the camera to shoot;

[0051] The time code processing module of the drone slave obtains the current time of the drone system as accurately and with low latency as possible. The accuracy is one order of magnitude higher than the time synchronization accuracy. A hardware-level high-precision timer can be used to directly access registers in kernel mode from the FPGA to obtain high-precision time information from the hardware.

[0052] Step 5: The camera of the drone slave is aimed at the direction of the drone host lighting module, and the video images are continuously captured according to the external trigger signal, and the images are transmitted to the time code processing module;

[0053] The camera has a high acquisition frame rate, which is 10 to 20 times higher than the light flicker frequency, so as to fully record the light flicker process and ensure that the extracted light flicker brightness information has high time accuracy. The camera exposure is controlled by the external trigger signal generated by the time code processing module, and the trigger signal adopts TTL level or differential signal.

[0054] Step 6: The time encoding module of the drone host obtains the system time

[0055] The time encoding module of the drone host obtains the current time of the drone system as accurately and with low latency as possible. The accuracy is one order of magnitude higher than the time synchronization accuracy. A hardware-level high-precision timer can be used to directly access registers in kernel mode from the FPGA to obtain high-precision time information from the hardware.

[0056] Step 7: The time coding module of the drone host calculates the time when the lamp starts flashing arrive At this moment, the lamp module starts to flash according to the coding method;

[0057] The coding method of light flickering uses the number of light flickering to encode, and the time period of each flickering is fixed to T seconds. One flickering is from dark to bright and then to dark. The dark to brightest period in one flickering cycle is T / 2 seconds, and the brightest to dark period is T / 2 seconds. There is an interval ΔT greater than T seconds between each flickering cycle, which represents the interval between a coding value. The time coding module uses a high-precision switch circuit to control the flickering of the light-emitting device of the lamp module. The function of the light brightness changing with time in one cycle is known or can be calibrated, such as sinusoidal wave change, pulse change, etc. The specific implementation method of the specific coding form is shown in Figure 3 ;

[0058] like Figure 3 As shown, multiple code values ​​are used to sequentially encode the time information to transmit, for example: the synchronization progress bit is 1ms, and the transmitted time information includes seconds and milliseconds, and the format is S2S1, M3M2M1, S2, S1, M3, M2, and M1 are all decimal numbers, where S1 is the units digit of the second value, S2 is the tens digit of the second value, M1 is the units digit of the millisecond value, M2 is the tens digit of the millisecond value, and M3 is the hundreds digit of the millisecond value. Then, the lights flash M1 times, M2 times, M3 times, S1 times, and S2 times in sequence to complete the time information display. Let S2S1, M3M2M1 be the moment when the last light flash corresponding to the last code is completed. The total duration of the light flashing can be calculated as:

[0059] T sum =T*(S2+S1+M3+M2+M1)+4*ΔT

[0060]

[0061] judge Is it greater than If it is greater than, the time coding module controls the lighting module to The lights start flashing at this moment; if it is not greater than, then S2S1, M3M2M1 moments are too small, set a larger value and repeat the above calculation and judgment process.

[0062] Multiple lamps can also be used to represent different bit values ​​for encoding to improve the efficiency of encoding. For example, 5 lamps are used to encode the values ​​of S2, S1, M3, M2, and M1 respectively, and the total length of the flashing time of the light is:

[0063] T sum =T*Max(S2, S1, M3, M2, M1)

[0064]

[0065] Use the same method as above to determine the time to start flashing lights

[0066] Step 8: After the time coding module of the drone host controls the lamp to complete the flashing set in step 7, the drone slave camera ends the video image shooting;

[0067] Step 9: The time code processing module of the drone slave processes the video image and decodes the time information:

[0068] The time code processing module receives each frame of image captured by the camera in step 5. The exposure time of each frame of image is based on the external triggering time in step 4. As a benchmark, the time gradually accumulated according to the external trigger signal cycle, such as Figure 4 As shown by the black dot, Where n represents the nth frame image, ΔT c is the external trigger signal period;

[0069] Light feature extraction for each frame of image can be achieved by using light source detection algorithms, such as threshold method, edge extraction and shape recognition, etc. Traditional computer vision algorithms obtain the center position of each light source in the image and record the grayscale value of the center position to represent the light brightness value b n , you can also use deep learning CNN target detection network to directly detect the light source and its center position, such as YOLO series network, CenterNet and other target detection networks;

[0070] Filter the values ​​whose brightness is greater than 0, which represents the moment when the light is on, such as Figure 4 middle This group of 6 points is grouped according to whether the time interval is greater than T, and the brightness values ​​of each group corresponding to each coding value are obtained; the multiple brightness values ​​b in each group of data are n and its corresponding exposure time According to the UAV host light brightness change function in step 7, curve fitting is performed, such as Figure 4 The sine curve in is fitted to obtain the brightness curve of each group of signals; the time corresponding to the maximum value of each group of curves is calculated, and the maximum values ​​can be divided into multiple groups according to whether the difference between the maximum value moments is greater than ΔT. The number of maximum values ​​in each group is the number of flashes corresponding to each coding value, such as Figure 4 middle As a group, it corresponds to the M1-bit value in step seven. It has 7 maximum values, indicating 7 flashes, and the code value is 7, that is, M1=7; the last code is S2=3, and the code values ​​of S1, M3, and M2 can be obtained in the same way. The system time S2S1, M3M2M1 sent by the drone host can be obtained by decoding;

[0071] If the drone host uses multiple lamps to represent different bit values ​​for encoding, then multiple light sources and center positions need to be extracted and detected when processing each frame of the image. Each lamp is processed separately according to the above method to obtain the encoding value of each lamp and the system time sent by the drone host. This will not be repeated.

[0072] Step 10: The time code processing module of the drone slave calculates the deviation between its own system time in the image and the host system time obtained by decoding;

[0073] Since the camera captures video images in a discrete sampling process, the exposure time of the last frame of the flashing cycle will deviate from the time value of the last moment of the flashing cycle of the drone host light in the slave system time. Therefore, according to the last set of fitting curves in step nine, the time when the brightness is maximum is calculated. For example Figure 4 middle Based on this value, the time value of the last moment of the light flashing cycle in the slave system time can be calculated

[0074] Therefore, the deviation between the slave system time and the host system time is

[0075] Step 11: The slave machine corrects its own system time according to the time deviation. The corrected time

[0076] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and range of equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0077] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A time synchronization device between unmanned aerial vehicles based on light flashing coding, characterized in that: It includes: A light flashing coding unit, which is installed on the drone host; and a camera and processing unit, which is mounted on each slave drone; The host drone controls the lights to flash at the corresponding time synchronously and in real time according to its system time and the light flashing coding rules set by the light flashing coding unit; other slave drones use the camera and processing unit to continuously shoot the flashing lights of the host drone, and process the captured video images to obtain the decoded time information. After comparing it with the time of their own system, they correct the deviation between their system time and the system time of the host drone to complete the time synchronization between drones.

2. The device for time synchronization between unmanned aerial vehicles based on light flashing coding according to claim 1, characterized in that: The light flashing coding unit is composed of a time coding module and a lamp module. The time coding module controls the lamp module to flash at an actual moment matching the time information to be reflected by the light flashing according to the system time of the drone host and the set light flashing coding method, thereby realizing a light flashing time coding signal synchronized with the host system time.

3. The device for time synchronization between unmanned aerial vehicles based on light flashing coding according to claim 2, characterized in that: The video and processing unit is composed of a camera and a time code processing module. The camera is used to continuously shoot the flashing signal of the lighting module of the drone host to form a sequence of video images, and record the exposure time of each frame of the image based on its own system time. The time code processing module processes the video image shot by the camera, decodes the time information represented by the flashing of the light in the image, and compares it with the image exposure time to obtain the deviation between the drone slave system time and the host system time, and corrects the slave system time according to the deviation.

4. The device for time synchronization between unmanned aerial vehicles based on light flashing coding according to claim 3 is characterized in that: The drone host and the drone slave are combined into a flight platform using existing drones. The drone itself has a time system and obtains a relatively stable system time when working.

5. The device for time synchronization between unmanned aerial vehicles based on light flashing coding according to claim 3 is characterized in that: The time coding module of the drone host accurately and with low latency obtains the current time of the drone system, with an accuracy one order of magnitude higher than the time synchronization accuracy. A hardware-level high-precision timer is used to directly access registers from the FPGA in kernel mode to obtain high-precision time information from the hardware. The time code processing module of the drone slave uses a hardware-level high-precision timer to directly access registers from the FPGA in kernel mode to obtain high-precision time information from the hardware.

6. The device for time synchronization between unmanned aerial vehicles based on light flashing coding according to claim 3, characterized in that: The light flashing coding method of the light flashing coding unit adopts the number of light flashes for coding, and the time period of each flashing is fixed. The time coding module adopts a high-precision switching circuit to control the flashing of the light emitting device of the lamp module. The lamp module includes a light emitting device flashing multiple times for coding or includes multiple independent light emitting devices flashing simultaneously, representing different bits. The flashing start time is set according to all the time required for a complete flashing of the coded time information, and the flashing starts at that time.

7. The device for time synchronization between unmanned aerial vehicles based on light flashing coding according to claim 3, characterized in that: The camera has a high acquisition frame rate, which is 10 to 20 times higher than the light flickering frequency, and the camera exposure is controlled by an external trigger signal generated by a time code processing module, accurately matching the system time of the drone slave, and the trigger signal uses a TTL level or a differential signal.

8. The device for time synchronization between unmanned aerial vehicles based on light flashing coding according to claim 3 is characterized in that: The time code processing module realizes light feature extraction and time encoding and decoding processing, and performs time synchronization correction after calculating the time deviation, and is implemented using a mature SoC chip or FPGA chip.

9. A method for time synchronization between unmanned aerial vehicles based on light flashing, which adopts the device for time synchronization between unmanned aerial vehicles based on light flashing coding as claimed in any one of claims 1 to 8, characterized in that: It includes the following steps: Step 1: Initialize the system time of each drone; Step 2: The drone starts to perform the mission; Step 3: Determine whether to start time synchronization. If synchronization is started, go to step 4; otherwise, go to step 2. Step 4: Obtain system time from the drone And synchronously generate a camera external trigger signal to trigger the camera to shoot; Step 5: The camera of the drone slave is aimed at the direction of the drone host lighting module, and the video images are continuously captured according to the external trigger signal, and the images are transmitted to the time code processing module; Step 6: The time encoding module of the drone host obtains the system time Step 7: The time coding module of the drone host calculates the time when the lamp starts flashing arrive At this moment, the lamp module starts to flash according to the coding method; Step 8: After the time coding module of the drone host controls the lamp to complete the flashing set in step 7, the drone slave camera ends the video image shooting; Step 9: The time code processing module of the drone slave processes the video image and decodes the time information; Step 10: The time code processing module of the drone slave calculates the deviation between its own system time in the image and the host system time obtained by decoding; Step 11: The drone slave corrects its own system time according to the time deviation. The corrected time

Images