Android device camera and radar data time synchronization method

The time stamp of radar data frames is calculated and updated through software methods, and combined with timely grabbing the camera preview screen, the problem of time-consuming synchronization between radar and camera data is solved, and the rapid timestamp synchronization and high-frequency photography is achieved, which improves the user experience.

CN119070938APending Publication Date: 2024-12-03SHANDONG SENTER ELECTRONICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310644942.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the time stamp synchronization process of radar and camera data takes a long time, especially on handheld scanning devices, resulting in poor user experience and Android device cameras cannot meet the requirements of high-frequency photography.

Method used

通过软件方法同步相机数据和雷达数据,计算网络时延和系统时间差,逐帧更新雷达数据帧的时间戳,使其对齐到安卓系统时间,并通过定时抓取相机预览画面获取相机照片,实现相机 Synchronize with the timestamp of the radar data frame.

Benefits of technology

The time stamp synchronization of camera photos and radar data frames without hardware synchronization is realized, which improves synchronization speed, meets the requirements of high-frequency photography, and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119070938A_ABST
    Figure CN119070938A_ABST
Patent Text Reader

Abstract

The invention discloses an Android device camera and radar data time synchronization method, and belongs to the field of laser radars. In order to solve the problem that two sets of timestamps of camera data and radar data are not related to each other, a mode of synchronizing the camera data and the radar data through a software method is provided, and the timestamps of camera photos and radar data frames can be synchronized as much as possible under the condition that hardware synchronization is not carried out. A data coloring algorithm is available, so that the data coloring algorithm can be used by a color fusion algorithm. Compared with a hardware synchronization mode, the method is short in time consumption and high in speed. And when the camera data is acquired, the camera data is acquired in a manner of capturing the preview picture from the camera instead of a conventional one-time complete photographing process of the camera, so that the problem that the photographing frequency cannot be met due to overlong time from sending of a photographing instruction to obtaining of image data from a photographing interface in normal one-time photographing is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of laser radar, and in particular to a method for synchronizing time between a camera of an Android device and radar data. Background Art

[0002] Some handheld scanning devices are equipped with both cameras and lidars. When the radar data needs to be colored, the hardware time synchronization method is generally used to synchronize the radar and camera time, so that the timestamps of the radar data and the camera data are synchronized. The camera data with the same timestamp uses a coloring algorithm to color the radar data.

[0003] Radar hardware synchronization can be achieved through PTP (Precision Time Synchronization Protocol), but in actual use, it is found that radar time synchronization takes a long time, and it is uncertain when it can be synchronized. When the radar is mounted on a battery-powered device, such as a handheld scanning device, in order to save power, the radar is usually not powered on when it is not collecting data. It is only temporarily powered on when it is really necessary to scan. The process from the power-on command to the radar startup is completed and data can be collected takes a long time. If the long time of PTP synchronization is added, the user experience is very poor.

[0004] In addition, radar data coloring requires regular photography, and the photos need to obtain camera data stably at a frequency of 10 to 50 Hz during radar data collection. Taking 10 Hz as an example, a photo needs to be taken every 100 ms. Android device camera photography is triggered by the Android system Camera architecture. During the photography process, there are also controls for other conditions such as focus and exposure. A normal photography session takes more than 100 ms from the issuance of the photography command to the acquisition of image data from the photography interface. Therefore, Android device cameras cannot meet the minimum requirements of the above application scenarios. Summary of the invention

[0005] In order to solve the defects of the prior art, the present invention provides a method for time synchronization of camera and radar data of an Android device, which synchronizes camera data and radar data through a software method, and synchronizes the timestamps of camera photos and radar data frames as much as possible without performing hardware synchronization.

[0006] The present invention provides the following technical solutions:

[0007] A method for synchronizing time between an Android device camera and radar data, the method comprising:

[0008] S1: The acquisition application of the Android system notifies the system to power on the LiDAR and establish a communication connection with the LiDAR, and starts the camera of the Android system at the same time;

[0009] S2: Calculate the network delay between the lidar and the acquisition application;

[0010] S3: The acquisition application receives radar data frames sent by the laser radar frame by frame, wherein the radar data frames carry radar system timestamps;

[0011] S4: Calculate the system time difference TimeOffset between the Android system and the radar system using the following formula;

[0012] TimeOffset=T_sys_1-T_lidar_1-Delay

[0013] Among them, T_lidar_1 is the radar system timestamp of the first radar data frame, and T_sys_1 is the time of the Android system when the acquisition application receives the first radar data frame;

[0014] S5: For each radar data frame received, the radar system timestamp is updated as follows to synchronize with the Android system time;

[0015] T_lidar_n'=T_lidar_n+TimeOffset

[0016] Wherein, T_lidar_n is the radar system timestamp of the n-th radar data frame, and T_lidar_n' is the timestamp of the n-th radar data frame after synchronization alignment;

[0017] S6: starting from the time T_base, a preview image is captured from the camera once at every Ti time interval as a camera photo, and the Android system time when the preview image is captured is used as the timestamp of the camera photo;

[0018] T_base=T_sys_1-Delay+Ti

[0019] S7: Using or storing the radar data frame with the synchronized time stamp and the camera photo with the time stamp.

[0020] Furthermore, the S1 includes:

[0021] S11: The acquisition application of the Android system sends a command to the Android system to power on the LiDAR and simultaneously starts the camera of the Android system;

[0022] S12: After the laser radar is powered on successfully, it is initialized and broadcasts its unique broadcast code;

[0023] S13: After receiving the broadcast code, the acquisition application establishes a communication connection with the laser radar.

[0024] Further, the S2 includes:

[0025] S21: The acquisition application sends a control command to the laser radar and records the time T1 of the Android system when the control command is sent;

[0026] S22: After receiving the control command, the laser radar processes it and returns a command reply message to the acquisition application;

[0027] S23: Collect the application receiving the command reply message, and record the time T2 of the Android system when the command reply message is received;

[0028] S24: Calculate the network delay Delay using the following formula;

[0029] Delay=(T2-T1) / 2.

[0030] Further, the S3 includes:

[0031] S31: The acquisition application sends an acquisition instruction to the laser radar;

[0032] S32: After receiving the collection instruction, the laser radar starts to collect radar data, and sends the collected radar data to the collection application frame by frame in the form of radar data frames;

[0033] The radar data frame carries a radar system timestamp, and the radar system timestamp is the time of the radar system when the radar data frame is collected;

[0034] S33: The acquisition application receives the radar data frame by frame.

[0035] Further, the S7 includes:

[0036] S71: discarding the radar data frames whose timestamps after synchronization alignment are before the time T_base;

[0037] S72: Use or store the remaining radar data frames with the synchronized time stamps and the camera photos with the time stamps.

[0038] Furthermore, the interval time Ti is determined according to the photo taking frequency required by the external algorithm to ensure that the timestamp error of the camera photo after synchronization alignment with the corresponding radar data frame is within a set range, so as to achieve a state in which the external algorithm can be used.

[0039] Furthermore, Ti=1000 / f (milliseconds), where f is the capture frequency of the camera preview image, and the value range of f is generally 10 to 50 Hz.

[0040] The present invention has the following beneficial effects:

[0041] In order to solve the problem that the two sets of timestamps of camera data and radar data are unrelated, the present invention proposes a method of synchronizing camera data and radar data by software method, which can synchronize the timestamps of camera photos and radar data frames as much as possible without hardware synchronization, so as to achieve a state that the data coloring algorithm is available, so that it can be used by the color fusion algorithm. Compared with the method of hardware synchronization, the present invention takes less time and has a higher speed. In addition, when acquiring camera data, it is not a conventional complete photographing process of the camera, but the camera data is acquired in the form of a preview screen captured by the camera, which solves the problem that the normal photographing time from the issuance of the photographing instruction to the acquisition of the image data from the photographing interface is too long to meet the photographing frequency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a method for synchronizing time between an Android device camera and radar data of the present invention;

[0043] Figure 2 This is a schematic diagram before the timestamps of the radar data frame and the camera photo are aligned;

[0044] Figure 3 This is a schematic diagram after the timestamps of the radar data frame and the camera photo are aligned. DETAILED DESCRIPTION

[0045] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0046] The embodiment of the present invention provides a method for synchronizing the time between the camera and radar data of an Android device, such as Figure 1 As shown, the method includes:

[0047] S1: The acquisition application of the Android system notifies the system to power on the LiDAR and establish a communication connection with the LiDAR, and starts the camera of the Android system at the same time.

[0048] One way to implement this step is to:

[0049] S11: After the acquisition application of the Android system is started, a command to power on the LiDAR is sent to the Android system, notifying the system to power on the LiDAR and starting the camera of the Android system at the same time.

[0050] The options for starting the Android system's camera also include timely startup. It takes a long time for the laser radar to be powered on and ready to collect data. In this process, the camera can be started.

[0051] S12: After the laser radar is powered on successfully, it is initialized according to historical configurations such as IP, scanning mode, echo mode, etc., and broadcasts its own unique broadcast code.

[0052] S13: After receiving the broadcast code, the acquisition application establishes a communication connection with the laser radar.

[0053] S2: Calculate the network delay between the lidar and the acquisition application.

[0054] After the communication connection is established, it is necessary to obtain the average network delay between the laser scanner and the Android device. One implementation method includes:

[0055] S21: Before initiating the acquisition instruction, the acquisition application sends a control instruction to the laser radar (such as obtaining the current scanning mode) and records the time T1 of the Android system when the control instruction is sent.

[0056] S22: After receiving the control command, the laser radar processes it and returns a command reply message to the acquisition application.

[0057] S23: Collect the command reply message received by the application, and record the time T2 of the Android system when the command reply message is received.

[0058] Since there is no time information in the data packet of the command reply message of the control command reply, the receiving time T2 of the command reply message is also obtained by the Android system.

[0059] S24: Calculate the network delay Delay using the following formula.

[0060] Delay=(T2-T1) / 2.

[0061] The calculated value is an approximate network delay.

[0062] S3: The acquisition application receives radar data frames sent by the lidar frame by frame. The radar data frames carry radar system timestamps.

[0063] One way to implement this step is to:

[0064] S31: After calculating the network delay, the acquisition application sends an acquisition instruction to the lidar.

[0065] S32: After receiving the collection instruction, the laser radar starts to collect radar data, and sends the collected radar data to the collection application frame by frame in the form of radar data frames.

[0066] The radar data frame carries a radar system timestamp, and the radar system timestamp is the time of the radar system when the radar data frame is collected.

[0067] S33: The acquisition application receives radar data frames frame by frame.

[0068] S4: Calculate the system time difference TimeOffset between the Android system and the radar system using the following formula.

[0069] TimeOffset=T_sys_1-T_lidar_1-Delay

[0070] Among them, T_lidar_1 is the radar system timestamp carried by the first radar data frame, and T_sys_1 is the time of the Android system when the acquisition application receives the first radar data frame.

[0071] The present invention takes the first frame of radar data frame as the reference frame for calculating the system time difference, takes out the timestamp T_lidar_1, and takes out the current system timestamp T_sys_1 of the Android system. Since the timestamp of the camera image can only take the time of the Android system, it is preferred to align the radar time to the time of the Android system and calculate the time difference TimeOffset between the two systems.

[0072] Current system time (T_sys_1) = radar data scanning time (T_lidar_1) + system time difference (TimeOffset) + network delay (Delay)

[0073] Therefore, the calculation formula of the system time difference is as follows:

[0074] TimeOffset=T_sys_1-T_lidar_1-Delay.

[0075] S5: For each radar data frame received, the radar system timestamp is updated as follows to synchronize with the Android system time.

[0076] T_lidar_n'=T_lidar_n+TimeOffset

[0077] Wherein, T_lidar_n is the radar system timestamp of the n-th radar data frame, and T_lidar_n' is the timestamp of the n-th radar data frame after synchronization alignment.

[0078] Without hardware synchronization, the time between the radar data frame and the camera data is the time of the two systems. Therefore, this solution preferably aligns the time of the radar data frame to the time of the host system (Android system). After the time is aligned, the camera and radar data can be synchronized. Specifically, after calculating the system time difference TimeOffset, all packets of radar data frames received by subsequent applications need to extract the timestamp in the packet and use TimeOffset for alignment processing. The formula for aligning the timestamp is as above.

[0079] S6: Starting from time T_base, a preview image is captured from the camera at intervals of Ti as a camera photo, and the Android system time when the preview image is captured is used as the timestamp of the camera photo.

[0080] T_base=T_sys_1-Delay+Ti.

[0081] The radar scan completes and obtains the first frame of data. There is a delay between the radar end sending it out and the acquisition application receiving it. The corresponding camera photo acquisition time should be the moment when the radar obtains the scan data, not the moment when the acquisition application receives the data. Therefore, in theory, the first photo of the Android system camera should be T_sys_1-Delay, but because the time has advanced to T_sys_1, the photo at T_sys_1-Delay is missing. After starting to capture photos, the preview screen is captured every Ti interval, so the capture time of the next photo is T_sys_1-Delay+Ti, and photos are captured every Ti interval thereafter. The captured camera photos are packaged with the timestamp for use.

[0082] For example, if an Android device wants to obtain photos at a speed of 10 photos per second through the camera architecture of the Android system, the present invention uses a method of regularly capturing preview images instead of the normal camera photo process. The method of regularly capturing preview images solves the problem that a normal photo shoot takes more than 100ms from the issuance of a photo command to the acquisition of image data from the photo interface, resulting in the Android device camera being unable to meet the minimum requirements of the above application scenarios.

[0083] The interval time Ti is determined according to the shooting frequency required by the external algorithm to ensure that the timestamp error of the camera photo and the corresponding radar data frame after synchronization is within the set range, that is, the error is as small as possible to achieve the state that the external algorithm can be used.

[0084] Exemplarily, Ti=1000 / f (milliseconds), where f is the capture frequency of the camera preview image, and the value range of f is generally 10 to 50 Hz.

[0085] For example, the camera interval time Ti is determined according to the number of photos N that the camera needs to acquire per second, and the calculation formula is: Ti = 1000 / N, the unit is ms. For example, if 10 photos are acquired per second, the interval time is 1000 / 10 = 100 ms.

[0086] S7: Using or storing the radar data frame with the synchronized time stamp and the camera photo with the time stamp.

[0087] The camera photos and radar data frames can be used after the timestamp synchronization and alignment. If the algorithm that needs to use radar data and camera images at the same time is real-time, the image data and time information need to be passed to the algorithm for use; if it is an offline algorithm, the captured camera photos can be stored in the system-specified directory. When storing, the timestamp must be stored together with the extended information.

[0088] Considering that the first camera photo is stored at T_base, the radar data frame also needs some processing, that is, the radar data before T_base after the aligned radar timestamp T_lidar_n' is directly discarded. Specifically, it includes:

[0089] S71: discarding radar data frames whose timestamps after synchronization alignment are before the time T_base.

[0090] S72: Use or store the remaining radar data frames with the synchronized time stamps and the camera photos with the time stamps.

[0091] In order to solve the problem that the two sets of timestamps of camera data and radar data are unrelated, the present invention proposes a method of synchronizing camera data and radar data by software method, which can synchronize the timestamps of camera photos and radar data frames as much as possible without hardware synchronization, so as to achieve a state that the data coloring algorithm is available, so that it can be used by the color fusion algorithm. Compared with the method of hardware synchronization, the present invention takes less time and has a higher speed. In addition, when acquiring camera data, it is not a conventional complete photographing process of the camera, but the camera data is acquired in the form of a preview screen captured by the camera, which solves the problem that the normal photographing time from the issuance of the photographing instruction to the acquisition of the image data from the photographing interface is too long to meet the photographing frequency.

[0092] like Figure 2As shown in the figure, before the timestamps of the camera photo and the radar data frame are aligned, the two sets of timestamps are unrelated. After alignment, the timestamps of the first camera photo and the first radar data frame (the first frame after discarding the radar data frame before T_base) are the same. The subsequent camera captures data regularly, and the radar data frames are aligned through TimeOffset according to their own time rules, as shown in the figure. Figure 3 shown.

[0093] The communication of the synchronization process of the present invention is within the local area network. The time delay within the local area network is generally no more than 10 milliseconds. The camera is calculated according to the minimum requirement of 10 frames per second. It can be known that Ti is a maximum of 100 milliseconds. Therefore, the overall time consumed by the above process will not exceed 150 milliseconds. When the radar is scanning data, the official recommendation is to keep still for a few seconds at the beginning of the scan. Within a few seconds, the above process can work normally, that is, the image and radar data are synchronized by software without affecting the actual user experience.

[0094] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for synchronizing time between an Android device camera and radar data, characterized in that: The method comprises: S1: The acquisition application of the Android system notifies the system to power on the LiDAR and establish a communication connection with the LiDAR, and starts the camera of the Android system at the same time; S2: Calculate the network delay between the lidar and the acquisition application; S3: The acquisition application receives radar data frames sent by the laser radar frame by frame, wherein the radar data frames carry radar system timestamps; S4: Calculate the system time difference TimeOffset between the Android system and the radar system using the following formula; TimeOffset=T_sys_1-T_lidar_1-Delay Among them, T_lidar_1 is the radar system timestamp of the first radar data frame, and T_sys_1 is the time of the Android system when the acquisition application receives the first radar data frame; S5: For each radar data frame received, the radar system timestamp is updated as follows to synchronize with the Android system time; T_lidar_n'=T_lidar_n+TimeOffset Wherein, T_lidar_n is the radar system timestamp of the n-th radar data frame, and T_lidar_n' is the timestamp of the n-th radar data frame after synchronization alignment; S6: starting from the time T_base, a preview image is captured from the camera at every Ti time interval as a camera photo, and the Android system time when the preview image is captured is used as the timestamp of the camera photo; T_base=T_sys_1-Delay+Ti S7: Using or storing the radar data frame with the synchronized time stamp and the camera photo with the time stamp.

2. The method for synchronizing the time of the camera and radar data of an Android device according to claim 1, characterized in that: The S1 includes: S11: The acquisition application of the Android system sends a command to the Android system to power on the LiDAR and simultaneously starts the camera of the Android system; S12: After the laser radar is powered on successfully, it is initialized and broadcasts its unique broadcast code; S13: After receiving the broadcast code, the acquisition application establishes a communication connection with the laser radar.

3. The method for synchronizing the time of the camera and radar data of an Android device according to claim 2, characterized in that: The S2 includes: S21: The acquisition application sends a control command to the laser radar and records the time T1 of the Android system when the control command is sent; S22: After receiving the control command, the laser radar processes it and returns a command reply message to the acquisition application; S23: Collect the application receiving the command reply message, and record the time T2 of the Android system when the command reply message is received; S24: Calculate the network delay Delay using the following formula; Delay=(T2-T1) / 2.

4. The method for synchronizing the time between the camera and radar data of an Android device according to claim 3, characterized in that: The S3 includes: S31: The acquisition application sends an acquisition instruction to the laser radar; S32: After receiving the collection instruction, the laser radar starts to collect radar data, and sends the collected radar data to the collection application frame by frame in the form of radar data frames; The radar data frame carries a radar system timestamp, and the radar system timestamp is the time of the radar system when the radar data frame is collected; S33: The acquisition application receives the radar data frame by frame.

5. The method for synchronizing the time between the camera and radar data of an Android device according to any one of claims 1 to 4, characterized in that: The S7 includes: S71: discarding the radar data frames whose timestamps after synchronization alignment are before the time T_base; S72: Use or store the remaining radar data frames with the synchronized time stamps and the camera photos with the time stamps.

6. The method for synchronizing time between an Android device camera and radar data according to claim 5, characterized in that: The interval time Ti is determined according to the shooting frequency required by the external algorithm to ensure that the timestamp error of the camera photo after synchronization and alignment with the corresponding radar data frame is within the set range, so that the external algorithm can be used.

7. The method for synchronizing time between an Android device camera and radar data according to claim 6, characterized in that: Ti=1000 / f, where f is the capture frequency of the camera preview image, and the value range of f is 10~50HZ.