Camera and laser sensor time synchronization method and device with real-time MEMS posture matching

Abstract

This application relates to a method and apparatus for time synchronization of cameras and laser sensors with real-time MEMS attitude matching. The method comprises: connecting a camera, a laser radar, and a MEMS device to form a multimodal visual perception system, acquiring MEMS accelerometer data from the laser radar and camera respectively, extracting features from the MEMS accelerometer data using a feature extraction algorithm, matching the features to achieve high-precision time alignment, and finally verifying the time alignment results based on the moment the device starts moving. This solves the problem that current consumer-grade cameras and laser radars lack advanced features such as external time trigger interfaces, making it difficult to achieve time synchronization with external devices, and resulting in poor multimodal data fusion accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of multi-sensor integration, and in particular to a method and device for time synchronization of a camera and a laser sensor with real-time MEMS posture matching. Background Art

[0002] The MEMS (Micro-Electro-Mechanical System) accelerometer is an advanced, highly integrated sensor technology specifically designed to measure the acceleration force of an object in a specific direction. The core of this sensor is its use of micro-electromechanical system technology to miniaturize traditional mechanical accelerometer components and integrate them onto a microelectronic chip. MEMS accelerometers contain extremely small masses and springs. When acceleration acts on the sensor, these microstructures will displace, and then the physical displacement will be converted into an electrical signal through capacitance changes or piezoelectric effects, achieving quantitative measurement of acceleration and angular acceleration. The advantages of MEMS accelerometers are their small size, light weight, and extremely low power consumption, while they can provide high accuracy and reliability. These characteristics make MEMS accelerometers an indispensable component in modern electronic and mobile devices.

[0003] The application of MEMS accelerometers is also crucial in devices such as consumer-grade sports cameras and lidar, which are mainly aimed at the general consumer market and emphasize cost-effectiveness and user-friendliness.

[0004] However, because MEMS accelerometers typically lack advanced features such as external time trigger interfaces, it is difficult to achieve time synchronization with external devices in scenarios such as fusing visual information from a camera with distance and orientation information from a lidar. This can lead to inaccurate or misinterpreted data. For example, in autonomous vehicles, unsynchronized data can lead to erroneous obstacle detection or distance estimation.

[0005] In summary, the existing application methods of MEMS accelerometers lack the limitation of external time trigger interface, making it difficult to achieve effective time synchronization between camera and lidar, which greatly affects the quality of data fusion and needs to be solved urgently. Summary of the Invention

[0006] This application provides a method and device for time synchronization of cameras and laser sensors with real-time MEMS attitude matching to address the problems that current consumer-grade cameras and lidars lack advanced features such as external time trigger interfaces, making it difficult to achieve time synchronization with external devices, resulting in poor multimodal data fusion accuracy.

[0007] A first aspect of the present application provides a method for time synchronization of a camera and a laser sensor with real-time MEMS posture matching, comprising the following steps: constructing a multimodal visual perception system based on a preset camera, a laser radar, and a MEMS device, and collecting visual perception data corresponding to the multimodal visual perception system, wherein the visual perception data includes camera data, camera MEMS data, laser radar data, and laser radar MEMS data; respectively obtaining MEMS acceleration time series corresponding to the camera MEMS data and the laser radar MEMS data, extracting feature information corresponding to the MEMS acceleration time series, and matching the MEMS acceleration time series according to the feature information to obtain MEMS matching data; respectively determining the start motion time corresponding to the camera and the laser radar, and calculating the motion time difference between the camera and the laser radar according to the start motion time, and comparing the motion time difference with the MEMS matching data to obtain a time alignment index corresponding to the camera and the laser radar, and judging whether the time alignment index meets a preset time synchronization condition, wherein if the time alignment index meets the time synchronization condition, it is determined that the camera and the laser radar are time synchronized.

[0008] Optionally, in one embodiment of the present application, the multimodal visual perception system is constructed based on a preset camera, lidar and MEMS device, and visual perception data corresponding to the multimodal visual perception system is collected, including: constructing a camera MEMS architecture based on the camera and the MEMS device, and constructing a lidar MEMS architecture through the lidar and the MEMS device; fixing the camera MEMS architecture and the lidar MEMS architecture according to a preset fixed connection relationship to construct the multimodal visual perception system; sequentially starting the lidar data acquisition function and the camera data acquisition function of the multimodal visual perception system, and leaving the lidar and the camera target for a certain period of time, The laser radar data and the laser radar MEMS data are synchronously collected through the laser radar MEMS architecture, and the camera data and the camera MEMS data are synchronously collected using the camera MEMS architecture; whether the laser radar MEMS architecture and the camera MEMS architecture have completed the data collection operation is determined respectively; if both the laser radar MEMS architecture and the camera MEMS architecture have completed the data collection operation, the laser radar and the camera are left stationary for the target time, and after the target time, the laser radar data collection function and the camera data collection function are stopped in turn, otherwise the data collection operation of the laser radar MEMS architecture and / or the camera MEMS architecture is continued.

[0009] Optionally, in one embodiment of the present application, the MEMS acceleration time series corresponding to the camera MEMS data and the lidar MEMS data are obtained respectively, and the feature information corresponding to the MEMS acceleration time series is extracted, and the MEMS acceleration time series is matched according to the feature information to obtain MEMS matching data, including: saving the camera MEMS data and the lidar MEMS data as MEMS acceleration time series in the target basic data cell format, wherein the MEMS acceleration time series includes a timestamp, yaw angular acceleration, roll angular acceleration and pitch angular acceleration; obtaining the fixed connection The method comprises the following steps: obtaining a rotation matrix corresponding to the connection relationship, and rotating the yaw angular acceleration, the roll angular acceleration and the pitch angular acceleration of the laser radar according to the rotation matrix, so that the Z axis of the coordinate system of the preset laser radar MEMS accelerometer is parallel to the Z axis of the coordinate system of the preset machine MEMS accelerometer; determining the time threshold corresponding to the MEMS acceleration information based on the yaw angular acceleration, and performing feature extraction on the MEMS acceleration time series according to the time threshold to obtain a peak sequence corresponding to the MEMS acceleration time series; matching the MEMS acceleration time series through the peak sequence to generate the MEMS matching data.

[0010] Optionally, in one embodiment of the present application, respectively determining the start movement moments corresponding to the camera and the lidar, and calculating the movement time difference between the camera and the lidar based on the start movement moments, includes: respectively obtaining the device still time before acquisition and the device still time after acquisition of the lidar and the camera, and respectively determining the start movement moment and the end movement moment corresponding to the lidar and the camera based on the device still time before acquisition and the device still time after acquisition; and calculating the movement time difference between the lidar and the camera based on the start movement moment and the end movement moment.

[0011] Optionally, in one embodiment of the present application, the mathematical expression of the time synchronization condition is:

[0012] |t lstart -t cstart -dt lj |<threshold

[0013] |t lend -t cend -dt lj |<threshold

[0014] Among them, t lstart represents the start time of the laser radar movement; t cstartIndicates the start time of the camera movement; t lend represents the end time of the laser radar movement; t cend Indicates the end motion moment of the camera; dt lj represents the MEMS matching data; threshold represents the preset time synchronization threshold.

[0015] According to a second aspect of the present application, an embodiment provides a camera and laser sensor time synchronization device with real-time MEMS posture matching, comprising: a construction module for constructing a multimodal visual perception system based on a preset camera, a laser radar, and a MEMS device, and collecting visual perception data corresponding to the multimodal visual perception system, wherein the visual perception data includes camera data, camera MEMS data, laser radar data, and laser radar MEMS data; a matching module for respectively obtaining MEMS acceleration time series corresponding to the camera MEMS data and the laser radar MEMS data, extracting feature information corresponding to the MEMS acceleration time series, and matching the MEMS acceleration time series according to the feature information to obtain MEMS matching data; a synchronization module for respectively determining the start motion time corresponding to the camera and the laser radar, and calculating the motion time difference between the camera and the laser radar according to the start motion time, and comparing the motion time difference with the MEMS matching data to obtain a time alignment index corresponding to the camera and the laser radar, and judging whether the time alignment index meets a preset time synchronization condition, wherein if the time alignment index meets the time synchronization condition, it is determined that the camera and the laser radar are time synchronized.

[0016] Optionally, in one embodiment of the present application, the construction module includes: an establishment unit for constructing a camera MEMS architecture based on the camera and the MEMS device, and constructing a lidar MEMS architecture through the lidar and the MEMS device; a fixing unit for fixing the camera MEMS architecture and the lidar MEMS architecture according to a preset fixed connection relationship to construct the multimodal visual perception system; a starting unit for sequentially starting the lidar data acquisition function and the camera data acquisition function of the multimodal visual perception system, and after the lidar and the camera are left stationary for a certain period of time, synchronously collecting the laser data through the lidar MEMS architecture. Radar data and the lidar MEMS data, and synchronously collect the camera data and the camera MEMS data using the camera MEMS architecture; a judgment unit, used to respectively judge whether the lidar MEMS architecture and the camera MEMS architecture have completed the data collection operation; a static unit, used to if both the lidar MEMS architecture and the camera MEMS architecture have completed the data collection operation, then static the lidar and the camera for the target time length, and after the target time length, sequentially stop the lidar data collection function and the camera data collection function, otherwise continue to execute the data collection operation of the lidar MEMS architecture and / or the camera MEMS architecture.

[0017] Optionally, in one embodiment of the present application, the matching module includes: a saving unit for saving the camera MEMS data and the lidar MEMS data as MEMS acceleration time series in the target basic data cell format, respectively, wherein the MEMS acceleration time series includes a timestamp, yaw angular acceleration, roll angular acceleration and pitch angular acceleration; a rotation unit for obtaining the rotation matrix corresponding to the fixed connection relationship, and rotating the yaw angular acceleration, roll angular acceleration and pitch angular acceleration of the lidar according to the rotation matrix so that the preset lidar MEMS accelerometer coordinate system Z axis is parallel to the preset machine MEMS accelerometer coordinate system Z axis; a determination unit for determining the time threshold corresponding to the MEMS acceleration information based on the yaw angular acceleration, and performing feature extraction on the MEMS acceleration time series according to the time threshold to obtain a peak sequence corresponding to the MEMS acceleration time series; a generation unit for matching the MEMS acceleration time series through the peak sequence to generate the MEMS matching data.

[0018] Optionally, in one embodiment of the present application, the synchronization module includes: an acquisition unit, used to respectively acquire the device still time before acquisition and the device still time after acquisition of the laser radar and the camera, and determine the start movement moment and end movement moment corresponding to the laser radar and the camera according to the device still time before acquisition and the device still time after acquisition; a calculation unit, used to calculate the movement time difference between the laser radar and the camera based on the start movement moment and the end movement moment.

[0019] Optionally, in one embodiment of the present application, the mathematical expression of the time synchronization condition is:

[0020] |t lstart -t cstart -dt lj |<threshold

[0021] |t lend -t cend -dt lj |<threshold

[0022] Among them, t lstart represents the start time of the laser radar movement; t cstart Indicates the start time of the camera movement; t lend represents the end time of the laser radar movement; t cend Indicates the end motion moment of the camera; dt lj represents the MEMS matching data; threshold represents the preset time synchronization threshold.

[0023] An embodiment of the third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the camera and laser sensor time synchronization method for real-time MEMS posture matching as described in the above embodiment.

[0024] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned real-time MEMS posture matching camera and laser sensor time synchronization method.

[0025] The fifth aspect of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned real-time MEMS posture matching camera and laser sensor time synchronization method.

[0026] Therefore, the embodiments of the present application have the following beneficial effects:

[0027] The embodiments of the present application can construct a multimodal visual perception system based on a preset camera, lidar and MEMS device, and collect visual perception data corresponding to the multimodal visual perception system, wherein the visual perception data includes camera data, camera MEMS data, lidar data and lidar MEMS data; obtain the MEMS acceleration time series corresponding to the camera MEMS data and the lidar MEMS data respectively, extract the feature information corresponding to the MEMS acceleration time series, and match the MEMS acceleration time series according to the feature information to obtain MEMS matching data; determine the start motion time corresponding to the camera and lidar respectively, and calculate the motion time difference between the camera and lidar according to the start motion time, and compare the motion time difference with the MEMS matching data to obtain the time alignment index corresponding to the camera and lidar, and determine whether the time alignment index meets the preset time synchronization condition, wherein if the time alignment index meets the time synchronization condition, it is determined that the camera and lidar are time synchronized. The present application can quickly complete high-precision time alignment of consumer-grade devices by matching the MEMS accelerometer data fixed to the device without relying on the time synchronization interface, thereby significantly improving the accuracy and reliability of data fusion. This solves the problem that current consumer-grade cameras and lidars lack advanced features such as external time trigger interfaces, making it difficult to achieve time synchronization with external devices, resulting in poor multimodal data fusion accuracy.

[0028] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A flowchart of a method for time synchronization of a camera and a laser sensor with real-time MEMS posture matching according to an embodiment of the present application is provided;

[0031] Figure 2 A schematic diagram of a laser radar and a panoramic camera provided for one embodiment of the present application;

[0032] Figure 3 A schematic diagram of a device coordinate system for a laser radar and a panoramic camera provided in one embodiment of the present application;

[0033] Figure 4 This is an example diagram of a camera and laser sensor time synchronization device for real-time MEMS posture matching according to an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0035] Among them, 10-real-time MEMS posture matching camera and laser sensor time synchronization device, 100-building module, 200-matching module, 300-synchronization module, 501-memory, 502-processor, 503-communication interface. DETAILED DESCRIPTION

[0036] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes a method and apparatus for synchronizing a camera and a laser sensor with real-time MEMS attitude matching according to an embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a method for synchronizing a camera and a laser sensor with real-time MEMS attitude matching, wherein a multimodal visual perception system is constructed based on a preset camera, a laser radar, and a MEMS device, and visual perception data corresponding to the multimodal visual perception system is collected, wherein the visual perception data includes camera data, camera MEMS data, laser radar data, and laser radar MEMS data; MEMS acceleration time series corresponding to the camera MEMS data and the laser radar MEMS data are obtained respectively, and feature information corresponding to the MEMS acceleration time series is extracted, and the MEMS acceleration time series is matched according to the feature information to obtain MEMS matching data; the start motion time corresponding to the camera and the laser radar is determined respectively, and the motion time difference between the camera and the laser radar is calculated based on the start motion time, and the motion time difference is compared with the MEMS matching data to obtain a time alignment index corresponding to the camera and the laser radar, and it is determined whether the time alignment index meets a preset time synchronization condition, wherein if the time alignment index meets the time synchronization condition, it is determined that the camera and the laser radar are time synchronized. By matching data from MEMS accelerometers connected to the device, this application can quickly achieve high-precision time alignment for consumer-grade devices without relying on a time synchronization interface, significantly improving the accuracy and reliability of data fusion. This solves the problem that current consumer-grade cameras and lidars lack advanced features such as external time trigger interfaces, making it difficult to achieve time synchronization with external devices, resulting in poor multimodal data fusion accuracy.

[0038] Specifically, Figure 1 This is a flowchart of a method for time synchronization of a camera and a laser sensor with real-time MEMS posture matching provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the camera and laser sensor time synchronization method for real-time MEMS posture matching includes the following steps:

[0040] In step S101, a multimodal visual perception system is constructed based on preset cameras, lidars and MEMS devices, and visual perception data corresponding to the multimodal visual perception system is collected, wherein the visual perception data includes camera data, camera MEMS data, lidar data and lidar MEMS data.

[0041] The embodiments of the present application can first connect the camera, lidar and MEMS device to form a multimodal visual perception system, and use the multimodal visual perception system to collect data to collect visual perception data. After the start and end of data collection, they are left stationary for more than 30 seconds respectively. After the collection is completed, the embodiments of the present application can select an axis with obvious angle changes during the collection process to obtain the MEMS accelerometer data (i.e., camera MEMS data and lidar MEMS data) in the visual perception data of the lidar and camera under the axis respectively.

[0042] Optionally, in one embodiment of the present application, a multimodal visual perception system is constructed based on a preset camera, a laser radar and a MEMS device, and visual perception data corresponding to the multimodal visual perception system is collected, including: constructing a camera MEMS architecture based on the camera and the MEMS device, and constructing a laser radar MEMS architecture through the laser radar and the MEMS device; fixing the camera MEMS architecture and the laser radar MEMS architecture according to a preset fixed connection relationship to construct a multimodal visual perception system; sequentially starting the laser radar data acquisition function and the camera data acquisition function of the multimodal visual perception system, and leaving the laser radar and the camera stationary. After the target time, the lidar data and the lidar MEMS data are synchronously collected through the lidar MEMS architecture, and the camera data and the camera MEMS data are synchronously collected using the camera MEMS architecture; it is determined whether the lidar MEMS architecture and the camera MEMS architecture have completed the data collection operation respectively; if both the lidar MEMS architecture and the camera MEMS architecture have completed the data collection operation, the lidar and the camera are kept stationary for the target time, and after the target time, the lidar data collection function and the camera data collection function are stopped in turn, otherwise the data collection operation of the lidar MEMS architecture and / or the camera MEMS architecture is continued.

[0043] In an embodiment of the present application, a camera can be fixedly connected to a low-cost MEMS to form a camera MEMS module (i.e., a camera MEMS architecture). Similarly, a laser radar can be fixedly connected to a low-cost MEMS to construct a laser radar MEMS module (i.e., a laser radar MEMS architecture). Thereafter, the embodiment of the present application can fixedly connect the camera MEMS module and the laser radar MEMS module according to a preset fixed connection relationship to construct a multimodal visual perception system.

[0044] As a feasible method, the embodiment of the present application can connect and fix the panoramic camera with built-in MEMS accelerometer and the laser radar together in a precise and stable manner, such as Figure 2 As shown in the figure, these sensors simultaneously collect data on their own status and surrounding space environment, and transmit the data to the built-in industrial computer for data processing and storage, so as to achieve all-round perception of indoor three-dimensional space information.

[0045] In the actual implementation process, the embodiments of the present application may use an industrial computer with a built-in 2.2GHz quad-core ARM Cortex-A73 and a 2.0GHz quad-core ARM Cortex-A53 CPU and 8GB of memory, and each sensor may use an existing product, as shown below:

[0046] Livox Mid-360 LiDAR with built-in MEMS

[0047] Insta360 OneRS 1-inch panoramic camera with built-in MEMS

[0048] In the specific implementation process, the embodiment of the present application can connect the laser radar to the industrial computer for data transmission and control, and the panoramic camera can be controlled independently; it should be noted that the embodiment of the present application can take the coordinate system of the panoramic camera as the carrier coordinate system of the multimodal visual perception system, such as Figure 3 As shown, the front direction of the multimodal visual perception system is the x-axis direction, the plane where the multimodal visual perception system is located is the xoy plane, and the xyz right-handed coordinate system is constructed. The lidar is installed below the panoramic camera and rotated 15° clockwise around the y-axis.

[0049] Afterwards, the embodiments of the present application can sequentially start the lidar data acquisition function and the camera data acquisition function of the multimodal visual perception system, and establish a unified time system respectively; secondly, after all the data acquisition functions are started, the devices (i.e., camera and lidar) are respectively left stationary for a target period of time (such as 30 seconds), and then data is collected according to the plan; secondly, the embodiments of the present application can respectively synchronously collect camera, lidar and their corresponding MEMS data through the camera MEMS architecture and the lidar MEMS architecture.

[0050] Determine whether the LiDAR MEMS architecture and the camera MEMS architecture have completed data acquisition operations. After the acquisition is completed, leave the device still for 30 seconds. When the acquisition is completed, stop the camera data acquisition first, and then stop the LiDAR data acquisition.

[0051] If one or both of the LiDAR and the camera have not completed the data collection, the data collection operation of the LiDAR and the camera will continue.

[0052] Finally, the embodiments of the present application can read the MEMS data recorded by the lidar and panoramic camera (i.e., camera MEMS data and lidar MEMS data) through an industrial computer and related programs.

[0053] Therefore, the embodiments of the present application connect cameras, lidars and MEMS devices and perform multimodal visual perception system data acquisition, thereby providing reliable data guidance and basis for the subsequent implementation of high-precision time synchronization.

[0054] In step S102, the MEMS acceleration time series corresponding to the camera MEMS data and the lidar MEMS data are respectively obtained, and the feature information corresponding to the MEMS acceleration time series is extracted, and the MEMS acceleration time series is matched according to the feature information to obtain MEMS matching data.

[0055] Furthermore, the embodiments of the present application also need to obtain the MEMS data of the lidar and camera respectively, select the axis with obvious angle change, extract the peak value of its MEMS acceleration information as a feature, and match the two sets of acceleration time series to obtain MEMS matching data.

[0056] Optionally, in one embodiment of the present application, MEMS acceleration time series corresponding to camera MEMS data and lidar MEMS data are obtained respectively, and feature information corresponding to the MEMS acceleration time series is extracted, and the MEMS acceleration time series is matched according to the feature information to obtain MEMS matching data, including: saving the camera MEMS data and lidar MEMS data as MEMS acceleration time series in the target basic data cell format, wherein the MEMS acceleration time series includes a timestamp, yaw angle angular acceleration, roll angle angular acceleration and pitch angle angular acceleration; obtaining A rotation matrix corresponding to the fixed connection relationship is used, and the yaw angular acceleration, roll angular acceleration, and pitch angular acceleration of the laser radar are rotated according to the rotation matrix so that the Z axis of the coordinate system of the preset laser radar MEMS accelerometer is parallel to the Z axis of the coordinate system of the preset machine MEMS accelerometer; based on the yaw angular acceleration, the time threshold corresponding to the MEMS acceleration information is determined, and the MEMS acceleration time series is feature extracted according to the time threshold to obtain a peak sequence corresponding to the MEMS acceleration time series; the MEMS acceleration time series is matched through the peak sequence to generate MEMS matching data.

[0057] In the embodiment of the present application, the specific steps of preprocessing and accurately matching the camera MEMS data and the lidar MEMS data are as follows:

[0058] Step 1: Extract the data stored in the MEMS accelerometers of the laser radar and panoramic camera (i.e., camera MEMS data and laser radar MEMS data), and save them according to the timestamp, yaw angle (Yaw) angular acceleration, roll angle (Roll) angular acceleration, and pitch angle (Pitch) angular acceleration as basic data units, denoted as I l =(t l , yaw l , roll l , pitch l );

[0059] Step 2: The rotation matrix R corresponding to the fixed connection relationship between the lidar and the panoramic camera lc , rotate the angular velocity vector of the laser radar according to formula (1) so that the three axes of the coordinate system of the laser radar MEMS accelerometer are parallel to the three corresponding axes of the coordinate system of the panoramic camera MEMS accelerometer;

[0060]

[0061] Step 3: Select the angular velocity of the yaw angle, set the time threshold t, perform feature extraction on the MEMS acceleration time series, and extract the peak sequence I of each set of data. l=[I l1 , I l2 , I l3 ,…,I ln ], each of which I li (i∈{1,2,3,…,n}), its yaw li are all local maxima within time t''

[0062] Step 4: Extract the peak sequences of the lidar MEMS and panoramic camera MEMS respectively, and record them as I ll and I lc , and calculate dt according to formula 2 l Sequence, dt l The length of the sequence should be the same as I ll consistent.

[0063] dt l =[t ll0 -t lc0 , t ll1 -t lc0 , t ll2 -t lc0 ,…,t lln -t lc0 ] (2)

[0064] Step 5: I lc Each item in t lci Add the same dt lj , and in I ll Find the corresponding item in lc Each term in the above is added with dt lj After that, you can lc If the corresponding item is found, it is considered that dt lj is the time difference between the lidar system and the panoramic camera system to generate MEMS matching data.

[0065] Therefore, the embodiments of the present application effectively ensure the execution and implementation of subsequent time alignment verification operations using static data by accurately matching the dual MEMS accelerometers that match the camera MEMS data and the lidar MEMS data.

[0066] In step S103, the start movement moments corresponding to the camera and the lidar are determined respectively, and the movement time difference between the camera and the lidar is calculated based on the start movement moment, and the movement time difference is compared with the MEMS matching data to obtain the time alignment indicators corresponding to the camera and the lidar, and determine whether the time alignment indicators meet the preset time synchronization conditions. If the time alignment indicators meet the time synchronization conditions, it is determined that the camera and the lidar are time synchronized.

[0067] Furthermore, the embodiments of the present application also need to extract the static point data of the laser radar and the camera, and detect the moment when they start moving based on the posture of the laser radar and the camera, and compare the time difference between the start of movement of the camera and the laser radar with the MEMS matching data to verify whether the time alignment result (i.e., the time alignment index) meets the preset time synchronization condition, and when the time alignment index meets the time synchronization condition, it is determined that the camera and the laser radar have achieved time synchronization.

[0068] Therefore, the embodiments of the present application synchronously integrate the panoramic camera and lidar integrated with MEMS accelerometers, match the data recorded by the MEMS accelerometers of the panoramic camera and lidar during data acquisition, and thus achieve time synchronization of the panoramic camera and lidar without an external time trigger interface.

[0069] Optionally, in one embodiment of the present application, the start movement moments corresponding to the camera and the lidar are determined respectively, and the movement time difference between the camera and the lidar is calculated based on the start movement moments, including: obtaining the device still time before acquisition and the device still time after acquisition of the lidar and the camera respectively, and determining the start movement moment and the end movement moment corresponding to the lidar and the camera respectively based on the device still time before acquisition and the device still time after acquisition; and calculating the movement time difference between the lidar and the camera based on the start movement moment and the end movement moment.

[0070] It should be noted that the embodiment of the present application can extract the start and end time points t of the laser radar and panoramic camera according to the device static time before the start and end of the acquisition. lstart , t lend , t cstart and t cend The motion time difference between the lidar and the camera is calculated by the start and end time of the motion, so as to determine whether the camera and the lidar have achieved effective time synchronization based on the motion time difference.

[0071] Therefore, the embodiments of the present application combine the device of the fixed MEMS with the time series feature extraction algorithm to realize the time synchronization of the panoramic camera and the lidar in the absence of an external time trigger interface, so as to achieve the purpose of effective time synchronization between the camera and the lidar in the absence of an external time trigger interface, thereby ensuring the accuracy and reliability of data fusion.

[0072] Optionally, in one embodiment of the present application, the mathematical expression of the time synchronization condition is:

[0073] |t lstart -t cstart -dt lj |<threshold

[0074] |t lend -t cend -dt lj |<threshold

[0075] Among them, t lstart Indicates the moment when the laser radar starts moving; t cstart Indicates the moment when the camera starts moving; t lend Indicates the end time of the laser radar movement; t cend Indicates the end time of the camera movement; dt lj Indicates MEMS matching data; threshold indicates the preset time synchronization threshold.

[0076] In an embodiment of the present application, the mathematical expression of the above time synchronization condition is:

[0077]

[0078] Among them, t lstart Indicates the moment when the laser radar starts moving; t cstart Indicates the moment when the camera starts moving; t lend Indicates the end time of the laser radar movement; t cend Indicates the end time of the camera movement; dt lj Indicates MEMS matching data; threshold indicates the preset time synchronization threshold.

[0079] Therefore, the embodiment of the present application compares the motion time difference and the MEMS matching data to obtain the time alignment index corresponding to the camera and the lidar, and determines whether the time alignment index meets the time synchronization condition shown in formula (3). If the time alignment index meets the time synchronization condition, it is determined that the time alignment result of the camera and the lidar is reliable.

[0080] To sum up, the time alignment results calculated by the embodiment of the present application are not only accurate and precise, experimentally true and effective, but also can serve as a solid foundation for subsequent sensor data fusion. According to the time alignment results of the embodiment of the present application, the color three-dimensional point cloud model that fuses the panoramic camera image data and the lidar point cloud data has a good effect, thereby verifying the effectiveness of the time synchronization of the present application.

[0081] According to the camera and laser sensor time synchronization method with real-time MEMS posture matching proposed in an embodiment of the present application, a multimodal visual perception system is constructed based on a preset camera, laser radar and MEMS device, and visual perception data corresponding to the multimodal visual perception system is collected, wherein the visual perception data includes camera data, camera MEMS data, laser radar data and laser radar MEMS data; the MEMS acceleration time series corresponding to the camera MEMS data and the laser radar MEMS data are obtained respectively, and the feature information corresponding to the MEMS acceleration time series is extracted, and the MEMS acceleration time series is matched according to the feature information to obtain MEMS matching data; the start movement time corresponding to the camera and the laser radar is determined respectively, and the movement time difference between the camera and the laser radar is calculated according to the start movement time, and the movement time difference and the MEMS matching data are compared to obtain the time alignment index corresponding to the camera and the laser radar, and it is judged whether the time alignment index meets the preset time synchronization condition, wherein if the time alignment index meets the time synchronization condition, it is determined that the camera and the laser radar are time synchronized. By matching the data of MEMS accelerometers connected to the device, this application can quickly complete high-precision time alignment of consumer-grade devices without relying on a time synchronization interface, thereby significantly improving the accuracy and reliability of data fusion.

[0082] Next, a camera and laser sensor time synchronization device with real-time MEMS posture matching proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0083] Figure 4 4 is a block diagram of a device for synchronizing a camera and a laser sensor with real-time MEMS posture matching according to an embodiment of the present application.

[0084] like Figure 4 As shown, the real-time MEMS attitude matching camera and laser sensor time synchronization device 10 includes: a construction module 100, a matching module 200 and a synchronization module 300.

[0085] Among them, the construction module 100 is used to build a multimodal visual perception system based on preset cameras, lidars and MEMS devices, and collect visual perception data corresponding to the multimodal visual perception system, wherein the visual perception data includes camera data, camera MEMS data, lidar data and lidar MEMS data.

[0086] The matching module 200 is used to obtain the MEMS acceleration time series corresponding to the camera MEMS data and the lidar MEMS data respectively, extract the feature information corresponding to the MEMS acceleration time series, and match the MEMS acceleration time series according to the feature information to obtain MEMS matching data.

[0087] The synchronization module 300 is used to determine the start movement time corresponding to the camera and the lidar respectively, and calculate the movement time difference between the camera and the lidar based on the start movement time, and compare the movement time difference with the MEMS matching data to obtain the time alignment index corresponding to the camera and the lidar, and judge whether the time alignment index meets the preset time synchronization condition. If the time alignment index meets the time synchronization condition, it is determined that the camera and the lidar have achieved time synchronization.

[0088] Optionally, in one embodiment of the present application, the construction module 100 includes: a building unit, a fixing unit, a starting unit, a judging unit and a static unit.

[0089] Among them, a unit is established to build a camera MEMS architecture based on a camera and a MEMS device, and to build a lidar MEMS architecture through a lidar and a MEMS device.

[0090] The fixing unit is used to fix the camera MEMS architecture and the lidar MEMS architecture according to a preset fixed connection relationship to build a multimodal visual perception system.

[0091] The starting unit is used to sequentially start the lidar data acquisition function and the camera data acquisition function of the multimodal visual perception system, and after the lidar and camera targets are stationary for a certain period of time, the lidar data and the lidar MEMS data are synchronously collected through the lidar MEMS architecture, and the camera data and the camera MEMS data are synchronously collected using the camera MEMS architecture.

[0092] The judgment unit is used to judge whether the laser radar MEMS architecture and the camera MEMS architecture have completed the data acquisition operation.

[0093] The static unit is used to statically store the laser radar and camera for a target time if both the laser radar MEMS architecture and the camera MEMS architecture have completed the data acquisition operation, and to stop the laser radar data acquisition function and the camera data acquisition function in turn after the target time, otherwise continue to execute the data acquisition operation of the laser radar MEMS architecture and / or the camera MEMS architecture.

[0094] Optionally, in one embodiment of the present application, the matching module 200 includes: a storage unit, a rotation unit, a determination unit, and a generation unit.

[0095] Among them, the saving unit is used to save the camera MEMS data and the lidar MEMS data as MEMS acceleration time series in the target basic data cell format, wherein the MEMS acceleration time series includes a timestamp, yaw angle acceleration, roll angle acceleration and pitch angle acceleration.

[0096] The rotation unit is used to obtain the rotation matrix corresponding to the fixed connection relationship, and rotate the yaw angular acceleration, roll angular acceleration and pitch angular acceleration of the laser radar according to the rotation matrix, so that the Z axis of the coordinate system of the preset laser radar MEMS accelerometer is parallel to the Z axis of the coordinate system of the preset machine MEMS accelerometer.

[0097] The determination unit is used to determine the time threshold corresponding to the MEMS acceleration information based on the yaw angle angular acceleration, and perform feature extraction on the MEMS acceleration time series according to the time threshold to obtain a peak sequence corresponding to the MEMS acceleration time series.

[0098] The generating unit is used to match the MEMS acceleration time series through the peak sequence to generate MEMS matching data.

[0099] Optionally, in one embodiment of the present application, the synchronization module 300 includes: an acquisition unit and a calculation unit.

[0100] Among them, the acquisition unit is used to respectively obtain the device still time before acquisition and the device still time after acquisition of the laser radar and camera, and determine the start movement time and end movement time corresponding to the laser radar and camera according to the device still time before acquisition and the device still time after acquisition.

[0101] The calculation unit is used to calculate the movement time difference between the lidar and the camera based on the start time and the end time of the movement.

[0102] Optionally, in one embodiment of the present application, the mathematical expression of the time synchronization condition is:

[0103] |t lstart -t cstart -dt lj |<threshold

[0104] |t lend -t cend -dt lj |<threshold

[0105] Among them, t lstart Indicates the moment when the laser radar starts moving; t cstart Indicates the moment when the camera starts moving; t lend Indicates the end time of the laser radar movement; t cend Indicates the end time of the camera movement; dt lj Indicates MEMS matching data; threshold indicates the preset time synchronization threshold.

[0106] It should be noted that the above explanation of the embodiment of the method for time synchronization of a camera and a laser sensor with real-time MEMS attitude matching is also applicable to the device for time synchronization of a camera and a laser sensor with real-time MEMS attitude matching in this embodiment, and will not be repeated here.

[0107] According to the embodiment of the present application, a camera and laser sensor time synchronization device with real-time MEMS posture matching proposed includes a construction module for constructing a multimodal visual perception system based on a preset camera, laser radar and MEMS device, and collecting visual perception data corresponding to the multimodal visual perception system, wherein the visual perception data includes camera data, camera MEMS data, laser radar data and laser radar MEMS data; a matching module for respectively obtaining MEMS acceleration time series corresponding to the camera MEMS data and the laser radar MEMS data, and extracting feature information corresponding to the MEMS acceleration time series, and matching the MEMS acceleration time series according to the feature information to obtain MEMS matching data; a synchronization module for respectively determining the start motion time corresponding to the camera and the laser radar, and calculating the motion time difference between the camera and the laser radar according to the start motion time, and comparing the motion time difference with the MEMS matching data to obtain the time alignment index corresponding to the camera and the laser radar, and judging whether the time alignment index meets the preset time synchronization condition, wherein if the time alignment index meets the time synchronization condition, it is determined that the camera and the laser radar have achieved time synchronization. By matching the data of MEMS accelerometers connected to the device, this application can quickly complete high-precision time alignment of consumer-grade devices without relying on a time synchronization interface, thereby significantly improving the accuracy and reliability of data fusion.

[0108] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0109] Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .

[0110] When the processor 502 executes the program, the camera and laser sensor time synchronization method for real-time MEMS posture matching provided in the above embodiment is implemented.

[0111] Furthermore, the electronic device further includes:

[0112] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0113] The memory 501 is used to store computer programs that can be run on the processor 502 .

[0114] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non- vo latil e memory), such as at least one disk storage.

[0115] If the memory 501, processor 502, and communication interface 503 are implemented independently, the communication interface 503, memory 501, and processor 502 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0116] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0117] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0118] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for time synchronization of a camera and a laser sensor with real-time MEMS posture matching.

[0119] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned camera and laser sensor time synchronization method for real-time MEMS posture matching.

[0120] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0121] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0122] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0123] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0124] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0125] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0126] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0127] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for real-time MEMS attitude matching camera and laser sensor time synchronization, characterized in that: The following steps are involved: Based on the preset camera, lidar and MEMS devices, a multimodal visual perception system is constructed, and visual perception data corresponding to the multimodal visual perception system is collected, wherein the visual perception data includes camera data, camera MEMS data, lidar data and lidar MEMS data; Acquire MEMS acceleration time series corresponding to the camera MEMS data and the lidar MEMS data, respectively, extract feature information corresponding to the MEMS acceleration time series, and match the MEMS acceleration time series according to the feature information to obtain MEMS matching data; Determine the start movement moments corresponding to the camera and the laser radar respectively, calculate the movement time difference between the camera and the laser radar based on the start movement moments, and compare the movement time difference with the MEMS matching data to obtain the time alignment indicators corresponding to the camera and the laser radar, and judge whether the time alignment indicators meet the preset time synchronization conditions, wherein if the time alignment indicators meet the time synchronization conditions, it is determined that the camera and the laser radar achieve time synchronization.

2. The method according to claim 1, characterized in that The multimodal visual perception system is constructed based on the preset camera, lidar and MEMS device, and the visual perception data corresponding to the multimodal visual perception system is collected, including: Based on the camera and the MEMS device, a camera MEMS architecture is constructed, and by using the laser radar and the MEMS device, a laser radar MEMS architecture is constructed; The camera MEMS architecture and the lidar MEMS architecture are fixedly connected according to a preset fixed connection relationship to construct the multimodal visual perception system; Sequentially starting the lidar data acquisition function and the camera data acquisition function of the multimodal visual perception system, and after the lidar and the camera are stationary for a target period, synchronously acquiring the lidar data and the lidar MEMS data through the lidar MEMS architecture, and synchronously acquiring the camera data and the camera MEMS data using the camera MEMS architecture; Determine whether the laser radar MEMS architecture and the camera MEMS architecture have completed data acquisition operations; If both the lidar MEMS architecture and the camera MEMS architecture complete the data acquisition operation, the lidar and the camera are left stationary for the target time, and after the target time, the lidar data acquisition function and the camera data acquisition function are stopped in turn, otherwise the data acquisition operation of the lidar MEMS architecture and / or the camera MEMS architecture is continued.

3. The method according to claim 2, characterized in that The obtaining of MEMS acceleration time series corresponding to the camera MEMS data and the lidar MEMS data, extracting feature information corresponding to the MEMS acceleration time series, and matching the MEMS acceleration time series according to the feature information to obtain MEMS matching data includes: Saving the camera MEMS data and the lidar MEMS data as MEMS acceleration time series in a target basic data cell format, respectively, wherein the MEMS acceleration time series includes a timestamp, yaw angle acceleration, roll angle acceleration, and pitch angle acceleration; Obtain a rotation matrix corresponding to the fixed connection relationship, and rotate the yaw angular acceleration, the roll angular acceleration, and the pitch angular acceleration of the laser radar according to the rotation matrix so that the coordinate system Z axis of the preset laser radar MEMS accelerometer is parallel to the coordinate system Z axis of the preset machine MEMS accelerometer; Determining a time threshold corresponding to the MEMS acceleration time series based on the yaw angle angular acceleration, and performing feature extraction on the MEMS acceleration time series according to the time threshold to obtain a peak sequence corresponding to the MEMS acceleration time series; The MEMS acceleration time series is matched using the peak sequence to generate the MEMS matching data.

4. The method according to claim 3, characterized in that The determining of the start movement times of the camera and the laser radar respectively, and calculating the movement time difference between the camera and the laser radar according to the start movement times, includes: Obtaining the pre-acquisition device stationary time and the post-acquisition device stationary time of the laser radar and the camera respectively, and determining the start and end movement times of the laser radar and the camera respectively according to the pre-acquisition device stationary time and the post-acquisition device stationary time; Based on the start time of movement and the end time of movement, the movement time difference between the laser radar and the camera is calculated.

5. The method according to claim 1, wherein The mathematical expression of the time synchronization condition is: in, Indicates the start time of the laser radar movement; Indicates the start time of the camera movement; Indicates the end motion moment of the laser radar; Indicates the end motion moment of the camera; Indicates the MEMS matching data; Indicates the preset time synchronization threshold.

6. A camera and laser sensor time synchronization device with real-time MEMS posture matching, characterized in that: include: A construction module is used to construct a multimodal visual perception system based on preset cameras, lidars, and MEMS devices, and to collect visual perception data corresponding to the multimodal visual perception system, wherein the visual perception data includes camera data, camera MEMS data, lidar data, and lidar MEMS data; a matching module, configured to respectively obtain MEMS acceleration time series corresponding to the camera MEMS data and the lidar MEMS data, extract feature information corresponding to the MEMS acceleration time series, and match the MEMS acceleration time series according to the feature information to obtain MEMS matching data; A synchronization module is used to respectively determine the start movement moments of the camera and the laser radar, calculate the movement time difference between the camera and the laser radar based on the start movement moments, and compare the movement time difference with the MEMS matching data to obtain the time alignment indicators corresponding to the camera and the laser radar, and determine whether the time alignment indicators meet the preset time synchronization conditions, wherein if the time alignment indicators meet the time synchronization conditions, it is determined that the camera and the laser radar have achieved time synchronization.

7. The device according to claim 6, characterized in that The building blocks include: An establishment unit, configured to construct a camera MEMS architecture based on the camera and the MEMS device, and to construct a lidar MEMS architecture using the lidar and the MEMS device; A fixing unit, configured to fix the camera MEMS architecture and the lidar MEMS architecture according to a preset fixed connection relationship to construct the multimodal visual perception system; a start-up unit, configured to sequentially start a lidar data acquisition function and a camera data acquisition function of the multimodal visual perception system, and after the lidar and the camera are left stationary for a target period, synchronously acquire the lidar data and the lidar MEMS data through the lidar MEMS architecture, and synchronously acquire the camera data and the camera MEMS data using the camera MEMS architecture; A judgment unit, configured to respectively judge whether the laser radar MEMS architecture and the camera MEMS architecture have completed data acquisition operations; A static unit is used to statically store the laser radar and the camera for a target time period if both the laser radar MEMS architecture and the camera MEMS architecture complete the data acquisition operation, and to sequentially stop the laser radar data acquisition function and the camera data acquisition function after the target time period, otherwise continue to execute the data acquisition operation of the laser radar MEMS architecture and / or the camera MEMS architecture.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the camera and laser sensor time synchronization method for real-time MEMS posture matching according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the camera and laser sensor time synchronization method for real-time MEMS posture matching as described in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that The computer program is executed to implement the camera and laser sensor time synchronization method for real-time MEMS posture matching according to any one of claims 1 to 5.

Images