Multi-sensor synchronization method, device and vehicle-mounted control system

Abstract

The present application discloses a multi-sensor synchronization method, device and vehicle-mounted control system. The method includes: obtaining the target parameters of the laser radar, wherein the target parameters include: the first angle corresponding to the current scanning position of the laser radar, the second angle corresponding to when the laser radar scans to the calibration position, and the rotation speed of the laser radar; obtaining the viewing angle range of the camera to be synchronized in the target scanning direction, wherein the camera to be synchronized is scanned in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar; determining the starting scanning time of the camera to be synchronized based on the target parameters and the viewing angle range, and controlling the camera to be synchronized to scan from the starting scanning time. The present application solves the technical problem that it is difficult to achieve accurate synchronization between multiple sensors in an autonomous driving system.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a multi-sensor synchronization method, device and vehicle-mounted control system. Background Art

[0002] In the field of autonomous driving, the perception system plays a vital role. It usually requires the use of multiple sensors in combination to make up for each other's shortcomings. That is, multi-sensor fusion is an indispensable core in the perception system. In the mainstream design scheme, commonly used sensors usually include: LiDAR and vehicle-mounted cameras. Among them, LiDAR is mainly a rotating mechanical LiDAR, which is used to measure the spatial information of the environmental background and obstacles, such as spatial position coordinates and distance; the vehicle-mounted camera is mainly a vehicle-mounted rolling exposure wide dynamic camera, which can obtain rich scene information, but the images taken by the camera are basically planar information and lack three-dimensional spatial information. By fusing the two sensors with each other, it is possible to make up for each other's shortcomings and obtain complete spatial scene information.

[0003] When the data from the two sensors are fused, the data in the time domain and space domain are crucial. If the deviation is too large, the fusion will fail, resulting in a significant decrease in perception ability, such as deviations in the predicted obstacle position. Therefore, the premise of sensor fusion is that the data returned by them maintains synchronization in time and space. Due to the differences in the scanning methods of LiDAR and on-board cameras, these two sensors have certain difficulties in data synchronization. Although frame-level synchronization between LiDAR and on-board cameras can be achieved in related technologies, that is, a certain point or area in the camera image is synchronized with the LiDAR, the degree of synchronization is still low and it is difficult to meet actual needs.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a multi-sensor synchronization method, device and vehicle-mounted control system to at least solve the technical problem that it is difficult to achieve accurate synchronization between multiple sensors in an autonomous driving system.

[0006] According to one aspect of an embodiment of the present application, a multi-sensor synchronization method is provided, including: obtaining target parameters of a laser radar, wherein the target parameters include: a first angle corresponding to a current scanning position of the laser radar, a second angle corresponding when the laser radar scans to a calibration position, and a rotation speed of the laser radar; obtaining a viewing angle range of a camera to be synchronized in a target scanning direction, wherein the camera to be synchronized scans in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar; determining a starting scanning moment of the camera to be synchronized based on the target parameters and the viewing angle range, and controlling the camera to be synchronized to start scanning from the starting scanning moment.

[0007] According to another aspect of an embodiment of the present application, a multi-sensor synchronization device is also provided, including: a first acquisition module, used to acquire target parameters of the laser radar, wherein the target parameters include: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding when the laser radar scans to a calibration position, and a rotation speed of the laser radar; a second acquisition module, used to acquire the viewing angle range of the camera to be synchronized in the target scanning direction, wherein the camera to be synchronized scans in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar; a control module, used to determine the starting scanning time of the camera to be synchronized based on the target parameters and the viewing angle range, and control the camera to be synchronized to start scanning from the starting scanning time.

[0008] According to another aspect of an embodiment of the present application, a vehicle-mounted control system is also provided, including: a vehicle-mounted camera, used to scan in a target scanning direction in a line-by-line exposure manner, wherein the target scanning direction is the same as the scanning direction of a vehicle-mounted laser radar; the vehicle-mounted laser radar, used to perform synchronous scanning with the vehicle-mounted camera; a vehicle-mounted control unit, used to obtain target parameters of the vehicle-mounted laser radar, wherein the target parameters include: a first angle corresponding to a current scanning position of the vehicle-mounted laser radar, a second angle corresponding when the vehicle-mounted laser radar scans to a calibration position, and a rotation speed of the vehicle-mounted laser radar; obtaining the viewing angle range of the vehicle-mounted camera in the target scanning direction; determining the starting scanning time of the vehicle-mounted camera based on the target parameters and the viewing angle range, and controlling the vehicle-mounted camera to start scanning from the starting scanning time.

[0009] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining target parameters of the laser radar, wherein the target parameters include: a first angle corresponding to a current scanning position of the laser radar, a second angle corresponding when the laser radar scans to a calibration position, and a rotation speed of the laser radar; obtaining a viewing angle range of a camera to be synchronized in a target scanning direction, wherein the camera to be synchronized scans in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar; determining a starting scanning time of the camera to be synchronized based on the target parameters and the viewing angle range, and controlling the camera to be synchronized to start scanning from the starting scanning time.

[0010] In an embodiment of the present application, the camera to be synchronized is rotated so that the target scanning direction of the camera to be synchronized is the same as the scanning direction of the laser radar, so as to facilitate data synchronization; thereafter, relevant parameters of the laser radar and the camera to be synchronized are obtained, including: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding when the laser radar scans to a calibration position, a rotation speed of the laser radar, a viewing angle range of the camera to be synchronized in the target scanning direction, etc.; the starting scanning time of the camera to be synchronized is determined according to the relevant parameters, and the camera to be synchronized is controlled to start scanning from the starting scanning time; since the scanning directions of the two are the same and the scanning times correspond to each other, pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar is achieved, thereby solving the technical problem that it is difficult to achieve accurate synchronization between multiple sensors in an autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 is a schematic diagram of the hardware structure of a computer terminal according to an embodiment of the present application;

[0013] Figure 2 is a flowchart of a multi-sensor synchronization method according to an embodiment of the present application;

[0014] Figure 3 is a schematic diagram of a scanning process of a camera and a laser radar to be synchronized according to an embodiment of the present application;

[0015] Figure 4 is a schematic diagram of another scanning process of a camera and a laser radar to be synchronized according to an embodiment of the present application;

[0016] Figure 5is a timing diagram of synchronous scanning of a camera to be synchronized and a laser radar according to an embodiment of the present application;

[0017] Figure 6 is a timing diagram of a camera to be synchronized when operating at a maximum frame rate according to an embodiment of the present application;

[0018] Figure 7 is a structural schematic diagram of a multi-sensor synchronization device according to an embodiment of the present application;

[0019] Figure 8 It is a structural schematic diagram of a vehicle control system according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0023] Lidar: used to measure spatial coordinates and distance. The mainstream lidar is a mechanical rotating radar.

[0024] Rolling shutter: A camera exposure method in which the image is exposed line by line.

[0025] Global shutter: A camera exposure method in which all pixels of the entire image are exposed at the same time.

[0026] Trigger mode: A camera working mode in which the camera starts exposing the shutter when it receives a trigger signal, thereby accurately controlling the camera's photo-taking time.

[0027] Slave mode is a camera working mode that controls the camera to expose and shoot stably at a certain frequency, such as taking pictures 30 times per second.

[0028] Example 1

[0029] In the related art, in order to achieve the synchronization of scanning data between the LiDAR and the vehicle-mounted camera, a timestamp matching scheme and a scheme for triggering camera exposure according to the rotation angle of the LiDAR were proposed respectively.

[0030] In the first solution, the onboard camera works in slave mode. When the lidar and the onboard camera receive data, they are time-stamped respectively, such as the camera image is time-stamped in frames and the lidar is time-stamped in circles, and then one-to-one matching is performed in the obtained data according to the timestamps. However, this method has obvious errors and can only ensure that the deviation between the lidar and camera data does not exceed one frame, and even accurate frame-level synchronization cannot be achieved.

[0031] In the second solution, the on-board camera works in trigger mode. Since the data returned by the laser radar includes the current rotation angle, the real-time rotation angle of the current laser radar can be obtained by parsing the laser radar data packet, and then when the laser radar rotates near the camera, the camera is triggered to start exposure. This method can ensure that a point or area in the camera image can be synchronized with the point cloud returned by the laser radar, and realize the frame-level synchronization of the two sensors. Among them, the camera's scanning methods include rolling exposure and global exposure. The scanning method of the laser radar is fixed to rotation scanning. From the camera's perspective, the laser radar is scanned column by column from left to right. No matter which scanning method the camera uses, it cannot be consistent with the scanning method of the laser radar. In the end, it can only align the pixels on a line from the upper left corner to the lower right corner of the camera image. Other pixels will have spatial and temporal position deviations. That is, the synchronization degree of this solution is still low and it is difficult to meet actual needs.

[0032] In order to solve the above problems, the embodiment of the present application provides an embodiment of a multi-sensor synchronization method to achieve pixel-level synchronization of scan data between a camera and a laser radar. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0033] Specifically, the multi-sensor synchronization method provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing the multi-sensor synchronization method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0034] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multi-sensor synchronization method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the vulnerability detection method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0036] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0038] Under the above operating environment, an optional multi-sensor synchronization method provided by the embodiment of the present application is as follows: Figure 2 As shown, the method at least includes steps S202-S206, wherein:

[0039] Step S202, obtaining target parameters of the laser radar, wherein the target parameters include: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding to when the laser radar scans to a calibration position, and a rotation speed of the laser radar.

[0040] Step S204, obtaining the viewing angle range of the camera to be synchronized in the target scanning direction, wherein the camera to be synchronized scans in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar.

[0041] Step S206, determining the start scanning time of the camera to be synchronized according to the target parameters and the viewing angle range, and controlling the camera to be synchronized to start scanning from the start scanning time.

[0042] At present, mainstream laser radars all use a mechanical rotation scanning method, and their installation method is usually horizontal, that is, the rotation axis of the laser radar is perpendicular to the horizontal plane; the camera to be synchronized is usually installed horizontally forward, backward, left or right. At this time, if you observe the scanning process of the laser radar from the perspective of the camera to be synchronized, you can find that the scanning method of the laser radar is a column-by-column scan from left to right. If you want to achieve pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar, at least you must first ensure that the scanning methods of the two are the same. Therefore, the camera to be synchronized in the embodiment of the present application is a camera to be synchronized in a rolling shutter exposure mode, that is, the camera to be synchronized will scan the image line by line.

[0043] by Figure 3For example, assuming that the laser radar rotates clockwise, in the view of the camera to be synchronized, the laser radar scans from left to right column by column, while the camera to be synchronized scans from top to bottom row by row. In order to make the scanning methods of the two sensors the same, in some optional embodiments of the present application, the camera to be synchronized can be rotated 90 degrees counterclockwise with the optical axis of the camera to be synchronized as the center, so as to obtain the scanning schematic diagram of the two sensors as shown in the figure below: Figure 4 As shown, in the perspective of the camera to be synchronized, the scanning mode of the laser radar is column-by-column scanning from left to right, and the scanning mode of the camera to be synchronized is also column-by-column scanning from left to right, that is, the scanning mode of the camera to be synchronized and the laser radar is the same.

[0044] It should be noted that the above process is only for illustration and does not constitute a specific limitation. For example, for the structure of the laser radar, if the scanning method of some solid-state radars is the same as that of mechanical rotating radars, the method of the embodiment of the present application can also be used; for the rotation direction of the laser radar, whether it is clockwise or counterclockwise, the scanning direction of the camera to be synchronized can be the same as that of the laser radar by rotating the camera to be synchronized to the corresponding target angle.

[0045] In the above steps, by making the scanning mode of the camera to be synchronized the same as that of the laser radar, the synchronization of the scanning data space has been achieved, and then the synchronization of the scanning data time needs to be achieved. Since the rotation of the laser radar itself is not controlled by an external sensor, the scanning time of the camera to be synchronized can be determined by calculating the relevant parameters of the laser radar and the camera to be synchronized, so that it corresponds to the laser radar, thereby achieving pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar.

[0046] Specifically, the data packet of the laser radar can be obtained by the control unit first, and the data packet can be parsed to obtain the first angle corresponding to the current scanning position of the laser radar as α, and the rotation speed of the laser radar as q (r / s). The control unit can be an embedded programmable device such as a single-chip microcomputer, an ARM processor, a field programmable gate array FPGA, etc.

[0047] Secondly, after determining the installation positions of the camera to be synchronized and the laser radar, the laser radar and the camera to be synchronized need to be calibrated, and the second angle β corresponding to when the laser radar scans to the front of the optical axis of the camera to be synchronized is determined according to the calibration results. Among them, although there are many kinds of methods used in the calibration process, the final calibration results are the same.

[0048] Optionally, after the camera to be synchronized is rotated, it is also necessary to determine the viewing angle range of the camera to be synchronized in its scanning direction. Figure 4For example, after the camera to be synchronized is rotated 90 degrees counterclockwise with its optical axis as the center, its scanning mode is horizontal scanning from left to right column by column. At this time, it can be determined that the viewing angle range angle of the rotated camera to be synchronized in the horizontal direction is H.

[0049] After obtaining the above parameters, the start scanning time of the camera to be synchronized can be determined to correspond to the laser radar, thereby achieving pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar. Among them, since the scanning process of the camera to be synchronized includes the exposure process and the reading process, the start scanning time is divided into the start exposure time t e and the starting reading time t0.

[0050] In some optional embodiments of the present application, the start reading time t0 can be determined first, and the first scanning time period t1 can be determined according to the first angle α, the second angle β and the rotation speed q, wherein the laser radar rotates from the current scanning position to the calibration position within the first scanning time period t1, wherein the calibration position is directly in front of the optical axis of the camera to be synchronized; the second scanning time period t f , where the camera to be synchronized is in the second scanning time period t f The laser radar scans a frame of image from the first line to the last line in the second scanning time period t f The rotation angle within is the viewing angle range H; according to the first scanning time period t1 and the second scanning time period t f Determine the start reading time t0.

[0051] Specifically, assuming that the laser radar rotates clockwise, its rotation angle gradually increases as it rotates, and the angle returns to zero after a full circle. The first scanning time period t1 required for the laser radar to rotate from the current scanning position α to the calibration position β can be calculated first. If α>β, then:

[0052] t1=(360+β-α) / (q*360)

[0053] If α≤β, then:

[0054] t1=(β-α) / (q*360)

[0055] The second scanning time period t f Also known as the single-frame reading time, it is the time period from the first line of the image to the last line of the image (the row here refers to the row scanned line by line in the scanning mode of the camera to be synchronized, which is equivalent to the column in the rotated camera's field of view). It is also the time required for the laser radar to rotate through the field of view range H of the camera to be synchronized.

[0056] t f =H / (q*360)

[0057] At this time, the first scanning time period t1 and the second scanning time period t f Determine the start reading time t0, where,

[0058]

[0059] At time t0, the camera to be synchronized is controlled to start reading the first column on the left side of the image. At this time, the laser radar starts scanning from the same position, which means that the synchronization of the scanning data of the camera to be synchronized and the laser radar is achieved. The timing diagram of the synchronous scanning of the two sensors is shown in Figure 5 As shown, each angle and time corresponds to the above calculation results, t2 corresponds to the time period when the laser radar rotates from the initial position (0 degrees) to the current scanning position, and has no effect on the above calculation process.

[0060] The scanning process of the camera to be synchronized includes the exposure process and the reading process. The timing diagram when it works in the maximum frame rate mode is as follows: Figure 6 As shown, the reading operation cannot be performed in parallel on different rows, and the reading operation of the second row must wait for the completion of the reading operation of the first row, while the exposure operation can be performed in parallel between different rows, that is, they can overlap in time. Therefore, the frame rate of the camera to be synchronized is actually determined by the reading operation. When the reading operation is fast enough, the camera to be synchronized can achieve a higher frame rate. When the reading operation of the first row starts and the reading operation of the last row is completed, the first frame of the image is scanned, and then the reading operation starts with the first row of the next frame of the image, and the cycle continues. The exposure operation is completed before the reading. After the exposure is completed, there may be a short time to wait for the reading operation of the row of data to start.

[0061] It can be seen that the starting reading time t0 of the camera to be synchronized cannot be directly controlled. Therefore, the starting exposure time t e , through the initial exposure time t e Realize the control of the starting reading time t0.

[0062] In some optional embodiments of the present application, the number of rows n to be scanned by the camera to be synchronized when scanning a frame of image can be determined according to the resolution of the camera to be synchronized; f The third scanning time period t of the camera to be synchronized is determined by the number of rows n r , where the camera to be synchronized is in the third scanning time period t r The reading process of one line of the image is completed within t0; according to the starting reading time t0, the second scanning time period t f and the third scanning period t r Determine the start exposure time t e .

[0063] Specifically, the third scanning time period t r Also known as the single-line reading time, it is the time required for the synchronized camera to complete the reading process of a line in the image.

[0064] t r =t f / n

[0065] Usually, the camera to be synchronized can modify the single-line reading time t by modifying the clock division register r , but t r It cannot be infinitely small, and there is a lower limit, that is, there is an upper limit to the frame rate of the camera to be synchronized. If t calculated according to the above process is small, the camera to be synchronized with a higher frame rate can apply the multi-sensor synchronization method of the embodiment of the present application.

[0066] Afterwards, the second scanning time period t f and the third scanning period t r Determine the start exposure time t e ,in,

[0067] t e =t0+t r -t f

[0068] In the actual control process, the control unit can start the exposure at the start time t e A trigger signal is sent to the camera to be synchronized, and the trigger signal is used to control the camera to be synchronized to start the exposure process.

[0069] The following is a specific calculation process of controlling the camera to be synchronized using the multi-sensor synchronization method of an embodiment of the present application, wherein the resolution of the camera to be synchronized is 1080x1920, and the horizontal viewing angle range H=60° after the camera to be synchronized is rotated; after calibration, it is determined that the optical axis direction of the camera to be synchronized is in the 90° direction of the laser radar rotation, that is, β=90°; the control unit receives the laser radar data packet and parses the current laser radar scanning position α=180°, and the rotation speed of the laser radar is 10 (r / s); calculate in sequence:

[0070] Single frame reading time of the camera to be synchronized:

[0071]

[0072] That is, the single-frame scanning period of the camera to be synchronized needs to be set to 1 / 60s, that is, the frame rate of the camera to be synchronized is 60FPS;

[0073] The time required for the laser radar to move from the current position α to β

[0074]

[0075] The time when the camera to be synchronized starts reading:

[0076]

[0077] Single line reading time of the camera to be synchronized:

[0078]

[0079] The camera to be synchronized starts exposure, i.e. the moment the trigger signal is sent:

[0080]

[0081] It should be noted that in the embodiment of the present application, since the time for the laser radar to scan a column is very short, and the scanning time of a single column after the camera to be synchronized rotates is also very short, the time deviation of the column scanning can be ignored, and the two sensors are approximately considered to be synchronized; since the rotation process of the laser radar is not a completely accurate uniform motion, there will be slight differences in speed between different circles, and therefore it is necessary to frequently repeat the above steps to set and control the camera to be synchronized, so as to ensure that the two sensors remain synchronized when the laser radar turns to the viewing range of the camera to be synchronized each circle.

[0082] In some optional embodiments of the present application, when it is detected that the laser radar stops rotating, the control unit controls the camera to be synchronized to stop scanning. Specifically, the camera to be synchronized is associated with the laser radar, and when the laser radar stops working, the camera to be synchronized no longer works independently, that is, when the calibrated laser radar point cloud data is projected into the image, it can no longer continue to form pixel-level synchronization.

[0083] In an embodiment of the present application, the camera to be synchronized is rotated so that the target scanning direction of the camera to be synchronized is the same as the scanning direction of the laser radar, so as to facilitate data synchronization; thereafter, relevant parameters of the laser radar and the camera to be synchronized are obtained, including: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding when the laser radar scans to a calibration position, a rotation speed of the laser radar, a viewing angle range of the camera to be synchronized in the target scanning direction, etc.; the starting scanning time of the camera to be synchronized is determined according to the relevant parameters, and the camera to be synchronized is controlled to start scanning from the starting scanning time; since the scanning directions of the two are the same and the scanning times correspond to each other, pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar is achieved, thereby solving the technical problem that it is difficult to achieve accurate synchronization between multiple sensors in an autonomous driving system.

[0084] Example 2

[0085] According to an embodiment of the present application, a multi-sensor synchronization device for implementing the multi-sensor synchronization method is also provided. Figure 7 As shown, the device includes a first acquisition module 70, a second acquisition module 72 and a control module 74, wherein:

[0086] The first acquisition module 70 is used to acquire the target parameters of the laser radar, wherein the target parameters include: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding to when the laser radar scans to a calibration position, and a rotation speed of the laser radar.

[0087] The second acquisition module 72 is used to obtain the viewing angle range of the camera to be synchronized in the target scanning direction, wherein the camera to be synchronized scans in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar.

[0088] The control module 74 is used to determine the start scanning time of the camera to be synchronized according to the target parameters and the viewing angle range, and control the camera to be synchronized to start scanning from the start scanning time.

[0089] Among them, mainstream laser radars all adopt a mechanical rotation scanning method, and their installation method is usually horizontal, that is, the rotation axis of the laser radar is perpendicular to the horizontal plane; the camera to be synchronized is usually installed horizontally forward, backward, left or right. At this time, if you observe the scanning process of the laser radar from the perspective of the camera to be synchronized, you can find that the scanning method of the laser radar is a column-by-column scan from left to right. If you want to achieve pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar, at least you must first ensure that the scanning methods of the two are the same. Therefore, the camera to be synchronized in the embodiment of the present application is a camera to be synchronized in a rolling shutter exposure mode, that is, the camera to be synchronized will scan the image line by line.

[0090] by Figure 3 For example, assuming that the laser radar rotates clockwise, in the view of the camera to be synchronized, the laser radar scans from left to right column by column, while the camera to be synchronized scans from top to bottom row by row. In order to make the scanning methods of the two sensors the same, in some optional embodiments of the present application, the camera to be synchronized can be rotated 90 degrees counterclockwise with the optical axis of the camera to be synchronized as the center, so as to obtain the scanning schematic diagram of the two sensors as shown in the figure below: Figure 4 As shown, in the perspective of the camera to be synchronized, the scanning mode of the laser radar is column-by-column scanning from left to right, and the scanning mode of the camera to be synchronized is also column-by-column scanning from left to right, that is, the scanning mode of the camera to be synchronized and the laser radar is the same.

[0091] By making the scanning mode of the camera to be synchronized the same as that of the laser radar, the spatial synchronization of the scanning data is achieved, and then the time synchronization of the scanning data needs to be achieved. Since the rotation of the laser radar itself is not controlled by an external sensor, the scanning time of the camera to be synchronized can be determined by calculating the relevant parameters of the laser radar and the camera to be synchronized, so that it corresponds to the laser radar, thereby achieving pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar.

[0092] Specifically, the data packet of the laser radar can be obtained by the control unit first, and the data packet can be parsed to obtain the first angle corresponding to the current scanning position of the laser radar as α, and the rotation speed of the laser radar as q (r / s). The control unit can be an embedded programmable device such as a single-chip microcomputer, an ARM processor, a field programmable gate array FPGA, etc.

[0093] Secondly, after determining the installation positions of the camera and the laser radar to be synchronized, the laser radar and the camera to be synchronized need to be calibrated. According to the calibration results, the second angle β corresponding to when the laser radar scans to the front of the optical axis of the camera to be synchronized is determined.

[0094] Optionally, after the camera to be synchronized is rotated, it is also necessary to determine the viewing angle range of the camera to be synchronized in its scanning direction. Figure 4 For example, after the camera to be synchronized is rotated 90 degrees counterclockwise with its optical axis as the center, its scanning mode is horizontal scanning from left to right column by column. At this time, it can be determined that the viewing angle range angle of the rotated camera to be synchronized in the horizontal direction is H.

[0095] After obtaining the above parameters, the start scanning time of the camera to be synchronized can be determined to correspond to the laser radar, thereby achieving pixel-level synchronization of the scanning data between the camera to be synchronized and the laser radar. Among them, since the scanning process of the camera to be synchronized includes the exposure process and the reading process, the start scanning time is divided into the start exposure time t e and the starting reading time t0.

[0096] In some optional embodiments of the present application, the start reading time t0 can be determined first, and the first scanning time period t1 can be determined according to the first angle α, the second angle β and the rotation speed q, wherein the laser radar rotates from the current scanning position to the calibration position within the first scanning time period t1, wherein the calibration position is directly in front of the optical axis of the camera to be synchronized; the second scanning time period t f , where the camera to be synchronized is in the second scanning time period t f The laser radar scans a frame of image from the first line to the last line in the second scanning time period t f The rotation angle within is the viewing angle range H; according to the first scanning time period t1 and the second scanning time period tf Determine the start reading time t0.

[0097] Specifically, assuming that the laser radar rotates clockwise, its rotation angle gradually increases as it rotates, and the angle returns to zero after a full circle. The first scanning time period t1 required for the laser radar to rotate from the current scanning position α to the calibration position β can be calculated first. If α>β, then:

[0098] t1=(360+β-α) / (q*360)

[0099] If α≤β, then:

[0100] t1=(β-α) / (q*360)

[0101] The second scanning time period t f Also known as the single-frame reading time, it is the time period from the first line of the image to the last line of the image (the row here refers to the row scanned line by line in the scanning mode of the camera to be synchronized, which is equivalent to the column in the rotated camera's field of view). It is also the time required for the laser radar to rotate through the field of view range H of the camera to be synchronized.

[0102] t f =H / (q*360)

[0103] At this time, the first scanning time period t1 and the second scanning time period t f Determine the start reading time t0, where,

[0104]

[0105] At time t0, the camera to be synchronized is controlled to start reading the first column on the left side of the image. At this time, the laser radar starts scanning from the same position, which means that the synchronization of the scanning data of the camera to be synchronized and the laser radar is achieved. The timing diagram of the synchronous scanning of the two sensors is shown in Figure 5 As shown, each angle and time corresponds to the above calculation results, t2 corresponds to the time period when the laser radar rotates from the initial position (0 degrees) to the current scanning position, and has no effect on the above calculation process.

[0106] The scanning process of the camera to be synchronized includes the exposure process and the reading process. The timing diagram when it works in the maximum frame rate mode is as follows: Figure 6As shown, the reading operation cannot be performed in parallel on different rows, and the reading operation of the second row must wait for the completion of the reading operation of the first row, while the exposure operation can be performed in parallel between different rows, that is, they can overlap in time. Therefore, the frame rate of the camera to be synchronized is actually determined by the reading operation. When the reading operation is fast enough, the camera to be synchronized can achieve a higher frame rate. When the reading operation of the first row starts and the reading operation of the last row is completed, the first frame of the image is scanned, and then the reading operation starts with the first row of the next frame of the image, and the cycle continues. The exposure operation is completed before the reading. After the exposure is completed, there may be a short time to wait for the reading operation of the row of data to start.

[0107] It can be seen that the starting reading time t0 of the camera to be synchronized cannot be directly controlled. Therefore, the starting exposure time t e , through the initial exposure time t e Realize the control of the starting reading time t0.

[0108] In some optional embodiments of the present application, the number of rows n to be scanned by the camera to be synchronized when scanning a frame of image can be determined according to the resolution of the camera to be synchronized; f The third scanning time period t of the camera to be synchronized is determined by the number of rows n r , where the camera to be synchronized is in the third scanning time period t r The reading process of one line of the image is completed within t0; according to the starting reading time t0, the second scanning time period t f and the third scanning period t r Determine the start exposure time t e .

[0109] Specifically, the third scanning time period t r Also known as the single-line reading time, it is the time required for the synchronized camera to complete the reading process of a line in the image.

[0110] t r =t f / n

[0111] Usually, the camera to be synchronized can modify the single-line reading time t by modifying the clock division register r , but t r It cannot be infinitely small. It has a lower limit, that is, the frame rate of the camera to be synchronized has an upper limit. If t is calculated according to the above process, r When the frame rate is small, the multi-sensor synchronization method of the embodiment of the present application can be applied to the cameras to be synchronized that require a higher frame rate.

[0112] Afterwards, the second scanning time period t f and the third scanning period tr Determine the start exposure time t e ,in,

[0113] t e =t0+t r -t f

[0114] In the actual control process, the control unit can start the exposure at the start time t e A trigger signal is sent to the camera to be synchronized, and the trigger signal is used to control the camera to be synchronized to start the exposure process.

[0115] Example 3

[0116] According to an embodiment of the present application, a vehicle-mounted control system based on the above-mentioned multi-sensor synchronization method is also provided, which can be applied to autonomous driving scenarios, such as Figure 8 As shown, the system includes a vehicle-mounted camera 80, a vehicle-mounted control unit 82 and a vehicle-mounted laser radar 84, wherein:

[0117] The vehicle-mounted camera 80 is used to scan in a target scanning direction in a line-by-line exposure manner, wherein the target scanning direction is the same as the scanning direction of the vehicle-mounted laser radar.

[0118] The vehicle-mounted laser radar 82 is used to perform synchronous scanning with the vehicle-mounted camera 80.

[0119] Among them, mainstream vehicle-mounted laser radars all adopt a mechanical rotation scanning method, and their installation method is usually horizontal, that is, the rotation axis of the vehicle-mounted laser radar is perpendicular to the horizontal plane; the vehicle-mounted camera is usually installed horizontally forward, backward, left or right. At this time, if you observe the scanning process of the vehicle-mounted laser radar from the perspective of the vehicle-mounted camera, you can find that the scanning method of the vehicle-mounted laser radar is a column-by-column scan from left to right. If you want to achieve pixel-level synchronization of scanning data between the vehicle-mounted camera and the vehicle-mounted laser radar, at least you must first ensure that the scanning methods of the two are the same. Therefore, the vehicle-mounted camera in the embodiment of the present application is a vehicle-mounted camera in rolling exposure mode, that is, the vehicle-mounted camera will scan the image line by line.

[0120] by Figure 3 For example, assuming that the onboard laser radar rotates clockwise, in the onboard camera's field of view, the onboard laser radar scans from left to right column by column, while the onboard camera scans from top to bottom row by row. In order to make the two scanning methods the same, in some optional embodiments of the present application, the onboard camera can be rotated 90 degrees counterclockwise around the optical axis of the onboard camera, so as to obtain the scanning schematic diagram of the two sensors as shown in the figure below: Figure 4As shown, in the vehicle camera's perspective, the vehicle-mounted laser radar scans column by column from left to right, and the vehicle-mounted camera scans column by column from left to right, that is, the scanning method of the vehicle-mounted camera and the vehicle-mounted laser radar is the same.

[0121] The vehicle-mounted control unit 84 is used to obtain the target parameters of the vehicle-mounted laser radar 82, wherein the target parameters include: a first angle corresponding to the current scanning position of the vehicle-mounted laser radar 82, a second angle corresponding when the vehicle-mounted laser radar 82 scans to a calibration position, and a rotation speed of the vehicle-mounted laser radar 82; obtain the viewing angle range of the vehicle-mounted camera 80 in the target scanning direction; determine the starting scanning time of the vehicle-mounted camera 80 based on the target parameters and the viewing angle range, and control the vehicle-mounted camera 80 to start scanning from the starting scanning time.

[0122] Among them, the vehicle-mounted control unit can be an embedded programmable device such as a single-chip microcomputer, an ARM processor, a field programmable gate array FPGA, etc.

[0123] Specifically, the data packet of the vehicle-mounted laser radar is obtained through the vehicle-mounted control unit, and the data packet is parsed to obtain the first angle α corresponding to the current scanning position of the vehicle-mounted laser radar, and the rotation speed of the vehicle-mounted laser radar is q (revolutions / second); the vehicle-mounted laser radar and the vehicle-mounted camera are calibrated, and according to the calibration results, it is determined that the second angle β corresponding to the vehicle-mounted laser radar scans to the front of the optical axis of the vehicle-mounted camera.

[0124] Optionally, after the vehicle-mounted camera is rotated, it is also necessary to determine the viewing angle range of the vehicle-mounted camera in its scanning direction. Figure 4 For example, after the vehicle-mounted camera is rotated 90 degrees counterclockwise with the optical axis of the vehicle-mounted camera as the center, its scanning method is to scan column by column from left to right in the horizontal direction. At this time, it can be determined that the viewing angle range angle of the rotated vehicle-mounted camera in the horizontal direction is H.

[0125] After obtaining the above parameters, the start scanning time of the vehicle camera can be determined to correspond to the vehicle laser radar, thereby achieving pixel-level synchronization of the scanning data between the vehicle camera and the vehicle laser radar. Among them, since the scanning process of the vehicle camera includes the exposure process and the reading process, the start scanning time is divided into the start exposure time t e and the starting reading time t0.

[0126] In some optional embodiments of the present application, the starting reading time t0 can be determined first, and the first scanning time period t1 can be determined according to the first angle α, the second angle β and the rotation speed q, wherein the vehicle-mounted laser radar rotates from the current scanning position to the calibration position within the first scanning time period t1, wherein the calibration position is directly in front of the optical axis of the vehicle-mounted camera; the second scanning time period t f, where the vehicle-mounted camera is in the second scanning time period t f The vehicle-mounted laser radar scans a frame of image from the first line to the last line in the second scanning time period t f The rotation angle within is the viewing angle range H; according to the first scanning time period t1 and the second scanning time period t f Determine the start reading time t0.

[0127] Specifically, assuming that the vehicle-mounted laser radar rotates clockwise, its rotation angle gradually increases as it rotates, and the angle returns to zero after a full circle. The first scanning time period t1 required for the vehicle-mounted laser radar to rotate from the current scanning position α to the calibration position β can be calculated first. If α>β, then:

[0128] t1=(360+β-α) / (q*360)

[0129] If α≤β, then:

[0130] t1=(β-α) / (q*360)

[0131] The second scanning time period t f Also known as the single-frame reading time, it is the time period from the first line of the image to the last line of the image (the row here refers to the row scanned line by line in the scanning mode of the vehicle-mounted camera, which is equivalent to the column in the rotating vehicle-mounted camera's viewing angle). It is also the time required for the vehicle-mounted laser radar to rotate through the viewing angle range H of the vehicle-mounted camera. Among them,

[0132] t f =H / (q*360)

[0133] At this time, the first scanning time period t1 and the second scanning time period t f Determine the start reading time t0, where,

[0134]

[0135] At time t0, the on-board camera is controlled to start reading the first column on the left side of the image. At this time, the on-board laser radar starts scanning from the same position, which means that the on-board camera and the on-board laser radar scan data are synchronized in time. The timing diagram of the synchronous scanning of the two sensors is shown in Figure 5 As shown, each angle and time corresponds to the above calculation results, and t2 corresponds to the time period of the vehicle-mounted laser radar rotating from the initial position (0 degrees) to the current scanning position, which has no effect on the above calculation process.

[0136] The scanning process of the vehicle-mounted camera includes the exposure process and the reading process. The timing diagram when it works in the maximum frame rate mode is as follows: Figure 6As shown, the reading operation cannot be performed in parallel on different rows, and the reading operation of the second row must wait for the completion of the reading operation of the first row, while the exposure operation can be performed in parallel between different rows, that is, they can overlap in time. Therefore, the frame rate of the vehicle-mounted camera is actually determined by the reading operation. When the reading operation of the first row starts and the reading operation of the last row is completed, the first frame of the image is scanned, and then the reading operation starts with the first row of the next frame of the image, and the cycle continues. The exposure operation is completed before the reading. After the exposure is completed, there may be a short time to wait for the reading operation of the row of data to start.

[0137] It can be seen that the starting reading time t0 of the vehicle-mounted camera cannot be directly controlled. Therefore, the starting exposure time t e , through the initial exposure time t e Realize the control of the starting reading time t0.

[0138] In some optional embodiments of the present application, the number of rows n to be scanned by the vehicle-mounted camera when scanning a frame of image can be determined according to the resolution of the vehicle-mounted camera; f The third scanning time period t of the vehicle-mounted camera is determined by the number of rows n r , where the vehicle-mounted camera is in the third scanning time period t r The reading process of one line of the image is completed within t0; according to the starting reading time t0, the second scanning time period t f and the third scanning period t r Determine the start exposure time t e .

[0139] Specifically, the third scanning time period t r Also known as the single-line reading time, it is the time required for the vehicle-mounted camera to complete the reading process of a line in the image.

[0140] t r =t f / n

[0141] Usually, the on-board camera can modify the single-line reading time t by modifying the clock division register r , but t r It cannot be infinitely small. It has a lower limit, that is, the frame rate of the on-board camera has an upper limit. If t is calculated according to the above process, r When the image size is small, only a vehicle-mounted camera with a higher frame rate can apply the multi-sensor synchronization method of the embodiment of the present application.

[0142] Afterwards, the second scanning time period t f and the third scanning period t rDetermine the start exposure time t e ,in,

[0143] t e =t0+t r -t f

[0144] In the actual control process, the vehicle control unit can start the exposure at the start time t e A trigger signal is sent to the vehicle-mounted camera, and the trigger signal is used to control the vehicle-mounted camera to start the exposure process.

[0145] In an embodiment of the present application, the vehicle-mounted control system rotates the vehicle-mounted camera so that the target scanning direction of the vehicle-mounted camera is the same as the scanning direction of the vehicle-mounted laser radar, so as to facilitate data synchronization; thereafter, by obtaining relevant parameters of the vehicle-mounted laser radar and the vehicle-mounted camera, including: a first angle corresponding to the current scanning position of the vehicle-mounted laser radar, a second angle corresponding when the vehicle-mounted laser radar scans to a calibration position, the rotation speed of the vehicle-mounted laser radar, the viewing angle range of the vehicle-mounted camera in the target scanning direction, etc.; the starting scanning time of the vehicle-mounted camera is determined according to the relevant parameters, and the vehicle-mounted camera is controlled to start scanning from the starting scanning time; since the scanning directions of the two are the same and the scanning times correspond to each other, pixel-level synchronization of scanning data between the vehicle-mounted camera and the vehicle-mounted laser radar is achieved, thereby solving the technical problem that it is difficult to achieve accurate synchronization between multiple sensors in the autonomous driving system.

[0146] Example 4

[0147] According to an embodiment of the present application, an electronic device is further provided, the electronic device comprising a processor and a memory, wherein: the memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps:

[0148] Obtain target parameters of the laser radar, where the target parameters include: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding when the laser radar scans to the calibration position, and the rotation speed of the laser radar; obtain the viewing angle range of the camera to be synchronized in the target scanning direction, where the camera to be synchronized scans in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar; determine the starting scanning time of the camera to be synchronized based on the target parameters and the viewing angle range, and control the camera to be synchronized to start scanning from the starting scanning time.

[0149] In some optional embodiments of the present application, the specific structure of the electronic device can refer to Figure 1 The computer terminal shown. It should be noted that Figure 1 The structure shown is for illustration only and does not limit the structure of the electronic device. The structure of the electronic device may be different from that of Figure 1The computer terminal shown may also include Figure 1 More or fewer components are shown, and there may also be Figure 1 Different configurations are shown.

[0150] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0151] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0156] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multi-sensor synchronization method, comprising: Obtain target parameters of the laser radar, wherein the target parameters include: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding to when the laser radar scans to a calibration position, and a rotation speed of the laser radar; Obtaining a viewing angle range of the camera to be synchronized in a target scanning direction, wherein the camera to be synchronized is scanned in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar; Determine a start scanning time of the camera to be synchronized according to the target parameter and the viewing angle range, and control the camera to be synchronized to start scanning from the start scanning time; The scanning process of the camera to be synchronized includes an exposure process and a reading process, and the start scanning time includes a start exposure time and a start reading time; The process of determining the start reading time t0 includes: determining a first scanning time period t1 according to the first angle, the second angle and the rotation speed, wherein the laser radar rotates from the current scanning position to the calibration position within the first scanning time period, and the calibration position is directly in front of the optical axis of the camera to be synchronized; determining a second scanning time period t1 according to the viewing angle range and the rotation speed. f , wherein the camera to be synchronized completes scanning of a frame of image from the first row to the last row in the second scanning time period, and the rotation angle of the laser radar in the second scanning time period is the viewing angle range; the start reading time is determined according to the first scanning time period and the second scanning time period The starting exposure time t e The determination process includes: determining the number of rows to be scanned when the camera to be synchronized scans a frame of image according to the resolution of the camera to be synchronized; determining the third scanning time period t of the camera to be synchronized according to the second scanning time period and the number of rows. r , wherein the camera to be synchronized completes the reading process of a row in the image within the third scanning time period; according to the start reading time, the second scanning time period and the third scanning time period determine the start exposure time t e =t0+t r -t f .

2. The method according to claim 1, wherein: Obtaining target parameters of the laser radar, including: Acquire a data packet of the laser radar through a control unit, and parse the data packet to obtain the first angle and the rotation speed; The laser radar and the camera to be synchronized are calibrated to determine a second angle corresponding to when the laser radar scans to the front of the optical axis of the camera to be synchronized.

3. The method according to claim 1, wherein: Controlling the camera to be synchronized to start scanning from the start scanning time, comprising: A control unit sends a trigger signal to the camera to be synchronized at the start exposure time, and the trigger signal is used to control the camera to be synchronized to start the exposure process.

4. The method according to any one of claims 1 to 3, wherein: The method further comprises: When it is detected that the laser radar stops rotating, the camera to be synchronized is controlled to stop scanning through a control unit; wherein the control unit includes one of the following: a single chip microcomputer, an ARM processor, and a field programmable gate array.

5. A multi-sensor synchronization device, comprising: A first acquisition module is used to acquire target parameters of the laser radar, wherein the target parameters include: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding to when the laser radar scans to a calibration position, and a rotation speed of the laser radar; A second acquisition module is used to acquire the viewing angle range of the camera to be synchronized in the target scanning direction, wherein the camera to be synchronized is scanned in a line-by-line exposure manner, and the target scanning direction is the same as the scanning direction of the laser radar; A control module, used to determine the start scanning time of the camera to be synchronized according to the target parameter and the viewing angle range, and control the camera to be synchronized to start scanning from the start scanning time; wherein the scanning process of the camera to be synchronized includes an exposure process and a reading process, and the start scanning time includes a start exposure time and a start reading time; the process of determining the start reading time t0 includes: determining a first scanning time period t1 according to the first angle, the second angle and the rotation speed, wherein the laser radar rotates from the current scanning position to the calibration position within the first scanning time period, and the calibration position is directly in front of the optical axis of the camera to be synchronized; determining a second scanning time period t1 according to the viewing angle range and the rotation speed f , wherein the camera to be synchronized completes scanning of a frame of image from the first row to the last row in the second scanning time period, and the rotation angle of the laser radar in the second scanning time period is the viewing angle range; the start reading time is determined according to the first scanning time period and the second scanning time period The starting exposure time t e The determination process includes: determining the number of rows to be scanned when the camera to be synchronized scans a frame of image according to the resolution of the camera to be synchronized; determining the third scanning time period t of the camera to be synchronized according to the second scanning time period and the number of rows. r , wherein the camera to be synchronized completes the reading process of a row in the image within the third scanning time period; according to the start reading time, the second scanning time period and the third scanning time period determine the start exposure time t e =t0+t r -t f 。 6. A vehicle-mounted control system, comprising: A vehicle-mounted camera, configured to scan in a target scanning direction in a line-by-line exposure manner, wherein the target scanning direction is the same as a scanning direction of the vehicle-mounted laser radar; The vehicle-mounted laser radar is used to perform synchronous scanning with the vehicle-mounted camera; A vehicle-mounted control unit, used to obtain target parameters of the vehicle-mounted laser radar, wherein the target parameters include: a first angle corresponding to the current scanning position of the vehicle-mounted laser radar, a second angle corresponding to when the vehicle-mounted laser radar scans to the calibration position, and a rotation speed of the vehicle-mounted laser radar; obtaining the viewing angle range of the vehicle-mounted camera in the target scanning direction; determining the starting scanning time of the vehicle-mounted camera according to the target parameters and the viewing angle range, and controlling the vehicle-mounted camera to scan from the starting scanning time; wherein the scanning process of the camera to be synchronized includes an exposure process and a reading process, and the starting scanning time includes a starting exposure time and a starting reading time; the process of determining the starting reading time t0 includes: determining a first scanning time period t1 according to the first angle, the second angle and the rotation speed, wherein the laser radar rotates from the current scanning position to the calibration position within the first scanning time period, and the calibration position is directly in front of the optical axis of the camera to be synchronized; determining a second scanning time period t1 according to the viewing angle range and the rotation speed f , wherein the camera to be synchronized completes scanning of a frame of image from the first row to the last row in the second scanning time period, and the rotation angle of the laser radar in the second scanning time period is the viewing angle range; the start reading time is determined according to the first scanning time period and the second scanning time period The starting exposure time t e The determination process includes: determining the number of rows to be scanned when the camera to be synchronized scans a frame of image according to the resolution of the camera to be synchronized; determining the third scanning time period t of the camera to be synchronized according to the second scanning time period and the number of rows. r , wherein the camera to be synchronized completes the reading process of a row in the image within the third scanning time period; according to the start reading time, the second scanning time period and the third scanning time period determine the start exposure time t e =t0+t r -t f .

7. An electronic device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Obtain target parameters of the laser radar, wherein the target parameters include: a first angle corresponding to the current scanning position of the laser radar, a second angle corresponding to when the laser radar scans to the calibration position, and a rotation speed of the laser radar; obtain the viewing angle range of the camera to be synchronized in the target scanning direction, wherein the camera to be synchronized is scanned in a line-by-line exposure mode, and the target scanning direction is the same as the scanning direction of the laser radar; determine the starting scanning time of the camera to be synchronized according to the target parameters and the viewing angle range, and control the camera to be synchronized to start scanning from the starting scanning time; wherein the scanning process of the camera to be synchronized includes an exposure process and a reading process, and the starting scanning time includes a starting exposure time and a starting reading time; the process of determining the starting reading time t0 includes: determining a first scanning time period t1 according to the first angle, the second angle and the rotation speed, wherein the laser radar rotates from the current scanning position to the calibration position within the first scanning time period, and the calibration position is directly in front of the optical axis of the camera to be synchronized; determining a second scanning time period t1 according to the viewing angle range and the rotation speed f , wherein the camera to be synchronized completes scanning of a frame of image from the first row to the last row in the second scanning time period, and the rotation angle of the laser radar in the second scanning time period is the viewing angle range; the start reading time is determined according to the first scanning time period and the second scanning time period The starting exposure time t e The determination process includes: determining the number of rows to be scanned when the camera to be synchronized scans a frame of image according to the resolution of the camera to be synchronized; determining the third scanning time period t of the camera to be synchronized according to the second scanning time period and the number of rows. r , wherein the camera to be synchronized completes the reading process of a row in the image within the third scanning time period; according to the start reading time, the second scanning time period and the third scanning time period determine the start exposure time t e =t0+t r -t f .

Images