Method and device for synchronizing fields of view between different devices and autonomous driving vehicle

Abstract

The present disclosure provides a method, apparatus, and autonomous driving vehicle for synchronizing the fields of view between different devices, relating to the field of data processing, and more particularly, to the field of autonomous driving. A specific implementation scheme comprises: determining the time deviation generated when the scanning device scans a preset angle based on the performance parameters of the scanning device; adjusting the initial acquisition time of the next frame of image captured by the camera based on the time deviation and the current acquisition time of the current frame of image captured by the camera to obtain a target acquisition time; obtaining the scanned image obtained when the scanning device scans the preset angle next time, and the next frame of image captured by the camera at the target acquisition time; and synchronizing the fields of view of the scanned image and the next frame of image.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, in particular to the field of autonomous driving, and specifically to a method and apparatus for synchronizing fields of view between different devices and an autonomous driving vehicle. Background Art

[0002] Autonomous driving systems enable unmanned driving, freeing people's hands and making life and work more convenient. Autonomous driving systems perceive the outside world through various sensors, with scanning devices (e.g., radar) and imaging devices (e.g., cameras) being common sensors used in autonomous driving systems.

[0003] In autonomous driving systems, different sensors are usually required to work together. For example, for scanning devices and shooting devices, the data collected by these two sensors in the same field of view at the same time needs to be transmitted to the perception model. Therefore, the field of view of the scanning device and the shooting device needs to be synchronized.

[0004] However, in the related art, usually, the scanning device sends the point cloud data message of the scanning device to the field of view synchronization program through Ethernet, switches, etc., so that the field of view synchronization program reads the point cloud data message of the scanning device to achieve the field of view synchronization of the scanning device and the shooting device, such as Figure 1 However, as the number of point clouds generated by the scanning device increases, as well as the number and accuracy of the scanning lines, the synchronization of the field of view between the scanning device and the camera will cause a large amount of system resource overhead, thereby reducing the system performance of the autonomous vehicle. Summary of the Invention

[0005] The present disclosure provides a method and apparatus for synchronizing fields of view between different devices, and an autonomous driving vehicle.

[0006] According to one aspect of the present disclosure, a method for synchronizing the field of view between different devices is provided, which is applied to a vehicle, where at least a scanning device and a shooting device are installed. The method includes: determining a time deviation generated when the scanning device scans a preset angle based on performance parameters of the scanning device; adjusting an initial acquisition time for the shooting device to acquire a next frame of image based on the time deviation and a current acquisition time at which the shooting device acquires a current frame of image, to obtain a target acquisition time; obtaining a scanned image obtained when the scanning device scans the preset angle next time, and a next frame of image acquired by the shooting device at the target acquisition time; and synchronizing the field of view of the scanned image and the next frame of image.

[0007] According to another aspect of the present disclosure, there is provided an apparatus for synchronizing the fields of view between different devices, which is applied to a vehicle, on which at least a scanning device and a photographing device are installed. The apparatus comprises: a deviation determination module, for determining, based on the performance parameters of the scanning device, a time deviation generated when the scanning device scans a preset angle; a time adjustment module, for adjusting, based on the time deviation and the current acquisition moment at which the photographing device acquires the current frame image, an initial acquisition moment at which the photographing device acquires the next frame image, to obtain a target acquisition moment; an image acquisition module, for acquiring a scanned image obtained when the scanning device scans the preset angle next time, and a next frame image acquired by the photographing device at the target acquisition moment; and a field of view synchronization module, for synchronizing the fields of view of the scanned image and the next frame image.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned method of synchronizing fields of view between different devices.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the above-mentioned method for synchronizing fields of view between different devices.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above-mentioned method for synchronizing fields of view between different devices when executed by a processor.

[0011] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising at least a scanning device and a shooting device, and a processor, wherein the processor is configured to synchronize the fields of view of the scanning device and the shooting device according to the above-mentioned method.

[0012] Based on the above content, it can be seen that the present disclosure uses the time deviation of the scanning device to adjust the acquisition time of the next frame of image by the shooting device, so as to realize the field of view synchronization between the scanning device and the shooting device. Because in the present disclosure, in the process of synchronizing the field of view of the scanning device and the shooting device, there is no need to read the point cloud message of the scanning device. Therefore, even if the amount of point cloud message data of the scanning device increases, the improvement of the number of scanning lines and accuracy requirements of the scanning device will not increase the system overhead, thereby improving the system performance of the autonomous driving vehicle.

[0013] It can be seen that the solution provided by the present disclosure achieves the purpose of synchronizing the field of view of the scanning device and the shooting device, thereby reducing the system overhead of the vehicle system, and thus avoiding the problem of high system overhead in the process of synchronizing the field of view of the scanning device and the shooting device in the related technology.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0016] Figure 1 This is a schematic diagram of field of view synchronization between a scanning device and a photographing device in the related art;

[0017] Figure 2 is a schematic diagram of synchronization between the laser radar and the camera field of view according to the first embodiment of the present disclosure;

[0018] Figure 3 is a flowchart of a method for synchronizing fields of view between different devices according to the first embodiment of the present disclosure;

[0019] Figure 4 is a schematic diagram of adjusting the acquisition time of the next frame of image according to the first embodiment of the present disclosure;

[0020] Figure 5 is a schematic diagram of an apparatus for synchronizing fields of view between different devices according to a third embodiment of the present disclosure;

[0021] Figure 6 The present invention is a block diagram of an electronic device for implementing the method for synchronizing fields of view between different devices according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] Example 1

[0024] According to one aspect of the present disclosure, a method for synchronizing the field of view between different devices is provided. The method can be applied to a vehicle, where at least a scanning device and a shooting device are installed. The scanning device can be, but is not limited to, a radar, such as a lidar; the shooting device can be an image acquisition device, such as a camera.

[0025] It should be noted that in this example, a laser radar is used as the scanning device and a camera is used as the image capturing device. Due to their operating principles, both the camera and the laser radar require a certain amount of time to generate each frame. Therefore, it is necessary to ensure that the triggering moment of each camera frame image is aligned with the moment when the laser radar scan line rotates to a specific angle to achieve the best field of view fusion effect.

[0026] Optional, Figure 2 The schematic diagram of the synchronization between the laser radar and the camera field of view is shown in FIG. Figure 2 As shown, based on the relationship between the camera's image frame frequency and the lidar's rotation frequency, the lidar is used as a reference (because changing its rotational phase is difficult). The deviation between the moment the lidar's scan line passes through 0 degrees and the exposure point at the center of the camera's field of view is calculated. This deviation is used to adjust the camera's trigger time for capturing the next frame. This trigger time is then adjusted by a controller, such as an FPGA (Field Programmable Gate Array). By repeating this process, the field of view of the two sensors can be synchronized at any time.

[0027] In an autonomous driving vehicle, it is necessary to ensure that other vehicles passing at a preset distance (for example, 30 meters) in front of the current vehicle and at a preset speed (30 km / h) perpendicular to the front of the vehicle, when passing through the center of the camera's field of view facing forward, have a difference in the left and right directions between the current vehicle's lidar point cloud projection and the current vehicle's position in the camera image that is less than a preset pixel point (for example, 10 pixels).

[0028] In order to achieve field of view synchronization between a scanning device and a shooting device, this embodiment provides a method for synchronizing the field of view between different devices, wherein: Figure 3 is a flow chart of the method for synchronizing the fields of view between different devices, such as Figure 3 As shown, the method includes the following steps:

[0029] In step S302 , the time deviation generated when the scanning device scans a preset angle is determined based on the performance parameters of the scanning device.

[0030] In step S302, the scanning device may be a laser radar, whose performance parameters include at least the laser radar's rotational speed. The preset angle may be, but is not limited to, 0 degrees. Furthermore, the aforementioned time offset is a fixed offset, meaning that when the laser radar's scan line scans 0 degrees, there will be a fixed time offset. This time offset is related to the scanning device's performance parameters and the vehicle's system performance consumption. In other words, given a fixed laser radar rotational speed and a fixed vehicle system performance consumption, the time offset generated when the laser radar scans 0 degrees is fixed.

[0031] Step S304 : adjusting the initial acquisition time for the next frame of image acquired by the camera based on the time deviation and the current acquisition time of the current frame of image acquired by the camera to obtain a target acquisition time.

[0032] In step S304, the capturing device may be a camera, which may be a forward-facing camera of the vehicle. Optionally, after determining the frame rate of the camera's image capture and the current capture time of the current frame, the controller may determine the initial capture time of the next frame of the camera based on the frame rate and the current capture time. That is, the initial capture time of the next frame of the camera is determined based on the camera's frame rate.

[0033] However, since it is difficult for the laser radar to change the rotation phase, in order to achieve synchronization between the laser radar and the camera field of view, the acquisition time of the camera image needs to be adjusted.

[0034] Optional, Figure 4 A schematic diagram showing an optional method for adjusting the acquisition time of the next frame of image is shown. Figure 4 It can be seen that in this embodiment, after obtaining the time deviation, the synchronization program transmits the time deviation to the controller through the PCIE (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) driver, so that the controller adjusts the acquisition time of the camera to capture the next frame of image according to the time deviation.

[0035] It should be noted that in the process of adjusting the acquisition time of the next frame of image by the camera, there is no need to read and parse the point cloud data message of the radar, which saves the resources of the vehicle's processor.

[0036] Step S306 , obtaining the scanned image obtained when the scanning device scans the preset angle next time, and the next frame of image captured by the photographing device at the target capture time.

[0037] Optional, such as Figure 4As shown in the figure, after determining the target capture time for the camera to capture the next frame, the camera captures the image at the target capture time, obtains the next frame, and transmits the captured next frame to the controller via GMSL (Gigabit Multimedia Serial Links). The controller transmits the next frame to the camera's software module via the PCIE driver. At the same time, the controller also obtains the scanned image obtained when the LiDAR scans at a preset angle to synchronize the field of view of the next frame with the scanned image.

[0038] Step S308: synchronize the scanned image and the next frame image with the field of view.

[0039] In step S308, Figure 4 As shown, the synchronization program can obtain the next frame image captured by the camera and the scanned image scanned by the lidar through the PCIE driver, and perform image fusion (for example, image superposition) on the scanned image and the next frame image to achieve field of view synchronization between the scanned image and the next frame image.

[0040] Based on the scheme defined in steps S302 to S308 above, it can be seen that the present disclosure utilizes the time offset of the scanning device to adjust the acquisition time of the next frame of image captured by the camera device to achieve field of view synchronization between the scanning device and the camera device. Based on the performance parameters of the scanning device, the time offset generated when the scanning device scans a preset angle is determined. Based on the time offset and the current acquisition time of the current frame of image captured by the camera device, the initial acquisition time of the next frame of image captured by the camera device is adjusted to obtain a target acquisition time. Then, the scanned image obtained when the scanning device scans the preset angle next time and the next frame of image captured by the camera device at the target acquisition time are obtained, and the field of view synchronization of the scanned image and the next frame of image is performed.

[0041] It is easy to notice that, since in the present disclosure, there is no need to read the point cloud message of the scanning device during the process of synchronizing the field of view of the scanning device and the shooting device, even if the amount of point cloud message data of the scanning device increases, the improvement of the number of scanning lines and accuracy requirements of the scanning device will not increase the system overhead, thereby improving the system performance of the autonomous driving vehicle.

[0042] It can be seen that the solution provided by the present disclosure achieves the purpose of synchronizing the field of view of the scanning device and the shooting device, thereby reducing the system overhead of the vehicle system, and thus avoiding the problem of high system overhead in the process of synchronizing the field of view of the scanning device and the shooting device in the related technology.

[0043] Example 2

[0044] According to one aspect of the present disclosure, a method for synchronizing the field of view between different devices is also provided. Figure 3 The provided method roadmap for synchronizing the fields of view between different devices explains each step mentioned in Example 1 in detail.

[0045] In an optional embodiment, before synchronizing the field of view of the scanning device and the shooting device, the time deviation when the scanning device scans the preset angle needs to be determined according to the characteristics of the scanning device, that is, before synchronizing the field of view of the scanning device and the shooting device, step S302 needs to be executed.

[0046] Specifically, a synchronization program is deployed in the vehicle's processor, which can determine the rotation speed of the scanning device and the system performance consumption of the vehicle from the performance parameters of the scanning device, and then determine the time deviation based on the rotation speed of the scanning device and the system performance consumption.

[0047] It should be noted that the above time deviation is a fixed time deviation for the scanning device. That is, when the scanning device's rotational speed and the vehicle's system performance consumption are fixed, the time deviation for the scanning device to scan 0 degrees is fixed. Optionally, system performance consumption is positively correlated with the time deviation. That is, the greater the system performance consumption, the greater the time deviation of the lidar.

[0048] In addition, from the above content, it can be seen that the time deviation is only related to the rotation speed of the scanning device and the system performance consumption of the vehicle, and has nothing to do with the point cloud data message of the scanning device. That is, as the number of scanning lines and the accuracy of the scanning device increase, the time deviation will not change. Furthermore, in the process of using the time deviation to achieve field of view synchronization between the scanning device and the shooting device, as the number of scanning lines and the accuracy of the scanning device increase, the processor overhead is always 0 and will not increase, thereby reducing the system overhead during the field of view synchronization between the scanning device and the shooting device.

[0049] Furthermore, after determining the time offset of the scanning device, the processor can adjust the initial acquisition time of the next image frame acquired by the camera based on the time offset and the current acquisition time of the current image frame acquired by the camera to obtain the target acquisition time. Prior to this, the processor must determine the initial acquisition time of the next image frame acquired by the camera.

[0050] Specifically, the processor obtains the current acquisition time and the frame rate of the image captured by the shooting device, and determines the initial acquisition time for the shooting device to capture the next frame of image based on the frame rate and the current acquisition time.

[0051] Optional, in Figure 4In [1], the synchronization program can read and analyze the current frame image captured by the camera to obtain the current capture time of the current frame image. In addition, if the camera captures images at a frame rate of 10fps, the synchronization program can predict the initial capture time of the next frame image based on the current capture time. For example, if the current capture time is 0:00:00, the initial capture time of the next frame image is 0:00:01.

[0052] It should be noted that the acquisition time of the next frame image is determined by the acquisition time of the current frame image, and then the acquisition time of the next frame image is adjusted to achieve field of view synchronization between the scanning device and the shooting device. This process forms a closed-loop control, realizes automatic synchronization of the field of view between the scanning device and the shooting device, and improves synchronization efficiency.

[0053] Further, such as Figure 3 As shown, after determining the initial acquisition time of the next frame image, the synchronization program executes step S304, that is, adjusting the initial acquisition time of the next frame image acquired by the shooting device based on the time deviation and the current acquisition time of the current frame image acquired by the shooting device to obtain the target acquisition time.

[0054] Specifically, after obtaining the current scanning moment when the scanning device scans the preset angle, the difference between the current scanning moment and the current acquisition moment is calculated to obtain the time difference. Then, the initial acquisition moment is adjusted based on the time difference and the time deviation to obtain the adjusted duration, and the sum of the adjusted duration and the initial acquisition moment is calculated to obtain the target acquisition moment.

[0055] It should be noted that the scanning device is a laser radar. Among them, the laser radar has the characteristic of aligning the moment of scanning a specific angle with an integer second in the real world. For example, if the rotation speed of the laser radar is 10Hz, then when the laser radar scan line passes through an angle of 0 degrees, the corresponding relative times are 0, 100ms, and 200ms respectively. However, in practice, due to time synchronization and system errors, there is a fixed deviation (i.e., time deviation) when the laser radar scans an angle of 0 degrees, and this deviation can be obtained through actual measurement.

[0056] In addition, the synchronization program can obtain the triggering time of the current frame image (i.e. the current acquisition time) T by reading the time in the camera image frame. cam , at this fixed deviation T ref When the value is known, the adjustment value (i.e., adjustment duration) C for adjusting the triggering moment (i.e., initial acquisition moment) of the next frame of image can be calculated by the following linear function f:

[0057] C=f(T ref ,T cam )

[0058] After obtaining the above adjustment time C, the synchronization program can adjust the acquisition time of the next frame of image by calculating the sum of the adjustment time and the initial acquisition time of the next frame of image.

[0059] From the above, it can be seen that in this embodiment, in the process of adjusting the acquisition time of the next frame of image by the shooting device, there is no need to read the point cloud message of the scanning device. Therefore, even if the amount of point cloud message data of the scanning device increases, the improvement of the number of scanning lines and accuracy requirements of the scanning device will not increase the system overhead, thereby improving the system performance of the autonomous driving vehicle.

[0060] In an optional embodiment, before obtaining the current scanning time when the scanning device currently scans the preset angle, the processor receives a scanning instruction for controlling the scanning device to scan, and detects, based on the scanning instruction, whether the scanning device is scanning an image for the first time. When determining that the scanning device is scanning the image for the first time, the scanning time when the scanning device first scans the preset angle is synchronized with the vehicle's system clock to obtain the first scanning time when the scanning device first scans the preset angle. When determining that the scanning device is not scanning the image for the first time, the scanning time when the scanning device currently scans the preset angle is determined based on the first scanning time.

[0061] Optionally, when the scanning device first scans an image, the processor initializes the scanning device to establish a relationship between the time when the scanning device first scans the 0-degree angle and universal time, thereby determining the universal time when the scanning device first scans the 0-degree angle. When the scanning device scans the 0-degree angle again, the processor can determine the universal time when the scanning device scans the 0-degree angle again based on the scanning device's rotation speed and the time when the scanning device first scans the 0-degree angle.

[0062] It should be noted that in the present disclosure, there is no need to read and parse the point cloud data message of the scanning device. The world time of the scanning device's current scan at a 0-degree angle can be determined only by the rotation speed of the scanning device. This process is unrelated to the number of scanning lines and accuracy of the scanning device. Therefore, even if the amount of point cloud message data of the scanning device increases and the requirements for the number of scanning lines and accuracy of the scanning device are increased, the system overhead will not increase, thereby improving the system performance of the autonomous driving vehicle.

[0063] Furthermore, Figure 3 As shown, after adjusting the initial capture time of the next frame of image captured by the camera, the processor can synchronize the field of view of the scanned image and the next frame of image. Specifically, the processor determines the target scanning time corresponding to the scanned image and the target capture time corresponding to the next frame of image. When the target scanning time and the target capture time match, the processor superimposes the scanned image and the next frame of image to obtain a synchronized result.

[0064] It should be noted that by superimposing the images collected by the scanning device and the shooting device at the same time and in the same field of view, the field of view synchronization of the images collected by different devices at the same time and in the same field of view can be achieved. In the field of autonomous driving, preparations for obstacle avoidance can be achieved, thereby improving the safety of autonomous driving vehicles.

[0065] As can be seen from the above, the present disclosure utilizes the characteristics of LiDAR to simplify the synchronization process while maintaining the same synchronization accuracy. This field of view synchronization process does not require parsing LiDAR data packets, saving processor resources consumed thereby. Moreover, as the number of scan lines and accuracy of LiDAR increase, the processor overhead does not increase, thus saving a large amount of processor resources. In addition, in the present disclosure, by calculating the adjustment duration of the moment when the camera captures the current image, and incorporating this adjustment duration into the adjustment of the capture time of the next frame of image, a closed-loop control of the field of view synchronization between LiDAR and camera is formed in this way.

[0066] Example 3

[0067] According to one aspect of the present disclosure, there is also provided a device for synchronizing the fields of view between different devices, the device being applied to a vehicle, wherein at least a scanning device and a photographing device are installed on the vehicle, wherein: Figure 5 is a schematic diagram of a device for synchronizing the fields of view between different devices, such as Figure 5 As shown, the device includes: a deviation determination module 501, a time adjustment module 503, an image acquisition module 505 and a field of view synchronization module 507.

[0068] The weight and deviation determination module 501 is used to determine the time deviation generated when the scanning device scans a preset angle based on the performance parameters of the scanning device; the time adjustment module 503 is used to adjust the initial acquisition time of the next frame image acquired by the shooting device based on the time deviation and the current acquisition time of the current frame image acquired by the shooting device to obtain the target acquisition time; the image acquisition module 505 is used to acquire the scanned image obtained when the scanning device scans the preset angle next time, and the next frame image acquired by the shooting device at the target acquisition time; the field of view synchronization module 507 is used to synchronize the field of view of the scanned image and the next frame image.

[0069] It should be noted here that the above-mentioned deviation determination module 501, time adjustment module 503, image acquisition module 505 and field of view synchronization module 507 correspond to steps S302 to S308 of the above-mentioned embodiment. The four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment.

[0070] Optionally, the deviation determination module includes: a parameter determination module and a first determination module. The parameter determination module is configured to determine the rotation speed of the scanning device and the system performance consumption of the vehicle from performance parameters of the scanning device; and the first determination module is configured to determine the time deviation based on the rotation speed of the scanning device and the system performance consumption.

[0071] Optionally, the apparatus for synchronizing fields of view between different devices further includes: a first acquisition module and a second determination module. The first acquisition module is configured to adjust the initial acquisition time for the next frame of image captured by the camera based on the time offset and the current acquisition time of the current frame of image captured by the camera, and to obtain the current acquisition time and the frame rate of the image captured by the camera before obtaining the target acquisition time. The second determination module is configured to determine the initial acquisition time for the next frame of image captured by the camera based on the frame rate and the current acquisition time.

[0072] Optionally, the time adjustment module includes: a second acquisition module, a first calculation module, a first adjustment module, and a second calculation module. The second acquisition module is configured to acquire the current scanning time when the scanning device scans the preset angle; the first calculation module is configured to calculate the difference between the current scanning time and the current acquisition time to obtain a time difference; the first adjustment module is configured to adjust the initial acquisition time based on the time difference and the time deviation to obtain an adjusted duration; and the second calculation module is configured to calculate the sum of the adjusted duration and the initial acquisition time to obtain a target acquisition time.

[0073] Optionally, the apparatus for synchronizing the fields of view between different devices further includes: an instruction receiving module, a detection module, a clock synchronization module, and a third determination module. The instruction receiving module is configured to receive a scanning instruction for controlling the scanning device to scan before obtaining the current scanning moment when the scanning device scans the preset angle; the detection module is configured to detect, based on the scanning instruction, whether the scanning device is scanning an image for the first time; the clock synchronization module is configured to synchronize the scanning moment when the scanning device first scans the preset angle based on the vehicle's system clock when determining that the scanning device has scanned the image for the first time, thereby obtaining the first scanning moment when the scanning device first scans the preset angle; and the third determination module is configured to determine, based on the first scanning moment, the scanning moment when the scanning device currently scans the preset angle when determining that the scanning device has not scanned the image for the first time.

[0074] Optionally, the field of view synchronization module includes: a fourth determination module and an overlay module. The fourth determination module is configured to determine a target scanning moment corresponding to the scanned image and a target acquisition moment corresponding to the next frame of image; and the overlay module is configured to overlay the scanned image and the next frame of image to obtain a synchronization result when the target scanning moment matches the target acquisition moment.

[0075] Example 4

[0076] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising at least a scanning device and a shooting device, and a processor, wherein the processor is used to synchronize the field of view of the scanning device and the shooting device according to the methods of the above-mentioned embodiments 1 and 2.

[0077] Optionally, the scanning device may be but is not limited to a radar, for example, a lidar; the shooting device may be but is not limited to a camera, wherein the camera may be a camera deployed with a CCD (Charged Coupled Device) image sensor, a camera deployed with a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a camera deployed with other image sensors.

[0078] In addition, the controller in this embodiment can execute the solutions provided in the above-mentioned embodiments 1 and 2. The relevant contents have been explained in embodiments 1 and 2 and will not be repeated here.

[0079] Example 5

[0080] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0081] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0082] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0083] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0084] The computing unit 601 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for synchronizing the field of view between different devices. For example, in some embodiments, the method for synchronizing the field of view between different devices can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for synchronizing the field of view between different devices described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured in any other appropriate manner (eg, by means of firmware) to execute the method for synchronizing fields of view between different devices.

[0085] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0089] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0090] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0091] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0092] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for synchronizing the field of view between different devices, applied to a vehicle, wherein at least a scanning device and a photographing device are installed on the vehicle, wherein: The method comprises: Determining the time deviation generated when the scanning device scans a preset angle based on the performance parameters of the scanning device; Adjusting the initial acquisition time for the next frame of image acquired by the photographing device based on the time deviation and the current acquisition time of the current frame of image acquired by the photographing device to obtain a target acquisition time; Acquire a scanned image obtained when the scanning device scans the preset angle next time, and a next frame of image captured by the photographing device at the target capture time; Performing field synchronization on the scanned image and the next frame image; Among them, based on the performance parameters of the scanning device, the time deviation generated when the scanning device scans a preset angle is determined, including: determining the rotation speed of the scanning device and the system performance consumption of the vehicle from the performance parameters of the scanning device; and determining the time deviation based on the rotation speed of the scanning device and the system performance consumption.

2. The method according to claim 1, before adjusting the initial acquisition time of the next frame of image acquired by the camera based on the time offset and the current acquisition time of the current frame of image acquired by the camera to obtain the target acquisition time, the method further comprises: Obtaining the current acquisition time and the frame rate of the image captured by the shooting device; Based on the frame rate and the current acquisition moment, an initial acquisition moment for the photographing device to acquire the next frame of image is determined.

3. The method according to claim 2, wherein: Adjusting the initial acquisition time for the next frame of image acquired by the photographing device based on the time deviation and the current acquisition time of the current frame of image acquired by the photographing device to obtain a target acquisition time includes: Obtaining a current scanning moment when the scanning device scans the preset angle this time; Calculating the difference between the current scanning time and the current acquisition time to obtain a time difference; Adjusting the initial acquisition time based on the time difference and the time deviation to obtain an adjusted duration; The sum of the adjusted time and the initial collection time is calculated to obtain the target collection time.

4. The method according to claim 3, before obtaining the current scanning time when the scanning device scans the preset angle, the method further comprises: receiving a scanning instruction for controlling the scanning device to scan; detecting, based on the scanning instruction, whether the scanning device is scanning an image for the first time; When determining that the scanning device scans the image for the first time, performing clock synchronization on the scanning time when the scanning device first scans the preset angle based on the system clock of the vehicle to obtain the first scanning time when the scanning device first scans the preset angle; When it is determined that this is not the first time that the scanning device scans the image, a scanning moment at which the scanning device currently scans the preset angle is determined based on the first scanning moment.

5. The method according to claim 4, wherein Performing field synchronization on the scanned image and the next frame image includes: Determining a target scanning time corresponding to the scanned image and a target acquisition time corresponding to the next frame of image; When the target scanning moment matches the target acquisition moment, the scanned image and the next frame image are superimposed to obtain a synchronization result.

6. A device for synchronizing the fields of view between different devices, used in a vehicle, wherein at least a scanning device and a photographing device are installed on the vehicle, wherein: The device comprises: a deviation determination module, configured to determine a time deviation generated when the scanning device scans a preset angle based on the performance parameters of the scanning device; a time adjustment module, configured to adjust an initial acquisition time for the next frame of image acquired by the photographing device based on the time deviation and a current acquisition time at which the photographing device acquires the current frame of image, to obtain a target acquisition time; An image acquisition module, configured to acquire a scanned image obtained when the scanning device scans the preset angle next time, and a next frame of image acquired by the photographing device at the target acquisition moment; A field of view synchronization module, configured to synchronize the field of view of the scanned image and the next frame of image; The deviation determination module is configured to determine the time deviation generated when the scanning device scans a preset angle based on the performance parameters of the scanning device by executing the following steps: determining the rotational speed of the scanning device and the system performance consumption of the vehicle from the performance parameters of the scanning device; and determining the time deviation based on the rotational speed of the scanning device and the system performance consumption.

7. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for synchronizing fields of view between different devices according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method for synchronizing fields of view between different devices according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for synchronizing fields of view between different devices according to any one of claims 1 to 5.

10. An autonomous driving vehicle comprising at least a scanning device, a photographing device, and a processor, wherein: The processor is configured to perform field of view synchronization on the scanning device and the shooting device according to the method according to any one of claims 1 to 5.

Images