Image frame synchronization method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a synchronization method and device of image frames, electronic equipment and storage medium, which relates to the technical fields of artificial intelligence, image processing, automatic driving and the like. The specific implementation scheme is: obtaining a first center point timestamp of an image acquisition device at restart; obtaining a second center point timestamp of a point cloud map acquisition device at restart; determining a timestamp difference value between the image acquisition device and the point cloud map acquisition device under the synchronization constraint condition of the first center point timestamp and the second center point timestamp; and synchronizing the point cloud map acquired by the point cloud map acquisition device and the image of the image acquisition device according to the timestamp difference value. By determining the timestamp difference value, the point cloud map and the image are synchronized, and the center time of the image acquisition device does not need to be calculated and adjusted at each point cloud map acquisition period, which can save system resources and increase the accuracy of point cloud map and image acquisition.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical fields of artificial intelligence, image processing, automatic driving, and the like, and particularly relates to a method and apparatus for synchronizing image frames, an electronic device, and a storage medium. BACKGROUND

[0002] In the current technology, based on the zero-degree angle alignment of the mechanical rotating radar hardware and the camera, the camera needs to recalculate the trigger delay time of the next frame every frame. Thus, the camera frame picture and the laser radar point cloud data frame information are aligned, which facilitates the fusion algorithm of perception and further improves the accuracy of judging obstacles in all time periods and all scenes. SUMMARY

[0003] The present disclosure provides a method and apparatus for synchronizing image frames, an electronic device, and a storage medium.

[0004] According to a first aspect of the present disclosure, a method for synchronizing image frames is provided, comprising: obtaining a first center point timestamp of an image acquisition device at a restart; obtaining a second center point timestamp of a point cloud map acquisition device at the restart; determining a timestamp difference between the image acquisition device and the point cloud map acquisition device under a synchronization constraint condition of the first center point timestamp and the second center point timestamp; and synchronizing a point cloud map acquired by the point cloud map acquisition device and an image of the image acquisition device according to the timestamp difference.

[0005] According to a second aspect of the present disclosure, a device for synchronizing image frames is provided, comprising: a first obtaining module configured to obtain a first center point timestamp of an image acquisition device at a restart; a second obtaining module configured to obtain a second center point timestamp of a point cloud map acquisition device at the restart; a calculation module configured to determine a timestamp difference between the image acquisition device and the point cloud map acquisition device under a synchronization constraint condition of the first center point timestamp and the second center point timestamp; and a synchronization module configured to synchronize a point cloud map acquired by the point cloud map acquisition device and an image of the image acquisition device according to the timestamp difference.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for synchronizing image frames according to the above-mentioned first aspect.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer readable storage medium having computer instructions stored thereon is provided, and the computer instructions are used to make the computer perform the method for synchronizing image frames according to the above-mentioned first aspect.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer programs / instructions which, when executed by a processor, implement the synchronization method of image frames according to the above-mentioned aspect.

[0009] By determining the timestamp difference, the point cloud image and the image are synchronized, without the need to calculate and adjust the center time of the image acquisition device at each point cloud acquisition cycle, which can save system resources and increase the accuracy of point cloud and image acquisition.

[0010] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0012] Figure 1 A flowchart of a synchronization method of image frames provided by an embodiment of the present disclosure is provided;

[0013] Figure 2 A schematic diagram of the time nodes of the image acquisition device and the point cloud acquisition device of the synchronization method of image frames provided by an embodiment of the present disclosure is provided;

[0014] Figure 3 A flowchart of another synchronization method of image frames provided by an embodiment of the present disclosure is provided;

[0015] Figure 4 A flowchart of another synchronization method of image frames provided by an embodiment of the present disclosure is provided;

[0016] Figure 5 A flowchart of another synchronization method of image frames provided by an embodiment of the present disclosure is provided;

[0017] Figure 6 A structural schematic diagram of a synchronization system of image frames provided by an embodiment of the present disclosure is provided;

[0018] Figure 7 A structural schematic diagram of a synchronization device of image frames provided by an embodiment of the present disclosure is provided;

[0019] Figure 8 A block diagram of an electronic device according to the synchronization method of image frames of an embodiment of the present disclosure is provided. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are cited as illustrative examples. Various details of the embodiments of the present disclosure are described herein in order to provide a thorough understanding thereof. It will be understood by those of ordinary skill in the art, however, that various embodiments described herein can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to not obscure the understanding of this description.

[0021] The synchronization method, apparatus and electronic device of image frames of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0022] Artificial intelligence (AI) is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of human life, including both hardware and software technologies. Artificial intelligence hardware technologies generally include computer vision technology, speech recognition technology, natural language processing technology, and learning / deep learning, big data processing technology, knowledge graph technology, etc.

[0023] Image processing technology is a technology that uses a computer to analyze images to achieve desired results. Also known as image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with an industrial camera, a video camera, a scanner, etc. The elements of the array are called pixels, and their values are called gray values. Image processing technology generally includes image compression, enhancement and restoration, matching, description and identification.

[0024] Automatic driving technology uses advanced communication, computer, network and control technologies to realize real-time and continuous control of trains. Modern communication means are used to directly face the train, and two-way data communication between the train and the ground can be realized, with fast transmission rate and large information volume. The subsequent tracking train and control center can learn the exact position of the preceding train in time, making the operation management more flexible, the control more effective, and more suitable for the needs of train automatic driving.

[0025] Figure 1 A flowchart of a synchronization method of image frames provided by the embodiments of the present disclosure is shown.

[0026] As shown in Figure 1 The synchronization method of image frames can include:

[0027] S101, obtaining a first center point timestamp of an image acquisition device at restart.

[0028] In the embodiments of the present disclosure, the image acquisition device can be various, which is not limited here, for example, it can be a camera, a video recorder, a camera, etc.

[0029] When taking a photo, due to the exposure time of the image acquisition device, the processing time of the image, and other reasons, there is a certain imaging delay before the image is imaged. In the embodiment of the present disclosure, the first center point timestamp is the middle time of the first imaging delay when the image acquisition device is restarted.

[0030] It should be noted that the imaging delay of different image acquisition devices can be different, which is determined by the exposure characteristics of the camera itself and the processing speed of the processor, and is not limited here.

[0031] In the embodiment of the present disclosure, Figure 2 The schematic diagram of the time nodes of the image acquisition device and the point cloud map acquisition device of the synchronization method of the image frame provided by the embodiment of the present disclosure is shown in Figure 2 The first center point timestamp can be determined by obtaining the starting acquisition timestamp and the ending acquisition timestamp of the image acquisition device. The first center point timestamp t 第一中心 = (t2-t1) / 2, where t2 is the ending acquisition time of the image acquisition device, and t1 is the starting acquisition timestamp of the image acquisition device.

[0032] It should be noted that the installation position of the image acquisition device and the center position of the field of view angle are set in advance, and can be changed according to the actual design needs, and are not limited here.

[0033] S102, obtaining the second center point timestamp of the point cloud map acquisition device when restarted.

[0034] In the embodiment of the present disclosure, the point cloud map acquisition device can be various, which is not limited here. For example, the point cloud map acquisition device can be a radar, a point cloud scanner, etc.

[0035] It should be noted that the point cloud map acquisition device needs to be scanned when collecting point cloud data, and has a certain data acquisition period, which is not limited here. It can be understood that the data acquisition period can be changed according to the actual design needs or different point cloud map acquisition devices.

[0036] In the embodiment of the present disclosure, the second center point timestamp is the center time of the first data acquisition period of the point cloud map acquisition device when the point cloud map acquisition device is restarted.

[0037] As shown in Figure 2 The second center point timestamp can be determined by obtaining the starting acquisition timestamp and the ending acquisition timestamp of the point cloud map acquisition device. The second center point timestamp t 第二中心 = (t3-t4) / 2, where t3 is the ending acquisition time of the point cloud map acquisition device, and t4 is the starting acquisition timestamp of the point cloud map acquisition device.

[0038] In the embodiments of the present disclosure, in order to realize better binding of the point cloud map and the image of the image acquisition device, the center of the point cloud map acquisition device needs to coincide with the center position of the field of view angle of the image acquisition device, so that a point cloud map with better effect can be obtained.

[0039] It should be noted that the acquisition cycles of the point cloud map acquisition device and the image acquisition device are the same, so that the point cloud map and the image can be matched.

[0040] The installation position of the point cloud map acquisition device is set in advance and can be changed according to actual design needs, which is not limited here.

[0041] S103, under the synchronization constraint condition of the first center point timestamp and the second center point timestamp, determining the timestamp difference between the image acquisition device and the point cloud map acquisition device.

[0042] It should be noted that the synchronization constraint condition is that the acquisition time of the point cloud map and the acquisition time of the image are unified as the same time, so as to facilitate subsequent binding. For example, as shown in Figure 2 When t 第一中心 and t 第二中心 are the same time, it is considered that the synchronization constraint condition is met.

[0043] In the embodiments of the present disclosure, the first center point timestamp and the second center point timestamp can be subtracted to determine the timestamp difference between the image acquisition device and the point cloud map acquisition device.

[0044] S104, according to the timestamp difference, synchronizing the point cloud map acquired by the point cloud map acquisition device and the image of the image acquisition device.

[0045] It should be noted that the method of synchronizing the point cloud map acquired by the point cloud map acquisition device and the image of the image acquisition device according to the timestamp difference can be various, which is not limited here.

[0046] Optionally, the exposure time of the image acquisition device can be adjusted according to the timestamp difference, so as to realize the same time as the center point timestamp of the point cloud map acquisition device, so as to meet the synchronization constraint condition.

[0047] Optionally, according to the timestamp difference, the image of the image acquisition device corresponding to the point cloud map acquired by the point cloud map acquisition device at a certain time can be determined, so as to realize the synchronization constraint condition.

[0048] In the embodiment of the present disclosure, first, the first center point timestamp of the image acquisition device at the time of restart is acquired, then the second center point timestamp of the point cloud map acquisition device at the time of restart is acquired, and then the timestamp difference between the image acquisition device and the point cloud map acquisition device is determined under the synchronization constraint condition of the first center point timestamp and the second center point timestamp, and finally, the point cloud map acquired by the point cloud map acquisition device and the image of the image acquisition device are synchronized according to the timestamp difference. By determining the timestamp difference, the point cloud map and the image are synchronized, and it is not necessary to calculate and adjust the center time of the image acquisition device at each point cloud map acquisition period, which can save system resources and increase the accuracy of point cloud map and image acquisition.

[0049] Taking an automatic driving scene as an example, in the automatic driving scene, the vehicle can acquire image data around the vehicle through the set vehicle-mounted image acquisition device, and at the same time, point cloud map data is acquired through the vehicle-mounted radar. In this way, the image data and the point cloud map data acquired can be used for fusion algorithm, and then the functions of generating a point cloud map and judging obstacles in all time periods and all scenes can be realized. At the time of starting, the vehicle-mounted image acquisition device and the vehicle-mounted radar need to be synchronized. First, the first center point timestamp of the vehicle-mounted image acquisition device and the second center point timestamp of the vehicle-mounted radar can be acquired, and then the timestamp difference between the vehicle-mounted image acquisition device and the vehicle-mounted radar is determined under the synchronization constraint condition of the first center point timestamp and the second center point timestamp, and the point cloud map of the vehicle-mounted radar and the image of the image acquisition device are synchronized according to the timestamp difference.

[0050] In the above embodiment, the first center point timestamp of the image acquisition device at the time of restart can also be acquired by Figure 3 Further explanation, the method comprises:

[0051] S301, acquiring a first initial timestamp of an image acquisition device at the time of restart.

[0052] In the embodiment of the present disclosure, the first initial timestamp is the initial time of image acquisition of the image acquisition device.

[0053] It should be noted that the first initial timestamp can be the time when the point cloud map acquisition device rotates to a certain position, and the position can present a preset angle with the center of the field of view angle of the image acquisition device at the time of restart. It should be noted that the preset angle can be changed according to the actual design needs. For example, the preset angle can be 30°.

[0054] S302, determining a first center point timestamp of the image acquisition device according to the first initial timestamp and a performance parameter of the image acquisition device.

[0055] In the embodiment of the present disclosure, the camera middle row time of the image acquisition device can be acquired first, then the camera sensor internal delay and the processing delay of the image processor (ISP) are acquired, and the first exposure time of the image acquisition device is acquired, and finally the first initial time stamp, the middle row time, the internal delay, the processing delay and the first exposure time are combined and operated to determine the first center point time stamp. In this way, the first center point time stamp can be determined for different configurations of the image acquisition device, which can improve the accuracy of the acquired first center point time stamp and improve the practicability of the present solution.

[0056] The camera middle row time, the camera sensor internal delay and the processing delay of the image processor (ISP) are set in advance and can be changed according to the actual type of the image acquisition device and the design needs, which are not limited here.

[0057] In the embodiment of the present disclosure, the first initial time stamp of the image acquisition device at the restart time is acquired first, and then the first center point time stamp of the image acquisition device is determined according to the first initial time stamp and the performance parameters of the image acquisition device. In this way, the first center point time stamp is determined according to the performance parameters of the image acquisition device for different image acquisition devices, which is more accurate and provides a data basis for subsequent synchronization.

[0058] In the above embodiment, the second center point time stamp of the point cloud image acquisition device at the restart time can also be acquired by Figure 4 Further explanation, the method comprises:

[0059] S401, acquiring a second initial time stamp of a point cloud image acquisition device.

[0060] In the embodiment of the present disclosure, the second initial time stamp is the initial time when the point cloud image acquisition device acquires the point cloud image.

[0061] It should be noted that the second initial time stamp can be the time when the point cloud image acquisition device rotates to a certain position, and the position can be at a preset angle from the center of the field of view angle of the image acquisition device at the restart time. It should be noted that the preset angle can be changed according to the actual design needs. For example, the preset angle can be 60°.

[0062] S402, determining a second center point time stamp of the point cloud image acquisition device according to the second initial time stamp and a second exposure time of the point cloud image acquisition device.

[0063] In the embodiments of the present disclosure, the second exposure time of the point cloud map acquisition device is set in advance and can be changed according to actual design needs or the configuration of the point cloud map acquisition device, which is not limited here. For example, the second exposure time can be 120 ms, and the second center point timestamp t=t1+120 / 2, where t1 is the second initial timestamp.

[0064] In the embodiments of the present disclosure, the second initial timestamp of the point cloud map acquisition device is first obtained, and then the second center point timestamp of the point cloud map acquisition device is determined according to the second initial timestamp and the second exposure time of the point cloud map acquisition device. In this way, according to the second exposure time of different point cloud map acquisition devices, an accurate second center point timestamp can be determined, providing a data basis for subsequent synchronization.

[0065] In the above embodiments, the point cloud map collected by the point cloud map acquisition device and the image collected by the image acquisition device are synchronized according to the timestamp difference, and the point cloud map collected by the point cloud map acquisition device and the image collected by the image acquisition device can also be synchronized by Figure 5 Further explanation, the method comprises:

[0066] S501, obtaining the timestamp of the first frame of point cloud map of the point cloud map acquisition device.

[0067] It should be noted that the timestamp of the first frame of point cloud map is the time when the radar starts to collect point cloud data. The timestamp can be obtained by analyzing the collected point cloud data.

[0068] S502, determining the timestamp of the first frame of image according to the timestamp of the first frame of point cloud map and the timestamp difference.

[0069] In the embodiments of the present disclosure, after obtaining the timestamp of the first frame of point cloud map, the timestamp of the first frame of image can be determined by subtracting the timestamp difference from the timestamp of the first frame of point cloud map.

[0070] S503, determining the timestamp of each subsequent frame of image according to the timestamp of the first frame of image and the sampling frequency.

[0071] It should be noted that after obtaining the timestamp of the first frame of image, the sampling period can be determined by the sampling frequency, and the timestamp of each subsequent frame of image can be determined by adding the sampling period to the timestamp of the first frame of image.

[0072] In the embodiments of the present disclosure, the point cloud map collected by the point cloud map acquisition device and the image collected by the image acquisition device are synchronized according to the timestamp difference, and the point cloud map collected by the point cloud map acquisition device and the image collected by the image acquisition device can also be synchronized by obtaining the timestamp of the nth frame of point cloud map of the point cloud map acquisition device, and determining the timestamp of the nth frame of image of the image acquisition device according to the timestamp of the nth frame of point cloud map and the timestamp difference. For example, when the sampling frequency is 10 Hz, the sampling period is 100 ms, and the timestamp of the image in the nth sampling period is t=(t 首+ 100 * n), where t 首 is the timestamp of the first frame image.

[0073] In the embodiment of the present disclosure, first, the timestamp of the first point cloud image collected by the point cloud image collection device is acquired, then the timestamp of the first frame image is determined according to the timestamp of the first point cloud image and the timestamp difference, and finally the timestamp of each subsequent frame image is determined according to the timestamp of the first frame image and the sampling frequency. In this way, it is not necessary to determine the timestamp of each subsequent frame image for each point cloud image collection, which reduces the occupancy rate of system resources and can reduce the prediction cost.

[0074] It should be noted that, in order to more accurately synchronize the point cloud image collected by the point cloud image collection device and the image collected by the image collection device, the time of the external clock can also be acquired based on the Precision Time Protocol (PTP), and the point cloud image collection device and the image collection device can be clock-synchronized based on the acquired time of the external clock. In this way, errors can be prevented when synchronizing due to the non-uniformity of the clocks of the point cloud image collection device and the image collection device.

[0075] Figure 6 The block diagram of the image frame synchronization system of the embodiment of the present disclosure is shown in Figure 6 The system includes a main controller 610, a detection synchronization module 620, an image collection device 630, and a point cloud image collection device 640.

[0076] The main controller 610 synchronizes the time of the external clock to the detection synchronization module 620, the image collection device 630, and the point cloud image collection device 640 based on the Precision Time Protocol (PTP).

[0077] The detection synchronization module 620 is configured to calculate the timestamp difference according to the data reported by the image collection device 630 and the point cloud image collection device 640, and to configure the timestamp difference to the image collection device 630, so as to realize the synchronization of the point cloud image collected by the point cloud image collection device and the image collected by the image collection device.

[0078] In the embodiment of the present disclosure, the main controller can be various, which is not limited here, for example, the main controller can be X86 or ARM.

[0079] Corresponding to the image frame synchronization method provided by the above several embodiments, one embodiment of the present disclosure also provides an image frame synchronization device. Since the image frame synchronization device provided by the embodiment of the present disclosure corresponds to the image frame synchronization method provided by the above several embodiments, the implementation manner of the above image frame synchronization method is also applicable to the image frame synchronization device provided by the embodiment of the present disclosure, which will not be described in detail in the following embodiments.

[0080] Figure 7 A structural diagram of an image frame synchronization device is provided for an embodiment of the present disclosure. The image frame synchronization device 700 comprises a first acquisition module 710, a second acquisition module 720, a calculation module 730, and a synchronization module 740.

[0081] The first acquisition module 710 is configured to acquire a first center point timestamp of an image acquisition device at a restart.

[0082] The second acquisition module 720 is configured to acquire a second center point timestamp of a point cloud map acquisition device at the restart.

[0083] The calculation module 730 is configured to determine a timestamp difference between the image acquisition device and the point cloud map acquisition device under a synchronization constraint condition of the first center point timestamp and the second center point timestamp.

[0084] The synchronization module 740 is configured to synchronize a point cloud map acquired by the point cloud map acquisition device and an image of the image acquisition device according to the timestamp difference.

[0085] In an embodiment of the present disclosure, the first acquisition module 710 is further configured to acquire a first initial timestamp of the image acquisition device at the restart; and determine the first center point timestamp of the image acquisition device according to the first initial timestamp and a performance parameter of the image acquisition device.

[0086] In an embodiment of the present disclosure, the first acquisition module 710 is further configured to acquire a middle row time of a camera of the image acquisition device; acquire an internal delay of a camera sensor of the image acquisition device and a processing delay of an image signal processor (ISP); acquire a first exposure time of the image acquisition device; and combine the first initial timestamp, the middle row time, the internal delay, the processing delay, and the first exposure time to determine the first center point timestamp.

[0087] In an embodiment of the present disclosure, the second acquisition module 720 is further configured to acquire a second initial timestamp of the point cloud map acquisition device; and determine the second center point timestamp of the point cloud map acquisition device according to the second initial timestamp and a second exposure time of the point cloud map acquisition device.

[0088] In an embodiment of the present disclosure, the synchronization module 740 is further configured to acquire a timestamp of a first frame of point cloud map of the point cloud map acquisition device; determine a timestamp of a first frame of image according to the timestamp of the first frame of point cloud map and the timestamp difference; and determine a timestamp of each subsequent frame of image according to the timestamp of the first frame of image and a sampling frequency.

[0089] In an embodiment of the present disclosure, the synchronization module 740 is further configured to acquire a timestamp of an n-th frame of point cloud map of the point cloud map acquisition device; and determine a timestamp of an n-th frame of image of the image acquisition device according to the timestamp of the n-th frame of point cloud map and the timestamp difference.

[0090] In one embodiment of the present disclosure, the synchronization module 740 is further configured to: acquire a time of an external clock based on a precision time protocol (PTP); and perform clock synchronization on the point cloud acquisition device and the image acquisition device based on the acquired time of the external clock.

[0091] In the technical solution of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0092] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0093] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0094] As shown in Figure 8 The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with computer programs / instructions stored in a read-only memory (ROM) 802 or loaded from a storage unit 806 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0095] Various components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806 such as a keyboard, a mouse, etc.; an output unit 807 such as various types of displays, speakers, etc.; a storage unit 808 such as a magnetic disk, an optical disk, etc.; and a communication unit 809 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0096] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the synchronization method of image frames. For example, in some embodiments, the synchronization method of image frames can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 806. In some embodiments, portions or all of the computer program / instructions can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program / instructions are loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the synchronization method of image frames described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the synchronization method of image frames by any other suitable means, such as by means of firmware.

[0097] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0098] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0099] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0100] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0101] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0102] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0103] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the steps disclosed in the present disclosure. For example, the steps disclosed in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0104] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for synchronizing image frames, wherein, include: Obtain the first center point timestamp of the image acquisition device at the time of restart, wherein the first center point timestamp is the center moment of the first imaging delay of the image acquisition device at the time of restart; Obtain the second center point timestamp of the point cloud map acquisition device at the time of restart, wherein the second center point timestamp is the center moment of the first data acquisition cycle of the point cloud map acquisition device at the time of restart; Under the synchronization constraint of the first center point timestamp and the second center point timestamp, the timestamp difference between the image acquisition device and the point cloud map acquisition device is determined; Based on the timestamp difference, the point cloud map acquired by the point cloud map acquisition device and the image acquired by the image acquisition device are synchronized.

2. The method according to claim 1, wherein, The process of obtaining the first center point timestamp of the image acquisition device upon restart includes: Obtain the first initial timestamp of the image acquisition device upon restart; The first center point timestamp of the image acquisition device is determined based on the first initial timestamp and the performance parameters of the image acquisition device.

3. The method according to claim 2, wherein, The step of determining the first center point timestamp of the image acquisition device based on the first initial timestamp and the performance parameters of the image acquisition device includes: Obtain the camera midline time of the image acquisition device; The internal delay of the camera sensor and the processing delay of the image processor (ISP) of the image acquisition device are obtained; Obtain the first exposure time of the image acquisition device; The first center point timestamp is determined by combining the first initial timestamp, the intermediate row time, the internal delay, the processing delay, and the first exposure time.

4. The method according to claim 1, wherein, The process of obtaining the second center point timestamp of the point cloud map acquisition device at the time of restart includes: Obtain the second initial timestamp of the point cloud acquisition device; The second center point timestamp of the point cloud acquisition device is determined based on the second initial timestamp and the second exposure time of the point cloud acquisition device.

5. The method according to any one of claims 1-4, wherein, The step of synchronizing the point cloud map acquired by the point cloud map acquisition device and the image acquired by the image acquisition device based on the timestamp difference includes: Obtain the timestamp of the first frame of point cloud image from the point cloud image acquisition device; The timestamp of the first frame image is determined based on the timestamp of the first frame point cloud map and the timestamp difference. The timestamp of each subsequent frame is determined based on the timestamp and sampling frequency of the first frame image.

6. The method according to any one of claims 1-4, wherein, The step of synchronizing the point cloud map acquired by the point cloud map acquisition device and the image acquired by the image acquisition device based on the timestamp difference includes: Obtain the timestamp of the nth frame of the point cloud image from the point cloud image acquisition device; The timestamp of the nth frame image of the image acquisition device is determined based on the timestamp of the nth frame point cloud map and the timestamp difference.

7. The method according to any one of claims 1-4, wherein, The method further includes: Time is obtained from an external clock based on the Precision Time Protocol (PTP). The point cloud acquisition device and the image acquisition device are synchronized based on the time of the acquired external clock.

8. A synchronization device for image frames, comprising: The first acquisition module is used to acquire the first center point timestamp of the image acquisition device at the time of restart, wherein the first center point timestamp is the center moment of the first imaging delay of the image acquisition device at the time of restart; the second acquisition module is used to acquire the second center point timestamp of the point cloud map acquisition device at the time of restart, wherein the second center point timestamp is the center moment of the first data acquisition cycle of the point cloud map acquisition device at the time of restart. The calculation module is used to determine the timestamp difference between the image acquisition device and the point cloud acquisition device under the synchronization constraint of the first center point timestamp and the second center point timestamp; The synchronization module is used to synchronize the point cloud map acquired by the point cloud map acquisition device and the image acquired by the image acquisition device according to the timestamp difference.

9. The apparatus according to claim 8, wherein, The first acquisition module is further configured to: Obtain the first initial timestamp of the image acquisition device upon restart; The first center point timestamp of the image acquisition device is determined based on the first initial timestamp and the performance parameters of the image acquisition device.

10. The apparatus according to claim 9, wherein, The first acquisition module is further configured to: Obtain the camera midline time of the image acquisition device; The internal delay of the camera sensor and the processing delay of the image processor (ISP) of the image acquisition device are obtained; Obtain the first exposure time of the image acquisition device; The first center point timestamp is determined by combining the first initial timestamp, the intermediate row time, the internal delay, the processing delay, and the first exposure time.

11. The apparatus according to claim 8, wherein, The second acquisition module is further configured to: Obtain the second initial timestamp of the point cloud acquisition device; The second center point timestamp of the point cloud acquisition device is determined based on the second initial timestamp and the second exposure time of the point cloud acquisition device.

12. The apparatus according to any one of claims 8-11, wherein, The synchronization module is also used for: Obtain the timestamp of the first frame of point cloud image from the point cloud image acquisition device; The timestamp of the first frame image is determined based on the timestamp of the first frame point cloud map and the timestamp difference. The timestamp of each subsequent frame is determined based on the timestamp and sampling frequency of the first frame image.

13. The apparatus according to any one of claims 8-11, wherein, The synchronization module is also used for: Obtain the timestamp of the nth frame of the point cloud image from the point cloud image acquisition device; The timestamp of the nth frame image of the image acquisition device is determined based on the timestamp of the nth frame point cloud map and the timestamp difference.

14. The apparatus according to any one of claims 8-11, wherein, The synchronization module is also used for: Time is obtained from an external clock based on the Precision Time Protocol (PTP). The point cloud acquisition device and the image acquisition device are synchronized based on the time of the acquired external clock.

15. An electronic device comprising: At least one processor, and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the synchronization method for image frames according to any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the image frame synchronization method according to any one of claims 1-7.

17. A computer program product comprising a computer program / instructions, wherein, When the computer program / instruction is executed by the processor, it implements the method for synchronizing image frames according to any one of claims 1-7.

Images