Image synchronization method of image acquisition system, vehicle, device and storage medium

Abstract

The invention discloses an image synchronization method of an image acquisition system, a vehicle, a device and a storage medium. The method comprises the following steps: in response to a synchronous reading signal, reading image data of at least two image acquisition devices in rows according to a rule that vertical effective durations corresponding to the image acquisition devices in the reading process are kept the same, the vertical effective duration is used for representing the duration from the beginning of the exposure reading operation of the first line of image data to the completion of the exposure reading operation of the last line of image data; and determining synchronous acquisition images of the at least two image acquisition devices based on the image data. By applying the technical scheme of the invention, the synchronism of the images acquired by multiple devices can be effectively improved, and the images are dynamically adjusted, so that the problem of keeping the space-time consistency of the image information in a quick moving scene of an object is solved, the problem of image accuracy, stability and reliability under different illumination conditions is solved, and the robustness of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to an image synchronization method, a vehicle, a device and a storage medium of an image acquisition system. Background Art

[0002] With the continuous development of image technology, in many fields involving video image analysis or video surveillance, there is a need to use cameras to capture images and process the images captured by cameras. For example, advanced assisted driving in cars requires the use of multiple cameras to analyze the driving environment. However, when multiple cameras output images to the main control, it is easy to cause the image information of multiple cameras to be out of sync in time and space due to differences in camera hardware performance from different manufacturers or differences in ambient brightness. For example, the position of the same object in the images captured by multiple cameras is out of sync, which may cause the assisted driving algorithm to misjudge the position of the object and make wrong decisions. If additional image processing algorithms are added to deal with the problem of time and space asynchrony, it may cause the algorithm to be complex or the calculation to take a long time, thereby consuming more processor resources.

[0003] In the prior art, it is possible to achieve image synchronization by synchronizing the camera exposure time. However, for scenes with different brightness or when the performance of cameras from different manufacturers varies greatly, this method will result in large differences in the brightness of the images obtained after being exposed by the camera, such as overexposure or underexposure of some images, inconsistent image brightness information, and unsatisfactory results. The exposure time synchronization method is more suitable for certain fixed brightness environments or scenes with small differences in ambient brightness, and is not well suited for driving environments where ambient brightness changes frequently.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide an image synchronization method, a vehicle, a device and a storage medium of an image acquisition system, so as to at least solve the technical problem of using different models of cameras to acquire images or the exposure is not synchronized in the related art.

[0006] According to one aspect of an embodiment of the present invention, there is provided an image synchronization method for an image acquisition system, comprising: responding to a synchronous reading signal, reading image data of at least two image acquisition devices row by row according to a rule that the vertical effective time lengths corresponding to the image acquisition devices respectively remain the same during the reading process, wherein the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; and determining the synchronous acquisition images of the at least two image acquisition devices based on the image data.

[0007] Optionally, the method further includes: generating a frame start signal of the image acquisition device in response to the synchronous reading signal, and controlling the start of an exposure reading operation of the image acquisition device; wherein the frame start signals of at least two image acquisition devices are synchronized.

[0008] Optionally, the method further includes: in response to completion of reading of image data by the image acquisition device, generating a frame end signal of the image acquisition device; wherein the frame end signals of at least two image acquisition devices are synchronized.

[0009] Optionally, in response to a synchronous read signal, a frame start signal of the image acquisition device is generated, and the exposure read operation of the image acquisition device is controlled to start, including: in response to an edge signal of a square wave synchronization signal in the synchronous read signal, a frame start signal of the image acquisition device is generated, and the exposure read operation of the image acquisition device is controlled to start; wherein the frame start signals of at least two image acquisition devices are synchronized with the edge signal of the square wave synchronization signal.

[0010] Optionally, according to the rule that the vertical effective time corresponding to each image acquisition device during the reading process remains the same, before the image data of at least two image acquisition devices are read row by row according to the preset rule, the method also includes: determining the minimum vertical effective time of the at least two image acquisition devices based on the device parameters of the image acquisition devices, the minimum vertical effective time being used to represent the shortest value of the effective time for the image acquisition device to expose and read the image; adjusting the pixel clock frequency and the row reading time of the at least two image acquisition devices so that the vertical effective time of the at least two image acquisition devices is the same.

[0011] Optionally, the method also includes: generating an internal timing adjustment strategy when it is determined that the frame start signal and the synchronous read signal of the image acquisition device meet a preset deviation condition; wherein the internal timing adjustment strategy is used to adjust the internal timing of the image acquisition device so that the frame start signal of the image acquisition device is synchronized with the synchronous read signal.

[0012] According to another aspect of an embodiment of the present invention, a vehicle is provided, including: an image acquisition system, wherein the image acquisition system is controlled by the image synchronization method of the above-mentioned image acquisition system.

[0013] According to another aspect of an embodiment of the present invention, there is provided an image synchronization device of an image acquisition system, comprising: a response module, the response module being used to start an exposure reading operation of a first row of image data of at least two image acquisition devices in response to a synchronous reading signal; a reading module, the reading module being used to respond to the synchronous reading signal and read the image data of at least two image acquisition devices according to preset rules, wherein the preset rules at least include that the vertical effective time for reading the image data of each image acquisition device is the same, and the vertical effective time is used to represent the time from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; and a determination module, the determination module being used to determine the synchronous acquisition of images of at least two image acquisition devices based on the image data.

[0014] According to another aspect of the present application, a computer program product is also provided, comprising computer instructions, which implement the steps of the image synchronization method of the above-mentioned image acquisition system when the computer instructions are executed by a processor.

[0015] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the image synchronization method of the image acquisition system when running.

[0016] In an embodiment of the present invention, in response to a synchronous reading signal, the image data of at least two image acquisition devices are read row by row according to the rule that the vertical effective durations corresponding to each image acquisition device during the reading process are kept the same, wherein the vertical effective duration is used to represent the duration from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; based on the image data, the synchronously acquired images of at least two image acquisition devices are determined. By adjusting the vertical effective durations of multiple image acquisition devices, sending synchronous reading signals, and multiple image acquisition devices reading image data according to the rule that the vertical effective durations corresponding to each image acquisition device during the reading process are kept the same, and then comprehensively analyzing and determining the synchronously acquired images, the synchronization of the images acquired by multiple devices can be effectively improved, and the images can be dynamically adjusted, solving the problem of maintaining the spatiotemporal consistency of image information in the scene of rapid object movement, and the problem of accurate, stable and reliable images under different lighting conditions, thereby improving the robustness of the system. When the image acquisition system and image synchronization method in the above embodiment are applied to vehicles, fast-moving vehicles can be captured and located more accurately, thereby improving the system's perception of the driving environment and the accuracy of decision-making, and ensuring vehicle safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a hardware structure block diagram of an electronic device of a vehicle according to an image synchronization method of an image acquisition system of one embodiment of the present invention;

[0019] Figure 2 is a flow chart of an image synchronization method of an image acquisition system according to one optional embodiment of the present invention;

[0020] Figure 3 4 is a structural block diagram of an image synchronization device of an image acquisition system according to one embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0024] SOF: start of frame, a signal in the MIPI protocol, which indicates the moment when the valid data part of the image frame starts to be output in the embodiment of the present application. It is output by the camera and finally input to the main control.

[0025] EOF: end of frame, a signal in the MIPI protocol, which indicates the moment when the output of the valid data part of the image frame ends in the embodiment of the present application. It is output by the camera and finally input to the main control.

[0026] Rolling shutter: The camera scans and reads line by line until all pixels are scanned. Generally, the exposure control of Rolling Shutter is performed in lines. Before exposing a line, it needs to be reset (cleared to 0), and then wait for a period of exposure time (in lines) before reading (readout) this line. The entire exposure process is completed after reading this line. Before the next reset, these lines are still in the exposure state.

[0027] Vertical effective time: the time from when the camera reads the first row of pixel matrix to when it reads the last row of pixel matrix.

[0028] Pixel array: refers to the many tiny photosensitive elements arranged on an image sensor (such as a CMOS or CCD sensor) that are responsible for capturing light and converting it into electronic signals.

[0029] According to one embodiment of the present invention, an embodiment of an image synchronization method of an image acquisition system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiment can be executed in an electronic device or similar computing device in a vehicle including a memory and a processor. For example, Figure 1 As shown, the electronic device of the vehicle may include one or more processors 102 (the processor may include but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microprocessor (MCU), a programmable logic device (FPGA), a neural network processor (NPU), a tensor processing unit (TPU), an artificial intelligence (AI) type processor, etc.) and a memory 104 for storing data. Optionally, the electronic device of the vehicle may also include a transmission device 106 for communication functions, an input and output device 108, and a display 110. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the electronic device of the vehicle. For example, the electronic device of the vehicle may include more or fewer components than those described in the above structure, or may have a configuration different from that described in the above structure.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image synchronization method of the image acquisition system in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the image synchronization method of the image acquisition system described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display 110 may be, for example, a touch screen liquid crystal display (LCD). The liquid crystal display may enable a user to interact with a user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), and a user may interact with the GUI by finger contact and / or gestures on a touch-sensitive surface, wherein the human-computer interaction functions here may optionally include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital videos, playing digital music, and / or web browsing, etc. The executable instructions for executing the above human-computer interaction functions are configured / stored in a computer program product or a readable storage medium executable by one or more processors.

[0034] This embodiment provides an image synchronization method of an image acquisition system of an electronic device running on the above vehicle. Figure 2 is a flow chart of an image synchronization method of an image acquisition system according to one embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0035] Step S21, in response to the synchronous reading signal, according to the rule that the vertical effective time lengths corresponding to the image acquisition devices are kept the same during the reading process, the image data of at least two image acquisition devices are read row by row, wherein the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data;

[0036] Specifically, in step S21, in response to a synchronous read signal, the synchronous read signal is sent by the same signal generating device (such as a vehicle main control board or a main control system), and the read signal is synchronously sent to each image acquisition device (such as a camera), and the reading of image data is triggered to start at the same time; during the reading process, the rule that the vertical effective time corresponding to each image acquisition device remains the same requires that the vertical effective time of all image acquisition devices when reading image data must be the same.

[0037] It should be noted that there may be more than two or more image acquisition devices. Due to differences in models and performance, there may be errors in the vertical effective time of different image acquisition devices. By adjusting the parameter configurations of different image acquisition devices (for example, adjusting the pixel clock frequency, row exposure time, etc.), image acquisition devices with different resolutions and performances can be adjusted to have the same vertical effective time, thereby ensuring subsequent image synchronization adjustments.

[0038] Step S22: determining, based on the image data, synchronous image acquisition of at least two image acquisition devices.

[0039] Specifically, after sending out the synchronous reading signal, each image acquisition device starts to read the image data synchronously according to the preset rules, and obtains image data with the same vertical effective time. The main control system analyzes and compares the characteristics of the image data obtained by each image acquisition device and finally determines the synchronous acquisition image of each image acquisition device.

[0040] Through the above steps, in response to the synchronous reading signal, the image data of at least two image acquisition devices are read row by row according to the rule that the vertical effective time lengths corresponding to each image acquisition device during the reading process are kept the same, wherein the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; based on the image data, the synchronous acquisition images of at least two image acquisition devices are determined. By adjusting the vertical effective time length of multiple image acquisition devices, sending a synchronous reading signal, multiple image acquisition devices read the image data according to the rule that the vertical effective time lengths corresponding to each image acquisition device during the reading process are kept the same, and then comprehensively analyzing and determining the synchronous acquisition image, the synchronization of the images collected by multiple devices can be effectively improved, and the image can be dynamically adjusted, which solves the problem of maintaining the spatiotemporal consistency of image information in the scene of rapid movement of objects, and the problem of accurate, stable and reliable images under different lighting conditions, thereby improving the robustness of the system. When the image acquisition system and image synchronization method in the above embodiments are applied to vehicles, fast-moving vehicles can be captured and located more accurately, thereby improving the system's perception of the driving environment and the accuracy of decision-making, and ensuring vehicle safety.

[0041] Optionally, in step S21, the method further includes the following execution steps:

[0042] Step S211, in response to the synchronous reading signal, generating a frame start signal of the image acquisition device, and controlling the exposure reading operation of the image acquisition device to start;

[0043] Specifically, the image acquisition device reads the image data starting from the exposure of the first row of data, and then reads step by step until the last row of image data is exposed. After the synchronous read signal is sent out, each image acquisition device receives the read signal synchronously, that is, starts the exposure and reading operation synchronously; after responding to the synchronous read signal, the image acquisition device simultaneously generates a frame start signal.

[0044] The frame start signals of at least two image acquisition devices are synchronized. The same frame start signals of each image acquisition device can ensure the consistency of the time when the image acquisition cycle starts, reduce the problem of image asynchrony caused by signal response time differences, and facilitate the adjustment of the vertical effective time of each image acquisition device.

[0045] Optionally, in step S21, the method further includes:

[0046] Step S212, in response to the image data reading of the image acquisition device being completed, generating a frame end signal of the image acquisition device;

[0047] Specifically, based on the image data reading logic of the image acquisition device, since multiple image acquisition devices generate frame start signals at the same time, it is guaranteed that multiple image acquisition devices start the exposure and reading operation at the same time. According to the rule that the vertical effective time corresponding to each image acquisition device remains the same during the reading process, under the premise of the same frame start signal, the vertical effective time of each image acquisition device is adjusted to the same time. The difference in vertical effective time may affect the valid number of rows of image data, and adjusting the vertical effective time to the same time can make the valid number of rows of image data read by each device consistent. After setting a consistent number of valid rows, image data is read. After reading from the first row to the end of the last row, each image acquisition device will generate a frame end signal.

[0048] Among them, the frame end signals of at least two image acquisition devices are synchronized. Since the vertical effective duration and the number of effective lines of each image acquisition device are synchronized, the frame end signals of each device will also be generated synchronously, ensuring that all devices can output frame end signals at the same time at the end of the image acquisition cycle to achieve synchronization of image data. By accurately controlling the vertical effective duration and the number of effective lines, it is ensured that each image acquisition device can maintain synchronization in time and space when reading image data, which can effectively reduce the delay of image processing and improve the response speed and real-time performance of the system.

[0049] Optionally, in step S211, in response to the synchronous reading signal, a frame start signal of the image acquisition device is generated, and the exposure reading operation of the image acquisition device is controlled to start, including:

[0050] Step S2111, in response to the edge signal of the square wave synchronization signal in the synchronization reading signal, generating a frame start signal of the image acquisition device, and controlling the exposure reading operation of the image acquisition device to start;

[0051] Specifically, the main control system or signal generating device generates a square wave synchronization signal with uniform frequency and phase, and sends it to all image acquisition devices. Since the frequency and phase of the signal are unified and in the form of a square wave, the edge signal of the square wave signal (i.e., the peak or trough signal of the square wave) is used as a trigger signal for the image acquisition device to start the exposure and reading operation, thereby generating a frame start signal, thereby controlling the start of the exposure and reading operation of the image acquisition device.

[0052] Among them, the frame start signals of at least two image acquisition devices are synchronized with the edge signal of the square wave synchronization signal. The frame start signal is synchronized with the edge signal of the square wave synchronization signal. When each image acquisition device receives the square wave synchronization signal, it responds to a specific edge signal (such as a rising edge or a falling edge in the square wave signal) and immediately starts the exposure reading operation of the first row of image data within a preset time, and each image acquisition device generates a frame start signal. This setting can accurately control all image acquisition devices to start the exposure reading operation of the first row of image data, and realize the synchronous start of image data acquisition.

[0053] Optionally, in step S21, according to the rule that the vertical effective durations corresponding to the respective image acquisition devices are kept the same during the reading process, before reading the image data of at least two image acquisition devices row by row, the method further includes:

[0054] Step S2101, based on the device parameters of the image acquisition devices, determining the minimum vertical effective time of at least two image acquisition devices, where the minimum vertical effective time is used to represent the shortest value of the effective time for the image acquisition devices to expose and read images;

[0055] Specifically, based on the different models of each image acquisition device, the parameters of each device are analyzed first, such as pixel clock frequency (PCLK), row exposure time (HTS), pixel matrix resolution (vertical height), and number of reading channels (Pipeline), and the minimum vertical effective time of each device is calculated based on the above parameters.

[0056] In this embodiment, the vertical valid time (Vvalid time) can be calculated based on the following formula:

[0057] Vvalid time=row time*valid lines=HTS / PCLK / Pipeline*height;

[0058] Among them, row time indicates the time it takes for the image acquisition device to read a row; valid lines indicates the number of valid exposure rows, which is generally equivalent to the vertical height of the image.

[0059] Step S2102, adjusting the pixel clock frequency and the row reading time of at least two image acquisition devices so that the vertical effective time lengths of at least two image acquisition devices are the same.

[0060] Specifically, after calculating and determining the minimum vertical effective time of each image acquisition device, the parameters of each device are adjusted to make the vertical effective time of each device the same or approximately the same to achieve the same vertical effective time of each device, thereby ensuring that each device can complete the reading of image data from the first line to the last line in the same time, which means synchronous reading of image data.

[0061] In this embodiment, by adjusting the pixel clock frequency, line reading time and other parameters of the image acquisition device, the vertical effective time of each image acquisition device is ensured to be the same, the synchronization of image data is achieved, the image processing error is reduced, and the adaptability and processing efficiency of the system are improved.

[0062] Optionally, in step 212, the method further includes:

[0063] Step S2121, generating an internal timing adjustment strategy when it is determined that the frame start signal and the synchronous read signal of the image acquisition device meet the preset deviation condition;

[0064] Specifically, the preset deviation condition is a pre-set deviation condition, which refers to the deviation that is not allowed to occur between the frame start signal and the synchronous read signal of the image acquisition device. When the synchronous read signal is sent by the corresponding central control system of each image acquisition device, an exposure read operation is performed and a frame start signal is generated. The time difference between the frame start signal and the synchronous read signal is detected to determine whether it meets the preset deviation condition. If the detected time difference meets the preset deviation condition, the system will generate an internal timing adjustment strategy. It should be understood that in actual applications, a reasonable deviation can also be set, that is, the time difference between the frame start signal and the synchronous read signal of the image acquisition device is allowed. When the time difference between the frame start signal and the synchronous read signal of the image acquisition device exceeds the reasonable deviation, it can be determined that the frame start signal and the synchronous read signal of the image acquisition device meet the preset deviation condition, and an internal timing adjustment strategy is generated.

[0065] The internal timing adjustment strategy is used to adjust the internal timing of the image acquisition device so that the frame start signal of the image acquisition device is synchronized with the synchronous reading signal.

[0066] In this embodiment, the internal timing adjustment strategy includes but is not limited to adjusting the exposure control timing, modifying the read control timing, changing the pixel clock frequency and other adjustment strategies. After the internal timing adjustment strategy is generated, the acquisition device will adjust the internal timing according to the strategy to achieve synchronization between the frame start signal and the synchronous read signal.

[0067] The present application also provides a preferred embodiment of an image synchronization method for an image acquisition system, which sends an external synchronization signal of the same frequency and phase to multiple image acquisition devices through the same signal generating device, sets the register configuration, and enables the image acquisition device to start performing an exposure reading operation after receiving the external synchronization signal, so that different image acquisition devices output the same frame start signal time, and then sets the vertical effective time of multiple image acquisition devices to the same or similar time, thereby ensuring that the frame end signal is also the same. The specific steps and principles are as follows:

[0068] Step 1: Generate a square wave signal with the same frequency and phase through the same signal generating device and send it to multiple image acquisition devices;

[0069] For example, a square wave signal is sent from the main control board and transmitted to the deserializer through the level signal of the pin. The deserializer encapsulates the square wave signal into a data packet of the GMSL protocol and passes it to the serializer connected to it. The serializer parses and obtains the square wave signal value and transmits the level signal to each image acquisition device through the pin.

[0070] Step 2: Set the synchronization method, use the square wave signal to trigger the image acquisition device to start reading image data (so that the image acquisition device generates a SOF signal), and when the image acquisition device receives the square wave edge signal, start reading the first row of data in the exposure dot matrix area of ​​the image acquisition device and output a SOF signal (frame start signal);

[0071] The square wave edge signal and the SOF signal are set to be synchronized, and the SOF signals output by different image acquisition devices are also set to be synchronized.

[0072] Step 3: Set the vertical effective time lengths of different image acquisition devices to be the same, that is, set the time between the image acquisition device reading the first row of pixel arrays and reading the last row to be the same;

[0073] Since in step 2, different image acquisition devices read the first line at the same time, then when the vertical effective duration is the same, the last line reading time is also the same. The vertical effective duration formula is as follows:

[0074] Vvalid time=row time*valid lines

[0075] =HTS / PCLK / Pipeline*height;

[0076] Among them, row time indicates the time it takes for the image acquisition device to read a row; valid lines indicates the number of valid exposure rows, which is simplified to be equivalent to height; HTS indicates the number of pixel clocks required to read a row; Pipeline indicates the channel through which the image acquisition device can read exposure data simultaneously; PCLK indicates the pixel clock, that is, the number of pixel clocks that can be generated per second, which determines the speed at which the image acquisition device reads image data; height indicates the vertical height of the image resolution.

[0077] The calculation formula of EOF is as follows:

[0078] EOF=SOF+Vvalid time

[0079] =SOF+HTS / PCLK / Pipeline*height;

[0080] From the above formula, we can see that EOF is affected by SOF and Vvalid time, while Vvalid time is determined by resolution, Pipeline, HTS, PCLK, etc. (that is, the parameters of the image acquisition device). Since the resolution and Pipeline of the image acquisition device are generally fixed, the EOF time needs to be adjusted through HTS and PCLK.

[0081] Step 4: Determine the minimum Vvalid time that the image acquisition device can support based on its performance, and adjust the HTS and PCLK of each image acquisition device so that the Vvalid time of each image acquisition device is the same or similar, thereby making the EOF the same.

[0082] For example, the resolutions of the two image acquisition devices are 1936*1550 and 3840*2160 respectively; their pipelines are 1 and 2 respectively; their output frame rates are both 30fps. Since the resolutions of the two image acquisition devices are fixed, it is necessary to adjust the PCLK and HTS values ​​to make the vertical effective time of the two image acquisition devices the same. If it is assumed that the vertical effective time is 30ms, after comprehensive adjustment, the HTS of the two image acquisition devices that can be designed are 2200 and 2830 respectively, and their PLCK are 115.7M and 102M.

[0083] It should be noted that sometimes the image acquisition device will adjust the exposure time according to different scenes, but it will not affect the EOF time. For example, when the exposure time changes, it actually only delays or advances the exposure start time, which will not affect the SOF time. EOF = SOF + Vvalid time, so the EOF time will not change, so the image acquisition devices still maintain synchronization.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0085] In this embodiment, an image synchronization device for an image acquisition system is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0086] Figure 3 is a structural block diagram of an image synchronization device of an image acquisition system according to one embodiment of the present invention, such as Figure 3 As shown, the device includes: a response module, the response module is used to start the exposure reading operation of the first row of image data of at least two image acquisition devices in response to the synchronous reading signal; a reading module, the reading module is used to respond to the synchronous reading signal and read the image data of at least two image acquisition devices according to preset rules, wherein the preset rules at least include that the vertical effective time for reading the image data of each image acquisition device is the same, and the vertical effective time is used to represent the time from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; a determination module, the determination module is used to determine the synchronous acquisition of images by at least two image acquisition devices based on the image data.

[0087] Through the above device, in response to the synchronous reading signal, the image data of at least two image acquisition devices are read row by row according to the rule that the vertical effective time lengths corresponding to each image acquisition device in the reading process are kept the same, wherein the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; based on the image data, the synchronous acquisition images of at least two image acquisition devices are determined. By adjusting the vertical effective time length of multiple image acquisition devices, sending a synchronous reading signal, multiple image acquisition devices read the image data according to the rule that the vertical effective time lengths corresponding to each image acquisition device in the reading process are kept the same, and then comprehensively analyzing and determining the synchronous acquisition image, the synchronization of the image acquisition of multiple devices can be effectively improved, and the image can be dynamically adjusted, which solves the problem of maintaining the temporal and spatial consistency of image information in the scene of rapid movement of objects, and the problem of accurate, stable and reliable images under different lighting conditions, and improves the robustness of the system. When the image acquisition system and image synchronization method in the above embodiment are applied to vehicles, fast-moving vehicles can be captured and located more accurately, thereby improving the system's perception of the driving environment and the accuracy of decision-making, ensuring vehicle safety.

[0088] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0089] Optionally, the image synchronization device of the image acquisition system provided in the present application may also include other modules. For example, the image synchronization device of the image acquisition system may also include a communication module, which is responsible for the transmission of image data and control signals to ensure the stability and real-time communication between the image acquisition device and the main control system; the image synchronization device of the image acquisition system may also include a compensation module, which performs compensation processing on the image data output by the image acquisition device to eliminate image quality difference problems caused by hardware differences or environmental factors.

[0090] The embodiment of the present application further provides a readable storage medium. Optionally, in this embodiment, the readable storage medium can be used to store the program code executed by the image synchronization method of the image acquisition system provided in the first embodiment.

[0091] Optionally, in this embodiment, the readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0092] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the image synchronization method of the image acquisition system.

[0093] Optionally, in this embodiment, the computer program is configured to perform the following steps:

[0094] Step S1, in response to a synchronous reading signal, according to a rule that the vertical effective time lengths corresponding to the image acquisition devices are kept the same during the reading process, the image data of at least two image acquisition devices are read row by row, wherein the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data;

[0095] Step S2: based on the image data, determine the synchronous image acquisition of at least two image acquisition devices.

[0096] The readable storage medium provided in the present application stores a computer program, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0097] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0098] Step S1, in response to a synchronous reading signal, according to a rule that the vertical effective time lengths corresponding to the image acquisition devices are kept the same during the reading process, the image data of at least two image acquisition devices are read row by row, wherein the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data;

[0099] Step S2: based on the image data, determine the synchronous image acquisition of at least two image acquisition devices.

[0100] The embodiment of the present application also provides a vehicle, the vehicle includes an image acquisition system, and the image acquisition system is controlled by the image synchronization method of the image acquisition system in the above embodiment. The method in the above embodiment can ensure data synchronization of multiple cameras of the vehicle, provide more accurate environmental perception for the vehicle, greatly reduce the risk of traffic accidents, and improve driving safety and comfort.

[0101] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0102] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0103] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0106] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0108] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image synchronization method for an image acquisition system, characterized in that: The method comprises: In response to the synchronous reading signal, the image data of at least two image acquisition devices are read row by row according to the rule that the vertical effective time lengths corresponding to the image acquisition devices are kept the same during the reading process, wherein the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; Based on the image data, synchronously captured images of at least two of the image capture devices are determined.

2. The method according to claim 1, characterized in that: The method further comprises: In response to the synchronous reading signal, a frame start signal of the image acquisition device is generated, and an exposure reading operation of the image acquisition device is controlled to start; Wherein, the frame start signals of at least two of the image acquisition devices are synchronized.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to completion of reading the image data of the image acquisition device, generating a frame end signal of the image acquisition device; Wherein, the frame end signals of at least two of the image acquisition devices are synchronized.

4. The method according to claim 2, characterized in that: In response to a synchronous reading signal, a frame start signal of the image acquisition device is generated, and an exposure reading operation of the image acquisition device is controlled to start, including: In response to the edge signal of the square wave synchronization signal in the synchronization reading signal, the frame start signal of the image acquisition device is generated, and the exposure reading operation of the image acquisition device is controlled to start; Wherein, the frame start signals of at least two of the image acquisition devices are synchronized with the edge signal of the square wave synchronization signal.

5. The method according to claim 1, characterized in that According to the rule that the vertical effective durations corresponding to the image acquisition devices respectively keep the same during the reading process, the method further comprises: Determine, based on device parameters of the image acquisition devices, a minimum vertical effective duration of at least two of the image acquisition devices, wherein the minimum vertical effective duration is used to represent the shortest value of the effective duration of the image acquisition devices for exposing and reading images; The pixel clock frequency and the row reading time of at least two of the image acquisition devices are adjusted so that the vertical effective time lengths of at least two of the image acquisition devices are the same.

6. The method according to claim 2, characterized in that The method further comprises: When it is determined that the frame start signal and the synchronous read signal of the image acquisition device meet the preset deviation condition, an internal timing adjustment strategy is generated; The internal timing adjustment strategy is used to adjust the internal timing of the image acquisition device so that the frame start signal of the image acquisition device is synchronized with the synchronous read signal.

7. A vehicle, characterized in that: It comprises an image acquisition system, and the image acquisition system is controlled by the image synchronization method of the image acquisition system described in any one of claims 1 to 6.

8. An image synchronization device for an image acquisition system, characterized in that: include: A response module, the response module is used to start the exposure reading operation of the first row of image data of at least two image acquisition devices in response to the synchronous reading signal; A reading module, the reading module is used to respond to the synchronous reading signal and read the image data of at least two image acquisition devices according to a preset rule, wherein the preset rule at least includes that the vertical effective time length of reading the image data of each image acquisition device is the same, and the vertical effective time length is used to represent the time length from the start of the exposure reading operation of the first row of image data to the completion of the exposure reading operation of the last row of image data; A determination module is used to determine, based on the image data, synchronously capture images of at least two of the image capture devices.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image synchronization method of the image acquisition system according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the computer-readable storage medium is controlled, and the device where the program is located executes the image synchronization method of the image acquisition system according to any one of claims 1 to 6.

Images