Long-distance synchronous acquisition method and system of multi-path common and dynamic visual sensors

Abstract

According to the long-distance synchronous acquisition method and system for the multiple common and dynamic vision sensors, the long-distance synchronous acquisition method and system for the multiple common and dynamic vision sensors comprise a plurality of common vision sensors, a plurality of dynamic vision sensors and a control processor, and the method is applied to the control processor. The method comprises the following steps: determining a target common vision sensor, and determining a target dynamic vision sensor; acquiring a first synchronizing signal of the target common visual sensor, performing first delay compensation processing on the first synchronizing signal, and then sending the first synchronizing signal to other common visual sensors; carrying out second delay compensation processing on the first synchronization signal and then sending the first synchronization signal to each dynamic visual sensor; and the second synchronizing signal sent by the target dynamic vision sensor is subjected to third delay compensation processing and then sent to the other dynamic vision sensors, so that synchronous data acquisition is carried out, and the accuracy and synchronism of image data acquisition of the multiple vision sensors are improved.

Description

Technical Field

[0001] The present application relates to the field of image acquisition technology, and in particular to a method and system for long-distance synchronous acquisition of multiple common and dynamic vision sensors. Background Art

[0002] In the automatic driving assistance system, it is necessary to observe the surrounding environment, so multiple visual sensors are needed to collect signals from different directions at the same time for subsequent processing and decision-making. At present, automatic driving assistance mainly relies on ordinary visual sensors. On the one hand, ordinary visual sensors have low frame rates, and when facing fast-moving objects, there will be blur and tailing phenomena; on the other hand, their dynamic range is relatively narrow, and unclear recognition will occur under strong or weak light conditions. Dynamic vision sensor DVS (Dynamic Vision Sensor) is a new type of event-based sensor, which triggers events by detecting the brightness change of each pixel: when the brightness change of a pixel reaches a certain threshold, the sensor will output an event, which contains pixel coordinates, occurrence time, and polarity. When the brightness increases above the high threshold, the polarity is positive; conversely, when the brightness decreases below the low threshold, the polarity is negative. Dynamic vision sensors have the advantages of microsecond time resolution and low latency output. At the same time, their dynamic range is as high as 120dB, which is much higher than the 60dB of visual sensors, and is suitable for extreme scene perception such as strong light and weak light. Dynamic vision sensors can collect scene change information, but lack scene texture information. Therefore, in the automatic driving assistance system, the combination of multiple ordinary vision sensors and dynamic vision sensors can complement each other's advantages and disadvantages. It can not only collect complete grayscale information of the image from multiple angles, but also solve the problems of blur or high latency when capturing fast-moving objects, as well as the problem of unclear recognition in strong or weak light conditions.

[0003] In the automated driving assistance system, the visual sensors are usually installed on the top or around the car, while the automated driving domain controller that controls the camera is usually placed in the trunk at the rear of the car, and the two are located several meters apart. To achieve high-bandwidth and long-distance transmission of multiple ordinary visual sensor data, the usual practice is to use the serial deserialization solution of GMSL or FPD-LINK. However, the message format of the dynamic visual sensor data is a custom format, which is completely different from the standard message format of ordinary visual sensors such as RAW / RGB / YUV, and the frame length is not fixed. If the same mode as ordinary visual sensors is used, it will cause packet errors. Therefore, how to solve the long-distance lossless transmission of dynamic visual sensor data has become a difficult problem.

[0004] For a combined array of multiple common vision sensors and multiple dynamic vision sensors, each sensor has its own local clock source. If time synchronization is not performed, the timestamps of data recorded by different sensors at the same time point will be inconsistent, which is not conducive to subsequent information processing and decision-making. However, in the relevant technology, there is no suitable solution for time synchronization processing of a combined array of multiple common vision sensors and multiple dynamic vision sensors. Summary of the invention

[0005] The embodiments of the present application provide a method and system for long-distance synchronous acquisition of multiple ordinary and dynamic vision sensors, which can improve the synchronization of image data acquisition by multiple ordinary vision sensors and multiple dynamic vision sensors.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors, wherein the multiple ordinary and dynamic vision sensors include multiple ordinary vision sensors, multiple dynamic vision sensors, and a control processor. The method is applied to the control processor, and the method includes:

[0007] Determine a target common vision sensor from the plurality of common vision sensors, and determine a target dynamic vision sensor from the plurality of dynamic vision sensors;

[0008] Acquire a first synchronization signal of the target common vision sensor, and send the first synchronization signal to the remaining common vision sensors after performing a first delay compensation process on the first synchronization signal;

[0009] The first synchronization signal is subjected to a second delay compensation process and then sent to the external event trigger pin of the target dynamic vision sensor;

[0010] A second synchronization signal is received from the target dynamic vision sensor, and the second synchronization signal is sent to the remaining dynamic vision sensors after a third delay compensation process, so that the plurality of ordinary vision sensors and the plurality of dynamic vision sensors can collect data synchronously.

[0011] In some embodiments, a serializer, a high-speed coaxial cable, and a deserializer are connected between each of the common vision sensors and the control processor, and the step of acquiring the first synchronization signal of the target common vision sensor includes:

[0012] The first synchronization signal is obtained after being sent by the target ordinary vision sensor and sequentially processed by the target ordinary serializer for serial conversion and the high-speed coaxial cable and the target ordinary deserializer for deserialization. The target ordinary serializer is the serializer connected to the target ordinary vision sensor, and the target ordinary deserializer is the deserializer connected to the target ordinary vision sensor.

[0013] In some embodiments, performing a first delay compensation process on the first synchronization signal and then sending it to the remaining common vision sensors includes:

[0014] Acquire a first serialization delay time generated when the first synchronization signal sequentially passes through the target common serializer and a first deserialization delay time generated when the first synchronization signal sequentially passes through the target common deserializer;

[0015] generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time;

[0016] The first synchronization signal is subjected to a first delay compensation process based on the first delay compensation process time, and the first synchronization signal after the first delay compensation process is sent to the remaining common vision sensors.

[0017] In some embodiments, generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time includes:

[0018] Obtaining a second serialization delay time generated when the first synchronization signal passes through other common serializers and a second deserialization delay time generated when the first synchronization signal passes through other common deserializers in sequence, wherein the other common serializers are the serializers connected to other common vision sensors, and the other common deserializers are the deserializers connected to other common vision sensors;

[0019] Acquire a first synchronization time period for the target common vision sensor to generate the first synchronization signal;

[0020] The first delay compensation processing time is obtained by sequentially subtracting the first serialization delay time, the second serialization delay time, the first deserialization delay time, and the second deserialization delay time from the first synchronization time period.

[0021] In some embodiments, a serializer, a high-speed coaxial cable, and a deserializer are connected between each of the dynamic vision sensors and the control processor, and the first synchronization signal is sent to each of the dynamic vision sensors after performing the second delay compensation processing, including:

[0022] Acquire a third serialization delay time generated when the first synchronization signal sequentially passes through the serializer connected to the dynamic vision sensor and a third deserialization delay time generated when the first synchronization signal sequentially passes through the deserializer connected to the dynamic vision sensor;

[0023] generating a second delay compensation processing time based on a first synchronization time period, the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time;

[0024] The first synchronization signal is subjected to a second delay compensation process based on the second delay compensation process time, and the first synchronization signal after the second delay compensation process is sent to each of the dynamic vision sensors.

[0025] In some embodiments, performing a third delay compensation process on the second synchronization signal and then sending it to the remaining dynamic vision sensors includes:

[0026] Acquire the change pulse phases of the remaining dynamic vision sensors when receiving the second synchronization signal;

[0027] generating a third delay compensation processing time based on the changed pulse phase;

[0028] The second synchronization signal is subjected to a third delay compensation process based on the third delay compensation process time, and the second synchronization signal after the third delay compensation process is sent to the remaining dynamic vision sensors.

[0029] In some embodiments, generating a third delay compensation processing time based on the changed pulse phase includes:

[0030] Acquire a second synchronization signal phase difference between the target dynamic vision sensor and the remaining dynamic vision sensors;

[0031] When the changed pulse phase is a pulse phase delay, obtaining a second synchronization time period for the target dynamic vision sensor to generate the second synchronization signal, and obtaining the third delay compensation processing time based on the difference between the second synchronization time period and the phase difference of the second synchronization signal;

[0032] When the changed pulse phase is a pulse phase advance, the third delay compensation processing time is obtained based on the second synchronization signal phase difference.

[0033] In some embodiments, the serial conversion processing and deserialization processing of the dynamic vision sensor are implemented based on the Tunnel mode of the GMSL chip.

[0034] In some embodiments, data is transmitted between the plurality of ordinary vision sensors, the plurality of dynamic vision sensors and the control processor via a COAX high-speed COAX cable.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a long-distance synchronous acquisition system of multiple ordinary and dynamic vision sensors, wherein the multiple ordinary and dynamic vision sensors include multiple ordinary vision sensors, multiple dynamic vision sensors, and a control processor, and the device is applied to the control processor, and the device includes:

[0036] a target sensor determination module, configured to determine a target common vision sensor from the plurality of common vision sensors, and to determine a target dynamic vision sensor from the plurality of dynamic vision sensors;

[0037] A first delay compensation module, used for acquiring a first synchronization signal of the target common vision sensor, and sending the first synchronization signal to the remaining common vision sensors after performing a first delay compensation process on the first synchronization signal;

[0038] A second delay compensation module, used for performing a second delay compensation process on the first synchronization signal and then sending the signal to each of the dynamic vision sensors;

[0039] The third delay compensation module is used to receive the second synchronization signal emitted by the target dynamic vision sensor, and send the second synchronization signal to the remaining dynamic vision sensors after performing third delay compensation processing, so that multiple ordinary vision sensors and multiple dynamic vision sensors can collect data synchronously.

[0040] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors as described in the first aspect.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, it implements the long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors described in the first aspect above.

[0042] The embodiment of the present application proposes a method and system for long-distance synchronous acquisition of multiple ordinary and dynamic vision sensors. The multiple ordinary and dynamic vision sensors include multiple ordinary vision sensors, multiple dynamic vision sensors and a control processor. The method is applied to the control processor, and the method includes: first, determining a target ordinary vision sensor from multiple ordinary vision sensors, and determining a target dynamic vision sensor from multiple dynamic vision sensors; then, obtaining a first synchronization signal of the target ordinary vision sensor, and performing a first delay compensation process on the first synchronization signal and sending it to the remaining ordinary vision sensors; next, performing a second delay compensation process on the first synchronization signal and sending it to the external event trigger pin of each dynamic vision sensor; finally, receiving a second synchronization signal emitted by the target dynamic vision sensor, and performing a third delay compensation process on the second synchronization signal and sending it to the remaining dynamic vision sensors, so that multiple ordinary vision sensors and multiple dynamic vision sensors can perform data synchronous acquisition. The embodiment of the present application adopts multiple ordinary vision sensors and multiple dynamic vision sensors to collect image data together, so as to utilize the data collection advantages of different types of vision sensors, which can not only collect scene texture information, but also solve the blur or high delay problems when capturing fast-moving objects, as well as the problem of unclear recognition under strong light or weak light conditions, thereby improving the accuracy of data collection; in addition, a first delay compensation process is performed on the first synchronization signal between multiple ordinary vision sensors, a second delay compensation process is performed on the first synchronization signal between ordinary vision sensors and dynamic vision sensors, and a third delay compensation process is performed on the second synchronization signal between multiple dynamic vision sensors, so that corresponding delay compensation processes are performed for different situations and synchronization signals, so that multiple vision sensors can collect data together at the same time, thereby improving the accuracy and synchronization of image data collection by multiple vision sensors.

[0043] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural diagram of a visual acquisition system provided in one embodiment of the present application.

[0045] Figure 2 This is a flow chart of a long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors provided by another embodiment of the present application.

[0046] Figure 3It is a schematic diagram of a process of transmitting a first synchronization signal from a target ordinary vision sensor to other ordinary vision sensors provided by another embodiment of the present application.

[0047] Figure 4 yes Figure 2 Flow chart of step 202 in FIG.

[0048] Figure 5 yes Figure 4 Flowchart of step 402 in FIG.

[0049] Figure 6 It is a schematic diagram of a time curve of a first synchronization signal after a first delay compensation process provided by another embodiment of the present application.

[0050] Figure 7 This is a schematic diagram of information collection triggering synchronization of a dynamic vision sensor provided by another embodiment of the present application.

[0051] Figure 8 It is a schematic diagram of a process of transmitting a first synchronization signal from a target ordinary vision sensor to a dynamic vision sensor provided by another embodiment of the present application.

[0052] Fig. 9 yes Figure 2 Flow chart of step 203 in FIG.

[0053] Fig.10 It is a schematic diagram of a flow chart of transmitting a second synchronization signal from a target dynamic vision sensor to other dynamic vision sensors provided by another embodiment of the present application.

[0054] Fig.11 yes Figure 2 Flow chart of step 204 in FIG.

[0055] Fig.12 yes Fig.11 Flowchart of step 1102 in FIG.

[0056] Fig.13 It is a schematic diagram of a signal waveform curve of a dynamic vision sensor receiving a second synchronization signal provided by another embodiment of the present application.

[0057] Fig.14 It is a structural schematic diagram of a long-distance synchronous acquisition system of multiple ordinary and dynamic vision sensors provided in another embodiment of the present application.

[0058] Fig.15 This is a schematic diagram of the hardware structure of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] In the automatic driving assistance system, it is necessary to observe the surrounding environment, so multiple visual sensors are needed to collect signals from different directions at the same time for subsequent processing and decision-making. At present, automatic driving assistance mainly relies on ordinary visual sensors. On the one hand, ordinary visual sensors have low frame rates, and there will be blur and tailing when facing fast-moving objects; on the other hand, their dynamic range is relatively narrow, and unclear recognition will occur under strong or weak light conditions. Dynamic vision sensor DVS (Dynamic Vision Sensor) is a new type of event-based sensor, which triggers events by detecting changes in the brightness of each pixel: when the brightness change of a pixel reaches a certain threshold, the sensor will output an event, which contains pixel coordinates, occurrence time, and polarity. When the brightness increases above the high threshold, the polarity is positive; conversely, when the brightness decreases below the low threshold, the polarity is negative. Dynamic vision sensors have the advantages of microsecond time resolution and low latency output. At the same time, their dynamic range is as high as 120dB, which is much higher than the 60dB of visual sensors, and is suitable for extreme scene perception such as strong light and weak light. Dynamic vision sensors can collect scene change information, but lack scene texture information. Therefore, in the automatic driving assistance system, the combination of multiple ordinary vision sensors and dynamic vision sensors can complement each other's advantages and disadvantages. It can not only collect complete grayscale information of the image from multiple angles, but also solve the problems of blur or high latency when capturing fast-moving objects, as well as the problem of unclear recognition in strong or weak light conditions.

[0063] In the automated driving assistance system, visual sensors are usually installed on the top or around the car, while the automated driving domain controller that controls the camera is usually placed in the trunk at the rear of the car, and the two are located several meters apart. To achieve high-bandwidth, long-distance transmission of multiple ordinary visual sensor data, the usual practice is to use the GMSL or FPD-LINK serial deserialization solution. However, the message format of dynamic visual sensor data is a custom format, which is completely different from the standard message formats such as RAW / RGB / YUV of ordinary visual sensors, and the frame length is not fixed. If the same mode as ordinary visual sensors is used, it will cause packet errors. Therefore, how to solve the long-distance lossless transmission of dynamic visual sensor data has become a difficult problem.

[0064] For a combined array of multiple common vision sensors and multiple dynamic vision sensors, each sensor has its own local clock source. If time synchronization is not performed, the timestamps of data recorded by different sensors at the same time point will be inconsistent, which is not conducive to subsequent information processing and decision-making. However, in the relevant technology, there is no suitable solution for time synchronization processing of a combined array of multiple common vision sensors and multiple dynamic vision sensors.

[0065] In order to improve the synchronization of image data acquisition by multiple ordinary vision sensors and multiple dynamic vision sensors, the embodiment of the present application adopts multiple ordinary vision sensors and multiple dynamic vision sensors to acquire image data together, so as to utilize the data acquisition advantages of different types of vision sensors, which can not only acquire scene texture information, but also solve the blur or high delay problems when capturing fast-moving objects, as well as the problem of unclear recognition under strong light or weak light conditions, thereby improving the accuracy of data acquisition; in addition, a first delay compensation process is performed on the first synchronization signal between multiple ordinary vision sensors, a second delay compensation process is performed on the first synchronization signal between ordinary vision sensors and dynamic vision sensors, and a third delay compensation process is performed on the second synchronization signal between multiple dynamic vision sensors, so that corresponding delay compensation processes are performed for different situations and synchronization signals, so that multiple vision sensors can acquire data together at the same time, thereby improving the accuracy and synchronization of image data acquisition by multiple vision sensors.

[0066] The following will further describe the long-distance synchronous acquisition method and system of multiple common and dynamic visual sensors provided by the embodiment of the present application. First, the visual acquisition system provided by the embodiment of the present application is described. Figure 1 , is a schematic diagram of the structure of a visual acquisition system provided in an embodiment of the present application. Figure 1As shown in , a plurality of common vision sensors, a plurality of dynamic vision sensors and a control processor are provided in the vision acquisition system. Each common vision sensor or each dynamic vision sensor is connected to the control processor via a GMSL serializer, a high-speed COAX cable and a GMSL deserializer. The control processor may be an FPGA or a CPLD.

[0067] Since the message format of the dynamic vision sensor is completely different from the standard message format of the ordinary vision sensor such as RAW / RGB / YUV, the GMSL chip Tunnel mode is used for serialization and deserialization to ensure the integrity and correctness of the data during the serialization process of the GMSL serializer connected to the dynamic vision sensor and the deserialization process of the GMSL deserializer to ensure the integrity and correctness of the data. In addition, in order to reduce the sensor connection, a high-speed COAX cable is used between multiple dynamic vision sensors, multiple ordinary vision sensors and the control processor to solve the connection problem of the Vsync signal of the ordinary vision sensor, the SYN and TRIG signals of the dynamic vision sensor to the FPGA / CPLD, and the GMSL control channel is used through serialization and deserialization to achieve the sharing of a COAX with the sensor data.

[0068] After collecting data, the visual sensor (including ordinary visual sensor and dynamic visual sensor) converts the MIPI CSI2 interface into GMSL serial data through the GMSL serializer. The GMSL serial data is transmitted to the GMSL deserializer through the high-speed COAX cable. The GMSL deserializer then deserializes the GMSL serial data into MIPI CSI2 signals and sends them to the control processor.

[0069] Since the message format of the dynamic vision sensor is completely different from the standard message format of the ordinary vision sensor such as RAW / RGB / YUV, the tunnel mode of the GMSL chip is used in this embodiment for serial deserialization to ensure the integrity and correctness of the data. In addition, in order to reduce the sensor connection, a high-speed COAX cable is used to solve the connection problem of the Vsync signal of the ordinary vision sensor, the SYN and TRIG signals of the dynamic vision sensor to the FPGA / CPLD, and the control channel of the GMSL is used through serial deserialization to realize the sharing of a COAX cable with the sensor data.

[0070] Ordinary vision sensors, also known as traditional image sensors, generally refer to those sensors that capture images at a fixed frame rate, such as CCD (charge-coupled device) or CMOS (complementary metal oxide semiconductor) sensors. Dynamic vision sensors are a new type of event-based vision sensor that, unlike traditional sensors, do not capture the entire image at a fixed frame rate, but only generate data when the pixel brightness changes. Ordinary vision sensors are suitable for conventional image capture tasks, while dynamic vision sensors show advantages in applications that require high temporal resolution, high dynamic range, and low latency. The two can be used complementary in areas such as autonomous driving assistance systems (ADAS) and robotic vision to achieve better environmental perception and decision-making.

[0071] FPGA (Field Programmable Gate Array) is a semiconductor device that can be configured by programming. It contains programmable logic blocks, input / output blocks, and internal connections. FPGA allows developers to design and modify hardware logic according to specific application requirements without changing the physical hardware.

[0072] CPLD (Complex Programmable Logic Device) is a simpler programmable logic device that uses EEPROM-based memory to store logic configuration.

[0073] COAX cable (Coaxial Cable) is a common transmission line, which consists of a central conductor (inner conductor) and an outer conductive layer (outer conductor), and the two layers are separated by insulating material.

[0074] A GMSL serializer is an integrated circuit that converts parallel data (such as video signals) into a high-speed serial data stream. This conversion allows data to be transmitted over long distances over a single coaxial or twisted pair cable without significantly degrading the signal quality. GMSL serializers are often paired with GMSL deserializers to enable data transmission and reception.

[0075] The GMSL deserializer is the counterpart of the GMSL serializer, which receives a high-speed serial data stream and converts it back to the original parallel data format. In video signal transmission, this means that the deserializer restores the received serial video data into a parallel video signal that can be processed by a display device or image processing system.

[0076] The following is a detailed description of the long-distance synchronous acquisition method of multiple common and dynamic visual sensors in the embodiment of the present application. Figure 2, which is an optional flow chart of a long-distance synchronous acquisition method of multiple common and dynamic visual sensors provided in an embodiment of the present application, Figure 2 The method may include but is not limited to steps 201 to 204. It can also be understood that this embodiment Figure 2 The order of steps 201 to 204 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or increased according to actual needs. The long-distance synchronous acquisition method of multiple common and dynamic visual sensors provided in the embodiment of the present application can be applied to the control processor in the visual acquisition system.

[0077] Step 201: determining a target common vision sensor from a plurality of common vision sensors, and determining a target dynamic vision sensor from a plurality of dynamic vision sensors.

[0078] The following is a detailed description of step 201.

[0079] In some embodiments, in response to an image data acquisition request from a vision acquisition system, the data acquisition synchronization of multiple vision sensors is improved.

[0080] First, it is necessary to determine an ordinary vision sensor from multiple ordinary vision sensors as the target ordinary vision sensor, that is, set one ordinary vision sensor among M ordinary vision sensors to Master mode, and keep the other ordinary vision sensors in Slave mode; similarly, determine a dynamic vision sensor from multiple dynamic vision sensors as the target dynamic vision sensor, that is, set one dynamic vision sensor among N dynamic vision sensors to Master mode, and keep the other dynamic vision sensors in Slave mode. Figure 1 As shown in , ordinary vision sensor No. 1 is used as the target ordinary vision sensor, and dynamic vision sensor No. 1 is used as the target dynamic vision sensor.

[0081] Step 202: Acquire a first synchronization signal of a target common vision sensor, and send the first synchronization signal to other common vision sensors after performing a first delay compensation process.

[0082] The following is a detailed description of step 202.

[0083] In some embodiments, after determining the target common vision sensor and the target dynamic vision sensor, in order to ensure the synchronization of data acquisition between multiple common vision sensors and between the common vision sensor and multiple dynamic vision sensors, the target common vision sensor will output a first synchronization signal Vsync and transmit the first synchronization signal Vsync to the target common serializer (i.e., Figure 11) for serial conversion, and a high-speed coaxial cable (i.e., Figure 1 high-speed COAX cable as shown in ) and the target conventional deserializer (i.e. Figure 1 The GMSL deserializer connected to the ordinary vision sensor No. 1 shown in the figure is transmitted to the control processor, and the control processor uses the first synchronization signal Vsync to synchronize the time between multiple ordinary vision sensors and between the target ordinary vision sensor and multiple dynamic vision sensors.

[0084] Next, multiple common vision sensors are synchronized. In order to achieve synchronous data acquisition of multiple common vision sensors, it is necessary to perform a first delay compensation process based on the first synchronization signal sent by the target common vision sensor and then send it to other common vision sensors. Figure 3 , is a schematic diagram of a process of transmitting a first synchronization signal from a target common vision sensor to other common vision sensors provided by an embodiment of the present application. Figure 3 As shown in FIG. 1 , taking two common vision sensors as an example, after the target common vision sensor sends out the first synchronization signal Vsync, the first synchronization signal Vsync will sequentially pass through the target common serializer (ie, Figure 1 1) and the target common deserializer (i.e., Figure 1 The GMSL deserializer connected to the ordinary visual sensor No. 1 shown in FIG) reaches the control processor; then the control processor performs a first delay compensation process on the first synchronization signal Vsync, and then passes through other ordinary deserializers (i.e., Figure 1 ) and other common serializers (i.e., Figure 1 The GMSL serializer connected to the No. 2 ordinary vision sensor shown in FIG) is sent to the No. 2 ordinary vision sensor.

[0085] How to perform the first delay compensation process on the first synchronization signal will be further described below.

[0086] Reference Figure 4 , performing a first delay compensation process on the first synchronization signal and then sending it to the remaining common vision sensors, including the following steps 401 to 403.

[0087] Step 401: Obtain a first serialization delay time generated when a first synchronization signal passes through a target common serializer and a first deserialization delay time generated when a first synchronization signal passes through a target common deserializer in sequence.

[0088] Step 402: Generate a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time.

[0089] Steps 401 to 402 are described in detail below.

[0090] In some embodiments, when the first synchronization signal passes through the target common serializer and the target common deserializer, processing delay time will inevitably be generated. For example, when the first synchronization signal is connected to the GPIO of the GMSL serial deserializer chip and the GPIO is configured as input, after the input end detects the signal change, the GPIO output end with the same ID as the input end on the other side will copy the input end signal, just like the two sides are straight through. This copying process will generate a delay of us level, but the GMSL chip can be configured as a delay compensation mode to ensure that the delay is a fixed value, thereby ensuring that the delay time of the synchronization signal each time it passes through the GMSL serializer and GMSL deserializer is fixed and can be determined in advance.

[0091] Therefore, after the control processor receives the first synchronization signal sent by the target common vision sensor, the first serialization delay time t0 generated when the first synchronization signal passes through the target common serializer and the first deserialization delay time t1 generated when the first synchronization signal passes through the target common deserializer in sequence are obtained. Next, based on the first serialization delay time t0 and the first deserialization delay time t1, a first delay compensation processing time for performing the first delay compensation processing on each of the remaining common vision sensors is generated, as described in detail below.

[0092] Reference Figure 5 , generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time, including the following steps 501 to 503.

[0093] Step 501: Obtain a second serialization delay time generated when a first synchronization signal passes through other common serializers and a second deserialization delay time generated when a first synchronization signal passes through other common deserializers in sequence.

[0094] Step 502: Acquire a first synchronization time period during which a target common vision sensor generates a first synchronization signal.

[0095] Step 503: Subtract the first serialization delay time, the second serialization delay time, the first deserialization delay time and the second deserialization delay time in sequence from the first synchronization time period to obtain a first delay compensation processing time.

[0096] Steps 501 to 503 are described in detail below.

[0097] In some embodiments, in addition to obtaining the first serialization delay time t0 and the first deserialization delay time t1, each of the remaining ordinary visual sensors is targeted one by one, and then the second serialization delay time t2 generated when the first synchronization signal passes through other ordinary serializers and the second deserialization delay time t3 generated when the first synchronization signal passes through other ordinary deserializers in sequence during the process of being transmitted to the corresponding ordinary visual sensor is obtained.

[0098] At the same time, it is also necessary to obtain the first synchronization time period ΔT of the target common vision sensor to generate the first synchronization signal, that is, the time interval between generating two consecutive first synchronization signals, which is also the frame period of the target common vision sensor.

[0099] Then, based on the first synchronization time period ΔT, the first serialization delay time t0, the second serialization delay time t2, the first deserialization delay time t1 and the second deserialization delay time t3 are subtracted in sequence, and the first delay compensation processing time Δt1 = ΔT-t0-t1-t2-t3 between the target ordinary vision sensor and each other ordinary vision sensor can be obtained.

[0100] Step 403: Perform a first delay compensation process on the first synchronization signal based on the first delay compensation process time, and send the first synchronization signal after the first delay compensation process to other common vision sensors.

[0101] Step 403 is described in detail below.

[0102] In some embodiments, after obtaining the first delay compensation processing time Δt1 corresponding to each other ordinary vision sensor, the control processor will perform a first delay compensation processing on the first synchronization signal Vsync according to the first delay compensation processing time Δt1, and send the first synchronization signal after the first delay compensation processing to the remaining corresponding ordinary vision sensors in turn through the GMSL deserializer and GMSL serializer connected to the remaining ordinary vision sensors, thereby ensuring the phase synchronization of all ordinary vision sensors (i.e., the rising edge or falling edge phase alignment), and then realizing synchronous data information acquisition.

[0103] Reference Figure 6 , is a schematic diagram of a time curve of a first synchronization signal after a first delay compensation process provided by an embodiment of the present application. Figure 6 As shown in FIG. 1 , taking the example that the first synchronization signal Vsync is output by the No. 1 common vision module working in the Master mode and finally sent to the Vsync input end of the No. 2 common vision module working in the Slave mode, the data flow of the first synchronization signal is as follows: Figure 3 As shown. Figure 6As shown, the No. 1 common vision sensor working in Master mode outputs a Vsync signal, which is delayed by t0 after passing through the GMSL serializer of module No. 1, delayed by t1 after passing through the deserializer of module No. 1, and then delayed by △t1 after delay compensation in the control processor, and delayed by t2 after passing through the deserializer of module No. 2, and finally delayed by t3 after passing through the serializer of module No. 2, and finally sent to the Vsync pin of the No. 2 common vision sensor working in Slave mode, the rising edge is aligned with T+ΔT, ΔT is a Vsync signal cycle. To ensure that the rising edge phase of the pulse received by the 2# common vision sensor is the same as the rising edge phase of the pulse sent by the 1# common vision sensor, the first delay compensation time of FPGA / CLPD is Δt1, which is Δt1=ΔT-t0-t1-t2-t3.

[0104] It is understandable that the delay values ​​of GPIOs with different IDs are configurable within a certain delay range, i.e. Figure 6 The delay values ​​t0-t3 in can be configured through registers, and the minimum granularity of the delay value is at the microsecond level. When the Vsync signal period ΔT and the serial-to-deserial delay t0-t3 values ​​are known, the total delay value of the first delay compensation time Δt1 can be calculated. The Vsync phase can be adjusted by counting the local high-frequency clock pulses inside the FPGA / CLPD to perform delay compensation.

[0105] Through the above steps 401 to 403, and steps 501 to 503, by using the GMSL serializer and GMSL deserializer that can set a fixed processing delay, the processing delay generated when the first synchronization signal is sent from the target ordinary vision sensor to each of the remaining ordinary vision sensors is predetermined, and combined with the first synchronization time period for generating the first synchronization signal, the first delay compensation time corresponding to the phase synchronization between each of the remaining ordinary vision sensors and the target ordinary vision sensor can be accurately determined, so that the first delay compensation time can be used to accurately realize the synchronous image data acquisition of multiple ordinary vision sensors together, thereby improving the synchronization of data acquisition of multiple ordinary and dynamic vision sensors.

[0106] Step 203: Perform a second delay compensation process on the first synchronization signal and then send the signal to the external event trigger pin of each dynamic vision sensor.

[0107] Step 203 is described in detail below.

[0108] In some embodiments, in order to achieve time synchronization between the common vision sensor and the dynamic vision sensor, the Vsync signal of the common vision sensor working in the Master mode needs to be sent to the external event trigger pins TRIG of the N dynamic vision sensors through the control processor. Figure 7 , is a schematic diagram of triggering synchronization of information collection of a dynamic visual sensor provided in an embodiment of the present application. Figure 7 As shown in , the TRIG signal of the dynamic vision sensor is an external event trigger signal. When the TRIG signal is set as input, Figure 7 As shown, when the TRIG signal detects a rising edge or a falling edge (the accuracy can reach the microsecond level), the dynamic vision sensor will generate a TRIG event, which will record the timestamp when the external TRIG signal changes, such as [T2, T0] or [T3, T1]. The time difference between T2 and T0 (or T3 and T1) is a frame period of an ordinary vision sensor (such as 30Hz or 60Hz, etc.). The dynamic vision sensor generates multiple positive polarity (increase in brightness) and negative polarity (decrease in brightness) event information representing brightness changes during the frame period. The event information contains timestamps, coordinates, and polarity. Therefore, with the alignment relationship between the timestamp recorded by the external trigger event of the dynamic vision sensor and the rising edge or falling edge of the Vsync signal of each frame of the ordinary vision sensor, the time synchronization of ordinary vision sensors of different sensor types and dynamic vision sensors can be achieved.

[0109] Reference Figure 8 , is a schematic diagram of a process of transmitting a first synchronization signal from a target ordinary visual sensor to a dynamic visual sensor provided by an embodiment of the present application. Figure 8 As shown in FIG. 1 , taking a common vision sensor and a dynamic vision sensor as examples, after the target common vision sensor sends out the first synchronization signal Vsync, the first synchronization signal Vsync will sequentially pass through the target common serializer (ie, Figure 1 1) and the target common deserializer (i.e., Figure 1 The GMSL deserializer connected to ordinary vision sensor No. 1 shown in the figure) arrives at the control processor; then the control processor performs a second delay compensation process on the first synchronization signal Vsync, and then sends it to the external event trigger pin TRIG of dynamic vision sensor No. 1 through the GMSL deserializer connected to dynamic vision sensor No. 1 and the GMSL serializer connected to dynamic vision sensor No. 1 in sequence.

[0110] How to perform the second delay compensation process on the first synchronization signal will be further described below.

[0111] Reference Fig. 9, the first synchronization signal is sent to each dynamic vision sensor after the second delay compensation processing, including the following steps 901 to 903.

[0112] Step 901: Obtain a third serialization delay time generated when a first synchronization signal passes through a serializer connected to a dynamic vision sensor and a third deserialization delay time generated when a first synchronization signal passes through a deserializer connected to a dynamic vision sensor.

[0113] Step 902: Generate a second delay compensation processing time based on a first synchronization time period, a first serialization delay time, a first deserialization delay time, a third serialization delay time, and a third deserialization delay time.

[0114] Step 903: Perform a second delay compensation process on the first synchronization signal based on the second delay compensation processing time, and send the first synchronization signal after the second delay compensation process to the target dynamic vision sensor.

[0115] Steps 901 to 903 are described in detail below.

[0116] In some embodiments, similar to the first delay compensation processing described in steps 401 to 403 above, when performing the second delay compensation processing on the first synchronization signal Vsync, it is necessary to pre-acquire the third serialization delay time t4 generated when the first synchronization signal passes through the GMSL serializer connected to each dynamic vision sensor and the third deserialization delay time t5 generated when the first synchronization signal passes through the GMSL deserializer connected to each dynamic vision sensor in sequence.

[0117] Then, based on the first synchronization time period ΔT, the first serialization delay time t0, the third serialization delay time t4, the first deserialization delay time t1 and the third deserialization delay time t5 are subtracted in sequence, and the second delay compensation processing time Δt2 = ΔT-t0-t1-t4-t5 corresponding to the target ordinary vision sensor and each dynamic vision sensor can be obtained.

[0118] Then, the control processor will perform a second delay compensation processing on the first synchronization signal Vsync according to the second delay compensation processing time Δt2, and send the first synchronization signal after the second delay compensation processing to the external event trigger pin TRIG of each corresponding dynamic vision sensor through the GMSL deserializer and GMSL serializer connected to each vision sensor in turn, thereby ensuring the time synchronization association between the target ordinary vision sensor and each dynamic vision sensor (that is, the rising edge or falling edge of the sending end of the target ordinary vision sensor Vsync is phase-aligned with the rising edge or falling edge of the Vsync signal received by the external event trigger pin TRIG of the dynamic vision sensor), thereby realizing synchronous data information acquisition.

[0119] Through the above steps 901 to 903, the GMSL serializer and GMSL deserializer that can set a fixed processing delay are used to predetermine the processing delay generated when the first synchronization signal is sent from the target ordinary vision sensor to the target dynamic vision sensor, and combined with the first synchronization time period for generating the first synchronization signal, the second delay compensation time can be accurately determined, so that the second delay compensation time can be combined with the timestamp recorded by the external trigger event of the dynamic vision sensor and the alignment relationship between the rising edge or falling edge of the first synchronization signal of each frame of the ordinary vision sensor, so as to achieve time synchronization between ordinary vision sensors and dynamic vision sensors of different sensor types, thereby improving the synchronization of data acquisition by multiple ordinary and dynamic vision sensors.

[0120] Step 204: receiving a second synchronization signal sent by the target dynamic vision sensor, and sending the second synchronization signal to the remaining dynamic vision sensors after performing a third delay compensation process, so that the multiple dynamic vision sensors can perform data synchronous collection.

[0121] Step 204 is described in detail below.

[0122] In some embodiments, in order to ensure the synchronization of data acquisition by multiple dynamic vision sensors, the target dynamic vision sensor working in the Master mode outputs a second synchronization signal SYN to be sent to the remaining dynamic vision sensors through the control processor, thereby achieving data acquisition synchronization between the multiple dynamic vision sensors. That is, the target dynamic vision sensor sends the second synchronization signal SYN to the SYN input pins of N-1 dynamic vision sensors working in the Slave mode through the FPGA / CPLD. The second synchronization signal SYN is the reference clock for time stamping the dynamic vision sensor event.

[0123] In some embodiments, after the ordinary GPIO of the control processor FPGA / CPLD receives the second synchronization signal SYN, the control processor FPGA / CPLD performs a third delay compensation process on the second synchronization signal SYN.

[0124] Reference Fig.10 , is a schematic diagram of a flow chart of a second synchronization signal transmitted from a target dynamic vision sensor to other dynamic vision sensors provided by an embodiment of the present application. Fig.10 As shown in FIG. 1 , taking two dynamic vision sensors as an example, after the target dynamic vision sensor sends the second synchronization signal SYN, the second synchronization signal SYN will sequentially pass through the target dynamic serializer (ie, Figure 1 1) and the target dynamic deserializer (i.e., Figure 1The GMSL deserializer connected to the dynamic vision sensor No. 1 shown in FIG) arrives at the control processor; then the control processor performs a third delay compensation process on the second synchronization signal SYN, and then passes through other dynamic deserializers (i.e., Figure 1 ) and other dynamic serializers (i.e., Figure 1 The GMSL serializer connected to other dynamic vision sensors shown in FIG) is sent to the remaining dynamic vision sensors.

[0125] The following will further describe how to perform the third delay compensation process on the second synchronization signal. Fig.11 , performing a third delay compensation process on the second synchronization signal and then sending it to the remaining dynamic vision sensors, including the following steps 1101 to 1103.

[0126] Step 1101: Acquire the changing pulse phases of the remaining dynamic vision sensors when receiving the second synchronization signal.

[0127] Step 1102: Generate a third delay compensation processing time based on the changed pulse phase.

[0128] Steps 1101 to 1102 are described in detail below.

[0129] For the phase alignment of the second synchronization signal SYN between dynamic vision sensors, the third delay compensation processing time corresponding to the phase compensation calculation method is different from the calculation method of the first delay compensation processing time and the second delay compensation processing time described above. Because compared with the first synchronization time period (generally tens of milliseconds) of the ordinary vision sensor to generate the first synchronization signal Vsync, the time period for the dynamic vision sensor to generate the second synchronization signal SYN is very short (generally in the microsecond level), and the minimum granularity of the fixed delay setting of the chip in the GMSL serializer and GMSL deserializer is also in the microsecond level, and the delay can be configured to range from a few microseconds to hundreds of microseconds, and the value of t0+t1+t2+t3 in the formula for calculating the first delay compensation processing time above has exceeded the time period of a second synchronization signal SYN. ​​Therefore, for the synchronization of multiple dynamic vision sensors, it is only necessary to ensure that the SYN signal phases of the target dynamic vision sensor and the remaining dynamic vision sensors are aligned.

[0130] Therefore, in order to align the SYN signal phases of the target dynamic vision sensor and the remaining dynamic vision sensors, it is first necessary to obtain the changing pulse phases of the remaining dynamic vision sensors when they receive the second synchronization signal SYN. ​​It is understandable that after the GPIO delay of the serializer and deserializer is set, the phase difference of the SYN signal between the target dynamic vision sensor and the remaining dynamic vision sensors is also determined. Therefore, it is only necessary to measure the changing pulse phase and phase difference between the target dynamic vision sensor and the remaining dynamic vision sensors during debugging, and it can be used in practical applications.

[0131] Then, according to the different changing pulse phase conditions between the target dynamic vision sensor and the remaining dynamic vision sensors, a third delay compensation processing time corresponding to each remaining dynamic vision sensor is generated, as described in detail below.

[0132] In some embodiments, generating a third delay compensation processing time based on a changing pulse phase includes the following steps 1201 to 1203 .

[0133] Step 1201: Acquire a second synchronization signal phase difference between a target dynamic vision sensor and other dynamic vision sensors.

[0134] Step 1202: When the changed pulse phase is a pulse phase delay, obtain the second synchronization time period of the target dynamic vision sensor to generate a second synchronization signal, and obtain the third delay compensation processing time based on the difference between the second synchronization time period and the phase difference of the second synchronization signal.

[0135] Step 1203: When the changed pulse phase is a pulse phase advance, a third delay compensation processing time is obtained based on the second synchronization signal phase difference.

[0136] Steps 1201 to 1202 are described in detail below.

[0137] In some embodiments, the second synchronization signal phase difference ΔT_dly between the target dynamic vision sensor and each of the remaining dynamic vision sensors is firstly acquired.

[0138] Then, for each of the remaining dynamic vision sensors one by one, when the changing pulse phase between the target dynamic vision sensor and the dynamic vision sensor is a pulse phase delay, the second synchronization time period ΔTp of the target dynamic vision sensor to generate a second synchronization signal is further obtained, and based on the difference between the second synchronization time period and the phase difference of the second synchronization signal, the third delay compensation processing time is obtained as Δt3=ΔTp-ΔT_dly.

[0139] When the pulse phase change between the target dynamic vision sensor and the dynamic vision sensor is a pulse phase lead, the third delay compensation processing time obtained based on the second synchronization signal phase difference is Δt3 = ΔT_dly.

[0140] Reference Fig.13 , is a schematic diagram of a signal waveform curve of a dynamic vision sensor receiving a second synchronization signal provided by an embodiment of the present application. Fig.13 As shown in , taking the waveforms of the synchronization SYN signals of three dynamic vision sensors as an example, compared with the waveform of the synchronization signal SYN of the target dynamic vision sensor No. 1 (i.e., channel 1), the waveform of the synchronization signal SYN of the dynamic vision sensor No. 2 is delayed (i.e., the change pulse phase is the rising edge pulse phase delay), and the second synchronization signal phase difference between them is ΔT_dly1; in contrast, the waveform of the synchronization signal SYN of the dynamic vision sensor No. 3 is advanced (i.e., the change pulse phase is the rising edge pulse phase advance), and the second synchronization signal phase difference between No. 3 and No. 1 is ΔT_dly2. Then, the corresponding third delay compensation processing time Δt3 is calculated according to the corresponding situation.

[0141] Step 1103: Perform a third delay compensation process on the second synchronization signal based on the third delay compensation processing time, and send the second synchronization signal after the third delay compensation process to the remaining dynamic vision sensors.

[0142] Step 1103 is described in detail below.

[0143] In some embodiments, after obtaining the third delay compensation processing time Δt3 corresponding to each of the remaining dynamic vision sensors, the control processor will perform a third delay compensation processing on the second synchronization signal SYN according to each third delay compensation processing time Δt3, and send the second synchronization signal SYN after the third delay compensation processing in turn through the GMSL deserializer and GMSL serializer connected to the remaining dynamic vision sensors to each corresponding dynamic vision sensor, thereby ensuring the phase synchronization of multiple dynamic vision sensors (i.e., the rising edge or falling edge phase alignment), and thus realizing synchronous data information acquisition.

[0144] Through the above steps 1101 to 1103, and steps 1201 to 1203, the third delay compensation processing time for compensating the phase synchronization between the target dynamic vision sensor and each of the remaining dynamic vision sensors is accurately determined according to the changing pulse phase between the target dynamic vision sensor and each of the remaining dynamic vision sensors and the phase difference of the second synchronization signal. Therefore, the third delay compensation time can be used to accurately realize synchronous image data acquisition of multiple dynamic vision sensors, thereby improving the synchronization of data acquisition of multiple ordinary and dynamic vision sensors.

[0145] In some embodiments, after all common vision sensors and dynamic vision sensors in the vision acquisition system are phase-synchronized according to the first synchronization signal or the second synchronization signal, data will be synchronously acquired together to effectively improve the accuracy and synchronization of image data acquisition by multiple vision sensors. The multi-channel common and dynamic vision sensors provided in this embodiment can be applied to long-distance transmission scenarios such as autonomous driving; the synchronization signals and external event trigger signals of the common and dynamic vision sensors are used to achieve the us-level timestamp synchronization of multiple common and dynamic vision sensors under long-distance transmission, and the synchronization signal, external event trigger signal and sensor data share a high-speed cable, which simplifies the hardware design and saves the cable cost.

[0146] The embodiment of the present application proposes a method and system for long-distance synchronous acquisition of multiple ordinary and dynamic vision sensors. The method is applied to a control processor, and the method includes: first, determining a target ordinary vision sensor from a plurality of ordinary vision sensors, and determining a target dynamic vision sensor from a plurality of dynamic vision sensors; then, obtaining a first synchronization signal sent by the target ordinary vision sensor and sequentially passing through a target ordinary serializer for serial conversion processing, a high-speed coaxial cable and a target ordinary deserializer for deserialization processing, and obtaining a first serialization delay time generated when the first synchronization signal sequentially passes through the target ordinary serializer and a first deserialization delay time generated when the first synchronization signal sequentially passes through other ordinary serializers; The second serialization delay time generated and the second deserialization delay time generated by other ordinary deserializers, other ordinary serializers are serializers connected to other ordinary visual sensors, and other ordinary deserializers are deserializers connected to other ordinary visual sensors. The first synchronization time period for the target ordinary visual sensor to generate a first synchronization signal is obtained. Based on the first synchronization time period, the first serialization delay time, the second serialization delay time, the first deserialization delay time and the second deserialization delay time are subtracted in sequence to obtain a first delay compensation processing time. Based on the first delay compensation processing time, the first synchronization signal is subjected to a first delay compensation processing, and the first synchronization signal after the first delay compensation processing is sent to the remaining ordinary visual sensors; next, the first synchronization signal is obtained. A synchronization signal sequentially passes through the third serialization delay time generated when the serializer is connected to the dynamic vision sensor and the third deserialization delay time generated when the dynamic vision sensor is connected to the deserializer, and generates a second delay compensation processing time based on the first synchronization time period, the first serialization delay time, the first deserialization delay time, the third serialization delay time and the third deserialization delay time. The first synchronization signal is subjected to a second delay compensation processing time, and the first synchronization signal after the second delay compensation processing is sent to each dynamic vision sensor; finally, the second synchronization signal emitted by the target dynamic vision sensor is received, and the change pulse phases of the remaining dynamic vision sensors when receiving the second synchronization signal are obtained, so as to obtain the target dynamic vision sensor. The second synchronization signal phase difference between the visual sensor and the remaining dynamic visual sensors, when the changing pulse phase is a pulse phase delay, obtains the second synchronization time period of the target dynamic visual sensor to generate the second synchronization signal, and based on the difference between the second synchronization time period and the second synchronization signal phase difference, obtains the third delay compensation processing time, when the changing pulse phase is a pulse phase advance, obtains the third delay compensation processing time based on the second synchronization signal phase difference, performs the third delay compensation processing on the second synchronization signal based on the third delay compensation processing time, and sends the second synchronization signal after the third delay compensation processing to the remaining dynamic visual sensors, so that multiple ordinary visual sensors and multiple dynamic visual sensors can perform data synchronous collection.

[0147] In the embodiment of the present application, a plurality of ordinary visual sensors and a plurality of dynamic visual sensors are used to collect image data together, so as to utilize the advantages of data collection of different types of visual sensors, so as to not only collect scene texture information, but also solve the problem of blur or high delay when capturing fast-moving objects, as well as the problem of unclear recognition under strong light or weak light conditions, thereby improving the accuracy of data collection; in addition, by using a GMSL serializer and a GMSL deserializer that can set a fixed processing delay, the processing delay generated when the first synchronization signal is sent from the target ordinary visual sensor to each of the remaining ordinary visual sensors is predetermined, and combined with the first synchronization time period for generating the first synchronization signal, the first delay compensation time corresponding to the phase synchronization between each of the remaining ordinary visual sensors and the target ordinary visual sensor can be accurately determined, so that the first delay compensation time can be used to accurately realize the synchronous image data collection of multiple ordinary visual sensors together, thereby improving the synchronization of data collection; and, by using a GMSL serializer and a GMSL deserializer that can set a fixed processing delay, the processing delay generated when the first synchronization signal is sent from the target ordinary visual sensor to the target dynamic visual sensor is predetermined. The processing delay generated in the process and combined with the first synchronization time period for generating the first synchronization signal can accurately determine the second delay compensation time, so that the second delay compensation time can be combined with the timestamp recorded by the external trigger event of the dynamic vision sensor and the alignment relationship of the rising edge or falling edge of the first synchronization signal of each frame of the ordinary vision sensor to achieve time synchronization between ordinary vision sensors and dynamic vision sensors of different sensor types, thereby improving the synchronization of data acquisition of multiple ordinary and dynamic vision sensors; and, according to the change pulse phase between the target dynamic vision sensor and each of the remaining dynamic vision sensors and the phase difference of the second synchronization signal, accurately determine the third delay compensation processing time for compensating the phase synchronization between the target dynamic vision sensor and each of the remaining dynamic vision sensors, so that the third delay compensation time can be used to accurately realize the synchronous image data acquisition of multiple dynamic vision sensors together, thereby improving the synchronization of data acquisition of multiple ordinary and dynamic vision sensors, so that multiple vision sensors can collect data together at the same time, further improving the accuracy and synchronization of image data acquisition of multiple vision sensors.

[0148] The embodiment of the present application also provides a long-distance synchronous acquisition system of multiple common and dynamic visual sensors, which can implement the long-distance synchronous acquisition method of multiple common and dynamic visual sensors, referring to Fig.14 , the device 1400 comprises:

[0149] The target sensor determination module 1410 is used to determine a target common vision sensor from a plurality of common vision sensors, and to determine a target dynamic vision sensor from a plurality of dynamic vision sensors;

[0150] A first delay compensation module 1420 is used to obtain a first synchronization signal of a target common vision sensor, and send the first synchronization signal to other common vision sensors after performing a first delay compensation process on the first synchronization signal;

[0151] The second delay compensation module 1430 is used to perform second delay compensation processing on the first synchronization signal and then send it to the target dynamic vision sensor;

[0152] The third delay compensation module 1440 is used to receive the second synchronization signal sent by the target dynamic vision sensor, and send the second synchronization signal to the remaining dynamic vision sensors after performing third delay compensation processing, so that multiple ordinary vision sensors and multiple dynamic vision sensors can collect data synchronously.

[0153] In some embodiments, the first delay compensation module 1420 is further configured to:

[0154] A first synchronization signal is obtained which is sent by a target common visual sensor and is sequentially processed by a target common serializer for serial conversion, a high-speed coaxial cable and a target common deserializer for deserialization.

[0155] In some embodiments, the first delay compensation module 1420 is further configured to:

[0156] Acquire a first serialization delay time generated when the first synchronization signal sequentially passes through a target common serializer and a first deserialization delay time generated when the first synchronization signal sequentially passes through a target common deserializer;

[0157] generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time;

[0158] The first synchronization signal is subjected to a first delay compensation process based on the first delay compensation process time, and the first synchronization signal after the first delay compensation process is sent to the remaining common vision sensors.

[0159] In some embodiments, the first delay compensation module 1420 is further configured to:

[0160] Obtaining a second serialization delay time generated when the first synchronization signal passes through other common serializers and a second deserialization delay time generated when the first synchronization signal passes through other common deserializers in sequence, where the other common serializers are serializers connected to other common vision sensors, and the other common deserializers are deserializers connected to other common vision sensors;

[0161] Acquire a first synchronization time period for a target common vision sensor to generate a first synchronization signal;

[0162] The first delay compensation processing time is obtained by sequentially subtracting the first serialization delay time, the second serialization delay time, the first deserialization delay time and the second deserialization delay time based on the first synchronization time period.

[0163] In some embodiments, the second delay compensation module 1430 is further configured to:

[0164] Acquire a third serialization delay time generated when the first synchronization signal sequentially passes through a target dynamic serializer and a third deserialization delay time generated when the first synchronization signal sequentially passes through a target dynamic deserializer;

[0165] generating a second delay compensation processing time based on the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time;

[0166] The first synchronization signal is subjected to a second delay compensation process based on the second delay compensation process time, and the first synchronization signal subjected to the second delay compensation process is sent to the target dynamic vision sensor.

[0167] In some embodiments, the third delay compensation module 1440 is further configured to:

[0168] Acquire the changing pulse phases of the remaining dynamic vision sensors when they receive the second synchronization signal;

[0169] generating a third delay compensation processing time based on the changed pulse phase;

[0170] The second synchronization signal is subjected to a third delay compensation process based on the third delay compensation process time, and the second synchronization signal subjected to the third delay compensation process is sent to the remaining dynamic vision sensors.

[0171] In some embodiments, the third delay compensation module 1440 is further configured to:

[0172] Acquire a second synchronization signal phase difference between the target dynamic vision sensor and the remaining dynamic vision sensors;

[0173] When the changed pulse phase is a pulse phase delay, a second synchronization time period of the target dynamic vision sensor generating a second synchronization signal is obtained, and a third delay compensation processing time is obtained based on a difference between the second synchronization time period and a phase difference of the second synchronization signal;

[0174] When the changed pulse phase is a pulse phase advance, the third delay compensation processing time is obtained based on the second synchronization signal phase difference.

[0175] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, the specific implementation method of the long-distance synchronous acquisition system of multiple ordinary and dynamic vision sensors is basically the same as the specific implementation method of the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors mentioned above, and will not be repeated here.

[0176] In the embodiment of the present application, the long-distance synchronous acquisition method and system of multiple ordinary and dynamic vision sensors use multiple ordinary vision sensors and multiple dynamic vision sensors to collect image data together, so as to utilize the data acquisition advantages of different types of vision sensors, not only to collect scene texture information, but also to solve the problem of blur or high delay when capturing fast-moving objects, as well as the problem of unclear recognition under strong light or weak light conditions, thereby improving the accuracy of data acquisition; in addition, by using a GMSL serializer and a GMSL deserializer that can set a fixed processing delay, the processing delay generated when the first synchronization signal is sent from the target ordinary vision sensor to each of the remaining ordinary vision sensors is predetermined, and combined with the first synchronization time period for generating the first synchronization signal, the first delay compensation time corresponding to the phase synchronization between each of the remaining ordinary vision sensors and the target ordinary vision sensor can be accurately determined, so that the first delay compensation time can be used to accurately realize the synchronous image data acquisition of multiple ordinary vision sensors together, thereby improving the synchronization of data acquisition; and, by using a GMSL serializer and a GMSL deserializer that can set a fixed processing delay, the first synchronization signal is predetermined from the target ordinary vision sensor to the target ordinary vision sensor. The processing delay generated by the pulse generator sent to the target dynamic vision sensor, combined with the first synchronization time period for generating the first synchronization signal, can accurately determine the second delay compensation time, so that the second delay compensation time can be combined with the timestamp recorded by the external trigger event of the dynamic vision sensor and the alignment relationship of the rising edge or falling edge of the first synchronization signal of each frame of the ordinary vision sensor, so as to achieve time synchronization between ordinary vision sensors and dynamic vision sensors of different sensor types, thereby improving the synchronization of data acquisition of multiple ordinary and dynamic vision sensors; and, according to the change pulse phase between the target dynamic vision sensor and each of the remaining dynamic vision sensors and the phase difference of the second synchronization signal, accurately determine the third delay compensation processing time for compensating the phase synchronization between the target dynamic vision sensor and each of the remaining dynamic vision sensors, so that the third delay compensation time can be used to accurately realize the synchronous image data acquisition of multiple dynamic vision sensors together, thereby improving the synchronization of data acquisition of multiple ordinary and dynamic vision sensors, so that multiple vision sensors can collect data together at the same time, further improving the accuracy and synchronization of image data acquisition of multiple vision sensors.

[0177] The present application also provides an electronic device, including:

[0178] at least one memory;

[0179] at least one processor;

[0180] at least one program;

[0181] The program is stored in the memory, and the processor executes the at least one program to implement the long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an autonomous driving domain controller, etc.

[0182] See also Fig.15 , Fig.15 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0183] The processor 1501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0184] The memory 1502 can be implemented in the form of NOR FLASH, NAND FLASH, EEPROM, etc. The memory 1502 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1502, and the processor 1501 calls and executes the long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors in the embodiments of this application;

[0185] Input / output interface 1503, used to implement information input and output;

[0186] Communication interface 1504, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0187] A bus 1505 that transmits information between the various components of the device (e.g., the processor 1501, the memory 1502, the input / output interface 1503, and the communication interface 1504);

[0188] The processor 1501 , the memory 1502 , the input / output interface 1503 and the communication interface 1504 are connected to each other in communication within the device via the bus 1505 .

[0189] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors is implemented.

[0190] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0191] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0192] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0195] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0196] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0201] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A long-distance synchronous acquisition method for multiple common and dynamic visual sensors, characterized in that: The method is applied to a control processor, and the method comprises: Determine a target ordinary vision sensor from a plurality of ordinary vision sensors, and determine a target dynamic vision sensor from a plurality of dynamic vision sensors; Acquire a first synchronization signal of the target common vision sensor, and send the first synchronization signal to the remaining common vision sensors after performing a first delay compensation process on the first synchronization signal; Performing a second delay compensation process on the first synchronization signal and then sending it to the external event trigger pin of each of the dynamic vision sensors; A second synchronization signal is received from the target dynamic vision sensor, and the second synchronization signal is sent to the remaining dynamic vision sensors after a third delay compensation process, so that the plurality of ordinary vision sensors and the plurality of dynamic vision sensors can collect data synchronously.

2. The long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors according to claim 1 is characterized in that: A serializer, a high-speed coaxial cable, and a deserializer are connected between each of the common vision sensors and the control processor, and the step of acquiring the first synchronization signal of the target common vision sensor includes: The first synchronization signal is obtained after being sent by the target ordinary vision sensor and sequentially processed by the target ordinary serializer for serial conversion and the high-speed coaxial cable and the target ordinary deserializer for deserialization. The target ordinary serializer is the serializer connected to the target ordinary vision sensor, and the target ordinary deserializer is the deserializer connected to the target ordinary vision sensor.

3. The long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors according to claim 2 is characterized in that: The step of performing a first delay compensation process on the first synchronization signal and then sending the first synchronization signal to the remaining common vision sensors includes: Acquire a first serialization delay time generated when the first synchronization signal sequentially passes through the target common serializer and a first deserialization delay time generated when the first synchronization signal sequentially passes through the target common deserializer; generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time; The first synchronization signal is subjected to a first delay compensation process based on the first delay compensation process time, and the first synchronization signal after the first delay compensation process is sent to the remaining common vision sensors.

4. The long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors according to claim 3 is characterized in that: The generating a first delay compensation processing time based on the first serialization delay time and the first deserialization delay time comprises: Obtaining a second serialization delay time generated when the first synchronization signal passes through other common serializers and a second deserialization delay time generated when the first synchronization signal passes through other common deserializers in sequence, wherein the other common serializers are the serializers connected to other common vision sensors, and the other common deserializers are the deserializers connected to other common vision sensors; Acquire a first synchronization time period for the target common vision sensor to generate the first synchronization signal; The first delay compensation processing time is obtained by sequentially subtracting the first serialization delay time, the second serialization delay time, the first deserialization delay time, and the second deserialization delay time from the first synchronization time period.

5. The long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors according to claim 3 is characterized in that: A serializer, a high-speed coaxial cable, and a deserializer are connected between each of the dynamic vision sensors and the control processor, and the first synchronization signal is sent to each of the dynamic vision sensors after performing a second delay compensation process, including: Acquire a third serialization delay time generated when the first synchronization signal sequentially passes through the serializer connected to the dynamic vision sensor and a third deserialization delay time generated when the first synchronization signal sequentially passes through the deserializer connected to the dynamic vision sensor; generating a second delay compensation processing time based on a first synchronization time period, the first serialization delay time, the first deserialization delay time, the third serialization delay time, and the third deserialization delay time; The first synchronization signal is subjected to a second delay compensation process based on the second delay compensation process time, and the first synchronization signal after the second delay compensation process is sent to each of the dynamic vision sensors.

6. The long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors according to claim 2 is characterized in that: The step of performing a third delay compensation process on the second synchronization signal and then sending the signal to the remaining dynamic vision sensors comprises: Acquire the change pulse phases of the remaining dynamic vision sensors when receiving the second synchronization signal; generating a third delay compensation processing time based on the changed pulse phase; The second synchronization signal is subjected to a third delay compensation process based on the third delay compensation process time, and the second synchronization signal after the third delay compensation process is sent to the remaining dynamic vision sensors.

7. The long-distance synchronous acquisition method of multiple common and dynamic visual sensors according to claim 6 is characterized in that: The step of generating a third delay compensation processing time based on the changed pulse phase comprises: Acquire a second synchronization signal phase difference between the target dynamic vision sensor and the remaining dynamic vision sensors; When the changed pulse phase is a pulse phase delay, obtaining a second synchronization time period for the target dynamic vision sensor to generate the second synchronization signal, and obtaining the third delay compensation processing time based on the difference between the second synchronization time period and the phase difference of the second synchronization signal; When the changed pulse phase is a pulse phase advance, the third delay compensation processing time is obtained based on the second synchronization signal phase difference.

8. The long-distance synchronous acquisition method of multiple common and dynamic visual sensors according to claim 2 is characterized in that: The serial conversion processing and deserialization processing of the dynamic vision sensor are implemented based on the Tunnel mode of the GMSL chip.

9. The long-distance synchronous acquisition method of multiple common and dynamic visual sensors according to claim 1 is characterized in that: Data is transmitted between the plurality of ordinary vision sensors, the plurality of dynamic vision sensors and the control processor via a COAX high-speed COAX cable.

10. A long-distance synchronous acquisition system of multiple ordinary and dynamic visual sensors, characterized in that: The multi-channel ordinary and dynamic vision sensors include a plurality of ordinary vision sensors, a plurality of dynamic vision sensors and a control processor, and the device is applied to the control processor, and the device includes: a target sensor determination module, configured to determine a target common vision sensor from the plurality of common vision sensors, and to determine a target dynamic vision sensor from the plurality of dynamic vision sensors; A first delay compensation module, used for acquiring a first synchronization signal of the target common vision sensor, and sending the first synchronization signal to the remaining common vision sensors after performing a first delay compensation process on the first synchronization signal; A second delay compensation module, used for performing a second delay compensation process on the first synchronization signal and then sending the signal to each of the dynamic vision sensors; The third delay compensation module is used to receive the second synchronization signal emitted by the target dynamic vision sensor, and send the second synchronization signal to the remaining dynamic vision sensors after performing third delay compensation processing, so that multiple ordinary vision sensors and multiple dynamic vision sensors can collect data synchronously.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the long-distance synchronous acquisition method of multiple ordinary and dynamic visual sensors according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the long-distance synchronous acquisition method of multiple ordinary and dynamic vision sensors according to any one of claims 1 to 9 is implemented.

Images