Time stamp determination method and device of image frame, medium, data processor and system

Abstract

The invention discloses an image frame timestamp determination method and device, a medium, a data processor and a system. The method comprises the following steps: determining whether a current image frame is normally exposed or not according to a trigger timestamp and an exposure timestamp of the current image frame; if the exposure of the current image frame is normal, determining a synchronization timestamp for the current image frame according to a receiving timestamp and the exposure timestamp; and if the exposure of the current image frame is abnormal, determining a synchronization timestamp for the current image frame according to the receiving timestamp and the triggering timestamp. According to the technical scheme, a fault-tolerant mechanism is provided for determining the synchronization timestamp, and the accuracy of the synchronization timestamp can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a method, device, medium, data processor and system for determining a timestamp of an image frame. Background Art

[0002] Multi-sensor data fusion technology is widely used in the fields of autonomous driving and surveying and mapping. Multi-sensor fusion is based on the timestamp information carried by each frame of sensor data. The accuracy of the timestamp information directly affects the effect of multi-sensor data fusion.

[0003] In the related art, when the data processing module receives the sensor data, the timestamp information sent by the synchronization control module is mostly bound to the sensor data as the timestamp information of the sensor data.

[0004] However, due to objective factors such as electromagnetic environment, network status and system load, sensor data such as image frames and timestamp information sent by the synchronization control module may be lost, which makes the accuracy of timestamp information determined for sensor data using related technologies low. Summary of the invention

[0005] The present application provides a method, device, medium, data processor and system for determining the timestamp of an image frame, which can achieve the purpose of improving the accuracy of timestamp information.

[0006] According to a first aspect of the present application, a method for determining a timestamp of an image frame is provided, the method comprising:

[0007] Determine the image exposure time corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp;

[0008] Determining whether the currently received image frame is normally exposed according to a relative size relationship between the image exposure time and a preset exposure time;

[0009] If the currently received image frame is exposed normally, determining the image transmission duration corresponding to the currently received image frame according to the receiving timestamp and the exposure timestamp; determining a synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration;

[0010] If the exposure of the currently received image frame is abnormal, the image acquisition duration corresponding to the currently received image frame is determined according to the receiving timestamp and the trigger timestamp; and the synchronization timestamp is determined for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration.

[0011] According to a second aspect of the present application, a device for determining a timestamp of an image frame is provided, the device comprising:

[0012] An image exposure time determination module is used to determine the image exposure time corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp;

[0013] An exposure condition determination module, used to determine whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure time and the preset exposure time;

[0014] A first synchronization timestamp determination module is configured to determine, if the currently received image frame is exposed normally, an image transmission duration corresponding to the currently received image frame according to a reception timestamp and an exposure timestamp; and determine a synchronization timestamp for the currently received image frame according to a relative size relationship between the image transmission duration and a preset transmission duration;

[0015] The second synchronization timestamp determination module is used to determine the image acquisition duration corresponding to the currently received image frame according to the receiving timestamp and the trigger timestamp if the exposure of the currently received image frame is abnormal; and determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration.

[0016] According to a third aspect of the present invention, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the timestamp of an image frame as described in the embodiment of the present application.

[0017] According to a fourth aspect of the present invention, an embodiment of the present application provides a data processor, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for determining the timestamp of an image frame as described in the embodiment of the present application is implemented.

[0018] According to a fifth aspect of the present invention, an embodiment of the present application provides a timestamp determination system for an image frame, the timestamp determination system comprising:

[0019] A data processor, used to send a trigger pulse to start the image acquisition module, and record the time of the trigger pulse as a trigger timestamp; also used to receive an image frame output by the image acquisition module, and record the time of receiving the image frame as a receiving timestamp; also used to receive an exposure timestamp output by the image acquisition module;

[0020] The data processor is further used to determine the image exposure duration corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp; determine whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure duration and the preset exposure duration; if the currently received image frame is normally exposed, determine the image transmission duration corresponding to the currently received image frame according to the receiving timestamp and the exposure timestamp; determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration; if the currently received image frame is abnormally exposed, determine the image acquisition duration corresponding to the currently received image frame according to the receiving timestamp and the trigger timestamp; determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration;

[0021] The image acquisition module is used to receive the trigger pulse sent by the data processor, expose and acquire image frames according to the trigger pulse, and send the acquired image frames to the data processor; record the time when the exposure is completed as the exposure timestamp, and send the exposure timestamp to the data processor.

[0022] The technical solution of the embodiment of the present application determines whether the current image frame is normally exposed according to the trigger timestamp and exposure timestamp of the current image frame; based on the different exposure conditions of the current image frame, different determination methods are selected to determine the synchronization timestamp for the current image frame. The present application provides a fault-tolerant mechanism for determining the synchronization timestamp. Even in the case of packet loss of timestamp information or image frame information, the image frame timestamp determination method provided by the present application can also determine an accurate synchronization timestamp for the current image frame, effectively improving the accuracy of the synchronization timestamp, and effectively avoiding the situation where data fusion cannot be performed due to the inaccuracy of the synchronization timestamp, or the data fusion result is erroneous.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flowchart of a method for determining a timestamp of an image frame provided in accordance with the first embodiment;

[0026] Figure 2 is a flowchart of a method for determining a timestamp of an image frame provided in Embodiment 2;

[0027] Figure 3 is a flowchart of a method for determining a timestamp of an image frame provided in Embodiment 3;

[0028] Figure 4A It is a structural schematic diagram of a system for determining a timestamp of an image frame provided in an embodiment of the present application;

[0029] Figure 4B It is a structural schematic diagram of another image frame timestamp determination system provided in an embodiment of the present application.

[0030] Figure 5 is a structural diagram of a device for determining a timestamp of an image frame provided in Embodiment 4 of the present application;

[0031] Figure 6 It is a structural diagram of a data processor provided in Example 5 of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", "target" and "candidate" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1This is a flowchart of a method for determining a timestamp of an image frame provided in Example 1. This embodiment can be applied to a multi-sensor fusion scenario to determine a synchronization timestamp for a currently received image frame. The method can be executed by a timestamp determination device for an image frame. The timestamp determination device for an image frame can be implemented in the form of hardware and / or software and can be integrated into an electronic device running the system, such as a data processor.

[0036] In order to facilitate understanding of the image frame timestamp determination method provided in the embodiment of the present application, the image frame timestamp determination system used in the method is first introduced. Figure 4A is a structural diagram of a system for determining a timestamp of an image frame provided by an embodiment of the present application. Figure 4A The image frame timestamp determination system 400 includes: a data processor 410 and an image acquisition module 420. The data processor 410 and the image acquisition module 420 are in communication connection.

[0037] The data processor 410 is used to send a trigger pulse to start the image acquisition module 420, and record the time of the trigger pulse as a trigger timestamp; it is also used to receive the image frame output by the image acquisition module 420, and record the time of receiving the image frame as a receiving timestamp; it is also used to receive the exposure timestamp output by the image acquisition module 420;

[0038] The image acquisition module 420 is used to receive the trigger pulse sent by the data processor 410, expose and acquire image frames according to the trigger pulse, and send the acquired image frames to the data processor 410; record the time of completion of exposure as the exposure timestamp, and send the exposure timestamp to the data processor 410.

[0039] The data processor 410 is further configured to determine a synchronization timestamp for a currently received image frame according to the trigger timestamp, the exposure timestamp and the receiving timestamp.

[0040] Figure 4B Schematic diagram of another system for determining the timestamp of an image frame provided by an embodiment of the present application. Figure 4B As shown, the time stamp determination system of the image frame includes: a first data processor 411, a second data processor 412 and an image acquisition module 420. The first data processor 411 and the second data processor 412 jointly undertake Figure 4A The operations performed by the data processor 410 in the process of determining the synchronization timestamp for the currently received image frame can improve the efficiency of determining the synchronization timestamp.

[0041] Among them, the first data processor 411 is connected to the image acquisition module 420 through a trigger interface (not shown), and the first data processor 411 is connected to the second data processor 412 through a communication interface (not shown). The second data processor 412 is connected to the image acquisition module 420. Exemplarily, the image acquisition module 420 can be a camera module. The first data processor 411 and the second data processor 412 are respectively deployed with a first data processing chip and a second data processing chip. The chip types of the first data processing chip and the second processing chip can be the same or different, which is not limited here, and are determined according to actual conditions. Exemplarily, the first data processing chip and the second data processing chip can be MCUs (Microcontroller Units, microcontroller units) with customized functions. The first data processing chip defines a first data processing logic, and the second data processing chip defines a second data processing logic. The first data processing logic is different from the second data processing logic, and the first data processing logic defines the data processing flow of the first data processor 411. The second data processing logic defines the data processing flow of the second data processor 412.

[0042] The first data processor 411 is used to send a trigger pulse to start the image acquisition module, and record the time of the trigger pulse as a trigger timestamp, and is also used to send a trigger pulse signal to the image acquisition module 420 through a trigger interface; wherein the trigger pulse signal is used to instruct the image acquisition module 420 to perform exposure and feed back an exposure pulse signal;

[0043] The image acquisition module 420 is used to receive a trigger pulse issued by the first data processor 411, expose and acquire image frames according to the trigger pulse, and send the acquired image frames to the second data processor 412; the image acquisition module 420 is also used to record the time of completion of exposure as an exposure timestamp, and send the exposure timestamp to the first data processor 411.

[0044] The first data processor 411 sends the trigger timestamp and the exposure timestamp to the second data processor 412 through the communication interface; the second data processor 412 determines a synchronization timestamp for the currently received image frame according to the trigger timestamp, the exposure timestamp and the receiving timestamp.

[0045] Of course, it is understandable that the image acquisition module 420 may directly send the exposure timestamp to the second data processor 412 instead of via the first data processor 411. The specific transmission path used to transmit the exposure timestamp to the second data processor 412 can be determined according to actual business needs and is not limited here.

[0046] In a specific embodiment, the first data processor 411 triggers a pulse signal to the image acquisition module 420 according to a preset image frame rate. The pulse output interface of the first data processor 411 is connected to the external trigger interface of the image acquisition module 420. The image acquisition module 420 receives the trigger pulse signal, starts the image frame exposure, and outputs the exposure pulse signal to the external trigger interface. The external trigger interface is connected to the pulse input interface of the first data processor 411 for identification by the first data processor 411. The image acquisition module 420 records the time of completion of the exposure as the exposure timestamp and sends the exposure timestamp to the first data processor 411. The image acquisition module 420 supports pulse trigger input and exposure pulse signal output. The data output interface of the image acquisition module 420 is connected to the data input interface of the second data processor 412. The second data processor 412 reads the image frame data through the data input interface and records the time of receiving the image frame as the receiving timestamp.

[0047] The second data processor 412 is directly connected to the first data processor 411 through a communication interface, and receives the trigger timestamp and exposure timestamp sent by the first data processor 411. The second data processor 412 determines a synchronization timestamp for the currently received image frame according to the trigger timestamp, exposure timestamp and receiving timestamp.

[0048] Optional, Figure 4A and Figure 4B The operation "determining a synchronization timestamp for a currently received image frame according to a trigger timestamp, an exposure timestamp and a receiving timestamp" performed by the data processor 410 and the second data processor 412 in the timestamp determination system of the image frame shown includes: determining an image exposure duration corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp; determining whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure duration and a preset exposure duration; if the currently received image frame is normally exposed, determining an image transmission duration corresponding to the currently received image frame according to the receiving timestamp and the exposure timestamp; determining a synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration; if the currently received image frame is abnormally exposed, determining an image acquisition duration corresponding to the currently received image frame according to the receiving timestamp and the trigger timestamp; and determining a synchronization timestamp for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration.

[0049] Next, a method for determining the timestamp of an image frame provided by an embodiment of the present application is introduced. Figure 1 is a flowchart of a method for determining a timestamp of an image frame according to the first embodiment. Figure 1 As shown, the method includes:

[0050] S110, determining an image exposure time length corresponding to a currently received image frame according to a currently received trigger timestamp and an exposure timestamp;

[0051] S120, determining whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure time and the preset exposure time;

[0052] S130, if the currently received image frame is exposed normally, determining the image transmission duration corresponding to the currently received image frame according to the receiving timestamp and the exposure timestamp; determining a synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration;

[0053] S140. If the exposure of the currently received image frame is abnormal, determine the image acquisition duration corresponding to the currently received image frame according to the receiving timestamp and the triggering timestamp; determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration.

[0054] The currently received image frame refers to an image frame received by the data processor in the current time window. For ease of description, the currently received image frame is referred to as the current image frame below.

[0055] The current image frame is acquired by the image acquisition module and sent to the data processor by the image acquisition module. The data processor has not yet updated the current image frame to the image frame sequence, that is, the synchronization timestamp of the current image frame has not been determined. Among them, the synchronization timestamp is the actual acquisition time of the current image frame, and the synchronization timestamp is an important basis for the subsequent data fusion of the current image frame and its sensor data frame.

[0056] The trigger timestamp refers to the time when the trigger pulse is sent, wherein the trigger pulse is used to start the exposure of the image acquisition module and perform image acquisition. The exposure timestamp is generated by the image acquisition module, and the exposure timestamp is the time when the image acquisition module completes the exposure. In addition to the exposure timestamp and the trigger timestamp, the data processor also includes the receiving timestamp of the current image frame. Among them, the receiving timestamp refers to the time when the data processor receives the current image frame. The receiving timestamp is generated by the data processor. When the data processor receives an image frame, the data processor will generate a receiving timestamp for the image frame, that is, the receiving timestamp in the data processor actually corresponds to the current image frame.

[0057] Due to objective factors such as electromagnetic environment, network status and system load, image frames and exposure timestamps may be lost, which may cause the trigger timestamp and exposure timestamp currently received in the data processor to not actually correspond to the current image frame.

[0058] In order to ensure the accuracy of the synchronization timestamp of the current image frame, it is necessary to determine whether the trigger timestamp and exposure timestamp currently received in the data processor actually correspond to the current image frame, that is, to determine the transmission conditions of the image frame and the exposure timestamp, such as determining whether the image frame and the exposure timestamp are lost. According to different transmission conditions of the image frame and the exposure timestamp, a synchronization timestamp is determined for the current image frame.

[0059] The transmission status of the image frame, the trigger timestamp and the exposure timestamp can be determined by the data processor according to the trigger timestamp, the exposure timestamp and the receiving timestamp. First, the data processor determines the image exposure duration corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp to determine whether the current image frame is normally exposed.

[0060] Among them, the time when the image acquisition module is started for exposure can be determined according to the trigger timestamp, and the time when the image acquisition module completes the exposure can be determined according to the exposure timestamp; from starting the exposure to completing the exposure, the image acquisition module needs to prepare for the exposure, for example, the image acquisition module needs to wake up and start related components. Among them, the image exposure duration is used to quantify the length of time from the start of the exposure to the completion of the exposure of the image acquisition module.

[0061] The image exposure duration is determined based on the wake-up time of the image module, the time to input a pulse, and the time for the image acquisition module to scan a row of pixels. When the image acquisition module is determined, the image exposure duration of the image acquisition module is also determined, and the exposure duration required for normal exposure of the image frame is determined as the reference exposure duration. Based on the reference exposure duration of the image acquisition module, it can be determined whether the current image frame is exposed normally. Abnormal transmission of the exposure timestamp and image frame, such as packet loss, will cause the image exposure duration to exceed the reference exposure duration.

[0062] According to the trigger timestamp and exposure timestamp of the current image frame, the image exposure duration corresponding to the current image frame is determined. Optionally, the relative time interval between the exposure timestamp and the trigger timestamp is calculated, and the relative time interval is determined as the image exposure duration of the current image frame.

[0063] The preset exposure time is related to the exposure wake-up parameter of the image acquisition module. The preset exposure time can be determined according to the reference exposure time of the image acquisition module and is not limited here. The preset exposure time is generally in milliseconds. The preset exposure time is used to measure whether the current image frame is normally exposed.

[0064] If the exposure time of the image frame is less than or equal to the preset exposure time, indicating that the exposure of the current image frame is normal, the image transmission time corresponding to the currently received image frame is determined according to the receiving timestamp and the exposure timestamp; according to the relative size relationship between the image transmission time and the preset transmission time, it is further determined whether the exposure timestamp actually corresponds to the current image frame, and based on this, a synchronization timestamp is determined for the currently received image frame.

[0065] Among them, the image transmission duration refers to the length of time between the image acquisition module completing the exposure and the data processing module receiving the current image frame. It can be known that when the network status is stable, the image transmission duration of the current image frame is also determined, and the transmission duration required for the normal transmission of the image frame is determined as the reference transmission duration. Among them, the preset transmission duration can be determined based on the reference transmission duration, which is not limited here. The preset transmission duration is generally in the millisecond level. The preset transmission duration is used to determine whether the exposure timestamp actually corresponds to the current image frame. It can then be determined whether the exposure timestamp can be directly used to determine the synchronization timestamp for the current image frame.

[0066] Conversely, if the image exposure time is greater than the preset exposure time, the current image frame is exposed abnormally, and the exposure timestamp of the current image frame cannot be used to determine the synchronization timestamp. It is necessary to determine the image acquisition time corresponding to the current image frame based on the receiving timestamp and triggering timestamp of the current image frame, and determine the synchronization timestamp for the currently received image frame based on the relative size relationship between the image acquisition time and the preset acquisition time.

[0067] The image acquisition duration is used to measure the length of time from when the image acquisition module starts exposure to when the data processor receives the current image frame. Optionally, the relative time interval between the receiving timestamp and the triggering timestamp is determined as the image acquisition duration. The image acquisition duration includes at least: image exposure duration and image transmission duration. The preset acquisition duration can be determined based on the reference exposure duration and the reference transmission duration.

[0068] Based on the relative size relationship between the image acquisition duration and the preset acquisition duration, it can be further determined whether the trigger timestamp actually corresponds to the current image frame, and based on this, a synchronization timestamp is determined for the current image frame.

[0069] It is worth noting that step 130 and step 140 are in parallel relationship and do not have a logical order relationship. In actual application, one of them is executed according to the exposure condition of the current image frame.

[0070] The technical solution of the embodiment of the present application determines whether the current image frame is normally exposed according to the trigger timestamp and exposure timestamp of the current image frame; based on the different exposure conditions of the current image frame, different determination methods are selected to determine the synchronization timestamp for the current image frame. The present application provides a fault-tolerant mechanism for determining the synchronization timestamp. Even in the case of packet loss of timestamp information or image frame information, the image frame timestamp determination method provided by the present application can also determine an accurate synchronization timestamp for the current image frame, effectively improving the accuracy of the synchronization timestamp, and effectively avoiding the situation where data fusion cannot be performed due to the inaccuracy of the synchronization timestamp, or the data fusion result is erroneous.

[0071] In an optional embodiment, after determining the synchronization timestamp for the currently received image frame, the timestamp determination method also includes: determining the offset address of the currently received image frame based on the starting address of the currently received image frame and the data size of the currently received image frame; and writing the synchronization timestamp into the currently received image frame based on the offset address.

[0072] The starting address refers to the starting address of the current image frame in the image frame sequence. The data size of the current image frame refers to the size of the storage space required for the current image frame. According to the starting address of the current image frame and the data size of the current image frame, the offset address of the current image frame is determined. Optionally, the offset address of the current image frame can be obtained by adding the data size of the current image frame to the starting address of the current image frame. For example, the offset address of the current image frame can be determined according to x+y=offset. Among them, x represents the starting address of the current image frame, y represents the data size of the current image frame, and offset represents the offset address of the current image frame.

[0073] Based on the offset address, the synchronization timestamp is written into the current image frame. Optionally, the synchronization timestamp is written at the offset address of the current image frame.

[0074] The above technical solution determines the offset address of the current image frame according to the starting address and data size of the current image frame, and writes the synchronization timestamp into the current image frame based on the offset address. When the current data frame is determined, the data size of the current data frame is determined, and based on the starting address and data size of the current image frame, the writing position of the synchronization timestamp in the current image frame can be determined, so that the synchronization timestamp can be quickly read from the current image frame later.

[0075] Embodiment 2

[0076] Figure 2This is a flow chart of a method for determining the timestamp of an image frame provided in Example 2. This embodiment is further optimized on the basis of the above-mentioned embodiment. Specifically, when the current image frame is exposed normally, the operation of "determining a synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration" is refined.

[0077] like Figure 2 As shown, the method includes:

[0078] S210: Determine an image exposure duration corresponding to a currently received image frame according to a currently received trigger timestamp and an exposure timestamp.

[0079] S220: Determine whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure time and the preset exposure time.

[0080] S230: If the exposure of the currently received image frame is normal, determine the image transmission duration corresponding to the currently received image frame according to the receiving timestamp and the exposure timestamp.

[0081] S240: If the exposure timestamp is earlier than the receiving timestamp, and the image transmission duration is less than or equal to the preset transmission duration, determine the currently received exposure timestamp as the synchronization timestamp of the currently received image frame.

[0082] It can be known that the current image frame received by the data processor is acquired by the image acquisition module through exposure. In theory, the exposure timestamp should be earlier than the receiving timestamp. If the exposure timestamp is earlier than the receiving timestamp, it cannot be directly determined that the exposure timestamp actually corresponds to the current image frame. It is also necessary to further determine the relative size relationship between the image transmission time and the preset transmission time.

[0083] When the exposure timestamp is earlier than the receiving timestamp and the image transmission duration is less than or equal to the preset transmission duration, it can be determined that the exposure timestamp actually corresponds to the current image frame, and the exposure timestamp can be used to determine the synchronization timestamp for the current image frame. Specifically, the exposure timestamp can be used to determine the synchronization timestamp of the current image frame.

[0084] S250: If the exposure timestamp is earlier than the receiving timestamp, and the image transmission time length is greater than the preset transmission time length, determine the synchronization timestamp of the currently received image frame according to the receiving timestamp.

[0085] In the case where the exposure timestamp is earlier than the receiving timestamp, if the image transmission duration is greater than the preset transmission duration, it indicates that the exposure timestamp does not actually correspond to the current image frame, and the exposure timestamp cannot be used to determine the synchronization timestamp for the current image frame. In the case where normal exposure is determined based on the trigger timestamp and the exposure timestamp, the trigger timestamp cannot be used to determine the synchronization timestamp for the current image frame. At this time, it is necessary to determine the synchronization timestamp for the current image frame based on the receiving timestamp. This is because the receiving timestamp is determined by the data processor based on the time when the current image frame is received, and the receiving timestamp does not require data transmission, there is no packet loss problem, and the receiving timestamp actually corresponds to the current image frame. In the case where the exposure timestamp is unavailable, the synchronization timestamp can also be determined based on the receiving timestamp.

[0086] In an optional embodiment, determining the synchronization timestamp of the currently received image frame according to the receiving timestamp includes: defining the synchronization timestamp of the currently received image frame as t, defining the receiving timestamp as t3, and t is calculated by the following formula:

[0087] t=t3-t λ

[0088] In the formula, t λ It is the time from the end of exposure to the completion of image output of the image acquisition module.

[0089] It can be known that when data transmission is normal, that is, there is no packet loss, the exposure timestamp will be used as the synchronization timestamp of the current image frame. The relative time interval determined by the exposure timestamp and the receiving timestamp is the image transmission duration. Based on this, the exposure timestamp of the current image frame can be inferred from the receiving timestamp. Optionally, the synchronization timestamp can be determined by subtracting the preset transmission duration from the receiving timestamp of the current image frame. Optionally, t is determined based on the preset transmission duration. λ .

[0090] S260: If the exposure timestamp is later than the receiving timestamp, determine the currently received image frame as the last received image frame, and re-determine the synchronization timestamp of the last received image frame according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

[0091] As mentioned above, the exposure timestamp actually corresponding to the current image frame should be earlier than the receiving timestamp in theory. If the exposure timestamp is later than the receiving timestamp, the currently received image frame is determined as the last received image frame. Based on the last received trigger timestamp, exposure timestamp and receiving timestamp, the synchronization timestamp of the last received image frame is re-determined.

[0092] It is worth noting that steps 240 to 260 all correspond to the situation where the current image frame is normally exposed. Step 240 corresponds to the situation where neither the timestamp information nor the image frame is lost; step 250 corresponds to the situation where both the exposure timestamp and the trigger timestamp actually corresponding to the current image frame are lost; and step 260 corresponds to the situation where the image frame is lost. That is to say, steps 240 to 260 are in a parallel relationship and do not have a logical relationship of sequence. For the convenience of description, steps 240 and 260 are shown in sequence. In actual application, one is executed according to the data loss situation.

[0093] The technical solution of the present application determines the synchronization timestamp for the currently received image frame based on the circumstance relationship between the exposure timestamp and the receiving timestamp, as well as the image transmission duration, on the premise that the currently received image frame is determined to be normally exposed according to the exposure timestamp and the trigger timestamp. The accuracy of the synchronization timestamp is guaranteed, and the fault tolerance of the synchronization timestamp determination method is improved. In the case where the exposure timestamp does not actually correspond to the currently received image frame, the synchronization timestamp can still be determined for the currently received image frame.

[0094] Embodiment 3

[0095] Figure 3 This is a flow chart of a method for determining the timestamp of an image frame provided in Example 3. This embodiment is further optimized on the basis of the above-mentioned embodiment. Specifically, when it is determined that the currently received image frame belongs to an exposure abnormality according to the exposure timestamp and the trigger timestamp, the operation of "determining a synchronization timestamp for the currently received image frame according to the relative size relationship between the image acquisition time and the preset acquisition time" is refined.

[0096] like Figure 3 As shown, the method includes:

[0097] S310: Determine an image exposure duration corresponding to a currently received image frame according to a currently received trigger timestamp and an exposure timestamp.

[0098] S320: Determine whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure time and the preset exposure time.

[0099] S330: If the exposure of the currently received image frame is abnormal, determine the image acquisition duration corresponding to the currently received image frame according to the receiving timestamp and the triggering timestamp.

[0100] The trigger timestamp and exposure time are key factors that affect whether the image frame exposure is normal. In the case of abnormal image frame exposure, it is necessary to further clarify which of the exposure timestamp and the trigger timestamp is lost.

[0101] S340: If the trigger timestamp is earlier than the receiving timestamp and the image acquisition duration is less than the preset acquisition duration, determine the synchronization timestamp of the currently received image frame according to the currently received trigger timestamp, the receiving timestamp and the exposure timestamp.

[0102] It is understandable that, when the trigger timestamp actually corresponds to the receiving timestamp, the trigger timestamp theoretically needs to be earlier than the receiving timestamp, and it is also necessary to ensure that the image acquisition duration determined by the trigger timestamp and the receiving timestamp is less than the preset acquisition duration. In other words, only when the trigger timestamp is earlier than the receiving timestamp and the image acquisition duration is less than the preset acquisition duration, the trigger timestamp can actually correspond to the current image frame. As for whether the trigger timestamp actually corresponds to the current image frame, it is necessary to further determine it in combination with the relative size relationship between the currently received trigger timestamp, exposure timestamp and receiving timestamp.

[0103] In an optional embodiment, determining the synchronization timestamp of the currently received image frame according to the currently received trigger timestamp, the receiving timestamp and the exposure timestamp includes:

[0104] If the exposure timestamp is earlier than the trigger timestamp, the synchronization timestamp of the currently received image frame is defined as t, and the trigger timestamp of the currently received image frame is defined as t1. t is calculated by the following formula:

[0105] t=t1+(t ζ *t ε +t δ )

[0106] Where, t ζ is the wake-up time of the image acquisition module; t ε The time for inputting a pulse to the image acquisition module; t δ The time it takes for the image acquisition module to scan a row of pixels;

[0107] Otherwise, the currently received image frame is determined as the last received image frame, and the synchronization timestamp of the last received image frame is re-determined according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

[0108] It can be known that, when both the exposure timestamp and the trigger timestamp actually correspond to the currently received image frame, the exposure timestamp should be later than the trigger timestamp. If the exposure timestamp is earlier than the trigger timestamp, it can be determined that the exposure abnormality is caused by the loss of the exposure timestamp. The currently received trigger timestamp actually corresponds to the currently received image frame. The currently received trigger timestamp can be used to determine the synchronization timestamp of the currently received image frame. Optionally, the synchronization timestamp is determined by superimposing a preset exposure time on the basis of the trigger timestamp. Optionally, t ζ *tε +t δ Indicates the preset exposure time. ζ The exposure wake-up parameter is related to the image acquisition module, and the exposure wake-up parameter is a positive integer. The exposure wake-up parameter is related to the image acquisition module, and the exposure wake-up parameters of different image acquisition modules may be different, which is not limited here. For example, the exposure wake-up parameter can be 61396. Where, t ε The time for inputting a pulse to the image acquisition module. δ is the time required for the image acquisition module to scan a row of pixels, t δ It can be determined by the following formula, t δ =row×tPCLK, where row is the number of pixels in a row of the current image frame, and tPCLK is the time required for the image acquisition module to scan one pixel. For ease of expression, the above situation is determined as situation 341.

[0109] Under the branch where step S340 is located, there is also a situation 342 corresponding to situation 341. Situation 342 is:

[0110] Otherwise, if the image exposure is abnormal, the image acquisition time is less than the preset acquisition time, and the exposure timestamp is later than the trigger timestamp, it may be caused by the loss of both the trigger timestamp and the image frame. In this case, the currently received image frame is determined as the last received image frame, and the synchronization timestamp of the last received image frame is re-determined based on the last received trigger timestamp, exposure timestamp and receiving timestamp.

[0111] S350: If the trigger timestamp is earlier than the receiving timestamp, and the image acquisition time length is greater than or equal to the preset acquisition time length, determine the synchronization timestamp of the currently received image frame according to the receiving timestamp and the exposure timestamp.

[0112] In the case where the trigger timestamp is earlier than the receiving timestamp, if the image acquisition duration is greater than or equal to the preset acquisition duration, it indicates that the trigger timestamp does not actually correspond to the receiving timestamp, and the trigger timestamp cannot be used to determine the synchronization timestamp for the current image frame. In this case, the synchronization timestamp can be determined for the currently received image frame based on the receiving timestamp and the exposure timestamp. Specifically, it can be determined whether the exposure timestamp actually corresponds to the currently received image frame based on the relative relationship between the receiving timestamp and the exposure timestamp.

[0113] In an optional embodiment, the synchronization timestamp of the currently received image frame is determined according to the receiving timestamp and the exposure timestamp, including: if the exposure timestamp is earlier than the receiving timestamp, and the image transmission duration is less than or equal to the preset transmission duration, the currently received exposure timestamp is determined as the synchronization timestamp of the currently received image frame; otherwise, the synchronization timestamp of the currently received image frame is defined as t, and the receiving timestamp is defined as t3, where t is calculated by the following formula:

[0114] t=t3-t λ

[0115] In the formula, t λ It is the time from the end of exposure to the completion of image output of the image acquisition module.

[0116] It can be known that the exposure timestamp may actually correspond to the current image frame only when the exposure timestamp is earlier than the receiving timestamp. As for whether the exposure timestamp actually corresponds to the receiving timestamp, it is also necessary to judge in combination with the image transmission time. Specifically, when the exposure timestamp is earlier than the receiving timestamp and the image transmission time is less than or equal to the preset transmission time, it can be determined that the currently received exposure timestamp actually corresponds to the current image frame, indicating that the currently received exposure timestamp can be used to determine the synchronization timestamp for the current image frame. Specifically, the currently received exposure timestamp is determined as the synchronization timestamp of the currently received image frame. For ease of expression, the above situation is determined as situation 351.

[0117] Under the branch where step S350 is located, there is also a situation 352 corresponding to situation 351. Situation 352 is: Otherwise, it indicates that the currently received exposure timestamp does not actually correspond to the current image frame, that is, the currently received exposure timestamp and trigger timestamp cannot be used to determine the synchronization timestamp for the current image frame. It can be known that the received timestamp actually corresponds to the current image frame. In this case, the synchronization timestamp can be determined for the current image frame based on the received timestamp. Based on this, the received timestamp can be used to infer the exposure timestamp of the current image frame. Optionally, the synchronization timestamp is determined by subtracting the preset transmission duration from the received timestamp of the current image frame. Optionally, t is determined according to the preset transmission duration. λ Among them, t λ It is the time from the end of exposure to the completion of image output of the image acquisition module.

[0118] S360: If the trigger timestamp is later than the receiving timestamp, determine the currently received image frame as the last received image frame, and re-determine the synchronization timestamp of the last received image frame according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

[0119] The trigger timestamp is later than the receiving timestamp, indicating that the trigger timestamp does not actually correspond to the current image frame. The currently received image frame is determined as the last received image frame, and the synchronization timestamp of the last received image frame is re-determined based on the trigger timestamp and receiving timestamp of the last reception.

[0120] It is worth noting that step 340 to step 360 are in parallel relationship and do not have a logical order. In actual application, one of them is executed according to the data loss situation.

[0121] The technical solution of the present application, on the premise that the currently received image frame is determined to be abnormally exposed according to the exposure timestamp and the trigger timestamp, determines the synchronization timestamp for the currently received image frame according to the causal relationship between the trigger timestamp and the receiving timestamp, combined with the image acquisition duration, and proposes a synchronization timestamp determination method suitable for the current image frame exposure abnormality situation, provides a fault-tolerant mechanism for the determination of the synchronization timestamp, ensures the accuracy of the synchronization timestamp, and improves the robustness of the synchronization timestamp determination method.

[0122] It can be known that, due to objective factors such as electromagnetic environment, network status and system load, image frames, trigger timestamps and exposure timestamps may be lost. Therefore, the exposure timestamp and trigger timestamp received by the data processor in the current time window may not actually correspond to the image frame received in the current time window. By using the timestamp information that actually corresponds to the currently received image frame to determine the synchronization timestamp for the currently received image frame, the accuracy of the synchronization timestamp can be guaranteed. Embodiments 2 and 3 describe methods for determining synchronization timestamps for image frames under different data loss situations. For ease of understanding, the following will take different loss situations of image frames, trigger timestamps and exposure timestamps as examples to further illustrate the method for determining the synchronization timestamp of the image frame described in Embodiments 2 and 3.

[0123] Assume that the image acquisition module generates two image frames in sequence (set as image 1 and image 2), so the data processor will have 6 timestamps, namely the trigger timestamp corresponding to image 1 (set as 1t1), the exposure timestamp corresponding to image 1 (set as 1t2), the receiving timestamp corresponding to image 1 (set as 1t3), the trigger timestamp corresponding to image 2 (set as 2t1), the exposure timestamp corresponding to image 2 (set as 2t2), and the receiving timestamp corresponding to image 2 (set as 2t3). Assume that the preset exposure time, the preset transmission time and the preset acquisition time are all set according to the normal working conditions of the corresponding system, wherein the preset exposure time, the preset transmission time and the preset acquisition time are all greater than 0. In the case that the timestamp information and the image frame are not lost, the data received by the data processor twice are (1t1, 1t2, 1t3, image 1) and (2t1, 2t2, 2t3, image 2). In addition, the above 6 timestamps have the following constraints on the time sequence:

[0124] 1t1<1t2<1t3<2t1<2t2<2t3

[0125] Based on the packet loss of timestamps and / or image frames, the data processor may experience the following reception conditions:

[0126] Situation 1: No data packet loss occurs, and the latest data received by the data processor is: 2t1, 2t2, 2t3, image 2;

[0127] Situation 2: Image 2 is lost, and the latest data received by the data processor is: 2t1, 2t2, 1t3, image 1;

[0128] Situation 3: Packet 2t1 is lost, and the latest data received by the data processor is: 1t1, 2t2, 2t3, image 2;

[0129] Situation 4: Packet 2t2 is lost, and the latest data received by the data processor is: 2t1, 1t2, 2t3, image 2;

[0130] Case 5: 2t1 and image 2 packets are lost, and the latest data received by the data processor is: 1t1, 2t2, 1t3, image 1;

[0131] Situation 6: 2t2 and image 2 packets are lost, and the latest data received by the data processor is: 2t1, 1t2, 1t3, image 1;

[0132] Situation 7: Packets 2t1 and 2t2 are lost, and the latest data received by the data processor is: 1t1, 1t2, 2t3, image 2.

[0133] The following is a specific analysis of various situations in combination with Example 2 and Example 3:

[0134] Case 1: Since no data packet loss occurs, 2t1, 2t2 and 2t3 all correspond to image 2, that is, 0<2t2-2t1≤preset exposure time, 0<2t3-2t2≤preset transmission time. Among them, 2t2-2t1 represents the image exposure time, 2t2-2t1≤preset exposure time, which means that the image exposure time is less than or equal to the preset exposure time, and the currently received image frame is exposed normally; 2t3-2t2 represents the image transmission time, 2t3-2t2≤preset transmission time, which means that the image transmission time is less than or equal to the preset transmission time, 0<2t2-2t1 and 0<2t3-2t2 respectively indicate that the currently received trigger timestamp is earlier than the exposure timestamp, and the exposure timestamp is earlier than the receiving timestamp. In this case, the exposure timestamp (2t2) can be determined as the synchronization timestamp of image 2. Corresponding to the branch where step S240 is located in the second embodiment.

[0135] Situation 2: Due to packet loss of image 2, the data processor has received the latest data (2t1, 2t2, 1t3, image 1), in which the timestamp corresponding to image 1 is only 1t3, and the synchronization timestamp of image 1 cannot be determined through 2t1 and 2t2. Therefore, the synchronization timestamp of image 1 can be re-determined based on the trigger timestamp (1t1), exposure timestamp (1t2) and reception timestamp (1t3) received last time. In the second and third embodiments, it is reflected as 0<2t2-2t1≤preset exposure time, 2t2-2t1 represents the image exposure time, 2t2-2t1≤preset exposure time, which means that the image exposure time is less than or equal to the preset exposure time, and the currently received image frame is exposed normally; 1t3-2t2<0, wherein 1t3-2t2<0 indicates that the currently received reception timestamp 1t3 is earlier than the currently received exposure timestamp 2t2. This corresponds to the branch where step S260 in the second embodiment is located.

[0136] Case 3: Due to packet loss at 2t1, the data processor receives the latest data (1t1, 2t2, 2t3, image 2), and the timestamps corresponding to image 2 are 2t2 and 2t3. Therefore, the exposure timestamp (2t2) can be determined as the synchronization timestamp of image 2. In the second and third embodiments, it is reflected as 2t2-1t1>preset exposure time, 2t3-1t1≥preset acquisition time, 0<2t3-2t2≤preset transmission time, wherein 2t2-1t1 represents the image exposure time, 2t2-1t1>preset exposure time, indicating that the image acquisition time is greater than the preset exposure time, and the currently received image frame is abnormally exposed; 2t3-1t1 represents the image acquisition time, 2t3-1t1≥preset acquisition time, indicating that the image acquisition time is greater than or equal to the preset acquisition time; 0<2t3-2t2 indicates that the exposure timestamp is earlier than the receiving timestamp; 2t3-2t2 represents the image transmission time, 2t3-2t2≤preset transmission time, indicating that the image transmission time is less than or equal to the preset transmission time. Corresponds to the branch where step S350 is located in the third embodiment. Specifically corresponds to the situation 351 under the branch of step S350. Specifically, situation 351 is: if the exposure timestamp is earlier than the receiving timestamp, and the image transmission duration is less than or equal to the preset transmission duration, it can be determined that the exposure timestamp actually corresponds to the currently received image frame, and the exposure anomaly is caused by the loss of the trigger timestamp, and the exposure timestamp is used as the synchronization timestamp of the currently received image frame;

[0137] Case 4: Due to packet loss at 2t2, the data processor has recently received data (2t1, 1t2, 2t3, image 2), and the timestamps corresponding to image 2 are 2t1 and 2t3. Therefore, the synchronization timestamp of image 2 can be determined based on the trigger timestamp (2t1). In Embodiments 2 and 3, |1t2-2t1|>preset exposure duration, 1t2-2t1<0, 0<2t3-2t1<preset acquisition duration, 1t2<2t3. It is worth noting that |1t2-2t1| represents the image exposure duration, |1t2-2t1|>preset exposure duration, indicating that the image exposure duration is greater than the preset exposure duration, and the current image frame is abnormally exposed. 1t2-2t1<0 means that the exposure timestamp 1t2 is earlier than the trigger timestamp 2t1. 2t3-2t1 represents the image acquisition duration, 2t3-2t1<preset acquisition duration, and the image acquisition duration is less than the preset acquisition duration; 0<2t3-2t1 means that the currently received trigger timestamp 2t1 is earlier than the received timestamp 2t3, corresponding to the branch where step S340 in embodiment three is located. Specifically, it corresponds to situation 341 under the branch of step S340. Specifically, situation 341 is: if the exposure timestamp is earlier than the trigger timestamp, it can be determined that the exposure abnormality is caused by the loss of the exposure timestamp. In this case, the currently received trigger timestamp actually corresponds to the currently received image frame. The synchronization timestamp of the currently received image frame can be determined based on the currently received trigger timestamp.

[0138] Case 5: Due to packet loss of 2t1 and image 2, the latest data received by the data processor is (1t1, 2t2, 1t3, image 1), and the timestamps corresponding to image 1 are 1t1 and 1t3. It is impossible to determine the synchronization timestamp of image 1 through 2t2. Therefore, the synchronization timestamp of image 1 can be re-determined based on the last received trigger timestamp (1t1), exposure timestamp (1t2) and reception timestamp (1t3). In the second and third embodiments, it is reflected as 2t2-1t1>preset exposure time, 0<1t3-1t1<preset acquisition time, 1t1<2t2, wherein 2t2-1t1 represents the image exposure time, 2t2-1t1>preset exposure time, indicating that the image exposure time is greater than the preset exposure time, and the exposure of the currently received image frame is abnormal; 1t3-1t1 represents the image acquisition time, 1t3-1t1<preset acquisition time, indicating that the image acquisition time is less than the preset acquisition time, 1t1<2t2, that is, the currently received trigger timestamp 1t1 is earlier than the currently received exposure timestamp 2t2, corresponding to the branch where step S340 is located in the third embodiment, and specifically can correspond to situation 342 of the branch where step S340 is located.

[0139] Situation 6: Due to packet loss of 2t2 and image 2, the latest data received by the data processor is (2t1, 1t2, 1t3, image 1), and the timestamps corresponding to image 1 are 1t2 and 1t3. Therefore, the synchronization timestamp of image 1 can be re-determined based on the last received trigger timestamp (1t1), exposure timestamp (1t2) and reception timestamp (1t3). In embodiments 2 and 3, it is reflected as 1t2-2t1<0, and 1t3-2t1<0, wherein 1t2-2t1<0 indicates that the currently received exposure timestamp is earlier than the currently received trigger timestamp, and 1t3-2t1<0 indicates that the reception timestamp is earlier than the currently received trigger timestamp, corresponding to the branch where step S360 in embodiment 3 is located.

[0140] Situation 7: Due to packet loss at 2t1 and 2t2, the latest data received by the data processor is (1t1, 1t2, 2t3, image 2). The timestamp corresponding to image 2 is only 2t3, and the synchronization timestamp of image 2 cannot be determined through 1t1 and 1t2. Therefore, the synchronization timestamp of image 2 can be determined according to the received timestamp (2t3). In the second and third embodiments, it is reflected as 0<1t2-1t1≤preset exposure duration, 2t3-1t2>preset transmission duration, wherein 1t2-1t1 represents the image exposure duration, 1t2-1t1≤preset exposure duration, indicating that the image exposure duration is less than or equal to the preset exposure duration; 0<1t2-1t1 indicates that the exposure timestamp is later than the trigger timestamp; 2t3-1t2 represents the image transmission duration, 2t3-1t2>preset transmission duration, indicating that the exposure timestamp is greater than the preset transmission duration, and at the same time indicating that the exposure timestamp is earlier than the receiving timestamp. This corresponds to the branch where step S250 in the second embodiment is located.

[0141] It should be noted that the above scenarios are merely hypothetical scenarios used to illustrate how the present application determines the synchronization timestamp of an image frame in the event of packet loss of various timestamps and / or image frames. The above scenarios are not intended to limit the scope of protection of the present application, and the decision-making method in the present application is not limited to the above hypothetical scenarios. As long as the different judgment conditions in the present application are met, the synchronization timestamp corresponding to the image frame can be determined through the present application.

[0142] Embodiment 4

[0143] Figure 5 1 is a schematic diagram of the structure of the device for determining the timestamp of an image frame provided in the fourth embodiment of the present application. This embodiment can be applied to the situation of determining the synchronization timestamp for the current image frame in a multi-sensor fusion scenario. The device can be implemented by software and / or hardware and can be integrated into a data processor such as an intelligent terminal.

[0144] like Figure 5As shown, the device may include: an image exposure duration determination module 510 , an exposure condition determination module 520 , a first synchronization timestamp determination module 530 , and a second synchronization timestamp determination module 540 .

[0145] An image exposure duration determination module 510 is used to determine the image exposure duration corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp;

[0146] An exposure condition determination module 520, configured to determine whether a currently received image frame is normally exposed based on a relative size relationship between the image exposure time and a preset exposure time;

[0147] The first synchronization timestamp determination module 530 is used to determine the image transmission duration corresponding to the currently received image frame according to the reception timestamp and the exposure timestamp if the currently received image frame is exposed normally; and determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration;

[0148] The second synchronization timestamp determination module 540 is used to determine the image acquisition duration corresponding to the currently received image frame based on the receiving timestamp and the triggering timestamp if the currently received image frame has an exposure abnormality; and determine the synchronization timestamp for the currently received image frame based on the relative size relationship between the image acquisition duration and the preset acquisition duration.

[0149] The technical solution of the embodiment of the present application determines whether the current image frame is normally exposed according to the trigger timestamp and exposure timestamp of the current image frame; based on the different exposure conditions of the current image frame, different determination methods are selected to determine the synchronization timestamp for the current image frame. The present application provides a fault-tolerant mechanism for determining the synchronization timestamp. Even in the case of packet loss of timestamp information or image frame information, the image frame timestamp determination method provided by the present application can also determine an accurate synchronization timestamp for the current image frame, effectively improving the accuracy of the synchronization timestamp, and effectively avoiding the situation where data fusion cannot be performed due to the inaccuracy of the synchronization timestamp, or the data fusion result is erroneous.

[0150] Optionally, the first synchronization timestamp determination module 530 includes: a first synchronization timestamp determination submodule, which is used to determine the currently received exposure timestamp as the synchronization timestamp of the currently received image frame if the exposure timestamp is earlier than the receiving timestamp and the image transmission duration is less than or equal to the preset transmission duration; a second synchronization timestamp determination submodule, which is used to determine the synchronization timestamp of the currently received image frame according to the receiving timestamp if the exposure timestamp is earlier than the receiving timestamp and the image transmission duration is greater than the preset transmission duration; and a third synchronization timestamp determination submodule, which is used to determine the currently received image frame as the last received image frame if the exposure timestamp is later than the receiving timestamp, and to re-determine the synchronization timestamp of the last received image frame according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

[0151] Optionally, the second synchronization timestamp determination submodule is specifically used to define the synchronization timestamp of the currently received image frame as t, and define the receiving timestamp as t3, where t is calculated by the following formula:

[0152] t=t3-t λ

[0153] Where, t λ It is the time from the end of exposure to the completion of image output of the image acquisition module.

[0154] Optionally, the second synchronization timestamp determination module 540 includes: a fourth synchronization timestamp determination submodule, which is used to determine the synchronization timestamp of the currently received image frame according to the currently received trigger timestamp, reception timestamp and exposure timestamp if the trigger timestamp is earlier than the reception timestamp and the image acquisition duration is less than the preset acquisition duration; a fifth synchronization timestamp determination submodule, which is used to determine the synchronization timestamp of the currently received image frame according to the reception timestamp and exposure timestamp if the trigger timestamp is earlier than the reception timestamp and the image acquisition duration is greater than or equal to the preset acquisition duration; and a sixth synchronization timestamp determination submodule, which is used to determine the currently received image frame as the last received image frame if the trigger timestamp is later than the reception timestamp, and to re-determine the synchronization timestamp of the last received image frame according to the last received trigger timestamp, exposure timestamp and reception timestamp.

[0155] Optionally, the fourth synchronization timestamp determination submodule includes: a first synchronization timestamp determination unit, which is used to define the synchronization timestamp of the currently received image frame as t if the exposure timestamp is earlier than the trigger timestamp, and define the currently received trigger timestamp as t1, where t is calculated by the following formula:

[0156] t=t1+(t ζ* t ε +t δ )

[0157] Where, t ζ is the wake-up time of the image acquisition module; t ε The time for inputting a pulse to the image acquisition module; t δ The second synchronization timestamp determination unit is used to determine the currently received image frame as the last received image frame, and re-determine the synchronization timestamp of the last received image frame according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

[0158] Optionally, the fifth synchronization timestamp determination submodule includes: a first synchronization timestamp determination unit, which is used to determine the currently received exposure timestamp as the synchronization timestamp of the currently received image frame if the exposure timestamp is earlier than the receiving timestamp and the image transmission duration is less than or equal to the preset transmission duration; a second synchronization timestamp determination unit, which, otherwise, defines the synchronization timestamp of the currently received image frame as t, defines the receiving timestamp as t3, and t is calculated by the following formula:

[0159] t=t3-t λ

[0160] Where, t λ It is the time from the end of exposure to the completion of image output of the image acquisition module.

[0161] Optionally, the device also includes: an offset address determination module, which is used to determine the offset address of the currently received image frame according to the starting address of the currently received image frame and the data size of the currently received image frame after determining the synchronization timestamp for the currently received image frame; and a synchronization timestamp writing module, which is used to write the synchronization timestamp into the currently received image frame based on the offset address.

[0162] The image frame timestamp determination device provided in the embodiment of the invention can execute the image frame timestamp determination method provided in any embodiment of the present application, and has the corresponding performance modules and beneficial effects for executing the image frame timestamp determination method.

[0163] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0164] Embodiment 5

[0165] Figure 6The schematic diagram of the structure of the data processor 610 of the embodiment that can be used to implement is shown. The data processor 610 includes at least one processor 611, and a memory connected to the at least one processor 611 in communication, such as a read-only memory (ROM) 612, a random access memory (RAM) 613, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 611 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 612 or the computer program loaded from the storage unit 618 to the random access memory (RAM) 613. In the RAM 613, various programs and data required for the operation of the data processor 610 can also be stored. The processor 611, the ROM 612, and the RAM 613 are connected to each other via a bus 614. An input / output (I / O) interface 615 is also connected to the bus 614.

[0166] A number of components in the data processor 610 are connected to the I / O interface 615, including: an input unit 616, such as a keyboard, a mouse, etc.; an output unit 617, such as various types of displays, speakers, etc.; a storage unit 618, such as a disk, an optical disk, etc.; and a communication unit 619, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 619 allows the data processor 610 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0167] The processor 611 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 611 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 611 performs the various methods and processes described above, such as a method for determining the timestamp of an image frame.

[0168] In some embodiments, the timestamp determination method for an image frame may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 618. In some embodiments, part or all of the computer program may be loaded and / or installed on the data processor 610 via the ROM 612 and / or the communication unit 619. When the computer program is loaded into the RAM 613 and executed by the processor 611, one or more steps of the timestamp determination method for an image frame described above may be performed. Alternatively, in other embodiments, the processor 611 may be configured to perform the timestamp determination method for an image frame in any other appropriate manner (e.g., by means of firmware).

[0169] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable image frame time stamp determination device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0172] To provide interaction with a user, the systems and techniques described herein may be implemented on a data processor having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the data processor. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0173] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a timestamp determination server for image frames), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0174] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0175] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0176] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for determining a timestamp of an image frame, characterized in that: The method comprises: Determine the image exposure time corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp; Determining whether the currently received image frame is normally exposed according to a relative size relationship between the image exposure time and a preset exposure time; If the currently received image frame is exposed normally, determining the image transmission duration corresponding to the currently received image frame according to the receiving timestamp and the exposure timestamp; determining a synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration; If the exposure of the currently received image frame is abnormal, the image acquisition duration corresponding to the currently received image frame is determined according to the receiving timestamp and the trigger timestamp; and the synchronization timestamp is determined for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration.

2. The timestamp determination method according to claim 1, characterized in that: Determining a synchronization timestamp for the currently received image frame according to a relative size relationship between the image transmission duration and a preset transmission duration includes: If the exposure timestamp is earlier than the receiving timestamp, and the image transmission duration is less than or equal to the preset transmission duration, determining the currently received exposure timestamp as the synchronization timestamp of the currently received image frame; If the exposure timestamp is earlier than the receiving timestamp, and the image transmission duration is greater than the preset transmission duration, determining the synchronization timestamp of the currently received image frame according to the receiving timestamp; If the exposure timestamp is later than the receiving timestamp, the currently received image frame is determined as the last received image frame, and the synchronization timestamp of the last received image frame is re-determined according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

3. The timestamp determination method according to claim 2, characterized in that: Determining a synchronization timestamp of the currently received image frame according to the receiving timestamp includes: The synchronization timestamp of the currently received image frame is defined as t, and the receiving timestamp is defined as t3, where t is calculated by the following formula: t=t3-t λ In the formula, the t λ It is the time from the end of exposure to the completion of image output of the image acquisition module.

4. The time stamp determination method according to any one of claims 1 to 3, characterized in that: Determining a synchronization timestamp for the currently received image frame according to a relative size relationship between the image acquisition time and a preset acquisition time, including: If the trigger timestamp is earlier than the receiving timestamp, and the image acquisition duration is less than the preset acquisition duration, determining the synchronization timestamp of the currently received image frame according to the currently received trigger timestamp, the receiving timestamp and the exposure timestamp; If the trigger timestamp is earlier than the receiving timestamp, and the image acquisition time length is greater than or equal to the preset acquisition time length, determining the synchronization timestamp of the currently received image frame according to the receiving timestamp and the exposure timestamp; If the trigger timestamp is later than the receiving timestamp, the currently received image frame is determined as the last received image frame, and the synchronization timestamp of the last received image frame is re-determined according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

5. The time stamp determination method according to claim 4, characterized in that: Determining a synchronization timestamp of the currently received image frame according to the currently received trigger timestamp, the receiving timestamp, and the exposure timestamp includes: If the exposure timestamp is earlier than the trigger timestamp, the synchronization timestamp of the currently received image frame is defined as t, and the currently received trigger timestamp is defined as t1, where t is calculated using the following formula: t=t1+(tζ*t ε +t δ ) In the formula, the t ζ is the wake-up time of the image acquisition module; ε The time for inputting a pulse into the image acquisition module; the t δ The time it takes for the image acquisition module to scan a row of pixels; Otherwise, the currently received image frame is determined as the last received image frame, and the synchronization timestamp of the last received image frame is re-determined according to the last received trigger timestamp, exposure timestamp and receiving timestamp.

6. The method for determining a timestamp according to claim 4, characterized in that: Determining a synchronization timestamp of the currently received image frame according to the receiving timestamp and the exposure timestamp includes: If the exposure timestamp is earlier than the receiving timestamp, and the image transmission duration is less than or equal to the preset transmission duration, determining the currently received exposure timestamp as the synchronization timestamp of the currently received image frame; Otherwise, the synchronization timestamp of the currently received image frame is defined as t, and the receiving timestamp is defined as t3, where t is calculated by the following formula: t=t3-t λ In the formula, the t λ It is the time from the end of exposure to the completion of image output of the image acquisition module.

7. A device for determining a timestamp of an image frame, characterized in that: The device comprises: An image exposure time determination module is used to determine the image exposure time corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp; An exposure condition determination module, used to determine whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure time and the preset exposure time; A first synchronization timestamp determination module is configured to determine, if the currently received image frame is exposed normally, an image transmission duration corresponding to the currently received image frame according to a reception timestamp and an exposure timestamp; and determine a synchronization timestamp for the currently received image frame according to a relative size relationship between the image transmission duration and a preset transmission duration; The second synchronization timestamp determination module is used to determine the image acquisition duration corresponding to the currently received image frame according to the receiving timestamp and the trigger timestamp if the exposure of the currently received image frame is abnormal; and determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for determining the timestamp of an image frame as described in any one of claims 1 to 6 is implemented.

9. A data processor comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for determining the timestamp of an image frame according to any one of claims 1 to 6 is implemented.

10. A system for determining a timestamp of an image frame, characterized in that: The timestamp determination system comprises: A data processor, used to send a trigger pulse to start the image acquisition module, and record the time of the trigger pulse as a trigger timestamp; also used to receive an image frame output by the image acquisition module, and record the time of receiving the image frame as a receiving timestamp; also used to receive an exposure timestamp output by the image acquisition module; The data processor is further used to determine the image exposure duration corresponding to the currently received image frame according to the currently received trigger timestamp and exposure timestamp; determine whether the currently received image frame is normally exposed according to the relative size relationship between the image exposure duration and the preset exposure duration; if the currently received image frame is normally exposed, determine the image transmission duration corresponding to the currently received image frame according to the receiving timestamp and the exposure timestamp; determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image transmission duration and the preset transmission duration; if the currently received image frame is abnormally exposed, determine the image acquisition duration corresponding to the currently received image frame according to the receiving timestamp and the trigger timestamp; determine the synchronization timestamp for the currently received image frame according to the relative size relationship between the image acquisition duration and the preset acquisition duration; The image acquisition module is used to receive the trigger pulse sent by the data processor, expose and acquire image frames according to the trigger pulse, and send the acquired image frames to the data processor; record the time when the exposure is completed as the exposure timestamp, and send the exposure timestamp to the data processor.

Images