Devices and methods for annotating image capture time, including equipment and media.

Abstract

This application provides an image capture time annotation device, method, apparatus, and medium. The device includes: a coarse code transmitting device, comprising a coarse code modulation circuit and multiple independently switchable coarse code emitters. The coarse code modulation circuit is configured to convert absolute time information into binary code, and the multiple coarse code emitters are configured to emit an array of light spots representing the binary code; a fine code transmitting device, comprising a fine code modulation circuit, a light guide, and a first and a second fine code emitter coupled to the light guide. The fine code modulation circuit is configured to generate a first and a second mutually orthogonal signal, and the first and second fine code emitters are respectively configured to emit the first and second signals; and a host computer configured to determine the absolute time of each pixel row in an image frame based on the image frame. The device and method of this application can accurately annotate image capture time at a low cost.

Description

Technical Field

[0001] This application relates to the field of machine vision, specifically to an image capture time annotation device and method, device, and medium. The device and method are applicable to both rolling shutter cameras and global shutter cameras, and can be widely used in scenarios such as robot control alignment, industrial inspection, and multi-camera joint measurement. Background Technology

[0002] In robotics and industrial automation applications, engineers often need to align image observations with the motion or control commands of the controlled object on the same timeline with high precision. Traditionally, this is done using dedicated industrial cameras that support external triggering and hardware timestamp output. Absolute time is recorded via shutter release cables or synchronization interfaces and uploaded to a host computer along with the image. Such equipment is expensive, has complex interfaces, and requires significant wiring in multi-camera scenarios. Furthermore, rolling shutter cameras inherently have a line-by-line exposure characteristic, meaning that the exposure center times of different lines within the same frame are different, making simple "frame timestamps" insufficient for line-level time registration.

[0003] Existing technologies also include methods that use the time of receiving frames from the host computer to deduce the shooting time. However, due to the instability of latency caused by drive buffering, transmission jitter, and operating system scheduling, the back-drilling error is often difficult to control stably within the range of 5 milliseconds, which cannot meet the stringent requirements of robots for tasks such as kinematic closed-loop, collision detection, or high-speed sorting.

[0004] Therefore, there is a need for a device and method that can accurately determine the time of image capture at a low cost. Summary of the Invention

[0005] In view of this, this application provides an image capture time annotation device and method to solve the above-mentioned technical problems in the prior art.

[0006] According to one aspect of this application, an image capture time annotation device is provided, the device comprising:

[0007] A coarse code transmitting device includes a coarse code modulation circuit and a plurality of coarse code emitters. The coarse code modulation circuit is configured to convert absolute time information into binary code, and the plurality of coarse code emitters are configured to emit an array of light spots representing the binary code.

[0008] A fine code transmitting device includes a fine code modulation circuit, a light guide, and a first fine code emitter and a second fine code emitter coupled to the light guide. The fine code modulation circuit is configured to generate a first signal and a second signal that are orthogonal to each other. The first fine code emitter and the second fine code emitter are respectively configured to emit the first signal and the second signal. The light guide includes a diffuse reflection light-emitting surface.

[0009] The host computer is configured as follows:

[0010] The binary code represented by the light spot array is read from the image frame captured by the camera, and the brightness of the first and second signals in each pixel row of the image frame is sampled from the diffuse reflection area of ​​the fine code emitting device in the image frame.

[0011] The absolute time of the image frame is obtained based on the binary code;

[0012] The relative time corresponding to each pixel row is obtained based on the brightness of the first and second signals in each pixel row of the image frame;

[0013] The absolute time corresponding to each pixel row is obtained by combining the relative time corresponding to each pixel row with the absolute time of the image frame.

[0014] According to a preferred embodiment of this application, obtaining the relative time corresponding to each pixel row based on the brightness of each pixel row in the image frame includes:

[0015] The first row brightness sequence and the second row brightness sequence are obtained based on the brightness of the first signal and the second signal in each pixel row of the image frame;

[0016] The row phase sequence is obtained based on the first row brightness sequence and the second row brightness sequence;

[0017] The relative time of each pixel row is obtained from the row phase sequence.

[0018] According to a preferred embodiment of this application, the row phase sequence is obtained by synthesizing the first row luminance sequence and the second row luminance sequence into an analytical signal and then performing phase extraction and de-envelopeing.

[0019] According to a preferred embodiment of this application, obtaining the relative time of each pixel row based on the row phase sequence includes obtaining the relative time of each pixel row by fitting the row phase sequence with a linear or low-order polynomial.

[0020] According to a preferred embodiment of this application, the host computer is configured to perform bandpass filtering and amplitude normalization processing on the first row brightness sequence and the second row brightness sequence before sampling the brightness of the first signal and the second signal in each pixel row of the image frame from the diffuse reflection area of ​​the fine code transmitting device in the image frame.

[0021] According to a preferred embodiment of this application, the binary code is Gray code, used to represent frame count or millisecond count.

[0022] According to a preferred embodiment of this application, the plurality of coarse code emitters are arranged in a straight line parallel to the scanning direction of the camera's photosensitive device.

[0023] According to a preferred embodiment of this application, the first signal and the second signal are sinusoidal signals of fixed frequency or chirped signals with linear frequency conversion.

[0024] According to a preferred embodiment of this application, the first signal and the second signal are in the infrared band.

[0025] According to a preferred embodiment of this application, the first signal and the second signal are in the near-infrared band.

[0026] According to a preferred embodiment of this application, the length direction of the light guide is set to be perpendicular to the scanning direction of the camera's photosensitive device.

[0027] According to a preferred embodiment of this application, the host computer is connected to the coarse code transmitter and the fine code transmitter via cables respectively, for synchronizing the transmission time of the coarse code transmitter and the fine code transmitter.

[0028] According to another aspect of this application, a method for annotating the time of image capture is provided, the method being executed by a host computer of the device described above, the method comprising:

[0029] The binary code represented by the light spot array is read from the image frame captured by the camera, and the brightness of the first and second signals in each pixel row of the image frame is sampled from the diffuse reflection area of ​​the fine code emitting device in the image frame.

[0030] The absolute time of the image frame is obtained based on the binary code;

[0031] The relative time corresponding to each pixel row is obtained based on the brightness of the first and second signals in each pixel row of the image frame;

[0032] The absolute time corresponding to each pixel row is obtained by combining the relative time corresponding to each pixel row with the absolute time of the image frame.

[0033] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0034] One or more processors;

[0035] Memory, which is used to store executable instructions;

[0036] The one or more processors are configured to implement the methods described above via executable instructions.

[0037] According to another aspect of this application, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, causes the processor to perform the method described above.

[0038] This application's technical solution does not rely on a shutter release cable or internal camera timestamps, but instead embeds "time" into the image content in a way that can be sampled by ordinary cameras. The coarse code provides a stable absolute time window and error correction capabilities, while the fine code provides a continuous and solvable phase reference within a single frame, thus restoring the exposure time to the line level under rolling shutter conditions. Because the fine code uses two quadrature modulation paths with a 90-degree phase difference, the phase estimation still exhibits good robustness even with the time integration effect of camera exposure on the signal. The coarse code, using Gray code supplemented with checksums, effectively resists reading instability caused by local occlusion, reflection, or overexposure. In practice, a single transmitting device can serve multiple cameras simultaneously, greatly reducing the hardware cost and wiring complexity of multi-camera systems. Attached Figure Description

[0039] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application, but do not constitute a limitation on the technical solutions of this application.

[0040] Figure 1 An image capture time annotation device illustrating an exemplary embodiment of this application;

[0041] Figure 2 A coarse code transmitting device is shown as an exemplary embodiment of this application;

[0042] Figure 3 The present application illustrates a code transmitting apparatus according to an exemplary embodiment of the present application;

[0043] Figure 4 A structural block diagram of the host computer provided by an exemplary embodiment of the present invention is shown. Detailed Implementation

[0044] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. The descriptions of the exemplary embodiments are merely illustrative and are not intended to limit the scope of this application or its application or use. This application can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to make the application thorough and complete, and to fully express the scope of this application to those skilled in the art.

[0045] Unless explicitly stated otherwise, an element may be one or more unless otherwise specified. The terms “multiple / several” mean two or more, the term “based on” should be interpreted as “at least partially based on,” and the terms “and / or” and “at least one of…” cover any one of the listed items and all possible combinations thereof. Furthermore, expressions such as “first,” “second,” etc., are for descriptive purposes only and do not indicate or imply their relative importance or implicitly specify the number of technical features indicated.

[0046] Overall Equipment Structure

[0047] Figure 1 An image capture time annotation device is shown as an exemplary embodiment of this application. For example... Figure 1 As shown, the image capture time annotation device of an exemplary embodiment of this application includes a coarse code transmitting unit 10, a fine code transmitting unit 20, and a host computer 30. The coarse code transmitting unit 10 and the fine code transmitting unit 20 are positioned within the field of view 41 of the camera 40. The host computer can receive image signals via the data transmission line 31 and communicate with the coarse code transmitting unit 10 and the fine code transmitting unit 20 via the time synchronization line 32.

[0048] Both the coarse code transmission unit 10 and the fine code transmission unit 20 include modulation circuits, which include a time base module for outputting the time base. The time base module can use a high-stability crystal oscillator as the local clock, or it can be synchronized by the host computer 30 through the time synchronization line 32 or other synchronization interfaces.

[0049] The coarse code transmitting unit 10 includes several independently switchable coarse code emitters arranged in a linear array. The axis of the coarse code emitters is preferably parallel to the row direction of the image sensor to maintain geometric stability of the bright and dark boundaries during line-by-line reading and to maintain a stable sampling interval for each emitter. The coarse code modulation circuit converts the frame count or millisecond count generated by the time base module into Gray code and adds redundant check bits when necessary. The same count value from the coarse code transmitting unit is fed to the host computer via an interface such as serial, USB, or Ethernet to synchronize the time window marked by the coarse code transmitting unit 10. The time window start signal of the coarse code transmitting unit 10 is also sent to the fine code transmitting unit 20 via a time synchronization line to synchronize the start time of the fine code.

[0050] The fine-code emission unit employs two emitters (a first fine-code emitter and a second fine-code emitter) tightly coupled to the light guide, each outputting a single-frequency or linearly chirped signal with orthogonal phase. The side of the light guide facing the camera is made into a diffuse reflective surface, so that the entire emission area appears as a long strip or narrow plane on the camera's imaging plane, forming a non-zero angle with the image frame line direction, preferably orthogonal, ensuring that most lines within the same frame can cover the emission area. To reduce interference with visible light imaging of the scene, the fine-code signal can operate in the infrared or near-infrared band, provided that the camera's image sensor has the corresponding spectral response.

[0051] Coding system and timing

[0052] The coarse code represents absolute time information. When the time information is a frame count, its refresh rate should be equal to the camera frame rate; when the time information is a millisecond, its refresh rate should be faster than the camera frame rate. The coarse code emitter in the coarse code transmitting unit 10 has only two states: on and off, corresponding to binary zero and one. The modulation circuit converts the integer frame count or time into binary Gray code and drives the corresponding emitter in the array to turn on and off. After the array is imaged by the camera, it forms a light and dark pattern with a stable spatial structure. Because adjacent Gray code words change by only one bit, even if there is sub-pixel jitter at the edges during line-by-line reading, it is not easy to cause code word misjudgment.

[0053] The fine code represents a high-precision time scale, and its two signals (I signal, or the first signal; and Q signal, or the second signal) follow an orthogonal relationship. Let the luminous intensity of the two signals (I and Q) be denoted as... and ,but

[0054]

[0055] Phase It can be either a single-frequency or linear chirp:

[0056] - Single frequency:

[0057] - Linear chirping: Its instantaneous frequency is .

[0058] Under the rolling shutter, the first The exposure center time of a row pixel can be approximated as:

[0059]

[0060] in For the first line of the frame time, This is the row time increment. When the exposure duration... When the relatively short modulation period is relatively short, the bar-shaped light-emitting region in the first... The average brightness of the row is approximately equal to the instantaneous luminous intensity. The value of is proportional, that is

[0061]

[0062] This allows for the generation of a phase sample sequence that varies with the line number within a single frame, providing observability for line-level time recovery. The light-emitting region of the fine code signal has a sufficient span in the readout direction of the image sensor (preferably orthogonal to the line direction) to ensure that most lines within the same frame can be effectively sampled. To avoid aliasing with power supply flicker and its harmonics, and to ensure that the time integral of the camera exposure does not completely smooth out phase differences, the operating frequency (or instantaneous frequency) of the fine code signal should preferably be selected in the range of 100 Hz to 1 kHz.

[0063] Upper computer algorithm

[0064] According to an exemplary embodiment of this application, the host computer 30 mainly performs the following steps:

[0065] The binary code represented by the light spot array is read from the image frame captured by the camera, and the brightness of the first and second signals in each pixel row of the image frame is sampled from the diffuse reflection area of ​​the light surface region of the fine code emitting device in the image frame.

[0066] The absolute time of the image frame is obtained from the binary code;

[0067] The relative time corresponding to each pixel row is obtained based on the brightness of the first and second signals in each pixel row of the image frame;

[0068] The absolute time corresponding to each pixel row is obtained by combining the relative time corresponding to each pixel row with the absolute time of the image frame.

[0069] Specifically, the host computer 30 first locates and identifies the coarse code area in the image frame captured by the camera, converts the read Gray code into binary counts, and determines the reliability of the code reading based on the check bit. Then, within the light-emitting area of ​​the fine code's bar shape, the program calculates the average brightness row by row from top to bottom according to the row number, obtaining two sequences. and Therefore, a complex analytic signal is constructed:

[0070]

[0071] And the phase curve that varies with the line number is obtained by phase operation and deenvelation:

[0072]

[0073] For single-frequency fine codes, due to We can obtain a linear relationship between the slope of the phase with respect to the row number and the inter-row time:

[0074]

[0075] In practice, least squares linear fitting is used:

[0076]

[0077] For linear chirped fine codes, we have:

[0078]

[0079] The host computer uses least squares or bounded one-dimensional search to... Performing joint estimation is equivalent to solving

[0080]

[0081] in The operator representation normalizes the phase residual to The envelope operator.

[0082] get After estimation, the absolute time of the first line of the frame still needs to be determined. Let the time window given by the coarse code be... Then, by searching or parsing the registration within this window, a unique identifier is determined. :

[0083]

[0084] in

[0085]

[0086] Finally, the host computer followed the following steps:

[0087]

[0088] Generate an absolute timestamp for each row of the frame; the frame-level timestamp can be a conventional value from the first row, the center row, or the center of the exposure integration. To assess reliability, the program also outputs the fitting residuals:

[0089]

[0090] and its statistics (such as) (as per p95), and reports effective line coverage and signal-to-noise ratio metrics for downstream screening or alignment with factor graph / dynamic time warping (DTW) of robot logs.

[0091] Robust strategy and parameter selection

[0092] To adapt to different illumination and exposure conditions, the host computer performs bandpass filtering and amplitude normalization on the fine code area before calculating the line average to mitigate the effects of gain variations and local overexposure. If a single-sided strip area suffers from insufficient line coverage due to occlusion, symmetrical fine code light strips can be placed on both sides of the scene, and the software can select the optimal one or fuse them. To minimize the impact on visible light images, the fine code preferably uses near-infrared emission and the infrared response channel is enabled at the camera end; the coarse code has a limited impact on the main image due to its lower refresh rate. The selection of the fine code frequency needs to balance avoiding aliases and maintaining sufficient phase slope. In practical engineering, the frequency can be set in the range of 200 to 800 Hz and finely adjusted according to the camera's line time and exposure duration. The coarse code bit width is usually 16 to 32 bits, with 4 to 8 bits of redundancy check to ensure reading reliability under long-distance or low-contrast conditions.

[0093] Time alignment with robot control logs in industrial applications

[0094] The system outputs line-by-line timestamps and can directly align them with the joint states or command sequences of the robot control system in terms of time. This is due to the fine-code recovery... It accurately reflects the exposure time of each line under the rolling shutter, and the host computer can map the observation time of any feature line or target edge within the image frame to the robot control timeline. To compensate for the slight drift of clocks from different devices during long-term operation, the program can use a sliding window approach. and Perform online calibration and, if necessary, use the coarse code count via the synchronization interface as an additional constraint to maintain absolute time base consistency across the entire system.

[0095] Industrial Applications and Effects

[0096] The device in this application embodiment can achieve absolute time stamping from frame level to line level using ordinary USB or Ethernet cameras, without relying on shutter release cables and internal camera timestamps. In practice, the device achieves sub-millisecond line alignment accuracy under rolling shutter conditions and sub-millisecond frame alignment accuracy under global shutter conditions. Since a single transmitting device can serve multiple cameras simultaneously, the overall hardware cost and wiring complexity are significantly reduced, making it suitable for deployment in robot workstations, production line vision units, and multi-camera motion capture platforms.

[0097] The following will combine Figures 1 to 4 A detailed description of the device according to embodiments of this application.

[0098] like Figure 1As shown, the image capture time annotation device of an exemplary embodiment of this application includes a coarse code transmitting unit 10, a fine code transmitting unit 20, and a host computer 30. The host computer 30 can be, for example, a common desktop computer, which communicates with the camera and the coarse and fine code transmitting units via a wired network. The camera 40 can be, for example, a common rolling shutter security camera, which can work in conjunction with the annotation device according to the embodiment of this application. The image resolution of the camera is, for example, [missing information]. The frame rate is The camera's image sensor uses a progressive scan mode, meaning that the 0th row of pixels in the same frame is exposed first, followed by the 1st row, the 2nd row, and so on, until the 3rd row is exposed. Therefore, the exposure center times of pixels in different rows of the same frame are not the same.

[0099] A coarse code emitting unit 10 and a fine code emitting unit 20 are arranged within the imaging field of view 41 of the camera 40. The coarse code emitting unit 10 is preferably placed in the bottom region of the image field of view 41, so that its light-emitting array forms a compact and easily identifiable bright / dark bit pattern in the final image. The fine code emitting unit 20 is preferably placed in the left edge region of the image field of view and extends along the vertical direction of the image, so that it appears as an approximately vertical bright band in the image as a strip of light-emitting area. The relative position and distance between the fine code emitting unit 20 and the camera 40 are adjusted so that this bright band covers no less than 90% of the effective pixel rows in the vertical direction of the image; that is, in this embodiment, the resolution... Under these conditions, the light-emitting area of ​​the fine barcode should be at least approximately Visible within the row range. This high-coverage arrangement allows the host computer to extract the fine code signal from the vast majority of rows in the same frame image, thereby recovering the exposure time of each row in subsequent steps.

[0100] The system also includes electrical connections. Specifically, both the coarse code transmission unit 10 and the fine code transmission unit 20 include their own modulation circuits (such as coarse code modulation circuit 103 and fine code modulation circuit 204). These circuits drive their respective light sources to emit light and output corresponding time reference information (such as the current millisecond count value and the chirp sequence trigger time) to the host computer via the time synchronization line 32. The camera 40 and the host computer transmit the acquired image frames via a conventional data transmission line 31 (such as Ethernet) without the need for additional hardware trigger lines. With this arrangement, this embodiment can enable the host computer to obtain row-level absolute exposure time information without modifying the camera 40 itself.

[0101] Figure 2 A coarse code transmitting device according to an exemplary embodiment of this application is shown. For example... Figure 2As shown, the coarse code transmitting unit 10 includes a housing 101, an emitter array 102, a coarse code modulation circuit 103, a light source signal line 104, and a time synchronization line 105. The emitter array consists of 32 infrared light-emitting diodes (LEDs) arranged sequentially in a horizontal direction. The center wavelength of the LEDs is preferably about 850 nm, used to form a distinguishable array of bright and dark light spots in the camera image frame. Each LED corresponds to one bit of the time code, with a bright light representing logic "1" and an off light representing logic "0". Because it uses, for example, infrared light, the emitter array has little impact on the visible light imaging of the scene, while still providing clear imaging on ordinary security cameras (which typically have infrared response).

[0102] The coarse code modulation circuit 103 internally includes a stable time base module for generating absolute time counts. For example, in this embodiment, the time base module counts system time in milliseconds. When the current time is 1 second, its millisecond count value is 1000. The coarse code modulation circuit 103 represents this count value as binary code and further converts it into Gray code representation. Taking 1000 milliseconds as an example, its binary code can be represented as 0000 0011 1110 1000 (16 bits), which can be converted into Gray code as 00000010 0001 1100. In this embodiment, the Gray code is extended into a 32-bit display vector and driven to 32 LEDs, where the high-order bits can be used for zero-padding, redundancy check, or subsequent extension bits, and the low-order bits correspond sequentially to the on / off state of the specific LEDs according to the Gray code order. The corresponding LED lighting sequence forms a distinct bit pattern in the image frame, facilitating subsequent frame-by-frame decoding by the host computer.

[0103] As the light-emitting array of the coarse code transmitting unit updates to the aforementioned Gray code pattern, the coarse code modulation circuit 103 also sends the corresponding millisecond count value (e.g., "the currently displayed millisecond time is 1000") to the host computer via the time synchronization line. Therefore, the host computer 30 can not only obtain this millisecond count value by decoding the light-emitting array in the image frame, but also directly confirm the meaning and update time of the count through the value of the time synchronization line 32. In the subsequent time registration process, the host computer 30 interprets this millisecond-level count as the absolute time reference interval of the image frame, thereby providing a coarse-grained time window for the capture time of that frame. This does not depend on the arrival timestamp of the camera driver layer.

[0104] By employing Gray code instead of direct binary code representation, the coarse code transmission unit 10 changes only a small amount of LED on / off states between adjacent count values, thereby reducing the probability of bit errors caused by image acquisition noise or pixel threshold jitter. Since the emitter array is located in a fixed area at the bottom of the image frame, the host computer 30 can stably extract the region of interest (ROI) of the array in each image frame, binarize it, and recover the corresponding millisecond count value. Therefore, in this embodiment, the coarse code transmission unit serves as a robust, verifiable, and approximately absolute time reference for each image frame.

[0105] Figure 3 A code transmitting apparatus according to an exemplary embodiment of this application is shown. Figure 3 As shown. The fine code emission unit 20 includes a light guide 201, an I-channel emitter (first fine code emitter) 202, a Q-channel emitter (second fine code emitter) 203, a fine code modulation circuit 204, a light source signal line 205, and a time synchronization line 206. The light guide 201 is preferably made of acrylic material with good light transmittance. Its surface facing the camera is frosted or sandblasted to form a strip-shaped light-emitting area with approximately diffuse reflection, so that this area appears as a bright band extending vertically on the camera's imaging plane. This bright band is orthogonal to the image frame line direction and continuously covers at least about 90% of the effective pixel lines in this embodiment in the vertical direction, thereby ensuring that the camera can extract the fine code signal from the vast majority of lines in the same frame image.

[0106] The I-channel and Q-channel emitters are two independent infrared LEDs, preferably both using a wavelength of approximately 850 nm, and physically coupled to the light guide, allowing their light to propagate inside the light guide and be uniformly emitted from the frosted surface. The fine-code modulation circuit 204 drives the I-channel and Q-channel LEDs to generate two amplitude-modulated optical signals (referred to as the first signal and the second signal), which are orthogonal in phase in time. Specifically, the luminous intensity of the I-channel LED can be expressed as:

[0107]

[0108] The luminous intensity of a Q-channel LED can be expressed as:

[0109]

[0110] in and These represent the amplitudes of the two light sources, It is a phase function that varies continuously over time. Preferably, Using linear chirp (linear frequency modulation):

[0111]

[0112] Its instantaneous frequency can be expressed as:

[0113]

[0114] in The initial instantaneous frequency is preferably in the range of 200 Hz to 400 Hz; For linear frequency modulation slope, preferably located at to The frequency design ensures that the instantaneous frequency changes by only tens to hundreds of hertz within the effective exposure range of a single frame (typically several milliseconds to tens of milliseconds). This gentle frequency change guarantees that the phase changes monotonically with time and has a slight secondary curvature, allowing the host computer to simultaneously estimate the inter-line time increment when fitting the phase curve. With frame first line time This allows for the restoration of line-by-line exposure time under a rolling shutter; on the other hand, it avoids the risks of severe integral smoothing and overexposure caused by excessively high frequency.

[0115] In this embodiment, the driving intensity of the I-path and Q-path LEDs (i.e., and The average brightness of the light-emitting strip area on the camera image is set by adjusting the drive current or pulse width modulation duty cycle so that it falls within the middle of the image sensor's dynamic range. This avoids both excessive darkness approaching the noise floor and excessive brightness causing saturation clipping. Preferably, the average brightness of this strip area on the camera image occupies approximately the middle segment of the full-scale grayscale value, while its brightness fluctuation (peak-valley variation) over time accounts for approximately 20% to 40% of the full-scale value. This preserves observable modulation under typical rolling shutter line-by-line integration conditions without significantly interfering with other areas of the image. Due to light guide coupling, LED batch variations, and viewing angle changes, the effective amplitude of the I and Q paths on the imaging plane... They may not be completely identical. In this case, the host computer will perform DC component subtraction and amplitude normalization on the progressive brightness curves obtained from the I and Q paths respectively during subsequent processing, so that they form orthogonal components with similar amplitudes in numerical terms, in order to construct a stable analytical signal for phase calculation.

[0116] Furthermore, the fine-code modulation circuit 204 initiates a new linear chirp sequence (i.e., restarts a new sequence). During the evolution of the chirped phase, a synchronization trigger signal can be sent to the host computer 30 via a time synchronization line. The host computer 30 records this trigger signal as the start time of that chirped phase segment. Using this marker, the host computer 30 can determine the chirped phase interval corresponding to each frame in the case of continuous acquisition across multiple frames, thereby enabling phase... To avoid cross-frame confusion during envelopment and fitting, and to provide a consistent reference for finally registering the phase information of each line with the millisecond-level absolute time window provided by the coarse code.

[0117] After receiving the image frame, the host computer 30 first locates and identifies the area where the coarse code emission unit 10 is located at the bottom of the image frame. The coarse code emission unit 10 consists of a light-emitting array of 32 sequentially arranged 850 nm infrared LEDs. The coarse code modulation circuit 103 converts the current millisecond-level time into a 32-bit binary Gray code and represents it with on / off states. For example, when the current absolute time is 1 second, that is, the millisecond time is 1000 ms, the coarse code modulation circuit first obtains the binary representation of this time 1000 (decimal) = 0000 0011 1110 1000 (binary), and then converts it into Gray code 0000 0010 00011100. This Gray code is extended to 32 bits and drives 32 LEDs respectively, so that the on / off pattern of the LEDs forms a stable light and dark pattern in the image. At the same time, the value of the millisecond time (e.g., 1000) is also sent to the host computer via the time synchronization line, so that the host computer can know the coarse time window corresponding to the current image on the software side. In this step, the host computer reads the pixel values ​​of the coarse code area, binarizes and decodes the on / off states of the 32 LEDs, inversely transforms the read Gray code into binary count values, and uses the redundant bits or check bits (such as CRC or parity check) to determine whether the code reading is reliable. Through this process, the host computer 30 obtains an absolute time reference interval, denoted as . This range defines the time window within which the frame should be captured, which is on the order of milliseconds.

[0118] After completing the coarse code parsing, the host computer 30 continues to process the fine code bar-shaped light-emitting area located on the left side of the image. The fine code emitting unit 20 includes an acrylic light guide, one side of which is frosted to form an approximately diffuse reflection bar-shaped surface light source. An I-channel emitter 202 and a Q-channel emitter 203, both 850nm infrared LEDs, are coupled to the light guide. The fine code modulation circuit 204 drives these two LEDs to generate continuously modulated light signals with a 90-degree phase difference. The luminous intensity of the I-channel LED is:

[0119]

[0120] The luminous intensity of the Q-channel LED is:

[0121]

[0122] in This is the phase function of the fine-code signal. In this embodiment, a linear chirped form is preferred, i.e.

[0123]

[0124] Its instantaneous frequency is:

[0125]

[0126] in The initial instantaneous frequency is preferably in the range of 200 Hz to 400 Hz; For linear frequency modulation slope, the preferred order of magnitude is... to The selection of the above parameters ensures that the instantaneous frequency varies only within the range of tens to hundreds of hertz within the single-frame exposure duration (typically several milliseconds to tens of milliseconds). This guarantees, on the one hand, that the phase exhibits monotonic and unenvelopable growth within the same frame, and on the other hand, maintains a sufficiently high signal-to-noise ratio under the line-by-line integration condition of a rolling shutter. The amplitudes of the I and Q paths... By adjusting the LED drive current or pulse width modulation duty cycle, the average brightness of the corresponding strip areas on the camera's imaging plane is made to fall within the middle of the image sensor's dynamic range (e.g., between 100 and 156 in an 8-bit grayscale image), and the peak-to-valley swing of the brightness is kept within approximately 20% to 40% of full scale. This avoids overexposure-induced saturation clipping while ensuring the modulation remains clearly visible. In subsequent processing, the host computer performs amplitude normalization and DC component subtraction on the I-channel and Q-channel line brightness sequences respectively to compensate for differences in LED coupling, light guide inhomogeneity, and ambient light background, ensuring that the two signals have approximately the same amplitude scale when calculating phase.

[0127] When extracting the fine code, the host computer 30 follows the camera's row number. (in The image is processed row by row sequentially. Within each row, the host computer defines a region of interest (ROI) covering the light-emitting area of ​​the thin barcode, and takes the row average of the pixels in the corresponding I-path and Q-path regions respectively, obtaining two one-dimensional brightness sequences that vary with the row number. and Based on these two sequences, the host machine constructs a complex analytic signal:

[0128]

[0129] And calculate its complex argument, i.e.

[0130]

[0131] Get a line number The estimated phase curve that changes gradually with time The above " "This indicates that the phase is continuously unencapsulated, and..." of The jump is restored to the real-valued curve of the monotonic evolution of the accompanying number.

[0132] Since the camera in this embodiment is a rolling shutter camera, the camera's image sensor starts exposure line by line. Therefore, the exposure center times of different lines within the same frame are not the same. It can be approximated that the first... The exposure center moment of the line is

[0133]

[0134] in The exposure center time is the first line of the frame (i.e., line 0). This is the exposure start time interval between two adjacent rows (i.e., the inter-row time increment). If the fine code signal is a single-frequency sine wave, then... ,So It approximately satisfies a linear relationship:

[0135]

[0136] Therefore, we can deduce that:

[0137]

[0138] In practice, the host computer will use a least-squares linear fitting method to... To perform fitting, i.e.

[0139]

[0140] In this embodiment, the fine code preferably uses linear chirp, therefore the phase includes a term that is quadratically related to time:

[0141]

[0142] Will use Substituting, we get:

[0143]

[0144] The host computer 30 treats the above relationship as a relationship with respect to unknown parameters. The model is constructed, and these parameters are jointly estimated through least-squares fitting or a one-dimensional search with bounded constraints. To avoid... To address the ambiguity caused by periodicity, the sum of squares of the envelope residuals is used as the objective function during fitting, i.e., the solution is:

[0145]

[0146] in The operator representation constrains the phase residual to The interval is set to maintain the continuity of the residuals. Through this fitting process, the host computer obtains the inter-row time increments. The estimated value And the first line time of the frame The relative estimate.

[0147] However, It still needs to be aligned with absolute time. Therefore, the host computer uses the millisecond-level time information transmitted by the coarse code transmission unit to establish an absolute time reference interval. The host computer processes the candidate data within that time window. Perform a search or analytical registration to select the phase that matches the predicted phase with the observed phase. The one that is most consistent is used as the absolute time of the first line of the final frame. This process can be formalized as follows:

[0148]

[0149] in This represents the theoretical phase reconstructed based on the fine-code modulation form. If the fine code is a single-frequency sine wave, then:

[0150]

[0151] If the finecode is linearly chirped, then:

[0152]

[0153] After the above calculations, the host computer finally obtained the absolute time of the first line exposure of the frame. and inter-line time increment Therefore, for any line in this frame... The absolute time of its exposure center can be expressed as:

[0154]

[0155] The host computer 30 writes this expression as a row-level time stamp result into the metadata of the frame. The frame-level time can be taken as the time of the 0th row, according to convention. Alternatively, you can take the middle row. Alternatively, the integral center time of the entire frame can be obtained by using the exposure model.

[0156] To report the reliability of this timestamp to subsequent processing stages (such as time alignment with the robotic arm control log), the host computer will simultaneously calculate the fitting residual:

[0157]

[0158] And give the root mean square value of the residual:

[0159]

[0160] And robust metrics such as the absolute residual at the 95th percentile (p95). The host computer 30 also counts how many rows in the current frame can successfully extract the I / Q row mean (i.e., the proportion of rows actually covered by the barcode to the total number of rows), and outputs this proportion as the effective row coverage rate. If the effective row coverage rate is lower than a predetermined threshold (e.g., lower than 80%), or If p95 exceeds the threshold, the row-level timestamp of that frame can be marked as "insufficient in quality" and downgraded to using only the frame-level timestamp during subsequent data fusion. In this way, the host computer 30 can not only generate absolute timestamps line by line, but also output the corresponding quality indicators at the same time, enabling downstream robot control alignment, trajectory reconstruction, or multi-camera fusion modules to select or filter based on quality information.

[0161] This application also provides a method for marking the time of image capture, which uses the host computer 30 described above to perform related processing steps.

[0162] According to embodiments of this application, the host computer is an electronic device that includes one or more processors and a memory for storing executable instructions. The one or more processors are configured to implement the operations of the host computer via the executable instructions. The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to perform the above-described methods. In the following portions of this specification, [further details will be provided]. Figure 4 Illustrative examples are provided to describe the aforementioned electronic devices and computer-readable storage media.

[0163] Figure 4 An example configuration of an electronic device 300 that can be used to implement the methods described in this invention is shown. The technical solutions of this invention can also be implemented wholly or at least partially by the electronic device 300 or similar devices / systems. The electronic device 300 can be of various different types. Examples of the electronic device 300 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices, wearable devices, entertainment devices, televisions or other display devices, automotive computers, etc.

[0164] Electronic device 300 may include at least one processor 302, memory 304, multiple communication interfaces 309, display device 301, other I / O devices 310, and one or more mass storage devices 303 capable of communicating with each other via system bus 311 or other suitable connection.

[0165] Processor 302 may be one or more processing units, and all processing units may include one or more computing units or multiple cores. Processor 302 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 302 may be configured to fetch and execute computer-readable instructions stored in memory 304, mass storage device 303, or other computer-readable media, such as program code of operating system 305, application program 306, or other program 307.

[0166] Memory 304 and mass storage device 303 are examples of computer-readable storage media for storing instructions executed by processor 302 to perform the various functions described above. For example, memory 304 can generally include both volatile and non-volatile memory. Furthermore, mass storage device 303 can generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks, storage arrays, network-attached storage, storage area networks, etc. Memory 304 and mass storage device 303 can both be collectively referred to as memory or computer-readable storage media in this invention, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which can be executed by processor 302 as a specific machine configured to perform the operations and functions described in the examples of this invention.

[0167] Multiple programs can be stored on mass storage device 303. These programs include operating system 305, one or more application programs 306, other programs 307, and program data 308, and they can be loaded into memory 304 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: the methods provided by the present invention (including any suitable steps of the method) and / or other embodiments described in the present invention.

[0168] Although Figure 4The data is shown as stored in the memory 304 of the electronic device 300, but the modular operating system 305, application program 306, other programs 307, and program data 308, or portions thereof, can be implemented using any form of computer-readable medium accessible by the electronic device 300. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer. A communication medium includes a medium through which communication signals, such as computer-readable instructions, data structures, program modules, or other data, are transmitted from one system to another. The communication medium can include guided transmission media and wireless media capable of propagating energy waves. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals, for example, in a wireless medium.

[0169] For example, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other known media or those developed hereafter capable of storing computer-readable information / data for use by a computer system.

[0170] One or more communication interfaces 309 are used to exchange data with other devices via means such as a network or direct connection. This communication interface can be one or more of the following: any type of network interface, wired or wireless (such as WLAN) interface, Wi-MAX interface, Ethernet interface, USB interface, cellular network interface, Bluetooth interface, NFC interface, etc. Communication interface 309 can facilitate communication across various network and protocol types, including wired and wireless networks, the Internet, etc. Communication interface 309 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0171] In some examples, a display device 301, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 310 may be devices that receive user input and provide output to the user, and may include touch / gesture input devices, cameras, keyboards, remote controls, mice, audio input / output devices, etc.

[0172] The technical solutions described in this invention can be supported by various configurations of the electronic device 300, and are not limited to the specific examples of the technical solutions described in this invention. The foregoing description and illustrations of this invention are not restrictive. It will be apparent to those skilled in the art that this invention is not limited to the details of the exemplary embodiments described above, and that this invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the scope of protection claimed by this invention is defined by the claims rather than the foregoing description, and all variations falling within the meaning and scope of equivalents of the claims are covered within the scope of protection of this invention.

Claims

1. A device for marking the time of image capture, characterized in that, The device comprises: a coarse code emitting device comprising a coarse code modulation circuit configured to convert absolute time information into a binary code, and a plurality of coarse code light emitters configured to emit an array of light spots representing the binary code; a fine code emitting device comprising a fine code modulation circuit configured to generate first and second mutually orthogonal signals, a light guide, and first and second fine code light emitters coupled to the light guide and configured to emit the first and second signals, respectively, the light guide comprising a diffuse light output surface; a host computer in communication with the camera, the coarse code emitting device and the fine code emitting device being arranged within an imaging field of view of the camera, the host computer being configured to: read the binary code represented by the array of light spots from an image frame captured by the camera, and sample the luminance of the first and second signals in each pixel row from a region of the diffuse light output surface of the fine code emitting device in the image frame, obtain an absolute time of the image frame from the binary code, obtain a relative time corresponding to each pixel row from the luminance of the first and second signals in each pixel row in the image frame, obtain an absolute time corresponding to each pixel row from the relative time corresponding to each pixel row and the absolute time of the image frame, wherein obtaining a relative time corresponding to each pixel row from the luminance of the first and second signals in each pixel row in the image frame comprises: obtaining a first row luminance sequence and a second row luminance sequence from the luminance of the first and second signals in each pixel row in the image frame, obtaining a row phase sequence from the first row luminance sequence and the second row luminance sequence, obtaining a relative time of each pixel row from the row phase sequence.

2. The apparatus of claim 1, wherein, The row phase sequence is obtained by synthesizing the first row luminance sequence and the second row luminance sequence into an analytic signal and performing phase extraction and de-enveloping.

3. The apparatus of claim 1, wherein, Obtaining a relative time of each pixel row from the row phase sequence comprises obtaining a relative time of each pixel row by linear or low-order polynomial fitting on the row phase sequence.

4. The apparatus of claim 1, wherein, The host computer is configured to, before sampling the luminance of the first and second signals in each pixel row from a region of the diffuse light output surface of the fine code emitting device in the image frame, perform band-pass filtering and amplitude normalization on the first row luminance sequence and the second row luminance sequence.

5. The apparatus of claim 1, wherein, The binary code is a Gray code used to represent frame count or millisecond count.

6. The apparatus of claim 1, wherein, The plurality of coarse code light emitters are arranged in a straight line parallel to the scanning direction of the camera photosensitive device.

7. The apparatus of claim 1, wherein, The first and second signals are sinusoidal signals of fixed frequency or chirp signals of linearly variable frequency.

8. The apparatus of claim 1, wherein, The wave band of the first and second signals is an infrared wave band.

9. The apparatus of claim 8, wherein, The wave band of the first and second signals is a near-infrared wave band.

10. The apparatus of claim 1, wherein, The length direction of the light guide is arranged perpendicular to the scanning direction of the camera photosensitive device.

11. The apparatus of claim 1, wherein, The host computer is connected to the coarse code emitting device and the fine code emitting device through a cable, respectively, for synchronizing the emission time of the coarse code emitting device and the fine code emitting device.

12. An image photographing time labeling method, the method being executed by a host computer of the device according to any one of claims 1 to 11, characterized in that, The method comprises: reading a binary code represented by the array of light points from an image frame captured by a camera, and sampling an image frame from the image frame for a diffusely reflecting light emitting surface area of the device for a luminance of the first signal and the second signal in each pixel row of the image frame, obtaining an absolute time of the image frame according to the binary code; obtaining a relative time corresponding to each pixel row respectively according to the luminance of the first signal and the second signal in each pixel row of the image frame; obtaining an absolute time corresponding to each pixel row respectively according to the relative time corresponding to each pixel row respectively and the absolute time of the image frame.

13. An electronic device, comprising: The electronic device comprises: one or more processors; a memory for storing executable instructions; the one or more processors are configured to implement the method of claim 12 via the executable instructions.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, causes the processor to perform the method of claim 12. The computer program, when executed by a processor, causes the processor to perform the method of claim 12.

Images