Verification system for multi-camera output time sequence adaptive image processing system input time sequence
By simulating the shooting timing of multiple cameras and adjusting the output timing, combined with the image data transmission timing verification of FPGA, the adaptation problem of the massive image data output timing of multiple cameras and the input timing of the image processing system is solved, and the real-time and adaptability of the system is improved.
Patent Information
- Application Number
- CN202510102142.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-22
- Publication Date
- 2025-04-18
AI Technical Summary
In a large field of view high-speed visual measurement system, there are challenges in adapting the output timing of massive image data of multiple cameras and the input timing of image processing system, resulting in the complexity of image processing system and the increased difficulty of camera adaptation.
The camera shooting timing simulation software generates multi-camera shooting timing, and the image transmission timing conversion and control software adjusts the single-camera output timing to form a multi-camera output timing suitable for the input timing of the image processing system, and uses FPGA to verify the image data transmission timing.
It realizes an effective balance between the output timing of massive image data of multiple cameras and the input timing of image processing system, reduces the difficulty of FPGA in controlling multi-camera shooting and completing image data transmission functions, and improves the real-time and adaptability of the system.
Smart Images
Figure CN119854646A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of camera image transmission, and in particular to a verification system for adapting the output timing of multiple cameras to the input timing of an image processing system. Background Art
[0002] In the fields of video surveillance, virtual reality, aerial detection, range testing, target interception, robot vision, biological and mechanical research, there are a large number of high-speed moving targets. Large-field-of-view high-speed visual measurement that combines large-field-of-view imaging with an image processing system is an effective method for capturing and measuring the motion state of high-speed moving targets. A multi-camera system composed of high-resolution cameras can complete large-field-of-view image capture. Through a single exposure of multiple cameras, it can capture real-time visual information in space, which can effectively solve the problems of limited observation field of view and insufficient resolution of traditional visual sensors. However, it brings about complex multi-camera shooting timing and image output timing. The complex multi-camera image output timing makes the image processing system complicated. A single image processor is difficult to handle image processing tasks, and the number of image processors needs to be increased accordingly. The input timing of the image processing system also becomes more complicated.
[0003] Based on the massive image data generated by the multi-camera shooting sequence, how to balance the output timing of the multi-camera massive image data and the requirements of the image processing system input timing is a key link that needs to be evaluated to improve the real-time performance of the large-field high-speed visual measurement system. At present, there are two main methods for adapting the camera output timing and the image processing system input timing. One is that the image processing system is adapted according to the camera output timing, but the resolution and shooting frame rate of different types of cameras are inconsistent, resulting in complex data output timing. At the same time, the output timing of various cameras provided by different suppliers is inconsistent, which increases the difficulty of image processing system adaptation. Another method is to change the camera output timing to meet the image processing input timing. The existing high-speed visual measurement image processing system mainly uses an embedded hardware acceleration platform, which has a single hardware interface, complex processing algorithm timing, and high image input timing requirements. At the same time, the output timing and interface of different cameras are not unified. In order to adapt the image processing system, the camera interface and transmission timing conversion are required, which increases the difficulty of camera adaptation. Under the massive image data of multiple cameras, the traditional method of adapting the camera image output timing and the image processing system input timing is challenging. Summary of the invention
[0004] The purpose of the present invention is to provide a verification system for adapting the output timing of multiple cameras to the input timing of an image processing system. By processing the image data and shooting timing information of multiple cameras, the output timing of image data of a single camera is changed, and the output timings of multiple single cameras are adjusted to the output timing of multiple cameras suitable for the input timing of the image processing system. FPGA can be used to transmit the image data to the image processing system according to the output timing of the multiple cameras to verify whether the output timing of the massive image data of the multiple cameras is adapted to the input timing of the image processing system, so as to effectively balance the requirements of the output timing of the massive image data of the multiple cameras and the input timing of the image processing system.
[0005] The object of the present invention is to achieve the following technical solutions:
[0006] A verification system for adapting the output timing of multiple cameras to the input timing of an image processing system, comprising:
[0007] The system is composed of camera shooting timing simulation software, image transmission timing conversion and control software, and an image data transmission timing verification module based on FPGA. The camera shooting timing simulation software is used to generate multi-camera shooting timings, the image transmission timing conversion and control software is used to configure image transmission configuration parameters according to the multi-camera shooting timings, and control the image data transmission timing verification module based on FPGA. The image data transmission timing verification module based on FPGA outputs transmission timing suitable for processing by the image processing system according to the image transmission configuration parameters configured by the image transmission timing conversion and control software.
[0008] Compared with the prior art, the present invention has the following beneficial effects:
[0009] (1) The system can easily simulate the shooting sequence of multiple cameras, and can support camera selection and determine the shooting sequence of multiple cameras during the design phase of a large-field-of-view high-speed visual measurement system.
[0010] (2) The system can realize the conversion from multi-camera shooting timing to multi-camera output timing, and can verify whether the multi-camera shooting timing and multi-camera image transmission output timing are reasonable when transmitting massive data.
[0011] (3) The system’s image data transmission rate and transmission mode are configurable, and the rationality of the image transmission configuration parameters can be verified.
[0012] (4) The system can assist in the design and implementation of image data timing for large-field-of-view high-speed vision measurement systems, verify whether the output timing of multi-camera massive image data is compatible with the input timing of the image processing system, reduce the difficulty of FPGA in controlling multi-camera shooting and completing image data transmission functions, and effectively balance the requirements of the output timing of multi-camera massive image data and the input timing of the image processing system. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 A schematic block diagram of a verification system for adapting output timings of multiple cameras to input timings of an image processing system according to an embodiment of the present invention is shown.
[0014] Figure 2 A method for transmitting and outputting images of eight infrared cameras provided by an embodiment of the present invention is shown.
[0015] Figure 3 A schematic diagram of an image transmission output cycle of eight infrared cameras provided by an embodiment of the present invention is shown.
[0016] Figure 4 The embodiment of the present invention shows an 8-frame short-wave infrared camera image interleaving transmission output timing. DETAILED DESCRIPTION
[0017] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms, and the present disclosure should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0018] Figure 1 FIG. 1 is a schematic block diagram of a verification system for adapting the output timing of multiple cameras to the input timing of an image processing system according to an embodiment of the present invention. Figure 1 As shown, the system includes camera shooting timing simulation software, image transmission timing conversion and control software, and an image data transmission timing verification module based on FPGA. The camera shooting timing simulation software is used to generate multi-camera shooting timing, the image transmission timing conversion and control software is used to configure image transmission configuration parameters according to the multi-camera shooting timing, and control the image data transmission timing verification module based on FPGA. The image data transmission timing verification module based on FPGA outputs a transmission timing suitable for processing by the image processing system according to the image transmission configuration parameters configured by the image transmission timing conversion and control software.
[0019] The above modules are described in detail below.
[0020] The camera shooting timing simulation software needs to generate camera information such as camera type and camera number according to the camera type, number of cameras, camera arrangement, whether the camera is moving and camera exposure method, generate multi-camera shooting timing information such as the exposure start and end time of each camera, exposure duration, image readout time, and generate a file for each multi-camera shooting parameter and shooting timing parameter information.
[0021] The image transmission timing conversion and control software processes the multi-camera shooting timing information, changes the single-camera output timing, and adjusts the output timing of multiple single cameras to the multi-camera output timing that is suitable for the input timing of the image processing system. The process includes:
[0022] 1) First, read the multi-camera shooting timing information generated by the camera shooting timing simulation software.
[0023] 2) According to the image input timing information processed by the image processing system, the image transmission parameters are configured, including the camera image output channel number configuration, the mapping relationship between each camera number and the camera image output channel number, the output image transmission time of each camera, the camera group number, whether the image data is transmitted in packet interleaving, and the mapping relationship between the camera number and the interleaving method.
[0024] 3) According to the mapping relationship between camera numbers and camera image output channel numbers, the camera numbers are mapped to the corresponding camera image output channel queues.
[0025] 4) For the data on each camera image output channel queue, calculate the start time of each camera image output according to the camera type, exposure start time, exposure end time, exposure time and image readout time.
[0026] 5) Sort the data labels of the data on each camera image output channel queue from small to large according to the label information at the start time of image output, and determine the image transmission queue of each camera image output channel.
[0027] 6) After the image transmission output start time is sorted, the transmission interval of the current image transmission output start time is obtained by subtracting the next image transmission output start time in the queue. According to the image transmission time configuration parameters of the camera image output channel, it is judged whether the multi-camera shooting timing and the image transmission time of the camera image output channel configured by FPGA are reasonable. The judgment method can be: if the transmission interval time of the current image transmission output start time is greater than the FPGA transmission time, it is reasonable, otherwise it is unreasonable. If it is reasonable, the image transmission time and pause time of the camera image output channel configured by FPGA are generated; if it is unreasonable, it is prompted that the image transmission time setting of the camera image output channel configured by FPGA and the multi-camera shooting timing are unreasonable, and the image transmission time of the camera image output channel configured by FPGA or the multi-camera shooting timing needs to be modified.
[0028] 7) After completing the transmission time and pause time settings, the image transmission start waiting time under the camera image output channel number is generated based on the first frame image transmission output start time minus the corresponding camera exposure start time.
[0029] 8) Configure the image transmission configuration parameters through the image transmission timing conversion and control software user interface, and finally convert the multi-camera shooting timing information into the following image transmission configuration parameters: image transmission start waiting time for each camera image output channel, start transmission time of each group of images (including multiple frames or single frame), number of image frames transmitted continuously, transmission time and transmission pause time of each group of images, number of camera images in each group of images and interleaving method, etc.
[0030] 9) Draw a timing diagram on each camera image output channel based on the image transmission start waiting time, the start transmission time, transmission time and transmission pause time of each group of images (including multiple frames or single frame) to verify whether the configured image transmission parameters and multi-camera shooting timing are reasonable.
[0031] The FPGA-based image data transmission timing verification module controls the camera image output channel, image transmission time, transmission pause time, and image interleaving method according to the image transmission timing conversion and image transmission configuration parameters configured by the control software, combined with the images transmitted in timing by the front-end multi-camera shooting. It changes the transmission rate of the image data and the transmission interval between two frames of images, and outputs a transmission timing suitable for processing by the image processing system.
[0032] The transmission timing suitable for image processing system processing, that is, the image input timing processed by the image processing system, includes: image processing image input channel number (that is, the corresponding camera image output channel number), image grouping (each group contains multiple camera types, the number of different types of cameras and the corresponding camera numbers), the transmission time of each frame of the image in the image group, whether the image data of each frame in the image group is transmitted in packets, the image size of each packet, the image frame interval, and the timing input relationship between multiple image processing image input channels.
[0033] A specific example of an embodiment of the present invention is described below.
[0034] 1. Multi-camera shooting sequence:
[0035] (1) 4 medium-wave infrared cameras and 4 short-wave infrared cameras take multiple frames, and 8 infrared cameras are exposed simultaneously. The exposure time of the medium-wave infrared camera (15ms) is shorter than that of the short-wave infrared camera (20ms). The stepping is 4 steps, and n frames are taken in each step, for a total of 32 n frames of images, each step has a step time;
[0036] (2) The camera shooting timing simulation software needs to generate the camera type, camera number, exposure start time, exposure duration, and image readout time of each camera based on the above (1) information, and generate a text file with each shooting parameter and shooting timing parameter information.
[0037] 2. Image processing image transmission input timing:
[0038] The infrared image is output in the form of interleaving packets from different cameras. After the 8 infrared cameras are exposed simultaneously, a single-channel optical fiber (taking optical fiber channel number C5 as an example) is used to transmit 8 frames of infrared images. The specific transmission method is as follows:
[0039] Due to the different exposure times of medium wave and short wave, two frames of medium wave images are first transmitted in a single channel optical fiber, then two frames of medium wave and two frames of short wave images are transmitted, and then two frames of short wave are transmitted, thus completing the transmission of four frames of medium wave and four frames of short wave images. Figure 2 shown.
[0040] like Figure 3 As shown in the figure, the interleaved transmission time of 2 frames of medium wave infrared is 90ms, the interleaved transmission time of 2 frames of short wave and 2 frames of medium wave is 80ms, and the interleaved transmission time of 2 frames of short wave is 40ms. The concentrated transmission time of infrared image is 90+80+40=210ms, and the interval period is 350ms, that is, the forward transmission time of the transmission frame is 210ms, and the reverse transmission time of the frame is 140ms.
[0041] 3. The process of converting multi-camera shooting sequence to multi-camera output sequence
[0042] (1) 8-frame infrared image interleaved transmission output
[0043] like Figure 4As shown in the figure, the infrared camera images are exposed by 8 cameras at the same time. The medium wave exposure ends first. The exposure time of the medium wave is longer than that of the short wave. After the exposure ends, the image transmission begins. Therefore, it is considered to first interleave two medium wave images (medium wave M1, M2, the first frame), then interleave two medium waves (medium wave M3, M4, the first frame) and two short waves (short wave S1, S2, the first frame) at the same time, a total of 4 images, and finally interleave 2 short wave images (short wave S3, S4, the first frame). These infrared cameras are all on the fiber channel number C5. The transmission time of each camera image is configured in the software. The medium wave infrared cameras M1~M2 are group G1, and the transmission time is 90ms; the medium wave infrared cameras M3~M4 and the short wave infrared cameras S1~S2 are group G2, and the transmission time is 80ms; the short wave infrared cameras S3~S4 are group G3, and the transmission time is 40ms. The start time of image transmission output of each infrared camera = exposure time + exposure time. After sorting the start time sequence of image transmission output of each camera on an optical fiber, the start time of image transmission output of medium wave M1~M4 is consistent, and the start time of image transmission output of short wave S1~S4 is consistent. Then, according to the grouping information and transmission time, the cameras with the same start time of image transmission output on the optical fiber are merged into groups, so the transmission pause time of group G1 and group G2 is 0, and the transmission pause time of group G3 is 140ms. The subsequent second frame, third frame, and last frame of the image are calculated using a method similar to the first frame of the image to start transmission, the number of continuously transmitted image frames, the transmission time, and the transmission pause time. Finally, according to the start time of the first frame of the medium wave M1 image transmission output minus the start time of the first frame of the medium wave M1 camera exposure, the fiber channel C5 image transmission start waiting time is 15ms.
[0044] (2) Draw a timing diagram on each fiber channel based on the image transmission start waiting time, the start transmission time of each group of images (including multiple frames or single frame), the transmission time and the transmission pause time.
[0045] 4. Image data transmission timing verification module based on FPGA
[0046] The FPGA-based image data transmission timing verification module controls the fiber channel, image transmission time, transmission pause time, and image interleaving method according to the parameters of the image transmission timing conversion and control software configuration, combined with the images transmitted in sequence by the front-end multi-camera shooting, changes the transmission rate of the image data and the transmission interval between two frames of images, and outputs a transmission timing suitable for processing by the image processing system.
[0047] The present invention can conveniently simulate the multi-camera shooting timing, realize the conversion from the multi-camera shooting timing to the multi-camera output timing, verify whether the multi-camera shooting timing and the multi-camera image transmission output timing are reasonable, and verify whether the image transmission configuration parameters are reasonable. At the same time, the system can provide assistance for the image data timing design and implementation in the scenario of multi-camera shooting and transmitting massive data, reduce the difficulty of FPGA to realize the control of multi-camera shooting and complete image data transmission functions, and effectively balance the requirements of the output timing of multi-camera massive image data and the input timing of the image processing system.
[0048] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0049] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is primarily selected for readability and instructional purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention.
Claims
1. A verification system for adapting the output timing of multiple cameras to the input timing of an image processing system, characterized in that: include: Camera shooting timing simulation software, image transmission timing conversion and control software and FPGA-based image data transmission timing verification module, wherein the camera shooting timing simulation software is used to generate multi-camera shooting timing, the image transmission timing conversion and control software is used to configure image transmission configuration parameters according to the multi-camera shooting timing, and control the FPGA-based image data transmission timing verification module, and the FPGA-based image data transmission timing verification module outputs a transmission timing suitable for processing by the image processing system according to the image transmission configuration parameters configured by the image transmission timing conversion and control software.
2. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 1, characterized in that: The camera shooting timing simulation software generates camera information according to the camera type, camera quantity, camera arrangement, whether the camera is moving and camera exposure method, including: camera type, camera number, and generates multi-camera shooting timing information for each camera, including: exposure start and end time, exposure duration, and image readout time.
3. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 2, characterized in that: The camera shooting sequence simulation software generates a file with the multi-camera shooting parameters and multi-camera shooting sequence information for each time.
4. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 2, characterized in that: The image transmission timing conversion and control software processes the multi-camera shooting timing information, changes the single-camera output timing, and adjusts the output timing of multiple single cameras to the multi-camera output timing suitable for the input timing of the image processing system.
5. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 4, characterized in that: The processing of image transmission timing conversion and control software includes: 1) First, read the multi-camera shooting timing information generated by the camera shooting timing simulation software; 2) Configure image transmission parameters according to the image input timing information processed by the image processing system; 3) According to the relationship between the camera number and the camera image output channel number in the image transmission parameters, the camera numbers are mapped to the corresponding camera image output channel queues; 4) Calculate the image output start time of each camera based on the camera type, exposure start time, exposure end time, exposure time and image readout time for the data in the image output channel queue of each camera; 5) Sort the data labels in the data queue on each camera image output channel from small to large according to the label information at the start time of image output, and determine the image transmission queue of each camera image output channel; 6) After the image output start time is sorted, the transmission interval of the current image output start time is obtained by subtracting the current image output start time from the next image output start time in the queue. According to the image transmission time configuration parameters of the camera image output channel, it is determined whether the multi-camera shooting timing and the image transmission time of the camera image output channel configured by the FPGA are reasonable. If reasonable, the image transmission time and pause time of the camera image output channel configured by the FPGA are generated; 7) After completing the setting of transmission time and pause time, the image transmission start waiting time under the camera image output channel number is generated according to the start time of the first frame image transmission output minus the corresponding camera exposure start time; 8) Configure the image transmission configuration parameters through the image transmission timing conversion and control software user interface, and finally convert the multi-camera shooting timing information into the following image transmission configuration parameters: image transmission start waiting time of each camera image output channel, start transmission time of each group of images, number of image frames transmitted continuously, transmission time and transmission pause time of each group of images, number of camera images in each group of images and interleaving mode; 9) Draw a timing diagram on each camera image output channel based on the image transmission start waiting time, the start transmission time of each group of images, the transmission time and the transmission pause time of each camera image output channel to verify whether the configured image transmission parameters and multi-camera shooting timing are reasonable.
6. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 5, characterized in that: The image transmission parameters include the camera image output channel number configuration, the mapping relationship between each camera number and the camera image output channel number, the output image transmission time of each camera, the camera group number, whether the image data is transmitted in packet interleaving, and the mapping relationship between the camera number and the interleaving method.
7. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 5, characterized in that: In 6), the judgment method is: if the transmission interval time at the start time of the current image output is greater than the image transmission time of the camera image output channel configured by the FPGA, it is reasonable, otherwise it is unreasonable; if it is unreasonable, it indicates that the image transmission time setting of the camera image output channel configured by the FPGA and the multi-camera shooting timing are unreasonable, and the image transmission time of the camera image output channel configured by the FPGA or the multi-camera shooting timing needs to be modified.
8. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 5, characterized in that: The FPGA-based image data transmission timing verification module controls the camera image output channel, image transmission time, transmission pause time, and image interleaving method according to the image transmission timing conversion and image transmission configuration parameters configured by the control software, combined with the images transmitted in timing by the front-end multi-camera shooting. It changes the transmission rate of the image data and the transmission interval between two frames of images, and outputs a transmission timing suitable for processing by the image processing system.
9. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 5, characterized in that: The transmission timing suitable for image processing system processing, that is, the image input timing processed by the image processing system, includes: image processing image input channel number, image grouping situation, transmission time of each frame image in the image group, whether each frame image data in the image group is transmitted interleaved by packet, the image size of each packet, image frame interval, and the timing input relationship between multiple image processing image input channels, wherein the image processing image input channel number corresponds to the camera image output channel number, and the image grouping situation includes each group containing multiple camera types, the number of cameras of different types and the corresponding camera numbers.
10. The verification system for adapting the output timing of multiple cameras to the input timing of the image processing system according to claim 5, characterized in that: In 8), each group of images includes a single frame image or multiple frame images.