Simulation image high-speed injection system with dynamically configurable transmission time sequence and operation method of simulation image high-speed injection system

Abstract

The invention provides a transmission time sequence dynamic configurable simulation image high-speed injection system and an operation method thereof, and the method comprises the steps: image transmission time sequence generation and system control software generates a simulation image transmission time sequence, controls NVMe SSD simulation image writing, and manages simulation image information and storage address information; recording a corresponding relation between a simulation image transmission time sequence and image storage information, and controlling the ARM-based transmission time sequence management module to start and stop working and select a task mode; the ARM-based transmission time sequence management module receives a simulation image transmission time sequence file through a network or a serial port for storage and management, and controls the FPGA-based simulation image transmission module to perform image transmission and start and stop; and the simulation image transmission module based on the FPGA receives the control command, reads a simulation image of the NVMe SSD, and outputs the simulation image through an optical fiber according to a transmission time sequence and a simulation image format requirement. According to the invention, a camera image transmission time sequence can be verified and a simulation image injection source can be provided for other equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of camera image transmission, and in particular to a simulation image high-speed injection system with dynamically configurable transmission timing and an operation method thereof. Background Art

[0002] For embedded image data processing systems, video image data with specific target features and background information are required for algorithm verification during algorithm development and verification, such as video images of scenes such as foreign objects at airports, drone flights, and aircraft takeoffs and landings. However, due to a series of limitations such as objective conditions, the above video image data is difficult to obtain at any time, especially for image data with a specific resolution. At the same time, the image data needs to be reused continuously to verify the processing algorithm. If real images are obtained using cameras through field experiments, they will be limited by many factors such as high verification costs, high risks, long test cycles, and weather. If the image data of the expected conditions is obtained by simulating the acquired target scene image data, and injecting it according to the data format required by the embedded image processing system, it can effectively speed up the processing algorithm development and verification process, shorten the R&D cycle, reduce R&D costs, reduce test risks, and provide an analytical basis for performance evaluation and improvement of image processing.

[0003] The high-speed simulated image injection system can provide simulated image data to the embedded image data processing system when the real image data source does not meet the working conditions by simulating the transmission protocol and data transmission timing of the real image source. On the one hand, it can provide image data verification processing algorithms, and on the other hand, it can verify the image data interface protocol of the embedded processing system. The bus rate of the traditional USB-based simulation injection system is 60MB / s, but due to factors such as protocol overhead and data transmission process priority, the actual rate is about 20MB / s, which cannot meet the application scenarios of multi-channel, high frame rate, and large target image simulation injection. Summary of the invention

[0004] The purpose of the present invention is to provide a high-speed simulation image injection system with dynamically configurable transmission timing and an operation method thereof, which can simulate and generate image data transmission timing and transmit the simulation image according to the transmission timing.

[0005] The object of the present invention is to achieve the following technical solutions:

[0006] A high-speed simulation image injection system with dynamically configurable transmission timing includes image transmission timing generation and system control software, an ARM-based transmission timing management module and an FPGA-based simulation image transmission module, wherein:

[0007] Image transmission timing generation and system control software generates simulation image transmission timing, controls NVMe SSD simulation image writing, manages simulation image information and storage address information, records the correspondence between simulation image transmission timing and image storage information, and controls the start and stop of the ARM-based transmission timing management module and the selection of task mode;

[0008] The ARM-based transmission timing management module receives the simulation image transmission timing file through the network or serial port. The ARM-based transmission timing management module stores the received simulation image transmission timing file and generates a task mode list. The image transmission timing generation and system control software can select the task mode through the network or serial port.

[0009] The FPGA-based simulation image transmission module receives control commands to inject simulation images.

[0010] An operating method of a high-speed simulation image injection system with dynamically configurable transmission timing is applicable to the high-speed simulation image injection system with dynamically configurable transmission timing, and comprises the following steps:

[0011] Step 1: Set the image type, image transmission channel, and transmission time through the image transmission timing generation and system control software to generate a simulation image transmission timing file;

[0012] Step 2: The image transmission timing generation and system control software generates a task mode according to the simulation image source and NVMe SSD storage information, writes the task mode and the simulation image into the NVMe SSD, and writes the simulation image storage location information in the NVMe SSD into the simulation image transmission timing file corresponding to the task mode; if the NVMe SSD has a corresponding simulation image, only the simulation image storage location information in the NVMe SSD matching the task mode is updated to the simulation image transmission timing file;

[0013] Step 3, the image transmission timing generation and system control software sends the simulation image transmission timing file to the ARM-based transmission timing management module through the network or serial port;

[0014] Step 4: Plug and unplug the SSD and install it into the FPGA-based simulation image transmission module;

[0015] Step 5: Power on the FPGA-based simulation image transmission module and the ARM-based transmission timing management module, select the task mode through the image transmission timing generation and system control software, and send a start transmission instruction to the ARM-based transmission timing management module;

[0016] Step 6: The ARM-based transmission timing management module generates a control command according to the transmission timing file information and the status information fed back by the FPGA-based simulation image transmission module; the FPGA-based simulation image transmission module receives the control command and reads the NVMe SSD simulation image, and outputs the simulation image through the fiber channel;

[0017] Step 7: After the simulation image transmission is completed, the ARM-based transmission timing management module waits for the control instruction, and the FPGA-based simulation image transmission module stops outputting the simulation image.

[0018] Compared with the prior art, the present invention has the following significant beneficial effects:

[0019] 1. The system is highly flexible and can be modified according to the data interface protocol of the embedded image processing system to achieve the adaptation of the data interface protocol;

[0020] 2. The system can achieve configurable transmission rate and transmission mode. In practical applications, the maximum simulation image transmission rate is better than 2GB / s;

[0021] 3. The system can realize the simultaneous transmission of multiple optical fiber images, and the timing within each image and between each image can be controlled, with the timing control error less than 1 microsecond;

[0022] 4. The system can simulate high-speed continuous output of simulation images. The storage capacity of simulation images is at the TB level, and the image resolution size is not limited, meeting the requirements of high-speed data transmission of the simulation injection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The system structure diagram of a high-speed simulation image injection system with dynamically configurable transmission timing provided by an embodiment of the present invention.

[0024] Figure 2 The present invention provides a flowchart of an operation method of a high-speed simulation image injection system with dynamically configurable transmission timing. DETAILED DESCRIPTION

[0025] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms, and the present disclosure should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0026] Figure 1 1 is a system structure diagram of a high-speed simulation image injection system with dynamically configurable transmission timing provided by an embodiment of the present invention. Figure 1 As shown, the system includes image transmission timing generation and system control software (located in Figure 1 The upper computer), the ARM-based transmission timing management module and the FPGA-based simulation image transmission module. The following introduces each module one by one.

[0027] First, the image transmission timing generation and system control software are introduced.

[0028] The image transmission timing generation and system control software generates the simulated image transmission timing, controls the writing of NVMe SSD simulated images, manages the simulated image information and storage address information, records the correspondence between the simulated image transmission timing and the image storage information, and controls the start and stop of the ARM-based transmission timing management module, task mode selection, etc.

[0029] Among them, the image transmission timing generation and system control software generate the simulated image transmission timing including: the image transmission timing generation and system control software generates the image transmission start waiting time for each optical fiber channel, the start transmission time of each group of images (including multiple frames or single frame), the number of image frames transmitted continuously, the transmission time and transmission pause time of each group of images, the number of camera images of each group of images and the interleaving method and other information.

[0030] The image transmission timing generation and system control software controls the writing of NVMe SSD simulation images and manages the simulation image information and storage address information, including: when generating the image transmission timing, the image transmission timing generation and system control software combines the simulation image information (image bit width, image size), writes the simulation image into the NVMe SSD, generates the storage address information of the simulation image in the SSD, and records the simulation image information and storage address information.

[0031] Among them, when the image transmission timing generation and system control software uses the existing task mode to generate a new simulated image transmission timing, it no longer writes data to the SSD repeatedly. The image transmission timing generation and system control software can simultaneously generate multiple fiber channel transmission timings and support multiple camera types. The transmission time of each type of camera can be configured.

[0032] The following describes the ARM-based transmission timing management module.

[0033] The ARM-based transmission timing management module receives the simulation image transmission timing information through the network or serial port. The ARM-based transmission timing management module stores the received simulation image transmission timing and generates a task mode. The image transmission timing generation and system control software can select the task mode through the network or serial port.

[0034] like Figure 1As shown, the ARM-based transmission timing management module may include: eMMC (embedded multimedia card), NorFlash (non-volatile flash memory), ARM (advanced reduced instruction set computer, Yulong810A chip is selected in this embodiment), DDR3 (third-generation double-rate synchronous dynamic random access memory), UART (universal asynchronous receiver and transmitter), and network, wherein UART is used for image transmission timing generation and data communication between system control software and the ARM-based transmission timing management module, the network is used for image transmission timing generation and data communication between system control software and the ARM-based transmission timing management module, eMMC is used to store system software and simulated image transmission timing files of the ARM-based transmission timing management module, Nor Flash is used to store application software of the ARM-based transmission timing management module, ARM is used to run system software and application software of the ARM-based transmission timing management module, DDR3 is used to cache intermediate data generated during the operation of system software and application software, Figure 1 The QSPI shown in the figure refers to the interface for communication between ARM and Nor Flash, SDIO refers to the interface for communication between ARM and eMMC, PCIe refers to the interface for communication between ARM and FPGA, and DDR is the interface for communication between ARM and DDR3.

[0035] The ARM-based transmission timing management module generates control commands and status information for interaction between ARM and FPGA according to the relationship list between the simulation image transmission timing and the image storage information. The control command content includes the number of cameras, transmission time, pause time, fiber channel number, camera number, camera type, image resolution, image storage address in NVMe SSD, DDR address parameters. The status information includes the number of remaining control commands to be executed for each fiber channel, status information of the simulation image stored in NVMe SSD, etc.

[0036] The following describes the FPGA-based simulation image transmission module.

[0037] The FPGA-based simulation image transmission module receives control commands to inject simulation images. The module includes large-capacity DDR3, NVMe SSD, FPGA and optical module. The DDR3 module accesses data read and write through the AXI bus. The simulation image stored in the NVMeSSD is cached in the DDR3 through DMA (direct memory access). The FPGA is used to run the program of the FPGA-based simulation image transmission module to drive and control the NVMe SSD and DDR3 to realize the reading and caching of simulation images, to communicate data with the ARM-based transmission timing management module, and to transmit the simulation image through optical fiber. The optical module is used to convert electrical signals into optical signals to realize long-distance data transmission. The interface of the NVMe SSD is the M.2 interface. Figure 1 The GPIO in the figure refers to the interface for communication between FPGA and DDR3, and GTH is the interface for communication between FPGA and optical module.

[0038] NVMe SSD has the characteristics of large storage capacity, high read and write speed, and low power consumption. At the PCIe3.0 rate, the theoretical rate of SSD can reach 4GB / s, which meets the requirements of being a high-speed simulation image injection data source. FPGA reads SSD simulation images and uses optical fiber for data transmission, which meets the requirements of high-speed data transmission of the simulation injection system. At the same time, FPGA has a high degree of flexibility and can be modified accordingly according to the data interface protocol of the embedded image processing system to achieve the adaptation of the data interface protocol.

[0039] Each fiber channel has an AXI bus interface for data access to DDR3. All fiber channels are interconnected with DDR3 through AXI Interconnect IP for data exchange. The single-channel fiber simulation image transmission channel reads the image from DDR3 and organizes the custom protocol data format. It transmits data by packet. It can control the direct transmission of a single image or the interleaved transmission of multiple images, and control the transmission rate of the simulation image and the transmission interval between two frames of images. A single fiber channel supports a maximum of 4 interleaved and simultaneous transmission of simulation images. The transmission link rate of a single fiber channel is 6.25Gbps, and the transmission protocol adopts the Aurora64B66B protocol. It supports a maximum of 5 fiber simulation images to be transmitted simultaneously.

[0040] In one embodiment, after the image transmission timing generation and system control software completes the storage of the simulated image, it is plugged and installed into the FPGA-based simulated image transmission module as a simulated image source for the high-speed injection system of the simulated image.

[0041] In one embodiment, the FPGA-based simulation image transmission module and the ARM-based transmission timing management module are connected through a flexible PCB, which can realize the stacking of modules, and the PCIe interface between the two can realize high-speed data communication, wherein the FPGA uses the Xilinx xc7vx690tffg1761 chip, and the ARM uses the Yulong810A chip, which integrates a Cortex-A9 processor. The host computer image transmission timing generation and system control software performs telemetry and remote control on the ARM-based transmission timing management module through the network or serial port.

[0042] Figure 2 The flowchart is a method for operating a high-speed simulation image injection system with dynamically configurable transmission timing. Figure 2 As shown, the method comprises the following steps:

[0043] Step 1: Set the image type, image transmission channel, and transmission time through the image transmission timing generation and system control software to generate a simulation image transmission timing file;

[0044] Step 2: The image transmission timing generation and system control software generates a task mode according to the simulation image source and NVMe SSD storage information, writes the task mode and the simulation image into the NVMe SSD, and writes the simulation image storage location information in the NVMe SSD into the simulation image transmission timing file corresponding to the task mode; if the NVMe SSD has a corresponding simulation image, only the simulation image storage location information in the NVMe SSD matching the task mode is updated to the simulation image transmission timing file;

[0045] Step 3, the image transmission timing generation and system control software sends the simulation image transmission timing file to the ARM-based transmission timing management module through the network or serial port;

[0046] Step 4: Plug and unplug the NVMe SSD and install it into the FPGA-based simulation image transmission module;

[0047] Step 5: Power on the FPGA-based simulation image transmission module and the ARM-based transmission timing management module, select the task mode through the image transmission timing generation and system control software, and send a start transmission instruction to the ARM-based transmission timing management module;

[0048] Step 6: The ARM-based transmission timing management module generates a control command according to the transmission timing file information and the status information fed back by the FPGA-based simulation image transmission module; the FPGA-based simulation image transmission module receives the control command and reads the NVMe SSD simulation image, and outputs the simulation image through the fiber channel;

[0049] Step 7: After the simulation image transmission is completed, the ARM-based transmission timing management module waits for the control instruction, and the FPGA-based simulation image transmission module stops outputting the simulation image.

[0050] When the ARM-based transmission timing management module receives the control command from the host computer, it will call the corresponding simulation image transmission timing file, and generate a control command for each group of images (including 1 to 4 images) according to the transmission protocol between the FPGA-based simulation image transmission module and the ARM-based transmission timing management module. The control command is sent to the FPGA-based simulation image transmission module, and the FPGA-based simulation image transmission module performs simulation image transmission according to the control command. The FPGA-based simulation image transmission module feedbacks the number of unexecuted control commands for each optical fiber channel, and the ARM-based transmission timing management module will determine whether to continue sending control commands based on the number of unexecuted control commands.

[0051] The FPGA-based simulation image transmission module receives and parses control commands, and caches them into the corresponding command queues according to the transmission channel number identified by the control command. The number of control commands increases by 1 for each control command received, and decreases by 1 for each control command executed. The command cache queue of a single transmission channel supports caching up to 20 control commands. The FPGA-based simulation image transmission module reads the simulation image from the corresponding address of the SSD according to the control command and transfers it to DDR3 in DMA mode. The fiber channel reads the simulation image from the corresponding address interval of DDR3 through the AXI bus and adds a custom fiber transmission protocol, which is then output through the optical fiber. The FPGA-based simulation image transmission module controls the transmission rate of the simulation image by controlling the transmission pause time between two data packets.

Claims

1. A high-speed simulation image injection system with dynamically configurable transmission timing, characterized in that: It includes image transmission timing generation and system control software, ARM-based transmission timing management module and FPGA-based simulation image transmission module, among which: Image transmission timing generation and system control software generates simulation image transmission timing, controls NVMe SSD simulation image writing, manages simulation image information and storage address information, records the correspondence between simulation image transmission timing and image storage information, and controls the start and stop of the ARM-based transmission timing management module and the selection of task mode; The ARM-based transmission timing management module receives the simulation image transmission timing file through the network or serial port. The ARM-based transmission timing management module stores the received simulation image transmission timing file and generates a task mode list. The image transmission timing generation and system control software can select the task mode through the network or serial port. The FPGA-based simulation image transmission module receives the control command of the ARM-based transmission timing management module to perform simulation image injection.

2. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 1 is characterized in that: The image transmission timing generation and system control software generate the simulated image transmission timing including: the image transmission timing generation and system control software generate the image transmission start waiting time for each optical fiber channel, the start transmission time of each group of images, the number of image frames transmitted continuously, the transmission time and transmission pause time of each group of images, the number of camera images of each group of images and the interleaving method.

3. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 2 is characterized in that: The image transmission timing generation and system control software controls the writing of the NVMe SSD simulation image and manages the simulation image information and storage address information, including: when the image transmission timing generation and system control software generates the image transmission timing, it combines the simulation image information, writes the simulation image into the NVMe SSD, generates the storage address information of the simulation image in the NVMe SSD, and records the simulation image information and the storage address information; When the image transmission timing generation and system control software uses the existing task mode to generate a new simulation image transmission timing, the simulation image is no longer repeatedly written to the NVMe SSD; The image transmission timing generation and system control software can simultaneously generate optical fiber multi-channel transmission timing, support multiple camera types, and the transmission time of each type of camera is configurable.

4. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 3 is characterized in that: The ARM-based transmission timing management module generates control commands and status information for interaction between ARM and FPGA according to the relationship list between the simulated image transmission timing and the image storage information. The control command content includes the number of cameras, transmission time, pause time, fiber channel number, camera number, camera type, image resolution, image storage address in NVMe SSD, DDR address. The status information includes the number of remaining control commands to be executed for each fiber channel and the status information of the simulated image stored in the SSD.

5. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 1 is characterized in that: The FPGA-based simulation image transmission module includes a large-capacity DDR3, NVMe SSD, FPGA and optical module, wherein the large-capacity DDR3 is accessed for data reading and writing through the AXI bus, and the simulation image stored in the NVMe SSD is cached in the DDR3 through DMA. The FPGA is used to run the control program of the FPGA-based simulation image transmission module to drive and control the NVMe SSD and DDR3 to realize the reading and caching of the simulation image, to communicate data with the ARM-based transmission timing management module, and to transmit the simulation image through optical fiber. The optical module is used to convert electrical signals into optical signals to realize long-distance transmission of data.

6. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 4 is characterized in that: Each fiber channel has an AXI bus interface for data access to DDR3. All fiber channels interact with DDR3 through AXIInterconnect IP. A single fiber channel reads the simulation image from DDR3 and organizes the custom protocol data format to transmit data in packets. A single fiber channel supports up to 4 simultaneous interleaved transmissions of simulation images. The transmission link rate of a single fiber channel is 6.25 Gbps. The transmission protocol adopts Aurora64B66B protocol, which supports up to 5 simultaneous transmissions of simulation images from fibers.

7. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 5 is characterized in that: The NVMe SSD interface is an M.2 interface. After the image transmission timing generation and system control software complete the simulation image storage, it is plugged and installed into the FPGA-based simulation image transmission module as a simulation image source for the high-speed injection system of simulation images.

8. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 5, characterized in that: The FPGA-based simulation image transmission module and the ARM-based transmission timing management module are connected through a flexible PCB to achieve stacking of modules, and a PCIe interface is used between the two to achieve high-speed data communication.

9. The high-speed simulation image injection system with dynamically configurable transmission timing according to claim 5, characterized in that: The FPGA uses the Xilinx xc7vx690tffg1761 chip, and the ARM uses the Yulong810A chip, which integrates the Cortex-A9 processor.

10. An operating method of a high-speed simulation image injection system with dynamically configurable transmission timing, applicable to the high-speed simulation image injection system with dynamically configurable transmission timing according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: Set the image type, image transmission channel, and transmission time through the image transmission timing generation and system control software to generate a simulation image transmission timing file; Step 2: The image transmission timing generation and system control software generates a task mode according to the simulation image source and NVMe SSD storage information, writes the task mode and the simulation image into the NVMe SSD, and writes the simulation image storage location information in the NVMe SSD into the simulation image transmission timing file corresponding to the task mode; If the NVMe SSD has a corresponding simulation image, only the simulation image storage location information in the NVMe SSD matching the task mode is updated to the simulation image transmission timing file; Step 3, the image transmission timing generation and system control software sends the simulation image transmission timing file to the ARM-based transmission timing management module through the network or serial port; Step 4: Plug and unplug the NVMe SSD and install it into the FPGA-based simulation image transmission module; Step 5: Power on the FPGA-based simulation image transmission module and the ARM-based transmission timing management module, select the task mode through the image transmission timing generation and system control software, and send a start transmission instruction to the ARM-based transmission timing management module; Step 6: The ARM-based transmission timing management module generates a control command according to the transmission timing file information and the status information fed back by the FPGA-based simulation image transmission module; the FPGA-based simulation image transmission module receives the control command and reads the NVMe SSD simulation image, and outputs the simulation image through the fiber channel; Step 7: After the simulation image transmission is completed, the ARM-based transmission timing management module waits for the control instruction, and the FPGA-based simulation image transmission module stops outputting the simulation image.

Images