FPGA-based virtual camera experimental system and method

Abstract

The application discloses a kind of virtual camera experimental system and method based on FPGA, the system is by the remote FPGA board card and local computer of interconnection;Remote FPGA board card, for the image data output of local computer sent by receiving, for user to use image data, write FPGA program to process image data, to complete the required experiment;Local computer is used to provide the hardware and software environment necessary for the client program of local computer, and provides the experimental environment required for FPGA development for user;Virtual camera display interface of local computer is used to display the real-time image of virtual camera;Image processing result display interface of local computer is used to display the image after being processed by remote FPGA board card.The application can ensure that the image data collected by all users is consistent, the experimental result obtained is more accurate and reliable, and the waste of experimental equipment resources is avoided.

Description

Technical Field

[0001] This invention relates to the field of FPGA technology, and in particular to an FPGA-based virtual camera experimental system and method. Background Technology

[0002] When using FPGA development boards and cameras as teaching aids in image processing-related courses, users typically program the FPGA chip to capture and process image data from the camera. These experiments are usually conducted offline, with each user equipped with a development board and camera, both of which must be present in the user's vicinity. When the FPGA development board is located remotely (e.g., due to its high cost and deployment in the cloud for shared use), this type of experiment is impossible because the camera is not locally located, and the user cannot control the captured image.

[0003] When conducting experiments using an FPGA development board and a camera, users collect experimental input data via the camera. Since each user uses a separate FPGA development board and camera, the collected image data will be inconsistent, potentially leading to differences in experimental results. This is especially problematic in exam or competition settings, where different camera data sources can result in varying levels of experimental difficulty. Summary of the Invention

[0004] In view of this, the present invention provides a virtual camera experimental system and method based on FPGA to solve the above-mentioned technical problems.

[0005] This invention discloses a virtual camera experimental system based on FPGA, which consists of interconnected remote FPGA boards and a local computer;

[0006] The remote FPGA board is used to output the image data received from the local computer through a virtual camera interface built inside the remote FPGA board, so that the user can use the image data through the virtual camera interface, write FPGA programs to process the image data, and complete the required experiments.

[0007] The local computer is used to provide the hardware and software environment necessary for the client program of the local computer, and to provide the experimental environment required for FPGA development for the user.

[0008] The virtual camera display interface of the local computer is used to display the real-time image of the virtual camera;

[0009] The local computer's image processing result display interface is used to display the image after it has been processed by the remote FPGA board.

[0010] Furthermore, the local computer is used to run a graphical operating system, run the software required for FPGA program development, and debug remote FPGA boards.

[0011] The local computer is equipped with a local camera or a webcam, or has a file system and stores video and image files necessary for the image processing experiment.

[0012] The client program on the local computer is used to acquire image data from a local camera, webcam, local video or image file; and to generate the image data stream required by the virtual camera and send the data stream to the remote FPGA board.

[0013] Furthermore, the remote FPGA board includes a storage module and an FPGA chip;

[0014] The storage module is used to store virtual image data sent by the local computer and to save image data processed by the user's experimental module; the FPGA chip is used for users to conduct image processing experiments.

[0015] The FPGA chip includes a processor, a storage control module, a VCAM module, and a user experiment module; the processor is connected to the storage module through the storage control module; the processor is connected to the user experiment module through the VCAM module; the user experiment module is connected to the storage module through the storage control module; and the processor is connected to the user experiment module.

[0016] Furthermore, the processor is connected to a local computer to receive TCP packets sent by the local computer, decompress the virtual camera image data into bitmap data format, and write the bitmap data into the storage module through the storage control module; wherein, the TCP packet contains virtual camera image data;

[0017] The VCAM module is used to read bitmap data stored in the storage module and output the bitmap data to the user experiment module through the virtual camera interface.

[0018] The user experiment module is used to acquire image data through a virtual camera interface, perform image processing on the acquired image data to obtain image data M, and then write the image data M into the storage module through the storage control module.

[0019] The processor is used to read image data M from the storage module, compress it, and send it to the local computer.

[0020] Furthermore, the processor is specifically used for:

[0021] Write the start address, end address, and completion status of the bitmap data in the storage module into the register of the VCAM module;

[0022] The VCAM module is specifically used for:

[0023] When the storage state in the register of the VCAM module is detected to be in the preset state, the bitmap data between the storage start address and storage end address in the storage module is read through the storage control module, and the bitmap data is output in sequence using the virtual camera.

[0024] Furthermore, the user experiment module is specifically used for:

[0025] Record the start and end addresses of the image data M and write them into the registers in the user experiment module;

[0026] The processor is also specifically used for:

[0027] After detecting that the storage state of the register in the user experiment module is in the preset state, the image data M in the storage module is read according to the storage start address and storage end address of the image data M;

[0028] The processor compresses the image data M into an image file Y and sends it to the local computer for display.

[0029] This invention also discloses an FPGA-based virtual camera experimental method, which includes:

[0030] Step 1: Obtain a frame of image data from the virtual camera's data source;

[0031] Step 2: After receiving the image file, the processor of the remote FPGA board decompresses it into bitmap data; the image file is the compressed image data.

[0032] Step 3: The processor of the remote FPGA board writes the bitmap data into the storage module through the storage control module, and records its storage start address START_ADDR1 and storage end address END_ADDR1;

[0033] Step 4: After the VCAM module detects that the storage completion status of register A3 has changed to the preset state, i.e., the value of register A3 is 0x01, the storage control module reads bitmap data sequentially from the storage start address START_ADDR1 in the storage module, and uses the virtual camera interface to output the read data to the user test module in a timing sequence; when the read data address reaches the storage end address END_ADDR1, reading stops and register A3 is written with the value 0x00.

[0034] Step 5: After the user experiment module obtains the bitmap data, it performs the desired image processing experiment and writes the image data M obtained after image processing into the storage module through the storage control module; and records its storage start address START_ADDR2 and storage end address END_ADDR2.

[0035] Step 6: The user experiment module writes START_ADDR2 and END_ADDR2 into registers B1 and B2 respectively, and writes the value 0x01 into register B3;

[0036] Step 7: After the processor detects that the value of register B3 is in the preset state, it reads the image data M corresponding to the start address START_ADDR2 to the end address END_ADDR2 in the storage module; after the data reading is completed, the processor writes the value 0x00 into register B3 in the user experiment module to indicate that the data reading has been completed.

[0037] Step 8: The processor compresses the image data M into an image file Y and sends it to the local computer for display.

[0038] Furthermore, prior to step 1, the procedure also includes:

[0039] The client program on the local computer establishes a connection with the remote FPGA board;

[0040] Users select the data source for the virtual camera through the client program; the data source is divided into two types: device data source and file data source. Device data source includes local camera and network camera; file data source includes local or network video files and local or network image files.

[0041] After the user selects a data source, the client program opens the device-type data source or the file-type data source.

[0042] Furthermore, in step 3:

[0043] The processor writes the start address START_ADDR1 and end address END_ADDR1 into registers A1 and A2 in the VCAM module, respectively; the processor writes the value 0x01 into register A3 in the VCAM module to indicate that the bitmap data has been stored in the memory module.

[0044] Further, step 8 includes:

[0045] The processor compresses the image data M into an image file Y, and sends the image file Y to the local computer via Ethernet in the form of TCP packets;

[0046] The local computer displays the image file Y on the image processing results display interface.

[0047] Because of the adoption of the above technical solution, the present invention has the following advantages:

[0048] 1. A system and method have been invented that can be deployed in the cloud and can be used on shared FPGA development boards to conduct image processing-related experiments using a variety of image data sources.

[0049] 2. A system and method were invented to solve the problem that FPGA development boards cannot perform camera-related image processing experiments because they lack a camera interface or the camera interface is incompatible with existing cameras. This allows for the reuse of these FPGA development boards, avoiding waste of equipment resources.

[0050] 3. Resolve the issue of inconsistent experimental results caused by inconsistent camera data sources during image processing using FPGA and camera.

[0051] 4. Through the system and method of this invention, users can not only conduct image processing experiments using FPGA, but also remotely conduct experiments related to camera interface timing and AXI interface timing. This is particularly suitable for situations where students are away from the laboratory. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0053] Figure 1 This is a schematic diagram of the framework of a virtual camera experimental system based on FPGA according to an embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further described in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0055] See Figure 1 This invention provides an embodiment of an FPGA-based virtual camera experimental system, which consists of a remote FPGA board and a local computer; the remote FPGA board and the local computer are typically connected via Ethernet. The local computer runs a client program, which includes a virtual camera display interface and an image processing result display interface.

[0056] The primary function of the local computer is to provide the necessary hardware and software environment for client programs and to offer students the experimental environment required for FPGA development. Typically, it should possess the following characteristics:

[0057] 1) Runs a graphical operating system, such as Windows or Linux, and has a network protocol stack, enabling communication with remote FPGA boards via Ethernet.

[0058] 2) Software necessary for running FPGA program development, such as Quartus II, ModelSim, Vivado, etc.

[0059] 3) It has the ability to debug remote FPGA boards. This function can be implemented by referring to the patent publication document (application number: CN202010228285.X).

[0060] 4) Equipped with a local camera or webcam, or with a file system that stores the video and image files necessary for the image processing experiment.

[0061] The main functions of the local client program are:

[0062] 1) Used to acquire image data from local cameras, webcams, local video or image files.

[0063] 2) Used to generate the image data stream required by the virtual camera and send the data stream to a remote FPGA board via Ethernet.

[0064] 3) Used to receive and display image data sent by a remote FPGA board after it has been processed by the user's experimental module.

[0065] The virtual camera display interface is used to display the real-time image from the virtual camera.

[0066] The image processing results display interface is used to display images processed by the remote FPGA board.

[0067] The remote FPGA board is used to receive image data sent by the local computer. At the same time, a virtual camera interface is built inside the FPGA to output image data. Users can use the image data through this interface, write FPGA programs to process the image data, and conduct specific experiments, such as using artificial intelligence models for image reasoning or using algorithms to process images.

[0068] Typically, this remote FPGA board should also possess the following features:

[0069] 1) The remote FPGA board should have an Ethernet interface, and the FPGA chip it carries should be able to run a soft-core processor or have a hard-core processor.

[0070] 2) The remote FPGA board supports remote FPGA debugging function, which can be implemented with reference to patent (application number): CN202010228285.X.

[0071] 3) The internal logic resources of the FPGA carried by the remote FPGA board should be large enough to meet the experimental requirements.

[0072] The storage module is used to store virtual image data sent by the local computer and to save image data processed by the user's experimental module. Random Access Memory (RAM) is typically used, and common RAM models include, but are not limited to, SDRAM, DDR2, DDR3, and DDR4.

[0073] FPGA chips are primarily used by users for image processing experiments. The logic modules running internally include: a processor, a memory control module, a VCAM module, and a user experiment module. The VCAM module and user experiment module are typically written in a hardware description language (VHDL or Verilog HDL).

[0074] A processor typically refers to a processor module embedded within an FPGA, and there are generally two types. One type is a processor built from the programmable logic units of the FPGA, called a soft-core processor, such as the NIOS II processor in the Intel FPGA platform. The other type is a hard-core processor within a heterogeneous FPGA chip, such as the ARM-Cortex A9 processor embedded in an Intel FPGA.

[0075] The processor is used to receive TCP packets sent by the local computer. The TCP packets contain virtual camera image data, which the processor can decompress into a 24-bit bitmap data format, that is, each pixel uses 3 bytes to store the RGB 3 primary color data.

[0076] The processor is used to write bitmap data to the storage module through the storage path.

[0077] The processor is used to write the start address, end address, and completion status of the bitmap data into the registers of the VCAM module via the AXI bus.

[0078] The processor is used to read the registers of the user experiment module via the AXI bus, thereby obtaining the start address, end address, and storage completion status of the image data processed by the user experiment module in the storage module.

[0079] The processor is used to read bitmap data from the storage module after it has been processed by the user experiment module, via the storage path.

[0080] The processor is used to compress the bitmap data obtained after processing by the user experiment module into an image file and send it to the local computer via Ethernet.

[0081] The storage control module has a storage module control IP core, which is used to perform status control and read / write operations on the storage module. The storage control module has three access interfaces: interface 1 is connected to the user experiment module through bus AXI_1, interface 2 is connected to the VCAM module through bus AXI_3, and interface 3 is connected to the processor through the DRAM path.

[0082] The user experiment module, VCAM module, and processor can use address access to read data stored in the storage module through the storage control module.

[0083] The VCAM module is used to read bitmap data stored in the storage module and output the data to the user experiment module through the virtual camera interface. The virtual camera interface control timing usually adopts interface timing such as DVP and CSI.

[0084] The VCAM module is connected to the storage control module via the AXI_3 bus, with the VCAM module acting as the MASTER end and the storage control module acting as the SLAVE end.

[0085] The VCAM module is connected to the processor via the AXI_4 bus, with the processor acting as the MASTER and the VCAM module acting as the SLAVE.

[0086] The VCAM module has three memory locations: registers A1, A2, and A3. The VCAM module and the processor can read and write to these three registers via the AXI bus. Register A1 stores the starting address of the bitmap data within the memory module, register A2 stores the ending address of the bitmap data within the memory module, and register A3 stores the completion status of writing the processed bitmap data into the memory module.

[0087] The user experiment module is written by student users to perform image processing experiments. It acquires image data through a virtual camera interface and processes the acquired image data, performing tasks such as image enhancement, image compression, and image recognition. The processed image data is then written to the storage module via the AXI_1 bus through the storage control module.

[0088] The user experiment module has two AXI interfaces. AXI interface 1 is connected to the processor via the AXI_1 bus, with the user experiment module acting as the MASTER end and the storage control module acting as the SLAVE end. AXI interface 2 is connected to the processor via the AXI_2 bus, with the processor acting as the MASTER end and the user experiment module acting as the SLAVE end.

[0089] The user experiment module has three memory locations: registers B1, B2, and B3. The user experiment module and the processor can read and write to these three registers via the AXI bus. Register B1 stores the starting address of the processed bitmap data within the memory module, register B2 stores the ending address of the processed bitmap data within the memory module, and register B3 stores the completion status of writing the bitmap data into the memory module.

[0090] This invention provides an embodiment of a virtual camera experiment method based on FPGA, the specific implementation steps of which are as follows:

[0091] S1. The client program establishes a TCP Socket connection with the remote FPGA board.

[0092] S2. Student users select the data source for the virtual camera through the client program. The data source is divided into two types: device data source and file data source. Device data source usually includes: local camera, network camera, etc.; file data source includes: local or network video files, local or network image files, etc.

[0093] S3. After a student user selects a data source, the client program opens the device-type data source or the file-type data source.

[0094] S4. The client program obtains a frame of image data from the data source. For example, this function can be implemented using computer vision and machine learning software libraries such as OpenCV.

[0095] S5. The client program compresses the image data into an image file Q.

[0096] S6. The client program displays the compressed image file Q on the virtual camera display interface.

[0097] S7. The client program sends the compressed image file Q to the remote FPGA board via Ethernet in the form of TCP packets.

[0098] S8. After receiving the image file Q, the processor on the remote FPGA board decompresses the image into 24-bit bitmap data, that is, each pixel uses 3 bytes to store the RGB 3 primary color data.

[0099] S9. The processor on the remote FPGA board writes the bitmap data into the storage module through the storage control module, and records its storage start address as START_ADDR1 and storage end address as END_ADDR1.

[0100] S10. The processor on the remote FPGA board writes START_ADDR1 and END_ADDR1 to registers A1 and A2 in the VCAM module via bus AXI_4, and writes the value 0x01 to register A3; where 0x01 represents 1 in hexadecimal format.

[0101] After detecting that register A3 has changed to 0x01, the S11.VCAM module uses the AXI_3 bus to read bitmap data sequentially from address START_ADDR1 via the storage control module, and outputs this data using the timing of the virtual camera. When the data address reaches END_ADDR1, reading stops and register A3 is written with the value 0x00; where 0x00 represents 0 in hexadecimal format.

[0102] S12. After acquiring bitmap data through the virtual camera interface, the user experiment module performs the desired experiments, such as image enhancement, image compression, and image recognition. After image processing, the user experiment module uses the AXI_1 bus to write the processed image data into the storage module via the storage control module. It records the starting address of the storage as START_ADDR2 and the ending address as END_ADDR2.

[0103] S13. The user experiment module writes START_ADDR2 and END_ADDR2 into registers B1 and B2 in the user experiment module, and writes the value 0x01 into register B3.

[0104] S14. After the processor detects that the value of register B3 is 0x01 through the bus AXI_2, it reads the starting address START_ADDR2 and the ending address END_ADDR2 of the processed bitmap data stored in the storage module from registers B1 and B2.

[0105] S15. The processor reads the processed bitmap data starting from the start address START_ADDR2 until it reaches the end address END_ADDR2. After reading, it writes the value 0x00 to register B3 in the user experimental module.

[0106] S16. The processor compresses the processed bitmap data into an image file Y.

[0107] S17. The processor sends the compressed image file Y to the local computer via Ethernet in the form of TCP packets.

[0108] S18. The local computer displays the image file Y on the image processing results display interface.

[0109] Repeat steps S4-S18 to process the image data for the next frame.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A virtual camera experimental system based on FPGA, characterized in that, It consists of interconnected remote FPGA boards and a local computer; The remote FPGA board is used to output the image data received from the local computer through a virtual camera interface built inside the remote FPGA board, so that the user can use the image data through the virtual camera interface, write FPGA programs to process the image data, and complete the required experiments. The local computer is used to provide the hardware and software environment necessary for the client program of the local computer, and to provide the experimental environment required for FPGA development for the user. The virtual camera display interface of the local computer is used to display the real-time image of the virtual camera; The local computer's image processing result display interface is used to display the image after it has been processed by the remote FPGA board. The local computer is used to run a graphical operating system, run the software required for FPGA program development, and debug remote FPGA boards. The local computer is equipped with a local camera or a webcam, or has a file system and stores video and image files necessary for the image processing experiment. The client program on the local computer is used to acquire image data from the local camera, webcam, local video or image files; Used to generate the image data stream required by the virtual camera and send the data stream to a remote FPGA board; The remote FPGA board includes a storage module and an FPGA chip; The storage module is used to store virtual image data sent by the local computer and to save image data processed by the user's experimental module; the FPGA chip is used for users to conduct image processing experiments. The FPGA chip includes a processor, a storage control module, a VCAM module, and a user experiment module; the processor is connected to the storage module through the storage control module; the processor is connected to the user experiment module through the VCAM module; the user experiment module is connected to the storage module through the storage control module; and the processor is connected to the user experiment module. The processor is connected to a local computer and is used to receive TCP packets sent by the local computer, decompress the virtual camera image data into bitmap data format, and write the bitmap data into the storage module through the storage control module; wherein, the TCP packet contains virtual camera image data; The VCAM module is used to read bitmap data stored in the storage module and output the bitmap data to the user experiment module through the virtual camera interface. The user experiment module is used to acquire image data through a virtual camera interface, perform image processing on the acquired image data to obtain image data M, and then write the image data M into the storage module through the storage control module. The processor is used to read image data M from the storage module, compress it, and send it to the local computer.

2. The FPGA-based virtual camera experimental system according to claim 1, characterized in that, The processor is specifically used for: Write the start address, end address, and completion status of the bitmap data in the storage module into the register of the VCAM module; The VCAM module is specifically used for: When the storage state in the register of the VCAM module is detected to be in the preset state, the bitmap data between the storage start address and storage end address in the storage module is read through the storage control module, and the bitmap data is output in sequence using the virtual camera.

3. The FPGA-based virtual camera experimental system according to claim 1, characterized in that, The user experiment module is specifically used for: Record the start and end addresses of the image data M and write them into the registers in the user experiment module; The processor is also specifically used for: After detecting that the storage state of the register in the user experiment module is in the preset state, the image data M in the storage module is read according to the storage start address and storage end address of the image data M; The processor compresses the image data M into an image file Y and sends it to the local computer for display.

4. An experimental method for an FPGA-based virtual camera experimental system, applicable to the FPGA-based virtual camera experimental system described in any one of claims 1-3, characterized in that, include: Step 1: Obtain a frame of image data from the virtual camera's data source; Step 2: After receiving the image file, the processor of the remote FPGA board decompresses it into bitmap data; the image file is the compressed image data. Step 3: The processor of the remote FPGA board writes the bitmap data into the storage module through the storage control module, and records its storage start address START_ADDR1 and storage end address END_ADDR1; Step 4: After the VCAM module detects that the storage completion status of register A3 has changed to the preset state, i.e., the value of register A3 is 0x01, the storage control module reads bitmap data sequentially from the storage start address START_ADDR1 in the storage module, and uses the virtual camera interface to output the read data to the user test module in a timing sequence; when the read data address reaches the storage end address END_ADDR1, reading stops and register A3 is written with the value 0x00. Step 5: After the user experiment module obtains the bitmap data, it performs the desired image processing experiment and writes the image data M obtained after image processing into the storage module through the storage control module; and records its storage start address START_ADDR2 and storage end address END_ADDR2. Step 6: The user experiment module writes START_ADDR2 and END_ADDR2 into registers B1 and B2 respectively, and writes the value 0x01 into register B3; Step 7: After the processor detects that the value of register B3 is in the preset state, it reads the image data M corresponding to the start address START_ADDR2 to the end address END_ADDR2 in the storage module; after the data reading is completed, the processor writes the value 0x00 into register B3 in the user experiment module to indicate that the data reading has been completed. Step 8: The processor compresses the image data M into an image file Y and sends it to the local computer for display.

5. The method according to claim 4, characterized in that, Before step 1, the following are also included: The client program on the local computer establishes a connection with the remote FPGA board; Users select the data source for the virtual camera through the client program; the data source is divided into two types: device data source and file data source. Device data source includes local camera and network camera; file data source includes local or network video files and local or network image files. After the user selects a data source, the client program opens the device-type data source or the file-type data source.

6. The method according to claim 4, characterized in that, In step 3: The processor writes the start address START_ADDR1 and end address END_ADDR1 into registers A1 and A2 in the VCAM module, respectively; the processor writes the value 0x01 into register A3 in the VCAM module to indicate that the bitmap data has been stored in the memory module.

7. The method according to claim 4, characterized in that, Step 8 includes: The processor compresses the image data M into an image file Y, and sends the image file Y to the local computer via Ethernet in the form of TCP packets; The local computer displays the image file Y on the image processing results display interface.

Images