Multipath camera image splicing system and method based on FPGA and ARM

Abstract

The invention discloses a multichannel camera image splicing system and method based on an FPGA and an ARM. The multichannel camera image splicing system comprises a multichannel camera unit, the ARM and the FPGA. The ARM issues configuration parameters to each path of camera unit through FPGA bridging via an IIC bus, and the FPGA comprises an image receiving unit for receiving image data acquired by the camera units and forwarding the image data to the data read-write architecture; the serial port receiving unit is used for receiving an instruction which is issued by the ARM and contains an image splicing strategy and analyzing the instruction to obtain the image splicing strategy; the data read-write framework is respectively connected with the serial port receiving unit and the image receiving unit, comprises read-write links of which the number is completely consistent with that of the camera units, and reads and writes the image data by adopting a ping-pong cache mode according to an image splicing strategy; and the data sending unit is used for sending the image data read by the data read-write architecture to the ARM to complete splicing. According to the invention, the problem that a large-capacity memory needs to be externally hung can be avoided, a storage chip is not needed to store image data, and the cost of the chip is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-channel camera image splicing system based on FPGA and ARM and a method thereof. Background Art

[0002] As an extension of human vision, image acquisition technology is widely used in people's production and life. Among them, the video acquisition system with multiple mobile industry processor interface (MIPI) cameras working together is becoming more and more widely used. Different design schemes in the industry have their own advantages and disadvantages, which will not be explained in detail here. However, the form in which the data collected by multiple cameras is displayed on the display is relatively fixed. There are two main schemes in the industry:

[0003] Solution 1 uses the ARM chip as the control unit of the display system. Since the ARM chip has fewer MIPI interfaces, it can flexibly perform image stitching when a small number of cameras (such as two) are collaboratively controlled. However, when the number of cameras is large, collaborative control of multiple cameras can only be performed by increasing the number of ARM chips. The main design idea is one master with multiple slaves. After the system is powered on, an ARM master chip sends instructions to multiple slave ARM chips through IO, and then each slave ARM chip configures the parameters of each camera through the IIC bus. After each camera obtains the parameter configuration, it passes the data to each ARM slave chip, and then the slave ARM chip processes the data and sends it to the master ARM chip. Finally, the data is integrated and spliced ​​in the master ARM chip.

[0004] Solution 2 is to use FPGA chip as the control unit of the display system. This solution solves the problem of few MIPI interfaces of ARM chips. It directly configures multiple cameras through FPGA to achieve camera data synchronization, and then integrates and splices the data by FPGA.

[0005] However, in the above solution 1, due to the large number of ARM chips, the clocks of each ARM chip are inconsistent, and the PCB wiring lengths are not equal during the design process. It will be impossible to send the camera parameter instructions at the same time, so that the camera data finally received cannot be completely synchronized. Secondly, the use of ARM chips for data integration must have a large-capacity storage chip. Therefore, considering the price of storage chips and multiple ARM chips, the cost of the entire system is relatively high.

[0006] Although Solution 2 perfectly solves the problem of camera data synchronization, if ARM is not used to configure the camera parameters, but only FPGA is used to configure the camera, because FPGA is a hardware design, its configuration parameters are fixed and cannot be as flexible and variable as ARM. This results in only being able to configure fixed parameters when configuring camera parameters. If you want to modify the parameters to debug some image algorithms, you can only modify the internal code of FPGA. The process of FPGA code modification and compilation is extremely cumbersome, time-consuming and inconvenient. In order to achieve flexible and adjustable parameters, many manufacturers choose FPGA chips with ARM cores, but this chip is relatively expensive and has no cost advantage over Solution 1. Summary of the invention

[0007] Purpose of the invention: In view of the above-mentioned deficiencies, the present invention proposes a multi-channel camera image stitching system and method based on FPGA and ARM, which can achieve the effect of multi-camera data synchronization and multiple stitching methods, and has better portability and scalability.

[0008] Technical solution:

[0009] The present invention provides a multi-channel camera image stitching system based on FPGA and ARM, comprising a multi-channel camera unit, ARM and FPGA; an IIC bridge unit is arranged on the FPGA, the ARM is connected to the IIC bridge unit through an IIC bus, the IIC bridge unit is simultaneously connected to each camera unit through the IIC bus, and the device address of each camera unit is the same;

[0010] ARM sends configuration parameters to each camera unit through the IIC bus via the IIC bridge unit. After the configuration is completed, the camera unit performs image acquisition;

[0011] Among them, FPGA includes:

[0012] An image receiving unit is provided corresponding to the camera unit and is connected to a camera unit respectively, and is used to receive image data collected by the camera unit and forward it to the data reading and writing architecture;

[0013] The serial port receiving unit is connected to the ARM and is used to receive the serial port instruction including the image stitching strategy issued by the ARM and parse and obtain the image stitching strategy therein;

[0014] A data read-write architecture, connected to the serial port receiving unit and the image receiving unit, respectively, including a number of read-write links exactly the same as that of the camera unit, for reading and writing image data in a ping-pong cache manner according to an image stitching strategy;

[0015] The data sending unit is used to send the image data read by the data reading and writing architecture to the ARM to complete the stitching.

[0016] Specifically, the IIC bridge unit leads out several SCL clock buses and SDA data buses respectively, so as to be connected to each camera unit at the same time through the IIC bus;

[0017] The ARM only sends the configuration parameters of the camera unit for the device address, and the IIC bridge unit forwards the configuration parameters of the camera unit to each camera unit respectively.

[0018] Specifically, the data read-write architecture includes a data write unit, a data storage unit and a data read unit connected in sequence, and the ping-pong cache method is as follows:

[0019] Dividing the storage area in the data storage unit into a plurality of areas, the number of the areas being twice that of the camera unit;

[0020] Each read-write link corresponds to two areas respectively. When the data writing unit receives the image data collected by a camera unit of a certain channel forwarded by the image receiving unit, starting from the first row of the image data, the image data of adjacent rows are stored in the two areas corresponding to the read-write link respectively. After a complete row of image data is written in a certain area of ​​the read-write link, the data reading unit reads the image data stored in the area according to the image stitching strategy, and at the same time, another area of ​​the read-write link starts to write the next row of image data.

[0021] More specifically, the data reading unit reads the image data stored in the area according to the image stitching strategy, specifically:

[0022] According to the image stitching strategy analyzed by the serial port receiving unit, the column range of the image data read by the data reading unit is determined, thereby reading the row data in the corresponding column range stored in the corresponding area of ​​the data storage unit.

[0023] Furthermore, it also includes a splicing control unit, which is connected to the serial port receiving unit and the data reading unit respectively, and determines the column range of the image data read by the data reading unit according to the image splicing strategy analyzed by the serial port receiving unit.

[0024] Furthermore, the splicing control unit monitors the image data read by the data reading unit in real time, and determines whether the image data corresponding to each channel is erroneous. If the image data corresponding to a certain channel is erroneous, the image data corresponding to the channel is directly discarded, and the corresponding storage area is directly filled with zeros.

[0025] Specifically, the ARM queries parameters of each camera unit through the IIC bus via the IIC bridge unit.

[0026] Specifically, the serial port instruction sent by the ARM to the serial port receiving unit also includes an image display instruction or an image data query instruction;

[0027] The image display instruction includes single-channel camera image display or multi-channel camera image display.

[0028] The present invention also provides a multi-channel camera image stitching method using the aforementioned multi-channel camera image stitching system, comprising the steps of:

[0029] S1, ARM sends configuration parameters to each camera unit through the IIC bus via the IIC bridge unit, and each camera unit performs parameter configuration accordingly;

[0030] ARM sends a serial port instruction containing an image stitching strategy to the serial port receiving unit, and the serial port receiving unit parses and obtains the image stitching strategy;

[0031] S2, after the configuration is completed, each camera unit collects images and transmits them to the image receiving unit;

[0032] S3, the data read and write architecture adopts a ping-pong cache method to read and write image data according to the image stitching strategy obtained in S1;

[0033] S4. The data sending unit sends the image data read by the data reading unit to the ARM to complete the stitching.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention stores data through ping-pong buffering, ensures the integrity of input image data row data, and ensures the consistency of image data row data input and output rates, can achieve the effect of multi-camera data synchronization and multiple splicing methods, and has better portability and scalability. The present invention can avoid the problem of needing an external large-capacity memory, does not require a storage chip to store image data, and greatly reduces the cost of the chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are merely embodiments of the present invention, and ordinary technicians in this field can obtain other drawings based on these drawings without creative work.

[0037] Figure 1 This is an architecture diagram of the multi-channel camera image stitching system based on FPGA and ARM of the present invention;

[0038] Figure 2This is a bridging example diagram of the IIC bridging unit in the present invention;

[0039] Figure 3 Workflow diagram for IIC bridging;

[0040] Figure 4 This is an example diagram of the data reading and writing architecture of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0042] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0043] Reference Figure 1 The multi-channel camera image stitching system based on FPGA and ARM of the present invention comprises multi-channel camera units, ARM and FPGA; wherein, an IIC bridge unit is arranged on the FPGA, ARM is connected to the IIC bridge unit through the IIC bus, the IIC bridge unit is connected to each camera unit through the IIC bus, ARM is used to send configuration parameters of each camera unit, and sends them to each camera unit through the bridge of FPGA; the camera unit performs image acquisition after the configuration is completed; FPGA comprises an image receiving unit, a data writing unit, a data storage unit, a data reading unit, a data sending unit, a stitching control unit and a serial port receiving unit; ARM, the serial port receiving unit and the data writing unit are connected in sequence, ARM sends a serial port instruction containing an image stitching strategy to the serial port receiving unit, the serial port receiving unit parses and obtains the image stitching strategy therein, and at the same time, ARM receives the image data sent by the data sending unit to complete the stitching, and transmits it to the display for use.

[0044] The IIC bus is composed of two buses, namely the SCL clock bus and the SDA data bus. Each device that supports the IIC protocol has a fixed device address, which can be used to identify the device on the IIC bus. The IIC bus connection includes two operation modes: IIC read operation and IIC write operation. However, in the prior art, whether it is a single device connection or a multi-device connection, in the IIC bus, when the SCL clock line is at a high level, the host must upload the device address and write instructions to the SDA data bus. The slave obtains the device address on the SDA data bus and determines that the device address is consistent with its own device address. It will reply with a response message on the SDA data bus to establish a communication channel. After that, the host sends a parameter instruction. After each 8-bit data is written, the host releases the control of the SDA data bus. The slave that has established the communication channel controls the SDA data bus at the current moment. After the slave responds, the control of the SDA data bus is released and controlled by the slave that establishes the communication channel next time; similarly, the read operation is similar. From the above principle, it can be seen that in the prior art, when there are multiple cameras, if the cameras are configured through ARM, since ARM adopts top-down programming, ARM needs to configure the multiple cameras one by one, that is, after configuring the first camera, the second camera will be configured, and so on, resulting in the parameter configuration of the multiple cameras being out of sync, and then the image data output by each camera is also out of sync. In the subsequent image splicing, images at different times will be spliced, which is prohibited in the image splicing design. In addition, in this process, it is necessary to parse and save the configuration parameters transmitted by ARM. The configuration parameters of the camera contain parameter information for hundreds of registers inside the camera, so the amount of information is large and requires large storage resources. At the same time, it is necessary to copy the configuration parameters of each camera, which also requires large logical resources and occupies a large amount of space, greatly increasing the resource usage pressure of the subsequent splicing algorithm.

[0045] In the present invention, an IIC bridge unit is provided on the FPGA. By utilizing the rich interfaces and editable characteristics of the FPGA, the IIC bridge unit can lead out several SCL clock buses and SDA data buses respectively, so that it can be connected to various camera units at the same time through the IIC bus. Figure 2As shown, the device address of each camera unit is the same, therefore, ARM only needs to send the configuration parameters of the camera unit according to the device address. In this process, FPGA only plays a forwarding role through the IIC bridge unit thereon, and forwards the configuration parameters of the camera unit to each camera unit respectively. There is no need to parse and save the configuration parameters transmitted by ARM, and the configuration parameters can be sent to each camera unit at the same time, which can solve the two problems of data synchronization of multiple cameras and flexible parameter configuration in low-cost FPGA, thereby realizing the one-to-many function of IIC instructions. In addition, in the present invention, the SDA data bus only needs one response bit after transmitting each 8 bits of data. After ARM receives the corresponding response, the configuration parameter sending is completed, and ARM's parameter reading operation for each camera is similar.

[0046] In the present invention, the data transmission between ARM and each camera unit only needs to determine whether the current operation is a read operation or a write operation. This is an arbitration mechanism. After determining the arbitration state, it is determined who controls the control right of the SDA data bus under different operations. Different control rights determine the data flow direction on the SDA data bus. Based on this, a bridge state machine is provided in the IIC bridge unit, and its working process is as follows: Figure 3 As shown, including:

[0047] 1) After the system is powered on, the bridge state machine enters state 0 (reset state);

[0048] 2) When the SCL clock bus in the IIC bus is at a high level and the SDA data bus undergoes a falling edge change, the bridge state machine enters state 1 (arbitration state);

[0049] 3) In the arbitration state, the bridge state machine starts the rising edge counter on the SCL clock bus. When the count reaches 8, that is, SCL = 1, it determines whether the SDA data bus is high or low at this time. If it is low (that is, SDA[8] = 0), it is a write state, and the bridge state machine enters state 2 (IIC write state), otherwise it enters state 4 (IIC read state);

[0050] 4) In state 2, the IIC bridge unit will copy the image data on the SDA data bus between ARM and FPGA to the SDA data bus between FPGA and the corresponding camera unit, so as to send the image data from ARM to the corresponding camera unit, and start the counter on the SCL clock bus at the same time. When the counter on the SCL clock bus is equal to 8, that is, after transmitting 8 bits of image data, the IIC bridge unit will copy the data on the SDA data bus between FPGA and the corresponding camera unit to the SDA data bus between ARM and FPAG, so as to realize the response of the corresponding camera unit, indicating that one byte of image data transmission is completed, that is, Scl_cnt=8, and the bridge state machine enters state 3 (IIC write wait state);

[0051] 5) In state 3, the IIC bridge unit will determine whether there is an edge change on the SCL clock bus. If yes, it will continue to enter state 2 (IIC write state) for write operation. Otherwise, when the SCL clock bus is at a high level, whether a rising edge is generated on the SDA data bus, if yes, the system will enter state 0 and wait for the next arbitration;

[0052] 6) In state 4, the IIC bridge unit will copy the image data on the SDA data bus between the FPGA and the corresponding camera unit to the SDA data bus between the ARM and the FPAG to realize image data reading; at the same time, the counter on the SCL clock bus is started. When the counter on the SCL clock bus is equal to 8, that is, after reading one byte of image data, the IIC bridge unit will copy the data on the SDA data bus between the ARM and the FPAG to the SDA data bus between the FPGA and the corresponding camera unit to realize the host response to indicate that one byte of image reading is completed, that is, Scl_cnt=8. At this time, the bridge state machine enters state 5 (IIC read wait state);

[0053] 7) In state 5, the IIC bridge unit will determine whether there are any edge changes on the SCL clock bus. If so, it will continue to enter state 4 (IIC read state) to perform a read operation. Otherwise, when the SCL clock bus is at a high level, a rising edge is generated on the SDA data bus, and the bridge state machine enters state 0 and waits for the next arbitration.

[0054] The present invention uses the IIC bus to make the FPGA only used for data transparent transmission and one-to-many functions in the configuration link of the multi-channel camera unit. Therefore, in essence, the FPGA does not need to parse the data transmitted by the ARM to the multi-channel camera unit through the IIC bus. At present, the design of parsing first and then sending is generally adopted in the market, because the camera unit needs to configure a large number of registers, such a design will greatly consume the internal logic and storage resources of the FPGA. When a low-cost FPGA chip is selected, it is extremely disadvantageous to the subsequent image stitching algorithm.

[0055] In the present invention, the camera unit adopts a MIPI interface camera, and after the multi-channel camera image stitching system of the present invention is powered on, it is configured according to the received configuration parameters, and after the configuration is completed, image acquisition is performed, and the image data is transmitted to the image receiving unit according to the MIPI_CSI-2 protocol.

[0056] Among them, in FPGA:

[0057] An image receiving unit is provided corresponding to the camera unit and is respectively connected to a camera unit, and is used to receive image data collected by the camera unit and forward it to the data writing unit;

[0058] In the present invention, the image receiving unit can directly use the MIPI receiving IP provided by the FPGA manufacturer to realize the function of transferring data from the MIPI protocol to the AXIS protocol, thereby reducing the complexity of the overall system architecture design, thereby reducing R&D time and cost.

[0059] The data writing unit, the data storage unit, the data reading unit, the splicing control unit and the data sending unit are connected in sequence; the splicing control unit is connected to the serial port receiving unit and the data reading unit respectively, and the column range of the image data read by the data reading unit is determined according to the image splicing strategy obtained by the serial port receiving unit; the data writing unit, the data storage unit and the data reading unit constitute a data reading and writing architecture, such as Figure 4As shown, it includes several completely identical read-write links, the number of which corresponds to the camera unit, and since the image data input rate of the camera unit is inconsistent with the output rate after the final image is spliced, in order to solve this problem, a ping-pong cache is added to the data read-write architecture, that is, the present invention divides the storage area in the data storage unit into several areas, and the number of areas is twice that of the camera unit; each read-write link corresponds to two areas respectively. Taking one of the read-write links as an example, when the data writing unit receives the image data collected by a camera unit of a certain channel forwarded by the image receiving unit, starting from the first row of the image data, the image data of adjacent rows are respectively stored in the two areas corresponding to the read-write link, and after writing a complete row of image data in a certain area of ​​the read-write link, the data reading unit reads the row data in the corresponding column range stored in the area, and then the other area of ​​the read-write link starts to write the next row of image data.

[0060] In the present invention, the image stitching strategy is a specific number of stitching, such as one-way, two-way or four-way stitching, and the column range of the image data read by the data reading unit is determined according to the number of ways in the stitching strategy; if it is one-way stitching, one of the image data can be read and output directly, and the specific one can be determined according to actual needs; if it is two-way stitching, the image data of each half row is selected according to the selected road for stitching; if it is four-way stitching, the image data of each quarter row is stitched in turn; through the above design, the relevant design can be flexibly tested and verified on the FPGA according to needs.

[0061] In a specific embodiment of the present invention, four camera units are selected as the input end of the image data, and the resolution of each camera unit is 1920*1080, that is, the row data volume of an image is 1920 points, totaling 1080 rows, then the workflow of the data reading and writing architecture is as follows:

[0062] 1) After the system is powered on, the write flag RAM_SEL in the data write unit is set to 0. At this time, the image data RAM_WR_0 collected by the corresponding camera unit is written to the storage area RAM_0 of the data storage unit through the data write unit. At the same time, the data write unit starts counting. When the count reaches 1919, the write flag RAM_SEL is set to 1, and the counter is cleared. Thereafter, the image data RAM_WR_0 collected by the corresponding camera unit is written to the storage area RAM_1 of the data storage unit through the data write unit. At the same time, the data write unit restarts counting. When the count reaches 1919 again, the write flag RAM_SE L is set to 0 again, and the counter is cleared. Thereafter, the image data collected by the corresponding camera unit is written to RAM_0, and the counting starts again. This cycle is repeated to realize the ping-pong write operation of data;

[0063] The writing principle of the image data RAM_WR_1, RAM_WR_2, and RAM_WR_3 collected by other camera units is the same, and the corresponding written storage areas are RAM_2 and RAM_3, RAM_4 and RAM_5, and RAM_6 and RAM_7.

[0064] 2) When the counter in step 1) reaches 1919, the read flag RD_FLAG in the data reading unit is set to 1 at the same time. At this time, the data reading unit queries the value of the write flag RAM_SEL. If the write flag RAM_SEL is 0, the corresponding read link RAM_RD_0 in the data reading unit reads the image data in the storage area RAM_1 of the data storage unit. If it is equal to 1, the data reading unit reads the image data in the storage area RAM_0 of the data storage unit. In other words, the reading of image data and the writing of image data are mutually exclusive in the storage space of the data storage unit. When the storage area RAM_0 is written with image data, the image data in the storage area RAM_1 can be read. When the storage area RAM_1 is written with image data, the image data in the storage area RAM_0 can be read. This ensures that a row of data is written and read completely without losing data, and it can be read as it is written, avoiding the problem of needing an external large-capacity memory, thereby greatly reducing costs. The reading principles of the reading links RAM_RD_1, RAM_RD_2, and RAM_RD_3 corresponding to the image data RAM_WR_1, RAM_WR_2, and RAM_WR_3 collected by other camera units are the same.

[0065] In the present invention, considering the cost, a large-capacity storage chip is not used. Therefore, the on-chip memory RAM alone cannot store a whole image data of four camera units at the same time. Therefore, when it is necessary to splice four images into one image, the present invention directly processes at the data source end by designing the above-mentioned architecture, that is, the image data is processed row by row from entering the FPGA, and then output row by row according to the corresponding column range, that is, the problem of needing an external large-capacity memory is avoided. At the same time, considering compatibility, if the received row data is directly processed in the data writing unit, since the clock of the data writing unit comes from the external camera, the design level of circuit engineers from different manufacturers is different, and the poor circuit design will make the clock stable and unreliable. If the image data is processed based on an unreliable clock, it will cause image data errors, thereby causing image distortion. Therefore, the present invention adopts the above-mentioned architecture to write a complete row of image data in a certain area in the data reading unit and then outputs it row by row, which can also ensure that the clock source of the data reading unit is an extremely stable clock inside the FPGA.

[0066] In the present invention, the splicing control unit can also perform error correction on the image data read by the data reading unit, specifically:

[0067] The splicing control unit monitors the image data read by the data reading unit in real time, and determines whether the image data corresponding to each channel is wrong, such as a blank image caused by a sudden power outage. If the image data corresponding to a certain channel is wrong, the image data corresponding to that channel is directly discarded, and the corresponding storage area is directly filled with zeros. In this way, the image data corresponding to the wrong channel has a black background to improve the beauty of the image interface.

[0068] The data sending unit sends the image data read by the data reading unit to the ARM to complete the stitching for use by subsequent image algorithms.

[0069] In the present invention, the data sending unit sends the image data read by the data reading unit to the ARM according to the standard MIPI protocol.

[0070] In the present invention, the configuration parameters include frame rate, resolution, light sensing information, data format, etc.

[0071] In the present invention, the serial port instructions issued by ARM also include image display instructions or image data query instructions; wherein, the image display instructions include single-channel camera image display or multi-channel camera image display. In a specific embodiment of the present invention, the camera unit is designed to be four-channel, then the image display instructions may include single-channel camera image display, dual-channel camera image display or four-channel camera image display.

[0072] The present invention also provides a multi-channel camera image stitching method based on the aforementioned multi-channel camera image stitching system, comprising the steps of:

[0073] S1, ARM sends the configuration parameters of each camera unit to each camera unit through the IIC bus via the FPGA bridge, and each camera unit performs parameter configuration accordingly;

[0074] ARM sends a serial port instruction containing an image stitching strategy to the serial port receiving unit, and the serial port receiving unit parses and obtains the image stitching strategy;

[0075] S2, after the configuration is completed, each camera unit collects images and transmits them to the image receiving unit;

[0076] S3, the data read and write architecture adopts a ping-pong cache method to read and write image data according to the image stitching strategy obtained in S1;

[0077] S4. The data sending unit sends the image data read by the data reading unit to the ARM to complete the stitching.

[0078] The present invention sets a one-to-many bridge in the FPGA, so that the multi-channel camera units can be configured in the ARM. The FPGA only plays the role of bridge, thereby solving the problem that the parameter configuration of the camera unit needs to be modified and the compilation of the FPGA is time-consuming. In addition, during the bridging process, the FPGA only recognizes and judges the direction of the IIC bus, and does not need to parse the data therein. The corresponding operation can be completed using a low-cost FPGA chip, which greatly reduces the cost.

[0079] The present invention stores data through ping-pong buffering to ensure the integrity of the input image data row data, and at the same time ensure the consistency of the input and output rates of the image data row data, and then reads the corresponding column range in the image data row data collected by each camera unit according to the splicing strategy, and outputs the read data to complete the image splicing. During the reading, the output is uniformly performed row by row, which can ensure that the clock source of the data reading is an extremely stable clock inside the FPGA. The present invention processes the image data row from the moment it enters the FPGA, and directly outputs it row by row after processing, so as to avoid the problem of needing an external large-capacity memory, and does not require a storage chip to store the image data, which greatly reduces the cost of the chip.

[0080] The present invention can subsequently add corresponding stitching methods in the serial port instructions, and can also expand the image stitching method by modifying the data reading method of the data reading unit, thereby greatly increasing the scalability of the system of the present invention.

[0081] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention (including the claims) is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0082] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-channel camera image stitching system based on FPGA and ARM, characterized in that: It includes a multi-channel camera unit, an ARM and an FPGA; an IIC bridge unit is arranged on the FPGA, the ARM is connected to the IIC bridge unit through an IIC bus, the IIC bridge unit is simultaneously connected to each camera unit through the IIC bus, and the device address of each camera unit is the same; ARM sends configuration parameters to each camera unit through the IIC bus via the IIC bridge unit. After the configuration is completed, the camera unit performs image acquisition; Among them, FPGA includes: An image receiving unit is provided corresponding to the camera unit and is connected to a camera unit respectively, and is used to receive image data collected by the camera unit and forward it to the data reading and writing architecture; The serial port receiving unit is connected to the ARM and is used to receive the serial port instruction including the image stitching strategy issued by the ARM and parse and obtain the image stitching strategy therein; A data read-write architecture, connected to the serial port receiving unit and the image receiving unit, respectively, including a number of read-write links exactly the same as that of the camera unit, for reading and writing image data in a ping-pong cache manner according to an image stitching strategy; The data sending unit is used to send the image data read by the data reading and writing architecture to the ARM to complete the stitching.

2. The multi-camera image stitching system according to claim 1, characterized in that: The IIC bridge unit leads out several SCL clock buses and SDA data buses respectively, so as to be connected to each camera unit at the same time through the IIC bus; The ARM only sends the configuration parameters of the camera unit for the device address, and the IIC bridge unit forwards the configuration parameters of the camera unit to each camera unit respectively.

3. The multi-camera image stitching system according to claim 1, characterized in that: The data read-write architecture includes a data write unit, a data storage unit and a data read unit connected in sequence, and the ping-pong cache method is as follows: Dividing the storage area in the data storage unit into a plurality of areas, the number of the areas being twice that of the camera unit; Each read-write link corresponds to two areas respectively. When the data writing unit receives the image data collected by a camera unit of a certain channel forwarded by the image receiving unit, starting from the first row of the image data, the image data of adjacent rows are stored in the two areas corresponding to the read-write link respectively. After a complete row of image data is written in a certain area of ​​the read-write link, the data reading unit reads the image data stored in the area according to the image stitching strategy, and at the same time, another area of ​​the read-write link starts to write the next row of image data.

4. The multi-camera image stitching system according to claim 3, characterized in that: The data reading unit reads the image data stored in the area according to the image stitching strategy, specifically: According to the image stitching strategy analyzed by the serial port receiving unit, the column range of the image data read by the data reading unit is determined, thereby reading the row data in the corresponding column range stored in the corresponding area of ​​the data storage unit.

5. The multi-camera image stitching system according to claim 4, characterized in that: It also includes a splicing control unit, which is connected to the serial port receiving unit and the data reading unit respectively, and determines the column range of the image data read by the data reading unit according to the image splicing strategy analyzed by the serial port receiving unit.

6. The multi-camera image stitching system according to claim 5, characterized in that: The splicing control unit monitors the image data read by the data reading unit in real time, and determines whether the image data corresponding to each channel is wrong. If the image data corresponding to a certain channel is wrong, the image data corresponding to the channel is directly discarded, and the corresponding storage area is directly filled with zeros.

7. The multi-camera image stitching system according to claim 1, characterized in that: The ARM queries parameters of each camera unit through the IIC bus via the IIC bridge unit.

8. The multi-camera image stitching system according to claim 1, characterized in that: The serial port instructions sent by the ARM to the serial port receiving unit also include image display instructions or image data query instructions; The image display instruction includes single-channel camera image display or multi-channel camera image display.

9. A method for stitching images from multiple cameras using the multi-channel camera image stitching system according to any one of claims 1 to 8, characterized in that: Includes steps: S1, ARM sends configuration parameters to each camera unit through the IIC bus via the IIC bridge unit, and each camera unit performs parameter configuration accordingly; ARM sends a serial port instruction containing an image stitching strategy to the serial port receiving unit, and the serial port receiving unit parses and obtains the image stitching strategy; S2, after the configuration is completed, each camera unit collects images and transmits them to the image receiving unit; S3, the data read and write architecture adopts a ping-pong cache method to read and write image data according to the image stitching strategy obtained in S1; S4. The data sending unit sends the image data read by the data reading unit to the ARM to complete the stitching.

Images