An industrial camera and image processing method

Abstract

The embodiment of the present application provides an industrial camera and an image processing method, wherein the industrial camera comprises: an FPGA chip, an image sensor and an image signal processing (ISP) chip; the image sensor is used for collecting image data and sending the image data to the ISP chip; the ISP chip is used for performing wide dynamic range (WDR) processing on the image data after receiving the image data sent by the image sensor; sending the WDR-processed image data to the FPGA chip; and the FPGA chip is used for performing secondary processing on the WDR-processed image data after receiving the WDR-processed image data sent by the ISP chip, to obtain an image to be output, so that WDR processing can be directly performed on the collected image on the camera.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to an industrial camera and an image processing method. Background Technology

[0002] Industrial cameras are a key component of machine vision systems, their most fundamental function being the conversion of light signals into ordered electrical signals. Industrial cameras offer high image stability, fast data transmission speeds, and strong anti-interference capabilities, and are widely used in automated production scenarios for workpiece identification, inspection, and visual measurement.

[0003] In related technologies, industrial cameras implemented using a single FPGA (Field Programmable Gate Array) chip architecture utilize the FPGA chip to process image data acquired by the image sensor to obtain the output image. While FPGA chips can perform functions such as correcting inherent sensor defects, processing simple image defects, image enhancement, and color image processing, they are limited in implementing WDR (Wide Dynamic Range) functionality. This necessitates that WDR implementation be performed on a PC (Personal Computer), which consumes both CPU resources and memory bandwidth. Furthermore, if CPU processing is not timely, incomplete or misaligned images can easily occur. Summary of the Invention

[0004] The purpose of this application is to provide an industrial camera and an image processing method for directly performing WDR processing on images on the camera. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide an industrial camera, which includes: an FPGA chip, an image sensor, and an image signal processing ISP chip;

[0006] The image sensor is used to acquire image data and send the image data to the ISP chip;

[0007] The ISP chip is used to perform wide dynamic range (WDR) processing on the image data after receiving the image data sent by the image sensor; and to send the WDR-processed image data to the FPGA chip.

[0008] The FPGA chip is used to perform secondary processing on the WDR-processed image data sent by the ISP chip after receiving it, to obtain the image to be output.

[0009] In one possible implementation, the industrial camera further includes: a memory chip;

[0010] The memory chip is used to store the first program of the FPGA chip and the second program of the ISP chip;

[0011] The ISP chip is specifically used to read the second program from the storage chip after power-on and load the second program; after receiving image data sent by the image sensor, it performs wide dynamic range (WDR) processing on the image data based on the second program; and sends the WDR-processed image data to the FPGA chip.

[0012] The FPGA chip is specifically used to read the first program from the storage chip after power-on and load the first program; after receiving the WDR-processed image data sent by the ISP chip, it performs secondary processing on the WDR-processed image data based on the first program to obtain the image to be output.

[0013] In one possible implementation, the FPGA chip includes a bridge module, an active loading module, and a SoftCPU module, wherein the bridge module is connected to the memory chip, the ISP chip, the active loading module, and the SoftCPU module, respectively.

[0014] The active loading module is used to read the first program from the storage chip through the bridge module and load the first program after power-on.

[0015] The Soft CPU module is used to send a switching instruction to the bridge module after the first program has been loaded;

[0016] The bridge module is used to establish a communication connection between the storage chip and the ISP chip after receiving the switching instruction, and to switch the access address of the ISP chip to the fifth partition of the storage chip, wherein the second program is stored in the fifth partition of the storage chip.

[0017] The ISP chip is specifically used to read the second program from the fifth partition of the storage chip and load the second program after power-on, based on the communication connection with the storage chip.

[0018] In one possible implementation, the bridge module is further configured to monitor the communication between the memory chip and the ISP chip, and establish a communication connection between the memory chip and the Soft CPU module if the idle time of the communication between the memory chip and the ISP chip exceeds a preset time.

[0019] In one possible implementation, the second program includes firmware and software programs; the FPGA chip further includes a clock source module.

[0020] The clock source module is used to provide a first frequency clock source for the ISP chip during the firmware loading phase and a second frequency clock source for the ISP chip during the software loading phase. The firmware loading phase of the ISP chip comprises the process of the ISP chip reading and loading the firmware program; the software loading phase of the ISP chip comprises the process of the ISP chip reading and loading the software program.

[0021] The ISP chip is specifically used to read and load the firmware program from the memory chip based on the communication connection with the memory chip, using the first frequency clock source as the interface clock; to divide the second frequency clock source to obtain a third frequency clock source; and to read and load the software program from the memory chip based on the communication connection with the memory chip, using the third frequency clock source as the interface clock; wherein the third frequency is not greater than a preset communication frequency.

[0022] In one possible implementation, the preset communication frequency is half the reciprocal of the communication delay between the ISP chip and the memory chip.

[0023] In one possible implementation, the first program includes an FPGA bootloader and an FPGA application program;

[0024] The data storage area of ​​the memory chip is divided into a first partition, a second partition, a third partition, a fourth partition, and a fifth partition in ascending order of storage address. The first partition is used to store the FPGA bootloader; the second partition is used to store the embedded bootloader; the third partition is used to store the FPGA application program; the fourth partition is used to store the embedded application program; and the fifth partition is used to store the second program.

[0025] In one possible implementation, the industrial camera further includes a memory chip, and the FPGA chip further includes an image data receiving module, an ISP subsystem, and a GigE Vision subsystem.

[0026] The image data receiving module is used to receive the WDR-processed image data sent by the ISP chip and forward the WDR-processed image data to the ISP subsystem;

[0027] The ISP subsystem is used to perform secondary processing on the image data processed by WDR to obtain the image to be output;

[0028] The GigE Vision subsystem is used to split the image to be output and store the split image to be output in the memory chip according to the GigE Vision standard.

[0029] In one possible implementation, the industrial camera further includes a network chip;

[0030] The GigE Vision subsystem is also used to read the split image to be output from the memory chip and transmit the read split image to the network chip through the GigE Vision standard;

[0031] The network chip is used to transmit the split image to be output to an external device through a network interface.

[0032] Secondly, embodiments of this application provide an image processing method applied to an industrial camera, the industrial camera including: an FPGA chip, an image sensor, and an image signal processing ISP chip, the method including:

[0033] The image sensor acquires image data and sends the image data to the ISP chip;

[0034] After receiving the image data sent by the image sensor, the ISP chip performs wide dynamic range (WDR) processing on the image data and sends the WDR-processed image data to the FPGA chip.

[0035] After receiving the WDR-processed image data sent by the ISP chip, the FPGA chip performs secondary processing on the WDR-processed image data to obtain the image to be output.

[0036] In one possible implementation, the industrial camera further includes: a memory chip;

[0037] After receiving image data from the image sensor, the ISP chip performs wide dynamic range (WDR) processing on the image data and sends the WDR-processed image data to the FPGA chip, including:

[0038] After power-on, the ISP chip reads the second program from the memory chip and loads the second program; after receiving image data sent by the image sensor, it performs wide dynamic range (WDR) processing on the image data based on the second program; and sends the WDR-processed image data to the FPGA chip.

[0039] After receiving the WDR-processed image data sent by the ISP chip, the FPGA chip performs secondary processing on the WDR-processed image data to obtain the image to be output, including:

[0040] After power-on, the FPGA chip reads the first program from the storage chip and loads the first program; after receiving the WDR-processed image data sent by the ISP chip, it performs secondary processing on the WDR-processed image data based on the first program to obtain the image to be output.

[0041] In one possible implementation, the FPGA chip includes a bridge module, an active loading module, and a SoftCPU module, wherein the bridge module is connected to the memory chip, the ISP chip, the active loading module, and the SoftCPU module, respectively.

[0042] After power-on, the FPGA chip reads the first program of the FPGA chip from the storage chip and loads the first program, including: after power-on, the active loading module reads the first program of the FPGA chip from the storage chip through the bridge module and loads the first program;

[0043] The method further includes:

[0044] After the first program is loaded, the Soft CPU module sends a switching command to the bridge module;

[0045] After receiving the switching instruction, the bridge module establishes a communication connection between the storage chip and the ISP chip, and switches the access address of the ISP chip to the fifth partition of the storage chip, wherein the second program is stored in the fifth partition of the storage chip.

[0046] After power-on, the ISP chip reads the second program from the fifth partition of the storage chip based on its communication connection with the storage chip, and loads the second program.

[0047] In one possible implementation, the method further includes:

[0048] The bridge module monitors the communication between the memory chip and the ISP chip, and establishes a communication connection between the memory chip and the SoftCPU module when the idle time of communication between the memory chip and the ISP chip exceeds a preset time.

[0049] In one possible implementation, the second program includes firmware and software programs; the FPGA chip further includes a clock source module; the method further includes:

[0050] The clock source module provides a first frequency clock source to the ISP chip during the firmware loading phase and a second frequency clock source to the ISP chip during the software loading phase. The firmware loading phase of the ISP chip involves the ISP chip reading and loading the firmware program; the software loading phase of the ISP chip involves the ISP chip reading and loading the software program.

[0051] The ISP chip reads the second program from the memory chip and loads the second program, including:

[0052] The ISP chip uses the first frequency clock source as the interface clock, reads the firmware program from the memory chip based on the communication connection with the memory chip, and loads the firmware program; it divides the second frequency clock source to obtain a third frequency clock source; it uses the third frequency clock source as the interface clock, reads the software program from the memory chip based on the communication connection with the memory chip, and loads the software program; wherein, the third frequency is not greater than a preset communication frequency.

[0053] In one possible implementation, the preset communication frequency is half the reciprocal of the communication delay between the ISP chip and the memory chip.

[0054] In one possible implementation, the first program includes an FPGA bootloader and an FPGA application program; the method further includes:

[0055] The data storage area of ​​the memory chip is divided into a first partition, a second partition, a third partition, a fourth partition, and a fifth partition in ascending order of storage address. The first partition is used to store the FPGA bootloader; the second partition is used to store the embedded bootloader; the third partition is used to store the FPGA application program; the fourth partition is used to store the embedded application program; and the fifth partition is used to store the second program.

[0056] In one possible implementation, the industrial camera further includes a memory chip, and the FPGA chip further includes an image data receiving module, an ISP subsystem, and a GigE Vision subsystem; the method further includes:

[0057] The image data receiving module receives the WDR-processed image data sent by the ISP chip and forwards the WDR-processed image data to the ISP subsystem.

[0058] The ISP subsystem performs secondary processing on the image data processed by WDR to obtain the image to be output.

[0059] The GigE Vision subsystem performs splitting processing on the image to be output and stores the split image to be output in the memory chip through the GigE Vision standard.

[0060] In one possible implementation, the industrial camera further includes a network chip; the method further includes:

[0061] The GigE Vision subsystem reads the split image to be output from the memory chip and transmits the read split image to the network chip through the GigE Vision standard;

[0062] The network chip transmits the split image to be output to an external device through a network interface.

[0063] Beneficial effects of the embodiments in this application:

[0064] This application provides an industrial camera and image processing method that utilizes the camera's ISP chip to directly perform WDR processing on the images acquired by the image sensor, thereby eliminating the need for WDR processing on a PC. This saves PC memory bandwidth and CPU resources, reduces the burden on the computer CPU, and minimizes image incompleteness or misalignment caused by untimely CPU processing.

[0065] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0067] Figure 1 This is a schematic diagram of the structure of an industrial camera according to an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of a camera loading architecture according to an embodiment of this application;

[0069] Figure 3 This is a schematic diagram of a chip access link according to an embodiment of this application;

[0070] Figure 4 This is a schematic diagram illustrating a clock frequency according to an embodiment of this application;

[0071] Figure 5 This is a schematic diagram of an industrial camera according to an embodiment of this application;

[0072] Figure 6 This is a schematic flowchart of an image processing method according to an embodiment of this application. Detailed Implementation

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0074] To address the issues in related technologies where Wide Dynamic Range (WDR) processing is performed on a PC, consuming significant CPU and memory bandwidth and potentially causing incomplete or misaligned images, this application provides an industrial camera comprising an FPGA chip, an image sensor, and an ISP (Image Signal Processing) chip. The image sensor acquires image data and sends it to the ISP chip. Upon receiving the image data from the image sensor, the ISP chip performs WDR processing on the image data and sends the WDR-processed image data to the FPGA chip. The FPGA chip, upon receiving the WDR-processed image data from the ISP chip, performs secondary processing on the WDR-processed image data to obtain the output image. The industrial camera provided in this application utilizes the camera's ISP chip to directly perform WDR processing on the image acquired by the image sensor, eliminating the need for WDR processing on a PC. This saves PC memory bandwidth and CPU resources, reduces the burden on the computer's CPU, and minimizes image incompleteness or misalignment caused by untimely CPU processing.

[0075] The following is a detailed explanation. Figure 1 This is a schematic diagram of an industrial camera provided in an embodiment of this application. The industrial camera 100 includes: an FPGA chip 130, an image sensor 110, and an ISP chip 120.

[0076] The image sensor 110 is used to acquire image data and send the image data to the ISP chip 120.

[0077] The ISP chip 120 is used to perform WDR processing on the image data after receiving the image data sent by the image sensor 110, and then send the WDR-processed image data to the FPGA chip 130.

[0078] The FPGA chip 130 is used to perform secondary processing on the WDR-processed image data sent by the ISP chip 120 to obtain the image to be output.

[0079] Wide dynamic range (WDR) processing refers to the ability to clearly distinguish both exceptionally bright and exceptionally dark areas in an image after processing. WDR is the ratio of the brightest luminance signal value to the darkest luminance signal value that an image can distinguish. The ISP chip 120 is equipped with WDR processing capabilities; the specific process of WDR processing can be found in related technologies. In one example, the ISP chip 120 performs WDR processing on the image data acquired by the image sensor 110 using a pre-set ratio of the brightest luminance signal value to the darkest luminance signal value, enabling the WDR-processed image data to clearly distinguish both exceptionally bright and exceptionally dark areas. The FPGA chip 130 further performs secondary processing on the WDR-processed image data, including correction of inherent sensor defects, processing of simple image defects, image enhancement processing, and color image processing. These functions, such as image defect processing, image enhancement processing, and color image processing, yield the output image.

[0080] The industrial camera provided in this application embodiment uses the camera's ISP chip to directly perform WDR processing on the images acquired by the image sensor, thereby eliminating the need for WDR processing on a PC. This saves the PC's memory bandwidth and CPU resources, reduces the burden on the computer's CPU, and also reduces image incompleteness or misalignment caused by untimely CPU processing.

[0081] An ISP chip requires a corresponding program to function properly. In some embodiments, corresponding storage chips can be provided for the ISP chip 120 and the FPGA chip 130 in the industrial camera 100. The first storage chip corresponding to the ISP chip 120 is used to store the file data of the ISP chip; the second storage chip corresponding to the FPGA chip 130 is used to store the file data of the FPGA chip. The program file for the ISP chip may include the system program and data processing program of the ISP chip, and the program file for the FPGA chip may include the system program and data processing program of the FPGA chip.

[0082] In some embodiments, to save costs, the file data of the ISP chip 120 and the FPGA chip 130 can be stored in the same memory chip. In this case, the industrial camera 100 may further include a memory chip for storing the first program of the FPGA chip 130 and the second program of the ISP chip 120.

[0083] In one example, the memory chip can be a NorFlash (non-volatile flash memory) memory chip. The first program can include the boot program, system program, and data processing program of the FPGA chip 130, and the second program can include the boot program, system program, and data processing program of the ISP chip 120. The memory chip can also be a NANDFlash memory chip (a type of flash memory), etc.

[0084] Accordingly, the ISP chip 120 is specifically used to read the second program from the storage chip after power-on and load the second program; and after receiving the image data sent by the image sensor 110, to perform WDR processing on the image data based on the second program and send the WDR-processed image data to the FPGA chip 130.

[0085] The aforementioned FPGA chip 130 is specifically used to read the first program from the storage chip after power-on and load the first program; and after receiving the WDR-processed image data sent by the ISP chip 120, to perform secondary processing on the WDR-processed image data based on the first program to obtain the image to be output.

[0086] After power-on, FPGA chip 130 accesses the memory chip, reads the first program from the memory chip, and writes the first program into FPGA chip 130 to complete the loading of FPGA chip 130. Similarly, after power-on, ISP chip 120 accesses the memory chip, reads the second program from the memory chip, and writes the second program into ISP chip 120 to complete the loading of ISP chip 120. In one example, the memory chip is accessed using an SPI (Serial Peripheral Interface) bus.

[0087] After the ISP chip 120 starts up, upon receiving image data from the image sensor 110, it performs WDR processing on the image data using a second program to obtain WDR-processed image data, and then sends the WDR-processed image data to the FPGA chip 130. Upon receiving the WDR-processed image data from the ISP chip 120, the FPGA chip 130 performs secondary processing on the WDR-processed image data using the image correction parameters, image enhancement parameters, and image color processing parameters configured in the first program, to obtain the image to be output.

[0088] In this embodiment, a single memory chip is used to load the ISP chip and the FPGA chip. Compared to configuring a memory chip for each of the ISP chip and the FPGA chip, this effectively saves the hardware cost of the camera and also saves the circuit board space of the camera.

[0089] To ensure the orderly loading of the ISP chip and FPGA chip, in some embodiments, the FPGA chip 130 may include a bridge module, an active loading module, and a Soft CPU (software reconfigurable processor) module. The bridge module is connected to the memory chip, the ISP chip 120, the active loading module, and the Soft CPU module, respectively. The bridge module may be an SPI bridge module, and the active loading module may be an FPGA active loading module.

[0090] An active loading module is used to read the first program from the memory chip via the bridge module and load the first program after power-on.

[0091] The Soft CPU module is used to send a switching command to the bridge module after the first program has finished loading.

[0092] The bridge module is used to establish a communication connection between the storage chip and the ISP chip after receiving a switching command, and to switch the access address of the ISP chip to the fifth partition of the storage chip, where the fifth partition of the storage chip stores the second program.

[0093] The ISP chip is specifically used to read and load the second program from the fifth partition of the memory chip after power-on, based on the communication connection with the memory chip.

[0094] In one example, after power-on, the FPGA active loading module accesses the memory chip via the SPI bridge module, reads the first program from the memory chip using the SPI bus, and writes the first program into the FPGA chip 130 to complete the loading of the FPGA chip 130. After the FPGA chip 130 is loaded, the FPGA active loading module releases control of the SPI bus over the memory chip, and the Soft CPU module takes over control of the SPI bus. After the first program is loaded, the Soft CPU module sends a switching command to the SPI bridge module. Upon receiving the switching command from the Soft CPU module, the SPI bridge module establishes a communication connection between the memory chip and the ISP chip 120. After establishing the communication connection between the memory chip and the ISP chip 120, the SPI bridge module switches the access address of the ISP chip 120 to the fifth partition of the memory chip, so that after power-on, the ISP chip 120 can access the memory chip using the SPI bus via the communication connection, read the second program from the fifth partition of the memory chip, and write the second program into the ISP chip 120 to complete the loading of the ISP chip 120. The fifth partition of the memory chip is used to store the second program of the ISP chip 120.

[0095] For example, the second program of the ISP chip 120 is stored at address 0x700000 in the fifth partition of the memory chip, while the access address issued by the ISP chip 120 starts from 0x000000. After establishing a communication connection between the memory chip and the ISP chip 120, the SPI bridge module switches the access address issued by the ISP chip 120 from 0x000000 to address 0x700000 in the fifth partition of the memory chip.

[0096] In this embodiment, the SPI bridge module is used to arbitrate the loading of the ISP chip and the FPGA chip, ensuring that the FPGA chip is loaded first and the ISP chip is loaded later. This reduces the conflict between the loading of the ISP chip and the FPGA chip after power-on, enabling the camera to operate stably.

[0097] In some embodiments, the bridge module is further configured to monitor the communication connection status between the memory chip and the ISP chip, and establish a communication connection between the memory chip and the SoftCPU module when the idle time of the communication connection between the memory chip and the ISP chip exceeds a preset time.

[0098] The memory chip uses the SPI bus for access. The SPI bridge module establishes a communication connection between the memory chip and the ISP chip 120, meaning the SPI bus control of the SPI bridge module is held by the ISP chip 120. A timeout mechanism is configured in the SPI bridge module. When the ISP chip 120 holds SPI bus control, the timeout mechanism detects the SPI bus status between the memory chip and the ISP chip 120. When the SPI bus is idle, it further monitors whether the idle time exceeds a preset duration. If it does, it is assumed that the ISP chip 120 has finished loading, triggering a timeout operation and automatically switching SPI bus control back to the Soft CPU module, establishing a communication connection between the memory chip and the Soft CPU module. Conversely, if the SPI bus is not idle, or the idle time has not exceeded the preset duration, it indicates that the ISP chip 120 has not yet finished loading, and no timeout operation will be triggered to switch SPI bus control. The preset duration can be set according to actual needs, such as 300 milliseconds, 500 milliseconds, or 800 milliseconds, etc.

[0099] In one example, the Soft CPU module is connected to the image sensor 110. The Soft CPU module accesses the memory chip through a communication connection with the memory chip, reads image correction-related parameters and user configuration parameters from the memory chip, and then updates the parameters of the image sensor 110.

[0100] For example, such as Figure 2 As shown, after the FPGA chip 130 completes the first program loading, the FPGA active loading module releases control of the SPI bus of the NorFlash memory chip, which is then controlled by the Soft CPU module. The Soft CPU module controls the selection switch, switching the SPI bridge module to access the ISP chip 120 (i.e., the Soft CPU module sends a switching command to the SPI bridge module, and the SPI bridge module establishes a communication connection between the NorFlash memory chip and the ISP chip 120). The Soft CPU module releases the reset pin of the ISP chip 120. Figure 2(Including the reset function), the ISP chip 120 accesses the NorFlash memory chip via the SPI bus, reads the second program from the NorFlash memory chip, and writes the second program into the ISP chip 120 to complete the loading of the ISP chip 120. The SPI bridge module monitors the communication connection status between the NorFlash memory chip and the ISP chip 120. When the idle time of the communication connection between the NorFlash memory chip and the ISP chip 120 exceeds a preset time, it is considered that the ISP chip 120 has completed the loading of the second program. It automatically switches the SPI bus control of the SPI bridge module back to the Soft CPU module, establishes a communication connection between the NorFlash memory chip and the Soft CPU module, and the Soft CPU module is connected to the SPI bridge module through the local bus.

[0101] The loading of the ISP chip 120 is accomplished after a communication connection is established between the NorFlash memory chip and the ISP chip 120, by the ISP chip 120 accessing the NorFlash memory chip via the SPI bus. For example, as shown... Figure 3 As shown, Figure 3 The diagram illustrates the link delay when the ISP chip 120 accesses the NorFlash memory chip. The clock frequency of the SPI bus is generated by the ISP chip 120 to read (or sample) data from the NorFlash memory chip. Due to the delay on the FPGA chip 130, the data read from the NorFlash memory chip may be out of sync with the clock generated by the ISP chip 120, which may lead to errors in the data read by the ISP chip 120 and cause the ISP chip 120 to fail to load.

[0102] like Figure 3 As shown, the IO (Input / Output) site delay and the inner delay of the NorFlash memory chip are collectively referred to as delay. The total delay of the ISP chip 120 accessing the NorFlash memory chip via the SPI bus can be expressed as: total delay = clk delay + clkdelay + inner delay + miso delay + miso delay ≈ IO Site Delay * 4 + inner delay. Here, clkdelay represents the clock delay, and miso (master input, slave output) delay represents the master-slave delay. Figure 3In this context, CS represents the chip select signal, and MOSI (saster output, slave input) represents master output and slave input.

[0103] The NorFlash memory chip modulates data on the falling edge of the SPI clock (clk), while the ISP chip 120 samples data on the rising edge of the SPI clock. To ensure successful loading of the ISP chip 120, half a cycle of the SPI clock must be greater than the total delay. To eliminate data sampling (or reading) errors caused by delay, the SPI clock can be downclocked to 1 / (total delay*2) Hz. However, the ISP chip 120 itself does not support such a low clock frequency, therefore the FPGA chip 130 is used to control the clock frequency of the ISP chip 120.

[0104] In some embodiments, the loading of the ISP chip 120 can be divided into two stages: a firmware loading stage (ROMBOOT) and a software loading stage (RAMBOOT). Accordingly, the second program mentioned above may include firmware and software programs. The FPGA chip 130 mentioned above may also include a clock source module.

[0105] The clock source module is used to provide a first frequency clock source for the ISP chip 120 during the firmware loading phase and a second frequency clock source for the ISP chip 120 during the software loading phase. The firmware loading phase of the ISP chip 120 involves the process of the ISP chip 120 reading and loading the firmware program; the software loading phase involves the process of the ISP chip 120 reading and loading the software program.

[0106] The ISP chip 120 is specifically used to read and load firmware from the memory chip based on the communication connection with the memory chip, using a first frequency clock source as the interface clock; to divide the second frequency clock source to obtain a third frequency clock source; and to read and load software from the memory chip based on the communication connection with the memory chip, using the third frequency clock source as the interface clock; wherein the third frequency is not greater than a preset communication frequency.

[0107] In one example, during the firmware loading phase of the ISP chip 120, the clock source module of the FPGA chip 130 provides a first frequency clock source for the ISP chip 120. Furthermore, the ISP chip 120 provides a first frequency clock source for the SPI bus, accesses the NorFlash memory chip via a communication connection, reads the firmware program from the NorFlash memory chip, and loads the firmware program. During the software loading phase of the ISP chip 120, the clock source module of the FPGA chip 130 provides a second frequency clock source for the ISP chip 120. Furthermore, the ISP chip 120 divides the second frequency clock source to obtain a third frequency clock source. This third frequency is not greater than a preset communication frequency. In one example, the preset communication frequency can be half the reciprocal of the communication delay between the ISP chip 120 and the memory chip (i.e., 1 / (total delay*2) mentioned above). The ISP chip 120 provides a third frequency clock source for the SPI bus, accesses the NorFlash memory chip via a communication connection, reads the software program from the NorFlash memory chip, and loads the software program. The first frequency and the second frequency are determined according to the type of ISP chip 120.

[0108] For example, such as Figure 4 As shown, Figure 4 This demonstrates the clock frequency during the two-stage loading process of the ISP chip 120. During the firmware loading stage of the ISP chip 120, the ISP chip 120 accesses the NorFlash memory chip via the SPI bus. The clock frequency provided by the ISP chip 120 to the SPI bus during this stage can be arbitrarily changed by the FPGA chip 130, depending on the type or model of the ISP chip 120. Figure 4 As shown, the clock source module of FPGA chip 130 provides a 6MHz clock source to ISP chip 120, and ISP chip 120 provides a 6MHz clock source to access the NorFlash memory chip via the SPI bus. Within 200ms after the firmware loading phase of ISP chip 120 ends, the clock source module of FPGA chip 130 switches the 6MHz clock source to a 24MHz clock source for ISP chip 120. During the software loading phase of ISP chip 120, the clock source module of FPGA chip 130 provides a 24MHz clock source to ISP chip 120. After internal phase-locked loop frequency division, ISP chip 120 obtains a 5.625MHz clock source, which is then used by ISP chip 120 to access the NorFlash memory chip via the SPI bus, thus loading ISP chip 120.

[0109] In this embodiment, the ISP chip accesses the NorFlash memory chip via the SPI bus. To ensure successful loading of the ISP chip, an FPGA chip provides a clock source for the ISP chip, which in turn provides a clock source for the SPI bus, eliminating data reading errors caused by latency. By switching the clock frequency during the ISP chip loading process using the FPGA chip, loading errors are reduced.

[0110] In some embodiments, the first program may include an FPGA bootloader and an FPGA application program; the data storage area of ​​the memory chip is divided into a first partition, a second partition, a third partition, a fourth partition, and a fifth partition in order of increasing memory address; wherein, the first partition is used to store the FPGA bootloader; the second partition is used to store the embedded bootloader; the third partition is used to store the FPGA application program; the fourth partition is used to store the embedded application program; and the fifth partition is used to store the second program.

[0111] The memory chip stores the first program of the FPGA chip 130 and the second program of the ISP chip 120, realizing the function of loading the FPGA chip 130 and the ISP chip 120 with a single memory chip. The data storage area of ​​the memory chip is reasonably partitioned according to the memory address from low to high, and is divided into the first partition for storing the FPGA bootloader, the second partition for storing the embedded bootloader, the third partition for storing the FPGA application, the fourth partition for storing the embedded application, and the fifth partition for storing the second program.

[0112] During the camera loading process, the FPGA chip 130 is loaded first, followed by the ISP chip 120. The program storage space of the FPGA chip 130 and the ISP chip 120 is partitioned in ascending order of storage address to improve the speed of data reading and loading of the FPGA chip 130 and the ISP chip 120.

[0113] The data storage area of ​​a memory chip may also include attribute data partitions, which can be used to store attribute data of each partition.

[0114] For example, the partition table of a NorFlash memory chip is shown in Table 1 below:

[0115] Table 1 Partition Table of NorFlash Memory Chips

[0116]

[0117] In Table 1, the first partition stores the FPGA bootloader, with the FPGA uboot starting at address 0x000000; the second partition stores the embedded bootloader, with the embedded uboot file starting at address 0x100000; the third partition stores the FPGA application, with the FPGA app starting at address 0x1a0000; the fourth partition stores the embedded application, with the embedded app file starting at address 0x450000; and the fifth partition stores the second program, which is the ISP chip file, starting at address 0x700000. The FLASH capacity represents the attribute data partition, with a size of 8MB, indicating that the space capacity of the first, second, third, fourth, and fifth partitions is 8MB.

[0118] In this embodiment, the data storage area of ​​the memory chip is reasonably partitioned, enabling the camera to be loaded with an 8MB NorFlash memory chip, further reducing the cost of the camera.

[0119] In some embodiments, the industrial camera 100 may further include a memory chip, and the FPGA chip 130 may further include an image data receiving module, an ISP subsystem, and a GigE Vision subsystem. In one example, the memory chip may be a DDR (Double Data Rate) synchronous dynamic random access memory chip.

[0120] The image data receiving module is used to receive the WDR-processed image data sent by the ISP chip 120 and forward the WDR-processed image data to the ISP subsystem.

[0121] The ISP subsystem is used to perform secondary processing on the image data after WDR processing to obtain the image to be output.

[0122] The GigE Vision subsystem is used to split the image to be output and store the split image to the memory chip according to the GigE Vision standard.

[0123] The ISP subsystem can perform functions such as correcting inherent sensor defects, processing simple image defects, image enhancement processing, and color image processing. Furthermore, the ISP subsystem can perform secondary processing on the image data after WDR processing, such as correcting inherent sensor defects, processing simple image defects, image enhancement processing, and color image processing, to obtain the image to be output.

[0124] GigE Vision is a standard for image transmission based on Gigabit Ethernet, initiated and specified by the AIA (Automated Imaging Association), and is suitable for industrial imaging applications. The GigE Vision standard includes GVCP (GigE Vision Control Protocol) and GVSP (GigE Vision Streaming Protocol). GVCP describes the control command transmission protocol between the host and the device; it is bidirectional data that can originate from either the host or the device, used to send and receive control information. GVSP describes the image data transmission protocol between the host and the device; it is unidirectional data that can only be sent from the device to the host.

[0125] For example, the GigE Vision standard-based industrial camera 100 (or gigabit Ethernet industrial camera) splits each frame of the image to be output into a giant frame packet format of about 9KB in size and stores the image to be output in the memory chip.

[0126] In this embodiment, the ISP subsystem is used to perform secondary processing on the WDR-processed image data, including correction of inherent sensor defects, processing of simple image defects, image enhancement processing, and color image processing. The GigEVision subsystem is used to split the output image obtained from the secondary processing, and the split output image is stored in the memory chip through the GigE Vision standard for easy access to the output image at any time.

[0127] In some embodiments, the industrial camera 100 described above may further include a network chip. In one example, the network chip may be a PHY (Physical Layer) network chip.

[0128] The aforementioned GigE Vision subsystem is also used to read the split image to be output from the memory chip and transmit the read split image to the network chip through the GigE Vision standard; the network chip is used to transmit the split image to be output to external devices through the network interface.

[0129] In one example, the GigE Vision subsystem stores the split images to be output in a DDR memory chip via the GigE Vision standard. The DDR memory chip buffers and retransmits the split images. The GigE Vision subsystem reads the split images from the DDR memory chip and transmits them to the PHY network chip via the GigE Vision standard. The PHY network chip then transmits the split images to an external device via an RJ45 network interface, with a maximum transmission rate of 1Gb / s. The external device decodes the received discrete data packets and reassembles them into image frames to achieve complete image reception.

[0130] For example, such as Figure 5 As shown, Figure 5 The industrial camera includes: an image sensor, an ISP chip, a NorFlash memory chip, an FPGA chip, a PHY network chip, a DDR memory chip, and an RJ45 network interface. The FPGA chip includes: an image data receiving module, an ISP subsystem, a GigE Vision subsystem, an SPI bridge module (i.e., the bridge module mentioned above), an FPGA active loading module (hard core) (i.e., the active loading module mentioned above), and a Soft CPU (the Soft CPU module mentioned above).

[0131] After power-on, the FPGA active loading module reads and loads the first program from the NorFlash memory chip via the SPI bridge module. Once the first program is loaded, the FPGA active loading module releases control of the NorFlash memory chip's SPI bus, which is then taken over by the Soft CPU module. The Soft CPU module sends a switching command to the SPI bridge module, which, upon receiving the command, establishes a communication connection between the NorFlash memory chip and the ISP chip. After power-on, the ISP chip, based on its communication connection with the NorFlash memory chip, reads and loads its second program from the NorFlash memory chip via the SPI bus. The SPI bridge module monitors the communication connection status between the NorFlash memory chip and the ISP chip. If the idle time of this connection exceeds a preset duration, it considers the ISP chip to have completed loading the second program and automatically switches control back to the Soft CPU module, establishing a new communication connection between the NorFlash memory chip and the Soft CPU module. The Soft CPU module accesses the NorFlash memory chip via the SPI bus to read image correction-related parameters and user configuration parameters from the NorFlash memory chip in order to update the parameters of the image sensor.

[0132] The image sensor acquires image data and sends it to the ISP chip. Upon receiving the image data, the ISP chip performs WDR processing on the image data based on a second program and sends the WDR-processed image data to the FPGA chip. The FPGA chip's image data receiving module receives the WDR-processed image data from the ISP chip and forwards it to the ISP subsystem. The ISP subsystem performs secondary processing on the WDR-processed image data to obtain the output image. The GigE Vision subsystem splits the output image and stores the split image in a DDR memory chip via the GigE Vision standard. It also reads the split image from the DDR memory chip and transmits it to the PHY network chip via the GigE Vision standard. The PHY network chip then transmits the split image to an external device via an RJ45 network interface.

[0133] In this embodiment, the SPI bridge module arbitrates the loading of the ISP chip and FPGA chip, ensuring that the FPGA chip is loaded before the ISP chip, reducing conflicts between the ISP and FPGA chip loading after power-on and enabling stable camera operation. This achieves loading of the ISP and FPGA chips onto a single NorFlash memory chip, effectively saving camera hardware costs and circuit board space. By utilizing the camera's ISP chip to directly perform WDR processing on the images acquired by the image sensor, WDR processing is no longer required on the PC, saving PC memory bandwidth and CPU resources, reducing the burden on the computer's CPU, and minimizing image incompleteness or misalignment caused by untimely CPU processing.

[0134] See Figure 6 This application also provides an image processing method applied to the aforementioned industrial camera, which includes an FPGA chip, an image sensor, and an image signal processing (ISP) chip. The method includes:

[0135] S601, the image sensor acquires image data and sends the image data to the ISP chip;

[0136] S602, after receiving image data sent by the image sensor, the ISP chip performs wide dynamic range (WDR) processing on the image data and sends the WDR-processed image data to the FPGA chip.

[0137] S603: After receiving the WDR-processed image data sent by the ISP chip, the FPGA chip performs secondary processing on the WDR-processed image data to obtain the image to be output.

[0138] In this embodiment, the image acquired by the image sensor is directly processed by the camera's ISP chip, thus eliminating the need for WDR processing on the PC. This saves the PC's memory bandwidth and CPU resources, reduces the burden on the computer's CPU, and also reduces image incompleteness or misalignment caused by untimely CPU processing.

[0139] In some embodiments, the industrial camera described above further includes: a memory chip;

[0140] After receiving image data from the image sensor, the aforementioned ISP chip performs wide dynamic range (WDR) processing on the image data and then sends the WDR-processed image data to the FPGA chip, including:

[0141] After power-on, the ISP chip reads the second program from the memory chip and loads the second program; after receiving image data sent by the image sensor, it performs wide dynamic range (WDR) processing on the image data based on the second program; and sends the WDR-processed image data to the FPGA chip.

[0142] After receiving the WDR-processed image data from the ISP chip, the FPGA chip performs secondary processing on the WDR-processed image data to obtain the image to be output, including:

[0143] After powering on, the FPGA chip reads the first program from the memory chip and loads the first program; after receiving the WDR-processed image data sent by the ISP chip, it performs secondary processing on the WDR-processed image data based on the first program to obtain the image to be output.

[0144] In some embodiments, the FPGA chip includes a bridge module, an active loading module, and a soft CPU module, wherein the bridge module is connected to the memory chip, the ISP chip, the active loading module, and the soft CPU module, respectively.

[0145] The above-mentioned FPGA chip reads the first program of the FPGA chip from the storage chip after power-on and loads the first program, including: after power-on, the active loading module reads the first program of the FPGA chip from the storage chip through the bridge module and loads the first program.

[0146] The above methods also include:

[0147] After the first program is loaded, the Soft CPU module sends a switching command to the bridge module;

[0148] After receiving the switching command, the bridge module establishes a communication connection between the storage chip and the ISP chip, and switches the access address of the ISP chip to the fifth partition of the storage chip, where the fifth partition of the storage chip stores the second program.

[0149] After power-on, the aforementioned ISP chip reads and loads the second program from the memory chip, including: after power-on, the ISP chip reads and loads the second program from the fifth partition of the memory chip based on the communication connection with the memory chip.

[0150] In some embodiments, the above method further includes:

[0151] The bridge module monitors the communication connection status between the memory chip and the ISP chip. When the idle time of the communication connection between the memory chip and the ISP chip exceeds a preset time, it establishes a communication connection between the memory chip and the Soft CPU module.

[0152] In some embodiments, the second program includes firmware and software programs; the FPGA chip further includes a clock source module; the method further includes:

[0153] The clock source module provides a first frequency clock source for the ISP chip during the firmware loading phase and a second frequency clock source during the software loading phase. The firmware loading phase of the ISP chip consists of the process of the ISP chip reading and loading the firmware program, and the software loading phase consists of the process of the ISP chip reading and loading the software program.

[0154] The aforementioned ISP chip reads the second program from the memory chip and loads the second program, including:

[0155] The ISP chip uses a first frequency clock source as the interface clock, reads and loads the firmware program from the memory chip based on the communication connection with the memory chip; it divides the second frequency clock source to obtain a third frequency clock source; it uses the third frequency clock source as the interface clock, reads and loads the software program from the memory chip based on the communication connection with the memory chip; wherein, the third frequency is not greater than the preset communication frequency.

[0156] In some embodiments, the preset communication frequency is half the reciprocal of the communication delay between the ISP chip and the memory chip.

[0157] In some embodiments, the first program includes an FPGA bootloader and an FPGA application program; the method further includes:

[0158] The data storage areas of the memory chip are divided into five partitions in ascending order of memory address: Partition 1, Partition 2, Partition 3, Partition 4, and Partition 5. Partition 1 is used to store the FPGA bootloader; Partition 2 is used to store the embedded bootloader; Partition 3 is used to store the FPGA application program; Partition 4 is used to store the embedded application program; and Partition 5 is used to store the second program.

[0159] In some embodiments, the industrial camera further includes a memory chip, and the FPGA chip further includes an image data receiving module, an ISP subsystem, and a GigE Vision subsystem; the method further includes:

[0160] The image data receiving module receives the WDR-processed image data sent by the ISP chip and forwards the WDR-processed image data to the ISP subsystem;

[0161] The ISP subsystem performs secondary processing on the image data after WDR processing to obtain the image to be output;

[0162] The GigE Vision subsystem splits the image to be output and stores the split image to the memory chip according to the GigE Vision standard.

[0163] In some embodiments, the industrial camera further includes a network chip; the method further includes:

[0164] The GigE Vision subsystem reads the split image to be output from the memory chip and transmits the read split image to the network chip through the GigE Vision standard;

[0165] The network chip transmits the split image to be output to an external device via the network interface.

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the method embodiments are basically similar to the system embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the system embodiments.

[0168] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An industrial camera, characterized in that, The industrial camera includes: an FPGA chip, an image sensor, and an image signal processing ISP chip; The image sensor is used to acquire image data and send the image data to the ISP chip; The ISP chip is used to perform wide dynamic range (WDR) processing on the image data after receiving the image data sent by the image sensor; and to send the WDR-processed image data to the FPGA chip. The FPGA chip is used to perform secondary processing on the WDR-processed image data after receiving the WDR-processed image data sent by the ISP chip to obtain the image to be output. The industrial camera also includes: a memory chip; The memory chip is used to store the first program of the FPGA chip and the second program of the ISP chip; the second program is used to perform wide dynamic range (WDR) processing on the image data, and the first program is used to perform secondary processing on the WDR-processed image data; The second program includes firmware and software programs; the FPGA chip includes a clock source module; The clock source module is used to provide a first frequency clock source for the ISP chip during the firmware loading phase and a second frequency clock source for the ISP chip during the software loading phase. The firmware loading phase of the ISP chip comprises the process of the ISP chip reading and loading the firmware program; the software loading phase of the ISP chip comprises the process of the ISP chip reading and loading the software program. The ISP chip is specifically used to read and load the firmware program from the memory chip based on the communication connection with the memory chip, using the first frequency clock source as the interface clock; to divide the second frequency clock source to obtain a third frequency clock source; and to read and load the software program from the memory chip based on the communication connection with the memory chip, using the third frequency clock source as the interface clock; wherein the third frequency is not greater than a preset communication frequency.

2. The industrial camera according to claim 1, characterized in that, The ISP chip is specifically used to read the second program from the storage chip after power-on and load the second program; after receiving image data sent by the image sensor, it performs wide dynamic range (WDR) processing on the image data based on the second program; and sends the WDR-processed image data to the FPGA chip. The FPGA chip is specifically used to read the first program from the storage chip after power-on and load the first program; after receiving the WDR-processed image data sent by the ISP chip, it performs secondary processing on the WDR-processed image data based on the first program to obtain the image to be output.

3. The industrial camera according to claim 2, characterized in that, The FPGA chip also includes a bridge module, an active loading module, and a soft CPU module. The bridge module is connected to the memory chip, the ISP chip, the active loading module, and the soft CPU module, respectively. The active loading module is used to read the first program from the storage chip through the bridge module and load the first program after power-on. The Soft CPU module is used to send a switching instruction to the bridge module after the first program has been loaded; The bridge module is used to establish a communication connection between the storage chip and the ISP chip after receiving the switching instruction, and to switch the access address of the ISP chip to the fifth partition of the storage chip, wherein the second program is stored in the fifth partition of the storage chip. The ISP chip is specifically used to read the second program from the fifth partition of the storage chip and load the second program after power-on, based on the communication connection with the storage chip.

4. The industrial camera according to claim 3, characterized in that, The bridge module is also used to monitor the communication between the memory chip and the ISP chip, and to establish a communication connection between the memory chip and the Soft CPU module when the idle time of the communication between the memory chip and the ISP chip exceeds a preset time.

5. The industrial camera according to claim 1, characterized in that, The preset communication frequency is half the reciprocal of the communication delay between the ISP chip and the memory chip.

6. The industrial camera according to claim 2, characterized in that, The first program includes an FPGA bootloader and an FPGA application program; The data storage area of ​​the memory chip is divided into five partitions in order of increasing memory address: the first partition, the second partition, the third partition, the fourth partition, and the fifth partition. The first partition is used to store the FPGA bootloader; The second partition is used to store the embedded bootloader; The third partition is used to store the FPGA application; The fourth partition is used to store embedded applications; The fifth partition is used to store the second program.

7. The industrial camera according to claim 2, characterized in that, The industrial camera also includes a memory chip, and the FPGA chip also includes an image data receiving module, an ISP subsystem, and a GigE Vision subsystem. The image data receiving module is used to receive the WDR-processed image data sent by the ISP chip and forward the WDR-processed image data to the ISP subsystem; The ISP subsystem is used to perform secondary processing on the image data processed by WDR to obtain the image to be output; The GigE Vision subsystem is used to split the image to be output and store the split image to be output in the memory chip according to the GigE Vision standard.

8. The industrial camera according to claim 7, characterized in that, The industrial camera also includes: a network chip; The GigE Vision subsystem is also used to read the split image to be output from the memory chip and transmit the read split image to the network chip through the GigE Vision standard; The network chip is used to transmit the split image to be output to an external device through a network interface.

9. An image processing method, characterized in that, Applied to an industrial camera, the industrial camera comprising: an FPGA chip, an image sensor, and an image signal processing ISP chip, the method includes: The image sensor acquires image data and sends the image data to the ISP chip; After receiving the image data sent by the image sensor, the ISP chip performs wide dynamic range (WDR) processing on the image data and sends the WDR-processed image data to the FPGA chip. After receiving the WDR-processed image data sent by the ISP chip, the FPGA chip performs secondary processing on the WDR-processed image data to obtain the image to be output. The industrial camera further includes: a memory chip; the memory chip stores a first program of the FPGA chip and a second program of the ISP chip; the second program is used to perform wide dynamic range (WDR) processing on the image data, and the first program is used to perform secondary processing on the WDR-processed image data; The second program includes firmware and software programs; the FPGA chip includes a clock source module; The method further includes: The clock source module provides a first frequency clock source to the ISP chip during the firmware loading phase and a second frequency clock source to the ISP chip during the software loading phase. The firmware loading phase of the ISP chip involves the ISP chip reading and loading the firmware program; the software loading phase of the ISP chip involves the ISP chip reading and loading the software program. The ISP chip uses the first frequency clock source as the interface clock, reads the firmware program from the memory chip based on the communication connection with the memory chip, and loads the firmware program; it divides the second frequency clock source to obtain a third frequency clock source; it uses the third frequency clock source as the interface clock, reads the software program from the memory chip based on the communication connection with the memory chip, and loads the software program; wherein, the third frequency is not greater than a preset communication frequency.

Images