Image processing system and method for industrial camera

Abstract

The invention provides an image processing system and method of an industrial camera. Relates to the technical field of industrial camera image processing. The system comprises a programmable logic end and a processing system end, the programmable logic end comprises a first flow exchange IP core and a multi-path image processing module; each path of image processing module has at least one image processing function; the first flow exchange IP core comprises a first input interface and a plurality of first output interfaces, and each first output interface is connected with one path of image processing module; the first input interface receives a first image data stream; the processing system end is connected with the first flow exchange IP core through the advanced extensible interface and is used for dynamically configuring a routing rule of the first flow exchange IP core; and the first stream exchange IP core transmits the first image data stream to the corresponding image processing module through the corresponding first output interface according to a routing rule. According to the invention, an image processing algorithm and data flow flexible routing management can be dynamically switched, and an efficient and flexible image processing function is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing of industrial cameras, and in particular to an image processing system and method of industrial cameras. BACKGROUND

[0002] With the rapid development of intelligent manufacturing and machine vision technology, Gigabit Ethernet (GigE) based industrial cameras have been widely used in industrial detection, automated production, medical imaging, security monitoring and other fields. GigE cameras are based on GigE Vision protocol, which transmits image data through Gigabit Ethernet. They have the advantages of high bandwidth (1Gbps), long distance transmission (up to 100 meters), low cost and standardization, and can be seamlessly integrated into industrial automation systems. In particular, in industrial detection, GigE cameras are used to capture high-resolution images in real time to support tasks such as defect detection, dimension measurement and target recognition; in the security field, GigE cameras transmit video streams through the network to support remote monitoring and intelligent analysis. These applications have put forward higher and higher requirements for the real-time performance, image quality and processing capacity of the cameras.

[0003] Currently, most of the processing systems of industrial cameras are based on Field-Programmable Gate Array (FPGA) GigE camera processing systems. This scheme generally uses a separate FPGA to collect image sensor data and output Bayer format (12 bits / pixel). The FPGA internally implements fixed-function image processing modules, including Bayer format image to RGB format image, fixed gain white balance and simple denoising filter, etc. The image data stream processed by these modules is buffered through the First-In-First-Out queue (FIFO) inside the FPGA, and then directly transmitted to the external Ethernet controller chip to realize GigE Vision data transmission.

[0004] However, the existing camera system integrates fixed-function image processing modules in hardware or firmware, which cannot dynamically adjust the image processing algorithm according to the actual task requirements. The image processing function is fixed, and if you want to redevelop the image processing module, you often need to redesign the hardware or firmware, which has a high development cycle and cost. Moreover, the data flow path is single, and the lack of dynamic routing capability leads to insufficient data flow management capability. SUMMARY

[0005] The embodiment of the present application provides an image processing system and method of an industrial camera, to solve the problem that the prior art cannot dynamically adjust the image processing algorithm according to the actual task demand, resulting in the solidification of the image processing function and the poor data flow dynamic routing capability.

[0006] In a first aspect, the embodiment of the present application provides an image processing system of an industrial camera, comprising: a programmable logic end and a processing system end; the programmable logic end comprises: a first flow exchange IP core and a plurality of image processing modules; wherein each image processing module has at least one image processing function;

[0007] The first flow exchange IP core comprises a first input interface and a plurality of first output interfaces, and each first output interface is connected with an image processing module; the first input interface receives a first image data stream;

[0008] The processing system end is connected with the first flow exchange IP core through a high-level extensible interface, and is configured to dynamically configure a routing rule of the first flow exchange IP core through the high-level extensible interface;

[0009] The first flow exchange IP core transmits the first image data stream to the corresponding image processing module through the corresponding first output interface according to the routing rule.

[0010] In a possible implementation, the programmable logic end further comprises: a second flow exchange IP core;

[0011] The second flow exchange IP core comprises a plurality of second input interfaces and a second output interface; each second input interface is connected with an image processing module, and is configured to receive a second image data stream output by the image processing module;

[0012] The processing system end is connected with the second flow exchange IP core through a high-level extensible interface, and is configured to dynamically configure a routing rule of the second flow exchange IP core, so as to transmit the second image data stream output by each image processing module to a next module through the second output interface.

[0013] In a possible implementation, the plurality of image processing modules comprises a direct data flow module; each image processing module except the direct data flow module comprises at least one image processing unit, and each image processing unit has one image processing function; wherein the direct data flow module is configured to directly transmit the first image data stream.

[0014] In a possible implementation, the programmable logic end further comprises: a direct memory access module connected with the second output interface;

[0015] The processing system end comprises a storage controller and a memory connected with the storage controller;

[0016] The direct memory access module is also connected with the storage controller through an advanced extensible interface, for storing the second image data stream into the storage through the storage controller.

[0017] In a possible implementation, the processing system end further comprises a processor connected with the storage controller.

[0018] The processor is configured to read the second image data stream in the storage controller, and send the second image data stream to an external host computer through a gigabit Ethernet.

[0019] In a possible implementation, the processor comprises a data encapsulation module and a data packet processing module connected with the data encapsulation module.

[0020] The data encapsulation module is configured to encapsulate the second image data stream based on a GigE Vision streaming protocol, and send the encapsulated image data to the data packet processing module.

[0021] The data packet processing module is configured to send the encapsulated image data to an external host computer through a gigabit Ethernet.

[0022] In a possible implementation, the processor is connected with an interconnection interface of an advanced extensible interface of the programmable logic end through a general interface of a master advanced extensible interface; and the interconnection interface of the advanced extensible interface is connected with each module of the programmable logic end.

[0023] In a possible implementation, the programmable logic end further comprises an image data acquisition and conversion module and a data stream conversion module connected with the image data acquisition and conversion module; and the data stream conversion module is further connected with a first input interface of the first stream exchange IP core.

[0024] The image data acquisition and conversion module is connected with an image sensor, configured to acquire raw image data obtained by the image sensor, and convert the raw image data to obtain a raw image data stream in a target format.

[0025] The data stream conversion module is configured to convert the raw image data stream in the target format into the first image data stream based on an advanced extensible interface 4-stream protocol.

[0026] In a second aspect, an embodiment of the present application provides an image processing method of an industrial camera, applied to a processor in an image processing system of the industrial camera, and the method comprises the following steps.

[0027] acquire a GVCP format control command data packet sent by an external host computer; the GVCP format control command data packet is configuration data input by a user on the host computer;

[0028] analyze the GVCP format control command data packet to obtain a routing switching instruction;

[0029] send the routing switching instruction to a first flow switching IP core at a programmable logic end, to configure a routing rule of the first flow switching IP core, so that the first flow switching IP core transmits a first image data stream received by the first flow switching IP core through a corresponding first output interface to a corresponding image processing module according to the routing rule, so that the corresponding image processing module processes the first image data stream according to a corresponding image processing function.

[0030] In a possible implementation, before the routing switching instruction is sent to the first flow switching IP core at the programmable logic end, the method further includes:

[0031] determine whether the length of the routing switching instruction is greater than 8 bytes;

[0032] if the length is greater than 8 bytes, determine whether the length of the routing switching instruction is greater than 16 bytes;

[0033] if the length is greater than 16 bytes, determine whether to execute a reconfigurable image processing function according to the content of the first 8 bytes and the last 8 bytes of the routing switching instruction;

[0034] if it is determined to execute the reconfigurable image processing function, send the routing switching instruction to the first flow switching IP core at the programmable logic end; otherwise, acquire again the GVCP format control command data packet sent by the external host computer.

[0035] In a third aspect, an electronic device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the method of the second aspect when executing the computer program.

[0036] In the embodiment of the present application, the first stream exchange IP core and the multi-path image processing module are arranged at the programmable logic end; the first stream exchange IP core comprises a first input interface and a plurality of first output interfaces, each first output interface being connected with one image processing module; the first input interface receives the first image data stream; since the processing system end is connected with the first stream exchange IP core through the advanced extensible interface, the routing rules of the first stream exchange IP core can be dynamically configured through the advanced extensible interface, and the data stream can be flexibly routed between the input interface and the plurality of output interfaces; therefore, the first image data stream input from the first input interface can be transmitted to the corresponding image processing module from any one first output interface, the flexible data stream switching between the multi-path image processing modules is realized, the flexibility and reconfigurability of the system are enhanced; since each image processing module has at least one image processing function, the system can be applied to various application scenarios requiring real-time switching of image processing algorithms, the dynamic switching of the image processing algorithm and the dynamic reconfigurable template design of the flexible routing management of the data stream are realized, and thus the efficient and flexible image processing function is realized. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a structural schematic diagram of an image processing system of an industrial camera provided by an embodiment of the present application;

[0038] Figure 2 is a structural schematic diagram of an image processing system of an industrial camera provided by another embodiment of the present application;

[0039] Figure 3 is a structural schematic diagram of a reconfigurable image processing function module provided by an embodiment of the present application;

[0040] Figure 4 is a path block diagram of a reconfigurable image processing data stream provided by an embodiment of the present application;

[0041] Figure 5 is an interface design schematic diagram of a stream exchange IP core provided by an embodiment of the present application;

[0042] Figure 6 is a flow schematic diagram of an image processing method of an industrial camera provided by an embodiment of the present application;

[0043] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0045] At present, most industrial cameras are based on FPGA GigE camera image processing systems. Such image processing systems usually have the following defects: fixed image processing capability: fixed function image processing modules are integrated in hardware or firmware, which cannot be dynamically adjusted according to task requirements, such as switching white balance, gamma correction or edge detection algorithms. Poor flexibility: after the fixed function of the FPGA design is modified, if new processing algorithms need to be added, the hardware or firmware often needs to be redesigned, which has a long development cycle and high cost. This design without dynamic reconfigurable capability limits the adaptability of the camera. Long development cycle: traditional FPGA image processing module development relies on register transfer level (RTL) design, which has a long development cycle and is difficult to iterate quickly, increasing development costs.

[0046] Based on the above defects, the present application proposes the following idea: design the system architecture of the processing system (PS) end and the programmable logic (PL) end, the PL end develops multiple image processing modules using high-level synthesis (HLS), each image processing module integrates at least one image processing algorithm, and each image algorithm corresponds to an image processing function; and design two flow exchange IP cores, the first flow exchange IP core has one input interface and multiple output interfaces, each output interface is connected to the input end of an image processing module, and is used for path splitting of the input image data stream; the second flow exchange IP core has multiple input interfaces and one output interface, each input interface is connected to the output end of an image processing module, and one of the multiple inputs is selected as an output; the PS end is connected to each module of the PL end through a lightweight advanced extensible (AXI_Lite) interface, and the routing rules of the first flow exchange IP core and the second flow exchange IP core are dynamically configured, so as to control the opening of different channels according to the routing rules, realize the selective input of the data stream to the specified path output, and flexibly route the data stream transmission path, thereby realizing the flexible switching of multiple image processing algorithms.

[0047] The embodiments of the present application will be described in detail below with reference to the drawings.

[0048] Figure 1 FIG. 1 is a structural schematic diagram of an image processing system of an industrial camera according to an embodiment of the present application.

[0049] As shown in FIG. 1, the image processing system of the industrial camera according to the embodiment of the present application comprises a programmable logic end (PL end) and a processing system end (PS end). Figure 1 The PL end comprises a first stream switching IP core and a plurality of image processing modules, each of which has at least one image processing function. The first stream switching IP core comprises a first input interface and a plurality of first output interfaces, each of which is connected to one of the image processing modules. The first input interface receives a first image data stream. The PS end is connected to the first stream switching IP core through an advanced extensible interface, and is configured to dynamically configure a routing rule of the first stream switching IP core through the advanced extensible interface. The first stream switching IP core transmits the first image data stream to the corresponding image processing module through the corresponding first output interface according to the routing rule.

[0050] It should be noted that the PL end is an FPGA programmable logic end, the PS end is an ARM processing system end, the first stream switching IP core is an advanced extensible interface stream switch (AXI-Stream Switch 0 for short), which is a device for realizing data switching and routing between a plurality of AXI Stream interfaces and has an AXI4_Lite interface through which the routing rule is configured.

[0051] It should be noted that each image processing module can support a plurality of image processing algorithms at the same time, realize the image processing function of one scene, and also support direct data stream transmission, and usually has at least one image processing function.

[0052] In the embodiment, as shown in FIG. 1, the image processing system of the industrial camera comprises a programmable logic end (PL end) and a processing system end (PS end). Figure 1As shown, the solid arrow represents the data flow direction, and the dashed arrow represents the control flow direction of the PS end; the processor of the PS end configures the routing rule of the first stream switching IP core (AXI-Stream Switch 0) through the AXI4_Lite interface, and the routing rule includes the mapping relationship from the source port to the target port, wherein the source port is the first input port, and the target port is one of the plurality of first output ports; the AXI-Stream Switch 0 inputs the received first image data stream to the image processing module connected with the target port through the target port according to the routing rule, the image processing module processes the first image data stream based on the image processing algorithm, and outputs the processed second image data stream, and the second image data stream is sent to the display module of the PC host computer (hereinafter referred to as PC host computer) through the processor of the PS end via the gigabit Ethernet, so that the user can view the processed image.

[0053] For example, the AXI-Stream Switch 0 includes a first input interface (00), five first output interfaces (00, 01, 02, 03, and 04), and the multi-path image processing module includes five image processing modules connected with the five first output interfaces of the AXI-Stream Switch 0; the mapping relationship from the source port to the target port specified by the routing rule of the AXI-Stream Switch 0 configured by the processor of the PS end is 00-04, and the AXI-Stream Switch 0 transmits the received first image data stream to the fifth image processing module through the interface 04, and the fifth image processing module processes the first image data stream with corresponding functions; when it is needed to switch to other image processing function algorithms, the processor of the PS end can realize flexible switching of the image data stream transmission path by reconfiguring the routing rule of the AXI-Stream Switch 0 through the AXI4_Lite interface, thereby realizing dynamic switching of the image data stream among the multi-path image processing modules.

[0054] In the embodiment, the first stream switching IP core and the multi-path image processing module are arranged at the programmable logic end; the first stream switching IP core comprises a first input interface and a plurality of first output interfaces, each first output interface is connected with an image processing module; the first input interface receives the first image data stream; since the processing system end is connected with the AXI4_Lite interface of the first stream switching IP core through the advanced extensible interface, the routing rules of the first stream switching IP core can be dynamically configured through the AXI4_Lite interface, and the data stream can be flexibly routed between the input interface and the plurality of output interfaces; therefore, the first image data stream input from the first input interface can be transmitted to the corresponding image processing module from any one of the first output interfaces, flexible data stream switching between the multi-path image processing modules is realized, the flexibility and reconfigurability of the system are enhanced; since each image processing module has at least one image processing function, the system can be applied to various application scenarios requiring real-time switching of image processing algorithms, dynamic switching of image processing algorithms and dynamic reconfigurable template design of flexible routing management of data streams are realized, and therefore efficient and flexible image processing functions are realized.

[0055] Figure 2 is a structural schematic diagram of an image processing system of an industrial camera provided by another embodiment of the present application, and the embodiment is used to completely describe the image processing system of the industrial camera.

[0056] It should be noted that the Figure 2 solid arrows represent data flow directions, and the dashed arrows represent control flow directions.

[0057] As shown in Figure 2 , the image processing system of the industrial camera provided by the embodiment comprises the modules in the embodiment shown in Figure 1 , and the programmable logic end further comprises a second stream switching IP core (AXI-Stream Switch 1); the second stream switching IP core comprises a plurality of second input interfaces and a second output interface; each second input interface is connected with an image processing module and is used to receive the second image data stream output by each image processing module; the processing system end is connected with the second stream switching IP core through the advanced extensible interface and is used to dynamically configure the routing rules of the second stream switching IP core, so as to transmit the second image data stream output by each image processing module to the next module through the second output interface.

[0058] In the embodiment, the second stream switch IP core (AXI-Stream Switch 1) includes a plurality of second input interfaces and a second output interface, each input interface is connected to an image processing module, the input and output interfaces are designed reversely to the first stream switch IP core (AXI-Stream Switch 0), the processor of the PS end is connected to the AXI4_Lite interface of the second stream switch IP core (AXI-Stream Switch 1) through the AXI interface, the routing rules of the second stream switch IP core AXI-Stream Switch 1 are configured through the AXI4_Lite interface, the mapping relationship from the source port to the target port contained in the routing rules is also reversed to the routing rules of the first stream switch IP core (AXI-Stream Switch 0), the source port is one of the plurality of second input interfaces, and the target port is the second output interface.

[0059] In the embodiment, the first stream switch IP core (AXI-Stream Switch 0), the plurality of image processing modules and the second stream switch IP core (AXI-Stream Switch 1) jointly constitute a reconfigurable image processing functional module, the routing rules of the first stream switch IP core (AXI-Stream Switch 0) and the second stream switch IP core (AXI-Stream Switch 1) are dynamically configured through the PS end, the dynamic routing of the image data stream can be realized, and thus the dynamic switching of the image data stream between the plurality of image processing modules can be realized.

[0060] For example, the reconfigurable image processing functional module is as shown in Figure 3As shown, the first stream switching IP core (AXI-StreamSwitch 0) includes a first input interface (00), five first output interfaces (respectively marked as 00, 01, 02, 03, and 04), and the first stream switching IP core (AXI-Stream Switch 0) splits the data flow into paths (1 input to 5 outputs). Each output is connected to an image processing module, which are module A, module B, module C, module D, and module E. Each image processing module integrates at least one functional algorithm; the second stream switching IP core (AXI-Stream Switch 1) includes five second input interfaces (respectively marked as 00, 01, 02, 03, and 04) and a second output interface (marked as 00). Each input interface of the second stream switching IP core (AXI-Stream Switch 1) is connected to an image processing module, and the 5 inputs are combined into 1 output. If the routing rules of the first stream switching IP core configured by the PS processor specify a source port to destination port mapping relationship of 00-04, then the routing rules of the second stream switching IP core configured by the PS processor specify a source port to destination port mapping relationship of 04-00; the first stream switching IP core transmits the received first image data stream to the fifth image processing module E via interface 04 (as indicated by the red solid arrow). The fifth image processing module E performs corresponding functional processing on the first image data stream, resulting in a processed second image data stream that is transmitted to interface 04 of the second stream switching IP core (as indicated by the red solid arrow). Finally, the second image data stream is transmitted to the next module via the second output interface 00. When switching to other image processing algorithms is required, the PS processor reconfigures the routing rules of AXI-Stream Switch 0 and AXI-Stream Switch 1 via the AXI4_Lite interface to achieve flexible switching of image data stream transmission paths, thereby enabling dynamic switching of image data streams between multiple image processing modules.

[0061] It should be noted that the number of image processing modules can be designed according to actual application requirements, and the number of image processing algorithms supported by each image processing module can also be designed according to actual application requirements to achieve different algorithm combinations. For example, white balance, gamma correction, edge detection and other algorithm functions can be arbitrarily combined into one image processing module to meet various image processing requirements.

[0062] In one possible implementation, Figure 2As shown, the programmable logic end (PL) further comprises a direct memory access module connected with the second output interface; the processing system end comprises a storage controller and a memory connected with the storage controller; the direct memory access module is further connected with the storage controller through an advanced extensible interface, for storing the second image data stream into the memory through the storage controller. Wherein, the storage controller is a double data rate (DDR) storage controller (namely, a DDRC storage controller), and the memory is a DDR3 memory.

[0063] Specifically, as shown in the figure, Figure 2 the direct memory access (VDMA) module (hereinafter referred to as VDMA module) is connected with a high-performance interface (S-AXI-HP) of a slave advanced extensible interface of the PS end through an intelligent interconnection interface (AXI_Smart Connect interface) of the AXI interface, and then the S-AXI-HP interface is connected with a DDRC storage controller of the PS end, and the VDMA module directly writes the second image data stream output by the second stream switching IP core into the DDRC storage controller, and then the DDRC storage controller stores the second image data stream into a DDR3 memory.

[0064] In this embodiment, the VDMA module is mainly used for realizing high-speed and efficient data transfer between the DDR3 memory of the PS end and the image processing module of the PL end, so as to avoid the overhead of the processor participating in data transmission.

[0065] In a possible implementation manner, the processing system end (PS) further comprises a processor connected with the storage controller; the processor is used for reading the second image data stream in the storage controller, and sending the second image data stream to an external host computer through a gigabit Ethernet.

[0066] Specifically, as shown in the figure, Figure 2 the processor acquires the second image data stream in the DDR3 memory through the DDRC storage controller, and then sends the second image data stream to a display module of a PC host computer through a gigabit Ethernet, so as to facilitate a user to view the processed image data.

[0067] In this embodiment, the DDRC storage controller is used for managing and controlling the DDR3 memory, and is responsible for managing and scheduling read and write operations on the DDR3 memory, so as to ensure efficient and reliable data transmission between the DDR3 memory and the processor.

[0068] In a possible implementation, the processor comprises a data encapsulation module and a data packet processing module connected with the data encapsulation module; the data encapsulation module is configured to encapsulate the second image data stream based on the GigE Vision streaming protocol and send the encapsulated image data to the data packet processing module; and the data packet processing module is configured to send the encapsulated image data to an external PC through the Gigabit Ethernet.

[0069] Specifically, as shown in Figure 2 the data encapsulation module is a data encapsulation module based on the GigE Vision streaming protocol (GVSP) (hereinafter referred to as GVSP data encapsulation module), which is connected with the DDRC storage controller; after the DDRC storage controller transmits the second image data stream to the GVSP data encapsulation module in the processor, the GVSP data encapsulation module packs and organizes the second image data stream according to the GVSP protocol to obtain a series of data packets for transmission in the network; and the encapsulated image data packets are transmitted to the data packet processing module, which transmits the encapsulated image data packets to the PC to display.

[0070] It should be noted that the data packet processing module is a User Datagram Protocol Server (UDP-Server) (hereinafter referred to as UDP server), which interacts with the UDP client program running on the PC as a data sender or receiver. Figure 2

[0071] In this embodiment, the GVSP data encapsulation enables the image data to be transmitted in the Gigabit Ethernet network in an efficient and reliable manner, ensuring that the receiver can accurately parse and restore the original image information, thereby realizing the rapid acquisition and processing of image data in the machine vision system.

[0072] In a possible implementation, the processor is connected with the advanced extensible interface of the programmable logic end through a general interface of the advanced extensible interface of the host device; and the advanced extensible interface of the processor is connected with each module of the programmable logic end.

[0073] Specifically, as shown in Figure 2 ​As shown, the Master Advanced eXtensible Interface General Purpose (M-AXI-GP) interface is connected with an Advanced eXtensible Interface Interconnect (AXI Interconnect) interface, and the AXI Interconnect interface is connected with each module at the PL end.

[0074] In this embodiment, the PS end configures the IP core in the PL end through the M-AXI-GP interface, and the VDMA module directly accesses the DDR3 memory of the PS end through the S-AXI-HP interface to realize efficient data transmission.

[0075] In a possible implementation, as shown in Figure 2 The programmable logic end (PL) further includes an image data acquisition and conversion module and a data stream conversion module connected with the image data acquisition and conversion module; the data stream conversion module is further connected with a first input interface of the first stream exchange IP core; the image data acquisition and conversion module is connected with an image sensor, and is configured to acquire original image data obtained by the image sensor, and perform format conversion on the original image data to obtain an original image data stream in a target format; and the data stream conversion module is configured to convert the original image data stream in the target format into a first image data stream based on an AXI4-Stream protocol.

[0076] Specifically, the image sensor acquires an original image data stream, the image data acquisition and conversion module acquires the original image data stream from the image sensor, and performs signal synchronization and format conversion processing on the acquired original image data stream, for example, converts the original image data stream in an RGB565 format (16 bits / pixel) into an image in an RGB888 format (24 bits / pixel), and finally outputs the original image data stream in the RGB888 format (24 bits / pixel) (a data stream of a pixel clock, a line synchronization, and a field synchronization) to the data stream conversion module; the data stream conversion module converts the original image data stream in the RGB888 format (24 bits / pixel) into an AXI4-Stream (i.e., the first image data stream) to facilitate the internal modules at the PL end to transmit data in a "stream" mode.

[0077] In a possible implementation, as shown in Figure 2 The processor of the PS end further includes a camera device information configuration module and a GVCP control protocol analysis module connected with a UDP server; where GVCP refers to a GigE Vision Control Protocol (GigE Vision Control Protocol).

[0078] Specifically, the user camera device information configuration module inputs a parameter configuration instruction, the GVCP control protocol analysis module analyzes the parameter configuration instruction based on the GVCP control protocol, obtains configuration information (such as resolution, frame rate, gain, exposure parameter, and the like) after analysis, and then transmits the configuration information after analysis to the image sensor through a serial camera control bus (SCCB) to configure the image sensor register (such as resolution, frame rate, gain, and the like), so as to realize adjustment of the camera parameter.

[0079] In a possible implementation, the multi-path image processing module includes one direct data stream module; each image processing module other than the direct data stream module includes at least one image processing unit, and each image processing unit has one image processing function; and the direct data stream module is configured to directly transmit the first image data stream.

[0080] In this embodiment, each image processing module can support multiple image processing algorithms, implement image processing functions in one scene, not only can dynamically switch between the multi-path image processing modules, but also can dynamically switch different image processing algorithms according to actual needs, and is suitable for various application scenarios that need to switch image processing algorithms in real time; and by designing at least one image processing unit in each image processing module, each image processing unit integrates one image processing algorithm to implement one image processing function, and the combination of multiple image processing algorithms is also implemented to adapt to various application scenarios that need to process images by multiple algorithms.

[0081] For example, as shown in FIG. 5, there are five image processing modules in total, of which the first image processing module is a direct data stream module, that is, the image data is not processed at all, and the image data stream is directly transmitted to the next module; of the remaining four image processing modules, the second image processing module includes an image filtering unit, the third image processing module includes a white balance unit, a gamma correction unit and an edge detection unit, the fourth image processing module includes an adaptive binaryzation unit, and the fifth image processing module includes an image enhancement unit and an edge detection unit. Figure 4 When image enhancement processing is needed, the routing rules of the first stream switching IP core and the second stream switching IP core are set to switch the image data stream to the second image processing module; when white balance, gamma correction and edge detection processing are needed, the routing rules are reconfigured to switch the image data stream to the third image processing module, and so on.

[0082] It should be noted that the number of image processing modules and the combination of different algorithms are only illustrative, and in actual implementation, the number of image processing modules, the number of image processing units included in each image processing module, and how to combine algorithms between different image processing units can be determined according to actual scene requirements.

[0083] In order to better understand the present application, the following will be described in combination with Figure 4 The paths of the reconfigurable image processing data flow of the entire image processing system are explained. This embodiment is explained by taking 5 paths of image transmission modules as an example, and in actual application, the number of paths of image transmission modules and the number of image processing units of each path of image processing modules can be set according to different scene requirements.

[0084] As shown in Figure 4 Path 1 (direct transmission path): image sensor→image data acquisition conversion module→data flow conversion module→AXI-Stream Switch 0 (1 to 5)→four-to-one (such as selecting an edge detection algorithm unit)→AXI-Stream Switch 1 (5 to 1)→VDMA→DDR3 memory→GVSP data packaging module→UDP server→gigabit Ethernet→PC host display; this path 1 is suitable for high-speed transmission scenes without image processing.

[0085] Path 2 (reconfigurable image processing path): image sensor→image data acquisition conversion module→data flow conversion module→AXI-Stream Switch 0 (1 to 5)→four-to-one (such as selecting an edge detection algorithm unit)→AXI-Stream Switch 1 (5 to 1)→VDMA→DDR3 memory→GVSP data packaging module→UDP server→gigabit Ethernet→PC host display; this path 2 is suitable for scenes requiring corresponding algorithm processing of images.

[0086] Path switching: the routing rules of Switch 0 and Switch 1 are dynamically configured through the AXI4_Lite interface, supporting multi-task scenarios, such as enabling image enhancement processing, edge detection, or direct data flow display modules to process different image data flows, respectively, and users can independently design the two IP cores of Switch 0 and Switch 1.

[0087] In order to more clearly understand the design of the reconfigurable image processing function in the image processing system of the present application, the interface design and parameter configuration of the flow exchange IP core are described below.

[0088] It should be noted that the AXI-Stream Switch IP core mainly introduces three configuration parameters:

[0089] From Interface Number: This parameter specifies the number of AXI4-Stream slave interfaces present on the IP, which is used to represent the total number of connectable slave interfaces; for AXI-Stream Switch 0, the number of slave interfaces is 1; for AXI-Stream Switch 1, the number of slave interfaces is multiple, such as 5.

[0090] Master Interface Number: This parameter specifies the number of AXI4-Stream master interfaces present on the IP, which is used to represent the total number of connectable master interfaces; for AXI-Stream Switch 0, the number of master interfaces is configured to be multiple, for example, 5; for AXI-Stream Switch 1, the number of master interfaces is configured to be 1.

[0091] Control Register Routing: This parameter specifies the routing module, and the control register routing enables the AXI4-Lite control register interface to handle the transmission routing; in actual use, this parameter is set to "Yes", and the PS is used to flexibly configure the data flow path through the AXI4-Lite interface.

[0092] Figure 5 is an interface design schematic diagram of a stream switching IP core provided by an embodiment of the present application, Figure 5 a in the figure is an interface design of a first stream switching IP core (AXI-Stream Switch 0), Figure 5 b in the figure is an interface design of a second stream switching IP core (AXI-Stream Switch 1). This embodiment takes a 5-way image processing design as an example to illustrate the interfaces of the two stream switching IP cores.

[0093] It should be noted that the AXI-Stream Switch IP core is a high-performance data flow routing module for FPGA design, which is based on the AXI4-Stream protocol. It is mainly used to flexibly route data flow between multiple AXI4-Stream input ports (Slave Interfaces, abbreviated as SI) and output ports (Master Interfaces, abbreviated as MI). The core function of the AXI-Stream Switch IP core is to realize dynamic or static routing of AXI4-Stream data flow, which is similar to a hardware data switch. It can selectively route data flow from multiple input data sources to one or more output targets. The PS flexibly configures the data flow path through the AXI4-Lite control interface, and the dynamic routing is suitable for scenarios that need to switch data sources in real time.

[0094] As Figure 5AXI-Stream Switch 0 IP core: input interface: 1 AXI4-Stream Slave interface (AXI4-Stream Slave interface, abbreviated as S_AXIS

[00] in the figure), connected to the AXI stream data stream output by the upstream module.

[0095] Output interface: 5 AXI4-Stream Master interfaces (AXI4-Stream Master interface, abbreviated as M_AXIS

[00] , M_AXIS

[01] , M_AXIS

[02] , M_AXIS

[03] , and M_AXIS

[05] in the figure), supporting 1-to-5 downstream module AXI stream data stream output.

[0096] Control interface: AXI4-Lite interface (abbreviated as AXI4-Lite interface, denoted as S_AXI_CTRL in the figure) connected with the AXI Interconnect interface, controlled by the processor through the AXI bus, used for dynamically configuring the route.

[0097] ap_clk is a clock interface for providing a synchronous clock signal; ap_rst_n is a reset interface for providing a reset signal.

[0098] As shown in Figure 5 AXI-Stream Switch 1 IP core: input interface: 5 AXI4-Stream Slave interfaces (S_AXIS

[00] , S_AXIS

[01] , S_AXIS

[02] , S_AXIS

[03] , and S_AXIS

[04] ), connected to the AXI stream data stream output by the image processing module.

[0099] Output interface: 1 AXI4-Stream Master interface (M_AXIS

[00] ), supporting 5-to-1 output, connected to the next module (VDMA).

[0100] Control interface: AXI4-Lite interface (S_AXI_CTRL), connected with the AXI Interconnect interface, used for dynamically configuring the route.

[0101] ap_clk is a clock interface for providing a synchronous clock signal; ap_rst_n is a reset interface for providing a reset signal.

[0102] The image processing system provided in the embodiment of the application is responsible for path shunting of a data stream according to an external instruction, and realizes flexible data stream management in combination with an AXI-Stream Switch. After the PS end receives an external routing switching instruction, the data stream path is flexibly configured through an AXI4-Lite control interface of the AXI-Stream Switch, so as to switch different image processing algorithm modules, realize the function of freely and dynamically switching the data stream routing, and be suitable for a scene requiring real-time switching of a data source, and the system design has good flexibility and reconfigurability.

[0103] The specific process that the PS end realizes the data stream path switching according to the routing switching instruction will be described below in combination with the flowchart. Figure 6

[0104] Figure 6 The flowchart is a flowchart of an image processing method of an industrial camera provided in an embodiment of the application, and the execution subject of the method is a processor in an image processing system of the industrial camera.

[0105] In step S601, a GVCP format control command data packet sent by an external host computer is acquired; the GVCP format control command data packet is configuration data input by a user on the host computer.

[0106] In this step, when the user needs to adjust the image processing algorithm, the corresponding configuration data is input through the PC host computer, and the configuration data is transmitted in the form of a GVCP format control command data packet; the processor of the PS end listens to the port of the PC host computer client in real time and receives the data packet sent by the client.

[0107] In step S602, the GVCP format control command data packet is parsed to obtain a routing switching instruction.

[0108] In this step, the processor parses the data packet to obtain the content of the routing switching instruction, judges whether the reconfigurable image processing function (i.e., switching the image processing algorithm) needs to be executed, and if yes, continues to execute the subsequent step S603.

[0109] It should be noted that the specific implementation manner of judging whether the reconfigurable image processing function needs to be executed in this step will be described in detail in subsequent embodiments.

[0110] In step S603, the routing switching instruction is sent to a first stream switching IP core of a programmable logic end, so as to configure the routing rule of the first stream switching IP core, so that the first stream switching IP core transmits the received first image data stream to a corresponding image processing module through a corresponding first output interface according to the routing rule, and the corresponding image processing module processes the first image data stream according to the corresponding image processing function. ​

[0111] In a possible implementation, the processor sends the routing switching instruction to the first stream switching IP core and the second stream switching IP core of the programmable logic end, configures routing rules of the two IP cores, and transmits, according to the routing rules, the received first image data stream to a corresponding image processing module through a corresponding first output interface by the first stream switching IP core, processes the first image data stream by the corresponding image processing module, and outputs a second image data stream, which is transmitted to a next module by the second stream switching IP core.

[0112] In the embodiment, the user sends different configuration data through the PC host computer, the processor drives the dynamic configuration routing, controls the routing switching of the image data stream, switches different algorithm modules, realizes the free dynamic switching of the data stream routing function, ensures that there is no data stream interruption in the switching process, and guarantees the efficiency, real-time performance and stability of the algorithm switching.

[0113] In a possible implementation, before the routing switching instruction is sent to the first stream switching IP core of the programmable logic end, the method further includes: determining whether the length of the routing switching instruction is greater than 8 bytes; if greater than 8 bytes, determining whether the length of the routing switching instruction is greater than 16 bytes; if greater than 16 bytes, determining whether to execute the reconfigurable image processing function according to the content of the first 8 bytes and the last 8 bytes of the routing switching instruction; if it is determined to execute the reconfigurable image processing function, sending the routing switching instruction to the first stream switching IP core of the programmable logic end; otherwise, reacquiring the data packet sent by the external host computer.

[0114] Specifically, after receiving a data packet, the PS processor parses the packet for GVCP control commands and extracts the routing switch instructions. Routing switch instructions are defined as a fixed length (typically greater than or equal to 16 bytes), with the first eight bytes representing the standard GVCP write register header (i.e., the GVCP packet header), and the following eight bytes representing the write register address and register contents. For reconfigurable image processing, the custom reconfigurable path configuration boot register is 0x0000B000, and the last four bytes contain the field for selecting the path number. For example, the complete write register command (WRITEREG_CMD) is: 0x42 010082 00 08AC AC 00 00 0B00 00 00 00 01, where 00 00 0B00 represents the reconfigurable path configuration boot register and 00 00 00 01 represents the path number (01 refers to path 1). Therefore, after parsing the route switching instruction, the first step is to determine whether its length is greater than or equal to 8 bytes. If it is less than 8 bytes, it indicates that the instruction is incorrect or is not a true route switching instruction, and the process ends. If it is greater than or equal to 8 bytes, the further step is to determine whether its length is greater than or equal to 16 bytes. If it is less than 16 bytes, it indicates that the instruction is not a true route switching instruction, and the process ends. If it is greater than or equal to 16 bytes, the first 8 bytes of the GVCP packet header are used to determine whether it is a write register command (WRITEREG_CMD), and then the register content of the last 8 bytes is parsed. If the first 8 bytes are not a write register command and the last 8 bytes are not a reconfigurable path configuration guide register, the reconfigurable image processing function is not executed. If so, the reconfigurable image processing function is executed, that is, the route switching instruction is sent to the first stream switching IP core and the second stream switching IP core on the programmable logic side to configure the routing rules of the first stream switching IP core and the second stream switching IP core to achieve dynamic data flow switching between multiple image processing modules.

[0115] It should be noted that, in its implementation, the first step in reconfigurable image processing is to initialize the AXI-Stream Switch instance, ensuring that Switch 0 and Switch 1 function properly and support subsequent routing configuration. This initialization involves configuring the parameters of AXI-StreamSwitch 0 and AXI-StreamSwitch 1 (number of slave interfaces, number of master interfaces, and the AXI4-Lite control interface). The configuration process is described in the aforementioned embodiments and will not be repeated here.

[0116] The second step of reconfigurable image processing function: AXI-Stream Switch IP core, write routing rules into the register of AXI-Stream Switch IP core through AXI4-Lite interface, adjust the data stream path; when PS writes data to AXI4-Lite interface, store the data transmitted from the interface into the path configuration register of AXI-Stream Switch IP core, and switch the path.

[0117] The third step of reconfigurable image processing function: realize the execution processing of routing switching instruction and dynamic switching of data stream path, the PS end dynamically configures the routing of two AXI-Stream Switch IP cores by parsing the received routing switching instruction, supports multiple image processing modes (such as Sobel edge detection, adaptive binarization, direct data stream display, image enhancement, etc.). For example, when the path number is 1, the PS end determines the target path and functional module of the data stream, triggers the subsequent AXI4-Lite bus operation, and the AXI4-Lite configuration configures the slave interface register of AXI-Stream Switch 0 to 0x00 and the master interface register to 0x01, that is, S_AXIS

[00] is connected to M_AXIS

[01] ; similarly, S_AXIS

[01] of AXI-Stream Switch 1 is connected to M_AXIS

[00] , realizing the switching of the path.

[0118] In this embodiment, the system designs a kind of on-chip interconnection global dynamic reconfigurable technology, which can dynamically change the ability of on-chip interconnection structure when the system runs, to meet the requirements and functional needs of different applications, thereby improving the adaptability of the system. Specifically, the PS end dynamically configures the routing of the first and second stream switching IP cores through the AXI4_Lite interface, realizes data stream path selection and function switching. The parameter adjustment takes effect in real time, without interrupting data stream transmission and display, and after the routing adjustment is completed, the PS end continues to listen to the instruction information sent by the client. To achieve the switching of different algorithm modules, dynamic routing is suitable for scenarios that need to switch data sources in real time, and realizes the reconfigurability of the system. The PS end dynamically configures the routing rules and reconfigurable image processing function module parameters of AXI-Stream Switch through AXI4-Lite, realizes data stream distribution and function switching. The low-speed control of AXI4-Lite ensures stability and accuracy, combined with part of the reconfiguration technology, significantly improves the flexibility and efficiency of the system, and can be applied to industrial detection, automation production, medical imaging, security monitoring and other diversified application scenarios.

[0119] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 7 the electronic device 7 of this embodiment includes a processor 70 and a memory 71. The memory 71 stores a computer program 72. The processor 70 implements the steps in each of the above method embodiments when executing the computer program 72. Alternatively, the processor 70 implements the functions of each module / unit in each of the above device embodiments when executing the computer program 72.

[0121] For example, the computer program 72 can be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 72 in the electronic device 7.

[0122] The electronic device 7 can include, but is not limited to, the processor 70 and the memory 71. Those skilled in the art can understand that Figure 7 The electronic device 7 is only an example and does not constitute a limitation on the electronic device 7, and can include more or fewer components than shown, or combine certain components, or different components, for example, the electronic device 7 can also include an input / output device, a network access device, a bus, etc.

[0123] The processor 70 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0124] The memory 71 can be an internal storage unit of the electronic device 7, such as a hard disk or a memory of the electronic device 7. The memory 71 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Further, the memory 71 can include both the internal storage unit and the external storage device of the electronic device 7. The memory 71 is used to store the computer program 72 and other programs and data required by the electronic device 7. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0125] For the convenience and brevity of description, only the above-mentioned division of functional modules / units is exemplified, and in actual application, the above-mentioned functions can be completed by different functional modules / units according to needs. The above-mentioned modules / units can be realized in the form of hardware, software or a combination of hardware and software.

[0126] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method in each method embodiment described above is realized.

[0127] The embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method in each method embodiment described above is realized.

[0128] The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0129] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments. If there is no special description and no logical conflict, the terms and / or descriptions of different embodiments are consistent and can be mutually referenced. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0130] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An image processing system for an industrial camera, characterized in that: include: Programmable logic side and processing system side; The programmable logic end includes: a first stream switching IP core and multiple image processing modules; wherein each image processing module has at least one image processing function; The first stream switching IP core includes a first input interface and multiple first output interfaces, each first output interface is connected to an image processing module; the first input interface receives a first image data stream; The processing system end is connected to the first flow switching IP core via an advanced extensible interface, and is used to dynamically configure routing rules of the first flow switching IP core via the advanced extensible interface; The first stream switching IP core transmits the first image data stream to the corresponding image processing module through the corresponding first output interface according to the routing rule.

2. The image processing system for an industrial camera according to claim 1, characterized in that: The programmable logic end further includes: a second stream switching IP core; The second stream exchange IP core includes a plurality of second input interfaces and second output interfaces; each second input interface is connected to an image processing module, and is used to receive the second image data stream output by each image processing module; The processing system end is connected to the second stream switching IP core through an advanced extensible interface, and is used to dynamically configure the routing rules of the second stream switching IP core to transmit the second image data stream output by each image processing module to the next module through the second output interface.

3. The image processing system for an industrial camera according to claim 1, characterized in that: The multi-channel image processing module includes a direct data stream module; each image processing module other than the direct data stream module includes at least one image processing unit, and each image processing unit has an image processing function; wherein, the direct data stream module is used to directly transmit the first image data stream.

4. The image processing system for an industrial camera according to claim 2, characterized in that: The programmable logic terminal further includes: a direct memory access module connected to the second output interface; The processing system end includes a storage controller and a memory connected to the storage controller; The direct memory access module is further connected to the storage controller via an advanced extensible interface, and is configured to store the second image data stream into the memory via the storage controller.

5. The image processing system for an industrial camera according to claim 4, characterized in that: The processing system end further includes: a processor connected to the storage controller; The processor is used to read the second image data stream in the storage controller and send the second image data stream to an external host computer via Gigabit Ethernet.

6. The image processing system for an industrial camera according to claim 5, characterized in that: The processor includes: a data encapsulation module and a data packet processing module connected to the data encapsulation module; The data encapsulation module is used to encapsulate the second image data stream based on the GigE Vision stream transmission protocol, and send the encapsulated image data to the data packet processing module; The data packet processing module is used to send the encapsulated image data to an external host computer via Gigabit Ethernet.

7. The image processing system for an industrial camera according to claim 5, characterized in that: The processor is connected to the interconnection interface of the advanced extensible interface of the programmable logic end through the universal interface of the master device advanced extensible interface; the interconnection interface of the advanced extensible interface is connected to each module of the programmable logic end.

8. The image processing system for an industrial camera according to any one of claims 1 to 7, characterized in that: The programmable logic terminal further includes: an image data acquisition and conversion module and a data stream conversion module connected to the image data acquisition and conversion module; the data stream conversion module is also connected to the first input interface of the first stream exchange IP core; The image data acquisition and conversion module is connected to the image sensor, and is used to acquire the original image data obtained by the image sensor and perform format conversion on the original image data to obtain an original image data stream in a target format; The data stream conversion module is used to convert the original image data stream in the target format into the first image data stream based on the Advanced Extensible Interface 4-Stream protocol.

9. An image processing method for an industrial camera, applied to a processor in an image processing system of an industrial camera, characterized in that: The method comprises: Obtaining a GVCP format control command data packet sent by an external host computer; the GVCP format control command data packet is configuration data input by the user in the host computer; Parsing the GVCP format control command data packet to obtain a routing switching instruction; The routing switching instruction is sent to the first stream switching IP core at the programmable logic end to configure the routing rules of the first stream switching IP core, so that the first stream switching IP core transmits the received first image data stream to the corresponding image processing module through the corresponding first output interface according to the routing rules, so that the corresponding image processing module processes the first image data stream according to the corresponding image processing function.

10. The image processing method for an industrial camera according to claim 9, characterized in that: Before sending the routing switching instruction to the first stream switching IP core of the programmable logic end, the method further includes: Determine whether the length of the routing switching instruction is greater than 8 bytes; If it is greater than 8 bytes, determining whether the length of the routing switching instruction is greater than 16 bytes; If it is greater than 16 bytes, determining whether to execute the reconfigurable image processing function according to the contents of the first 8 bytes and the last 8 bytes of the routing switching instruction; If it is determined to execute the reconfigurable image processing function, the routing switching instruction is sent to the first stream switching IP core of the programmable logic end; otherwise, the GVCP format control command data packet sent by the external host computer is re-acquired.

Images