Real-time multi-target detection system based on ZYNQ

Abstract

The application discloses a real-time multi-target detection system based on ZYNQ, which comprises a host computer, an image acquisition module and a ZYNQ board card; the image acquisition module is used for completing Bayer interpolation processing by means of an ISP module of a CMOS camera, and outputs original video stream data in RGB565 format after color restoration; the ZYNQ board card comprises an FPGA end and an ARM end, wherein the FPGA end comprises a pixel filtering enhancement module, an adaptive threshold calculation module, a connected domain marking module and an AXI uplink module; the pixel filtering enhancement module is connected with the adaptive threshold calculation module and the connected domain marking module respectively, the adaptive threshold calculation module is connected with the connected domain marking module, and the connected domain marking module is connected with the AXI uplink module; the ZYNQ board card outputs bounding box position information and effective area information; and the application has the advantages of low resource consumption, high real-time performance, high flexibility and high stability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-target detection, and particularly relates to a real-time multi-target detection system based on ZYNQ. BACKGROUND

[0002] With the rapid development of computer vision and Internet of Things technology, target detection technology has wide application prospects in industrial production, security monitoring, automatic driving and other fields. However, in complex scenes, the current detection equipment often faces the problems of poor real-time performance and low precision, which limits its application in practice.

[0003] For example, in cloth defect detection, in order to improve efficiency, the cloth runs at a very fast speed in the transmission shaft, so that the defect part may exist in only 1-2 frames in the video image collected by the camera, which has quite high requirements for the real-time performance of the detection system. Although the image processing system based on ARM can meet the high complexity processing task, because the processor sampling rate is low and the structure of the serial processing instruction task is difficult to meet the requirements of high real-time performance and high bandwidth. SUMMARY

[0004] The purpose of the application is to provide a real-time multi-target detection system based on ZYNQ, which places the video image processing part collected in the ZYNQ architecture to realize acceleration, obtains the bounding box position information, realizes system task scheduling and host computer communication through the ARM end, can store the current frame image when the object is detected, achieves the purpose of real-time processing and automatic capture and storage of target image, and has the flexibility advantage that both software and hardware can be programmed.

[0005] The technical scheme of the application is a real-time multi-target detection system based on ZYNQ, comprising a host computer, an image acquisition module and a ZYNQ board card.

[0006] The image acquisition module is used to complete Bayer interpolation processing by using the ISP module of the CMOS camera, and outputs the original video stream data in RGB565 format after color restoration.

[0007] The ZYNQ board card comprises an FPGA end and an ARM end, wherein the FPGA end comprises a pixel filtering enhancement module, an adaptive threshold calculation module, a connected domain marking module and an AXI uplink module; the pixel filtering enhancement module is connected with the adaptive threshold calculation module and the connected domain marking module respectively, the adaptive threshold calculation module is connected with the connected domain marking module, and the connected domain marking module is connected with the AXI uplink module.

[0008] The ARM end is used for overall task scheduling, and a DMA channel is opened to complete video path control; a threshold parameter of a connected domain marking module is changed according to an instruction of an upper computer, and a maximum number of detection bounding boxes is set according to a requirement of an application scene; and an initial bias value of an adaptive threshold calculation module is set.

[0009] The pixel filtering enhancement module is configured to receive the original video stream data of the image acquisition module, filter and denoise the original video stream data using a switch-type median-Gaussian filtering algorithm, enhance the data using a Retinex algorithm, and perform color space conversion to obtain enhanced gray video stream data.

[0010] The adaptive threshold calculation module is configured to count the enhanced gray video stream data and calculate a maximum inter-class variance to obtain an adaptive global threshold.

[0011] The connected domain marking module is configured to convert the enhanced gray video stream data into binary video stream data using the adaptive global threshold, perform multi-target positioning, generate a bounding box of each target, and output position information and effective area information of the bounding box.

[0012] The AXI uplink module is configured to receive the position information of the bounding box from the connected domain marking module, package the position information into an AXI protocol, and store the position information in a specified address for reading by the ARM end.

[0013] Further, the pixel filtering enhancement module includes a pixel splicing circuit, a video frame storage circuit, a switch-type median-Gaussian filtering circuit, a Retinex enhancement unit, and a color space conversion circuit.

[0014] The original video stream data in the RGB565 format is input into the pixel filtering enhancement module, and during a high period of an input row valid signal HREF, one standard spliced video stream data pixel_data is output every two clocks through the pixel splicing circuit.

[0015] The obtained pixel_data is buffered in a DDR4 through the video frame storage circuit, and a standard video interface with a video valid signal vga_de, a field valid signal vga_vs, and a row valid signal vga_hs is generated according to a VGA video standard protocol; the video stream data buffered in the DDR4 is read using the video interface to obtain standard VGA protocol video stream data ori_vga.

[0016] The ori_vga is input into the switch-type median-Gaussian filtering circuit to determine the type of a current pixel point of the ori_vga, and different methods are selected for denoising processing according to different types of the pixel point to obtain filtered video stream data post_fliter, and the specific content is as follows:

[0017] (1) When the pixel value of the pixel point is 0 or 255, the pixel point is a salt and pepper noise point, then the pixel point is processed using median filtering, and the specific formula is:

[0018] Mid(x,y)=med{G(x+i,y+j),(i,j)∈[-r,r]}

[0019] Wherein, r represents the convolution kernel radius, x, y respectively represent the horizontal and vertical coordinates of the pixel point, Mid(x, y) represents the result after median filtering processing, med represents the median operation, G() represents the Gaussian filtering result, i represents the horizontal offset relative to the current pixel point, j represents the vertical offset relative to the current pixel point;

[0020] (2) When the pixel point is not a salt and pepper noise point, then the pixel point is processed using Gaussian filtering, and the specific formula is:

[0021]

[0022] Wherein, Fth(x, y) represents the switching function, Mean represents the mean value of the target pixel neighborhood, P(x, y) represents the target point pixel value, and T represents the bias value;

[0023] The calculation formula of the Gaussian filtering result is as follows:

[0024]

[0025] The shift-ram cascade method is used to establish a 3*3 matrix to obtain the neighborhood pixels of the center pixel of the switching type median-Gaussian filter circuit.

[0026] The post_filter is transmitted to the Retinex enhancement unit, and after the reflection component extraction circuit, the enhanced video stream data is obtained by using the SSR Reitnex calculation method based on the center wrap theory, and the specific formula is:

[0027] log(R(x,y))=log(S(x,y))-log(Gs(x,y)*S(x,y))

[0028] Wherein, log represents logarithmic operation, R(x, y) represents reflection component, S(x, y) represents unprocessed image video stream data, and Gs(x, y) represents second-order Gaussian function.

[0029] The enhanced video stream data is processed using the form of ROM lookup table, and the data is enlarged by 1024 times to reduce the precision loss, the coefficient of Gaussian standard deviation is 80, the established Gaussian filter template is 15*15 size; After obtaining the log(R(x, y)) component, it is sent to the stretching circuit for processing.

[0030] The stretching circuit re-stretches the reflected component after logarithmic operation into an 8-bit wide channel, with the formula being:

[0031]

[0032] wherein post_retin represents the stretched video stream data, MIN and MAX represent the minimum and maximum values of log(R(x, y)) respectively.

[0033] The post_retin is converted into data in Ycbcr space using the color space conversion circuit to obtain the enhanced gray video stream data, with the formula being:

[0034] Y = (77 * R + 150 * G + 29 * B) » 8

[0035] Cb = (-43 * R - 85 * G + 128 * B + 32768) » 8

[0036] Cr = (128 * R - 107 * G - 21 * B + 32768) » 8

[0037] wherein Y represents the Y channel classification, R represents the red channel pixel value, G represents the green channel pixel value, B represents the blue channel pixel value, Cb represents the blue channel offset, and Cr represents the red channel offset.

[0038] Further, the adaptive threshold calculation module is configured to perform the following actions:

[0039] When the valid enablement VGA_DE of the enhanced gray video stream data is 1, the memory RAM1 is used to perform gray histogram statistics on the enhanced gray video stream data; when the video field synchronization signal vga_vs of the enhanced gray video stream data is 1, the cumulative sum of the product of all gray levels and their occurrence times in the current frame is calculated by traversing the RAM1, with the formula being:

[0040]

[0041] wherein S represents the cumulative sum, M represents the total number of gray levels, and Q(is) represents the occurrence times of the is-th gray level in the current frame.

[0042] According to the cumulative sum, the inter-class variance when all gray values are the segmentation values is calculated, with the formula being:

[0043]

[0044] Wherein, U(jt) represents the inter-class variance, j represents the gray level distinguishing the foreground and background of the binary image, G(is) represents the pixel number of the is-th gray level, and G represents the total pixel number of the image.

[0045] Record the maximum inter-class variance U(jt) max The corresponding segmentation value, which is used as the adaptive global threshold value of the current frame together with the bias value preset from the host computer.

[0046] Further, the connected domain marking module comprises a video binarization conversion unit, a deburring operation unit, and a connected domain marking unit.

[0047] The enhanced gray video stream data is input into the connected domain marking module, and is converted into binary video stream data by the video binarization conversion unit according to the adaptive global threshold value, and the specific formula is as follows:

[0048]

[0049] Wherein, post_bin represents the binary video stream data, PAE represents the enhanced gray video stream data, and Th represents the adaptive global threshold value.

[0050] The deburring operation unit performs morphological filtering on the binary video stream data post_bin, and obtains the deburred video stream data post_burr through two-stage threshold-based image erosion operation, and the specific formula is as follows:

[0051]

[0052] Wherein, Ps represents the set structure template, and Pd represents the pixel position in the image.

[0053] In the connected domain marking unit, the single-scan marking algorithm is used to perform connected region marking processing on the deburred video stream data, the positioning of the image target is completed, and the position information and effective area information of the surrounding frame of different targets are output.

[0054] Further, the deburring operation unit is configured to perform the following actions:

[0055] N rows of data of the binary video stream data post_bin are buffered by using a FIFO, the output of the FIFO and the current time post_bin form a filtering matrix with a length and a width of N+1; the number of black pixels in the filtering matrix is counted, and when the number of black pixels in the filtering matrix is less than or equal to The pixel at the center position of the filtering matrix is changed to black; otherwise, the pixel at the center position of the filtering matrix is changed to white, and the deburred video stream data is obtained.

[0056] Further, the connected domain marking unit is configured to perform the following actions:

[0057] A dual-port RAM 2 is used to combine the label storage table and the equivalence table, and store the label label corresponding to the region data information, which includes the four boundaries of the bounding box, the foreground region area, and the equivalent label.

[0058] The FIFO is used to complete the buffering of the deburring video stream data, and the values of the current pixel point and the adjacent upper left, upper, upper right, and left pixels are obtained under 8-connected judgment, and are denoted as P, A, B, C, and D, respectively. The preliminary label assignment, equivalence relationship recording, and equivalence table merging are performed.

[0059] Preliminary label assignment: the label is initialized as 1. When P is a foreground pixel, the effective conditions of ABCD are observed: if ABCD are all background pixels, the label is assigned to P, and label+1. In the next clock, the coordinate information of the current P point is written into the dual-port RAM at the address of label, and the area information bit is 1. If there is one foreground pixel point in ABCD, the label of the foreground pixel point is assigned to the label of P. If there are two or three foreground pixel points in ABCD, the label of the smallest foreground pixel point is assigned to the label of P. In the next clock, the value of the address of label in the dual-port RAM is read, and the coordinate information of the four boundaries is compared with the coordinate information of the current P point. If P is outside the rectangle composed of the coordinate information of the four boundaries, the coordinate of P is updated in the RAM, and the area information bit is +1 and written back to the RAM.

[0060] Equivalence relationship recording: when P is white and not on the four boundaries, there are two kinds of label equivalence conditions: ① AC are both white and have different labels, and AC are in the same region; according to the preliminary label assignment rule, the data in the dual-port RAM at the address of the smaller label is updated in the next clock, and the larger label is written into the [9:0] bit of the smaller address, recording the equivalence relationship; ② CD are both black and have different labels, and the larger label is written into the [9:0] bit of the recording table in the next clock; when there is no equivalence relationship, the maximum value 10’b1111111111 is written into the [9:0] bit.

[0061] Equivalence table merging: when the transmission is not completed, the address and the count value are both 0. After the video frame transmission is completed and the field blanking period is entered, the count value is +1. The data in the RAM at the address 1 is checked. If the [9:0] bit of the read data is the maximum value 10’b1111111111, it means that the region corresponding to the data in the address 1 has no equivalence relationship with other regions, and the data is written back directly.

[0062] If the read [9:0] bit is not the maximum value or 0, it indicates that the region in address 1 and the region in which the new read data [9:0] is stored have equivalence relationship, and they are the same region, at this time, the read address of the RAM is replaced by the read [9:0] bit, the information of the region is read out in the next clock and is combined with the existing region 1 information, the four boundary values are updated, the area information bit is directly added, and it is judged again whether the [9:0] bit of the new read data is the maximum value or 0; the operation is repeated until the [9:0] bit in the new address is the maximum value, at this time, it indicates that the current equivalence chain is completed.

[0063] After the completion of the merging, only the K regions with the maximum effective area are retained, and the information of the K equivalent regions is output, including the bounding box position information and the effective area.

[0064] Further, the AXI uplink module receives the bounding box position information of the connected region marking module, and simultaneously transmits the bounding box position information at the FPGA end and the ARM end, the bounding box position information is read at the ARM end by using the AXI-lite protocol cluster, and the bounding box position information is superimposed on the copy channel of ori_vga in the form of an external rectangle at the FPGA end and the bounding box position information and the effective area information are output.

[0065] Compared with the prior art, the above technical scheme of the present application has the following significant technical effects:

[0066] (1) The present application completes the image processing operation while transmitting the video, and the internal algorithm is completed during the frame blanking period, which fully utilizes the gap of the video transmission protocol, does not consume additional time delay, and has good real-time performance.

[0067] (2) The row filtering, enhancement, deburring, adaptive threshold and single scan connected region marking algorithms used in the present application are improved according to the hardware structure, which consumes less RAM to complete the algorithm function and saves hardware resources.

[0068] (3) The present application fully combines the advantages of ZYNQ architecture, has an Ethernet interface for communication with the upper computer on the basis of ensuring real-time performance, and is programmable in software and hardware, which has certain flexibility. And it can automatically complete the target image capture task, is suitable for image information collection of edge devices with high real-time performance, has high automation degree, and has very broad application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 It is a schematic diagram of the overall structure of the present application.

[0070] Figure 2 It is a structure schematic diagram of the pixel filtering enhancement module of the present application.

[0071] Figure 3This is a schematic diagram of the connected component marking module of the present invention. Detailed Implementation

[0072] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0073] This invention proposes a real-time multi-target detection system based on ZYNQ, such as... Figure 1 As shown, it includes a host computer, an image acquisition module, and a ZYNQ board.

[0074] The image acquisition module is used to perform Bayer interpolation processing using the ISP module of the CMOS camera and output the original video stream data in RGB565 format after color restoration.

[0075] The ZYNQ board includes an FPGA side and an ARM side. The FPGA side includes a pixel filtering enhancement module, an adaptive threshold calculation module, a connected component marking module, and an AXI uplink module. The pixel filtering enhancement module is connected to the adaptive threshold calculation module and the connected component marking module, the adaptive threshold calculation module is connected to the connected component marking module, and the connected component marking module is connected to the AXI uplink module.

[0076] The ARM side is used for overall task scheduling, enabling the DMA channel to complete video path control; it changes the threshold parameters of the connected component marking module according to the host computer instructions, sets the maximum number of detection bounding boxes according to the application scenario requirements, and sets the initial bias value of the adaptive threshold calculation module.

[0077] The pixel filtering enhancement module receives the raw video stream data from the image acquisition module, uses a switch-type median-Gaussian filtering algorithm to filter and denoise the raw video stream data, then uses the Retinex algorithm to enhance the data and performs color space conversion to obtain enhanced grayscale video stream data.

[0078] like Figure 2 As shown, the pixel filtering enhancement module includes a pixel stitching circuit, a video frame storage circuit, a switched median-Gaussian filter circuit, a Retinex enhancement unit, and a color space conversion circuit. All circuits are designed using a PL-side FPGA to achieve hardware acceleration of the algorithm. This embodiment uses an OmniVision OV5640 CMOS image sensor.

[0079] The original video stream data in RGB565 format under 1280*720@60 is input into a pixel filtering enhancement module, and after a pixel splicing circuit, a standard spliced video stream data pixel_data is output every two clocks during the pull-up period of the input row valid signal HREF.

[0080] The obtained pixel_data is cached in DDR4 through a video frame storage circuit and a VDMA IP core of Xilinx, and a standard video interface with a video valid signal vga_de, a field valid signal vga_vs and a row valid signal vga_hs is generated according to the VGA video standard protocol. The video stream data cached in the DDR4 is read by using the video interface to obtain standard VGA protocol video stream data ori_vga.

[0081] The ori_vga is processed by a switch-type median-Gaussian filter circuit to judge the type of the current pixel point of the ori_vga, and different methods are selected for denoising processing according to the different types of the pixel point to obtain filtered video stream data post_fliter, and the specific content is as follows:

[0082] (1) When the pixel value of the pixel point is 0 or 255, the pixel point is a salt and pepper noise point, and the median filter is used to process the pixel point, and the specific formula is as follows:

[0083] Mid(x,y)=med{G(x+i,y+j),(i,j)∈[-r,r]}

[0084] Wherein, r represents the convolution kernel radius, x and y represent the horizontal and vertical coordinates of the pixel point respectively, Mid(x,y) represents the result of the median filter processing, med represents the median operation, G() represents the Gaussian filter result, i represents the horizontal offset relative to the current pixel point, and j represents the vertical offset relative to the current pixel point.

[0085] (2) When the pixel point is not a salt and pepper noise point, the Gaussian filter is used to process the pixel point, and the specific formula is as follows:

[0086]

[0087] Wherein, Fth(x,y) represents a switch function, Mean represents the mean value of the target pixel neighborhood, P(x,y) represents the target point pixel value, and T represents a bias value.

[0088] The calculation formula of the Gaussian filter result is as follows:

[0089]

[0090] The 3*3 matrix is established by shift-ram cascade to obtain the neighborhood pixels of the center pixel of the switch type median-Gaussian filter circuit.

[0091] The post_filter is transmitted to the Retinex enhancement unit, and the enhanced video stream data is obtained by the reflection component extraction circuit and the SSR Reitnex calculation method based on the center surround theory, and the specific formula is:

[0092] log(R(x,y))=log(S(x,y))-log(Gs(x,y)*S(x,y))

[0093] Wherein, log represents logarithmic operation, R(x,y) represents reflection component, S(x,y) represents unprocessed image video stream data, and Gs(x,y) represents second-order Gaussian function.

[0094] The enhanced video stream data is processed by using the form of ROM lookup table, and the data is enlarged by 1024 times to reduce the precision loss, the coefficient of Gaussian standard deviation is 80, the Gaussian filter template is established with the size of 15*15; After obtaining the log(R(x,y)) component, it is sent to the stretching circuit for processing.

[0095] The stretching circuit re-stretches the reflection component after logarithmic operation to 8-bit wide channel, and the specific formula is:

[0096]

[0097] Wherein, post_retin represents the stretched video stream data, MIN and MAX represent the minimum and maximum values of log(R(x,y)) respectively.

[0098] The post_retin is converted into data in Ycbcr space by using the color space conversion circuit, and the enhanced gray video stream data is obtained, and the specific formula is:

[0099] Y=(77*R+150*G+29*B)>>8

[0100] Cb=(-43*R-85*G+128*B+32768)>>8

[0101] Cr=(128*R-107*G-21*B+32768)>>8

[0102] Wherein, Y represents Y channel classification, R represents red channel pixel value, G represents green channel pixel value, B represents blue channel pixel value, Cb represents blue channel offset, and Cr represents red channel offset.

[0103] An adaptive threshold calculation module is configured to count the enhanced gray video stream data and calculate the maximum between-class variance to obtain an adaptive global threshold. The specific content is as follows:

[0104] When the valid enablement VGA_DE of the enhanced gray video stream data is 1, the memory RAM1 is used to perform gray histogram statistics on the enhanced gray video stream data; when the video field synchronization signal vga_vs of the enhanced gray video stream data is 1, the sum of the products of all gray levels and their occurrence times in the current frame is calculated by traversing the RAM1, and the specific formula is as follows:

[0105]

[0106] wherein S represents the sum, M represents the total number of gray levels, and Q(is) represents the occurrence times of the is-th gray level in the current frame.

[0107] The between-class variance when all the gray values are the segmentation values is calculated according to the sum, and the specific formula is as follows:

[0108]

[0109] wherein U(jt) represents the between-class variance, jt represents the gray level for distinguishing the foreground and the background of the binary image, G(is) represents the pixel number of the is-th gray level, and G represents the total number of pixels of the image.

[0110] The maximum between-class variance U(jt) is recorded max The corresponding segmentation value is used as the adaptive global threshold of the current frame together with a bias value from the host computer.

[0111] In this embodiment, 256 gray levels are used, and therefore the self-delay of this module is the delay of traversing the gray levels twice, that is, 512 clocks, that is, the binary threshold used for the next frame is obtained after 512 clocks after the start of the field blanking.

[0112] A connected domain marking module is configured to convert the enhanced gray video stream data into binary video stream data by using the adaptive global threshold, and perform multi-target positioning to generate an enclosing frame of each target and output the position information and the effective area information of the enclosing frame.

[0113] As shown in Figure 3 the connected domain marking module includes a video binary conversion unit, a deburring operation unit and a connected domain marking unit.

[0114] The enhanced gray video stream data is input into the connected domain marking module, and is converted into binary video stream data by the video binary conversion unit according to the adaptive global threshold, and the specific formula is as follows:

[0115]

[0116] Wherein, post_bin represents the binarization video stream data, PAE represents the enhanced gray video stream data, and Th represents the adaptive global threshold.

[0117] The deburring operation unit performs morphological filtering on the binarization video stream data post_bin, and obtains the deburred video stream data post_burr through two-stage image erosion operation with threshold, and the specific formula is as follows:

[0118]

[0119] Wherein, Ps represents the set structure template, and Pd represents the pixel position in the image.

[0120] The FIFO is used to complete 5-row data buffering of the binarization video stream data post_bin, the output of the FIFO and the current time post_bin form a filtering matrix with length and width of 6; the number of black pixels in the filtering matrix is counted, and when the number of black pixels in the filtering matrix is less than or equal to 18, the pixel corresponding to the center of the filtering matrix is changed to black; otherwise, the pixel corresponding to the center of the filtering matrix is changed to white, and the deburred video stream data is obtained.

[0121] In the connected domain marking unit, the single-scan marking algorithm is used to perform connected region marking processing on the deburred video stream data, to complete the positioning of the image target, and to output the position information and effective area information of the bounding box of different targets. The specific content is as follows:

[0122] A block of dual-port RAM 2 is used to combine the label storage table and the equivalence table into one, to store the region data information corresponding to the label label, which includes the four boundaries of the bounding box, the foreground area and the equivalent label.

[0123] The FIFO is used to complete the buffering of the deburred video stream data, and the values of the current pixel point and the adjacent upper left, upper, upper right and left pixels are obtained under the condition of 8-connected judgment, which are respectively recorded as P and A, B, C and D, to perform preliminary label allocation, equivalence relationship recording and equivalence table merging.

[0124] Label initial assignment: initialize label as 1, if P is a foreground pixel, observe the effective situation of ABCD pixels at this time: if ABCD are all background pixels, assign label to P, at the same time label + 1, the next clock will write the coordinate information of the current P point into the dual-port RAM at the address of label, and the area information bit is 1; if there is one foreground pixel point in ABCD four points, assign the label of the foreground pixel point to the label of P point; if there are two or three foreground pixel points in ABCD four points, assign the label of the smallest point in these foreground pixel points to the label of P point, the next clock also reads the value of the address label in the dual-port RAM, compares the coordinate information of the four boundaries with the coordinate information of the current P point, if P point is outside the rectangle composed of the coordinate information of the four boundaries, update the P point coordinate to the RAM, at the same time the area information bit + 1 and write back to the RAM.

[0125] Equivalent relationship record: if P is white and not on the four boundaries, there will be two kinds of label equivalence situations: ① AC are both white and the labels are different, for example, the label of A is 5 and the label of C is 6 at this time, AC are in the same region; according to the label initial assignment rule, the next clock will update the data in the dual-port RAM at address 5, write the larger label 6 into [9:0] bit of address 5, record the equivalent relationship; ② CD are both black and the labels are different, also write the larger label into [9:0] bit of the record table at the next clock; when there is no equivalent relationship, write the maximum value 10'b1111111111 into [9:0] bit.

[0126] Equivalent table merging: when the transmission is not completed, the address and the count value are both 0, after the video frame transmission is completed and the field blanking period is entered, the count value + 1, check the data in the RAM at address 1, if the read data [9:0] bit is the maximum value 10'b1111111111, it means that the region corresponding to the data in address 1 has no equivalent relationship with other regions, at this time the data is written back directly.

[0127] If the read [9:0] bit is not the maximum value or 0, it means that the region in address 1 has equivalent relationship with the region stored in the address whose [9:0] bit is the newly read data, they are the same region, at this time replace the read address of the RAM with the read [9:0] bit, read the information of the region at the next clock and merge it with the existing region 1 information, update the four boundary values, directly add the area information bit, and judge again whether the [9:0] bit of the newly read data is the maximum value or 0; repeat this operation until the [9:0] bit in the new address is the maximum value, at this time it means that the current equivalent chain merging is completed.

[0128] After merging is completed, only the 10 regions with the largest effective area are retained, and the information of the 10 equivalent regions is output, including the enclosing frame position information and the effective area.

[0129] In the AXI uplink module, the OV5640 camera sends a start signal to the IIC timing controller through the AXI bus, then the ARM end enters a waiting state, starts to configure the internal registers of the OV5640 camera, and sends 253 SCCB protocol parameter data in total; when the camera parameter configuration is completed, a camera_init_done signal is fed back to the ARM end through a hardware interrupt; a VDMA channel is started, the IP address of the development board is configured as 192.168.1.17, a UDP judgment function is mounted, and then a cyclic work flow is entered; the hardware interrupt in the ZYNQ is connected to IRQ_F2P, and the interrupt number ID has [91:84] and [68:61] in total, which are configured as a rising edge trigger mode to enter an interrupt processing function. After entering the interrupt function, the bounding box position information in the AXI uplink module is read through the AXI interface, for subsequent recognition of the ARM end.

[0130] The above shows and describes the basic principles, main features and advantages of the present application. However, the above only describes specific embodiments of the present application, and the technical features of the present application are not limited to this. Any other implementation derived by those skilled in the art without departing from the technical solution of the present application should be covered in the patent range of the present application.

Claims

1. A real-time multi-target detection system based on ZYNQ, characterized in that, The system comprises a host computer, an image acquisition module and a ZYNQ board card; The image acquisition module is used for completing Bayer interpolation processing by using an ISP module of a CMOS camera and outputting original video stream data in an RGB565 format after color restoration; The ZYNQ board card comprises an FPGA end and an ARM end, wherein the FPGA end comprises a pixel filtering enhancement module, an adaptive threshold calculation module, a connected domain marking module and an AXI uplink module; the pixel filtering enhancement module is connected with the adaptive threshold calculation module and the connected domain marking module respectively, the adaptive threshold calculation module is connected with the connected domain marking module, and the connected domain marking module is connected with the AXI uplink module; The ARM end is used for overall task scheduling, opening a DMA channel to complete video path control, changing the threshold parameter of the connected domain marking module according to the instruction of the host computer, setting the maximum number of detection bounding boxes according to the application scene requirement, and setting the initial bias value of the adaptive threshold calculation module; The pixel filtering enhancement module is used for receiving the original video stream data of the image acquisition module, filtering and denoising the original video stream data by using a switch type median-gaussian filtering algorithm, enhancing the data by using a Retinex algorithm, and performing color space conversion to obtain enhanced gray video stream data; The adaptive threshold calculation module is used for counting the enhanced gray video stream data and calculating the maximum inter-class variance to obtain an adaptive global threshold value; The connected domain marking module is used for converting the enhanced gray video stream data into binary video stream data by using the adaptive global threshold value, positioning multiple targets, generating a bounding box of each target, and outputting the position information and effective area information of the bounding box; The AXI uplink module is used for receiving the position information of the bounding box from the connected domain marking module, packaging the position information into an AXI protocol and storing the AXI protocol in a specified address, and waiting for the ARM end to read.

2. The ZYNQ based real time multi-target detection system as claimed in claim 1, wherein, The pixel filtering enhancement module comprises a pixel splicing circuit, a video frame storage circuit, a switch type median-gaussian filtering circuit, a Retinex enhancement unit and a color space conversion circuit; The original video stream data in the RGB565 format is input into the pixel splicing circuit, and one standard spliced video stream data pixel_data is output every two clocks during the pull-up period of an input row valid signal HREF of the original video stream data; The obtained pixel_data is input into the video frame storage circuit, the pixel_data is buffered in a DDR4, a standard video interface with a video valid signal vga_de, a field valid signal vga_vs and a row valid signal vga_hs is generated according to a VGA video standard protocol, the buffered video stream data in the DDR4 is read by using the video interface, and standard vga protocol video stream data ori_vga is obtained; The ori_vga is input into the switch type median-gaussian filtering circuit, the type of a current pixel point of the ori_vga is judged, different methods are selected for denoising processing according to different types of the pixel point, and filtered video stream data post_fliter is obtained, and the specific content is as follows: (1) When the pixel value of the pixel point is 0 or 255, the pixel point is a salt and pepper noise point, then the pixel point is processed using median filtering, and the specific formula is: Mid(x, y) = med{G(x+i, y+j), (i, j) e [-r, r]} Wherein, r represents the convolution kernel radius, x and y represent the horizontal and vertical coordinates of the pixel point respectively, Mid(x, y) represents the result after median filtering processing, med represents the median operation, G() represents the Gaussian filtering result, i represents the horizontal offset relative to the current pixel point, and j represents the vertical offset relative to the current pixel point; (2) When the pixel point is not a salt and pepper noise point, then the pixel point is processed using Gaussian filtering, and the specific formula is: Wherein, Fth(x, y) represents the switching function, Mean represents the mean value of the target pixel neighborhood, P(x, y) represents the target point pixel value, and T represents the bias value; The calculation formula of the Gaussian filtering result is as follows: A 3*3 matrix is established by using shift-ram cascade to obtain the neighborhood pixels of the center pixel of the switching median-Gaussian filter circuit; The post_filter is transmitted to the Retinex enhancement unit, and after the reflection component extraction circuit, the enhanced video stream data is obtained by using the SSR Reitnex calculation method based on the center wrap theory; The specific formula is: log(R(x, y)) = log(S(x, y))-log(Gs(x, y)*S(x, y)) Wherein, log represents logarithmic operation, R(x, y) represents reflection component, S(x, y) represents unprocessed image video stream data, and Gs(x, y) represents second-order Gaussian function; The enhanced video stream data is processed in the form of ROM lookup table, and the data is enlarged by 1024 times to reduce the precision loss, the coefficient of Gaussian standard deviation is 80, and the Gaussian filter template with a size of 15*15 is established; After obtaining the log(R(x, y)) component, it is sent to the stretching circuit for processing; The stretching circuit re-stretches the reflection component after logarithmic operation to an 8-bit wide channel, and the specific formula is: Wherein, post_retin represents the video stream data after stretching, MIN and MAX represent the minimum and maximum values of log(R(x, y)) respectively; The post_retin is converted into data in Ycbcr space by using the color space conversion circuit, and the enhanced gray video stream data is obtained, and the specific formula is: Y = (77*R+150*G+29*B)>>8 Cb = (-43*R-85*G+128*B+32768)>>8 Cr = (128*R-107*G-21*B+32768)>>8 Wherein, Y represents Y channel classification, R represents red channel pixel value, G represents green channel pixel value, B represents blue channel pixel value, Cb represents blue channel offset, and Cr represents red channel offset.

3. The ZYNQ based real time multi-target detection system as claimed in claim 1, wherein, The adaptive threshold calculation module is configured to perform the following actions: When the valid enable VGA_DE of the enhanced gray video stream data is 1, the memory RAM1 is used to perform gray histogram statistics on the enhanced gray video stream data; when the video field synchronization signal vga_vs of the enhanced gray video stream data is 1, the accumulation sum of the product of all gray levels and the number of occurrences of the gray levels in the current frame is calculated by traversing the RAM1, and the specific formula is: Wherein, S represents the accumulation sum, M represents the total number of gray levels, and Q(is) represents the number of occurrences of the is-th gray level in the current frame; The inter-class variance when all gray levels are the segmentation values is calculated according to the accumulation sum, and the specific formula is: Wherein, U(jt) represents the inter-class variance, jt represents the gray level for distinguishing the foreground and the background of the binary image, G(is) represents the pixel number of the is-th gray level, and G represents the total pixel number of the image; Record the maximum inter-class variance U(jt) max The corresponding segmentation value, which is used as the adaptive global threshold value of the current frame together with the bias value preset from the host computer.

4. The ZYNQ based real time multi-target detection system as claimed in claim 1, wherein, The connected domain marking module comprises a video binary conversion unit, a deburring operation unit and a connected domain marking unit; The enhanced gray video stream data is input into the connected domain marking module, and the video stream data is converted into binary video stream data according to the adaptive global threshold value through the video binary conversion unit, and the specific formula is: Wherein, post_bin represents the binary video stream data, PAE represents the enhanced gray video stream data, and Th represents the adaptive global threshold value; The deburring operation unit performs morphological filtering on the binary video stream data post_bin, and obtains the deburred video stream data post_burr through two-stage image erosion operation with threshold value, and the specific formula is: Wherein, Ps represents the set structure template, and Pd represents the pixel position in the image; In the connected domain marking unit, the single-scan marking algorithm is used to perform connected region marking processing on the deburred video stream data, the positioning of the image target is completed, and the position information and the effective area information of the surrounding frame of different targets are output.

5. The ZYNQ based real time multi-target detection system as claimed in claim 4, wherein, The deburring operation unit is configured to perform the following actions: The N-row data buffer of the binary video stream data post_bin is completed using the FIFO, and the output of the FIFO and the current time post_bin form a filtering matrix with a length and a width of N+1; Counting the number of black pixels in the filter matrix, when the number of black pixels in the filter matrix is less than or equal to changing the pixel at the position corresponding to the center of the filter matrix to black; otherwise, changing the pixel at the position corresponding to the center of the filter matrix to white, to obtain the deburred video stream data.

6. The ZYNQ based real time multi-target detection system as claimed in claim 4, wherein, The connected domain marking unit is configured to perform the following actions: A block of dual-port RAM2 is used to combine the label storage table and the equivalence table into one, and the region data information corresponding to the label label is stored, which includes the four boundaries of the surrounding frame, the foreground area and the equivalent label; The FIFO is used to complete the buffer of the deburred video stream data, and the values of the current pixel point and the adjacent upper left, upper, upper right and left pixels are obtained under the 8-connected judgment, which are respectively denoted as P, A, B, C and D, and the preliminary label allocation, equivalence relationship recording and equivalence table merging are performed. Label initial assignment: initialize label as 1, if P is a foreground pixel, observe the effective situation of ABCD pixels at this time: if ABCD are all background pixels, assign label to P, at the same time, label+1, the next clock will write the coordinate information of current P point into the address label of the dual-port RAM, and the area information bit is 1; if there is one foreground pixel in ABCD four points, assign the label of the foreground pixel to the label of P point; If there are two or three foreground pixels in ABCD four points, assign the label of the smallest point in these foreground pixels to the label of P point, the next clock also reads the value of the address label in the dual-port RAM, compares the coordinate information of the four boundaries with the coordinate information of the current P point, if P point is outside the rectangle composed of the coordinate information of the four boundaries, update the P point coordinates to the RAM, at the same time, the area information bit+1 and write back to RAM; Equivalent relation record: if P is white and not on the four boundaries, there will be two kinds of label equivalence situations: ①AC are both white and the labels are different, then AC are in the same region; according to the label initial assignment rule, the next clock will update the data in the dual-port RAM with the smaller label, write the larger label into the [9:0] bit of the smaller label, and record the equivalent relationship; ②CD are both black and the labels are different, also write the largest label into the [9:0] bit of the record table in the next clock; when there is no equivalent relationship, write the maximum value 10'b1111111111 into [9:0] bit; Equivalent table merging: when the transmission is not completed, the address and the count value are both 0, after the video frame transmission is completed and the field blanking period is entered, the count value+1, check the data in the RAM at address 1, if the [9:0] bit of the read data is the maximum value 10'b1111111111, it means that the region corresponding to the data in address 1 has no equivalent relationship with other regions, at this time, directly write back the data; If the read [9:0] bit is not the maximum value or 0, it means that the region in address 1 has equivalent relationship with the region stored in the memory with the read [9:0] bit as the new data, they are the same region, at this time, replace the read address of the RAM with the read [9:0] bit, the next clock reads the information of this region and merges with the existing region 1 information, updates the four boundary values, and directly adds the area information bit, again judges whether the [9:0] bit of the new read data is the maximum value or 0; repeat this operation until the [9:0] bit in the new address is the maximum value, at this time, it means that the current equivalent chain merging is completed; After merging is completed, only the K regions with the largest effective area are retained, and the information of the K equivalent regions is output, including the enclosing frame position information and the effective area.

7. The ZYNQ based real time multi-target detection system as claimed in claim 1, wherein, The AXI uplink module receives the connected area marking module bounding box position information, and simultaneously transmits the bounding box position information at the FPGA end and the ARM end. The bounding box position information is read at the ARM end using the AXI-lite protocol cluster, and the bounding box position information is superimposed in the form of an external rectangle to the copy channel of ori_vga at the FPGA end, and the bounding box position information and the effective area information are output.

Images