Target snapshot method and system based on Zynq UltraScale + MPSoC

By performing pixel data processing and object detection algorithm deployment on the Zynq UltraScale+MPSoC chip, the image clarity and invalid data processing problems of the target capture system in harsh environments are solved, and the detection accuracy and system response speed are improved.

CN119012020APending Publication Date: 2024-11-22CHANGCHUN UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411020825.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing target capture system has reduced image detail clarity in bad weather such as low light or haze, resulting in missed or incorrect judgments. The transmission, storage and processing of invalid data in massive images increases the burden on the server, resulting in slow response speed.

Method used

Adopting the target capture system based on Zynq UltraScale+MPSoC, the pixel data processing module is added to the PL end to perform adaptive histogram equalization operation, enhance image clarity, and deploy the object detection algorithm on the Zynq UltraScale+MPSoC chip to achieve preliminary image Filter, transmit only image data containing the target.

Benefits of technology

It improves the accuracy of target detection, reduces the transmission and storage requirements of invalid data, reduces the consumption of storage and computing resources, and improves the operating efficiency and response speed of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119012020A_ABST
    Figure CN119012020A_ABST
Patent Text Reader

Abstract

The invention discloses a target snapshot method and system based on a Zynq UltraScale + MPSoC. The system comprises a camera, a Zynq UltraScale + MPSoC chip, two pieces of DDR3SDRAM (Double Data Rate 3Synchronous Dynamic Random Access Memory) and an SFP (Small Form-factor Pluggable) + optical module, wherein the camera sends generated pixel data to the Zynq UltraScale + MPSoC chip, the Zynq UltraScale + MPSoC chip receives the pixel data, image enhancement operation is carried out on the pixel data by utilizing the high-performance parallel processing capability of a PL end, and frame numbers and time information are added to form frame-by-frame image data; according to the invention, the accuracy of target snapshot can be improved, and the problems of low real-time response speed and waste of storage resources and computing resources of an existing target snapshot system are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of sound or image recording or reproducing equipment, and in particular to a target capture system based on Zynq UltraScale+MPSoC. Background Art

[0002] With the rapid development of society, the demand for target capture systems in the fields of security, traffic management, industrial automation, etc. is growing. Existing target capture systems often face problems such as misidentification and omission, data processing and storage challenges.

[0003] When faced with bad weather conditions such as low light or haze, the clarity of image details captured by the camera of the target capture system is reduced, resulting in missed judgments or incorrect judgments.

[0004] Secondly, the existing target capture system sends the captured images to the terminal or back-end server for processing without screening. The massive images include a large number of images that do not contain the target. The transmission, storage and processing of these images that do not contain the target require a large amount of storage resources and computing resources, which greatly increases the burden on the terminal or back-end server, causing processing delays, that is, slow real-time response speed. Summary of the invention

[0005] In view of the above problems, the present invention provides a target capture system based on Zynq UltraScale+MPSoC, which improves the accuracy of target capture and solves the problems of slow real-time response speed and waste of storage resources and computing resources in the existing target capture system.

[0006] The present invention solves the above-mentioned technical problems through the following technical means: a target capture system based on Zynq UltraScale+MPSoC, comprising a camera, a Zynq UltraScale+MPSoC chip, two DDR3SDRAMs and an SFP+ optical module, the ZYNQ chip comprising a PL end (programmable logic part) and a PS end (processing system part), the PL end comprising a pixel data conversion module, a pixel data processing module, a frame information adding module, an image data preprocessing module, a Data_to_AXI_Full_1 module, an image data sending completion interrupt module, an AXI_Interconnect_1 module, a Data_to_AXI_Full_2 module, an AXI_Interconnect_2 module, a DDR3 control module, an Aurora_Send module, an interrupt register module, an AXI_Interconnect_3 module, and a DPU (Deep Learning Processing Units, Deep Learning Processing Units) module, and the PS end comprises a camera driver, a DPU driver, and an application.

[0007] The pixel data conversion module at the PL end is used to convert the pixel data received at the camera output data interface timing into a pixel data stream, and convert the pixel data stream into a pixel data stream in RGB888 format, and finally output it to the pixel data processing module.

[0008] The pixel data processing module at the PL end is used to receive pixel data converted into RGB888 format, perform adaptive histogram equalization operation on the pixel data, and output the pixel data after image enhancement to the frame information adding module.

[0009] The frame information adding module at the PL end is used to add a frame number and a time tag to each frame of pixel data to form a frame of image data, and output the image data with the added frame number and time tag to the image data preprocessing module and the Data_to_AXI_Full_2 module.

[0010] The image data preprocessing module at the PL end is used to resize the size of a frame of image to (640, 640), and output the resized image data to the Data_to_AXI_Full_1 module.

[0011] The Data_to_AXI_Full_1 module at the PL end is used to convert image data into AXI_Full image data, and send the converted AXI_Full image data to the AXI_Interconnect_1 module, and is also used to output a pulse signal to the image data sending completion interrupt module after sending a frame of image data.

[0012] The image data sending completion interrupt module of the PL end is used to receive the pulse signal, convert the pulse signal into a high-level signal, store it in the register as an interrupt signal and output it to the PS end. When the PS end detects the interrupt signal, it will make the register zero through the AXI_Interconnect_1 module.

[0013] The AXI_Interconnect_1 module at the PL end is used to receive image data of the AXI_Full protocol, and output the image data to the PS end, and then store it in the second DDR3SDRAM at the PS end. It is also used to receive instruction data transmitted from the PS end and output it to the image data sending completion interrupt module to clear the register.

[0014] The Data_to_AXI_Full_2 module at the PL end is used to convert the image data into AXI_Full image data, and send the converted AXI_Full image data to the AXI_Interconnect_2 module.

[0015] The AXI_Interconnect_2 module at the PL end is used to receive image data of the AXI_Full protocol and output the image data to the DDR3 control module. In addition, it is also used to receive image data taken out from the first DDR3SDRAM by the DDR3 control module and output the image data to the Aurora_Send module.

[0016] The Aurora_Send module at the PL end is used to retrieve the image data containing the target from the first DDR3SDRAM through the AXI_Interconnect_2 module and the DDR3 control module according to the control information of the interrupt register module, and convert the image data containing the target into data of the Aurora protocol and send it to the SFP+ optical module through the GTH interface. The SFP+ optical module converts the electrical signal into an optical signal and sends it to the back end or terminal server through the optical fiber. After the data is sent, the value of the register in the interrupt register module is rewritten.

[0017] The interrupt register module at the PL end is used to store the image reasoning result information containing the target sent from the PS end, and output the information to the Aurora_Send module. When the Aurora_Send module finishes sending the image data containing the target, it will clear the stored image reasoning information.

[0018] The AXI_Interconnect_3 module on the PL side is used to receive the image reasoning result information on the PS side and output the information to the interrupt register module.

[0019] The DPU module on the PL side is configured by the PS side, obtains instructions from the second DDR3SDRAM on the PS side, processes the received image data, infers the image, and outputs the inference information to the second DDR3SDRAM on the PS side.

[0020] The AXI_Interconnect_4 module on the PL side is used to receive image data sent from the PS side and output the image data to the DPU module. It is also used to receive image reasoning information generated by the DPU module and output the image reasoning information to the PS side.

[0021] The AXI_Interconnect_5 module on the PL side is used to receive configuration instruction information sent from the PS side and output the configuration instruction information to the DPU, and is also used to receive status information of the DPU module and send the status information to the PS side.

[0022] The AXI_Interconnect_6 module on the PL side is used to receive instructions obtained from the second DDR3SDRAM on the PS side and output them to the DPU module.

[0023] The camera driver on the PS side is used to read the camera parameters to obtain the functions that the camera can support, configure the camera parameters, initialize the camera, that is, configure the camera's resolution, frame rate, pixel data bit width, pixel data format, etc., and determine whether the camera parameters are set successfully.

[0024] The DPU driver on the PS side is used to initialize the DPU, that is, configure various parameters of the DPU, map the memory space of the DPU module to the system memory, realize fast access to data, receive calculation requests from applications, and arrange the execution order according to priority or policy.

[0025] The application on the PS side is used to read the interrupt signal of completing sending a frame of image data and the DPU task completion interrupt on the PL side, control the DPU driver, read the inference information of each frame of image data stored in the DPU from the second DDR3SDRAM, and perform a series of operations such as post-processing, and output the final image inference information to the interrupt register module through the AXI_Interconnect_3 module on the PL side, and then control the Aurora_Send module on the PL side to send the image data containing the target stored in the first DDR3SDRAM to the SFP+ optical module through the GTH interface, and the SFP+ optical module converts the electrical signal into an optical signal and sends it to the back end or terminal server through the optical fiber.

[0026] The SFP+ optical module is used to convert the received electrical signal into an optical signal and send it to the back end or terminal server through the optical fiber.

[0027] The present invention adds a pixel data processing module to the PL end of the Zynq UltraScale+MPSoC chip, performs an adaptive histogram equalization operation on the pixel data, enhances the clarity and contrast of the image, can reduce the omission discrimination and error discrimination caused by environmental factors, and improves the accuracy of target detection in the target capture system and the back-end or terminal server. In addition, the present invention introduces a target detection algorithm, deploys the target detection algorithm on the Zynq UltraScale+MPSoC chip, realizes the preliminary screening of images, identifies and filters out images without targets, and only transmits the image data containing targets to the back-end or terminal server for further processing. This not only greatly reduces the transmission and storage requirements of invalid data, but also reduces the consumption of storage resources and computing resources, and improves the overall operating efficiency and response speed of the system. The Zynq UltraScale+MPSoC used in the present invention integrates a multi-processor system (PS end) and a programmable logic (PL end), realizing the close coupling and efficient collaboration of software and hardware. This heterogeneous computing platform enables the system to complete the entire process from pixel data reception, processing, detection to transmission on a single chip, reduces the overall complexity of the system, and improves processing speed and energy efficiency. The present invention adopts a dual DDR3SDRAM access strategy in the system design. Two DDR3 SDRAMs serve the PL end and the PS end respectively. The first DDR3SDRAM on the PL end stores the image data after image enhancement, and the second DDR3SDRAM on the PS end stores the image data after image resizing. Such a design ensures the high efficiency of data transmission. As long as the inference information of the target detection result is transmitted to the PL end, the image data of the first DDR3SDRAM on the PL end can be immediately read and sent to the SFP+ optical module through the GTH interface, and then sent to the back end or terminal server, ensuring that the image data containing the target can be quickly transmitted to the back end or terminal server. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A target capture system based on Zynq UltraScale+MPSoC provided by the present invention;

[0029] Figure 2 This is the working block diagram of the Zynq UltraScale+ MPSoC chip. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0032] In one embodiment, a target capture system based on Zynq UltraScale+MPSoC is provided. Figure 1 As shown, a target capture system based on Zynq UltraScale+MPSoC of this embodiment includes a camera, a ZynqUltraScale+MPSoC chip, two DDR3SDRAMs and an SFP+ optical module. The camera sends the generated pixel data to the Zynq UltraScale+MPSoC chip, which receives the pixel data and uses the high-performance parallel processing capability of the PL end to perform image enhancement operations on the pixel data, and adds frame numbers and time information to form frame-by-frame image data. On the one hand, the image data is stored in the first DDR3SDRAM on the PL side. On the other hand, the image data after image enhancement is stored in the second DDR3SDRAM on the PS side after resizing operation on the PL side. The application on the PS side controls the DPU module on the PL side to perform target detection on the image data stored in the second DDR3SDRAM, reads the inference data of the DPU module for post-processing, and sends the post-processed image inference information containing the target to the PL side. The PL side sends the image data containing the target stored in the first DDR3SDRAM to the SFP+ optical module through the GTH interface. The SFP+ optical module converts the electrical signal into an optical signal and sends it to the back end or terminal server.

[0033] like Figure 2The figure shows the working block diagram of the Zynq UltraScale+MPSoC chip. The Zynq UltraScale+MPSoC chip includes a PL end (programmable logic part) and a PS end (processing system part). The PL end includes a pixel data conversion module, a pixel data processing module, a frame information adding module, an image data preprocessing module, a Data_to_AXI_Full_1 module, an image data sending completion interrupt module, an AXI_Interconnect_1 module, a Data_to_AXI_Full_2 module, an AXI_Interconnect_2 module, a DDR3 control module, an Aurora_Send module, an interrupt register module, an AXI_Interconnect_3 module, and a DPU (Deep Learning Processing Units) module. The PS end includes a camera driver, a DPU driver, and an application.

[0034] The pixel data conversion module at the PL end receives the pixel data from the camera output data interface, converts the pixel data stream into a pixel data stream in RGB888 format, and outputs it to the pixel data processing module.

[0035] The pixel data processing module on the PL side performs image enhancement on the pixel data through an improved adaptive histogram equalization algorithm to improve the visibility of the image and the clarity of the image details, and outputs the enhanced pixel data to the frame information adding module.

[0036] The frame information adding module on the PL side adds a frame number and a time tag to each frame of pixel data to form a frame of image data, and outputs the image data with the added frame number and time tag to the image data preprocessing module (resize) and the Data_to_AXI_Full_2 module.

[0037] The image data preprocessing module at the PL end selectively receives pixel data from a frame of image data to form a frame of image data with a resolution of (640, 640), that is, resizing the size of a frame of image to (640, 640), and outputs the resized image data to the Data_to_AXI_Full_1 module.

[0038] The Data_to_AXI_Full_1 module on the PL side converts the image data into AXI_Full image data, and sends the converted AXI_Full image data to the AXI_Interconnect_1 module. In addition, after sending a frame of image data, a pulse signal is output to the image data sending completion interrupt module.

[0039] The image data sending completion interrupt module on the PL side can receive the pulse signal, convert the pulse signal into a high-level signal, store it in the register as an interrupt signal and output it to the PS side. When the PS side detects the interrupt signal, it will clear the register through the AXI_Interconnect_1 module.

[0040] The AXI_Interconnect_1 module on the PL side is used to receive image data of the AXI_Full protocol and output the image data to the S_AXI_HP interface on the PS side, and then store it in the second DDR3SDRAM on the PS side. It is also used to receive command data from the PS side and output it to the image data sending completion interrupt module to clear the register.

[0041] The Data_to_AXI_Full_2 module on the PL side converts the image data into AXI_Full image data, and sends the converted AXI_Full image data to the AXI_Interconnect_2 module.

[0042] The AXI_Interconnect_2 module on the PL side receives the image data of the AXI_Full protocol and outputs the image data to the DDR3 control module. In addition, it also receives the image data taken out from the first DDR3 SDRAM by the DDR3 control module and outputs the image data to the Aurora_Send module.

[0043] The Aurora_Send module on the PL side retrieves the target image data from the first DDR3SDRAM through the AXI_Interconnect_2 module and the DDR3 control module according to the control information of the interrupt register module, and converts the target image data into Aurora protocol data and sends it to the SFP+ optical module through the GTH interface. The SFP+ optical module converts the electrical signal into an optical signal and sends it to the back-end or terminal server through the optical fiber. After sending the data, the register value in the interrupt register module is cleared.

[0044] The interrupt register module on the PL side registers the image inference information sent from the PS side and outputs the information to the Aurora_Send module. When the Aurora_Send module sends the image data containing the target, it clears the registered information.

[0045] The AXI_Interconnect_3 module on the PL side receives the image inference result information from the PS side and outputs the information to the interrupt register module.

[0046] The DPU module on the PL side is configured by the PS side, obtains instructions from the second DDR3SDRAM on the PS side, processes the received image data, infers the image, and outputs the inference information to the second DDR3SDRAM on the PS side.

[0047] The AXI_Interconnect_4 module on the PL side is used to receive image data sent from the PS side and output the image data to the DPU module. It is also used to receive image inference information generated by the DPU module and output the image inference information to the PS side.

[0048] The AXI_Interconnect_5 module on the PL side is used to receive configuration instruction information sent from the PS side and output this configuration instruction information to the DPU. It is also used to receive status information of the DPU module and send this status information to the PS side.

[0049] The AXI_Interconnect_6 module on the PL side is used to receive instructions from the second DDR3SDRAM on the PS side and output them to the DPU module.

[0050] The camera driver on the PS side is used to read the camera parameters to obtain the functions that the camera can support, configure the camera parameters, initialize the camera, that is, configure the camera's resolution, frame rate, pixel data bit width, pixel data format, etc., and determine whether the camera parameters are set successfully.

[0051] The DPU driver on the PS side is used to initialize the DPU, that is, to configure various parameters of the DPU, map the memory space of the DPU module to the system memory, realize fast access to data, receive computing requests from applications, and arrange the execution order according to priority or policy.

[0052] The application on the PS side is used to process the completion interrupt of sending a frame of image data on the PL side and the task completion interrupt of the DPU, provide image calculation requests to the DPU driver, read the inference information of each frame of image data stored in the DPU module from the second DDR3SDRAM, and perform a series of operations such as post-processing, and output the final image inference information to the interrupt register module through the AXI_Interconnect_3 module on the PL side, and then control the Aurora_Send module on the PL side to send the target image data stored in the first DDR3SDRAM through the GTH interface.

[0053] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0054] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A target capture method based on Zynq UltraScale+MPSoC, characterized in that: The steps include: Step 1: The Zynq UltraScale+ MPSoC chip receives pixel data and uses the high-performance parallel processing capability of the PL end to perform image enhancement operations on the pixel data and add frame number and time information to form frame-by-frame image data; Step 2: The Zynq UltraScale+ MPSoC chip stores the image data with enhanced image and added frame number and time information into the first DDR3SDRAM on the PL side. On the other hand, the enhanced image data is stored into the second DDR3SDRAM on the PS side after resizing operation on the PL side. Step 3: The application on the PS side of the Zynq UltraScale+ MPSoC chip controls the DPU module on the PL side to perform target detection on the image data stored in the second DDR3SDRAM, reads the inference data of the DPU module for post-processing, and sends the post-processed image inference information containing the target to the PL side. The PL side sends the image data containing the target stored in the first DDR3SDRAM to the SFP+ optical module through the GTH interface, and then sends it to the terminal or back-end server through the optical fiber; 2. The target capture method based on Zynq UltraScale+MPSoC according to claim 1, characterized in that: In step 1, the Zynq UltraScale+ MPSoC chip uses high-performance parallel capabilities on the PL side to perform image enhancement operations on the pixel data through adaptive histogram equalization operations, and then adds frame number and time information.

3. The target capture method based on Zynq UltraScale+MPSoC according to claim 1, characterized in that: In step 2, the Zynq UltraScale+ MPSoC chip uses high-performance parallel capabilities on the PL side to resize the image data, reducing the time spent on subsequent image reasoning; 4. The target capture method based on Zynq UltraScale+MPSoC according to claim 1, characterized in that: In step 2, after the PL side of the Zynq UltraScale+MPSoC chip sends a frame of image data after the resize operation to the second DDR3SDRAM on the PS side, the Data_to_AXI_Full_1 module on the PL side will output a pulse signal to the image data sending completion interrupt module. After receiving the pulse signal, the image data sending completion interrupt module converts the pulse signal into a high-level signal, stores it in the register, and then outputs it to the PS side as an interrupt signal. When the PS side detects the interrupt signal, it will perform image inference on the frame of image data and clear the register through the AXI_Interconnect_1 module.

5. The target capture method based on Zynq UltraScale+MPSoC according to claim 1, characterized in that: In step 2, two DDR3SDRAMs are used to store the image data to be sent to the backend or terminal server and the image data to be image inference, respectively, to ensure the efficiency of data transmission. As long as the inference information of the target detection result is transmitted to the PL end, the image data of the first DDR3SDRAM on the PL end can be immediately read and sent to the SFP+ optical module through the GTH interface, and then sent to the backend or terminal server.

6. The target capture method based on Zynq UltraScale+MPSoC according to claim 1, characterized in that: In step 3, the PS side of the Zynq UltraScale+MPSoC chip sends the post-processed image reasoning information containing the target to the interrupt register module of the PL side through the AXI bus. When the image data containing the target is sent to the backend or terminal server, the image reasoning information stored in the interrupt register module will be cleared.

7. A target capture system based on Zynq UltraScale+MPSoC, characterized in that: The method described in any one of claims 1 to 6 is performed, wherein the system includes a camera, a Zynq UltraScale+MPSoC chip, two DDR3SDRAMs and an SFP+ optical module, wherein the camera configuration interface SCCB is connected to the PS end of the Zynq UltraScale+MPSoC chip, the camera pixel data interface is connected to the PL end of the Zynq UltraScale+MPSoC chip, the first DDR3SDRAM is connected to the PL end of the ZynqUltraScale+MPSoC chip, the second DDR3SDRAM is connected to the PS end of the Zynq UltraScale+MPSoC chip, and the image data containing the target is output to the SFP+ optical module through the GTH interface of the Zynq UltraScale+MPSoC chip, and the SFP+ optical module converts the electrical signal into an optical signal and sends it to the back end or terminal server through optical fiber.