A design method of a ZYNQ internet of things terminal with edge AI video analysis function

Abstract

The application provides a ZYNQ Internet of Things terminal design method with edge AI video analysis function, and relates to the technical field of Internet of Things edge computing terminal design. The ZYNQ Internet of Things terminal design method with edge AI video analysis function comprises a ZYNQ processor, a PL end DPU video module of the ZYNQ processor, a PS end DPU control module and an AI DPU module. The ZYNQ processor is connected with the AI DPU module through the DPU video module and the DPU control module. The PS end of the ZYNQ processor collects camera video, performs decoding operation and obtains image data. The DPU video module performs format conversion and packaging on the image data, obtains video data and transmits the video data to the AI DPU module. The AI DPU module performs artificial intelligence identification on the video data, obtains AI video identification results and sends the AI video identification results to the ZYNQ processor. The ZYNQ processor comprehensively analyzes the identification results and obtains analysis results. The application can reduce the deployment time, cost and power consumption of software and hardware.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things edge computing terminal design, in particular to a ZYNQ Internet of Things terminal design method and system with edge AI video analysis function. BACKGROUND

[0002] With the development of Internet of Things and monitoring video technology, the demand for video monitoring of related smart environment monitoring projects is gradually increasing, regardless of whether the communication is in a convenient area. The analysis of video is increasingly not satisfied with the evidence and accountability after the accident. The demand for event monitoring and early warning through video monitoring is increasing, and the real-time video processing and recognition such as face recognition, people flow monitoring, behavior and action recognition, and suspicious person recognition application demand is becoming more and more widespread.

[0003] Traditional cloud computing has insufficient communication bandwidth, excessive power consumption and insufficient AI computing power for real-time analysis of large-scale video data generated at the edge, so real-time video processing needs to be solved on edge devices, and small-flow video recognition results are used to replace large-flow high-definition video streams reported to the platform or remote server.

[0004] However, the current development of edge-oriented video collection and AI analysis function terminal has the following status and problems: (1) mature AI video analysis equipment can have more recognition modes and high computing power, but the cost and power consumption are generally high, which is not suitable for deployment at the edge; (2) the cost of special AI camera is high, the AI algorithm is single, it is not convenient for secondary development, and replacing the camera in the area where the ordinary camera has been deployed causes waste, in addition, it cannot integrate other sensors to analyze and utilize the environmental information other than video; (3) in remote areas where network is not convenient, the cost of building a video private network is very high, and the support capability of edge devices for wireless communication needs to be improved; (4) the AI DPU (Deep learning Processing Unit, deep learning processor) module is a special algorithm module with low power consumption and certain flexibility, but it usually requires a specific video transmission interface and a non-compressed video timing and format, needs to be integrated as a slave device in the edge device, and reserves the corresponding command control interface and firmware upgrade interface, which has certain requirements for the performance of the main processor of the edge device; the existing Internet of Things terminal device has video access capability, but the ability to expand AI function is weak, and there is no guiding method for integrating different AI DPU modules, which has great development difficulty. SUMMARY

[0005] In order to solve the technical problems of the prior art that the ability of expanding AI function is weak, there is lack of guiding method for integration of different AI DPU modules, and the development is difficult, and the cost and power consumption of the mature AI video analysis device are generally high, the embodiment of the present application provides a ZYNQ Internet of Things terminal design method and system with edge AI video analysis function.The technical solution is as follows:

[0006] In one aspect, a ZYNQ Internet of Things terminal design method with edge AI video analysis function is provided, which is realized by a ZYNQ Internet of Things terminal device with edge AI video analysis function, and the method comprises the following steps:

[0007] S1, the ZYNQ processor is connected with the AI DPU module through the DPU video module of the PL end of the ZYNQ processor and the DPU control module of the PS end of the ZYNQ processor as the main processor;

[0008] S2, the PS end of the ZYNQ processor collects camera video and performs decoding operation to obtain image data; the DPU video module of the PL end of the ZYNQ processor performs format conversion and packaging on the image data to obtain video data; and the DPU video module of the PL end of the ZYNQ processor transmits the video data to the AI DPU module;

[0009] S3, the AI DPU module receives the video data, performs artificial intelligence identification on the video data to obtain AI video identification result; and the AI DPU module sends the AI video identification result to the ZYNQ processor;

[0010] S4, the ZYNQ processor receives the AI video identification result, comprehensively analyzes the AI video identification result, and obtains an analysis result.

[0011] Optionally, the system of the ZYNQ Internet of Things terminal further comprises a memory module, a camera interface module, a sensor control module, an indicator light module, a network communication module, a wireless transmission module and a voice processing module;

[0012] The sensor control module comprises a processor chip internal sensor, a sensor on a system mainboard and an interface for external sensor;

[0013] The memory module comprises a running memory DDR, a FLASH and an SD card.

[0014] Optionally, the indicator light module is used for displaying and alarming the status of the ZYNQ Internet of Things terminal; wherein the indicator light module comprises: a power indicator light 1-way, a network connection status indicator light 2-way, an AI DPU module status indicator light 1-way, and a sensor indicator light 1-way.

[0015] The power indicator light 1-way is used for indicating whether the equipment is powered on.

[0016] The network connection status indicator light 2-way comprises: a first way for indicating whether the wireless network link is connected; and a second way for indicating the wireless network data reporting state.

[0017] The AI DPU module status indicator light 1-way is used for indicating whether the AI DPU module is working, hibernating or malfunctioning.

[0018] The sensor indicator light 1-way is used for indicating the alarm state of the sensor value.

[0019] Optionally, the DPU control module of the PS end of the ZYNQ processor comprises: a UART interface and a firmware upgrade interface.

[0020] The DPU control module of the PS end of the ZYNQ processor is used for the ZYNQ processor PS end to allocate the UART interface to command control the AI DPU module, the ZYNQ processor sends a control or request instruction, and the AI DPU module sends the execution result to the ZYNQ processor; the firmware upgrade interface allocated by the ZYNQ processor PS end is used for sending a firmware upgrade package to the AI DPU module, the AI DPU module runs the firmware upgrade program after decompressing the file, and updates the firmware version.

[0021] Optionally, the DPU video module of the PL end of the ZYNQ processor is located at the PL end of the ZYNQ processor, and comprises: a VDMA IP core, a video format conversion IP core, a VO-BUFFER IP core and a DPU video interface IP core; wherein,

[0022] The VDMA IP core is an official IP of XILINX, uses an AXI HP port to read and write the image by the PL end DDR and to transmit the stream data of the image video by the PS to the PL end, and uses an AXI GP port to control the register of the VDMA IP core.

[0023] The video format conversion IP core is a self-defined IP, and reading and writing of the video stream data on the DDR are completed by connecting the VDMA IP core; the video format conversion IP core obtains the image containing video information buffered on the DDR after the PS end performs video decoding frame by frame, converts the image into an image in a video format required by the AI DPU module through a format conversion algorithm, and writes the image into the DDR;

[0024] The VO-BUFFER IP core and the DPU video interface IP core are self-defined IPs; the VO-BUFFER IP core completes DDR cache reading by connecting the VDMA IP core, encapsulates the image according to a frame format required by the AI DPU module, obtains the encapsulated image, and transmits the image to the DPU video interface IP; the DPU video interface IP core constructs a video data transmission interface according to a frame timing required by the AI DPU module, leads out an electrical pin connected to the AI DPU module in a physical manner, and transmits the processed video data to the AI DPU module; wherein the electrical pin of the video transmission interface includes a data line pin and a clock line pin.

[0025] Optionally, the PS end and the PL end of the ZYNQ processor communicate through AXIBUS; wherein an AXIGP interface type in the AXIBUS is used for a general-purpose peripheral interface of the PS end and low-speed communication between the PL end and the PS end; an AXIHP interface in the AXIBUS is used for a DDR interface of the PS end and high-speed communication between the PL end and the PS end, and high-speed reading and writing of the DDR by the PL end are completed through the AXIHP.

[0026] Optionally, the ZYNQ processor of the S4 receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result, including:

[0027] S41, the system mainboard of the ZYNQ Internet of Things terminal is powered on, the ZYNQ processor and the AI DPU module are initialized, the ZYNQ processor allocates memory space for a video transmission and format conversion thread and creates a local file; wherein the local file is saved on an SD card and includes a running log and an AI video processing result folder;

[0028] S42, the DPU control module of the PS end of the ZYNQ processor sends a request video processing command to the AI DPU module through UART, the ZYNQ processor listens to the reply of the UART, the AI DPU module parses the command after receiving the command, and calls an internal program of the AI DPU module;

[0029] S43, the camera interface module acquires a video stream according to an external camera, the ZYNQ processor decodes the video stream, and stores video frames into DDR through a PS end; a video format conversion IP core of a PL end of the ZYNQ processor converts the video frames into images in a format required by an AI DPU, and writes the images back to DDR; a VO-BUFFER encapsulates the images into a video stream in a format required by the AI DPU according to a time sequence, and sends the video stream to the AI DPU module through a DPU video interface;

[0030] S44, a video interface unit of the AI DPU module receives a video, identifies a target to be detected in the video through an artificial intelligence algorithm, obtains an AI video recognition result, and sends the AI video recognition result to the ZYNQ processor through a UART interface; wherein the AI video recognition result includes a feature map and text information;

[0031] S45, the ZYNQ processor analyzes and filters data of the received AI video recognition result, stores the AI video recognition result in an AI video processing result folder, records a DPU running log, and packs the AI video recognition result and other sensor information to be reported, and reports the information through a network communication module.

[0032] On the other hand, a ZYNQ Internet of Things terminal design system with edge AI video analysis function is provided, which is applied to a ZYNQ Internet of Things terminal design method with edge AI video analysis function, and the system comprises:

[0033] The ZYNQ processor is used as a main processor to connect with the AI DPU module through a DPU video module of a PL end of the ZYNQ processor and a DPU control module of a PS end of the ZYNQ processor; the PS end of the ZYNQ processor collects camera video, performs decoding operation, and obtains image data; the ZYNQ processor receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result;

[0034] The DPU video module of the PL end of the ZYNQ processor is used to perform format conversion and encapsulation on the image data, and obtain video data; the DPU video module of the PL end of the ZYNQ processor is used to transmit the video data to the AI DPU module;

[0035] The AI DPU module is used to receive the video data, perform artificial intelligence identification on the video data, and obtain an AI video recognition result; the AI DPU module is used to send the AI video recognition result to the ZYNQ processor.

[0036] The DPU control module at the PS end of the ZYNQ processor is connected with the DPU video module at the PL end of the ZYNQ processor and the DPU control module at the PS end of the ZYNQ processor and the AI DPU module.

[0037] Optionally, the system of the ZYNQ Internet of Things terminal further comprises a memory module, a camera interface module, a sensor control module, an indicator light module, a network communication module, a wireless transmission module and a voice processing module.

[0038] The sensor control module comprises a processor chip internal sensor, a system mainboard sensor and an interface for an external sensor.

[0039] The memory module comprises a running memory DDR, a FLASH and an SD card.

[0040] Optionally, the indicator light module is used for displaying and alarming the state of the ZYNQ Internet of Things terminal, and the indicator light module comprises one-way power indicator light, two-way network connection state indicator light, one-way AI DPU module state indicator light and one-way sensor indicator light.

[0041] The one-way power indicator light is used for indicating whether the device is powered on.

[0042] The two-way network connection state indicator light comprises a first way for indicating whether a wireless network link is connected and a second way for indicating a wireless network data reporting state.

[0043] The one-way AI DPU module state indicator light is used for indicating whether the AI DPU module is working, sleeping or malfunctioning.

[0044] The one-way sensor indicator light is used for indicating an alarm state of a sensor value.

[0045] Optionally, the DPU control module at the PS end of the ZYNQ processor comprises a UART interface and a firmware upgrade interface.

[0046] The DPU control module at the PS end of the ZYNQ processor is used for distributing a UART interface at the PS end of the ZYNQ processor to command control the AI DPU module, the ZYNQ processor sends a control or request instruction, and the AI DPU module sends an execution result to the ZYNQ processor; a firmware upgrade interface at the PS end of the ZYNQ processor is distributed to send a firmware upgrade package to the AI DPU module, the AI DPU module decompresses the file and runs a firmware upgrade program to update the firmware version.

[0047] Optionally, the DPU video module at the PL end of the ZYNQ processor is located at the PL end of the ZYNQ processor and includes a VDMA IP core, a video format conversion IP core, a VO-BUFFER IP core, and a DPU video interface IP core.

[0048] The VDMA IP core is an official IP of XILINX, uses an AXI HP port to perform read and write of images by the PL end DDR and stream data transmission of image videos by the PS to the PL end, and uses an AXI GP port to control registers of the VDMA IP core.

[0049] The video format conversion IP core is a self-defined IP, and read and write of video stream data on the DDR are completed by connecting the VDMA IP core; the video format conversion IP core obtains images containing video information buffered on the DDR after video decoding by the PS end frame by frame, converts the images into images in a video format required by the AI DPU module by a format conversion algorithm, and writes the images into the DDR.

[0050] The VO-BUFFER IP core and the DPU video interface IP core are self-defined IPs; the VO-BUFFER IP core completes DDR cache reading by connecting the VDMA IP core, encapsulates images in a frame format required by the AI DPU module, obtains encapsulated images, and transmits the images to the DPU video interface IP; the DPU video interface IP core constructs a video data transmission interface according to a frame timing required by the AI DPU module, leads out electrical pins and is physically connected to the AI DPU module, and transmits processed video data to the AI DPU module; wherein the electrical pins of the video transmission interface include data line pins and clock line pins.

[0051] Optionally, the PS end and the PL end of the ZYNQ processor communicate through an AXIBUS; wherein an AXI GP interface type in the AXIBUS is used for a general-purpose peripheral interface of the PS end and low-speed communication between the PL end and the PS end; an AXI HP interface in the AXIBUS is used for a DDR interface of the PS end and high-speed communication between the PL end and the PS end, and high-speed read and write of the PL end to the DDR are completed through the AXI HP.

[0052] Optionally, the ZYNQ processor receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result, including:

[0053] The system mainboard of the ZYNQ Internet of Things terminal is powered on, and the ZYNQ processor and the AI DPU module complete initialization, and the ZYNQ processor allocates memory space for video transmission and format conversion threads and creates a local file; wherein, the local file is saved on an SD card, including a running log and an AI video processing result folder;

[0054] The DPU control module of the PS end of the ZYNQ processor sends a request video processing command to the AI DPU module through UART, and the ZYNQ processor listens to the reply of UART, and the AI DPU module parses after receiving the command, and calls the internal program of the AI DPU module;

[0055] The camera interface module acquires a video stream according to an external camera, the ZYNQ processor decodes the video stream, and buffers the video frame to DDR through the PS end; the video format conversion IP core of the PL end of the ZYNQ processor converts the video frame into an image in a format required by the AI DPU, and writes the image back to DDR; the VO-BUFFER encapsulates the image into a video stream in a format required by the AI DPU according to a time sequence, and sends the video stream to the AI DPU module through the DPU video interface;

[0056] The video interface unit of the AI DPU module receives a video, identifies a target to be detected in the video through an artificial intelligence algorithm, and obtains an AI video recognition result; the AI video recognition result is sent to the ZYNQ processor through the UART interface; wherein, the AI video recognition result includes a feature map and text information;

[0057] The ZYNQ processor parses the received AI video recognition result and performs data screening processing, stores the AI video recognition result in the AI video processing result folder, records the DPU running log, and packs the AI video recognition result and other sensor information to be reported, and reports the information through a network communication module.

[0058] On the other hand, a ZYNQ Internet of Things terminal device with edge AI video analysis function is provided, which comprises a processor and a memory, wherein the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to realize any one of the above-mentioned ZYNQ Internet of Things terminal design methods with edge AI video analysis function.

[0059] On the other hand, a computer readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to realize any one of the above-mentioned ZYNQ Internet of Things terminal design methods with edge AI video analysis function.

[0060] The technical scheme provided by the embodiment of the present application has at least the following beneficial effects:

[0061] The embodiment of the present application connects the ZYNQ processor as the main processor through the DPU video module of the PL end of the ZYNQ processor and the DPU control module and the AI DPU module of the PS end of the ZYNQ processor; the PS end of the ZYNQ processor collects camera video and performs decoding operation to obtain image data; the DPU video module of the PL end of the ZYNQ processor performs format conversion and packaging on the image data to obtain video data; the DPU video module of the PL end of the ZYNQ processor transmits the video data to the AI DPU module; the AI DPU module receives the video data, performs artificial intelligence identification on the video data to obtain AI video identification results; the AI DPU module sends the AI video identification results to the ZYNQ processor; the ZYNQ processor receives the AI video identification results, comprehensively analyzes the AI video identification results, and obtains analysis results.

[0062] The embodiment of the present application can make the power and algorithm of the AI DPU module independent of the development of the function of the ZYNQ processor end by quickly integrating the AI video processing function based on the ZYNQ Internet of Things terminal device; the ZYNQ processor acts as a bridge for video transmission between different types of coded cameras and different types of AI DPU modules, but does not participate in the process of deep neural network inference, logic resources and power consumption are controllable; the ZYNQ processor can comprehensively process the processing results of the AI DPU module and the sensor data, can enrich data analysis and data fusion application development, and has good functional expansibility; the embodiment of the present application utilizes the software and hardware collaborative design characteristics of the ZYNQ heterogeneous processor, divides resources and designs modules for the processor end and the programmable logic end of the ZYNQ, enhances interface diversity, and is compatible with multiple interfaces including cameras, AI DPU modules, sensors and wireless transmission modules; fully utilizes the characteristics of AXI bus and FPGA to improve data throughput and real-time computing capability. The use of the present application can make the ZYNQ Internet of Things terminal quickly integrate the AI video analysis function, which is suitable for Internet of Things transmission and edge computing, and reduces the deployment time, cost and power consumption of software and hardware. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1It is a ZYNQ Internet of Things terminal design method flow chart provided by the embodiment of the application with edge AI video analysis function.

[0065] Figure 2 It is a system mainboard function module design and hardware connection schematic diagram provided by the embodiment of the application.

[0066] Figure 3 It is a hardware schematic diagram of a ZYNQ processor part AI video analysis function module provided by the embodiment of the application.

[0067] Figure 4 It is an AI DPU module and interface schematic diagram corresponding to a ZYNQ processor DPU interface provided by the embodiment of the application.

[0068] Figure 5 It is a flow schematic diagram of a method for running an AI video analysis function provided by the embodiment of the application.

[0069] Figure 6 It is a ZYNQ Internet of Things terminal design system block diagram provided by the embodiment of the application with edge AI video analysis function.

[0070] Figure 7 It is a structure schematic diagram of a ZYNQ Internet of Things terminal device provided by the embodiment of the application with edge AI video analysis function. DETAILED DESCRIPTION

[0071] The technical solutions in the application will be described below with reference to the drawings.

[0072] In the embodiments of the application, the words such as "example", "for example" and the like are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0073] In the embodiments of the application, "image" and "picture" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "corresponding" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0074] In the embodiments of the application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0075] In order to make the technical problems, technical solutions and advantages to be solved by the present application more clear, the following will be described in detail in combination with the drawings and specific embodiments.

[0076] The embodiment of the present application provides a ZYNQ Internet of Things terminal design method with edge AI video analysis function, which can be realized by a ZYNQ Internet of Things terminal device with edge AI video analysis function. The ZYNQ Internet of Things terminal device with edge AI video analysis function can be a terminal or a server. As shown in the flow chart of the ZYNQ Internet of Things terminal design method with edge AI video analysis function, the processing flow of the method can include the following steps: Figure 1

[0077] S1, the ZYNQ processor as the main processor is connected with the AI DPU module through the DPU video module of the PL end of the ZYNQ processor and the DPU control module of the PS end of the ZYNQ processor.

[0078] The ZYNQ Internet of Things terminal with edge AI video analysis function can be used for monitoring pedestrians at the intersection on the mountain in the scenic area.

[0079] Optionally, the PS end and the PL end of the ZYNQ processor communicate through AXIBUS; wherein the AXIGP interface type in AXIBUS is used for the general-purpose peripheral interface of the PS end and the low-speed communication between the PL end and the PS end; the AXIHP interface in AXIBUS is used for the DDR interface of the PS end and the high-speed communication between the PL and the PS end, and the high-speed reading and writing of the DDR by the PL end is completed through AXIHP.

[0080] Optionally, the system of the ZYNQ Internet of Things terminal further includes a memory module, a camera interface module, a sensor control module, an indicator light module, a network communication module, a wireless transmission module and a voice processing module.

[0081] The sensor control module includes a processor chip internal sensor, a sensor on a system mainboard and an interface for an external sensor.

[0082] The memory module includes a running memory DDR, a FLASH and an SD card.

[0083] Optionally, the indicator light module is used for displaying and alarming the state of the ZYNQ Internet of Things terminal; wherein the indicator light module includes: a power indicator light 1 way, a network connection state indicator light 2 ways, an AI DPU module state indicator light 1 way and a sensor indicator light 1 way.

[0084] The power indicator light 1 way is used for indicating whether the device is powered on.

[0085] ​The network connection status indicator light 2-way includes: the first way is used for indicating whether the wireless network link is connected; the second way is used for indicating the wireless network data reporting state;

[0086] The AI DPU module state indicator light 1-way is used for indicating whether the AI DPU module is working, sleeping or malfunctioning.

[0087] The sensor indicator light 1-way is used for indicating the alarm state of the sensor value.

[0088] Optionally, the DPU control module at the PS end of the ZYNQ processor includes: a UART interface and a firmware upgrade interface.

[0089] The DPU control module at the PS end of the ZYNQ processor is used for the ZYNQ processor PS end to allocate the UART interface to command control the AI DPU module, the ZYNQ processor sends control or request instructions, and the AI DPU module sends the execution result to the ZYNQ processor; the firmware upgrade interface allocated by the ZYNQ processor PS end is used for sending the firmware upgrade package to the AI DPU module, the AI DPU module runs the firmware upgrade program after decompressing the file, and updates the firmware version.

[0090] Optionally, the DPU video module at the PL end of the ZYNQ processor is located at the PL end of the ZYNQ processor and includes: a VDMA IP core, a video format conversion IP core, a VO-BUFFER IP core and a DPU video interface IP core; wherein,

[0091] The VDMA IP core is an official IP of XILINX, uses an AXI HP port to read and write the image on the PL end DDR and to transmit the stream data of the image video from the PS to the PL end, and uses an AXI GP port to control the register of the VDMA IP core;

[0092] The video format conversion IP core is a self-defined IP, and reads and writes the video stream data on the DDR by connecting the VDMA IP core; the video format conversion IP core obtains the image containing video information buffered on the DDR after the PS end decodes the video frame by frame, converts the image into the video format required by the AI DPU module through a format conversion algorithm, and writes the image into the DDR;

[0093] The VO-BUFFER IP core and the DPU video interface IP core are self-defined IP; the VO-BUFFER IP core completes DDR cache reading by connecting the VDMA IP core, encapsulates images according to the frame format required by the AI DPU module, and transmits the encapsulated images to the DPU video interface IP; the DPU video interface IP core constructs a video data transmission interface according to the frame timing required by the AI DPU module, leads out electrical pins and is physically connected to the AI DPU module, and transmits the processed video data to the AI DPU module; wherein the electrical pin of the video transmission interface includes a data line pin and a clock line pin.

[0094] The network communication module can report edge data to the cloud platform in wired gigabit Ethernet, Wi-Fi, LoRa and 4G transmission modes; when the wireless communication mode is LoRa, the ZYNQ Internet of Things terminal device reports the edge data to the base station, and the base station sends the edge data to the cloud platform.

[0095] The ZYNQ main processor manages all modules integrated in the system, which can reduce development time and difficulty.

[0096] In an available implementation, the ZYNQ processor is a processor chip of the XILINX 7 series with ARM A9 dual-core+FPGA heterogeneous multi-core architecture, which has the characteristics of strong processing performance, rich logic and interface resources, and low power consumption, and can be applied to embedded edge terminals.

[0097] The ZYNQ processor is a processor chip of the XILINX 7 series with ARM A9 dual-core+FPGA heterogeneous multi-core architecture, which has the characteristics of strong processing performance, rich logic and interface resources, and low power consumption, and can be applied to embedded edge terminals. Figure 2 is a system mainboard function module design and hardware connection schematic diagram provided by an embodiment of the application; the system at least includes: a ZYNQ chip, a memory module, a camera interface module, a sensor control module, an AI DPU module, an indicator light module, a wireless transmission module and a voice processing module; the sensor control module includes a sensor in a processor chip, a sensor on a system mainboard and an interface for an external sensor.

[0098] The AI DPU module is a slave device of the ZYNQ processor and can identify AI targets of a video stream, but the interface configuration is usually low, a non-compressed video stream with specific image format and specific video interface needs to be provided, and the camera cannot be directly accessed and managed; the ZYNQ chip is adopted to decompress and decode the video stream of the collected images and convert the format, and the video is sent to the AI DPU module through the interface integrated by the IP core.

[0099] The ZYNQ chip is a main processor managing each interface and module, the PS end of the ZYNQ chip is an ARM core and an MIO controller, is responsible for sensor interface control and collection function, camera module access and collection, indicator light function, wireless module access management and communication program and DPU control module function, and the PL end of the ZYNQ chip is a programmable logic part, and the functions of image format conversion, DPU video interface and voice module interface are integrated by building an IP core.

[0100] In an available implementation, the ZYNQ processor chip is an XC7Z010-CLG400 processor, the external camera adopts a network camera, usually in H.264 and H.265 video formats, the AI DPU model is a deep identification DPB1101 module, the module main processor is a ZYNQ7020 chip, integrates an artificial intelligence recognition algorithm and a BT1120 video interface, can process an image at a speed of 30Fps / 1080P, completes face and pedestrian recognition, and the obtained recognition result is a feature map and a text sequence, which are transmitted to the ZYNQ processor through UART; the firmware upgrade interface of the DPB1101 module is SPI; when upgrading, the ZYNQ sends a firmware upgrade request to the DPB1101 through UART, sends an upgrade file through the ZYNQ SPI interface, and the DPB1101 module receives, parses and executes the upgrade after receiving.

[0101] The external camera can adopt a USB interface camera, usually adopts an MPEG compression mode, or can adopt a CMOS camera based on OmniVision, and is connected with a CMOS camera interface IP core designed in the PL end of the ZYNQ, and one set of logic resources of the AI video analysis terminal based on the ZYNQ71010 processor in the embodiment can complete real-time video processing of one network camera or USB camera or CMOS camera at 30Fps / 1080P.

[0102] The sensor control module in the embodiment of the application integrates a GPS positioning module, a temperature sensor and an infrared sensor, and the external sensor interface includes RS232, RS485 and CAN ports, and is used for connecting the data interface of the external sensor.

[0103] S2, the PS end of the ZYNQ processor collects camera video and performs decoding operation to obtain image data; the DPU video module of the PL end of the ZYNQ processor performs format conversion and packaging on the image data to obtain video data; and the DPU video module of the PL end of the ZYNQ processor transmits the video data to the AI DPU module.

[0104] The PS end of the ZYNQ processor collects camera video in 720P or 1080P video.

[0105] The collected video content can include pedestrians and vehicles at an intersection on a mountain in a scenic area.

[0106] In a feasible implementation manner, the AI video processing function module includes a ZYNQ processor and an AI DPU module. Figure 3 is a hardware schematic diagram of a ZYNQ processor part AI video analysis function module provided by an embodiment of the application; wherein the camera interface is composed of an Ethernet controller, a GigEPHY and an ETHSwitch module at the PS end; the DPU control interface is composed of a PS end UART controller, a DPU firmware upgrade interface and a GPIO control pin; the DPU video interface is composed of a plurality of AXI-VDMA IP cores, an image format conversion IP, a VO-BUFFER IP core and a DPU video interface IP.

[0107] The PS end of the ZYNQ processor communicates through an AXIBUS, and an AXIGP interface type in the AXIBUS is used for processor interconnection at the PS end and general peripheral interface, low-speed communication between the PL end and the PS end; an AXIHP interface in the AXIBUS is used for the PS end DDR interface and high-speed communication between the PL end and the PS end, and the PL end can perform high-speed reading and writing on the DDR through the AXIHP.

[0108] In a feasible implementation manner, the AXI-VDMA IP core in the embodiment of the application is an official IP of XILINX, plays a key role in high-speed image carrying and DDR reading and writing, and simultaneously applies AXI HP and AXI GP ports to change the obtained video information stream into an AXI STEAM; the cooperation relationship between the VDMA IP core and other IPs is that the image format conversion IP is matched with the VDMA IP core to enable the image to perform the processes of DDR reading, image format conversion and image writing into the DDR; the VO-BUFFER IP is matched with the VDMA IP core to enable the image to perform the processes of DDR reading, image format packaging and transmission of the packaged result to the DPU video interface IP.

[0109] The DPU video interface includes data pins and clock pins, and is connected to the AI DPU module video interface pins through the PL end EMIO in a physical manner.

[0110] The process of AI video processing by the main processor includes that the ZYNQ processor sends a request command through the UART controller, listens to the UART serial port, the PS end starts to collect the camera video and decodes the video into an image written into the DDR, the VDMA IP core carries the image at the PL end, the image format conversion IP writes back the DDR after converting the image into a format, the VO-BUFFER module packages the video format, and the DPU video interface transmits the video to the AI DPU module.

[0111] The ZYNQ processor adopts a LINUX operating system and transplants an FFMPEG library to perform video decoding.

[0112] S3, the AI DPU module receives video data, obtains an AI video recognition result by performing artificial intelligence recognition on the video data, and sends the AI video recognition result to the ZYNQ processor.

[0113] In the embodiment of the application, Figure 4 is a ZYNQ processor DPU interface corresponding AI DPU module and interface schematic diagram provided by the embodiment of the application, also an AI DPU module external interface schematic diagram; wherein, the AI DPU module external interface corresponding to the ZYNQ processor includes: a video transmission interface, a UART communication interface, a firmware upgrade interface, a RESET GPIO interface, a power module and an indicator light module.

[0114] In a feasible implementation manner, the AI DPU module is initialized, the UART communication interface is listened to, and the received command sent by the ZYNQ processor is parsed; if the received command is an AI video analysis command, the video is obtained from the video transmission interface, the artificial intelligence recognition algorithm is run, and the AI video processing result is obtained; the AI video processing result is transmitted to the ZYNQ processor through the UART communication interface, and if the implementation is unsuccessful, the timeout report and the link text where the problem occurs are transmitted to the ZYNQ processor through the UART communication interface;

[0115] The artificial intelligence recognition algorithm can be a convolutional neural network algorithm CNN and a multi-layer perceptron MLP, and the embodiment of the application does not limit this.

[0116] The command sent by the ZYNQ processor also includes a command for requesting AI DPU information, a configuration modification command and a firmware upgrade command.

[0117] The indicator light of the AI DPU module includes a power indicator light and a running state indicator light, which are used for debugging indication in the integrated work of the ZYNQ Internet of Things end.

[0118] The AI DPU module is powered by a separate power supply channel, and the embodiment of the application controls the power-on and shutdown of the AI DPU power supply by using the GPIO interface on the ZYNQ main processor chip on the system mainboard.

[0119] S4, the ZYNQ processor receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result.

[0120] The analysis result is determined according to the function of the AI DPU module, and the AI DPU module usually returns a text, a screenshot or a cutout.

[0121] The specific content of the comprehensive analysis of the AI video recognition result can include a face recognition result, a people count, and a vehicle recognition result.

[0122] The method for running the AI video analysis function is performed after the system mainboard of the ZYNQ Internet of Things terminal is powered on and the ZYNQ processor and the AI DPU complete initialization operations.

[0123] In a possible implementation manner, the method for running the AI video analysis function comprises the following steps: Figure 5 is a flowchart of the method for running the AI video analysis function; in a possible implementation manner, the process of completing sensor collection and reporting once by the method for running the AI video analysis function comprises the following steps:

[0124] (1) The sensor control module collects external environment information at regular time intervals, processes the information, and stores the processed information;

[0125] (2) The camera interface module collects a video stream, decodes the video stream through the PS end, and writes the decoded video stream into a DDR cache through format conversion of the image format conversion IP of the PL end;

[0126] (3) The VO-BUFFER and the video interface IP obtain an image, encapsulate the image into a specific format, and send the encapsulated image to the AI DPU module for processing; the AI DPU runs an artificial intelligence recognition algorithm to obtain an AI video processing result;

[0127] (4) The AI video processing result is returned to the ZYNQ processor through the DPU control module;

[0128] (5) The ZYNQ processor stores the AI video processing result, processes and packs the AI video processing result and sensor information according to a network reporting requirement, and reports the AI video processing result and the sensor information to a northbound interface through a network communication module.

[0129] The ZYNQ processor performs comprehensive analysis on the obtained sensor information and the AI video processing result, determines whether voice prompt broadcasting is needed, and if so, runs a voice broadcasting related program to broadcast the voice prompt; if not, the voice broadcasting related program is not executed.

[0130] The indicator light module comprises five indicator lights, which are used for device state display and alarm, wherein a power indicator light is indicator light 1, which indicates whether the device is powered on; a network connection state is indicator light 2, wherein indicator light 2 indicates whether a wireless network link is connected, and indicator light 3 indicates a wireless network data reporting state; an AI DPU module state indicator light is indicator light 4, which is used for indicating whether the AI DPU module is working, sleeping, or malfunctioning; and a sensor indicator light is indicator light 5, which indicates an alarm state of a sensor value.

[0131] In a feasible implementation, after power-on, the indicator light 1 is always on; after power-on, the wireless network link is always on, when data is reported to the northbound interface, the indicator light 2 is always on, when the AIDPU module is running, the indicator light 3 is always on, when the sensor is not working, the indicator light 4 is not on, and when the sensor is working abnormally, the indicator light 4 is flashing; when the sensor is working normally, the indicator light 5 is always on, when the value appears an alarm, the indicator light 5 is flashing, and when the sensor is not working, the indicator light 5 is not on.

[0132] Optionally, the specific implementation process of S4 can include S41-S45.

[0133] S41, the system mainboard of the ZYNQ Internet of Things terminal is powered on, the ZYNQ processor and the AIDPU module complete initialization, the ZYNQ processor allocates memory space for the video transmission and format conversion thread and creates a local file; wherein the local file is saved on the SD card, including a running log and an AI video processing result folder;

[0134] S42, the DPU control module of the PS end of the ZYNQ processor sends a request video processing command to the AIDPU module through the UART, the ZYNQ processor listens to the reply of the UART, the AIDPU module parses the command after receiving the command, and calls the internal program of the AIDPU module;

[0135] S43, the camera interface module acquires a video stream according to an external camera, the ZYNQ processor decodes the video stream, and buffers the video frame to the DDR through the PS end; the video format conversion IP core of the PL end of the ZYNQ processor converts the video frame into an image in a format required by the AIDPU, and writes the image back to the DDR; the VO-BUFFER encapsulates the image into a video stream in a format required by the AIDPU according to a time sequence, and sends the video stream to the AIDPU module through the DPU video interface;

[0136] Wherein, the external camera can be a 1-way CMOS camera, a USB digital camera, or a 1-way RJ45 network camera.

[0137] S44, the video interface unit of the AIDPU module receives a video, identifies a to-be-detected target in the video through an artificial intelligence algorithm, and obtains an AI video recognition result; the AI video recognition result is sent to the ZYNQ processor through the UART interface; wherein the AI video recognition result includes a feature map and text information;

[0138] Wherein, the text information can include a coordinate text of a video in which a face appears and a corresponding cutout, and a number of people or a license plate recognition result.

[0139] Wherein, the to-be-detected target can include a face or a number of people.

[0140] S45, the ZYNQ processor parses the received AI video recognition result and performs data filtering processing, stores the AI video recognition result in an AI video processing result folder, records a DPU operation log, and packs the AI video recognition result and other sensor information to be reported, and reports the information through a network communication module.

[0141] In the embodiment of the application, the ZYNQ processor is used as a main processor, and the DPU video module at the PL end of the ZYNQ processor and the DPU control module at the PS end of the ZYNQ processor are connected with the AI DPU module; the PS end of the ZYNQ processor collects camera video and performs decoding operation to obtain image data; the DPU video module at the PL end of the ZYNQ processor performs format conversion and packaging on the image data to obtain video data; the DPU video module at the PL end of the ZYNQ processor transmits the video data to the AI DPU module; the AI DPU module receives the video data, performs artificial intelligence identification on the video data to obtain an AI video recognition result; the AI DPU module sends the AI video recognition result to the ZYNQ processor; the ZYNQ processor receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result.

[0142] In the embodiment of the application, the ZYNQ Internet of Things terminal device is quickly integrated with the AI video processing function based on the ZYNQ, so that the power and algorithm of the AI DPU module are independent of the development of the function of the ZYNQ processor end; the ZYNQ processor is used as a bridge for video transmission between different types of coded cameras and different types of AI DPU modules, but does not participate in the process of deep neural network inference, logic resources and power consumption are controllable; the ZYNQ processor can comprehensively process the processing result of the AI DPU module and the sensor data, can enrich the development of data analysis and data fusion application, and has good functional expansibility; the embodiment of the application utilizes the software and hardware collaborative design characteristics of the ZYNQ heterogeneous processor, divides resources and designs modules for the processor end and the programmable logic end of the ZYNQ, enhances interface diversity, and is compatible with multiple interfaces including cameras, AI DPU modules, sensors and wireless transmission modules; the characteristics of the AXI bus and the FPGA are fully utilized to improve data throughput and real-time computing capability. The ZYNQ Internet of Things terminal can be quickly integrated with the AI video analysis function by using the application, which is suitable for Internet of Things transmission and edge computing, and reduces the deployment time, cost and power consumption of software and hardware.

[0143] Figure 6 A ZYNQ Internet of Things terminal design system block diagram with edge AI video analysis function is shown according to an exemplary embodiment, and the system is used for a ZYNQ Internet of Things terminal design method with edge AI video analysis function. Referring to Figure 6The system comprises a ZYNQ processor 310, a DPU video module 320 at the PL end of the ZYNQ processor, an AI DPU module 330, and a DPU control module 340 at the PS end of the ZYNQ processor. Among them:

[0144] The ZYNQ processor 310 is configured to connect with the AI DPU module through the DPU video module at the PL end of the ZYNQ processor and the DPU control module at the PS end of the ZYNQ processor as a master processor; the PS end of the ZYNQ processor collects camera video, performs decoding operation, and obtains image data; the ZYNQ processor receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result;

[0145] The DPU video module 320 at the PL end of the ZYNQ processor is configured to perform format conversion and packaging on the image data, and obtain video data; the DPU video module at the PL end of the ZYNQ processor is configured to transmit the video data to the AI DPU module;

[0146] The AI DPU module 330 is configured to receive the video data, and obtain an AI video recognition result by performing artificial intelligence recognition on the video data; the AI DPU module is configured to send the AI video recognition result to the ZYNQ processor.

[0147] The DPU control module 340 at the PS end is configured to connect the DPU video module at the PL end of the ZYNQ processor and the DPU control module at the PS end of the ZYNQ processor with the AI DPU module.

[0148] Optionally, the system of the ZYNQ Internet of Things terminal further comprises a memory module, a camera interface module, a sensor control module, an indicator light module, a network communication module, a wireless transmission module, and a voice processing module.

[0149] The sensor control module comprises a processor chip internal sensor, a system mainboard sensor, and an interface for external sensors.

[0150] The memory module comprises a running memory DDR, a FLASH, and an SD card.

[0151] Optionally, the indicator light module is configured to display and alarm the state of the ZYNQ Internet of Things terminal; the indicator light module comprises a power indicator light 1-way, a network connection state indicator light 2-way, an AI DPU module state indicator light 1-way, and a sensor indicator light 1-way.

[0152] The power indicator light is 1-way, used for indicating whether the device is powered on or not.

[0153] The network connection state indicator light is 2-way, including: the first way is used for indicating whether the wireless network link is connected or not; and the second way is used for indicating the wireless network data reporting state.

[0154] The AI DPU module state indicator light is 1-way, used for indicating whether the AI DPU module is working, hibernating or malfunctioning.

[0155] The sensor indicator light is 1-way, used for indicating the alarm state of the sensor value.

[0156] Optionally, the DPU control module at the PS end of the ZYNQ processor includes: a UART interface and a firmware upgrade interface.

[0157] The DPU control module at the PS end of the ZYNQ processor is used for the ZYNQ processor PS end to allocate the UART interface to command control the AI DPU module, the ZYNQ processor sends a control or request instruction, and the AI DPU module sends the execution result to the ZYNQ processor; the firmware upgrade interface allocated by the ZYNQ processor PS end is used for sending a firmware upgrade package to the AI DPU module, the AI DPU module decompresses the file and runs the firmware upgrade program to update the firmware version.

[0158] Optionally, the DPU video module at the PL end of the ZYNQ processor is located at the PL end of the ZYNQ processor, and includes: a VDMA IP core, a video format conversion IP core, a VO-BUFFER IP core and a DPU video interface IP core; wherein,

[0159] The VDMA IP core is an official IP of XILINX, uses an AXI HP port to perform the reading and writing of the image by the PL end DDR and the streaming data transmission of the image video by the PS end to the PL end, and uses an AXI GP port to control the register of the VDMA IP core;

[0160] The video format conversion IP core is a self-defined IP, and the reading and writing of the video streaming data on the DDR are completed by connecting the VDMA IP core; the video format conversion IP core obtains the image containing the video information buffered on the DDR frame by frame after the video decoding by the PS end, converts the image into the image of the video format required by the AI DPU module through a format conversion algorithm, and writes the image into the DDR;

[0161] The VO-BUFFER IP core and the DPU video interface IP core are self-defined IPs; the VO-BUFFER IP core completes DDR cache reading by connecting the VDMA IP core, encapsulates images according to the frame format required by the AI DPU module, obtains the encapsulated images, and transmits the encapsulated images to the DPU video interface IP; the DPU video interface IP core constructs a video data transmission interface according to the frame timing required by the AI DPU module, leads out electrical pins and is physically connected to the AI DPU module, and transmits the processed video data to the AI DPU module; wherein the electrical pin of the video transmission interface includes a data line pin and a clock line pin.

[0162] Optionally, the PS end and the PL end of the ZYNQ processor communicate through AXIBUS; wherein the AXIGP interface type in AXIBUS is used for the general-purpose peripheral interface of the PS end and the low-speed communication between the PL end and the PS end; the AXIHP interface in AXIBUS is used for the DDR interface of the PS end and the high-speed communication between the PL and the PS end, and the high-speed reading and writing of the PL end to the DDR are completed through AXIHP.

[0163] Optionally, the ZYNQ processor receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result, including:

[0164] The system mainboard of the ZYNQ Internet of Things terminal is powered on, the ZYNQ processor and the AI DPU module complete initialization, and the ZYNQ processor allocates memory space for a video transmission and format conversion thread and creates a local file; wherein the local file is saved on an SD card, including a running log and an AI video processing result folder;

[0165] The DPU control module of the PS end of the ZYNQ processor sends a request video processing command to the AI DPU module through UART, the ZYNQ processor listens to the reply of UART, the AI DPU module parses the command after receiving the command, and calls an internal program of the AI DPU module;

[0166] The camera interface module acquires a video stream according to an external camera, the ZYNQ processor decodes the video stream and buffers video frames to DDR through the PS end; the video format conversion IP core of the PL end of the ZYNQ processor converts the video frames into images in a format required by the AI DPU, and writes the images back to DDR; the VO-BUFFER encapsulates the images into a video stream in a format required by the AI DPU according to a timing sequence, and sends the video stream to the AI DPU module through the DPU video interface;

[0167] The video interface unit of the AI DPU module receives a video, identifies a to-be-detected target in the video through an artificial intelligence algorithm, and obtains an AI video recognition result; the AI video recognition result is sent to the ZYNQ processor through a UART interface; wherein the AI video recognition result includes a feature map and text information.

[0168] The ZYNQ processor analyzes and filters the received AI video recognition result, stores the AI video recognition result in an AI video processing result folder, records a DPU operation log, and packs the AI video recognition result and other sensor information to be reported, and reports the information through a network communication module.

[0169] In the embodiment of the application, the ZYNQ processor is used as a main processor, and the DPU video module at the PL end of the ZYNQ processor and the DPU control module at the PS end of the ZYNQ processor are connected with the AI DPU module; the PS end of the ZYNQ processor collects camera video and performs decoding operation to obtain image data; the DPU video module at the PL end of the ZYNQ processor performs format conversion and packaging on the image data to obtain video data; the DPU video module at the PL end of the ZYNQ processor transmits the video data to the AI DPU module; the AI DPU module receives the video data, performs artificial intelligence identification on the video data to obtain an AI video recognition result; the AI DPU module sends the AI video recognition result to the ZYNQ processor; the ZYNQ processor receives the AI video recognition result, comprehensively analyzes the AI video recognition result, and obtains an analysis result.

[0170] In the embodiment of the application, the ZYNQ Internet of Things terminal device is quickly integrated with the AI video processing function, so that the performance and algorithm of the AI DPU module are independent of the development of the function of the ZYNQ processor end; the ZYNQ processor is used as a bridge for video transmission between different types of camera and different types of AI DPU module, but does not participate in the process of deep neural network inference, logic resources and power consumption are controllable; the ZYNQ processor can comprehensively process the processing result of the AI DPU module and sensor data, can enrich data analysis and data fusion application development, and has good function expansion; in the embodiment of the application, the hardware and software collaborative design characteristics of the ZYNQ heterogeneous processor are used to divide resources and design modules for the processor end and the programmable logic end of the ZYNQ, enhance interface diversity, and be compatible with multiple interfaces including camera, AI DPU module, sensor and wireless transmission module; the characteristics of AXI bus and FPGA are fully utilized to improve data throughput and real-time computing capability. The ZYNQ Internet of Things terminal can be quickly integrated with the AI video analysis function by using the application, which is suitable for Internet of Things transmission and edge computing, and reduces the deployment time, cost and power consumption of software and hardware.

[0171] Figure 7 is a structural schematic diagram of a ZYNQ Internet of Things terminal device with edge AI video analysis function provided by an embodiment of the present application, as shown in Figure 7 The ZYNQ Internet of Things terminal device with edge AI video analysis function can include the ZYNQ Internet of Things terminal design system with edge AI video analysis function shown in Figure 6 Optionally, the ZYNQ Internet of Things terminal device with edge AI video analysis function 410 can include a first processor 2001.

[0172] Optionally, the ZYNQ Internet of Things terminal device with edge AI video analysis function 410 can further include a memory 2002 and a transceiver 2003.

[0173] The first processor 2001 can be connected with the memory 2002 and the transceiver 2003 through a communication bus.

[0174] The various constituent components of the ZYNQ Internet of Things terminal device with edge AI video analysis function 410 will be specifically introduced below: Figure 7

[0175] The first processor 2001 is the control center of the ZYNQ Internet of Things terminal device with edge AI video analysis function 410, and can be one processor or a collective term of multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), can be an application specific integrated circuit (ASIC), or be one or more integrated circuits configured to implement the embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0176] Optionally, the first processor 2001 can execute various functions of the ZYNQ Internet of Things terminal device with edge AI video analysis function 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0177] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, such as the CPU0 and CPU1 shown in Figure 7

[0178] ​​In a specific implementation, as an embodiment, the ZYNQ IoT terminal device 410 with edge AI video analysis function may also include multiple processors, such as Figure 7 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0179] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0180] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and accessed through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0181] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0182] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0183] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently and be coupled with the first processor 2001 through an interface circuit (not shown in the figure) of the ZYNQ Internet of Things terminal device 410 with edge AI video analysis function. Figure 7 The present embodiment is not limited in this regard.

[0184] It should be noted that, Figure 7 The structure of the ZYNQ Internet of Things terminal device 410 with edge AI video analysis function shown in the figure does not constitute a limitation on the router, and the actual knowledge structure identification device can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0185] In addition, the technical effects of the ZYNQ Internet of Things terminal device 410 with edge AI video analysis function can refer to the technical effects of the ZYNQ Internet of Things terminal design method with edge AI video analysis function described in the above method embodiments, which will not be repeated here.

[0186] It should be understood that the first processor 2001 in the present embodiment can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0187] It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0188] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0189] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0190] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0191] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0192] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0193] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, systems and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0194] In several embodiments provided by the present application, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, systems or units, which can be electrical, mechanical or other forms.

[0195] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0196] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0197] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0198] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A ZYNQ IoT terminal design method with edge AI video analysis function, characterized in that: The ZYNQ IoT terminal design with edge AI video analysis function is implemented by the system of the ZYNQ IoT terminal, including: a ZYNQ processor, a DPU video module on the PL side of the ZYNQ processor, a DPU control module on the PS side of the ZYNQ processor, and an AI DPU module; the method includes: S1, the ZYNQ processor as the main processor is connected to the AI ​​DPU module through the DPU video module on the PL side of the ZYNQ processor and the DPU control module on the PS side of the ZYNQ processor; The DPU control module on the PS side of the ZYNQ processor includes: a UART interface and a firmware upgrade interface; Among them, the DPU control module on the PS side of the ZYNQ processor is used to allocate a UART interface on the ZYNQ processor PS side to control the AIDPU module. The ZYNQ processor sends a control or request instruction, and the AI ​​DPU module sends the execution result to the ZYNQ processor. The firmware upgrade interface is allocated by the ZYNQ processor PS side to send the firmware upgrade package to the AI ​​DPU module. The AIDPU module decompresses the file and runs the firmware upgrade program to update the firmware version. S2. The PS side of the ZYNQ processor collects camera video and performs decoding operations to obtain image data; the DPU video module of the PL side of the ZYNQ processor performs format conversion and encapsulation on the image data to obtain video data; the DPU video module of the PL side of the ZYNQ processor transmits the video data to the AI ​​DPU module; The DPU video module of the PL side of the ZYNQ processor is located at the PL side of the ZYNQ processor, and includes: VDMA IP core, video format conversion IP core, VO-BUFFER IP core and DPU video interface IP core; The VDMA IP core is an official XILINX IP. It uses the AXI HP port to read and write images from the PL side DDR and stream data from the PS to the PL side. The AXI GP port is used to control the registers of the VDMA IP core. The video format conversion IP core is a custom IP that reads and writes video stream data on DDR by connecting to the VDMA IP core. The video format conversion IP core obtains the image containing video information cached on DDR after the PS end performs video decoding frame by frame, converts the image into the video format required by the AI ​​DPU module through the format conversion algorithm, and writes it to DDR. The VO-BUFFER IP core and DPU video interface IP core are custom IPs; the VO-BUFFER IP core completes DDR cache reading by connecting to the VDMA IP core, encapsulates the image according to the frame format required by the AI ​​DPU module, and transmits the encapsulated image to the DPU video interface IP; the DPU video interface IP core constructs a video data transmission interface according to the frame timing required by the AI ​​DPU module, leads to electrical pins that are physically connected to the AI ​​DPU module, and transmits the processed video data to the AI ​​DPU module; wherein the electrical pins of the video data transmission interface include data line pins and clock line pins; S3. The AI ​​DPU module receives the video data and obtains an AI video recognition result by performing artificial intelligence recognition on the video data; the AI ​​DPU module sends the AI ​​video recognition result to the ZYNQ processor; S4. The ZYNQ processor receives the AI ​​video recognition result, performs a comprehensive analysis on the AI ​​video recognition result, and obtains an analysis result; The ZYNQ processor of S4 receives the AI ​​video recognition result, performs a comprehensive analysis on the AI ​​video recognition result, and obtains an analysis result, including: S41: The system motherboard of the ZYNQ IoT terminal is powered on, the ZYNQ processor and the AI ​​DPU module are initialized, and the ZYNQ processor allocates memory space for the video transmission and format conversion threads and creates local files; wherein the local files are saved on the SD card, including the operation log and the AI ​​video processing result folder; S42: The ZYNQ processor parses the received AI video recognition results and performs data screening processing, stores the AI ​​video recognition results in the AI ​​video processing result folder, records the DPU operation log, and packages the AI ​​video recognition results and other sensor information to be reported, and reports the information through the network communication module; The system of the ZYNQ IoT terminal further includes: a memory module, a camera interface module, a sensor control module, an indicator light module, a network communication module, a wireless transmission module and a voice processing module; The sensor control module includes sensors in the processor chip, sensors on the system mainboard, and interfaces for external sensors; Wherein, the memory module includes running memory DDR, FLASH and SD card; The indicator light module is used to display and alarm the status of the ZYNQ IoT terminal. The indicator light module includes: 1 power indicator light, 2 network connection status indicator lights, 1 AI DPU module status indicator light, and 1 sensor indicator light. Among them, the power indicator light 1 is used to indicate whether the device is powered on; The two network connection status indicator lights include: a first indicator light for indicating whether the wireless network link remains connected; a second indicator light for indicating the wireless network data reporting status; Among them, the AI ​​DPU module status indicator light 1 is used to indicate whether the AI ​​DPU module is working, dormant or faulty; Among them, the sensor indicator light 1 is used to indicate the alarm status of the sensor value; Among them, the PS side and PL side of the ZYNQ processor communicate through AXIBUS; among them, the AXIGP interface type in AXIBUS is used for the general peripheral interface of the PS side and the low-speed communication between the PL side and the PS side; the AXIHP interface in AXIBUS is used for the DDR interface of the PS side and the high-speed communication between the PL side and the PS side, and the high-speed reading and writing of the DDR by the PL side is completed through AXIHP.

2. A ZYNQ IoT terminal design system with edge AI video analysis function, wherein the ZYNQ IoT terminal design system with edge AI video analysis function is used to implement the ZYNQ IoT terminal design method with edge AI video analysis function as claimed in claim 1, characterized in that: The system includes: a ZYNQ processor, a DPU video module on the PL side of the ZYNQ processor, a DPU control module on the PS side of the ZYNQ processor, and an AI DPU module; The ZYNQ processor is used as a main processor to connect to the AI ​​DPU module through the DPU video module on the PL side of the ZYNQ processor and the DPU control module on the PS side of the ZYNQ processor; the PS side of the ZYNQ processor collects camera video, performs decoding operations, and obtains image data; the ZYNQ processor receives the AI ​​video recognition result, performs comprehensive analysis on the AI ​​video recognition result, and obtains an analysis result; The DPU video module on the PL side of the ZYNQ processor is used to convert and encapsulate the image data to obtain video data; the DPU video module on the PL side of the ZYNQ processor is used to transmit the video data to the AIDPU module; The AI ​​DPU module is used to receive the video data and obtain an AI video recognition result by performing artificial intelligence recognition on the video data; the AI ​​DPU module is used to send the AI ​​video recognition result to the ZYNQ processor; The DPU control module on the PS side of the ZYNQ processor is used to connect the DPU video module on the PL side of the ZYNQ processor and the DPU control module on the PS side of the ZYNQ processor with the AI ​​DPU module.

3. A ZYNQ IoT terminal device with edge AI video analysis function, characterized in that: The ZYNQ IoT terminal devices with edge AI video analysis capabilities include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to claim 1 is implemented.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to claim 1.

Images