Artificial intelligence edge computing platform and method based on localized edge computing platform
Through the domestic edge computing platform, combined with domestic FPGA chips and artificial intelligence edge computing modules, the problems of high costs, unstable supply chain and high power consumption in the existing technology are solved, and efficient neural network inference and video stream processing are achieved. , supports neural networks with magnitude 100 megabytes of parameters, improving system stability and hardware compatibility.
Patent Information
- Application Number
- CN202411350986.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-27
AI Technical Summary
The existing artificial intelligence edge computing platform relies on imported chips, which has high costs, unstable supply chain, mutually exclusive power consumption and computing power, making it difficult to deploy neural networks with a parameter volume of 100 megabytes or above, and data transmission delay and privacy There are security problems.
Adopt a domestic edge computing platform, including domestic FPGA chips and artificial intelligence edge computing modules, combined with the domestic open source edge computing end-side operating system, uses EMC as a storage medium, has rich external communication interfaces, and uses image output modules and data exchange. and system control module, OSD information overlay module and neural network working module to realize efficient video stream processing and data communication.
It realizes artificial intelligence edge computing network inference capability up to 20 TOPS, supports neural network inference with a scale of 100 megabytes of parameters, has rich video and communication interfaces, improves the hardware compatibility and stability of the system, reduces power consumption, and is suitable for Construction of a domestic hardware ecosystem.
Smart Images
Figure CN119201462A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision and edge computing technology, and specifically to an artificial intelligence edge computing platform and method based on a domestically produced edge computing platform. Background Art
[0002] With the rapid development of artificial intelligence technology, artificial intelligence edge computing platforms have been widely used in security monitoring, intelligent transportation, unmanned driving and other fields. However, most of the existing artificial intelligence edge computing platforms rely on imported chips and computing modules, and have problems such as high cost and unstable supply chain. In addition, the existing artificial intelligence edge computing platforms have the problem of power consumption and computing power being mutually exclusive. Low power consumption means low computing power, which is not convenient for deploying neural networks with hundreds of megabytes and above; high computing power means high power consumption, and excessive power consumption on the edge computing side is not friendly to the system's long-term battery operation. Therefore, it is usually necessary to rely on cloud computing, which has problems such as data transmission delay and privacy security. Therefore, it is of great practical significance to develop an artificial intelligence edge computing platform based on a purely domestically produced edge computing platform. Summary of the invention
[0003] In view of the shortcomings of the prior art, the present invention discloses an artificial intelligence edge computing platform and method based on a domestically produced edge computing platform to solve the problems raised in the above background technology.
[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an artificial intelligence edge computing platform based on a domestic edge computing platform, including a domestic FPGA chip and a domestic artificial intelligence edge computing module to form a core computing architecture, using a domestic open source edge computing terminal side operating system as an embedded underlying system, using emmc as a system installation and storage medium, including several external communication interfaces;
[0005] The communication interface includes 4 serial ports, 2 CAN ports for command transmission and reception, two Gigabit network ports for network format video input / output, 1 bidirectional 3G-SDI analog interface for analog video input / output, and 1 dual-channel fan interface for external fan to cool the system;
[0006] It also includes an image output module, a data exchange and system control module, an OSD information overlay module, and a neural network working module;
[0007] The image output module is used to process the video stream data collected by the external camera and then output it into a network format video;
[0008] The data exchange and system control module is used to realize the communication between the terminal data and to process and forward the received data;
[0009] The OSD information overlay module is used to overlay system status, tracking and identification related information into the video;
[0010] The neural network working module is used to perform neural network inference and prediction on the input image to obtain a detection result.
[0011] Preferably, the image video stream is received through the FPGA chip, and data interaction is performed with the artificial intelligence edge computing module through the PCIE2.0 protocol to realize bidirectional reading and writing of the video stream.
[0012] Preferably, the emmc is expanded to a maximum of 256GB, and its onboard emmc package is compatible with various capacity emmc particles that conform to the BGA153 package. The range of particles selected during actual mounting is 8GB-256G.
[0013] Preferably, the serial port is compatible with three communication serial ports: RS422, RS485 and RS232, which is convenient for switching according to user needs.
[0014] Preferably, the reserved external fan interface provides 1A current and supports external fans with a maximum speed of 20,000 rpm, making it easy to select fans of different specifications according to user needs.
[0015] The present invention also provides a method for using an artificial intelligence edge computing platform based on a domestically produced edge computing platform, comprising the following steps:
[0016] S1. Start the hardware in this platform. First, load the Atlas kernel. Then the system loads the burned operating system image from the EMMC storage chip. Then, start the Taishan CPU core and initialize the serial port and network port peripheral hardware.
[0017] S2. After initialization, the system loads the docker container and starts the SSH user login service inside the container, which is convenient for users to log in and manage remotely through the network; finally, the self-start script is executed to initialize the system environment variables and start the system services; after executing each instruction, the system determines whether each module is executed successfully; if successful, the "self-test passed" mark is sent, indicating that the system initialization is successful; if failed, the startup program is re-executed until all modules of the system are initialized successfully;
[0018] S3. After initialization is completed, the system uses the camera in the image output module to encode and decode the collected video stream data through the FPGA, and then sends it to the neural network working module in the AI module for tracking and recognition. The recognition results and control instructions are then sent to the OSD information overlay module and the data exchange and system control module to display the tracking results and control the external tracking device.
[0019] Preferably, the image output module specifically comprises the following steps:
[0020] Video stream data is collected from an external camera and transmitted to the AI module via FPGA. The AI module parses, encodes, decodes, reads information, determines the resolution and adjusts the image size. It is then sent to the neural network for processing, and the result and system status are superimposed with OSD information. The result is then converted back to the original size, encoded in H.264 and output through the Gigabit Ethernet port.
[0021] Preferably, the specific steps of the data exchange and system control module include:
[0022] Communication is achieved through the onboard serial port and CAN port. When the serial port module is working, it receives external terminal data by first reading the frame header to determine the address and other information, waiting for the complete data packet and judging whether the length is correct. If it is correct, it verifies the data. If it is correct, it determines the data type according to the source address and forwards it accordingly. Among them, the feedback information of the visible light and infrared terminals is forwarded to the data processing module or the display and control terminal, and the control command of the display and control terminal is forwarded to the infrared and white light terminals.
[0023] Preferably, the OSD information overlay module specifically comprises the following steps:
[0024] After entering the module, the information is first read in parallel at the overlay position in the video frame, and then the information to be loaded is read. The system status information is directly overlaid. For tracking and identification information, it is necessary to first determine whether the function is turned on. If it is turned on, it is overlaid. If it is turned off, it is read in a loop until it is turned on and then overlaid.
[0025] Preferably, the specific steps of the neural network working module include:
[0026] After the system enters the recognition mode, the original image is input into the neural network for preliminary processing, including scaling and format conversion, followed by feature extraction, processing and fusion, followed by bounding box prediction and category prediction. The prediction results are first threshold filtered and then non-maximum suppression is performed, and finally the detection results are output to post-processing.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] 1. The present invention has an artificial intelligence edge computing network reasoning capability of up to 20TOPS, and has a wealth of video and communication interfaces for users to use. It can simultaneously access up to 6 video streams in different formats, including 3 cameralink base video streams, 1 bidirectional 3G-SDI video stream and 2 bidirectional gigabit network video streams, 4 high-speed serial ports and 2 CAN ports, and 2 reserved fan power supply interfaces, which can realize neural network reasoning with hundreds of megabytes of parameters.
[0029] 2. In the present invention, the Atlas kernel is used and the operating system image is loaded from the EMMC storage chip, combined with the use of the Taishan CPU core, to make full use of domestic hardware resources. This combination helps to improve the hardware compatibility of the system and ensure stable startup on the domestic edge computing platform; the serial port and network port peripheral hardware are initialized so that the system can communicate well with external devices. This helps to build a complete domestic hardware ecosystem, facilitate the connection of various domestic sensors, actuators and other equipment, and realize data collection and accurate transmission of control instructions. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0031] In the attached picture:
[0032] Figure 1 It is a schematic diagram of the interface of the present invention;
[0033] Figure 2 It is the overall architecture diagram of the present invention;
[0034] Figure 3 It is a startup initialization flow chart of a specific implementation scheme of the present invention;
[0035] Figure 4 is a workflow diagram of an image output function of a specific embodiment of the present invention;
[0036] Figure 5 It is a workflow diagram of data exchange and system control functions of a specific implementation scheme of the present invention;
[0037] Figure 6 It is a flowchart of OSD information overlaying process of a specific implementation scheme of the present invention;
[0038] Figure 7 It is a neural network workflow diagram of a specific implementation scheme of the present invention. DETAILED DESCRIPTION
[0039] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0040] Example: Figure 1As shown in the figure, the artificial intelligence edge computing platform based on the domestic edge computing platform uses domestic FPGA chips and domestic artificial intelligence edge computing modules to form the core computing architecture of the system. It adopts 100% pure domestic design, and the usage rate of imported devices is 0; it has up to 20Tops edge computing power, and can deploy large-scale artificial intelligence networks on the offline side to achieve higher-precision target recognition tasks; it receives up to three cameralink camera video streams through the FPGA chip, and interacts with the artificial intelligence edge computing module through the PCIE2.0 protocol to realize two-way reading and writing of video streams; it uses the domestic open source edge computing end-side operating system as the embedded underlying system, and uses the eMMC that can be expanded to 256GB as the system installation and storage medium, which has Rich external communication interfaces, including up to 4 serial ports, 2 CAN ports for command sending and receiving, two Gigabit network ports for network format video input / output, 1 bidirectional 3G-SDI analog interface for analog video input / output, and 1 dual-channel fan interface for external fan cooling of the system; its reserved fan interface can provide up to 1A current and supports external fans with a maximum speed of 20,000 rpm, which is convenient for selecting fans of different specifications according to user needs; its serial port is compatible with three common communication serial ports: RS422, RS485 and RS232, which is convenient for switching according to user needs; its onboard emmc package is compatible with various capacity emmc particles that conform to the BGA153 package. When actually mounting, the minimum particle is 8GB and the maximum particle is 256G.
[0041] Among them, J2 and J4 are cameralink video input interfaces, J2 is a cameralink medium interface with a dual tap design, which can be expanded to two independent cameralinkbase interfaces, J4 is an independent cameralinkbase interface, J5 is a USB-typeC interface for debugging, J6 is a system 12V power supply interface, J7 is a 4-way serial port, J8 is a 2-way network port, J10 is a 2-way CAN port, and X1 is a 1-way bidirectional 3G-SDI analog interface.
[0042] like Figure 2As shown, the cameralink, 3G-SDI and other image video streams are connected to the system through interfaces such as J2, J4, and X1. The FPGA chip in the system reads the serial video information of the cameralink and the analog video information of the 3G-SDI, converts the video into parallel information including line and field synchronization signals and sampling clock signals, and eliminates the blanking time in the video line and field signals through the FIFO cache mechanism, and stores the image information frame by frame in the video into the onboard DDR3 particles through the built-in VDMA channel of the FPGA. The FPGA and the artificial intelligence edge computing module are connected through the PCIE2.0 protocol. When 1 frame of data is stored in the DDR3, the FPGA sends a handshake signal to the artificial intelligence edge computing module, prompting that the current data can be read. After receiving the handshake signal, the artificial intelligence edge computing module successfully shakes hands and starts to continuously read image data through the PCIE channel. Since the blanking wait time has been eliminated in the FPGA, the artificial intelligence edge computing module can obtain continuous image data uninterruptedly at this time.
[0043] The artificial intelligence edge computing platform in the present invention also includes an image output module, a data exchange and system control module, an OSD information overlay module and a neural network working module;
[0044] The image output module is used to process the video stream data collected by the external camera and then output it into a network format video;
[0045] The data exchange and system control module is used to realize the communication between the terminal data and to process and forward the received data;
[0046] The OSD information overlay module is used to overlay system status, tracking and identification related information into the video;
[0047] The neural network working module is used to perform neural network inference and prediction on the input image to obtain a detection result.
[0048] The present invention also provides a method for using an artificial intelligence edge computing platform based on a domestically produced edge computing platform, such as Figure 3 As shown, the following steps are included:
[0049] First is the execution process of the hardware startup program of this platform. When the computing platform is turned on, the system executes the following commands in sequence: load the Atlas kernel, then the system loads the burned operating system image from the EMMC storage chip, then starts the Taishan CPU core, and initializes peripheral hardware such as serial ports and network ports.
[0050] After initialization, the system loads the docker container and starts the SSH user login service inside the container, which facilitates users to log in and manage remotely through the network. Finally, the self-start script is executed to initialize the system environment variables and start the system services. After executing each instruction, the system determines whether the above modules are executed successfully. If successful, the "self-test passed" mark is sent, indicating that the system is initialized successfully; if it fails, the startup program is re-executed until all modules of the system are initialized successfully.
[0051] After initialization is completed, the system will encode and decode the video stream data collected by the camera through FPGA, and then send it to the AI module for tracking and recognition. The recognition results and control instructions are then sent to the OSD information overlay module and the serial port module to display the tracking results and control the external tracking device.
[0052] The entire process involves an image output module, a data exchange and system control module, an OSD information overlay module, and a neural network working module.
[0053] The following is the specific workflow of each module:
[0054] like Figure 4 The figure shows the workflow of the image output function module. First, the video stream data collected by the external camera is input into the FPGA through the cameralink. The FPGA transmits the collected video stream data to the AI module through the PCIE2.0 interface. After receiving the video stream data, the module will first parse and re-encode the video stream, then decode the video and read the video information. Then, it will determine whether it meets the video size required for reasoning based on the resolution of the video. If so, the image will be homogenized so that the data can better adapt to the deep learning model structure for subsequent image analysis and processing. If it does not meet the video size required for reasoning, the image will be converted to a video size suitable for model reasoning through image scaling, and then the image will be homogenized. After the image homogenization is completed, the video data can be sent to the neural network for video image processing. After the processing is completed, the processed results and the system operation status are superimposed with OSD information so that the tracking target position and system status information can be displayed on the video later. Then the video is converted to the original size when it was input, and finally the video data is encoded in H.264 and sent to the Gigabit network port for network format video output.
[0055] like Figure 5The figure shows the functional workflow of the data exchange and system control module. The communication between the terminal data is realized through the onboard 4-way high-speed serial port and two-way CAN port. At the beginning of the serial port module, the system will enter the serial port function module to prepare for data reception and transmission. When the module receives data from the external terminal, it will first read the format of the serial port data in the frame header to determine the source terminal and destination terminal address of the data communication process. Then the system will wait for the data information of the serial port to receive the complete data packet. After receiving all the data and assembling the data, the system will determine whether the length of the data packet meets the length specified in the data format. If the data length is correct, it will start to read the serial port data information. If not, it will discard the data packet and wait for the data information sent by the serial port again until the received data length meets the length of the data format. After that, data verification will be performed to detect whether there is an error in the data transmission process. If the transmission is incorrect, the transmission will be re-initiated and the serial port data information will be read. If the transmission is correct, the data valid bit will be read. Then, the data type is determined based on the source address of the data format, whether it is feedback information from a visible light or infrared terminal or a control command from a display and control terminal. If it is feedback information from a visible light or infrared terminal, the feedback information is forwarded to the data processing module based on the destination address for subsequent video data processing; or forwarded to the display and control terminal, where the video tracking results and system status information are displayed. If it is a control command from a display and control terminal, the command is directly forwarded to the infrared or white light terminal, thereby controlling the zoom and magnification of infrared or white light cameras.
[0056] like Figure 6 The figure shows the operation flow chart of the OSD information overlay module, which is mainly responsible for overlaying the system status information, tracking-related information, and tracking and recognition results onto the video. First, the system enters the OSD information overlay module and executes three processes in parallel, reading the overlay positions of the system information, tracking information, and recognition information in the video frame respectively. After the reading is completed, the system status, tracking, and recognition information to be loaded will continue to be read. Finally, the system status information will be directly output and overlaid onto the video, while the tracking information and recognition information will first determine whether the corresponding tracking and recognition functions are turned on. If turned on, the tracking status, tracking information, and output recognition box and recognition results will be output and overlaid onto the video. If it is turned off, the system will return to the steps of reading the tracking information and reading the recognition information overlay position and repeat the cycle until the tracking and recognition functions are turned on before the next step is taken.
[0057] like Figure 7The figure shows the workflow of the neural network working module in the AI module for neural network reasoning prediction. When the system enters the recognition mode, the original image will be input into the neural network, and then the network will perform preliminary processing such as scaling and format conversion on the image so that the image meets the size, format and other requirements required for reasoning. After the preliminary processing is completed, feature extraction, feature processing and feature fusion will be performed, and then the results will be sent to the neural network for prediction. The neural network will perform bounding box prediction and category prediction respectively to predict the position of the selected object in the image and the category of the object. After the neural network predicts the image, a series of bounding box prediction results will be obtained. After that, the network will first apply threshold filtering to quickly filter out high-confidence predictions, reduce the number of bounding boxes for subsequent processing, thereby improving the overall processing efficiency, and then perform non-maximum suppression (NMS) processing to further reduce these overlapping bounding boxes, retaining only the best prediction results, and finally output the detection results to post-processing.
[0058] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An artificial intelligence edge computing platform based on a domestically produced edge computing platform, characterized by: The core computing architecture consists of domestic FPGA chips and domestic artificial intelligence edge computing modules, with a domestic open source edge computing terminal operating system as the embedded underlying system, and eMMC as the system installation and storage medium, including several external communication interfaces; The communication interface includes 4 serial ports, 2 CAN ports for command transmission and reception, two Gigabit network ports for network format video input / output, 1 bidirectional 3G-SDI analog interface for analog video input / output, and 1 dual-channel fan interface for external fan to cool the system; It also includes an image output module, a data exchange and system control module, an OSD information overlay module, and a neural network working module; The image output module is used to process the video stream data collected by the external camera and then output it into a network format video; The data exchange and system control module is used to realize the communication between the terminal data and to process and forward the received data; The OSD information overlay module is used to overlay system status, tracking and identification related information into the video; The neural network working module is used to perform neural network inference and prediction on the input image to obtain a detection result.
2. The artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 1 is characterized in that: The image video stream is received through the FPGA chip, and data is interacted with the artificial intelligence edge computing module through the PCIE2.0 protocol to achieve bidirectional reading and writing of the video stream.
3. The artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 1 is characterized in that: The emmc can be expanded up to 256GB, and its onboard emmc package is compatible with various capacity emmc particles that conform to the BGA153 package. The actual mounting range is 8GB-256G particles.
4. The artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 1 is characterized in that: The serial port is compatible with three communication serial ports: RS422, RS485 and RS232.
5. The artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 1 is characterized in that: The reserved external fan interface provides 1A current and supports an external fan with a maximum speed of 20,000 rpm.
6. A method for using an artificial intelligence edge computing platform based on a domestically produced edge computing platform, characterized in that: The following steps are involved: S1. Start the hardware in this platform. First, load the Atlas kernel. Then the system loads the burned operating system image from the EMMC storage chip. Then, start the Taishan CPU core and initialize the serial port and network port peripheral hardware. S2. After initialization, the system loads the docker container, starts the SSH user login service inside the container, and finally executes the self-start script to initialize the system environment variables and start the system services. After executing each instruction, the system determines whether each module is executed successfully. If successful, the "self-test passed" mark is sent, indicating that the system initialization is successful. If it fails, the startup program is re-executed until all modules of the system are initialized successfully. S3. After initialization is completed, the system uses the camera in the image output module to encode and decode the collected video stream data through the FPGA, and then sends it to the neural network working module in the AI module for tracking and recognition. The recognition results and control instructions are then sent to the OSD information overlay module and the data exchange and system control module to display the tracking results and control the external tracking device.
7. The method for using the artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 6 is characterized in that: The specific steps of the image output module include: Video stream data is collected from an external camera and transmitted to the AI module via FPGA. The AI module parses, encodes, decodes, reads information, determines the resolution and adjusts the image size. It is then sent to the neural network for processing, and the result and system status are superimposed with OSD information. The result is then converted back to the original size, encoded in H.264 and output through the Gigabit Ethernet port.
8. The method for using the artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 1 is characterized in that: The specific steps of data exchange and system control module include: Communication is achieved through the onboard serial port and CAN port. When the serial port module is working, it receives external terminal data by first reading the frame header to determine the address and other information, waiting for the complete data packet and judging whether the length is correct. If it is correct, it verifies the data. If it is correct, it determines the data type according to the source address and forwards it accordingly. Among them, the feedback information of the visible light and infrared terminals is forwarded to the data processing module or the display and control terminal, and the control command of the display and control terminal is forwarded to the infrared and white light terminals.
9. The method for using the artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 1, characterized in that: The specific steps of the OSD information overlay module include: After entering the module, the information is first read in parallel at the overlay position in the video frame, and then the information to be loaded is read. The system status information is directly overlaid. For tracking and identification information, it is necessary to first determine whether the function is turned on. If it is turned on, it is overlaid. If it is turned off, it is read in a loop until it is turned on and then overlaid.
10. The method for using the artificial intelligence edge computing platform based on the domestic edge computing platform according to claim 1, characterized in that: The specific steps of the neural network working module include: After the system enters the recognition mode, the original image is input into the neural network for preliminary processing, including scaling and format conversion, followed by feature extraction, processing and fusion, followed by bounding box prediction and category prediction. The prediction results are first threshold filtered and then non-maximum suppression is performed, and finally the detection results are output to post-processing.