Multi-modal heterogeneous computing processing system

Abstract

The invention discloses a multi-mode heterogeneous computing processing system, and belongs to the technical field of embedded systems and high-performance computing. According to the system, a Ginseng 310P processor and a Xilinx Zynq UltraScale + FPGA form a heterogeneous computing core, an LPDDR4x memory, an eMMC memory, various high-speed interfaces and a power management module are integrated, and the system is connected with an external function module through a 688 PIN BGA connector. The system supports multiple communication protocols such as SERDES high-speed serial communication, multi-channel gigabit Ethernet, USB 3.1, SATA, SDI video interfaces and the like, gives full play to the AI reasoning capability of the Gigabit 310P and the hardware programmability of the FPGA, has strong data processing capability and flexible expansibility, and is suitable for application scenes such as video coding and decoding, artificial intelligence reasoning, industrial control and the like.

Description

Technical Field

[0001] This invention belongs to the field of embedded systems and high-performance computing technology, and in particular relates to a multimodal heterogeneous computing processing system. Background Technology

[0002] With the rapid development of artificial intelligence, the Internet of Things and Industry 4.0 technologies, modern embedded systems have put forward higher requirements for processing power, energy efficiency and interface richness. Traditional single-architecture processing systems have the following shortcomings: (1) Limited processing power: a single processor is difficult to meet the needs of control tasks, AI inference and real-time data processing at the same time; (2) Low energy efficiency: general-purpose processors have relatively low energy efficiency when processing specific tasks; (3) Poor interface scalability: fixed architecture is difficult to adapt to the diverse peripheral interface requirements; (4) Insufficient system flexibility: it is difficult to dynamically adjust computing resources according to application scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal heterogeneous computing processing system to solve the problems of limited processing power, low energy efficiency and poor interface scalability in the prior art.

[0004] The first aspect of this invention discloses a multimodal heterogeneous computing processing system, the system comprising: An embedded processor; as the main processing unit, the embedded processor is used to perform system control, task scheduling, and AI inference calculations; wherein, the embedded processor adopts the Ascentage 310P processor; Field-programmable gate arrays (FPGAs) serve as coprocessor units, enabling hardware acceleration and interface expansion. These FPGAs utilize a combination of Xilinx Zynq UltraScale and FPGA. Storage modules; including LPDDR4x memory, eMMC memory, and SPI NOR Flash; High-speed interface module; including SERDES interface, PCIe interface, and USB 3.1 interface; Network communication module; including multiple Gigabit Ethernet interfaces; Expansion interface module; connects to external functional modules via a 688-pin BGA connector; Power management module; provides stable power supply to all components of the system.

[0005] Preferably, the embedded processor is connected to the field-programmable gate array (FPGA) via a high-speed bus, and the two form a master-slave collaborative heterogeneous computing architecture, wherein the embedded processor runs the operating system and application programs, and the FPGA implements hardware acceleration and interface expansion.

[0006] Preferably, the storage module includes 12 sets of LPDDR4x memory, 64GB eMMC memory, and SPI NOR Flash, which are connected to the embedded processor and the field-programmable gate array via DDRC, eMMC, and SFC interfaces, respectively.

[0007] Preferably, the high-speed interface module includes: Four SERDES interfaces: M2.0-M2.3, M2.4-M2.5, M3, and M4. PCIe to USB 3.1 controller; provides two USB 3.1 ports; SATA interface; connects to a field-programmable gate array via SERDES M4.

[0008] Preferably, the SERDES M2.4-M2.5 interface connects to the video codec card, supports SDI video input and output, and the embedded processor performs video encoding and decoding processing. The field-programmable gate array is used to perform video data scheduling.

[0009] Preferably, the network communication module includes three Gigabit Ethernet interfaces, each implemented through an RTL8521 PHY chip, one of which is a management port, with network protocol processing handled by an embedded processor.

[0010] Preferably, the system further includes: an RTC real-time clock module, an INA226 power monitoring chip, an LM75 temperature sensor, and peripheral devices connected via IIC0 and IIC2 buses.

[0011] Preferably, the system further includes: a UART0 debug serial port, a UART1 RS485 interface, a UART2 RS232 interface, and a PWM-controlled cooling fan.

[0012] Preferably, the power management module includes: a PSIP power input subsystem, a VRD+DRMOS voltage regulator, and a PMU6421 power management unit.

[0013] A second aspect of this invention discloses a data processing method, which is implemented using a multimodal heterogeneous computing processing system as described in the first aspect of this invention. The method includes: Receive external data via a 688-pin BGA connector; Embedded processors perform system control, task scheduling, and AI inference computation; Field-programmable gate arrays (FPGAs) are used for hardware-accelerated processing and interface data exchange. Dynamic power consumption control of the system is achieved through a power management module.

[0014] The beneficial technical effects brought about by this invention include: (1) strong processing power: the 310P provides powerful AI inference capabilities, and the FPGA provides flexible hardware acceleration. The two work together to improve the overall performance of the system; (2) high energy efficiency: the heterogeneous architecture allocates appropriate computing resources according to the characteristics of the task to improve the energy efficiency ratio; (3) rich interfaces: it supports a variety of communication protocols and peripheral interfaces to adapt to different application scenarios; (4) flexible expansion: it supports functional expansion through the 688PIN BGA connector; (5) precise power consumption control: multi-level power management and temperature monitoring ensure stable operation of the system. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a multimodal heterogeneous computing processing system. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The first aspect of this invention discloses a multimodal heterogeneous computing processing system, the system comprising: An embedded processor; as the main processing unit, the embedded processor is used to perform system control, task scheduling, and AI inference calculations; wherein, the embedded processor adopts the Ascentage 310P processor; Field-programmable gate arrays (FPGAs) serve as coprocessor units, enabling hardware acceleration and interface expansion. These FPGAs utilize a combination of Xilinx Zynq UltraScale and FPGA. Storage modules; including LPDDR4x memory, eMMC memory, and SPI NOR Flash; High-speed interface module; including SERDES interface, PCIe interface, and USB 3.1 interface; Network communication module; including multiple Gigabit Ethernet interfaces; Expansion interface module; connects to external functional modules via a 688-pin BGA connector; Power management module; provides stable power supply to all components of the system.

[0019] Preferably, the embedded processor is connected to the field-programmable gate array (FPGA) via a high-speed bus, and the two form a master-slave collaborative heterogeneous computing architecture, wherein the embedded processor runs the operating system and application programs, and the FPGA implements hardware acceleration and interface expansion.

[0020] Preferably, the storage module includes 12 sets of LPDDR4x memory, 64GB eMMC memory, and SPI NOR Flash, which are connected to the embedded processor and the field-programmable gate array via DDRC, eMMC, and SFC interfaces, respectively.

[0021] Preferably, the high-speed interface module includes: Four SERDES interfaces: M2.0-M2.3, M2.4-M2.5, M3, and M4. PCIe to USB 3.1 controller; provides two USB 3.1 ports; SATA interface; connects to a field-programmable gate array via SERDES M4.

[0022] Preferably, the SERDES M2.4-M2.5 interface connects to the video codec card, supports SDI video input and output, and the embedded processor performs video encoding and decoding processing. The field-programmable gate array is used to perform video data scheduling.

[0023] Preferably, the network communication module includes three Gigabit Ethernet interfaces, each implemented through an RTL8521 PHY chip, one of which is a management port, with network protocol processing handled by an embedded processor.

[0024] Preferably, the system further includes: an RTC real-time clock module, an INA226 power monitoring chip, an LM75 temperature sensor, and peripheral devices connected via IIC0 and IIC2 buses.

[0025] Preferably, the system further includes: a UART0 debug serial port, a UART1 RS485 interface, a UART2 RS232 interface, and a PWM-controlled cooling fan.

[0026] Preferably, the power management module includes: a PSIP power input subsystem, a VRD+DRMOS voltage regulator, and a PMU6421 power management unit.

[0027] A second aspect of this invention discloses a data processing method, which is implemented using a multimodal heterogeneous computing processing system as described in the first aspect of this invention. The method includes: Receive external data via a 688-pin BGA connector; Embedded processors perform system control, task scheduling, and AI inference computation; Field-programmable gate arrays (FPGAs) are used for hardware-accelerated processing and interface data exchange. Dynamic power consumption control of the system is achieved through a power management module.

[0028] First implementation example Figure 1 As shown, the multimodal heterogeneous computing processing system of the present invention includes the following main components.

[0029] (1) Heterogeneous computing core The system uses the Ascentage 310P processor and Xilinx Zynq UltraScale+ FPGA to form a heterogeneous computing core: Ascentage 310P processor: As the main processing unit, it integrates the AI ​​computing core, is responsible for running the operating system, applications and AI inference tasks, and provides powerful neural network computing capabilities; Xilinx Zynq UltraScale+ FPGA: As a coprocessor unit, it utilizes programmable logic to implement hardware acceleration functions and expands various peripheral interfaces; Collaborative working mechanism: The two are connected via a high-speed bus. The 310P is responsible for task scheduling and data distribution, while the FPGA is responsible for hardware acceleration of specific algorithms and interface data processing.

[0030] (2) Storage module LPDDR4x memory: 12 sets of LPDDR4x memory are connected to the processor and FPGA via DDRC interface to provide high-speed data caching; eMMC storage: A 64GB eMMC storage device is connected via an eMMC interface and used for system program storage and large-capacity data storage; SPI NOR Flash: Connected via SFC interface, used to store boot configuration and firmware program; EEPROM: Used to store system configuration parameters.

[0031] (3) The high-speed interface module system provides multiple sets of high-speed serial interfaces: SERDES interfaces: including four SERDES interfaces: M2.0-M2.3, M2.4-M2.5, M3 and M4, supporting multiple high-speed communication protocols; PCIe interface: Provides two USB 3.1 ports via a PCIe to USB 3.1 controller; SATA interface: Connects via SERDES M4 interface, supporting high-speed storage devices; Video interface: SERDES M2.4-M2.5 interface connects to video codec cards and supports SDI video input and output.

[0032] (4) The network communication module system integrates three gigabit Ethernet interfaces: Management port: Used for system management and monitoring; Data ports: Two gigabit Ethernet ports are used for data communication, both implemented through the RTL8521 PHY chip; Control interface: Manages the network PHY chip through the MDIO0 interface.

[0033] (5) The expansion interface module system connects to external functional modules via a 688-pin BGA connector, which includes: SERDES M2.0-M2.3 interface signals; Power and reset control signals; Other auxiliary control signals.

[0034] (6) The power management module system adopts an advanced power management solution: PSIP power input subsystem: provides stable power input; VRD+DRMOS voltage regulator: provides precise voltage output for different components; PMU6421 Power Management Unit: Enables dynamic power consumption management of the system; INA226 Power Monitoring: Real-time monitoring of system power consumption; LM75 temperature sensor: monitors system temperature and works with PWM cooling fan to achieve temperature control.

[0035] (7) System configuration and debugging Startup Configuration: Set the system startup mode via DIP switches; Debugging interface: The UART0 debug serial port is used for system development and fault diagnosis; Industrial interfaces: UART1 supports RS485 communication, and UART2 supports RS232 communication; RTC Real-Time Clock: Provides a precise time base.

[0036] The second embodiment is a data processing method, which includes the following steps.

[0037] (1) System startup phase: After power-on, the PMU6421 power management unit sequentially activates each power domain; The Ascentage 310P processor loads its boot configuration from SPI NOR Flash; The system determines the startup mode based on the DIP switch settings; The Shengteng 310P initializes the FPGA and loads the hardware acceleration program.

[0038] (2) Data processing stage: External data is input to the system via a 688-pin BGA connector; The Ascentage 310P processor performs system control, task scheduling, and AI inference calculations; FPGAs are used for hardware-accelerated processing, such as image preprocessing and data encryption / decryption. The two exchange data and perform collaborative computing via a high-speed bus.

[0039] (3) Communication transmission stage: Network data is sent and received via a gigabit Ethernet interface, and the network protocol stack is handled by the SYM 310P. Video data is input and output via the SDI interface and encoded / decoded by the SYM 310P. High-speed data is transmitted through the SERDES interface, with protocol conversion and data scheduling handled by the FPGA.

[0040] (4) System monitoring phase: INA226 monitors system power consumption in real time; The LM75 temperature sensor monitors the system temperature. Dynamically adjust the PWM cooling fan speed based on temperature data; The Shengteng 310P dynamically adjusts its operating frequency and voltage according to the system load.

[0041] In summary, the multimodal heterogeneous computing processing system disclosed in this invention uses the Ascentage 310P processor as the main processing unit, responsible for system control, task scheduling, and AI inference computing; and the Xilinx Zynq UltraScale+ FPGA as the coprocessing unit, responsible for hardware acceleration and interface expansion. The two are connected through a high-speed bus to form a master-slave collaborative heterogeneous computing architecture. The beneficial technical effects brought by this invention include: (1) strong processing power: the Ascentage 310P provides powerful AI inference capabilities, and the FPGA provides flexible hardware acceleration. The two work together to improve the overall system performance; (2) high energy efficiency: the heterogeneous architecture allocates appropriate computing resources according to the characteristics of the task, improving the energy efficiency ratio; (3) rich interfaces: it supports a variety of communication protocols and peripheral interfaces to adapt to different application scenarios; (4) flexible expansion: it supports functional expansion through a 688PIN BGA connector; (5) precise power consumption control: multi-level power management and temperature monitoring ensure stable system operation.

[0042] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multimodal heterogeneous computing processing system, characterized in that, The system includes: An embedded processor; as the main processing unit, the embedded processor is used to perform system control, task scheduling, and AI inference calculations; wherein, the embedded processor adopts the Ascentage 310P processor; Field-programmable gate arrays (FPGAs) serve as coprocessor units, enabling hardware acceleration and interface expansion. These FPGAs utilize a combination of Xilinx Zynq UltraScale and FPGA. Storage modules; including LPDDR4x memory, eMMC memory, and SPI NOR Flash; High-speed interface module; including SERDES interface, PCIe interface, and USB 3.1 interface; Network communication module; including multiple Gigabit Ethernet interfaces; Expansion interface module; connects to external functional modules via a 688-pin BGA connector; Power management module; provides stable power supply to all components of the system.

2. The multimodal heterogeneous computing processing system according to claim 1, characterized in that, The embedded processor is connected to the field-programmable gate array (FPGA) via a high-speed bus, forming a master-slave collaborative heterogeneous computing architecture. The embedded processor runs the operating system and applications, while the FPGA implements hardware acceleration and interface expansion.

3. The multimodal heterogeneous computing processing system according to claim 2, characterized in that, The storage module includes 12 sets of LPDDR4x memory, 64GB eMMC memory, and SPI NOR Flash, which are connected to the embedded processor and the field-programmable gate array via DDRC, eMMC, and SFC interfaces, respectively.

4. The multimodal heterogeneous computing processing system according to claim 3, characterized in that, The high-speed interface module includes: Four SERDES interfaces: M2.0-M2.3, M2.4-M2.5, M3, and M4. PCIe to USB 3.1 controller; provides two USB 3.1 ports; SATA interface; connects to a field-programmable gate array via SERDES M4.

5. The multimodal heterogeneous computing processing system according to claim 4, characterized in that, SERDES M2.4-M2.5 interfaces connect to video codec cards, supporting SDI video input and output. Video encoding and decoding are performed by an embedded processor, and a field-programmable gate array is used to perform video data scheduling.

6. The multimodal heterogeneous computing processing system according to claim 5, characterized in that, The network communication module includes three Gigabit Ethernet interfaces, each implemented through an RTL8521 PHY chip. One of these is a management port, where the embedded processor handles network protocol processing.

7. The multimodal heterogeneous computing processing system according to claim 6, characterized in that, The system also includes: an RTC real-time clock module, an INA226 power monitoring chip, an LM75 temperature sensor, and peripheral devices connected via IIC0 and IIC2 buses.

8. The multimodal heterogeneous computing processing system according to claim 7, characterized in that, The system also includes: a UART0 debug serial port, a UART1 RS485 interface, a UART2 RS232 interface, and a PWM-controlled cooling fan.

9. A multimodal heterogeneous computing processing system according to claim 8, characterized in that, The power management module includes: PSIP power input subsystem, VRD+DRMOS voltage regulator, and PMU6421 power management unit.

10. A data processing method, characterized in that, The method is implemented using a multimodal heterogeneous computing processing system as described in any one of claims 1-9, and the method includes: Receive external data via a 688-pin BGA connector; Embedded processors perform system control, task scheduling, and AI inference computation; Field-programmable gate arrays (FPGAs) are used for hardware-accelerated processing and interface data exchange. Dynamic power consumption control of the system is achieved through a power management module.

Images