Photonic NPU embedded neural network processor

Abstract

The invention discloses a photon NPU (Network Processing Unit) embedded neural network processor, which comprises (1) an electromagnetic wave bus, (2) a multiply-add module, (3) an activation function module, (4) a two-dimensional data operation module and (5) a decompression module, the NPU embedded neural network processor is divided into independent modules with different sizes according to functions and purposes, each functional module is provided with an independent input and output end, and each functional module is provided with an independent input end and an independent output end. All input and output ends are connected with a transmitting end and a receiving end of an electromagnetic wave bus, the bandwidth advantage of electromagnetic waves is utilized, a processing terminal is further formed through interconnection, and the electromagnetic wave bus is used for replacing a control bus, a data bus, an address bus and all replaceable circuits to transmit needed control signals, data and data addresses. An electromagnetic wave bus is used as a carrier of data needing to be processed, a data address and control signal transmission, and a photon (NPU) embedded neural network processor processes and outputs a result after passing through a radio frequency chip and a baseband chip.

Description

TECHNICAL FIELD

[0001] The technical field of photonics chips. BACKGROUND

[0002] Over the past two decades, the clock frequency of microprocessors has stagnated, and the continuation of Moore's law is being challenged like never before as nanofabrication technology approaches its inevitable physical limits. In this post-Moore era, tremendous efforts are being made across disciplines and research areas to develop high-energy-efficient and ultrafast computers.

[0003] As integrated circuits develop to nanometer technology, they are approaching the physical limit and the so-called red wall problem appears: first, the delay of the line is more and more important than that of the gate. Long lines not only have transmission delay problems, but also have energy consumption problems. Second, the feature size has been small enough to make chip manufacturing defects inevitable, and fault-tolerant and error-tolerant technologies need to be researched from three aspects of defect tolerance, fault tolerance and error tolerance. Third, leakage current and power consumption become very important, and self-management technology of power consumption needs to be adopted. In the next few decades, autonomous driving technology will reshape the entire society. When passengers in the car can release their hands and attention from driving, this will overturn most of the current business models, such as shopping, video, and games. At the same time, the efficiency of transportation will be greatly improved, and human society will be greatly improved. But now for the autonomous driving and intelligent car industry, it is still in a very early stage, only L2 and L3 solutions are deployed on high-end cars, and above L4 currently there is no mature and stable solution. Tesla, Huawei, Xiaopeng, and Weilai have launched L3 / L4 autonomous driving solutions, while traditional car manufacturers such as Volkswagen, Ford, Jaguar, and Land Rover have also launched autonomous driving solutions not lower than L2 in some products. However, in order to realize this dream, it requires unlimited computing power, unlimited storage space, and super-strong reliability, of course, this is not only for luxury cars, but also for affordable models that everyone can afford. This is exactly the same as what happened in the smartphone market over the past decade. It is believed that China can become one of the leading marketing of autonomous driving in the next decade, as China has many new electric car giants such as Weilai, Xiaopeng, Ideal, BYD, and Geely to achieve industrial upgrading.

[0004] It is projected that by 2025, over 50% of new car sales will offer Level 2 / Level 3 autonomous driving capabilities, increasing to 90% by 2030. By 2030, the overall automotive chip market will reach $115 billion, accounting for 11% of the entire chip market. There are many reasons hindering the widespread adoption of autonomous driving. Besides the lack of sufficient testing and validation data from fleets in all complex real-world environments, I believe computing power is the primary factor limiting Level 4 and above autonomous driving. A typical Level 4 autonomous driving system has 4-6 radars, 1-6 lidars, 6-12 cameras, and 8-16 ultrasonic sensors. These sensors will generate a total of 3 Gbit / s (~1.4 TB / h) to 40 Gbit / s (~19 TB / h) of data. For example, Waymo's autonomous vehicles have 8 cameras, 6 radars, and 6 lidars. To process this data, the computing power requirement for Level 4 autonomous driving may be over 1000 TOPS, while Level 5 may require 10 times that, reaching 10000 TOPS. However, the current mainstream solutions only have around 100-250 TOPS.

[0005] However, NPUs used in autonomous driving systems are not merely about computing power; they may possess characteristics independent of general-purpose NPUs. Designing a competitive autonomous driving NPU requires understanding the challenges faced by both general-purpose NPUs and ADAS NPUs. With Moore's Law failing, traditional NPUs and high-performance chipsets face significant challenges.

[0006] The development history of computer buses includes the early PC bus and ISA bus, PCI / AGP bus, PCI-X bus, and the mainstream PCI Express and HyperTransport high-speed serial buses. From the PC bus to the ISA and PCI buses, and then from PCI to the PCI Express and HyperTransport architecture, computers have undergone three major leaps in development during these three major transformations. Correspondingly, the processing speed, functions, and software platforms of computers have all evolved in the same way. Obviously, without the advancement of bus technology as a foundation, the rapid development of computers would be impossible. The industry's thirst for high-speed buses is endless. In computer systems, various functional components are addressed through the address bus, and the speed of the bus has a great impact on system performance. For this reason, the bus is hailed as the nerve center of the computer system. However, compared with functional components such as the CPU, graphics card, memory, and hard drive, the pace of bus technology advancement has been much slower. In the more than 20-year history of PC development, the bus has only undergone three upgrades, but each of these changes has given computers a completely new look. In today's era of artificial intelligence computing based on big data and cloud computing, the existing "memory wall" problem has become more prominent. The von Neumann architecture requires frequent data exchange, which brings about bandwidth, latency and power consumption problems. The real root cause is that the outdated circuit bus has become a bottleneck that limits computing power.

[0007] Compared to electrons, millimeter waves have inherent advantages as information carriers, including ultra-high speed, ultra-high bandwidth, and ultra-low loss. In particular, millimeter waves and centimeter waves (Sub-6) are especially outstanding. When transmitting information, millimeter waves and centimeter waves in the electromagnetic spectrum have extremely fast response times, and the information rate can reach tens of Tb / s, which can improve performance by hundreds of times. Secondly, they have extremely high information capacity, which is 3 to 4 orders of magnitude higher than that of electrons. New technologies can be used to further improve computing speed and power efficiency. Summary of the Invention

[0008] In view of this, a photonic NPU (Neural Networks Process Units) embedded neural network processor is invented, including: (1) an electromagnetic wave bus, (2) a multiply-accumulate module, (3) an activation function module, (4) a two-dimensional data operation module, and (5) a decompression module. The NPU (Neural Networks Process Units) embedded neural network processor is divided into independent modules of different sizes according to their functions. Each module has independent input and output terminals. All input and output terminals are connected to the millimeter wave transmitter and receiver. Taking advantage of the bandwidth of millimeter waves, a processing terminal is further formed by interconnection. Millimeter waves are used to replace the control bus circuit, data bus circuit, address bus circuit, and all other replaceable circuits to transmit the control signals, data, and data addresses required for processing. Millimeter waves are used as the carrier for transmitting the data, data addresses, and control signals to be processed. After passing through the radio frequency chip and baseband chip, the photonic NPU (Neural Networks Process Units) embedded neural network processor processes the data and outputs the results.

[0009] This photonic NPU (Neural Networks Processing Unit) embedded neural network processor completes the output calculation results through repeated transmission, reception, and processing. Millimeter waves have extremely fast response times, with information rates reaching tens of Tb / s, which is hundreds of times faster than traditional circuit data transmission. Secondly, they have extremely high information capacity, 3-4 orders of magnitude higher than electrons. Employing novel technologies, low switching latency and high transmission bandwidth can be achieved. The aforementioned photonic NPU embedded neural network processor is easy to implement, relatively inexpensive, and capable of ultra-fast computation.

Claims

1. This invention discloses a photonic NPU (Neural Networks Processing Unit) embedded neural network processor, comprising: (1) Electromagnetic wave bus, (2) Multiply-accumulate module, (3) Activation function module, (4) Two-dimensional data operation module, (5) Decompression module. It adopts the architecture of "data-driven parallel computing" and is particularly good at processing video to accelerate the operation of neural networks, solving the problem of low efficiency of traditional chips in neural network operation. The multiply-accumulate module is used to calculate matrix multiplication, convolution, dot multiplication and other functions. The activation function module adopts the highest 12th order parameter fitting method to realize the activation function in the neural network. The two-dimensional data operation module is used to realize the operation on a plane, such as downsampling, plane data copying, etc. The decompression module is used to decompress the weight data. The characteristic of the electromagnetic wave bus is that it replaces the control bus circuit, data bus circuit, address bus circuit and all replaceable circuits as the carrier for transmitting the control signal data and data address required for calculation.

2. The electromagnetic wave bus: Due to different application scenarios, millimeter waves can be carrier signals, and centimeter waves (sub-6) can also be carrier signals. Since they are all electromagnetic waves, here we refer to the control bus, data bus, address bus, and all other circuits that can be used as alternatives as carriers for transmitting the control signal data and data address required by the graphics processor. The carriers for transmitting the control signal data and data address required by the graphics processor are collectively referred to as electromagnetic wave buses.

3. Its characteristic is that it uses an electromagnetic wave bus as a carrier to repeatedly transmit and receive electromagnetic waves and process them to complete the calculation and output the calculation results. 4.4 It adopts a "data-driven parallel computing" architecture, which is particularly good at processing video to accelerate neural network operations and solves the problem of low efficiency of traditional chips in neural network operations.

5. A photonic NPU (Neural Networks Processing Unit) embedded neural network processor device, characterized in that... The control bus circuit, data bus circuit, address bus circuit, and all other replaceable circuits are replaced by an electromagnetic wave bus as the carrier for transmitting control signals, data, and data addresses required for computation. The calculated results are output after passing through the methods and steps described in claims 1 to 5.