Event driven integrated circuit with interface system

Abstract

The present invention relates to an event-driven integrated circuit, which includes a sensor, an interface system and a processor. The scheme asynchronously generates and processes events with addresses, and the interface system includes a replication module, a fusion module, a secondary sampling module, an area of ​​interest module, an event routing module, etc., which form a programmable daisy chain. The sensor, the interface system and the processor are coupled on a single chip through an adapter board, and different bare crystals can be manufactured using different processes. The scheme can eliminate signal loss and noise interference in the prior art, and can achieve technical effects such as high-speed signal processing and lower chip occupation area and manufacturing cost, solving the technical problems of large chip area and weak signal processing capability in the prior art. In addition, the ingenious design of the interface system enriches the function and configurability of the interface system, and provides various power consumption, function and speed advantages for subsequent processing.

Description

Technical Field

[0001] The invention relates to an event-driven integrated circuit, and in particular to a low-power integrated circuit with an interface module for asynchronously processing events. Background Art

[0002] There are event-driven sensors in the prior art. One type of event-driven sensor is an event-driven camera, which includes a pixel array having pixels. According to the brightness change of the pixels, the event-driven camera generates an event, where the event includes a changing identifier, such as -1 for darker and +1 for brighter. Such cameras are called dynamic vision sensors (DVS). There are other known event-driven sensors, such as one-dimensional sensors and sound sensors.

[0003] DVS generates events in an asynchronous manner, and it is an event-driven sensor. Traditional clock-based cameras need to read all frames or lines of all pixels, which is incomparable to DVS. DVS provides ultra-fast image processing while still maintaining a low rate because DVS only records changes.

[0004] However, subsequent event processing steps require a fundamentally different processing architecture than the clocked, synchronously operating processors that the traditional Von-Neumann architecture relies on.

[0005] Designing a versatile processing structure for an event-driven system is a challenging task, especially considering the interconnection between various preprocessing and processing components. Compared with clock systems, event-driven systems must process different events according to different processes, especially when exchanging data between different components, involving the AD-HOC handshake mechanism. Such a handshake mechanism includes the necessary exchange of data requests, confirmation of requests, and subsequent data exchange.

[0006] Components in an event-driven system, especially sensors, may have different specifications considering pixel array size, different hot-pixels, event addressing, etc. When assembling different components into a system, it will be a tedious, time-consuming and expensive task to adapt the processing pipeline of the system. The term "processing pipeline" specifically refers to the wiring, such as the interconnection between different components, and also refers to the data processing of components and the data transmission between different components. The term "processing pipeline" also specifically refers to the specific way how various output ports or various first components of the system are connected to various input ports or various second components of the system.

[0007] The most fundamental design concept of event-driven systems is to pursue extreme low power consumption to adapt to edge computing. In order to achieve extreme low power consumption, technicians in this field have tried every possible way from various angles.

[0008] In the prior art, on August 8, 2014, Science magazine first introduced IBM's brain-inspired chip TrueNorth, which has 5.4 billion transistors, 4096 neural synaptic cores, 1 million programmable pulse neurons, and 2.56 million configurable synapses. The chip structure adopts an event-driven design and is an asynchronous-synchronous hybrid chip: the router, scheduler, and controller use a quasi-delay non-sensitive clock-free asynchronous design, while the neuron is a traditional clocked synchronous circuit solution, and its clock is generated by an asynchronous controller with a global clock frequency of 1kHz. If measured with a video input of 30 frames / second and 400*240 pixels, the power consumption of the chip is 63mW. For details, please refer to:

[0009] Prior art 1: "A million spiking-neuron integrated circuit with ascalable communication network and interface", Paul A.Merolla, John V. Arthur etal, Vol.345, Issue 6197, SCIENCE, 8 Aug 2014.

[0010] The CVPR conference held on July 21-26, 2017 disclosed a gesture recognition system based on the IBM TrueNorth chip: refer to Fig. 1 and Fig. 4 of the article (or the appendix of the present invention).Fig.11 , please refer to the original text for details of this figure, the present invention will not repeat it), the TrueNorth processor on the NSIe development board receives events output from the DVS128 via USB2.0. In other words, the DVS and the processor are connected via a USB cable. For details, please refer to:

[0011] Prior Art 2: "A Low Power, Fully Event-Based Gesture Recognition System", Arnon Amir, Brian Taba etal, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 21-26 July 2017.

[0012] In November 2017, Intel disclosed its brain-like chip Loihi. From the public presentation materials, it is known that the power consumption of the chip is 74mW. In the neuromorphic computing progress disclosed on Intel's official website, it is pointed out that Loihi hardware is available for research by INRC members in fields that require direct access to hardware (such as robotics). These systems include a USB form factor code-named "Kapoho Bay". In addition to providing a USB interface for Loihi, KapohoBay also provides an event-driven hardware interface for the DAVIS 240CDVS silicon retinal camera from iniVation. In other words, for Loihi, it is also connected to DVS via USB. For details, please refer to the network link (the link content can also be obtained in the Internet archive):

[0013] https: / / newsroom.intel.com / news / intel-announces-neuromorphic-computing-research-collaborators / #gs.vj5gmb, 6Dec 2018.

[0014] On August 1, 2019, Nature magazine reported on the cover article of Tianjic, a brain-like chip developed by Tsinghua University. Its power consumption is 400mW at a typical voltage of 0.9V (950mW at 1.2V). In other subsequent articles by the team, other technical details of Tianjic were disclosed. The article points out: The single-chip PCB is equipped with Altera Cyclone 4FPGA and communication interfaces USB and SPI. Referring to Fig-2b of the article, the camera is connected to the FPGA via USB and finally to Tianjic. In other words, the visual sensor and processor in this disclosed technology are also connected via a USB cable. The article can be referenced at:

[0015] Prior Art 3: "A hybrid and scalable brain-inspired robotic platform", ZheZou, Rong Zhao etal, Scientific Reports, 23 Oct 2020.

[0016] The above are examples of the efforts made by the world's top event-driven system designers to pursue extreme low power consumption. Compared with the power consumption of about 100-200W of traditional CPUs based on the von Neumann architecture, its energy efficiency has been greatly improved. In order to connect sensors and AI processors, the above existing technologies all use USB cables / interfaces or other similar interface technologies to achieve their connection. These world's top event-driven designers did not realize what was wrong with this.

[0017] However, the inventor independently discovered and realized that: in the above-mentioned prior art, USB cables (or other cable implementation technologies) have a certain connection length, so the system may suffer from signal loss and noise interference coupled to the cable, and because it is a cable, each handshake between devices, such as necessary communication before and after data transmission, will consume more energy and slow down the processing speed of the system, which has an adverse effect on the performance of brain-like chips. In the prior art in this field, these top designers did not realize the adverse effects of this factor and believed that the proposed technical solutions have exhausted all efforts to pursue extreme low power consumption and the solutions have met the needs of various technical indicators. They also did not point out that in the face of the above-mentioned adverse factors, they should take or imply that actions should be taken to eliminate the adverse effects. Therefore, without the clear inspiration of other technical solutions in this field, these world-class designers lack the recognition / motivation that the above-mentioned technical means need to be further improved.

[0018] In addition, the inventors further discovered in their research / certain solutions to solve the above problems that for DVS, in order to obtain better quality image information, those skilled in the art need to select a special semiconductor process to manufacture DVS, such as a CIS-CMOS process image sensor. However, for AI processors (such as the sCNN processor described later), if conventional CMOS processes are used, this conventional CMOS process is not suitable for manufacturing high-quality image sensors (imaging effects are not ideal). If all CIS-CMOS processes are used to manufacture sensors and AI processors, especially on the same chip / die, the AI ​​processor will occupy a large amount of chip area and increase chip costs, which will lose commercial competitiveness in view of the development trend of pursuing smaller and smaller chip footprints. Therefore, how to eliminate signal loss and noise interference, and preferably further pursue lower chip footprints and manufacturing costs, is an important problem to be solved for the industrialization / commercialization of brain-like chips.

[0019] Completely different from the traditional von Neumann architecture, the event generation and event processing involve the event transmission process, which can be regarded as an interface system. The design of this interface system will be a completely different technical challenge from the past. How to design this interface system so that it can transmit events efficiently, flexibly, and with low power consumption is also one of the comprehensive challenges faced by those skilled in the art.

[0020] The present invention is proposed to solve one of the above-mentioned technical problems or a combination of several technical problems. The technical solution of the invention can solve or alleviate the combination of one or more technical problems mentioned above.

[0021] In addition to clearly stating the source of the technology, the technology mentioned in the above background technology may be wholly or partially owned or undisclosed technology, that is, the applicant does not admit that the technology mentioned in the background technology must belong to the prior art in the sense of patent law, unless there is substantial evidence to prove it. At the same time, the technical solutions and technical features disclosed in the above background technology have been disclosed with the disclosure of the patent document of this invention. Summary of the invention

[0022] In order to solve or alleviate the combination of one or more of the above technical problems, the technical solution proposed by the present invention is implemented in the following way:

[0023] An integrated circuit comprises an event-driven sensor (10), an event-driven interface system (20) and an event-driven processor (30), wherein the event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are coupled to a single chip (3).

[0024] In a certain class of embodiments, the event-driven sensor (10) is configured to: asynchronously generate and asynchronously output an event (100) after an input device (11) of the event-driven sensor (10) detects an event generation signal or / and a change in the event generation signal, wherein the event (100) includes or is associated with an event address indicating the input device (11), and the output end of the event-driven sensor (10) is coupled to the input end of the event-driven interface system (20);

[0025] The event-driven interface system (20) is configured to: asynchronously receive the event (100) and pre-process the received event (100), the output end of the event-driven interface system (20) being coupled to the input end of the event-driven processor (30);

[0026] The event-driven processor (30) is configured to: receive the event (101) pre-processed by the event-driven interface system (20), and process the received event (101) in an asynchronous manner;

[0027] The event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are coupled to a single chip (3) via an adapter board (40).

[0028] In certain embodiments, the event-driven interface system (20) and the event-driven processor (30) are both located in the first die (1-1); or, the event-driven sensor (10) and the event-driven interface system (20) are both located in the second die (1-2); or, a portion of the event-driven interface system (20) and the event-driven processor (30) are both located in the first die (1-1) and another portion of the event-driven interface system (20) and the event-driven sensor (10) are both located in the second die (1-2).

[0029] In a certain class of embodiments, the event-driven interface system (20) and the event-driven processor (30) are both located on a first bare die (1-1), and a second bare die (1-2) where the event-driven sensor (10) is located is stacked on the first bare die (1-1) where the event-driven interface system (20) and the event-driven processor (30) are located.

[0030] In certain embodiments, the adapter plate (40) is a silicon adapter plate or a glass adapter plate.

[0031] In certain embodiments, the event-driven sensor (10), the event-driven interface system (20), and the event-driven processor (30) are packaged on a single chip (3) using 2.5D or 3D packaging technology.

[0032] In certain embodiments, the event-driven sensor (10) is a combination of one or more of the following types: a point sensor, a 1D sensor, or a 2D sensor.

[0033] In certain embodiments, the event-driven sensor (10) is a combination of one or more of the following types: sound / vibration sensor, dynamic visual sensor.

[0034] In certain embodiments, the event-driven processor (30) is configured with a spiking neural network.

[0035] In certain embodiments, the event-driven processor (30) is configured with a spiking convolutional neural network.

[0036] In some embodiments, the first bare die and the second bare die are manufactured using different processes.

[0037] In a certain class of embodiments, the event-driven interface system (20) includes at least one interface module (200), wherein the interface module (200) is configured in a programmable daisy chain form to asynchronously process events (100) received from the event-driven sensor (10).

[0038] In some embodiments, at least one interface module (200) includes a replication module (201), which is configured to: receive an event (100) and perform a replication operation to obtain a replicated event (100c), wherein the event (100) comes from the event-driven sensor (10) or from other interface modules (200) of the event-driven interface system (20) (in some embodiments, it is a fusion module), and send the replicated event (100c) to an external processing pipeline, and send the event (100) along the daisy chain.

[0039] In certain embodiments, at least one interface module (200) includes a fusion module (202), which is configured to: receive events (100, 100e) from at least two different places, wherein the events (100) come from other interface modules (200) of the event-driven interface system (20) (in certain embodiments, they are replication modules) or the event-driven sensor (10); the events (100e) also come from components / modules of the integrated circuit or other integrated circuits or other event-driven sensors, and send part or all of the received events (100, 100e) to subsequent interface modules (200) along the programmable daisy chain.

[0040] In certain embodiments, the at least one interface module (200) includes a secondary sampling module (203) configured to assign a single address to a plurality of received events (100).

[0041] In a certain class of embodiments, the separation module (203-4) included in the secondary sampling module (203) is configured to: route the event (100) to the scaling register in the associated secondary sampling module (203) according to the address value of the received event (100); the scaling register is configured to: split, secondary sample, pool or / and shift the received address value, and output the address value to the address reorganization module (203-5) in the secondary sampling module (203), and the address reorganization module (203-5) is configured to: adjust the event address according to the scaled address value, and then send the event (100) with the adjusted address along the programmable daisy chain.

[0042] In certain embodiments, the at least one interface module (200) includes an area of ​​interest module (204) configured to: adjust an attribute of at least one event address, wherein the adjustment method includes one or more of the following methods: shifting, flipping, exchanging or / and rotating an attribute of at least one event address; or / and

[0043] Events (100) having address attribute values ​​outside a programmable address attribute value range are discarded, and events (100) that are not discarded are sent along the programmable daisy chain.

[0044] In a certain class of embodiments, the at least one interface module (200) includes an event routing module (205), which is configured to: receive an event (100), add header information to the received event (100), and send the event (100) together with the header information of the event (100) to the event-driven processor (30) and / or other event-driven processors or other processing pipelines.

[0045] In certain embodiments, the at least one interface module (200) includes a rate control module configured to: when a maximum speed is exceeded, only send a portion of the events (100) along the programmable daisy chain to limit the rate of events to not exceed the maximum rate.

[0046] In certain embodiments, the at least one interface module (200) includes a mapping module (206) configured to map an event address to another event address.

[0047] In certain embodiments, the mapping module (206) includes one or a combination of the following:

[0048] An area of ​​interest module, a lookup table module, and a flipping or / and rotating module; wherein the flipping or / and rotating module is configured to flip or / and rotate the event address of the event (100).

[0049] In certain embodiments, the at least one interface module (200) includes an event address rewriting module (207) configured to convert a received event address into a unified address format, thereby transmitting the unified event address format on the programmable daisy chain.

[0050] In some embodiments, the at least one interface module (200) includes an event address filtering module (208) configured to filter out a series of events (100) having a specifically selected event address.

[0051] In a certain embodiment, the event address filtering module (208) is specifically a hot pixel filtering module (208'), which is configured to: filter events (100) with specific event addresses, and store a preset list of event addresses to be filtered through a CAM memory (208'-3).

[0052] In certain embodiments, any one or more interface modules (200) of the event-driven interface system (20) can be bypassed by a programmable switch.

[0053] In addition, the present invention also provides an event-driven interface system (20), which is coupled to an event-driven sensor (10) and an event-driven processor (30) to form an integrated circuit, wherein the event-driven sensor (10) generates and asynchronously outputs an event (100), wherein the event (100) includes or is associated with an event address of an input device (11) on the event-driven sensor (10) that indicates the generation of the event; the event-driven interface system (20) includes at least one interface module (200), wherein the interface module (200) is configured in a programmable daisy chain form to asynchronously process the event (100) received from the sensor (10).

[0054] In certain embodiments, the at least one interface module (200) includes one or more of the following: a replication module (201), a fusion module (202), a secondary sampling module (203), a region of interest module (204), and an event routing module (205); wherein:

[0055] The replication module (201) is configured to: receive an event (100) and perform a replication operation to obtain a replication event (100c), wherein the event (100) comes from the event-driven sensor (10) or from other interface modules (200) of the event-driven interface system (20), and send the replication event (100c) to an external processing pipeline, and send the event (100) along the daisy chain;

[0056] The fusion module (202) is configured to: receive events (100, 100e) from at least two different places, wherein the event (100) comes from other interface modules (200) of the event-driven interface system (20) or the event-driven sensor (10); the event (100e) also comes from the integrated circuit or other integrated circuit components / modules or other event-driven sensors, and send part or all of the received events (100, 100e) to subsequent interface modules (200) along the programmable daisy chain;

[0057] The secondary sampling module (203) is configured to: assign a single address to a plurality of received events (100);

[0058] The region of interest module (204) is configured to:

[0059] Adjusting the attributes of at least one event address, wherein the adjustment method includes one or more of the following methods: shifting, flipping, exchanging and / or rotating the attributes of at least one event address; or / and

[0060] discarding events (100) whose address attribute values ​​are outside a programmable address attribute value range, and sending events (100) that are not discarded along the programmable daisy chain;

[0061] The event routing module (205) is configured to:

[0062] Receive an event (100), add header information to the received event (100), and send the event (100) together with the header information of the event (100) to the event-driven processor (30) and / or other event-driven processors or other processing pipelines.

[0063] In certain embodiments, along the event transmission direction of the programmable daisy chain, the at least one interface module (200) has the following interface module coupling sequence:

[0064] A replication module (201), a fusion module (202), a subsampling module (203), a region of interest module (204), and an event routing module (205); or

[0065] A fusion module (202), a replication module (201), a secondary sampling module (203), a region of interest module (204) and an event routing module (205).

[0066] In a certain class of embodiments, for the replication module (201), the event (100) comes from other interface modules (200) of the event-driven interface system (20), specifically the fusion module (202); or / and for the fusion module (202), the event (100) comes from other interface modules (200) of the event-driven interface system (20), specifically the replication module (201).

[0067] In a certain embodiment, the upstream of the interface module coupling sequence further includes: an event address rewriting module (207) or / and an event address filtering module (208); wherein the event address rewriting module (207) is configured to: convert the received event address into a unified address format, thereby transmitting the unified event address format on the programmable daisy chain;

[0068] The event address filtering module (208) is configured to filter out a series of events (100) having a specifically selected event address.

[0069] In certain embodiments, the event (100) is first processed by the event address rewriting module (207) and then processed by the event address filtering module (208).

[0070] In certain embodiments, the event address filtering module (208) is specifically a hot pixel filtering module (208'), which is configured to:

[0071] Events with specific event addresses are filtered (100), and a preset event address list to be filtered is stored in a CAM memory (208'-3).

[0072] In a certain class of embodiments, the at least one interface module (200) further includes a mapping module (206), the mapping module (206) including one or a combination of the following: an area of ​​interest module, a lookup table module, a flipping and / or rotation module; wherein the flipping and / or rotation module is configured to flip and / or rotate the event address of the event (100).

[0073] In certain embodiments, the at least one interface module (200) further comprises a rate control module configured to, when a maximum speed is exceeded, send only a portion of the events (100) along the programmable daisy chain to limit the rate of events to not exceed the maximum rate.

[0074] In certain embodiments, any one or more interface modules (200) of the event-driven interface system (20) can be bypassed by a programmable switch.

[0075] In certain embodiments, the event-driven sensor (10), the event-driven interface system (20), and the event-driven processor (30) are coupled to a single chip (3) via an adapter board (40), or are manufactured in the same bare die.

[0076] Compared with the prior art, the technical solution of the present invention has the following beneficial effects, including but not limited to:

[0077] 1. Provides an integrated solution of sensors, interfaces, and event-driven processors that can eliminate signal loss and noise interference, and can further pursue lower chip footprint and manufacturing costs, and the solution can achieve sub-milliwatt power consumption.

[0078] 2. An event-driven interface system is provided, which can transmit events efficiently, flexibly and with low power consumption, and provide event pre-processing function for the processor to process events efficiently and conveniently.

[0079] The technical solutions, technical features, and technical means disclosed above may not be completely identical or consistent with the technical solutions, technical features, and technical means described in the subsequent specific implementation methods. However, the new technical solutions disclosed in this part also belong to a part of the many technical solutions disclosed in the present invention document. The new technical features, technical means disclosed in this part and the technical features and technical means disclosed in the subsequent specific implementation methods are in a reasonable combination with each other, and more technical solutions are disclosed, which is a useful supplement to the specific implementation methods. Similarly, some details in the drawings of the specification may not be clearly described in the specification, but if the technical personnel in this field can infer their technical meaning based on the description of other relevant texts or drawings of the present invention, the common technical knowledge in the field, and other prior arts (such as conferences, journal articles, etc.), then the technical solutions, technical features, and technical means that are not clearly recorded in the text of this part also belong to the technical content disclosed in the present invention, and can be used in combination as described above to obtain the corresponding new technical solutions. The technical solution combined by all the technical features disclosed in any position of the present invention is used to support the generalization of the technical solution, the modification of the patent document, and the disclosure of the technical solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a schematic diagram of an event-driven circuit system according to a certain embodiment of the invention;

[0081] Figure 2 is a schematic diagram of a circuit system according to another embodiment of the invention;

[0082] Figure 3 is a cross-sectional view of a chip according to an embodiment of the invention;

[0083] Figure 4 is a schematic cross-sectional view of a 3D chip geometric structure according to an embodiment of the invention;

[0084] Figure 5 is a schematic diagram of a circuit system including a sound / vibration sensor according to an embodiment of the invention;

[0085] Figure 6 It is the event processing flow chart produced by the sensor;

[0086] Figure 7 It is a flow chart of the event processing of the sound sensor recording the vibration signal;

[0087] Figure 8 It is a daisy chain diagram of the interface system;

[0088] Fig. 9 is a schematic diagram of a hot pixel filtering module having the ability to filter events having a selected event address;

[0089] Fig.10 It is a schematic diagram of the sub-sampling module;

[0090] Fig.11 It is a gesture recognition system based on IBM TrueNorth in the prior art. DETAILED DESCRIPTION

[0091] (I) Statement on the description of the technical solution

[0092] In order to clearly describe and fully understand the technical solution disclosed in the present invention, the following agreements are made in any patent document of the present invention:

[0093] The embodiments described in this part, even in the same figure and the same area, are not just descriptions of a specific embodiment, but selective descriptions of potential embodiments with certain technical features. The embodiments disclosed in this invention document are all reasonable combinations of the following technical features, as long as such combinations are not logically contradictory or meaningless, so some technical features are not necessarily present in certain types of specific implementations.

[0094] Method and product embodiments may describe certain technical features separately. Usually, the invention document implies that other corresponding embodiments also have corresponding technical features / devices / steps that match / correspond to / coordinate with the technical features, but they are just not clearly described in words. For example, method embodiments implicitly include certain steps / instructions / functions executed / implemented by a device / module / component of product embodiments, and product embodiments implicitly include devices / modules / components that implement certain steps / instructions / functions executed by method embodiments. These implied technical features are also used to support the summary of technical solutions, the modification of patent documents, and the disclosure of technical solutions.

[0095] The English words / phrases in brackets of non-English terms involved in any position of the present invention are further supplementary explanations of the meaning of the term and are used to assist in explaining its common English expression in this field. When non-English terms cannot be correctly understood, interpreted, or there is a contradiction, the English meaning can assist in understanding the meaning here, and the English meaning shall prevail when necessary. In addition, not all of the above non-English terms appearing in any position are explicitly interpreted in the above manner, but the meaning interpreted in one place should also be able to explain the same term appearing in other positions.

[0096] In the specification, a specific term may be given a specific explanation, and the same term may be interpreted according to this explanation in any document related to the present invention, such as the attached claims.

[0097] The "module", "module group", "component", "part", "assembly", "part", "system" (if any) and the like mentioned in any document of the present invention refer to a product or a part of a product implemented only by hardware, only by software, or a combination of software and hardware. Unless otherwise clearly indicated by the context, the present invention does not imply that the above terms can only be implemented by hardware or software.

[0098] The comma "、" in Chinese usually expresses a pause between parallel words or phrases, separates similar parallel things, or separates a sequence number and the text that is numbered with Chinese characters. Therefore, it usually does not clearly determine the expression of the logic "or / and" by itself, and the logic it expresses usually needs to be analyzed based on the context.

[0099] In any patent document of any language of the present invention, it is agreed that: similar to "A, B, C" means: usually express A or B or C, except that it must be "and" logic according to the context; similar to "A, B or C" means: A or B or C; similar to "A, B and C" means: A and B and C. Similar to "A, B or C" means: A or B or C; similar to "A, B and C" means: A and B and C.

[0100] The multiple-solution description methods "A and / or B", "A or / and B", "A and / or B", and "A or / and B" all include three parallel technical solutions: (1) A; (2) A and B; (3) B.

[0101] In non-mathematical formulas, " / " generally means logical "or".

[0102] In any patent document of the present invention, a description similar to "the first input terminal of a certain module" is equivalent to "the first input terminal of a certain module" and refers to the same object, unless "a certain module" itself is not sufficient to distinguish the specific object or causes ambiguity, and the most reasonable interpretation should be made according to the context.

[0103] The words “may / may be” (may, may be, might, indicating choice, except when the context expresses “capability”) appearing anywhere in the present invention document are a way of describing a preferred embodiment, which implies that there may be potential, other types of reasonable alternatives. When the technical terms “approximately”, “approximately”, “close to”, etc. appearing anywhere in the present invention express descriptors with approximate meanings, what they are meant to express is: the technical solution is allowed to be implemented with errors without affecting the solution of the technical problem, that is, it is not required that the data obtained after strict measurement of actual parameters strictly conform to the general mathematical definition (because there is no physical entity that completely conforms to the mathematical definition). This term is not vague and ambiguous, so as to make it unclear what the scope defined / expressed by the technical solution is. In fact, whether the technical problem can still be solved should be the criterion for determining whether it falls within the limited scope.

[0104] In any position of the present invention, there may be situations where the terms used to express the same object are not exactly the same. Usually, this is due to the difference between the abbreviation and the full name, the insertion of function words such as auxiliary words, etc. In the context, if there is no other clear reference, both refer to the same object.

[0105] The expression "B corresponding to A" means that B is associated with A and B can be determined according to A. However, it should be understood that determining B according to A does not mean determining B only according to A, but B can also be determined according to A and / or other information.

[0106] The expressions "first", "second", etc. are usually used to identify and distinguish objects, but they do not constitute a limit on the number of objects of the same type. Although they usually refer to a single object, this does not mean that there is only one object of the same type. For example, it may be for the purpose of effect enhancement, pressure sharing, equivalent replacement, etc.

[0107] Even if it is not emphasized in the previous text, when a certain type of embodiment is mentioned in the specific technical solution, further beneficial technical effects will be emphasized, and such further technical effects are also obtained after the inventor's creative work.

[0108] The drawings of the present invention are drawn for the convenience of describing the technical solution to be disclosed, and therefore it is difficult to objectively and comprehensively display the technical solution. The dimensions, proportions, quantities, etc. shown in the schematic diagram may not be an accurate depiction of the actual product, and the technical features not shown do not necessarily mean that the actual technical solution does not have such technical features. Therefore, the details of the drawings should not constitute an improper restriction on the true intention of the present invention.

[0109] The legislative purpose of the patent system is not to require applicants to describe in detail the technical solutions they want to disclose. Although the applicant has disclosed as many technical details as possible, it is still impossible and no one can make an infinitely detailed and infinitely in-depth description of any technical features. For some technical features / goals / means that are not introduced in much detail in the present invention document, they are not technical features / means that those skilled in the art want to implement but have not been able to implement, or technical goals that they want to achieve but have not been able to achieve, even if the existing technology may not provide a perfect solution. The applicant believes that those skilled in the art can know / combine / explore / experiment out a certain / certain implementable methods based on all existing technologies such as general technical knowledge, patents, journals, books, technical manuals, Internet technical documents, etc. in this field or other related fields.

[0110] (II) Description of technical solution

[0111] The field described in the scheme belongs to an event-driven integrated circuit system, so its sensors, interface systems, and processors are all event-driven. In the present invention, the meaning of "integrated circuit system" and "integrated circuit" are basically equivalent, and the meaning of "interface system" and "interface circuit" are basically equivalent. The "system" here has the meaning of product attributes. The term "couple" means an electrical / electrical connection between two or more components having this relationship.

[0112] refer to Figure 1 , which shows a schematic diagram of an integrated circuit layout of a certain embodiment. An event-driven integrated circuit system 1 includes an event-driven sensor 10 (hereinafter referred to as the sensor), an event-driven interface system 20 (hereinafter referred to as the interface system), and an event-driven processor 30 (hereinafter referred to as the processor).

[0113] It should be noted that the above-mentioned event-driven sensor 10, event-driven interface system 20, and event-driven processor 30 are divided according to function for the convenience of description, but this does not mean that the above components must be physically independent. They can be implemented as three separate independent components, or multiple components can be combined together to achieve the integration / combination of multiple functions with a single component, such as the sensor 10 and the interface system 20 can be combined together (especially on the same bare crystal (die, also called bare chip, bare chip)), and the interface system 20 and the processor 30 can be combined together. Perhaps some of the selection methods will cause a reduction in performance in some aspects, but the present invention does not limit its physical division form.

[0114] In order to solve the signal loss, noise interference and transmission delay caused by the USB cable and pursue more extreme low power consumption, a certain embodiment of the present invention is: at least three components of the above-mentioned event-driven sensor 10, event-driven interface system 20, and event-driven processor 30 are coupled to a single chip (chip) ( Figure 1 In some embodiments, the three components are coupled to a single chip (only a single die). In this case, the sensor 10 and the processor 30 use the same manufacturing process, such as a conventional 65nm CMOS process, but this suboptimal solution sacrifices the image quality of the sensor 10.

[0115] In a preferred embodiment, at least three components of the sensor 10, the interface system 20, and the processor 30 are connected via an interposer. Figure 1 The interposer described in the present invention includes but is not limited to: a silicon interposer and a glass interposer. The present invention does not limit the material type of the interposer.

[0116] In any place of the present invention, the term "single chip" means: including more than one bare die coupled by an adapter, or: only one bare die but no adapter is required. It should be noted that in some cases the context of the term may imply / limit the term to represent only one of the above meanings.

[0117] In certain embodiments, the event-driven sensor 10 is an event-driven 2D array sensor (e.g., an event-driven camera), and the sensor 10 generally includes one or more event-driven sensor input devices 11. The event-driven camera includes a large number of pixels, and each pixel is an event-driven input device 11. The event-driven sensor input device 11 is configured to asynchronously generate an event after detecting an event generating signal or / and a change in the event generating signal (e.g., a change in light intensity on a pixel). Each event is associated with or includes an event address, which includes / indicates an identifier of the input device 11, such as the X and Y coordinates of a pixel in the 2D array.

[0118] In certain embodiments, the event-driven sensor is a 1D, 2D, 3D or other type of sensor.

[0119] Regarding the design of event-driven 2D array sensor, at least the following prior art can be referred to (European patent, title: Photoarray for detecting time-dependent image data, published on June 27, 2018):

[0120] Prior art 4: EP1958433B1;

[0121] The present invention incorporates the above contents into the disclosure of the present invention in its entirety. Due to space limitations, the specific contents will not be described in detail herein. The embodiments of the sensor of the present invention are not limited thereto.

[0122] In certain embodiments, the sensor 10 is coupled to the input terminal 21 of the event-driven interface system 20 via the output terminal 12 of the sensor 10, and the sensor 10 outputs the event 100 in an asynchronous manner. Each event received by the interface system input terminal 21 of the event-driven interface system 20 can be processed by various pre-processing tasks to achieve a unified subsequent processing, which will achieve independence from the personality of the sensor 10. In particular, the interface system 20 may include a series of interface modules 200, wherein each interface module 200 is configured to process the incoming events 100 in a programmable manner. In this way, from the perspective of a unified event address structure and a possible event header, all events 100 processed by the processor 30 have the same format. Furthermore, in certain embodiments, the interface module 200 may be configured to include performing 1) a filtering step; or / and 2) an address manipulation step. For example, to limit the incident event rate within the processing capability of the processor 30, or / and to provide a predefined event address format. Each of the above operations can effectively reduce the energy consumption of the circuit system 1 .

[0123] For parallel signal processing, the interface system 20 includes a series / multiple parallel input terminals 21 (such as 21-1, 21-2) for receiving events 100 from the sensor 10, and also includes a series / multiple parallel output terminals 22 (such as 22-1, 22-2) coupled to the input terminals of the processor 30, and is configured to transmit the pre-processed events 101 to the processor 30 in a parallel manner. This configuration allows multiple events to be transmitted simultaneously, achieving reduced power consumption and fast event processing.

[0124] Figure 2 What is shown is a certain type of alternative embodiment. In this case, the event-driven sensor 10 is a 1D event-driven sensor, such as an event-driven mechanical pressure sensor, which is used to detect mechanical vibration. Similarly, the sensor 10, the interface system 20 and the processor 30 are assembled on a single chip 3 and coupled through an adapter board 40.

[0125] In some embodiments, the processor 10, the interface system 20 and the processor 30 are coupled to the same side of the chip 3; in another embodiment, the processor 10, the interface system 20 and the processor 30 are coupled to both sides of the chip 3. The present invention does not limit whether the above three components are assembled on the same side.

[0126] In some embodiments, the sensor 10 and the interface system 20 are in the same die; and in another embodiment, the interface system 20 and the processor 30 are in the same die.

[0127] In addition, the sensor 10 may also be a point sensor, in which case the event addresses of the point sensor are all the same address. As for the sound sensor type, in some embodiments, it may include at least two sound sensors at different physical locations to achieve stereo sound collection.

[0128] Compared with conventional circuit systems that are not event-driven or / and not on a single chip 3, the design of this event-driven system can achieve particularly low application energy consumption and can operate asynchronously, which is particularly suitable for battery-powered applications with ultra-long operating time.

[0129] The synergy of assembling multiple event-driven components on a single chip 3 enables the circuit system 1 to operate at high speed with an extremely low energy footprint. These operations can be feature detection and classification methods. For these purposes, the processor 30 is configured as an event-driven spiking artificial neural network (event-driven spiking artificial neural network, or simply event-driven spiking neural network, also commonly referred to as spiking neural network SNN in the art). SNN includes a variety of network algorithms, preferably, the above-mentioned neural network is configured as an event-driven spiking convolutional neural network (sCNN), which is particularly suitable for ultra-fast application needs, such as object recognition.

[0130] The specific implementation of the sCNN can at least refer to the prior art (PCT patent application document, title: Event-driven spiking convolutional neural network, publication date: 15, Oct, 2020):

[0131] Prior art 5: WO2020 / 207982A1;

[0132] The present invention is incorporated as part of the disclosure of the present invention by way of full text introduction. Due to space limitations, the specific content is not repeated herein. The implementation of the sCNN of the present invention is not limited thereto, and the present invention does not exclude that a heterogeneous fusion network such as SNN+ANN can also be used in a specific embodiment.

[0133] The event-driven processor 30 configured as an event-driven spiking neural network or sCNN, combined with the geometric structure of circuit integration, can further adapt to the needs of long-term, low-power application scenarios.

[0134] An integrated circuit system configured in this way can output only relevant information including the detected object, such as "[table]", "[chair]", "[** is approaching]" and other similar messages. Compared with traditional technologies, it does not need to record and upload a large amount of data information, eliminates the information transmission delay of connecting to cloud data, massive computing power requirements, and power consumption. It is very suitable for low-power, low-latency, low data storage pressure, and long-life application scenarios, such as IoT and edge computing. According to a specific embodiment of the present invention, the average power consumption of the solution adapted to 64*64 DVS under 65nm process is as low as 0.1mW, and the peak power consumption is only 1mW; the average power consumption of the solution adapted to 128*128 DVS is as low as 0.3mW, and the peak power consumption is only 3mW.

[0135] In certain embodiments, the processor 30 includes at least two (or more) processors, and in particular, each processor is configured to perform different tasks. The processor 30 is also configured to process events asynchronously.

[0136] refer to Figure 3 , which is a cross-sectional view of a chip in a certain embodiment. On the package substrate 4, for example, the adapter board 40 can be connected through a C4 bump 43, and a plurality of through holes 42 are provided in the adapter board 40. A plurality of micro bumps 41 are provided on the adapter board 40, and two bare crystals are provided on the micro bumps 41: a first bare crystal (or a first integrated circuit structure) 1-1, and a second bare crystal (or a second integrated circuit structure) 1-2. Based on the setting of the adapter board 40, the coupling of the first bare crystal 1-1 and the second bare crystal 1-2 can be achieved through some optional specific means such as micro bumps (μbumps) 41 and through holes 42.

[0137] In certain embodiments, the event-driven interface system (20) and the event-driven processor (30) are both located in the first die (1-1).

[0138] In certain embodiments, the event-driven sensor (10) and the event-driven interface system (20) are both located in the second die (1-2).

[0139] In certain embodiments, a portion of the event-driven interface system (20) and the event-driven processor (30) are both located in the first die (1-1) and another portion of the event-driven interface system (20) and the event-driven sensor (10) are both located in the second die (1-2).

[0140] The through via 42 of the present invention includes but is not limited to: through silicon via (TSV) and through glass via (TGV).

[0141] In some embodiments, the different dies are coupled via Cu-Cu technology.

[0142] The adapter plate 40 of the present invention includes but is not limited to: a silicon adapter plate or a glass adapter plate.

[0143] Figure 4 2 is a cross-sectional view of a 3D chip geometry structure in a certain embodiment. For example, the interface system 20 and the processor 30 are mounted on the first bare die 1-1, and the sensor 10 is mounted on the second bare die 1-2. This is a possibility of achieving spatial isolation between the first circuit structure 1-1 and the second circuit structure 1-2 through the adapter board 40.

[0144] For example, there may be C4 bumps 43 between the package substrate 4 and the transfer board 40. The transfer board 40 includes a plurality of through holes 42 therein.

[0145] A micro-bump 41 is included between the first bare die 1-1 and the adapter board 40. The interface system 20 and the processor 30 are assembled on the same bare die, namely the first bare die 1-1. The electrical / electrical coupling between the interface system 20 and the processor 30 can be achieved through the micro-bump 41 and / or the through hole 42. A through hole 42 is provided in the interface system 20, and the second bare die 1-2 where the sensor 10 is located is coupled to the first bare die 1-1 through the micro-bump 41, so that the second bare die 1-2 is stacked on the first bare die 1-1.

[0146] This type of 3D chip structure greatly reduces the chip's footprint, increases the speed of information transmission between different components, and reduces the overall power consumption of the system.

[0147] Particularly and advantageously, for a chip having such a structural design, the characteristic size / manufacturing process (technology node) of the second die 1-2 and the first die 1-1 can be different, for example, the process of the second die 1-2 can be greater than 65nm, while the process of the interface system and the processor is less than 65nm, such as 22 / 14 / 10 / 7 / 5nm or less. This allows a more cost-effective manufacturing process to be selected during the production and manufacturing process of the chip, and the present invention does not limit this selection and combination.

[0148] Optionally, the processor 30 and the interface system 20 may also adopt the same manufacturing process and be manufactured on the same integrated circuit structure / bare die or on different integrated circuit structures.

[0149] exist Figure 5 In a certain class of embodiments illustrated in the example, the system includes a sound / vibration sensor 10', which is connected to an event-driven amplifier 13 and is configured to amplify the sensor signal and output an event 100 indicating a change in the intensity of the recorded sound / vibration power spectrum (power spectrum), especially wherein the event is asynchronously generated for each frequency in the power spectrum. The amplifier 13 and the sound sensor 10' thus constitute an event-driven sensor 10. The amplifier 13 is connected to an interface system 20, wherein the interface system 20 is configured to process any event 100 generated by the amplifier 13 and transmit the processed event to the processor 30. In this class of embodiments, the sensor 10' and the amplifier 13 are arranged on the same chip, which can achieve the advantage of maintaining a single-chip structure.

[0150] In certain embodiments, the event-driven sensor 10 is a combination of one or more of the following types: a point sensor, a 1D sensor, and a 2D sensor.

[0151] In some embodiments, the event-driven sensor 10 is a combination of one or more of the following types: sound / vibration sensor, dynamic visual sensor.

[0152] exist Figure 6 1 is a flow chart of processing an event 100 generated by a sensor 10 in a circuit system 1. For example, a 2D sensor 10 including a series of pixels 11, pre-processing the event 100 is implemented by an interface system 20, and the pre-processed event 100 is output to a processor 30.

[0153] In the present invention, the term "upstream" refers to the side closer to the sensor, and the term "downstream" refers to the side closer to the processor, both of which are related to the order of event processing, such as Figure 6 The middle replication module 201 is located upstream of the event routing module 205 .

[0154] In the figure, each geometric figure between the sensor 10 and the processor 30 represents at least one processing step, which is performed by the interface system 20. To achieve this purpose, the interface system 20 includes a series of interface modules 200 (including but not limited to 201, 202, 203, 204, 205, 206, 207, 208, 208', etc.). The interface modules 200 can be independently programmable, and any one or more interface modules (200) can be bypassed by a programmable switch to support more network configurations. Preferably, the interface modules 200 can be specifically implemented as hardware circuits and constitute a programmable daisy chain, which is configured to asynchronously process events 100 incoming from the sensor 10 in an event-driven and asynchronous manner.

[0155] Copy module 201 (copy module): Figure 6 In a certain example, the first interface module 201 is a replication module, which is configured in the subsequent stage of the sensor 10, and includes an input terminal 201-1 coupled to the output terminal 10-1 of the sensor 10, and also includes a first output terminal 201-2a and a second output terminal 201-2b. The input terminal 201-1 of the replication module 201 is configured to receive the event 100 and perform a replication operation, that is, to replicate the received addressed event 100, and is configured to forward the replicated event 100c and its replicated event address to the first output terminal 201-2a of the replication module 201, and to transmit the received event 100 to the second output terminal 201-2b of the replication module 201 along the daisy chain.

[0156] The replicated event 100c can be fed into external processing pipelines (not shown) of different systems. The replication module 201 allows the event to be coupled into other systems before any processing is performed on it.

[0157] Fusion Module 202 (merge module): In a certain embodiment, the fusion module 202 is coupled to the replication module 201, and the first input terminal 202-1a of the fusion module 202 is coupled to the second output terminal 201-1b of the replication module 201. The fusion module 202 is also provided with a second input terminal 202-1b, which is configured to receive events 100e from other components / modules, and these components and modules may come from the present circuit system or other circuit systems (not shown), such as events output by a second sensor or a daisy chain. The coupling module 202 also includes an output terminal 202-2, which is configured to send all received events 100 or / and events 100e along the daisy chain. Thereafter, the event 100e is merged into the stream constituted by the event 100, and thus the event 100e and the event 100 are no longer distinguished and are collectively referred to as events 100. This embodiment allows more components and / or information to be integrated into the processing pipeline of the circuit system, and has a more flexible system configuration capability.

[0158] Second sampling module 203 (sub-sampling / sum pooling module): The output terminal 202-2 of the fusion module 202 is coupled to the input terminal 203-1 of the sub-sampling module 203. The sub-sampling module 203 is configured to assign a single address to the received events 100, so that the number of different event addresses can be reduced. In this way, for example, several event addresses representing several different pixels 11 of the 2D array sensor 10 can be sub-sampled into fewer pixels. The sub-sampling process is called pixel binning in some specific application scenarios.

[0159] The sub-sampling module 203 may be bypassed by a programmable switch (not shown).

[0160] Area of ​​Interest Module 204(Region Of Interest module, ROI): The output terminal 203-2 of the secondary sampling module 203 is coupled to the input terminal 204-1 of the region of interest module 204, wherein the region of interest module 204 is configured to adjust the property of at least one event address, specifically, the adjustment method can be through shifting, flipping, swapping or / and rotating the property of at least one event address, and the operation performed in the ROI module can be through the event address rewriting operation. In addition, the region of interest module 204 can be further configured to discard those events whose address property values ​​are outside the programmable address property value range. The region of interest module 204 is programmable and is configured to store the programmable address property value range, which is set for each address property. In addition, the region of interest module 204 is also configured to send the received events 100, as long as these events 100 are not discarded, together with the adjusted addresses, to the next level along the daisy chain.

[0161] In certain implementations, from a 2D image perspective, the region of interest module 204 allows cropping of images and / or other basic geometric operations at event 100 pixel coordinates.

[0162] Event Routing Module 205 (Event routing module): The input terminal 205-1 of the event routing module 205 is coupled to the output terminal 204-2 of the region of interest module 204, and receives the event 100. The event routing module 205 is configured to: optionally, associate header information to the received event 100, and output the event 100 together with its header information to the first output terminal 205-2a of the event routing module 205. Optionally, the event routing module 205 is configured to copy the event 100 including the header information and the adjusted event address, and output the copied event 100 together with its copied header information to the second output terminal 205-2b of the event routing module 205, and the second output terminal 205-2b can be coupled to other event-driven processors or other processing pipelines.

[0163] The event routing module 205 configured in this way thus adds the following capability to the circuit system 1: it can provide pre-processing information from events for any type of processor, processor or processor with any type of input format required by the program it runs.

[0164] The first output terminal 205-2a of the event routing module 205 is coupled to the processor 30, and the processed event 100 with the event address and header information is then transmitted to the processor 30 and executed for processing tasks, such as pattern or feature recognition tasks or other applications.

[0165] Optionally, the circuit system 1 may further include other interface modules 200, which may be configured to perform tasks on the events 100 received from the sensor 10 or the like, such as rate control tasks, hot pixel filtering tasks, event address rewriting tasks (see Figure 8-9 ).

[0166] Rate Control Module (rating limit module): configured to limit the rate of events to not exceed a maximum rate. For example: in particular, limiting the rate of events with the same address; when the maximum speed is exceeded, only a fraction of events are sent along the daisy chain, for example, every nth received event will not be sent along the daisy chain, n is a value determined according to the current event rate; the maximum rate can be programmably and adjustable and stored in a memory, which can belong to the rate control module or come from outside the module. The rate control module may include or be connected to a processing unit with a clock for determining the event reception rate of the module. In this way, the event rate on the daisy chain will not exceed the maximum rate, and the data rate sent to the processor 30 is also limited.

[0167] For each interface module 200, which is programmable, it is obvious that it is easy to bypass the interface module by issuing appropriate instructions to the module. For example, when coordinate flipping is not required, it can be achieved by bypassing the entire region of interest module 204, thus achieving a direct connection between the subsampling module 203 and the event routing module 205.

[0168] Figure 7 Shown is a certain embodiment of a sensor for recording vibrations and sound sensors. The sensor includes an amplifier (reference Figure 5 ) and a filter, and the sensor is configured to generate events from the sound sensor, asynchronously encode the events according to the power spectrum recorded by the sensor 100. In addition, the amplifier can be configured to shift a channel. As a combined unit, the sound sensor and the amplifier can be considered as an event-driven sensor 10 as a whole.

[0169] In the manner described above, the event-driven sensor 10 transmits the event 100 to the fusion module 202, and the event 100e is transmitted from different processing pipelines and fused into the event 100 generated by the event-driven sensor 10. Also as described in the previous example, the fused event 100 is then transmitted to the replication module 201, so the fused event 100 is replicated. The replicated event 100c is then transmitted to the same processing pipeline, which is the source of the event 100e received in the fusion module 202, or is transmitted to other processing pipelines (not shown). The effect achieved by this is to allow a large degree of freedom in designing or processing daisy chains. It is not difficult to find that in this type of embodiment, many events 100, 100e may have been sent in at the beginning of the daisy chain. The advantage of the programmable module of the interface system 20 including the daisy chain is that in the processor 30, based on the unified event format and / or event address format, the event 100 can be processed and the processor performs its intended purpose without further processing.

[0170] The cooperation between the replication module 201 and the fusion module 202 , especially in the early stage of processing the event 100 , greatly enhances the interoperability of the system.

[0171] Mapping module 206 (mapping module): between the secondary sampling module 203 responsible for pooling (which is basically equivalent to secondary sampling) and the event routing module 205 responsible for event routing of the interface module 200 (see Figure 6 ), a mapping module 206 (which itself includes the ROI module 204) can be placed therebetween, in order to implement rich event address mapping operations.

[0172] According to certain embodiments, a certain interface module is / includes a mapping module (such as the above-mentioned mapping module 206), wherein the mapping module 206 is configured to map an event address to another event address. The mapping module 206 includes one or a combination of the following contents:

[0173] 1. Region of interest module (ROI 204);

[0174] 2. Look-up table (LUT) module;

[0175] 3. Flip and / or rotation module; it is configured as the event address of the flip and / or rotation event.

[0176] Preferably, all interface modules of the present invention (including those described later) can be bypassed by programmable switches inside them.

[0177] Figure 8One class of embodiments shown is a daisy chain implemented via a programmable interface module 200 , which is incorporated into the interface system 20 .

[0178] Event address rewriting module 207 (Event address rewrite module): The event-driven sensor 10 provides a stream of events 10, which are sent to an optional event address rewrite module 207. The event address rewrite module 207 is configured to rewrite the event format to a common format for subsequent processing steps, i.e., converting the event address received from the sensor to a unified address format, thereby passing the unified event address format on the daisy chain. The event address rewrite module 207 can be programmed for a specific sensor model in order to accommodate any type of sensor 10 that provides virtually any event format. The format of the event may be related to the byte-order and event address format stored in the generated event. The unified event address format is a pre-defined data format, which enables subsequent processing to rely on the pre-defined data format, thereby omitting double check instances on the event format, achieving faster processing.

[0179] Event address filtering module 208 (Event address filter module): Once the common events and event address formatting are completed by the event address rewrite module 207, the events will be further processed by the event address filter module 208. The event address filter module 208 is configured to filter out a series of events with specifically selected event addresses. The selected event addresses can be stored, read, and written in the CAM memory (Content-addressable memory). These filters allow filtering out hot pixels or the like. So the event address filter module 208 can be a hot pixel filter module. After the event address rewrite module 207, the event address filter module 208 is the first module or part of it. It can reduce the number of events transmitted on the daisy chain at an early stage, and this processing method will reduce the energy consumption of the daisy chain. If the processor 30 also has the ability to filter addresses, the event address rewrite module 207 can be bypassed.

[0180] The filtered events 100 will then be sent to the replication module 201 and / or the fusion module 202, which can respectively provide the sensor's replication events 100c to the external system 100c and incorporate the external event 100e source. The replication module 201 and the fusion module 202 can be independently bypassed by programming.

[0181] In the technical solution including both the copy module 201 and the fusion module 202, the order of the two can have two different processing orders: copy module 201 first and then fusion module 202, or vice versa. Figure 6-7 Description of embodiments.

[0182] After the copy module 201 and / or the fusion module 202, the sub-sampling module 203 will process the input events in the manner of the above-mentioned embodiments. Placing the sub-sampling module 203 at a specific position of the daisy chain can process all events, even if the events are from external sources.

[0183] The region of interest module 204 is located in the subsequent part of the subsampling module 203 and processes all events sent by the subsampling module 203. The region of interest module 204 reduces the number of event addresses, thereby reducing the workload. The region of interest module 204 can also be configured to flip and / or rotate the X, Y coordinates of the event address.

[0184] Arranged in the subsequent part of the region of interest module 204 is an event routing module 205 , which is configured to: prepare an event, such as providing header information of the event 10 , and the event is sent to the processor 30 .

[0185] Figure 8 The daisy chain shown provides a unified approach for efficiently, quickly, and flexibly processing events from sensors 10 or other sources.

[0186] Hot pixel processing module 208' (hot pixel filter module): Fig. 9 A specific embodiment of the event address filter module 208 is a hot pixel filter module 208'. The function of the hot pixel filter module 208' is to filter events with a specific event address. For example, this allows reducing or completely removing such events on the daisy chain. Such events are removed because the pixels of the input device, such as a 2D array sensor, are compromised or damaged. The hot pixel filter module 208' includes an input terminal 208'-1 for receiving the event 100.

[0187] After receiving event 100, after the hot pixel filter enable judgment step S800, if it is judged to be disabled (No, Disabled), the hot pixel filter module 208' can be directly bypassed by programming the switch and the event 100 can be directly sent to the output end 208'-2 of the hot pixel filter module 208'. If S800 is judged to be enabled (Yes, Enabled), preferably, the preset event address list to be filtered is read from the CAM memory 208'-3. In the address comparison / matching step S802, it is verified whether the address of event 100 belongs to one of the addresses to be filtered in the list. If the received address of event 100 matches an address in the list, the event 100 will be filtered out S803 (filtered out) in the daisy chain and discarded (dropped). If there is no address matching the address of the event 100 in the above list, the event 100 will be output to the pipeline at the output terminal 208 ′- 2 of the hot pixel filtering module 208 ′.

[0188] exist Fig.10 , which shows a workflow diagram of the sub-sampling module 203 in a certain embodiment. The input event 100 is processed by evaluating its associated address, especially its address coordinates, such as X, Y, Z coordinates.

[0189] For example, the address of event 100 is Separation module 203-4 After the above coordinates (X, Y, Z) are shifted or divided (division) operation S901, the coordinates X, Y, Z are subsampled into a low-data-volume coordinate set, which effectively reduces the number of pixel coordinates. The event address processed in this way is then Address reorganization module 203- 5 The adjusted address is then sent to the next stage along with event 100 for further processing.

[0190] Specifically, the separation module 203-4 is configured to: route event 100 to a scaling register in an associated secondary sampling module according to the address value of the received event (such as X, Y, Z coordinates), the scaling register being configured to split, secondary sample, pool or / and shift the received address value, and output the address value to an address reorganization module 203-5 in the secondary sampling module, the address reorganization module 203-5 being configured to adjust the event address according to the scaled address value, and then send the event with the adjusted address along the daisy chain.

[0191] For example, by pooling the event addresses of adjacent input devices, the subsampling module can adjust the pixel resolution of the 2D array sensor, thereby reducing the number of pixels on the X and Y axes. For example, the resolution of the image processed by the processor can be changed from 256*256 to 64*64. For 1D sensors, they can also be processed by the subsampling module with similar configuration as above.

[0192] In some embodiments, the event 100 may also include a channel identifier, which is not processed separately as described above, but only loops through S900 in the resampling module 203 alone.

[0193] (III) Statement on circumvention design technology solutions

[0194] Although the present invention has been described with reference to the specific features and embodiments of the present invention, various modifications and combinations thereof may be made without departing from the present invention. Therefore, the specification and the drawings should be simply regarded as an illustration of some embodiments of the present invention as defined by the appended claims, and it is intended to cover any and all modifications, variations, combinations or equivalents falling within the scope of the present invention. Therefore, although the present invention and its advantages have been described in detail, various changes, substitutions and modifications may be made without departing from the present invention as defined by the appended claims. In addition, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, devices, methods and steps described in the specification.

[0195] It will be readily apparent to one of ordinary skill in the art from the disclosure of the present invention that currently existing or later developed processes, machines, manufactures, compositions of matter, devices, methods, or steps that perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein may be applied in accordance with the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, devices, methods, or steps.

[0196] In order to achieve better technical effects or meet the needs of certain applications, those skilled in the art may make further improvements to the technical solution based on the present invention. However, even if such partial improvement / design is creative and / or progressive, as long as the technical features covered by the claims of the present invention are utilized, the technical solution should also fall within the protection scope of the present invention according to the "comprehensive coverage principle".

[0197] Some of the technical features mentioned in the attached claims may have alternative technical features, or the order of certain technical processes and the order of material organization may be reorganized. After knowing the present invention, ordinary technicians in this field can easily think of these replacement means, or change the order of technical processes and the order of material organization, and then adopt basically the same means to solve basically the same technical problems and achieve basically the same technical effects. Therefore, even if the above-mentioned means and / or order are clearly defined in the claims, these modifications, changes, and substitutions should fall within the scope of protection of the claims based on the "principle of equivalents".

[0198] For the clear numerical limitation in the claims, such as 1.5V voltage, it should not be understood as a strict limitation that the voltage value must be 1.5V. Generally, those skilled in the art will understand that 1.2V and 1.8V voltages can also be applied to a specific implementation. These design solutions that circumvent the details without departing from the concept of the present invention also fall within the scope of protection of the claims.

[0199] In conjunction with the various method steps and units described in the embodiments disclosed herein, they can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection claimed by the present invention.

[0200] In addition, any module, component, or device that executes instructions exemplified herein may include or otherwise access a non-transitory computer / processor readable storage medium or medium for storing information, such as computer / processor readable instructions, data structures, program modules, and / or other data. Any such non-transitory computer / processor storage medium may be part of the device or accessible or connectable to the device. Any application or module described herein may be implemented using computer / processor readable / executable instructions, which may be stored or otherwise maintained by such non-transitory computer / processor readable storage medium.

[0201] Table 1: List of reference numbers

[0202]

Claims

1. An integrated circuit comprising an event-driven sensor (10), an event-driven interface system (20) and an event-driven processor (30), characterized in that: The event-driven sensor (10), the event-driven interface system (20), and the event-driven processor (30) are coupled to a single chip (3); The event-driven sensor (10) is configured to: asynchronously generate and asynchronously output an event (100) after an input device (11) of the event-driven sensor (10) detects an event generation signal or / and a change in the event generation signal, wherein the event (100) includes or is associated with an event address indicating the input device (11), and the output end of the event-driven sensor (10) is coupled to the input end of the event-driven interface system (20); The event-driven interface system (20) is configured to: asynchronously receive the event (100) and pre-process the received event (100), the output end of the event-driven interface system (20) being coupled to the input end of the event-driven processor (30); The event-driven processor (30) is configured to: receive the event (101) pre-processed by the event-driven interface system (20), and process the pre-processed event (101) in an asynchronous manner; The event-driven sensor (10), the event-driven interface system (20), and the event-driven processor (30) are coupled to a single chip (3) via an adapter board (40).

2. The integrated circuit according to claim 1, wherein: The event-driven interface system (20) and the event-driven processor (30) are both located in the first die (1-1); or, the event-driven sensor (10) and the event-driven interface system (20) are both located in the second die (1-2); or, a portion of the event-driven interface system (20) and the event-driven processor (30) are both located in the first die (1-1) and another portion of the event-driven interface system (20) and the event-driven sensor (10) are both located in the second die (1-2).

3. The integrated circuit according to claim 1, wherein: The event-driven interface system (20) and the event-driven processor (30) are both located on a first bare die (1-1), and a second bare die (1-2) where the event-driven sensor (10) is located is stacked on the first bare die (1-1) where the event-driven interface system (20) and the event-driven processor (30) are located.

4. The integrated circuit according to claim 1, wherein: The adapter plate (40) is a silicon adapter plate or a glass adapter plate.

5. The integrated circuit according to claim 1, wherein: The event-driven sensor (10), the event-driven interface system (20), and the event-driven processor (30) are packaged on a single chip (3) using 2.5D or 3D packaging technology.

6. The integrated circuit of claim 1, wherein: The event-driven sensor (10) is one or more of the following types: a point sensor, a 1D sensor, a 2D sensor, and a 3D sensor.

7. The integrated circuit of claim 1, wherein: The event-driven sensor (10) belongs to one or more combinations of the following types: sound / vibration sensor, dynamic visual sensor.

8. The integrated circuit of claim 1, wherein: The event-driven processor (30) is configured with a spiking neural network.

9. The integrated circuit of claim 1, wherein: The event-driven processor (30) is configured with a spiking convolutional neural network.

10. The integrated circuit according to claim 2 or 3, characterized in that: The first bare die and the second bare die are manufactured using different processes.

11. The integrated circuit according to any one of claims 1 to 9, characterized in that: The event-driven interface system (20) comprises at least one interface module (200), wherein the interface module (200) is configured in a programmable daisy chain form and asynchronously processes events (100) received from the event-driven sensor (10).

12. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises a replication module (201), which is configured to: receive an event (100) and perform a replication operation to obtain a replication event (100c), wherein the event (100) comes from the event-driven sensor (10) or from other interface modules (200) of the event-driven interface system (20), and send the replication event (100c) to an external processing pipeline, and send the event (100) along the daisy chain.

13. The integrated circuit of claim 11, wherein: The at least one interface module (200) includes a fusion module (202), which is configured to: receive events (100, 100e) from at least two different places, wherein the event (100) comes from other interface modules (200) of the event-driven interface system (20) or the event-driven sensor (10); the event (100e) also comes from the integrated circuit or other integrated circuit components / modules or other event-driven sensors, and send part or all of the received events (100, 100e) to subsequent interface modules (200) along the programmable daisy chain.

14. The integrated circuit of claim 12, wherein: The other interface module (200) of the integrated circuit is a fusion module (202).

15. The integrated circuit of claim 13, wherein: The other interface module (200) of the integrated circuit is a copy module (201).

16. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises a sub-sampling module (203) configured to assign a single address to a number of received events (100).

17. The integrated circuit of claim 16, wherein: The secondary sampling module (203) includes a separation module (203-4) configured to: route the event (100) to a scaling register in the associated secondary sampling module (203) according to the address value of the received event (100); the scaling register is configured to: split, secondary sample, pool or / and shift the received address value, and output the address value to an address reassembly module (203-5) in the secondary sampling module (203), and the address reassembly module (203-5) is configured to: adjust the event address according to the scaled address value, and then send the event (100) with the adjusted address along the programmable daisy chain.

18. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises a region of interest module (204) configured to: Adjusting the attributes of at least one event address, wherein the adjustment method includes one or more of the following methods: shifting, flipping, exchanging and / or rotating the attributes of at least one event address; or / and Events (100) with address attribute values ​​outside a programmable address attribute value range are discarded, and events (100) that are not discarded are sent along the programmable daisy chain.

19. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises an event routing module (205) configured to: Receive an event (100), add header information to the received event (100), and send the event (100) together with the header information of the event (100) to the event-driven processor (30) and / or other event-driven processors or other processing pipelines.

20. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises a rate control module configured to: When the maximum rate is exceeded, only a portion of the events (100) are sent along the programmable daisy chain to limit the rate of events to not exceed the maximum rate.

21. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises a mapping module (206) configured to: Maps an event address to another event address.

22. The integrated circuit of claim 21, wherein: The mapping module (206) includes one or a combination of the following contents: An area of ​​interest module, a lookup table module, and a flipping or / and rotating module; wherein the flipping or / and rotating module is configured to flip or / and rotate the event address of the event (100).

23. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises an event address rewriting module (207) configured to convert a received event address into a unified address format, thereby transmitting the unified event address format on the programmable daisy chain.

24. The integrated circuit of claim 11, wherein: The at least one interface module (200) comprises an event address filtering module (208) configured to: Filters out a set of events (100) with a specific selected event address.

25. The integrated circuit of claim 24, wherein: The event address filtering module (208) is specifically a hot pixel filtering module (208'), which is configured as follows: Events with specific event addresses are filtered (100), and a preset event address list to be filtered is stored in a CAM memory (208'-3).

26. The integrated circuit of claim 11, wherein: Any one or more interface modules (200) of the event-driven interface system (20) can be bypassed by a programmable switch.

27. An event-driven interface system (20) coupled to an event-driven sensor (10) and an event-driven processor (30) to form an integrated circuit, wherein the event-driven sensor (10) generates and asynchronously outputs an event (100), wherein the event (100) includes or is associated with an event address of an input device (11) on the event-driven sensor (10) that generates the event; characterized in that: The event-driven interface system (20) comprises at least one interface module (200), wherein the interface module (200) is configured in a programmable daisy chain form and asynchronously processes events (100) received from the sensor (10); The at least one interface module (200) comprises one or more of the following: a replication module (201), a fusion module (202), a secondary sampling module (203), an area of ​​interest module (204) and an event routing module (205); wherein: The replication module (201) is configured to: receive an event (100) and perform a replication operation to obtain a replication event (100c), wherein the event (100) comes from the event-driven sensor (10) or from other interface modules (200) of the event-driven interface system (20), and send the replication event (100c) to an external processing pipeline, and send the event (100) along the daisy chain; The fusion module (202) is configured to: receive events (100, 100e) from at least two different places, wherein the event (100) comes from other interface modules (200) of the event-driven interface system (20) or the event-driven sensor (10); the event (100e) also comes from the integrated circuit or other integrated circuit components / modules or other event-driven sensors, and send part or all of the received events (100, 100e) to subsequent interface modules (200) along the programmable daisy chain; The secondary sampling module (203) is configured to: assign a single address to a plurality of received events (100); The region of interest module (204) is configured to: Adjusting the attributes of at least one event address, wherein the adjustment method includes one or more of the following methods: shifting, flipping, exchanging and / or rotating the attributes of at least one event address; or / and discarding events (100) whose address attribute values ​​are outside a programmable address attribute value range, and sending events (100) that are not discarded along the programmable daisy chain; The event routing module (205) is configured to: Receiving an event (100), adding header information to the received event (100), and sending the event (100) together with the header information of the event (100) to the event-driven processor (30) and / or other event-driven processors or other processing pipelines; The event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are coupled to a single chip (3) via an adapter board (40).

28. The event-driven interface system (20) according to claim 27, characterized in that: Along the event transmission direction of the programmable daisy chain, the at least one interface module (200) has the following interface module coupling sequence: a replication module (201), a fusion module (202), a subsampling module (203), a region of interest module (204), and an event routing module (205); or A fusion module (202), a replication module (201), a secondary sampling module (203), a region of interest module (204) and an event routing module (205).

29. The event-driven interface system (20) according to claim 28, characterized in that: For the replication module (201), the event (100) comes from another interface module (200) of the event-driven interface system (20), specifically the fusion module (202); or / and For the fusion module (202), the event (100) comes from another interface module (200) of the event-driven interface system (20), specifically the replication module (201).

30. The event-driven interface system (20) of claim 28, characterized in that: The upstream of the interface module coupling sequence also includes: an event address rewriting module (207) or / and an event address filtering module (208); wherein the event address rewriting module (207) is configured to: convert the received event address into a unified address format, thereby transmitting the unified event address format on the programmable daisy chain; The event address filtering module (208) is configured to filter out a series of events (100) having a specifically selected event address.

31. The event-driven interface system (20) of claim 30, characterized in that: The event (100) is first processed by the event address rewriting module (207), and then processed by the event address filtering module (208).

32. The event-driven interface system (20) of claim 30, characterized in that: The event address filtering module (208) is specifically a hot pixel filtering module (208'), which is configured as follows: Events with specific event addresses are filtered (100), and a preset event address list to be filtered is stored in a CAM memory (208'-3).

33. The event-driven interface system (20) of claim 27, characterized in that: The at least one interface module (200) further comprises a mapping module (206), wherein the mapping module (206) comprises one or a combination of the following contents: an area of ​​interest module, a lookup table module, a flipping and / or rotation module; wherein the flipping and / or rotation module is configured to flip and / or rotate the event address of the event (100).

34. The event-driven interface system (20) of claim 27, characterized in that: The at least one interface module (200) further comprises a rate control module configured to: when a maximum rate is exceeded, only send part of the event (100) along the programmable daisy chain to limit the rate of the event to not exceed the maximum rate.

35. The event-driven interface system (20) according to any one of claims 26 to 34, characterized in that: Any one or more interface modules (200) of the event-driven interface system (20) can be bypassed by a programmable switch.

Images