Event-driven integrated circuit with interface system - Patents.com
The integration of event-driven sensors, interfaces, and processors into a single chip with a programmable daisy chain addresses signal loss and noise interference, achieving efficient, low-power event processing for event-driven systems.
Patent Information
- Application Number
- JP2023552014
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-04-19
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2041-04-19
AI Technical Summary
Existing event-driven systems face challenges in efficiently transmitting events between sensors and processors due to signal loss, noise interference, and high power consumption, particularly when using USB cables, which also increase system latency and cost.
An integrated circuit design that combines event-driven sensors, interfaces, and processors into a single chip using 2.5D or 3D packaging, employing an interposer and a programmable daisy chain of interface modules for efficient, flexible, and low-power event processing.
This design reduces signal loss and noise interference, lowers power consumption, and minimizes chip footprint while enabling high-speed, low-latency event processing, suitable for applications requiring low power and efficient data transfer.
Smart Images

Figure 0007588246000001 
Figure 0007588246000002 
Figure 0007588246000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an event-driven integrated circuit, and in particular to a low-power integrated circuit with an interface module for asynchronously processing events. [Background technology]
[0002] Event-driven sensors already exist in the prior art. One type of event-driven sensor is an event-driven camera, which is made up of a pixel array containing pixels. In response to changes in the brightness of a pixel, an event-driven camera generates an event, which includes an identifier that indicates the change, such as -1 for darker and +1 for brighter. Such cameras are called dynamic vision sensors (DVS). In addition, event-driven sensors such as one-dimensional sensors and sound sensors are known.
[0003] DVS is an event-driven sensor, and events generated are asynchronous. Traditional clock-based cameras have to read every frame or line of every pixel, which is no match for DVS. DVS only records the changes, providing ultra-fast image processing while maintaining a low rate.
[0004] However, the subsequent event processing steps require a processing architecture that is fundamentally different from clock and synchronization operating processors based on traditional Von-Neumann architectures.
[0005] Designing a versatile processing structure for an event-driven system is an extremely challenging task, especially when considering the interconnections between various preprocessing and processing components. Compared to clock-driven systems, event-driven systems need to process flows depending on different events. In particular, when exchanging transfer data between different components, an AD-HOC handshake mechanism is involved. Such a handshake mechanism includes the required exchange data request, confirmation of the request, and subsequent data exchange.
[0006] Components of an event-driven system, especially sensors, can have different specifications considering pixel array size, different hot-pixels / noises, event addressing, etc. When assembling various components into a system, it becomes a laborious and expensive task to adapt them to the processing pipeline of the system. The term "processing pipeline" specifically refers to the wiring, i.e. interconnections between different components, and also refers to the data processing of components and the transfer of data between different components. The term "processing pipeline" also refers to the specific way how various output ports or various first components of the system are connected to various input ports or various second components of the system.
[0007] The most basic design concept of an event-driven system is to aim for extremely low power consumption to accommodate edge calculations. In order to achieve the ultimate in low power consumption, those skilled in the art have repeatedly tried and tested various approaches.
[0008] In the prior art, IBM's brain-inspired chip TrueNorth was first introduced in Science magazine on August 8, 2014, with 5.4 billion transistors, 4096 synaptic cores, 1 million programmable spiking neurons, and 2.56 million configurable synapses. The chip is an asynchronous-synchronous hybrid chip constructed using an event-driven design. That is, the routing, scheduler, and controller adopt quasi-delay insensitive clockless asynchronous design, while the neurons are traditional clock synchronous circuit means, and the clock is generated by the asynchronous controller, with a global clock frequency of 1 kHz. Calculated with a video input of 30 frames / s and 400*240 pixels, the power consumption of the chip is 63 mW. For more details, please see below.
[0009] Public document 1: “A million spiking-neuron integrated circuit with a scalable communication network and interface” Paul A. Merolla, John V. Arthur et al., Volume: 345, Issue: 6197, SCIENCE, Publication date: August 8, 2014.
[0010] At the CVPR conference held on July 21-26, 2017, a gesture recognition system based on the IBM TrueNorth chip was unveiled. Referring to Figures 1 and 4 of the article (or Figure 11 of the present invention, for details of the figure, please refer to the original text and this invention will not overlap), the TrueNorth processor placed on the NSIe development board receives events output from the DVS128 via USB2.0. In other words, the connection between the DVS and the processor is made via a USB cable. (For details, see below.)
[0011] Public literature 2: “A Low Power, Fully Event-Based Gesture Recognition System,” Arnon Amir, Brian Taba et al., 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21-26, July 2017.
[0012] In November 2017, Intel unveiled its self-developed brain-type chip Loihi, and the public speech materials revealed that the chip consumes 74mW of power. Advances in Neuromorphic Computing published on Intel's official website points out that Loihi hardware is available for INRC members to use in research in fields that require direct access to hardware, such as robotics. These systems include a USB form factor codenamed "Kapoho Bay." In addition to providing a USB interface to Loihi, Kapoho Bay also provides an event-driven hardware interface to iniVation's DAVIS 240C DVS silicon retina camera. In other words, in the case of Loihi, it is similarly connected to the DVS via USB (see the network link for details, the contents of the link can also be obtained from the Internet Archive. https: / / newsroom.intel.com / news / intel-announces-neuromorphic-computing-research-collaborators / #gs.vj5gmb, December 6, 2018).
[0013] On August 1, 2019, Nature magazine reported in a cover article on the brain-type chip Tianjic developed by Tsinghua University. Its power consumption is 400mW at a typical voltage of 0.9V (950mW at 1.2V). The team subsequently disclosed other technical details of Tianjic in other articles. The article pointed out that a single-chip PCB is equipped with an Altera Cyclone 4 FPGA and communication interfaces USB and SPI. Referring to Figure 2b in the article, the camera is connected to the FPGA via USB and finally to the Tianjic. In other words, in the disclosed technology, the vision sensor and the processor are also connected via a USB cable. The article can be found below.
[0014] Public document 3: “A hybrid and scalable brain-inspired robotic platform”, Zhe Zou, Rong Zhao et al., Scientific Reports, October 23, 2020.
[0015] The above is an example where the world's top event-driven system designers are seeking the ultimate in low power consumption, and the energy efficiency ratio is greatly improved compared to the power consumption of about 100-200W of traditional CPUs based on the von Neumann architecture. To connect the sensor and the AI processor, all of the above prior art technologies realize their connection via USB cable / interface or other similar interface technologies, and these world's top event-driven designers are not aware of the drawbacks of this.
[0016] However, according to the inventor's own findings, in the above prior art, the USB cable (or other cable-based technology) requires a connection with a certain length, so the system may suffer from signal loss and noise interference due to coupling to the cable. Also, because it is a cable, each handshake between devices, such as the communication required before and after data transfer, consumes more energy and slows down the processing speed of the system, which has a negative impact on the performance of the brain-type chip. In the prior art of this field, these top designers do not recognize the negative impact of the factors, and believe that the proposed technical means have made all efforts to aim for extremely low power consumption and that the solution meets the requirements of various technical indicators, and do not suggest or imply that measures should be taken to eliminate the above negative effects of the above negative factors. Therefore, without a clear revelation of other technical means in this field, these world's top designers lack the awareness / motivation to further improve the above technical means.
[0017] Furthermore, in the research / several solutions to solve the above problems, the inventor further discovered that in order to obtain better quality image information for the DVS, those skilled in the art need to select a special semiconductor process for manufacturing the DVS, such as a CIS-CMOS process image sensor. However, in the case of an AI processor (sCNN processor described later), if a conventional CMOS process is used, such a conventional CMOS process is not suitable for manufacturing high-quality image sensors (the imaging effect is not ideal). In addition, if the CIS-CMOS process is used for everything to manufacture the sensor and the AI processor, especially on the same chip / die, the AI processor will occupy a lot of chip area, increasing the cost of the chip, which will lose business competitiveness in light of the development trend of seeking even smaller chip area. Therefore, how to eliminate signal loss and noise interference, and preferably further reduce the chip area and manufacturing cost, is an important issue to be solved for the industrialization / commercialization of brain-type chips.
[0018] Since it is completely different from the traditional Von Neumann architecture, an event transmission process is involved between event generation and event processing, and this process can be regarded as an interface, and the design of this interface is a completely different technical challenge from the past. How to design this interface so that events can be transmitted efficiently, flexibly, and with low power consumption is also one of the comprehensive challenges faced by those skilled in the art. Summary of the Invention [Problem to be solved by the invention]
[0019] The present invention is proposed to solve one or a combination of one or more of the above technical problems, and the technical means of the present invention can solve or alleviate one or a combination of one or more of the above technical problems.
[0020] Unless the source is clearly stated, the technologies described in the above background art may belong to undisclosed technologies in whole or in part, that is, the applicant does not consider the technologies described in the background art to belong to the prior art as defined in the Patent Law, unless it can be proved by substantial evidence. At the same time, the technical means and technical features disclosed in the above background art are disclosed together with the disclosure of the patent document of the present invention.
[0021] In order to solve or mitigate one or a combination of the above-mentioned technical problems, the technical solution according to the present invention is realized in the following manner. [Means for solving the problem]
[0022] The integrated circuit includes an event-driven sensor (10), an event-driven interface system (20), and an event-driven processor (30), the event-driven sensor (10), the event-driven interface system (20), and the event-driven processor (30) being combined on a single chip (3).
[0023] In one embodiment, the event-driven sensor (10) is configured to asynchronously generate and output an event (100) when an input device (11) of the event-driven sensor (10) detects an event generating signal and / or a change in the event generating signal, the event (100) including or associated with an event address pointing to the input device (11), and an output of the event-driven sensor (10) is coupled to an input of the event-driven interface system (20); the event-driven interface system (20) is configured to asynchronously receive the events (100), pre-process the received events (100), and an output of the event-driven interface system (20) is coupled to an input of the event-driven processor (30); the event-driven processor (30) is configured to receive pre-processed events (101) from the event-driven interface system (20) and asynchronously process the received events (101); The event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are coupled to a single chip (3) via an interposer (40).
[0024] In one embodiment, the event-driven interface system (20) and the event-driven processor (30) are both located on a first die (1-1), or the event-driven sensor (10) and the event-driven interface system (20) are both located on a second die (1-2), or a part of the event-driven interface system (20) and the event-driven processor (30) are both located on a first die (1-1) and another part of the event-driven interface system (20) and the event-driven sensor (10) are both located on a second die (1-2).
[0025] In one embodiment, the event-driven interface system (20) and the event-driven processor (30) are both located on a first die (1-1) and a second die (1-2) on which the event-driven sensor (10) is located is stacked on the first die (1-1) on which the event-driven interface system (20) and the event-driven processor (30) are located.
[0026] In one embodiment, the interposer (40) is a silicon interposer or a glass interposer.
[0027] In one embodiment, the event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are implemented on a single chip (3) using 2.5D or 3D implementation technologies.
[0028] In one embodiment, the event-driven sensor (10) is a point sensor, a 1D sensor, a 2D sensor, or a combination of more of these.
[0029] In one embodiment, the event-driven sensor (10) is one or more combinations of audio / vibration sensors, dynamic visual sensors.
[0030] In one embodiment, a spiking neural network is disposed in the event-driven processor (30).
[0031] In one embodiment, a spiking convolutional neural network is disposed in the event-driven processor (30).
[0032] In one embodiment, the first die and the second die are manufactured by different processes.
[0033] In one embodiment, the event-driven interface system (20) includes at least one interface module (200) configured in a programmable daisy-chain configuration to asynchronously process events (100) received from the event-driven sensor (10).
[0034] In one embodiment, the at least one interface module (200) includes a duplication module (201) configured to receive an event (100), perform a duplication operation to obtain a duplicated event (100c), the event (100) being from the event-driven sensor (10) or another interface module (200) (which in one embodiment is a fusion module) of the event-driven interface system (20), send the duplicated event (100c) to an external processing pipeline, and send the event (100) along the daisy chain.
[0035] In one embodiment, the at least one interface module (200) receives events (100, 100e) from at least two different locations, the events (100) being from other interface modules (200) of the event-driven interface system (20) (which in one embodiment is a duplicate module) or the event-driven sensor (10), and the events (100e) being from the integrated circuit or other integrated circuit components / modules or other event-driven sensors, and includes a fusion module (202) configured to transmit some or all of the received events (100, 100e) to a subsequent interface module (200) along the programmable daisy chain.
[0036] In one embodiment, the at least one interface module (200) includes a sub-sampling module (203) configured to assign a single address to a plurality of received events (100).
[0037] In one embodiment, a separation module (203-4) included in the sub-sampling module (203) is configured to route a received event (100) to a scaling register in the associated sub-sampling module (203) according to an address value of the received event (100), the scaling register being configured to divide, sub-sample, pool and / or shift the received address value and output the address value to an address reconstruction module (203-5) in the sub-sampling module (203), the address reconstruction module (203-5) being configured to adjust an event address according to the scaled address value and then transmit the address-adjusted event (100) along the programmable daisy chain.
[0038] In one embodiment, the at least one interface module (200) adjusts a property of the at least one event address, the adjustment method comprising one or more of shifting, flipping, swapping, and / or rotating the property of the at least one event address; and / or and a region of interest module (204) configured to discard events (100) with address property values outside a range of programmable address property values and to transmit events (100) that are not discarded along the programmable daisy chain.
[0039] In one embodiment, the at least one interface module (200) includes an event routing module (205) configured to receive an event (100), add header information to the received event (100), and send the event (100) together with the header information of the event (100) to the event-driven processor (30) and / or other event-driven processors or other processing pipelines.
[0040] In one embodiment, the at least one interface module (200) includes a rate control module configured to transmit only a portion of the events (100) along the programmable daisy chain when a maximum speed is exceeded, to limit the event rate to not exceed the maximum rate.
[0041] In one embodiment, the at least one interface module (200) includes a mapping module (206) configured to map one event address to another event address.
[0042] In one embodiment, the mapping module (206) includes one or a combination of a region of interest module, a lookup table module, and a flip and / or rotate module, where the flip and / or rotate module is configured to flip and / or rotate the event addresses of the events (100).
[0043] In one embodiment, the at least one interface module (200) includes an event address rewriting module (207) configured to convert received event addresses into a common address format, thereby transmitting the common event address format on the programmable daisy chain.
[0044] In one embodiment, the at least one interface module (200) includes an event address filter module (208) configured to filter out a set of events (100) having a particular selected event address.
[0045] In one embodiment, the event address filter module (208), specifically a hot pixel filter module (208'), is configured to filter events (100) having a specific event address and store a list of predefined event addresses waiting to be filtered via a CAM memory (208'-3).
[0046] In one embodiment, any one or more of the interface modules (200) of the event-driven interface system (20) can be bypassed by a programmable switch.
[0047] The present invention further provides an event-driven interface system (20) coupled to an event-driven sensor (10) and an event-driven processor (30) to form an integrated circuit, the event-driven sensor (10) generating and asynchronously outputting events (100), the events (100) including or associated with an event address pointing to an input device (11) on the event-driven sensor (10) that generated the event, the event-driven interface system (20) including at least one interface module (200) configured in a programmable daisy-chain format and asynchronously processing the events (100) received from the sensor (10).
[0048] In one embodiment, the at least one interface module (200) includes one or more of a duplication module (201), a fusion module (202), a subsampling module (203), a region of interest module (204), and a routing module (205); the replication module (201) is configured to receive an event (100) and perform a replication operation to obtain a replicated event (100c), the event (100) being from the event-driven sensor (10) or another interface module (200) of the event-driven interface system (20), and to transmit the replicated event (100c) to an external processing pipeline and to transmit the event (100) along the daisy chain; the fusion module (202) is configured to receive events (100, 100e) from at least two different sources, the events (100) being from other interface modules (200) of the event-driven interface system (20) or the event-driven sensor (10), the events (100e) being from the integrated circuit or other integrated circuit components / modules or other event-driven sensors, and to transmit some or all of the received events (100, 100e) along the programmable daisy chain to a subsequent interface module (200); the sub-sampling module (203) is configured to assign a single address to a plurality of received events (100); the region of interest module (204) is configured to adjust at least one event address property in at least one or more ways of shifting, flipping, swapping, and / or rotating, and / or discarding events (100) whose address property values are outside a programmable address property value range, and to transmit any events (100) that are not discarded along the programmable daisy chain; The event routing module (205) is configured to receive an event (100), add header information to the received event (100), and send the event (100) together with the header information of the event (100) to the event-driven processor (30) and / or other event-driven processors or other processing pipelines.
[0049] In one embodiment, the coupling order of the at least one interface module (200) along the event transmission direction of the programmable daisy chain is: a duplication module (201), a fusion module (202), a subsampling module (203), a region of interest module (204) and an event routing module (205), or a fusion module (202), a duplication module (201), a subsampling module (203), a region of interest module (204) and an event routing module (205).
[0050] In one embodiment, for the replication module (201), the event (100) comes from another interface module (200) of the event-driven interface system (20), specifically the fusion module (202), and / or for the fusion module (202), the event (100) comes from another interface module (200) of the event-driven interface system (20), specifically the replication module (201).
[0051] In one embodiment, the upstream of the interface module coupling order further includes an event address rewriting module (207) and / or an event address filter module (208), where the event address rewriting module (207) is configured to convert a received event address into a common address format, thereby transmitting the common event address format in the programmable daisy chain; Here, the event address filter module (208) is configured to filter out the set of events (100) that have a particular selected event address.
[0052] In one embodiment, the event (100) is first processed by the event address rewriting module (207) and then by the event address filter module (208).
[0053] In one embodiment, the event address filter module (208) is specifically a hot pixel filter module (208'), It is configured to filter events (100) having a specific event address and store, via a CAM memory (208'-3), a list of event addresses waiting to be filtered.
[0054] In one embodiment, the at least one interface module (200) further comprises a mapping module (206), the mapping module (206) comprising one or a combination of a region of interest module, a lookup table module, and a flip and / or rotate module, where the flip and / or rotate module is configured to flip and / or rotate an event address of the event (100).
[0055] In one embodiment, the at least one interface module (200) includes a rate control module configured to transmit only a portion of the events (100) along the programmable daisy chain above a maximum rate to limit events from exceeding a maximum rate.
[0056] In one embodiment, any one or more of the interface modules (200) of the event-driven interface system (20) can be bypassed by a programmable switch.
[0057] In one embodiment, the event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are coupled to a single chip (3) via an interposer (40) or fabricated on the same die. Effect of the Invention
[0058] Compared with the prior art, the beneficial effects of the technical means of the present invention include, but are not limited to: 1. To provide an integrated solution of sensors, interfaces and event-driven processors, eliminating signal loss and noise interference, while also further reducing chip area and manufacturing costs, and the technology can achieve sub-milliwatt power consumption. 2. Provide an event-driven interface that can send events efficiently, flexibly, and with low power consumption, and provide event pre-processing functions for the processor to process events efficiently and conveniently.
[0059] The technical means, technical features and technical methods disclosed above may not be completely identical or consistent with the technical means, technical features and technical methods described in the following description of the embodiment. However, these new technical means disclosed in this section are also part of the many technical means disclosed in the present invention document, and these new technical features and technical means disclosed in this section will disclose more technical means by reasonably combining with the technical features and technical means described in the following description of the embodiment, and will be a useful supplement to the description of the embodiment of the embodiment. At the same time, some details of the drawings in the specification may not be clearly described in the specification, but if a person skilled in the art can infer the technical meaning based on other related contents or drawings of the present invention, the general technical knowledge in this field, and other prior art (conferences, journal articles, etc.), the technical means, technical features and technical methods not clearly described in the text of the said section also belong to the technical content disclosed by the present invention, and the corresponding new technical means can be obtained by combining and using them as described above. A technical means in which all technical features disclosed in any position of the present invention are combined is used to generalize the technical means, amend the patent document, and support the disclosure of the technical means. [Brief description of the drawings]
[0060] [Figure 1] FIG. 1 is a diagram showing an event-driven circuit system according to an embodiment of the present invention. [Diagram 2] FIG. 13 is a diagram showing a circuit system according to another embodiment of the present invention. [Diagram 3] FIG. 2 is a cross-sectional view of a chip according to an embodiment of the present invention. [Figure 4] FIG. 2 shows a cross-section of a 3D chip geometry according to one embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram showing a circuit system including a sound / vibration sensor according to an embodiment of the present invention. [Figure 6] 13 is a flowchart of a process for an event generated by a sensor. [Figure 7] 13 is an event processing flowchart in which a sound sensor records a vibration signal. [Figure 8] FIG. 2 shows a daisy chain of interfaces. [Figure 9] FIG. 1 illustrates a hot pixel filter module capable of filtering events having a selected event address. [Figure 10] FIG. 2 illustrates a sub-sampling module. [Figure 11] A gesture recognition system based on IBM TrueNorth in the prior art. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0061] (1) Statement regarding the method of describing technical means In order to clearly explain and fully understand the technical means disclosed in the present invention, the following provisions are made in any patent document related to the present invention.
[0062] The embodiments described in the partial content, even if described in the same drawing and in the same area, are not merely descriptions of a specific embodiment, but are selective descriptions of potential embodiments having a specific technical feature. The embodiments disclosed in the present invention are all rational combinations of the specific technical features described below (unless such combinations are logically contradictory or meaningless). Therefore, a specific technical feature is not essential in a specific embodiment.
[0063] Although the embodiments of the method and product may describe specific technical features individually, the present invention usually implies that another corresponding embodiment also has the corresponding technical features and has an apparatus / step that corresponds to / matches the technical features, but it is just not explicitly described in words. For example, the embodiments of the method imply specific steps / instructions / functions performed / realized by the devices / modules / components in the embodiments of the product, and the embodiments of the product imply devices / modules / components that perform certain steps / instructions / functions by implementing the embodiments of the method. These implied technical features are also used to generalize the technical means, amend the patent document, and support the disclosure of the technical means.
[0064] The English words / phrases in parentheses of non-English terms in any position of the present invention are intended to further supplement the meaning of the term and to supplement the explanation in the common English expression in the field. In case of non-English terms cannot be correctly understood and interpreted, or are in contradiction, the English meaning will help to understand the meaning here, and the English meaning will take precedence if necessary. In addition, the above interpretations are adopted for all the above non-English terms in any position, but the meaning interpreted in one particular place shall be the same as the same term appearing in other places.
[0065] In the specification, certain terms may be given a particular interpretation, and the same terms in any document related to the present invention, such as the claims, may be interpreted in accordance with that interpretation.
[0066] The terms "module," "component," "part," "member," "part," "system" (if any), and the like described in any document related to the present invention refer to a product or part of a product implemented by only hardware, only software, or a combination of software and hardware. Unless otherwise clearly indicated by the context, the present invention does not imply that the above terms refer to only hardware or only software embodiments.
[0067] The Chinese punctuation mark "," is usually used to indicate the juxtaposition of words or phrases, to separate similar parallel matters, or to separate a sequence of characters and a sentence, and does not usually clearly define a logical "or / and" expression, and the logic it expresses usually needs to be analyzed according to the contextual linguistic environment.
[0068] As a rule in all patent documents of any language of the present invention, a phrase like "A, B, C" normally means A or B or C, excluding what should be a logical "and" depending on the context. A phrase like "A, B or C" means A or B or C, and a phrase like "A, B and C" means A, B and C. A phrase like "A, B or C" means A or B or C, and a phrase like "A, B and C" means A, B and C.
[0069] The descriptions of multiple measures, such as "A and / or B," "A or / and B," "A and / or B," and "A or / and B," all include three parallel technical measures: (1) A, (2) A and B, and (3) B.
[0070] In non-mathematical formulas, " / " usually denotes logical disjunction ("or").
[0071] In all patent documents of the present invention, the description such as "the first input port of the ~ module" and "the first input port of the ~ module" have the same meaning and refer to the same object. Except when "~ module" itself is insufficient or ambiguous to distinguish a specific object, it shall be interpreted in the most reasonable manner according to the context.
[0072] The terms "may" (may be, may, might, choice, except when the context indicates "ability") in any position of this document are ways of describing a preferred embodiment, and imply that there are other types of potential reasonable alternatives. When technical terms such as "approximately", "approximately", and "close to" are described in any position of this invention, it means that the technical means allows for implementation with errors, provided that it does not affect the solution of the technical problem. In other words, after measuring the strict actual parameters, the obtained data does not need to strictly match the general mathematical definition (because there is no physical reality that perfectly matches the mathematical definition), and such terms should not make the scope limited / expressed by the technical means unclear due to vagueness and unclearness, but should actually be the criterion for judging whether it can solve the technical problem within the limited scope.
[0073] In any position of the present invention, there may be cases where the terms are not completely identical to refer to the same object. Usually, this is due to differences between abbreviations and universals, and the insertion of functional words such as particles. Unless otherwise indicated by the context, all refer to the same object.
[0074] The expression "B corresponding to A" means that B is related to A, and B can be determined according to A. However, determining B according to A does not mean that B is determined only according to A, but also means that B may be determined according to A and / or other information.
[0075] The expressions "first", "second" and the like are typically used to identify and distinguish between objects, but do not constitute a limitation on the number of objects of the same kind. As such, they typically refer to just one object, but do not mean that there is only one object of this kind, and may be for purposes of, for example, improved effectiveness, pressure sharing and equivalent replacement.
[0076] Even if not emphasized above, when an example is mentioned in a specific embodiment, further beneficial technical effects may be emphasized, and the further technical effects are also obtained by the creative labor of the inventors.
[0077] The reference drawings of the present invention are intended to facilitate the explanation of the disclosed technical means, and therefore may not objectively and comprehensively display the technical means. The size, ratio, quantity, etc. shown in the schematic diagram may not accurately depict the actual product, and technical features not shown do not mean that the actual technical means does not have the technical features, and therefore the details of the drawings do not constitute an inappropriate limitation on the true intent of the present invention.
[0078] The legislative intent of the patent system is not to require applicants to provide detailed descriptions of the technical means they intend to disclose, regardless of their importance. Although applicants disclose as many technical details as possible, it is not possible for anyone to provide an infinitely detailed and in-depth description of any technical feature. The technical features / objectives / means not detailed in the present invention document are not technical features / means that a person skilled in the art has attempted to implement but failed to implement, or technical goals that a person skilled in the art has attempted to achieve but failed to achieve, but that even the prior art may not yet be able to provide a complete solution. The applicant believes that a person skilled in the art can obtain / combine / discover / experiment a certain / several implementation methods based on all prior art, such as general technical knowledge in this field or other related fields, patents, magazines, books, technical manuals, internet technical documents, etc.
[0079] (II) Description of technical means The field described in the solution belongs to event-driven integrated circuit systems, so its sensors, interfaces and processors are all event-driven. In the present invention, "integrated circuit system" and "integrated circuit" have essentially the same meaning, "interface" and "interface circuit" have essentially the same meaning, and "system" here has the meaning of product property. The term "couple" refers to an electrical / electrical connection between two or more parts having the relevant relationship.
[0080] 1 is a diagram showing the layout of an integrated circuit according to an embodiment. The event-driven integrated circuit system 1 includes an event-driven sensor 10 (hereinafter referred to as the sensor), an event-driven interface 20 (hereinafter referred to as the interface), and an event-driven processor 30 (hereinafter referred to as the processor).
[0081] It should be noted that the above-mentioned event-driven sensor 10, event-driven interface 20, and event-driven processor 30 are all divided by function for convenience of explanation, but this does not necessarily mean that the above-mentioned components are physically independent. They may be implemented as three separate components, or multiple components may be combined to realize integration / combination of multiple functions as a single component. For example, the sensor 10 and the interface 20 may be combined (especially on the same die), or the interface 20 and the processor 30 may be combined. Depending on the selection method, there may be a performance degradation in some aspects, but the present invention does not limit the physical division form.
[0082] In order to solve the signal loss, noise interference and transfer delay caused by the USB cable and to seek for a more extreme low power consumption, one embodiment of the present invention combines at least three components, the event-driven sensor 10, the event-driven interface 20 and the event-driven processor 30, into a single chip (not shown in FIG. 1). In one embodiment, the above three components are combined into a single chip (when there is only one die). In this case, the sensor 10 and the processor 30 use the same manufacturing process, such as the conventional 65 nm CMOS technology, but this suboptimal solution sacrifices the image quality of the sensor 10.
[0083] In a preferred embodiment, at least three components, the sensor 10, the interface 20, and the processor 30, are coupled to a single chip (in the case of multiple dies) through an interposer (not shown in FIG. 1). The interposer according to the present invention includes, but is not limited to, a silicon interposer and a glass interposer. The present invention does not limit the material type of the interposer.
[0084] The term "single chip" in any location of the present invention may mean multiple dies coupled via an interposer, or just one die without the need for an interposer. Note that in some cases, the context may imply or limit the term to only one of the above meanings.
[0085] In one embodiment, the event-driven sensor 10 is an event-driven 2D array sensor (such as an event-driven camera), and the sensor 10 typically includes one or more event-driven sensor input devices 11. An event-driven camera includes a large number of pixels, each pixel being an event-driven input device 11. The event-driven sensor input device 11 is configured to asynchronously generate events upon detecting an event-generating signal and / or a change in the event-generating signal (such as a change in light intensity on a pixel), where each event is associated with or includes an event address that includes / points to an identifier of the input device 11, such as the X and Y coordinates of a pixel in the 2D array.
[0086] In one embodiment, the event-driven sensor is a 1D, 2D, 3D or other type of sensor.
[0087] With regard to the design means of the event-driven 2D array sensor, at least the following prior art can be referred to (European patent, invention title: “Photoarray for detection time-dependent image data”, published on June 27, 2018): Publicly known document 4: EP1958433B1
[0088] The present invention is incorporated herein by reference in its entirety as a part of the disclosure of the present invention. Due to space limitations, the specific contents will not be described repeatedly, but the embodiment of the sensor of the present invention is not limited thereto.
[0089] In one embodiment, the sensor 10 is coupled to the input port 21 of the event-driven interface 20 via the output port 12 of the sensor 10, and the sensor 10 asynchronously outputs the event 100. Each event received by the interface input port 21 of the event-driven interface 20 is tasked through various pre-processing processes, and then a common subsequent process is performed, which can be realized regardless of the individuality of the sensor 10. In particular, the interface 20 may include a series of interface modules 200, each of which is configured to process the received event 100 in a programmable manner. In this way, all the events 100 processed by the processor 30 have the same format in terms of a common event address structure and a possible event header. Furthermore, in one embodiment, the interface module 200 may be configured to include performing 1) a filtering step or / and 2) an address manipulation step. For example, limiting the incident event rate within the processing capacity of the processor 30 or / and providing a predefined event address format. Each of the above steps can efficiently reduce the energy consumption of the circuit system 1.
[0090] For parallel signal processing, the interface 20 includes a series / multiple parallel inputs 21 (e.g., 21-1, 21-2) for receiving events 100 from the sensors 10, and a series / multiple parallel outputs 22 (e.g., 22-1, 22-2), which are coupled to the inputs of the processor 30 and configured to transfer pre-processed events 101 in parallel to the processor 30. With this arrangement, multiple events can be transferred simultaneously, reducing power consumption and achieving high-speed event processing.
[0091] 2 shows an alternative embodiment, where the event driven sensor 10 is an ID event driven sensor, such as an event driven mechanical pressure sensor used to detect mechanical vibrations. Similarly, the sensor 10, interface 20 and processor 30 are mounted on a single chip 3 and coupled via an interposer 40.
[0092] In one embodiment, the sensor 10, the interface 20 and the processor 30 are coupled to the same side of the chip 3, and in another embodiment, the sensor 10, the interface 20 and the processor 30 are coupled to both sides of the chip 3. The present invention is not limited to whether the above three components are mounted on the same side.
[0093] In one embodiment, the sensor 10 and the interface 20 are on the same die, and in another embodiment, the interface 20 and the processor 30 are on the same die.
[0094] Also, the sensor 10 may be a point sensor, in which case the event addresses of the point sensor are the same. Regarding the type of sound sensor, in one embodiment, at least two sound sensors with different physical locations may be included to achieve stereo sound effect collection.
[0095] In contrast to conventional circuit systems that are non-event driven or / and not on a single chip, the design methodology of such an event driven system allows for very low application energy consumption and asynchronous operation, particularly suited to battery powered and ultra-long runtime usage requirements.
[0096] Due to the synergistic effect of mounting multiple event-driven components on a single chip 3, the circuit system 1 enables high-speed processing operations with a very low energy footprint. These operations may be feature detection and classification methods. For these purposes, the processor 30 is configured as an event-driven spiking artificial neural network (also called spiking neural network SNN in the art). SNN includes various network algorithms, but preferably, the above-mentioned neural network is configured as an event-driven convolutional neural network (sCNN), which is particularly suitable for ultra-high-speed application requirements such as object recognition.
[0097] Specific embodiments of the sCNN may refer to at least the following prior art: PCT patent application, title: "Event-driven spiking convolutional neural network", publication date: October 15, 2020. Publicly known document 5: WO2020 / 207982A1
[0098] The present invention incorporates the contents of the present invention in the form of a full citation. Due to space limitations, the specific contents will not be described repeatedly, but the embodiment of the sCNN of the present invention is not limited thereto. In addition, the present invention does not exclude that in a specific embodiment, it may be a heterogeneous fusion network such as SNN+ANN.
[0099] The event-driven processor 30 configured as an event-driven spiking neural network or sCNN, coupled with circuit integration geometry, can be further adapted to the needs of long duration and low power usage requirements.
[0100] The integrated circuit system configured in this manner can output only information including information related to the detected object, such as "[table]", "[chair]", and "[** is approaching]". Compared to the conventional technology, there is no need to record and upload a large amount of data information, and the information transfer delay for connecting cloud data, the demand for a large amount of computing power, and power consumption are avoided, and it is very suitable for use requirements of low power consumption, low latency, low data storage pressure, and battery life, such as in the IoT and edge computing fields. According to a specific embodiment of the present invention, the average power consumption of a technical means adopting a 64*64 DVS in a 65 nm process is as low as 0.1 mW, and the peak power consumption is only 1 mW, and the average power consumption of a means adopting a 128*128 DVS is as low as 0.3 mW, and the peak power consumption is only 3 mW.
[0101] In one embodiment, processor 30 includes at least two (or more) processors, each configured to perform a different task. Processor 30 is also configured to process events asynchronously.
[0102] 3 is a cross-sectional view of a chip in one embodiment. An interposer 40 can be connected to a mounting substrate 4 using, for example, C4 bumps 43, and the interposer 40 is provided with several through-hole vias 42. The interposer 40 is also provided with several micro-bumps 41, and two dies, namely a first die (or referred to as a first integrated circuit structure) 1-1 and a second die (or referred to as a second integrated circuit structure) 1-2, are provided on the micro-bumps 41. The attachment by the interposer 40 can realize the coupling between the first die 1-1 and the second die 1-2 by any specific means such as micro-bumps 41 and through-hole vias 42.
[0103] In one embodiment, the event-driven interface system (20) and the event-driven processor (30) are both located on the first die (1-1).
[0104] In one embodiment, the event driven sensor (10) and the event driven interface system (20) are both located on the second die (1-2).
[0105] In one embodiment, a portion of the event-driven interface system (20) and the event-driven processor (30) are both located on a first die (1-1), and another portion of the event-driven interface system (20) and the event-driven sensor (10) are both located on a second die (1-2).
[0106] Through-hole vias 42 according to the present invention include, but are not limited to, through silicon vias (TSVs) and through glass vias (TGVs).
[0107] In one embodiment, the different dies are bonded together by Cu-Cu technology.
[0108] The interposer 40 according to the present invention includes, but is not limited to, a silicon interposer or a glass interposer.
[0109] 4 is a cross-sectional view of a 3D chip geometry in one embodiment. By way of example, the interface 20 and the processor 30 are mounted on a first die 1-1, and the sensor 10 is mounted on a second die 1-2, showing the possibility of spatial separation between the first circuit structure 1-1 and the second circuit structure 1-2 via an interposer 40.
[0110] For example, there may be C4 bumps 43 between the mounting substrate 4 and the interposer 40. The interposer 40 includes several through-hole vias 42.
[0111] Microbumps 41 are included between the first die 1-1 and the interposer 40, the interface 20 and the processor 30 are mounted on the same die, i.e., the first die 1-1, and electrical / electronic coupling is achieved between the interface 20 and the processor 30 via the bumps 41 and / or through-hole vias 42. The interface 20 is provided with through-hole vias 42, and the second die 1-2 on which the sensor 10 is arranged and the first die 1-1 are coupled to each other via the microbumps 41, and the second die 1-2 is stacked on the first die 1-1.
[0112] Chips arranged in this way with a 3D structure significantly reduce the chip's footprint, improve the speed of information transfer between different components, and reduce the power consumption of the entire system.
[0113] Particularly preferably, in a chip having such a structure design, the critical dimension (CD) / technology node of the second die 1-2 and the first die 1-1 may be different, for example, the process of the second die 1-2 is larger than 65 nm, and the process of the interface and the processor is smaller than 65 nm, for example, 22 / 14 / 10 / 7 / 5 nm or less. This allows the selection of a manufacturing process with better cost performance in the chip manufacturing process, and the present invention does not limit the selection and combination method.
[0114] The processor 30 and the interface 20 may also use any of the same manufacturing processes and be fabricated on the same integrated circuit structure / die or on different integrated circuit structures.
[0115] In one embodiment shown in Fig. 5, the system comprises a sound / vibration sensor 10' coupled to an event-driven amplifier 13 configured to amplify the sensor signal and output events 100 indicative of intensity changes in the recorded sound / vibration power spectrum, in particular events generated asynchronously for each frequency in the power spectrum. Thus, the amplifier 13 and the sound sensor 10' constitute an event-driven sensor 10. The amplifier 13 is connected to an interface 20 configured to process any events 100 generated by the amplifier 13 and transmit the processed events to a processor 30. In this embodiment, the sensor 10' and the amplifier 13 are mounted on the same chip, which has the advantage of maintaining a single chip structure.
[0116] In one embodiment, the event-driven sensor 10 is a point sensor, a 1D sensor, a 2D sensor, or a combination of more of these.
[0117] In one embodiment, the event-driven sensor 10 is a combination of one or more of the following: a sound / vibration sensor, a dynamic vision sensor.
[0118] 6 shows, as an example, a flow chart for processing events 100 generated by a sensor 10 in a circuit system 1. For example, in the case of a 2D sensor 10 including a series of pixels 11, the pre-processing events 100 are realized via the interface 20 and output to the processor 30 after being pre-processed.
[0119] In the present invention, "upstream" refers to the side closer to the sensor, and "downstream" refers to the side closer to the processor, both of which relate to the order of event processing. As shown in FIG. 6, the replication module 201 is located upstream of the event routing module 205.
[0120] In the figure, every geometrical figure between the sensor 10 and the processor 30 represents at least one processing step performed by the interface 20. To achieve this goal, the interface 20 includes a series of interface modules 200 (including but not limited to 201, 202, 203, 204, 205, 206, 207, 208, 208', etc.). The interface modules 200 may be independently programmable, and any one or more of the interface modules (200) may be bypassed by a programmable switch to support more network configurations. Preferably, the interface modules 200 may be specially implemented as hardware circuits and form a programmable daisy chain configured to asynchronously process events 100 received from the sensor 10 in an event-driven and asynchronous manner.
[0121] Copy module 201 6, the first interface module 201 is a duplication module, which is disposed after the sensor 10 and includes an input 201-1 coupled to the output 10-1 of the sensor 10, and further includes a first output 201-2a and a second output 201-2b. The input 201-1 of the duplication module 201 is configured to receive the event 100, perform a duplication operation, i.e., duplicate the event 100 with the received address, and forward the duplicated event 100c and its duplicated event address to the first output 201-2a of the duplication module 201, and send the received event 100 along the daisy chain to the second output 201-2b of the duplication module 201.
[0122] The replicated event 100c may be fed into an external processing pipeline (not shown) in a different system. The replication module 201 allows the event to be bound to other systems before any processing is done on the event.
[0123] Merge module 202 In one embodiment, the fusion module 202 is coupled to the replica module 201, and a first input 202-1a of the fusion module 202 is coupled to a second output 201-1b of the replica module 201. The fusion module 202 is further provided with a second input 202-1b configured to receive an event 100e (e.g., an event output by a second sensor or a daisy chain) from another component / module from this circuit system or another circuit system (not shown). The fusion module 202 further includes an output 202-2 configured to transmit all received events 100 and / or events 100e along the daisy chain. After that, the events 100e are fused into a stream formed by the events 100, so that from now on, the events 100e and the events 100 are not distinguished from each other and are collectively referred to as the events 100. This embodiment allows more components and / or information to be integrated into the processing pipeline of the circuit system, and has a more flexible system configuration capability.
[0124] Sub-sampling module 203 (sub-sampling / sum pooling module) The output 202-2 of the fusion module 202 is coupled to an input 203-1 of the sub-sampling module 203. The sub-sampling module 203 is configured to assign a single address to several received events 100, thus reducing the number of different event addresses. In this way, for example, several event addresses representing several different pixels 11 of the 2D array sensor 10 may actually be sub-sampled to fewer pixels. In certain usage requirements, the sub-sampling process is called binning.
[0125] The sub-sampling module 203 may be bypassed by a programmable switch (not shown).
[0126] Region of Interest module 204 (ROI) The output 203-2 of the sub-sampling module 203 is coupled to an input 204-1 of the region of interest module 204, which is configured to adjust at least one property of the event address, in particular, the adjustment method is to shift, flip, swap or / and rotate the property of the at least one event address, and the operation performed in the ROI module may be an operation by rewriting the event address. Furthermore, the region of interest module 204 may be further configured to discard events whose address property value is outside a range of programmable address property values. The region of interest module 204 is programmable and configured to store a range of the above-mentioned programmable address property values, which is set for each address property. Furthermore, the region of interest module 204 is configured to send the received events 100 (unless these events 100 are discarded) with the adjusted address to the next stage along the daisy chain.
[0127] In one embodiment, from a 2D image perspective, the region of interest module 204 allows for manipulation of cropped images at event 100 pixel coordinates or / and other basic geometric operations.
[0128] Event routing module 205 An input 205-1 of the event routing module 205 is coupled to an output 204-2 of the region of interest module 204 to receive the event 100. The event routing module 205 may be configured to associate header information with the received event 100 and output the event 100 with the header information to a first output 205-2a of the event routing module 205. The event routing module 205 may be configured to replicate the event 100 including the header information and the adjusted event address and output the replicated event 100 with the replicated header information to a second output 205-2b of the event routing module 205, which may be coupled to another event-driven processor or other processing pipeline.
[0129] The event routing module 205 configured in this manner thereby adds to the circuit system 1 the capability to input format and provide pre-processing information from events to any type of processor, any type required by the processor or program executed by the processor.
[0130] A first output 205-2a of the event routing module 205 is coupled to the processor 30, and the processed event 100 with the event address and header information is sent to the processor 30 for processing tasks such as mode or feature identification tasks or other applications.
[0131] The circuit system 1 may further include other interface modules 200 configured to perform tasks such as rate control tasks, hot pixel filtering tasks, event address rewriting tasks, etc., on events 100 received from a sensor 10 or similar (see Figures 8-9).
[0132] Rate limit module The rate of events is configured to be limited so that it does not exceed a maximum rate. For example, the rate is limited, especially with the same event address, and if the maximum rate is exceeded, only a fraction of the events are sent along the daisy chain. For example, every nth received event is not sent along the daisy chain, where n is a value determined according to the current event rate. The maximum rate may be programmably and tunably stored in a memory, which may belong to the rate control module or may be external to the module. The rate control module may include or be connected to a processing unit having a clock for determining the event reception rate of the module. In this way, the event rate on the daisy chain does not exceed the maximum rate, and at the same time the rate of data fed into the processor 30 is also limited.
[0133] Each interface module 200 is programmable, and it goes without saying that the interface modules may be easily bypassed by issuing appropriate commands. For example, if coordinate inversion is not required, this can be achieved by bypassing the entire region of interest module 204, and a direct connection between the sub-sampling module 203 and the event routing module 205 is achieved.
[0134] Figure 7 shows an example when the sensor records a vibrating sound sensor. The sensor includes an amplifier (see Figure 5) and a filter, which is configured to generate events from the sound sensor and asynchronously encode the events 100 according to the power spectrum recorded by the sensor. Furthermore, the amplifier may be configured to shift a channel. As a combined unit, the sound sensor and the amplifier may be considered as a whole event-driven sensor 10.
[0135] As described above, the event-driven sensor 10 transmits the event 100 to the fusion module 202, and the event 100e is transmitted from the different processing pipelines and fused to the event 100 generated by the event-driven sensor 10. Also, as in the above example, the fused event 100 is sent to the duplication module 201, so that the fused event 100 is duplicated. The duplicated event 100c is then sent to the same processing pipeline as the event 100e (which is the source from which the event 100e is received by the fusion module 202) or to another processing pipeline (not shown). This allows for a great deal of freedom in the design and processing of the daisy chain. In this example, it is well understood that many events 100, 100e may be fed into the early stages of the daisy chain. The advantage of the programmable module including the daisy chain interface 20 is that the event 100 can be processed in the processor 30 based on a common event format or / and event address format, and the processor performs a given purpose and does not require further processing.
[0136] The cooperation between the replication module 201 and the fusion module 202 greatly improves the interoperability of the system, especially in the early stages of processing the event 100 .
[0137] Mapping module 206 The mapping module 206 (which itself includes the ROI module 204) may be placed between the subsampling module 203 responsible for pooling (essentially equivalent to subsampling) of the interface module 200 and the event routing module 205 responsible for routing events (see FIG. 6), with the purpose of realizing rich event address mapping operations.
[0138] According to one embodiment, an interface module is / includes a mapping module (such as mapping module 206 described above), which is configured to map one event address to another event address. Mapping module 206 includes one or a combination of the following: 1. Region of Interest module (ROI 204), 2. Look-up table (LUT) module, 3. A flip and / or rotate module configured to flip and / or rotate an event address of an event.
[0139] Preferably, all interface modules of the present invention (including those described below) may be bypassed by their internal programmable switches.
[0140] FIG. 8 illustrates a daisy chain implemented by a programmable interface module 200 that is incorporated into the interface 20 in one embodiment.
[0141] Event address rewrite module 207 The event-driven sensors 10 provide a stream of events 10, which are either sent to the event address rewriting module 207. The event address rewriting module 207 is configured to rewrite the event format into a shared / common format for subsequent processing steps, i.e. converting the event addresses received from the sensors into a common address format, which is then transmitted on the daisy chain. The event address rewriting module 207 may be programmable for a particular sensor model to accommodate any type of sensor 10 provided by practically any event format. The event format may relate to the byte-order and the format of the event address stored in the generated event. The common event address format is a predefined data format, which eliminates the instance of double checking the event format, as subsequent processing is according to the predefined data format, resulting in faster processing.
[0142] Event address filter module 208 Once the normal event and event address formatting is completed by the event address rewrite module 207, the event is further processed by the event address filter module 208. The event address filter module 208 is configured to filter out a series of events with a specific selected event address. The selected event address may be stored, read and written in a content-addressable memory (CAM) memory. These filters filter out hot pixels and the like. Thus, the event address filter module 208 may be a hot pixel filter module. After the event address rewrite module 207, the event address filter module 208, as the first module or part of it, can reduce the number of events sent on the daisy chain at an early stage, and this processing method reduces the energy consumption of the daisy chain. If the processor 30 also has the function of filtering addresses, the event address rewrite module 207 may be bypassed.
[0143] The filtered events 100 may be sent to a duplication module 201 or / and a fusion module 202, which may respectively provide sensor duplication events 100c to external systems and address external sources of events 100e. The duplication module 201 and the fusion module 202 may be independently bypassed by programming.
[0144] In the technical solution including the duplication module 201 and the fusion module 202 at the same time, the order of both may be two different processing orders, i.e., duplication module 201 first and fusion module 202 second, or vice versa (for details, see the description of the embodiment in the above Figs. 6-7).
[0145] After the duplication module 201 and / or the fusion module 202, the sub-sampling module 203 processes the input events in the manner of the above-mentioned embodiment. By placing the sub-sampling module 203 at a specific position in the daisy chain, it is possible to process all events (even if the events are from outside).
[0146] The region of interest module 204 is a follower of the sub-sampling module 203 and processes all events sent from the sub-sampling module 203. The region of interest module 204 reduces the number of event addresses, thereby reducing the workload. The region of interest module 204 may also be configured to flip and / or rotate the X, Y coordinates of the event addresses.
[0147] Following the region of interest module 204 is an event routing module 205 that is configured to prepare the event, such as providing header information for the event 10 to be sent to the processor 30 .
[0148] The daisy chain shown in FIG. 8 provides a common processing method for efficient, fast and flexible processing of events from sensors 10 or other sources.
[0149] Hot pixel filter module 208´ 9 is a hot pixel filter module 208' as a specific embodiment of the event address filter module 208. The function of the hot pixel filter module 208' is to filter events with a particular event address, for example to reduce or completely remove such events on a daisy chain. Such events are removed because they compromise or corrupt the pixels of an input device, such as a 2D array sensor. The hot pixel filter module 208' includes an input 208'-1 for receiving the event 100.
[0150] After receiving the event 100, if it is determined to be disabled (No, Disabled) through the hot pixel filtering enable determination step S800, the hot pixel filter module 208' is directly bypassed via a programming switch, and the event 100 is sent directly to the output terminal 208'-2 of the hot pixel filter module 208' (S801). If it is determined to be enabled (Yes, Enabled) in S800, a list of predetermined event addresses waiting to be filtered is preferably read from the CAM memory 208'-3. In the address comparison / matching step S802, it is verified whether the address of the event 100 belongs to the addresses waiting to be filtered in the list. If the address of the received event 100 matches an address in the list, the event 100 is filtered out in the daisy chain (S803) and dropped. If there is no address in the list that matches the address of the event 100, the event 100 is output to the pipeline at the output terminal 208'-2 of the hot pixel filter module 208'.
[0151] 10 shows a flow chart of the operation of the sub-sampling module 203 in one embodiment. An incoming event 100 is processed by evaluating the associated address, in particular the address coordinates such as the X, Y, and Z coordinates.
[0152] For example, the address of event 100 is separated into three different addresses by separation module 203-4. After a shift or division operation S901 of the above coordinates (X, Y, Z), the coordinates X, Y, Z are subsampled into a smaller set of coordinates, which significantly reduces the number of pixel coordinates. The event addresses thus processed are then fused in address reconstruction module 203-5, and the adjusted address is sent together with event 100 to the next stage for further processing.
[0153] In particular, the separation module 203-4 is configured to route the event 100 to a scaling register in an associated sub-sampling module according to the address value (e.g., X, Y, Z coordinates) of the received event, which is configured to divide, sub-sample, pool and / or shift the received address value and output the address value to an address reconstruction module 203-5 in the sub-sampling module, which is configured to adjust the event address according to the scaled address value and then transmit the address-adjusted event along the daisy chain.
[0154] For example, the subsampling module can adjust the pixel resolution of a 2D array sensor to reduce the number of pixels on the X and Y axes by pooling event addresses of adjacent input devices. For example, the resolution of the image processed by the processor can be from 256*256 to 64*64. For a 1D sensor, it can be processed by a subsampling module configured similarly to the above.
[0155] In one embodiment, the event 100 may further include a channel identifier that is not separately processed as described above, but simply loops through the sub-sampling module 203 by itself (S900).
[0156] (3) Statement on Avoidance Design Measures Although the present invention has been described with reference to specific features and examples thereof, various modifications and combinations can be made without departing from the present invention. For this reason, the specification and drawings should be considered merely as an introduction to some examples of the present invention as defined by the appended claims, and are intended to cover any modifications, variations, combinations, or equivalents within the scope of the present invention. Thus, although the present invention and its advantages have been described in detail, various improvements, substitutions, and changes can be made without departing from the present invention as defined by the appended claims. Furthermore, the scope of the present application is not intended to be limited to the particular embodiments of the process, equipment, manufacture, composition of matter, apparatus, method, and steps described in the specification.
[0157] Those skilled in the art will readily appreciate that the present disclosure enables the present invention to adapt and carry out existing or later developed processes, equipment, manufacture, compositions of matter, devices, methods, or steps that achieve substantially the same function or substantially the same result as the corresponding embodiments described herein, and therefore, the appended claims are intended to include within their scope such processes, equipment, manufacture, compositions of matter, devices, methods, or steps.
[0158] In order to achieve better technical effects or according to the needs of some applications, those skilled in the art may further modify the technical means based on the present invention. However, even if the partial improvement / design has creativity and / or inventive step, as long as the technical features included in the claims of the present invention are utilized, according to the "principle of complete coverage", the technical means should also fall within the protection scope of the present invention.
[0159] Some technical features described in the accompanying claims may have alternative technical features, or some technical flow sequences or material composition sequences may be rearranged. After knowing the present invention, a person skilled in the art can easily come up with some of the replacement means, or change the technical flow sequences or material composition sequences, and then use almost the same means to solve almost the same technical problems and achieve almost the same technical effects. Therefore, even if the claims explicitly limit the above means and / or sequences, all of the modifications, changes and substitutions should fall within the scope of protection of the claims according to the principle of equivalence.
[0160] In the case of a voltage that is numerically and explicitly limited in the claims, for example a voltage of 1.5V, it should not be understood as a strict limitation that the voltage value must be 1.5V. In general, those skilled in the art will understand that voltages of 1.2V and 1.8V are also applicable to a specific implementation. These design measures that are avoided through details without departing from the concept of the present invention are also included in the protection scope of the claims.
[0161] Each method step or unit described in the embodiments disclosed herein can be combined and realized by electronic hardware, computer software, or a combination of both, and in order to clearly describe the compatibility of hardware and software, the above description has generally described the steps and compositions of each embodiment according to their functions. Whether these functions are ultimately implemented in the form of hardware or software is determined by the specific application or design provisions of the technical means. Those skilled in the art may use different methods to realize the described functions for each specific application, but such realization should not be considered to be outside the scope of the present invention's protection.
[0162] Additionally, any module, component, or device that executes instructions illustrated herein may include or otherwise have access to a non-transitory computer / processor readable storage medium or media for storing information such as, for example, computer / processor readable instructions, data structures, program modules, and / or other data. Any such non-transitory computer / processor storage medium may be part of the device or accessible or connectable to the device. Any application or module described herein may be implemented using computer / processor readable / executable instructions, which may be stored or otherwise maintained by such non-transitory computer / processor readable storage medium. [Explanation of symbols]
[0163] 1 Event-driven integrated circuit system 1-1 First Die 1-2 Second Die 3. Chip 4 Mounting board 10 Event-driven sensors 10-1 Event-driven sensor output terminal 10' Sound / Vibration Sensor 11 Event-driven input devices / pixels 12 Sensor output terminal 13 Event-driven amplifier 20 Event-driven interface system 21 Interface system input 21-1 Interface system input terminal 1 21-2 Interface system input terminal 2 22 Interface system output 22-1 Interface system output terminal 1 22-2 Interface system output terminal 2 30 Event-Driven Processors 40 Interposer 41 μ bump 42 Through hole vias 43 C4 Bump 100, 100e Event 200 Interface Module 201 Replication Module 201-1 Input terminal of the duplication module 201-2a First output of the replication module 201-2b Second output of the replication module 202 Fusion Module 202-1a First input of fusion module 202-1b Second input of fusion module 202-2 Output end of fusion module 203 Subsampling Module 203-1 Subsampling module input terminal 203-2 Subsampling module output terminal 204 Region of Interest Module 204-1 Input terminal of region of interest module 204-2 Output terminal of region of interest module 205 Event Routing Module 205-1 Event routing module input terminal 205-2a First output of the event routing module 205-2b Second output of the event routing module 100c Replication Event 101 Pre-processing event 206 Mapping Module 207 Event Address Rewrite Module 208 Event Address Filter Module 208' Hot Pixel Filter Module 208'-1 Hot pixel filter module input terminal 208'-2 Hot pixel filter module output terminal 208'-3 CAM memory S800 Hot pixel filter enable judgment step S801 Direct feed event step S802 Address match step S803 Filtering removal step 203-4 Separation module 203-5 Address reconstruction module S900 Loop-Through Step S901 Shift or split operation
Claims
1. An integrated circuit including an event-driven sensor (10), an event-driven interface system (20) and an event-driven processor (30), the event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are combined on a single chip (3); The event-driven interface system (20) includes at least one interface module (200), the interface module (200) configured in a programmable daisy chain format and configured for asynchronously processing events (100) received from the event-driven sensor (10).
2. the event-driven sensor (10) is configured to asynchronously generate and output an event (100) when an input device (11) of the event-driven sensor (10) detects an event generating signal and / or a change in the event generating signal, the event (100) including or associated with an event address that points to the input device (11), and an output of the event-driven sensor (10) is coupled to an input of the event-driven interface system (20); the event-driven interface system (20) is configured to asynchronously receive the events (100), pre-process the received events (100), and an output of the event-driven interface system (20) is coupled to an input of the event-driven processor (30); The event-driven processor (30) is configured to receive pre-processed events (101) from the event-driven interface system (20) and asynchronously process the received events (101); 2. The integrated circuit of claim 1, wherein the event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are coupled to a single chip (3) via an interposer (40).
3. The integrated circuit of claim 2, wherein the event-driven interface system (20) and the event-driven processor (30) are both located on a first die (1-1), or the event-driven sensor (10) and the event-driven interface system (20) are both located on a second die (1-2), or a part of the event-driven interface system (20) and the event-driven processor (30) are both located on a first die (1-1), and another part of the event-driven interface system (20) and the event-driven sensor (10) are both located on a second die (1-2).
4. The integrated circuit of claim 2, wherein the event-driven interface system (20) and the event-driven processor (30) are both located on a first die (1-1) and a second die (1-2) on which the event-driven sensor (10) is located is stacked on the first die (1-1) on which the event-driven interface system (20) and the event-driven processor (30) are located.
5. The integrated circuit of claim 2, wherein the interposer (40) is a silicon interposer or a glass interposer.
6. 3. The integrated circuit of claim 2, wherein the event-driven sensor (10), the event-driven interface system (20) and the event-driven processor (30) are implemented on a single chip (3) using 2.5D or 3D implementation technology.
7. 3. The integrated circuit of claim 2, wherein the event-driven sensor (10) is any one or a combination of a point sensor, a 1D sensor, a 2D sensor, and a 3D sensor.
8. 5. The integrated circuit of claim 3, wherein the first die and the second die are manufactured by different processes.
9. The integrated circuit of any one of claims 1 to 8, characterized in that the at least one interface module (200) includes a duplication module (201) configured to receive an event (100) and perform a duplication operation to obtain a duplicated event (100c), the event (100) being from the event-driven sensor (10) or another interface module (200) of the event-driven interface system (20), transmit the duplicated event (100c) to an external processing pipeline, and transmit the event (100) along the programmable daisy chain.
10. The integrated circuit of any one of claims 1 to 8, characterized in that the at least one interface module (200) receives events (100, 100e) from at least two different sources, the events (100) being from other interface modules (200) of the event-driven interface system (20) or the event-driven sensor (10), the events (100e) being from the integrated circuit or other integrated circuit components / modules or other event-driven sensors, and further comprising a fusion module (202) configured to transmit some or all of the received events (100, 100e) along the programmable daisy chain to a subsequent interface module (200).
11. 9. The integrated circuit of claim 1, wherein a separation module (203-4) included in a sub-sampling module (203) is configured to route a received event (100) to a scaling register in the associated sub-sampling module (203) according to an address value of the received event (100), the scaling register being configured to divide, sub-sample, pool and / or shift the received address value and output the address value to an address reconstruction module (203-5) in the sub-sampling module (203), the address reconstruction module (203-5) being configured to adjust an event address according to the scaled address value and then transmit the address-adjusted event (100) along the programmable daisy chain.
12. The integrated circuit of any one of claims 1 to 8, characterized in that the at least one interface module (200) includes a rate control module configured to transmit only a portion of the events (100) along the programmable daisy chain when a maximum speed is exceeded, to limit the event rate to not exceed a maximum rate.
13. The at least one interface module (200) includes a mapping module (206) configured to map one event address to another event address; The mapping module (206) may include at least one or a combination of a region of interest module, a lookup table module, and a flip and / or rotation module; The integrated circuit of any one of claims 1 to 8, characterized in that the flip and / or rotate module is configured to flip and / or rotate event addresses of the events (100).
14. The at least one interface module (200) includes an event address filter module (208) configured to filter out a set of events (100) having a particular selected event address; The integrated circuit of any one of claims 1 to 8, characterized in that the event address filter module (208) is specifically a hot pixel filter module (208') configured to filter events (100) having a specific event address and store a predetermined event address list waiting to be filtered via a CAM memory (208'-3).
15. The integrated circuit of any one of claims 1 to 8, characterized in that any one or more interface modules (200) of the event-driven interface system (20) are bypassable by a programmable switch.
Citation Information
Patent Citations
Optical device module, and manufacturing method therefor
JP2005216970A
Semiconductor device
JP2015119038A
Low-power, always-on face detection, tracking, recognition, and / or analysis using event-based vision sensors.
JP2018501531A
Feature calculation in sensor element array
JP2018501675A
Imaging system and object recognition system
JP2020161993A
