Intelligent imager for intensive real-time image analysis

Abstract

An intelligent imager comprising a three-dimensional integrated circuit having a stack of at least two integrated circuit layers: a sensor layer with a pixel array, and one or more processing layers. The three-dimensional integrated circuit includes: a conversion block (30) comprising a plurality of analog-to-digital converters (31) at the base of each pixel column; a parallel processing processor (40) comprising several processing units (41), each comprising several elementary processors (42) and a shared memory (43); a data bus (50); and a control block (60) configured to route the digital data output from the conversion block (30) directly to the shared memories (43) of the processing units (41), with the digital data representing the same pixel row being routed simultaneously to different processing units (41). Figure for the abstract: Fig. 1

Description

Title of the invention: Intelligent imager for intensive real-time image analysis. Field of the invention

[0001] The present invention relates to the field of intelligent imagers. An intelligent imager is a system integrating a vision sensor and an image processing processor. The invention is particularly well suited to the case of an intelligent imager comprising a high-resolution vision sensor and a parallel processor adapted to implement intensive image analysis processing based on an artificial intelligence algorithm. State of the art

[0002] In an intelligent imager comprising a high-resolution vision sensor and a high-performance image analysis processor, the interface between the sensor and the processing processor is particularly difficult to implement. Due to both the large amount of data from the sensor and the complexity of the processing processor, the interface between the sensor and the processing processor often introduces latency in the processing performed by the processor.

[0003] The vision sensor generally takes the form of a pixel array. Each pixel has a photodetector configured to provide an analog signal representing the light intensity received by the photodetector. Each pixel is connected to an analog-to-digital converter (ADC) via a readout circuit to transform the analog signal into digital data. To enable real-time image analysis, the digital data representing the sensor pixels must be transmitted to the processing unit with minimal latency.

[0004] For intensive image analysis processing, it is advantageous to use a parallel architecture processing unit. Such a processor is organized into several parallel computing units. Each computing unit comprises several elementary processors and shared memory. Such a processor is particularly well-suited for implementing an artificial intelligence image analysis algorithm.

[0005] Conventional intelligent imagers include a frame buffer and a sequencer positioned between the sensor and the processing unit. To perform image processing, the digital data representing the sensor pixels are first stored in the frame buffer. buffer, then routed sequentially to the shared memories of the different computing units using the sequencer.

[0006] Patent application WO 2014 / 057106 A1 describes an image sensor comprising several pixel groups and column-foot analog-to-digital converters (each converter is configured to perform the analog-to-digital conversion of the pixel signals in one column of a pixel group). The digitized data is stored in a buffer formed by several memory banks. A sequencer is configured to read the memory banks and export the digital pixel data to an output interface for an image processing engine.

[0007] Patent application WO 2017 / 161060 A1 describes an imager in the form of a three-dimensional integrated circuit. A pixel matrix is ​​implemented on a first layer of the imager. A second layer of the imager comprises a readout circuit and analog-to-digital converters arranged in a cluster structure: the pixel matrix is ​​divided into several pixel groups, and each pixel group is associated with an analog-to-digital converter (this is referred to as an "ADC cluster" structure). The digital pixel data is stored in a buffer before being exported to an image analysis processor.

[0008] The document “3D Integration Technologies for the Stacked CMOS Image Sensors,” Y. Kagawa et al., 2019 International 3D Systems Integration Conference (3DIC), also describes an imager in the form of a three-dimensional integrated circuit. A pixel matrix is ​​implemented on a first layer of the imager, and analog-to-digital converters positioned at the base of the pixel matrix columns feed a buffer implemented on a second layer of the imager. This buffer allows the digital pixel data to be temporarily stored before being sequentially exported to an image analysis processor.

[0009] The imagers presented above all have the disadvantage of introducing latency in image analysis processing.

[0010] Patent application EP 3971979 A1 describes a three-dimensional integrated circuit architecture that optimizes the distribution of analog and digital components in a stack of at least three semiconductor layers. Such an architecture can be adapted, in particular, for the implementation of an integrated circuit for an imager (image sensor). Description of the invention

[0011] The present invention aims to remedy all or part of the drawbacks of the prior art, in particular those set out above.

[0012] To this end, and according to a first aspect, the present invention proposes an intelligent imager comprising a three-dimensional integrated circuit having a stack of at least two integrated circuit layers. Each layer is electrically connected to at least one other layer via through-hole electrical connections. The layers comprise an upper layer, referred to as the "sensor layer," and one or more lower layers, referred to as the "processing layers." The sensor layer comprises a photosensitive sensor having a pixel array, each pixel being configured to provide an analog signal representative of a received light intensity.The three-dimensional integrated circuit includes a conversion block implemented on the sensor layer and / or on one or more processing layers, a parallel processing unit implemented on one or more processing layers, a data bus, and a control block implemented on the processing layer(s) on which the parallel processing unit is implemented. The conversion block comprises a plurality of analog-to-digital converters, each associated with a column of the pixel matrix. The conversion block is configured to transform the analog signals provided by the pixels into digital data. The parallel processing unit comprises a plurality of processing units, each processing unit having several elementary processors and shared memory among the elementary processors of the processing unit.The data bus and control block are configured to route the digital data output from the conversion block directly to the shared memories of the processing units, with the digital data representing the same pixel line all being routed simultaneously to different processing units.

[0013] An "integrated circuit" is also called a "microelectronic circuit" or "electronic chip". An integrated circuit comprises active analog and / or digital electronic components made from a semiconductor material, as well as metallic traces for electrically interconnecting the electronic components.

[0014] In the present application, an "integrated circuit layer" corresponds to a planar (2D) microelectronic circuit that forms part of the three-dimensional (3D) integrated circuit forming the imager. The different layers (commonly referred to as "tiers") are stacked vertically one on top of the other (the vertical being defined by the direction of stacking of the layers). The term "layer" here has the same meaning as the term "level" (the 3D integrated circuit is formed by a stacking of several circuit "levels").

[0015] A layer of the stack can be electrically connected to another layer via vias. A via is a metallic connection allowing vertical passage through the semiconductor substrate of a layer to electrically connect two electronic components made in two different layers of the stack.

[0016] Different manufacturing methods can be considered for producing the various layers of the stack. According to a first example, known as "parallel 3D integration" ("3D stacking" or "3D packaging"), the different layers are produced independently of each other from different wafers, then the layers are superimposed and interconnected by TSV (Through-Silicon Via) type connections. According to another example, known as "sequential 3D integration" or "monolithic 3D integration"), the different layers are directly produced one on top of the other.For example, after fabricating a first integrated circuit layer from a first semiconductor substrate, a second semiconductor substrate is transferred onto the first layer (for example, by oxide-oxide bonding); the second substrate is then thinned; the electronic components of the second layer are then fabricated from the second substrate. In the case of sequential 3D integration, the electrical connections between the layers are of the MIV type (Monolithic Inter-tier Via). Electrical connections between the different layers can also be made using contact pads or microbeads. It is also possible to stack the imager's integrated circuit layers using parallel 3D integration for some of the layers and sequential 3D integration for others.

[0017] The sensor layer is the first layer of the stack (the top layer located at the top of the stack). The sensor layer can operate in the visible or infrared spectrum, depending on the intended application. The processing layers are located below the sensor layer.

[0018] Each pixel of the photosensitive sensor corresponds to a photodetector formed for example by a photodiode (PN junction), a phototransistor, a photodiode associated with transfer gates, or a photodetector made in an organic or colloidal photosensitive layer.

[0019] The sensor layer can for example be made according to a back-side illumination configuration (BSI sensor, for "Back-Side Illumination" in English) or according to a front-side configuration (FSI sensor, for "Front-Side Illumination" in English).

[0020] In this application, the term "block" refers to a portion of the three-dimensional integrated circuit forming the imager and performing a specific logic function. A block may comprise various analog and / or digital electronic components. A block may be implemented on a single layer or on several layers of the three-dimensional integrated circuit.

[0021] The conversion block's function is to transform the analog signals provided by the pixels into digital data. The conversion block can be implemented on a single layer, for example, on the processing layer located directly below the sensor layer, or on a lower processing layer, or even on the sensor layer itself. However, there is nothing preventing the conversion block from being implemented on several layers; for example, different parts of the conversion block can be implemented on several processing layers, or part of the conversion block can be implemented on the sensor layer and other parts of the conversion block can be implemented on one or more processing layers.

[0022] In particular embodiments, the conversion block is fully implemented on one or more processing layers.

[0023] In particular embodiments, the conversion block is entirely implemented on a single processing layer.

[0024] In particular embodiments, the conversion block is entirely implemented on the processing layer located immediately below the sensor layer.

[0025] The conversion block includes, in particular, analog-to-digital converters (ADCs). The conversion block may also include other electronic components, such as an analog signal amplification and shaping circuit, a pixel row or column decoder, a pixel readout circuit, etc. The analog-to-digital converters (ADCs) of the conversion block are arranged at the base of the columns of the photosensitive sensor's pixel array (column ADC arrangement). This arrangement allows for efficient sharing of the ADCs across the pixel columns. The area occupied by the ADCs is therefore optimized (the area occupied is significantly smaller than for a cluster ADC arrangement). The readout circuit is also significantly less complex.The arrangement of the ADCs at the base of the column allows for a higher density of photodetectors in the pixel matrix, which improves the sensor's performance (for example, in terms of signal-to-noise ratio or quantum efficiency).

[0026] The parallel processing unit can, in particular, be configured to perform intensive image analysis processing, for example for recognition of shapes, objects, faces, or for event detection (e.g., motion detection or gesture recognition) in images. This intensive image analysis processing can be based on one or more artificial intelligence algorithms.

[0027] Different implementation options are conceivable for the parallel processing processor: it can be implemented entirely on the processing layer located immediately below the sensor layer (second layer of the stack), it can be implemented partly on the second layer and partly on at least one other processing layer below the second layer, or it can be implemented on one or more processing layers below the second layer.

[0028] In particular embodiments, the three-dimensional integrated circuit has at least three layers, and the parallel processing processor is implemented partly on the processing layer that implements the conversion block and partly on at least one other processing layer.

[0029] Using a 3D integrated circuit to implement the intelligent imager offers several advantages. The layered structure reduces the length of electrical connections between the various electronic components built on the different layers, thus optimizing performance. The 3D structure also allows the different functional blocks to be distributed across different layers according to their specific characteristics. Each block can then be implemented using the most suitable technology (each layer can be implemented with a different technology). The 3D structure also enables the implementation of an architecture with a high level of parallelization of the tasks performed by the different circuits in the different layers.

[0030] Each processing unit of the processing unit is configured to process digital data corresponding to an area (a subset) of pixels of the pixel matrix of the photosensitive sensor.

[0031] The data bus and the control block provide the functionality of a communication network for routing the digital data output from the conversion block to the various processing units. It is important to note that the digital data is routed "directly" from the conversion block to the various processing units. This means that, in the proposed intelligent imager architecture, there is no intermediate buffer for storing the digital pixel data between an ADC and the processing unit. In other words, the digital data is routed directly from the conversion block to the shared memories of the processing units without being stored in any other intermediate memory.

[0032] The elementary processors of the processing unit are designed to be configured to read digital data stored in the shared memory of the processing unit to which they belong, and to execute an image analysis algorithm directly from the read data (for example, an artificial intelligence algorithm). This direct interface to the processing units, without an intermediate buffer, allows pixel data to be processed as soon as possible after digital conversion, resulting in a substantial reduction in processing latency. Furthermore, eliminating an intermediate buffer reduces the imager's size and power consumption.

[0033] The control block is configured so that numerical data representing the same row of pixels are all routed simultaneously to different processing units. Thus, processing units that have received numerical data relating to one or more rows of pixels in the matrix can begin their processing while numerical data relating to one or more other rows of pixels are being routed to other processing units. This also helps to reduce latency in image analysis processing.

[0034] A digital output data from the conversion block can be representative of a single pixel or a group (an area) of pixels of the photosensitive sensor matrix.

[0035] In particular embodiments, the conversion block comprises pixel grouping blocks, each pixel grouping block being connected to several analog-to-digital converters associated with several successive columns of the pixel matrix, each pixel grouping block being configured to provide a digital data representative of a group of pixels of the pixel matrix.

[0036] For example, a numerical value can represent a group of M x N pixels, where M is a divisor (strictly less than) of the number of columns in the matrix and N is a divisor (strictly less than) of the number of rows in the matrix. The numerical value can, in particular, correspond to an average (possibly weighted) of the M x N pixels. Such arrangements reduce the complexity of intensive analysis processing by reducing the amount of numerical data to be processed. This also allows for a reduction in the size and power consumption of the imager.

[0037] In particular embodiments, the control block is configured to route the same digital data to at least two separate computing units by transmitting said digital data only once on the data bus.

[0038] Such arrangements make it possible to implement a redundancy function (for example, when the same digital pixel data is transmitted to at least two separate processing units) or an image processing function with areas overlap (for example, when only part of the pixel data intended for one processing unit is also transmitted to another processing unit). The ability to transmit the same digital data to several separate processing units in a single transmission on the data bus avoids introducing latency into the processing.

[0039] In particular embodiments, the computing units are arranged in a matrix format along several rows and several columns, and the control block comprises several sub-control blocks, each sub-control block being connected to all the computing units in the same column via a branch of the data bus.

[0040] It is thus possible to transmit the same numerical data to several calculation units of the same column in a single step.

[0041] In particular embodiments, the three-dimensional integrated circuit further comprises an optimization processor separate from the parallel processing processor, implemented on one or more processing layers. Digital data output from the conversion block is routed to the optimization processor via a path separate from the data bus.

[0042] The optimization processor is for example adapted to execute an optimization or error correction algorithm, and to output an image stream on an output interface, such as a MIPI type interface (English acronym for "Mobile Industry Processor Interface", it is an alliance dedicated to the development of interface specifications for mobile products).

[0043] The use of two separate paths to route digital data to the parallel processing unit and to the optimization processor enables simultaneous image enhancement and intensive image analysis processing. Thanks to this decoupling, the image optimization and / or optical defect correction processing implemented by the optimization processor does not add any additional latency to the intensive image analysis processing implemented by the parallel processing unit (the enhancement and defect correction implemented by the optimization processor is not required for the image analysis processing implemented by the parallel processing unit).

[0044] Various options are possible for routing the digital data output from the conversion block to the computing units of the parallel processing processor.

[0045] In particular embodiments, data transmission is carried out "point-to-point" or "point-to-multipoint": the control block is then configured to encapsulate a numerical data item in a message containing at least one addressing identifier corresponding to at least one computing unit, and to broadcast said message on the data bus to several computing units; each computing unit has a filtering block configured to detect, based on said at least one addressing identifier, whether the numeric data encapsulated in the message should or should not be processed by the computing unit.

[0046] In particular embodiments, data routing is implemented by broadcast: each computing unit then includes a filtering block configured to detect, based on a number of received digital data, whether a received digital data should or should not be processed by the computing unit. Presentation of the figures

[0047] The invention will be better understood upon reading the following description, given by way of non-limiting example, and made with reference to the following figures which represent:

[0048] [Fig. 1] a schematic representation of the architecture of the intelligent imager according to the invention,

[0049] [Fig.2] a schematic representation of a first example of an embodiment of the intelligent imager in the form of a three-dimensional integrated circuit comprising three stacked layers,

[0050] [Fig.3] a detailed schematic representation of the first embodiment described in [Fig.2],

[0051] [Fig.4] a schematic representation of the intelligent imager sensor layer,

[0052] [Fig. 5] a schematic representation of a processing layer of the intelligent imager,

[0053] [Fig.6] a schematic representation of another processing layer of the intelligent imager,

[0054] [Fig.7] a schematic representation of a second embodiment of the intelligent imager in the form of a three-dimensional integrated circuit comprising five stacked layers,

[0055] [Fig.8] a detailed schematic representation of the second embodiment illustrated in [Fig.7],

[0056] [Fig.9] a schematic representation of a processing layer of the intelligent imager according to the second embodiment,

[0057] [Fig. 10] a schematic representation of a computing unit of the parallel processing processor integrated into the intelligent imager.

[0058] In these figures, identical reference numerals from one figure to another designate identical or analogous elements. For clarity, the elements shown are not necessarily to the same scale, unless otherwise stated. Detailed description of the invention

[0059] Figure 1 schematically represents a computing architecture of the intelligent imager according to the invention. This is a functional representation of various electronic components implemented on a three-dimensional (3D) integrated circuit forming the intelligent imager.

[0060] Figures 2 and 3 represent a first example of stacking three layers 10-1, 10-2, 10-3 of integrated circuit to form the three-dimensional integrated circuit 10 forming the intelligent imager.

[0061] The term "integrated circuit" means a microelectronic circuit comprising active analog and / or digital electronic components made from a semiconductor material, as well as metallic traces for electrically interconnecting the electronic components. The term "semiconductor material" encompasses any material exhibiting semiconducting properties (amorphous or polycrystalline semiconductors such as silicon, carbon nanotubes, semiconductor oxides, etc.).

[0062] As illustrated in [Fig. 2], the three layers 10-1, 10-2, 10-3 are stacked one on top of the other. Each layer of the stack is electrically connected to at least one other layer via through-hole electrical connections 11. Such a connection 11 allows the semiconductor substrate of a layer to be traversed vertically to electrically connect two electronic components made in two different layers of the stack.

[0063] As previously mentioned, various manufacturing methods can be considered for producing the different layers 10-1, 10-2, 10-3, such as parallel 3D integration, sequential 3D integration, or a combination of these two techniques. In the case of parallel 3D integration, the through-wire electrical connections 11 can take the form of TSV (Through-Silicon Vias) type connections; in the case of sequential 3D integration, the through-wire electrical connections 11 can take the form of MIV (Monolithic Inter-tier Via) type connections. The through-wire electrical connections 11 can also be made using contact pads or microbeads.

[0064] The first layer 10-1 of the stack (the top layer located on top of the stack) is called the "sensor layer". The sensor layer 10-1 is shown schematically in more detail in [Fig. 4]. The sensor layer 10-1 comprises a photosensitive sensor having a matrix 20 of pixels 21. The photosensitive sensor can operate in the visible or infrared spectrum, depending on the intended application.

[0065] Each pixel 21 of the matrix 20 of the photosensitive sensor corresponds to a photodetector. A photodetector is, for example, formed by a photodiode (PN junction), a phototransistor, a photodiode combined with transfer gates, or a photodetector made in an organic or colloidal photosensitive layer. Conventionally, each pixel is configured to provide an analog signal representative of a received light intensity.

[0066] The photosensitive sensor can be implemented in a back-side illumination (BSI) configuration. In a BSI configuration, the photodetectors are positioned on the upper surface of the 10⁻¹ sensor layer (the surface opposite the 10⁻² layer immediately below the 10⁻¹ sensor layer), and the metallic tracks (routing) are positioned on the lower surface of the 10⁻¹ sensor layer (the surface opposite the 10⁻² layer immediately below the sensor layer). The BSI configuration optimizes photon capture by the photodetectors.

[0067] According to another example, the photosensitive sensor can be implemented in a front-side configuration (FSI sensor). In an FSI configuration, the photodetectors are positioned on the lower surface of the 10⁻¹ sensor layer, and the metallic traces (routing) are positioned on the upper surface of the 10⁻¹ sensor layer. The FSI configuration simplifies the manufacturing of the sensor layer. However, the routing portion reflects some of the light and reduces the number of photons detected by the photodetectors.

[0068] In the example considered, the matrix 20 has 3072 rows of 4096 pixels (the matrix 20 therefore has 4096 columns of 3072 pixels; it is a 4096 x 3072 pixel matrix). Each pixel 21 occupies an area of ​​1 pm² (one square micrometer). The silicon surface area of ​​the sensor layer 10⁻¹ is approximately 12 mm². The thickness of the sensor layer 10⁻¹ is less than 300 pm.

[0069] The various functional blocks described in [Fig. 1] are implemented on layers 10-2 and 10-3, which are located below the sensor layer 10-1. Layers 10-2 and 10-3, located below the sensor layer, are called "processing layers".

[0070] Figure 5 schematically represents the processing layer 10⁻² located immediately below the sensor layer 10⁻¹. Layer 10⁻² corresponds to the second layer of the stack forming the three-dimensional integrated circuit 10. Figure 6 schematically represents the processing layer 10⁻³ located below of the second layer 10-2. Layer 10-3 corresponds to the third layer of the stack forming the three-dimensional integrated circuit 10.

[0071] As illustrated in [Fig.1], the three-dimensional integrated circuit 10 forming the intelligent imager according to the invention comprises a conversion block 30, a parallel processing processor 40, a data bus 50 and a control block 60.

[0072] The conversion block 30 has the function of transforming the analog signals provided by the pixels 21 of the matrix 20 of the photosensitive sensor into digital data. In the example considered, and as illustrated in Figures 3 and 5, the conversion block 30 is implemented on the processing layer 10-2 located immediately below the sensor layer 10-1. The conversion block 30 is also referred to as "CNV" in the figures.

[0073] It should be noted that the conversion block 30 could also be implemented on another layer, for example on a lower processing layer, or even on the sensor layer 10-1. Nor does anything prevent the conversion block 30 from being implemented on several layers (for example, different parts of the conversion block could be implemented on several processing layers, or part of the conversion block could be implemented on the sensor layer and other parts of the conversion block could be implemented on one or more processing layers).

[0074] As illustrated in [Fig. 1], the conversion block 30 comprises analog-to-digital converters 31 (ADCs 31). The conversion block 30 may also include other electronic components, such as an analog signal amplification and shaping circuit, a pixel row or column decoder, a pixel readout circuit, etc. These components are not shown in the figures. The ADCs 31 of the conversion block are arranged at the base of the columns of the pixel matrix 20 of the photosensitive sensor (column ADC arrangement). In other words, each column of pixels of the photosensitive sensor is associated with an ADC 31. In the example considered, the conversion block 30 therefore comprises 4096 ADCs.

[0075] As schematically illustrated in [Fig.2], each column of pixels of the matrix 20 of the photosensitive sensor of the sensor layer 10-1 is electrically connected to an ADC 31 of the conversion block 30 via a through-hole electrical connection 11 (via).

[0076] Conventionally, each ADC 31 is adapted to digitize the analog signals of the pixels in the column to which the ADC is attached (each analog signal being representative of a light intensity received by a pixel).

[0077] In the example considered, the parallel processing unit 40 is configured to perform intensive image analysis processing based on an algorithm Artificial intelligence is used, for example, for shape, object, or face recognition, or for event detection in images (e.g., motion detection). The artificial intelligence algorithm is, for example, a deep neural network algorithm.

[0078] To be implemented under the photosensitive sensor, the processing unit 40 must meet stringent constraints in terms of power consumption, heat generation, surface area, and computing capacity. To provide high computing capacity within these power, heat generation, and surface area constraints, the processing unit 40 follows a parallel architecture organized into a plurality of groups (“clusters”) of elementary processors. Thus, and as illustrated in [Fig. 1], the parallel processing unit 40 comprises a plurality of computing units 41. Each computing unit 41 comprises several elementary processors 42 and a memory 43 shared among the elementary processors 42 of the computing unit 41.

[0079] To enable real-time image analysis, the digital data representing the pixels of the sensor must be routed to the distributed memories 43 of the parallel processing processor 40 with a minimum of latency.

[0080] It should be noted that the shared memory 43 can be a monolithic memory shared between the different elementary processors 42, or a memory comprising several memory banks shared between the different elementary processors 42 (the different memory banks possibly being accessible simultaneously to increase bandwidth).

[0081] In the example considered and described with reference to Figures 2 to 6, the parallel processing processor 40 is implemented partly on the 10-2 processing layer (i.e., on the second 10-2 layer of the stack, which also corresponds to the first processing layer) and partly on the 10-3 processing layer (i.e., on the third 10-3 layer of the stack, which also corresponds to the second processing layer). As illustrated in Figures 1, 5, and 6, the computing units 41 of the parallel processing processor 40 are arranged in a matrix across several rows and several columns. As illustrated in Figures 5 and 6, part of the 41 computing unit lines is implemented on the 10-2 processing layer, and another part of the 41 computing unit lines is implemented on the 10-3 processing layer.

[0082] It should be noted that this example is by no means limiting. Nothing would prevent, for example, the parallel processing chip 40 from being implemented entirely on the processing layer 10-2 located immediately below the sensor layer (second layer of the stack). Nor would anything prevent it from being implemented on one or more processing layers located below the second 10-2 layer of the stack.

[0083] In the example considered, the parallel processing unit 40 is a PNeuro-type processor as described in the document "PNeuro: a scalable energy-efficient programmable hardware accelerator for neural networks," A. Carbon et al., Design, Automation and Test in Europe (DATA) 2018, pages 1045-1050. A computing unit 41 corresponds to a neural computing block (or NCB). The processing layer 10-2, for example, comprises three rows of sixteen computing units 41. The processing layer 10-3, for example, comprises three other rows of sixteen computing units 41. However, nothing would prevent one and / or the other of the processing layers 10-2 and 10-3 from comprising a different number (greater or less than three) of rows of computing units 41. Nothing would prevent a line of 41 calculation units from containing another number (greater or less than sixteen) of 41 calculation units.In the example considered, each of the 10-2 and 10-3 processing layers occupies a silicon surface area of ​​approximately 11 mm². The thickness of the 10-2 layer is between 10 and 30 µm. The thickness of the 10-3 layer is on the order of 200 to 300 µm. The total thickness of the three-dimensional integrated circuit 10 is less than 500 µm.

[0084] The data bus 50 and the control block 60 provide the functionality of a communication network for routing the digital data output from the conversion block 30 to the shared memories 43 of the various processing units 41. It is important to note that the digital data is routed "directly" from the conversion block 30 to the various processing units 41. This means that, in the proposed intelligent imager architecture, there is no intermediate buffer for storing the digital pixel data between an ADC 31 and the processing unit 40. In other words, the digital data is routed directly from the conversion block 30 to the shared memories 43 of the processing units 41 without being stored in any other intermediate memory. In the example considered with a PNeuro-type processor, each shared memory 43 has four memory banks.

[0085] As illustrated in Figures 5 and 6, the data bus 50 and the control block 60 are implemented on the processing layers 10-2 and 10-3 on which the parallel processing processor 40 is implemented.

[0086] Each computing unit 41 of the processing processor 40 is configured to process digital data corresponding to an area (a subset) of pixels 21 of the matrix 20 of the photosensitive sensor implemented on the sensor layer 10-1.

[0087] The elementary processors 42 of the computing units 41 of the processing processor 40 are intended to be configured to read digital data stored in the shared memory 43 of the processing unit 41 to which they belong, and to execute the image analysis algorithm directly from the read data. This direct interface to the processing units 41, without intermediate buffer memory, allows the pixel data to be processed as soon as possible after digital conversion.

[0088] The control block 60 is configured so that digital data representing the same row of pixels are all routed simultaneously to different processing units 41. Thus, the processing units 41 that have received digital data relating to one or more rows of pixels in the matrix can begin their processing while digital data relating to one or more other rows of pixels are being routed to other processing units. This makes it possible to significantly reduce latency in image analysis processing.

[0089] As illustrated in Figures 1, 5, and 6, the control block 60 comprises several control sub-blocks 61. The control sub-blocks 61 can, in particular, encapsulate the data to be transmitted in messages containing at least one destination address (an address identifying a processing unit 41 among the various processing units 41 of the parallel processing unit 40). The control sub-blocks 61 can also transmit control and / or synchronization information (for example, a signal indicating the start or end of image processing). In the figures, the control sub-blocks 61 are designated by the acronym ACS (Address, Control, and Synchronization). Advantageously, and as illustrated in Figures 1, 5 and 6, each control subblock 61 can be connected, via a branch of the data bus 50, to all the computing units 41 in the same column of computing units 41.

[0090] As illustrated in [Fig. 2], two control subblocks 61 located on two different processing layers 10-2, 10-3 can be connected by through-connections 11. This makes it possible to provide the pixel data to all the computing units 41 of the processing processor 40 distributed across the different processing layers.

[0091] It should be noted that a digital output data from the conversion block 30 can be representative of a single pixel 21 or of a group (an area) of pixels 21 of the matrix 20 of the photosensitive sensor.

[0092] In the example considered, and as illustrated in [Fig. 1], the conversion block 30 comprises five hundred and twelve pixel binning blocks 32. A binning block 32 is designated by the letter "B" (for "binning") in [Fig. 1]. Each pixel binning block 32 is connected to eight ADCs 31 associated with eight successive columns of the pixel matrix 20. Each binning block 32 is configured to provide a numerical value representing a group of 8 x 8 pixels 21 of the pixel matrix 20. In such a case, a numerical value output from a binning block 32 is representative of an area of Sixty-four (8 x 8 = 64) pixels of the photosensitive sensor. This digital data is, for example, a weighted average of the sixty-four pixels processed by the grouping block 32. This has the effect of reducing the overall image resolution by a factor of 8 x 8 (the resolution is reduced from 4096 x 3072 to 512 x 384) while retaining the visual information in the image. Because the resolution is reduced, intensive analysis processing requires less memory and fewer calculations, thus reducing processing time and power consumption. The grouping block 32 performs very simple online and on-the-fly processing that introduces no latency. Every eight ADC clock cycles, a new digital data point representing a group of 8 x 8 pixels is produced.

[0093] Each control subblock 61 simultaneously receives thirty-two digital data points supplied by thirty-two consecutive grouping blocks 32 (each digital data point representing an 8 x 8 pixel area). Each control subblock 61 serializes these thirty-two digital data points on the data bus 50 without loss of bandwidth (the operating frequency of a control subblock 61 is therefore at least thirty-two times higher than that of an ADC). These thirty-two digital data points are routed to at least one processing unit 41.

[0094] In the example considered, in which the processing unit 40 is organized into six rows of sixteen processing units 41, the sixteen processing units 41 in each row simultaneously receive thirty-two digital data points representing a total of eight rows of pixels (i.e., 4096 x 8 pixels). After sixty-four ADC clock cycles, a row of processing units 41 has received the digital data representing one-sixth of the pixels in an image (i.e., 512 rows of pixels, or 4096 x 512 pixels). Once they have received their digital data, the processing units 41 in the same row can then begin their processing while other digital data relating to other rows of pixels are routed to other processing units in another row.

[0095] It should be noted, however, that the use of the grouping blocks 32 is optional. Without the grouping blocks 32, a numerical output from the conversion block 30 represents only one pixel 21 of the matrix 20.

[0096] In particular embodiments, the control block 60 can be configured to route the same digital data to at least two separate processing units 41 by transmitting said digital data only once on the data bus 50. It is thus possible to implement a redundancy function (for example, when the same pixel digital data is transmitted to at least two separate processing units 41) or an image processing function with overlapping areas (for example, when only a portion of the pixel digital data intended for one processing unit 41 is also transmitted to another unit 41). of computing). The ability to transmit the same digital data to several distinct computing units 41 in a single transmission on the data bus 50 avoids introducing latency into the processing.

[0097] As illustrated in [Fig.1], the three-dimensional integrated circuit 10 forming the imager may also optionally include an optimization processor 70 separate from the parallel processing processor 40. The optimization processor 70 is designated by "ISP" in the figures (for the English acronym "Image Signal Processor", in French "processeur d'images").

[0098] The optimization processor 70 is for example suitable for executing an optimization or error correction algorithm, and for outputting an image stream on an output interface (for example a MIPI interface).

[0099] In the example considered, and as illustrated in Figures 3 and 5, the optimization processor 70 is implemented on the 10-2 processing layer. However, nothing would prevent the optimization processor 70 from being implemented on another processing layer, or from being implemented on several different processing layers.

[0100] Advantageously, the digital output data from the conversion block 30 is routed to the optimization processor 70 by a path separate from the data bus 50, for example by another data bus separate from the data bus 50 which feeds the shared memories 43 of the computing units 41 of the parallel processing processor 40.

[0101] The use of two separate paths to route digital data to the parallel processing unit 40 and to the optimization unit 70 enables simultaneous image enhancement and intensive image analysis processing. Thanks to this decoupling, the image optimization and / or optical defect correction processing implemented by the optimization unit 70 does not add any additional latency to the intensive image analysis processing implemented by the parallel processing unit 40.

[0102] It should be noted that the embodiment described above with reference to Figures 2 to 6 is by no means a limiting example. In particular, the number of processing layers in the stack forming the three-dimensional integrated circuit 10 could be different (for example, a single processing layer, or more than two processing layers). The same applies to the number of computing units 41 of the parallel processing unit 40.

[0103] Figures 7 to 9 show another embodiment of an intelligent imager according to the invention. In this example, the three-dimensional integrated circuit 10 comprises five layers: a sensor layer 10-1 and four processing layers 10-2 to 10-5. The sensor layer 10-1 is similar to that described above with reference to Figures 2 to 4. The processing layer 10-2 is similar to the processing layer 10-2 described above with reference to figures 2, 3 and 5. The 10-3 and 10-4 treatment layers are similar to the 10-3 treatment layer described above with reference to figures 2, 3 and 6.

[0104] The 10-5 processing layer is described in [Fig. 9]. The 10-5 processing layer includes a portion of the parallel processing unit 40, as well as a portion of the control block 60 and the data bus 50. In addition, the 10-5 processing layer includes a mass memory 80 which can be used, for example, to store weights and parameters of one or more artificial intelligence algorithms intended to be executed by the parallel processing unit 40. This mass memory 80 can be implemented using SRAM (Static Random Access Memory) technology or with non-volatile technology to achieve higher memory density. The 10-5 processing layer is connected to the 10-4 processing layer located above it by through-connections 11 at the level of the control subblocks 61, similarly to what was described previously with reference to [Fig. 2].The 10-5 treatment layer also occupies a silicon surface area of ​​approximately 11 mm2.

[0105] The three-dimensional integrated circuit 10 could also optionally include an approximate event detection preprocessor used to wake up the parallel processing processor 40 when an approximate event is detected. Such arrangements allow the parallel processing processor 40 to remain in standby mode until an approximate event is detected by the preprocessor. The preprocessor can, for example, be implemented on the processing layer 10-2, and / or on one or more other processing layers. An additional pixel grouping step can optionally be considered to further reduce the resolution of the data supplied to the preprocessor (for example, a resolution of 108 x 76 pixels).

[0106] In the example illustrated in Figures 7 to 9, the thickness of each of the layers 10-2 to 10-4 is between 10 and 30 µm. The thickness of the layer 10-5 is on the order of 200 to 300 µm.

[0107] [Fig. 10] schematically represents an example of an embodiment of a computing unit 41 of the parallel processing processor 40. [Fig. 10] also illustrates how a digital output data from the conversion block 30 can be routed to the shared memory of the computing unit 41 with the data bus 50 under the control of a control sub-block 61 (ACS block).

[0108] In the example considered and illustrated in [Fig. 10], a multiplexer 45 allows the selection of a data type to be written to the shared memory 43. The multiplexer 48 allows the selection of the address to be used for writing digital data, depending on whether the data to be written corresponds to pixel information from the data bus 50 or to information from the elementary processor group 42. Arrow 50-1 represents a digital data item representing one or more pixels transmitted on the data bus 50; arrow 50-2 represents a control or synchronization information issued by an ACS block on the data bus 50; arrow 46 represents data from the elementary processor group 42 to be written to the shared memory 43; arrow 47 represents data read by the elementary processor group 42. The processing unit 41 includes a filtering block 44 whose function is to detect whether a digital pixel data item transmitted on the data bus 50 should be processed by the processing unit 41 and, if so, to determine a local address of the shared memory 43 to which the data should be written.

[0109] Different options are possible for routing the digital data output from the conversion block 30 to the different calculation units 41.

[0110] According to a first example, data transmission is carried out "point-to-point" or "point-to-multipoint". In this case, the control block 60 is configured to encapsulate a digital pixel data item in a message containing at least one addressing identifier corresponding to at least one processing unit 41 and to broadcast said message on the data bus 50 to several processing units 41. The filtering block 44 is configured to detect, based on said at least one addressing identifier, whether or not the digital data included in the message should be processed by the processing unit 41.

[0111] According to a second example, data routing is implemented by broadcast. In this case, the filter block 44 is configured to detect, based on the number of received digital data points, whether or not a received digital data point should be processed by the processing unit 41. For example, the control block 60 is configured to send a synchronization signal on the data bus 50 to indicate the start of processing a new image; the filter block 44 knows that it will see a stream of digital data associated with this new image on the data bus; and the filter block 44 also knows the positions of the digital data points that the processing unit 41 to which it is connected must process in this stream.

[0112] The preceding description clearly illustrates that, through its various features and their advantages, the present invention achieves the stated objectives. In particular, the arrangement of the ADCs at the base of the column allows them to be shared efficiently across the pixel columns. The surface area occupied by the ADCs is thus optimized and the readout circuit is simplified. The arrangement of the ADCs at the base of the column makes it possible to obtain a higher density of photodetectors in the pixel matrix, which allows to improve sensor performance (e.g., in terms of signal-to-noise ratio or quantum efficiency).

[0113] The layered stacking structure also allows for performance optimization (reduced power consumption, strong parallelization of tasks performed by the circuits of the different layers).

[0114] Direct routing of digital data from the conversion block 30 to the shared memories 43 of the computing units 41 (without using an intermediate buffer memory) makes it possible to significantly limit the latency in image processing, the size of the imager, and the power consumed by the imager.

[0115] The use of a parallel processing unit 40 makes it possible to meet the constraints inherent in intensive image processing (constraints in terms of power consumption, heat generation, surface area, and computing capacity). Advantageously, the digital data representing the same line of pixels are all routed simultaneously to different computing units 41 of the parallel processing unit 40 in order to reduce processing latency.

Claims

1. Demands Intelligent imager comprising a three-dimensional integrated circuit (10) having a stack of at least two integrated circuit layers (10-1, 10-2, 10-3), each layer being electrically connected to at least one other layer via through-hole electrical connections (11), said layers comprising: - an upper layer (10-1), called the "sensor layer", comprising a photosensitive sensor having a matrix (20) of pixels, each pixel (21) being configured to provide an analog signal representative of a received light intensity, - one or more lower layers (10-2, 10-3), called "treatment layers", the three-dimensional integrated circuit (10) comprising: - a conversion block (30) implemented on the sensor layer (10-1) and / or on one or more processing layers (10-2, 10-3), the conversion block (30) comprising a plurality of analog-to-digital converters (31) each associated respectively with a column of the pixel matrix (20), the conversion block (30) being configured to transform into digital data the analog signals supplied by the pixels (21), - a parallel processing processor (40) implemented on one or more processing layers (10-2, 10-3), the parallel processing processor (40) comprising a plurality of computing units (41), each computing unit (41) comprising several elementary processors (42) and a memory (43) shared between the elementary processors (42) of the computing unit (41), - a data bus (50) and a control block (60) implemented on the processing layer(s) (10-2, 10-3) on which the parallel processing processor (40) is implemented, and configured to route the digital data output from the conversion block (30) directly to the shared memories (43) of the processing units (41), the data digital data representing the same line of pixels being all routed simultaneously to different computing units (41), characterized in that the control block (60) is configured to route the same digital data to at least two separate computing units (41) by transmitting said digital data only once on the data bus (50).

2. Intelligent imager according to claim 1 wherein the conversion block (30) is fully implemented on one or more processing layers (10-2, 10-3).

3. Intelligent imager according to claim 2 wherein the conversion block (30) is entirely implemented on a single processing layer (10-2) (10-2).

4. Intelligent imager according to claim 3 wherein the conversion block (30) is entirely implemented on the processing layer (10-2) located immediately below the sensor layer (10-1).

5. Intelligent imager according to any one of claims 3 to 4 wherein the three-dimensional integrated circuit (10) has at least three layers (10-1, 10-2, 10-3), and the parallel processing processor (40) is implemented partly on the processing layer (10-2) which implements the conversion block (30) and partly on at least one other processing layer (10-3).

6. Intelligent imager according to any one of claims 1 to 5 wherein the conversion block (30) comprises pixel grouping blocks (32), each pixel grouping block (32) being connected to several analog-to-digital converters (31) associated with several successive columns of the pixel matrix, each pixel grouping block (32) being configured to provide digital data representative of a group of pixels (21) of the pixel matrix (20).

7. Intelligent imager according to any one of claims 1 to 6 wherein the computing units (41) are arranged in a matrix pattern along several rows and several columns, and the control block (60) comprises several sub-control blocks (61), each sub-control block (61) being connected to all the computing units (41) in the same column via a branch of the data bus (50).

8. Intelligent imager according to any one of claims 1 to 7 wherein the three-dimensional integrated circuit (10) further comprises an optimization processor (70) separate from the parallel processing processor (40), implemented on one or more processing layers (10-2, 10-3), and wherein digital data output from the conversion block (30) are routed to the optimization processor (70) by a path separate from the data bus (50).

9. Intelligent imager according to any one of claims 1 to 8 wherein the control block (60) is configured to encapsulate digital data in a message comprising at least one addressing identifier corresponding to at least one computing unit (41), and to broadcast said message on the data bus (50) to several computing units (41), and wherein each computing unit (41) comprises a filtering block (44) configured to detect, based on said at least one addressing identifier, whether the digital data encapsulated in the message should be processed by the computing unit (41).

10. Intelligent imager according to any one of claims 1 to 8 wherein each computing unit (41) includes a filtering block (44) configured to detect, based on a number of received digital data points, whether a received digital data point should or should not be processed by the computing unit (41).

Images