Intelligent imager for intensive real-time image analysis

The 3D integrated circuit architecture in intelligent imagers directly routes digital data from a sensor layer to parallel processing units, addressing latency issues and enhancing real-time image analysis efficiency.

FR3149427A1Active Publication Date: 2024-12-06COMMISSARIAT A LENERGIE ATOMIQUE ET AUX ENERGIES ALTERNATIVES
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
FR2023005638
Authority / Receiving Office
FR · FR
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06
Estimated Expiration
2043-06-05

AI Technical Summary

Technical Problem

The interface between a high-resolution vision sensor and a processing processor in intelligent imagers introduces latency due to the large data volume and complexity, leading to inefficient image analysis processing.

Method used

A three-dimensional integrated circuit architecture with a stack of layers, including a sensor layer and processing layers, where analog signals are converted to digital data directly routed to shared memories of parallel processing units without intermediate buffers, optimizing the distribution of components and reducing latency.

Benefits of technology

This architecture significantly reduces processing latency, minimizes imager size and power consumption, and enables efficient real-time image analysis, particularly for tasks like shape recognition and event detection using artificial intelligence algorithms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000024_0000
    Figure 00000024_0000
  • Figure 00000025_0000
    Figure 00000025_0000
  • Figure 00000026_0000
    Figure 00000026_0000
Patent Text Reader

Abstract

An intelligent imager comprising a three-dimensional integrated circuit having a stack of at least two integrated circuit layers: a sensor layer with a pixel array, and one or more processing layers. The three-dimensional integrated circuit includes: a conversion block (30) comprising a plurality of analog-to-digital converters (31) at the base of each pixel column; a parallel processing processor (40) comprising several processing units (41), each comprising several elementary processors (42) and a shared memory (43); a data bus (50); and a control block (60) configured to route the digital data output from the conversion block (30) directly to the shared memories (43) of the processing units (41), with the digital data representing the same pixel row being routed simultaneously to different processing units (41). Figure for the abstract: Fig. 1
Need to check novelty before this filing date? Find Prior Art

Description

Title of the invention: Intelligent imager for intensive real-time image analysis Field of the invention

[0001] The present invention belongs to the field of intelligent imagers. A smart imager is a system integrating a vision sensor and an image processing processor. The invention is particularly well suited to the case of a smart imager comprising a high-resolution vision sensor and a parallel processor adapted to implement intensive image analysis processing based on an artificial intelligence algorithm. State of the art

[0002] In a smart imager comprising a high-resolution vision sensor and an intensive image analysis processing processor, the interface between the sensor and the processing processor is particularly difficult to implement. Due to the large amount of data from the sensor and the complexity of the processing processor, the interface between the sensor and the processing processor often introduces latency into the processing performed by the processor.

[0003] The vision sensor generally takes the form of a matrix of pixels. Each pixel comprises a photodetector configured to provide an analog signal representative of a light intensity received by the photodetector. Each pixel is connected to an analog-to-digital converter (ADC) via a readout circuit to transform the analog signal into digital data. To enable real-time image analysis, the digital data representative of the sensor pixels must be routed to the processing processor with minimal latency.

[0004] To perform intensive image analysis processing, it is advantageous to use a processing processor with parallel architecture. Such a processor is organized into several parallel computing units. Each computing unit comprises several elementary processors as well as a memory shared between these elementary processors. Such a processor is particularly well suited to the implementation of an artificial intelligence algorithm for image analysis.

[0005] Conventional smart imagers include a frame buffer and a sequencer positioned between the sensor and the processing processor. To perform image processing, digital data representative of the sensor pixels are first stored in the buffer, then routed sequentially to the shared memories of the different computing units using the sequencer.

[0006] Patent application WO 2014 / 057106 A1 describes an image sensor comprising several groups of pixels as well as analog-to-digital converters at the bottom of the column (each converter is configured to perform the analog-to-digital conversion of the pixel signals of a column of a group of pixels). The digitized data is stored in a buffer memory formed by several memory banks. A sequencer is configured to read the memory banks and export the digital pixel data to an output interface intended for an image processing processor.

[0007] Patent application WO 2017 / 161060 A1 describes an imager in the form of a three-dimensional integrated circuit. A pixel matrix is ​​implemented on a first layer of the imager. A second layer of the imager comprises a readout circuit and analog-to-digital converters according to a group structure: the pixel matrix is ​​divided into several groups of pixels and each group of pixels is associated with an analog-to-digital converter (this is referred to as an "ADC cluster" type structure in English). The digital pixel data is stored in a buffer memory before being exported to an image analysis processor.

[0008] The document “3D Integration Technologies for the Stacked CMOS Image Sensors”, Y. Kagawa et al., 2019 International 3D Systems Integration Conference (3DIC), also describes an imager taking the form of a three-dimensional integrated circuit. A pixel matrix is ​​implemented on a first layer of the imager, and analog-to-digital converters positioned at the foot of the columns of the pixel matrix feed a buffer memory implemented on a second layer of the imager. This buffer memory allows the digital pixel data to be temporarily stored before being sequentially exported to an image analysis processor.

[0009] The imagers presented above all have the disadvantage of introducing latency into the image analysis processing.

[0010] Patent application EP 3971979 A1 describes a three-dimensional integrated circuit architecture for optimizing the distribution of analog and digital components in a stack of at least three semiconductor layers. Such an architecture can in particular be adapted to the production of an integrated circuit for an imager (image sensor). Statement of the invention

[0011] The present invention aims to remedy all or part of the drawbacks of the prior art, in particular those set out above.

[0012] To this end, and according to a first aspect, the present invention proposes a smart imager comprising a three-dimensional integrated circuit comprising a stack of at least two integrated circuit layers. Each layer is electrically connected to at least one other layer via through-hole electrical connections. The layers comprise an upper layer, called the "sensor layer", and one or more lower layers, called "processing layers". The sensor layer comprises a photosensitive sensor comprising a matrix of pixels, each pixel being configured to provide an analog signal representative of a received light intensity.The three-dimensional integrated circuit comprises a conversion block implemented on the sensor layer and / or on one or more processing layers, a parallel processing processor implemented on one or more processing layers, a data bus and a control block implemented on the processing layer(s) on which the parallel processing processor is implemented. The conversion block comprises a plurality of analog-to-digital converters each associated respectively with a column of the pixel matrix. The conversion block is configured to transform the analog signals provided by the pixels into digital data. The parallel processing processor comprises a plurality of calculation units, each calculation unit comprising several elementary processors and a memory shared between the elementary processors of the calculation unit.The data bus and the control block are configured to route the digital data output from the conversion block directly to the shared memories of the computing units, the digital data representative of the same row of pixels being all routed simultaneously to different computing units.

[0013] An "integrated circuit" is also called a "microelectronic circuit" or "electronic chip". An integrated circuit comprises active analog and / or digital electronic components made from a semiconductor material, as well as metal tracks for electrically interconnecting the electronic components with each other.

[0014] In the present application, an “integrated circuit layer” corresponds to a planar (2D) microelectronic circuit which forms a part of the three-dimensional (3D) integrated circuit forming the imager. The different layers (commonly referred to as “tiers” in English) are stacked vertically on top of each other (the vertical being defined by the direction of the stacking of the layers). The term “layer” has here the same meaning as the term “level” (the 3D integrated circuit is formed by a stack of several circuit “levels”).

[0015] A layer of the stack may be electrically connected to another layer via vias. A via is a metallic connection allowing a layer to pass vertically through the semiconductor substrate to electrically connect two electronic components made in two different layers of the stack).

[0016] Different manufacturing methods can be envisaged to produce the different layers of the stack. According to a first example, known as “parallel 3D integration” (“3D stacking” or “3D packaging”), the different layers are produced independently of each other from different wafers, then the layers are superimposed and interconnected by TSV type connections (acronym for “Through-Silicon Via”). According to another example, known as “sequential 3D integration” or “monolithic 3D integration”), the different layers are directly produced on top of each other.For example, after having produced a first integrated circuit layer from a first semiconductor substrate, a second semiconductor substrate is transferred onto the first layer (for example by oxide-oxide bonding); the second substrate is then thinned; the electronic components of the second layer are then manufactured from the second substrate. In the case of sequential 3D integration, the electrical connections between the layers are of the MIV type (Monolithic Inter-tier Via). The electrical connections between the different layers can also be made via contact pads or microbeads. It is also possible to carry out the stacking of the imager's integrated circuit layers using parallel 3D integration for some of the layers and sequential 3D integration for another part of the layers.

[0017] The sensor layer is the first layer of the stack (top layer located at the top of the stack). The sensor layer can operate in the visible or infrared range, depending on the intended application. The processing layers are located below the sensor layer.

[0018] Each pixel of the photosensitive sensor corresponds to a photodetector formed for example by a photodiode (PN junction), a phototransistor, a photodiode associated with transfer gates, or a photodetector produced in an organic or colloidal photosensitive layer.

[0019] The sensor layer can for example be produced according to a rear-side illumination configuration (BSI sensor, for “Back-Side Illumination” in English) or according to a front-side configuration (FSI sensor, for “Front-Side Illumination” in English).

[0020] In the present application, the term "block" designates a part of the three-dimensional integrated circuit forming the imager and responding to a certain logic function. A block may consist of different analog and / or digital electronic components. A block may be implemented on a single layer or on several layers of the three-dimensional integrated circuit.

[0021] The conversion block has the function of transforming the analog signals provided by the pixels into digital data. The conversion block can be implemented on a single layer, for example on the processing layer located directly below the sensor layer, or on a lower processing layer, or even on the sensor layer. However, nothing prevents the conversion block from being implemented on several layers; for example, different parts of the conversion block can be implemented on several processing layers, or a part of the conversion block can be implemented on the sensor layer and other parts of the conversion block can be implemented on one or more processing layers.

[0022] In particular embodiments, the conversion block is implemented entirely on one or more processing layers.

[0023] In particular embodiments, the conversion block is implemented entirely on a single processing layer.

[0024] In particular embodiments, the conversion block is entirely implemented on the processing layer located immediately below the sensor layer.

[0025] The conversion block includes in particular analog-to-digital converters (ADCs). The conversion block may also include other electronic components, such as for example an analog signal amplification and conditioning circuit, a pixel row or column decoder, a pixel reading circuit (“readout circuit”), etc. The analog-to-digital converters (ADCs) of the conversion block are arranged at the bottom of the columns of the pixel matrix of the photosensitive sensor (“column ADC” type arrangement). This arrangement makes it possible to efficiently pool the ADCs on the pixel columns. The surface area occupied by the ADCs is consequently optimized (the surface area occupied is significantly smaller than for a “cluster ADC” type arrangement). The reading circuit (“readout circuit”) is also significantly less complex.The arrangement of the ADCs at the bottom of the column makes it possible to obtain a higher density of photodetectors in the pixel matrix, which improves the performance of the sensor (for example in terms of signal to noise or quantum efficiency).

[0026] The parallel processing processor may in particular be configured to execute intensive image analysis processing, for example for recognition of shapes, objects, faces, or for detection of events (for example movement detection or gesture recognition) in images. This intensive image analysis processing may be based on one or more algorithms artificial intelligence.

[0027] Different implementation options are possible for the parallel processing processor: it can be implemented entirely on the processing layer located immediately below the sensor layer (second layer of the stack), it can be implemented partly on the second layer and partly on at least one other processing layer lower than the second layer, or it can be implemented on one or more processing layers lower than the second layer.

[0028] In particular embodiments, the three-dimensional integrated circuit comprises at least three layers, and the parallel processing processor is implemented partly on the processing layer which implements the conversion block and partly on at least one other processing layer.

[0029] The use of a 3D integrated circuit to produce the intelligent imager provides several advantages. The layer stack structure makes it possible to reduce the length of the electrical connections between the different electronic components produced on the different layers, which optimizes performance. The 3D structure also makes it possible to distribute the different functional blocks on different layers according to their specificities. Each block can then be produced using the most suitable technology (each layer can be produced with a different technology). The 3D structure also makes it possible to implement an architecture with a high level of parallelization of the tasks carried out by the different circuits of the different layers.

[0030] Each computing unit of the processing processor is configured to process digital data corresponding to an area (a subset) of pixels of the pixel matrix of the photosensitive sensor.

[0031] The data bus and the control block provide the functionality of a communication network for routing the digital data output from the conversion block to the various computing units. It is important to note that the digital data is routed "directly" from the conversion block to the various computing units. This means that, in the proposed intelligent imager architecture, there is no intermediate buffer memory for storing the digital pixel data between an ADC and the processing processor. In other words, the digital data is routed directly from the conversion block to the shared memories of the computing units without being stored in another intermediate memory.

[0032] The elementary processors of the calculation units of the processing processor are intended to be configured to read digital data stored in the shared memory of the calculation unit to which they belong, and to execute an image analysis algorithm directly from the data read (for example a artificial intelligence algorithm). This direct interface to the computing units, without an intermediate buffer, allows pixel data to be processed as soon as possible after digital conversion, which allows for a substantial gain in processing latency. Doing without an intermediate buffer also limits the footprint of the imager and the power consumed by the imager.

[0033] The control block is configured such that digital data representative of the same row of pixels are all routed simultaneously to different computing units. Thus, the computing units having received digital data relating to one or more rows of pixels of the matrix can begin their processing while digital data relating to one or more other rows of pixels are routed to other computing units. This also makes it possible to reduce the latency in the image analysis processing.

[0034] Digital data at the output of the conversion block may be representative of a single pixel or of a group (a zone) of pixels of the matrix of the photosensitive sensor.

[0035] In particular embodiments, the conversion block comprises pixel grouping blocks, each pixel grouping block being connected to several analog-digital converters associated with several successive columns of the pixel matrix, each pixel grouping block being configured to provide digital data representative of a group of pixels of the pixel matrix.

[0036] For example, a digital data item may be representative of a group of M x N pixels, where M is a divisor number (strictly lower) of the number of columns of the matrix and N is a divisor number (strictly lower) of the number of rows of the matrix. The digital data item may in particular correspond to an average (possibly weighted) of the M x N pixels. Such arrangements make it possible to reduce the complexity of the intensive analysis processing by reducing the quantity of digital data to be processed. This also allows a reduction in size and power consumed by the imager.

[0037] In particular embodiments, the control block is configured to route the same digital data to at least two distinct calculation units by transmitting said digital data only once on the data bus.

[0038] Such arrangements make it possible to implement a redundancy function (for example when the same digital pixel data is transmitted to at least two separate computing units) or a thumbnail processing function with overlapping areas (for example when only part of the digital pixel data intended for one computing unit is also transmitted to another computing unit). The fact of being able to transmit the same digital data to several separate computing units in a single transmission on the data bus avoids introducing latency in processing.

[0039] In particular embodiments, the calculation units are arranged in a matrix manner according to several rows and several columns, and the control block comprises several control sub-blocks, each control sub-block being connected to all the calculation units of the same column via a branch of the data bus.

[0040] It is thus possible to transmit the same digital data to several calculation units of the same column in one go.

[0041] In particular embodiments, the three-dimensional integrated circuit further comprises an optimization processor distinct from the parallel processing processor, implemented on one or more processing layers. Digital data output from the conversion block are routed to the optimization processor by a path distinct from the data bus.

[0042] The optimization processor is for example adapted to execute an optimization or error correction algorithm, and to output a stream of images on an output interface, such as for example a MIPI type interface (English acronym for “Mobile Industry Processor Interface”, this is an alliance dedicated to the development of interface specifications for mobile products).

[0043] The use of two separate paths to convey the digital data to the parallel processing processor and to the optimization processor allows for simultaneous image enhancement and intensive image analysis processing. Due to this decoupling, the image optimization and / or optical defect correction processing implemented by the optimization processor does not add additional latency to the intensive image analysis processing implemented by the parallel processing processor (the enhancement and defect correction implemented by the optimization processor is not necessary for the image analysis processing implemented by the parallel processing processor).

[0044] Different options are possible for routing the digital data output from the conversion block to the calculation units of the parallel processing processor.

[0045] In particular embodiments, the routing of the data is carried out “point to point”, or “point to multipoint”: the control block is then configured to encapsulate a digital data item in a message comprising at least one addressing identifier corresponding to at least one calculation unit, and to broadcast said message on the data bus to several calculation units; each calculation unit comprises a filtering block configured to detect, as a function of said at least one addressing identifier, whether or not the digital data item encapsulated in the message must be processed by the calculation unit.

[0046] In particular embodiments, the routing of the data is implemented by broadcast: each calculation unit then comprises a filtering block configured to detect, as a function of a number of digital data received, whether or not a digital data received must be processed by the calculation unit. Presentation of figures

[0047] The invention will be better understood on reading the following description, given by way of non-limiting example, and made with reference to the following figures which represent:

[0048] [Fig. 1] a schematic representation of the architecture of the intelligent imager according to the invention,

[0049] [Fig.2] a schematic representation of a first example of embodiment of the intelligent imager in the form of a three-dimensional integrated circuit comprising three stacked layers,

[0050] [Fig.3] a detailed schematic representation of the first embodiment described in [Fig.2],

[0051] [Fig.4] a schematic representation of the sensor layer of the smart imager,

[0052] [Fig.5] a schematic representation of a processing layer of the intelligent imager,

[0053] [Fig.6] a schematic representation of another processing layer of the intelligent imager,

[0054] [Fig.7] a schematic representation of a second embodiment of the intelligent imager in the form of a three-dimensional integrated circuit comprising five stacked layers,

[0055] [Fig.8] a detailed schematic representation of the second embodiment illustrated in [Fig.7],

[0056] [Fig.9] a schematic representation of a processing layer of the intelligent imager according to the second exemplary embodiment,

[0057] [Fig. 10] a schematic representation of a computing unit of the parallel processing processor integrated in the intelligent imager.

[0058] In these figures, identical references from one figure to another designate identical or similar elements. For reasons of clarity, the elements represented are not necessarily on the same scale, unless otherwise stated. Detailed description of the invention

[0059] [Fig.l] schematically represents a computing architecture of the intelligent imager according to the invention. This is a functional representation of different electronic components implemented on a three-dimensional (3D) integrated circuit forming the smart imager.

[0060] Figures 2 and 3 represent a first example of stacking three layers 10-1, 10-2, 10-3 of integrated circuit to form the three-dimensional integrated circuit 10 forming the intelligent imager.

[0061] The term "integrated circuit" means a microelectronic circuit which comprises active analog and / or digital electronic components made from a semiconductor material, as well as metal tracks allowing the electronic components to be electrically interconnected with each other. The expression "semiconductor material" encompasses any material having semiconductor properties (amorphous or polycrystalline semiconductor such as, for example, silicon, carbon nanotubes, semiconductor oxide, etc.).

[0062] As illustrated in [Fig.2], the three layers 10-1, 10-2, 10-3 are stacked on top of each other. Each layer of the stack is electrically connected to at least one other layer via through electrical connections 11. Such a connection 11 makes it possible to vertically cross the semiconductor substrate of a layer to electrically connect two electronic components made in two different layers of the stack).

[0063] As indicated previously, different manufacturing methods can be envisaged to produce the different layers 10-1, 10-2, 10-3, such as for example parallel 3D integration, sequential 3D integration, or a mix of these two techniques. In the case of parallel 3D integration, the through-electrical connections 11 can take the form of TSV (“Through-Silicon Vias” type connections); in the case of sequential 3D integration, the through-electrical connections 11 can take the form of MIV (“Monolithic Inter-tier Via”) type connections. The through-electrical connections 11 can also be produced via contact pads or microbeads.

[0064] The first layer 10-1 of the stack (upper layer located on top of the stack) is called the “sensor layer”. The sensor layer 10-1 is shown schematically in more detail in [Fig.4]. The sensor layer 10-1 comprises a photosensitive sensor comprising a matrix 20 of pixels 21. The photosensitive sensor can operate in the visible or infrared range, depending on the intended application.

[0065] Each pixel 21 of the matrix 20 of the photosensitive sensor corresponds to a photodetector. A photodetector is for example formed by a photodiode (PN junction), a phototransistor, a photodiode associated with transfer gates, or a photodetector made in an organic or colloidal photosensitive layer. Conventionally, each pixel is configured to provide an analog signal representing indicative of a received light intensity.

[0066] The photosensitive sensor can be produced according to a backside illumination configuration (BSI sensor). In a BSI configuration, the photodetectors are positioned at the upper face of the sensor layer 10-1 (face which is opposite the layer 10-2 located immediately below the sensor layer 10-1), and the metal tracks (routing) are positioned at the lower face of the sensor layer 10-1 (face which is opposite the layer 10-2 located immediately below the sensor layer). The BSI configuration makes it possible to optimize the capture of photons by the photodetectors.

[0067] According to another example, the photosensitive sensor can be produced according to a front-face configuration (FSI sensor). In an FSI configuration, the photodetectors are positioned at the lower face of the sensor layer 10-1, and the metal tracks (routing) are positioned at the upper face of the sensor layer 10-1. The FSI configuration makes it possible to simplify the manufacture of the sensor layer. In return, the routing part reflects a portion of the light and reduces the quantity of photons detected by the photodetectors.

[0068] In the example considered, the matrix 20 comprises 3072 rows of 4096 pixels (the matrix 20 therefore comprises 4096 columns of 3072 pixels, it is a matrix of 4096 x 3072 pixels). Each pixel 21 occupies an area of ​​1 pm2 (one square micrometer). The silicon area of ​​the sensor layer 10-1 is approximately 12 mm2. The thickness of the sensor layer 10-1 is less than 300 pm.

[0069] The various functional blocks described in [Fig.l] are implemented on layers 10-2 and 10-3 which are located below the sensor layer 10-1. Layers 10-2 and 10-3 located below the sensor layer are called “processing layers”.

[0070] [Fig. 5] schematically represents the processing layer 10-2 located immediately below the sensor layer 10-1. The layer 10-2 corresponds to the second layer of the stack forming the three-dimensional integrated circuit 10. [Fig. 6] schematically represents the processing layer 10-3 located below the second layer 10-2. The layer 10-3 corresponds to the third layer of the stack forming the three-dimensional integrated circuit 10.

[0071] As illustrated in [Fig.l], the three-dimensional integrated circuit 10 forming the intelligent imager according to the invention comprises a conversion block 30, a parallel processing processor 40, a data bus 50 and a control block 60.

[0072] The conversion block 30 has the function of transforming into digital data the analog signals supplied by the pixels 21 of the matrix 20 of the photosensitive sensor. In the example considered, and as illustrated in FIGS. 3 and 5, the conversion block 30 is implemented on the processing layer 10-2 located immediately under the sensor layer 10-1. The conversion block 30 is also referred to as “CNV” in the figures.

[0073] It should be noted that the conversion block 30 could also be implemented on another layer, for example on a lower processing layer, or even on the sensor layer 10-1. Nothing also prevents the conversion block 30 from being implemented on several layers (for example different parts of the conversion block could be implemented on several processing layers, or a part of the conversion block could be implemented on the sensor layer and other parts of the conversion block could be implemented on one or more processing layers).

[0074] As illustrated in [Fig.l], the conversion block 30 comprises analog-to-digital converters 31 (ADCs 31). The conversion block 30 may also comprise other electronic components, such as for example an analog signal amplification and shaping circuit, a pixel row or column decoder, a pixel reading circuit (“readout circuit”), etc. These components are not shown in the figures. The ADCs 31 of the conversion block are arranged at the bottom of the columns of the pixel matrix 20 of the photosensitive sensor (“column ADC” type arrangement). In other words, each column of pixels of the photosensitive sensor is associated with an ADC 31. In the example considered, the conversion block 30 therefore comprises 4096 ADCs.

[0075] As schematically illustrated in [Fig.2], each column of pixels of the matrix 20 of the photosensitive sensor of the sensor layer 10-1 is electrically connected to an ADC 31 of the conversion block 30 via a through electrical connection 11 (via).

[0076] Conventionally, each ADC 31 is adapted to digitize the analog signals of the pixels of the column to which the ADC is attached (each analog signal being representative of a light intensity received by a pixel).

[0077] In the example considered, the parallel processing processor 40 is configured to execute intensive image analysis processing based on an artificial intelligence algorithm, for example for shape, object, or face recognition, or for event detection in images (for example motion detection). The artificial intelligence algorithm is for example a deep neural network type algorithm.

[0078] In order to be able to be implemented under the photosensitive sensor, the processing processor 40 must respect strong constraints in terms of power consumption, thermal heating, surface area, and computing capacity. To provide a large computing capacity, within the constraints of power, thermal heating and surface area, the processing processor 40 follows a parallel architecture organized in a plurality of groups (“clusters” in English) of elementary processors. Thus, and as illustrated in [Fig.l], the parallel processing processor 40 comprises a plurality of calculation units 41. Each calculation unit 41 comprises several elementary processors 42 and a memory 43 shared between the elementary processors 42 of the calculation unit 41.

[0079] To enable real-time image analysis, digital data representative of the sensor pixels must be routed to the distributed memories 43 of the parallel processing processor 40 with minimal latency.

[0080] It should be noted that the shared memory 43 may be a monolithic memory shared between the different elementary processors 42, or a memory comprising several memory banks shared between the different elementary processors 42 (the different memory banks possibly being able to be accessed simultaneously to increase the bandwidth).

[0081] In the example considered and described with reference to FIGS. 2 to 6, the parallel processing processor 40 is implemented partly on the processing layer 10-2 (i.e. on the second layer 10-2 of the stack, which also corresponds to the first processing layer) and partly on the processing layer 10-3 (i.e. on the third layer 10-3 of the stack, which also corresponds to the second processing layer). As illustrated in FIGS. 1, 5 and 6, the calculation units 41 of the parallel processing processor 40 are arranged in a matrix manner in several rows and several columns. As illustrated in Figures 5 and 6, a portion of the computational unit lines 41 are implemented on the processing layer 10-2, and another portion of the computational unit lines 41 are implemented on the processing layer 10-3.

[0082] It should be noted that this example is in no way limiting. Nothing would prevent, for example, the parallel processing processor 40 from being implemented entirely on the processing layer 10-2 located immediately below the sensor layer (second layer of the stack). Nothing would also prevent it from being implemented on one or more processing layers located below the second layer 10-2 of the stack.

[0083] In the example considered, the parallel processing processor 40 is a PNeuro type processor as described in the document “PNeuro: a scalable energy-efficient programmable hardware accelerator for neural networks”, A. Carbon et al., Design, Automation And Test in Europe (DATA) 2018, pages 1045-1050. A computing unit 41 corresponds to a neural computing block (or NCB, English acronym for “Neural Computing Block”). The processing layer 10-2 comprises for example three lines of sixteen computing units 41. The processing layer 10-3 comprises for example three other lines of sixteen computing units 41. However, nothing would prevent one and / or the other of the processing layers 10-2 and 10-3 comprises another number (greater or smaller than three) of lines of computing units 41. Nothing would prevent a line of computing units 41 from comprising another number (greater or smaller than sixteen) of computing units 41. In the example considered, each of the processing layers 10-2 and 10-3 occupies a silicon surface of approximately 11 mm2. The thickness of the layer 10-2 is between 10 and 30 qm. The thickness of the layer 10-3 is of the order of 200 to 300 qm. The total thickness of the three-dimensional integrated circuit 10 is less than 500 qm.

[0084] The data bus 50 and the control block 60 provide the functionality of a communication network for routing the digital data output from the conversion block 30 to the shared memories 43 of the various computing units 41. It is important to note that the digital data is routed “directly” from the conversion block 30 to the various computing units 41. This means that, in the proposed intelligent imager architecture, there is no intermediate buffer memory for storing the digital pixel data between an ADC 31 and the processing processor 40. In other words, the digital data is routed directly from the conversion block 30 to the shared memories 43 of the computing units 41 without being stored in another intermediate memory. In the example considered with a PNeuro type processor, each shared memory 43 comprises four memory banks.

[0085] As illustrated in Figures 5 and 6, the data bus 50 and the control block 60 are implemented on the processing layers 10-2 and 10-3 on which the parallel processing processor 40 is implemented.

[0086] Each calculation unit 41 of the processing processor 40 is configured to process digital data corresponding to an area (a subset) of pixels 21 of the matrix 20 of the photosensitive sensor implemented on the sensor layer 10-1.

[0087] The elementary processors 42 of the calculation units 41 of the processing processor 40 are intended to be configured to read digital data stored in the shared memory 43 of the calculation unit 41 to which they belong, and to execute the image analysis algorithm directly from the read data. This direct interface to the calculation units 41, without intermediate buffer memory, makes it possible to process the pixel data as soon as possible after the digital conversion.

[0088] The control block 60 is configured such that digital data representative of the same row of pixels are all routed simultaneously to different calculation units 41. Thus, the calculation units 41 having received digital data relating to one or more rows of pixels of the matrix can begin their processing while digital data relating to one or more other rows of pixels are routed to other calculation units. This allows to significantly limit latency in image analysis processing.

[0089] As illustrated in Figures 1, 5 and 6, the control block 60 comprises several control sub-blocks 61. The control sub-blocks 61 may in particular encapsulate the data to be transmitted in messages comprising at least one destination address (address identifying a calculation unit 41 among the different calculation units 41 of the parallel processing processor 40). The control sub-blocks 61 may also transmit control and / or synchronization information (for example a signal making it possible to signal the start or end of processing of an image). In the figures, the control sub-blocks 61 are designated by the acronym ACS (acronym for “Address, Control and Synchronization”). Advantageously, and as illustrated in Figures 1, 5 and 6, each control sub-block 61 can be connected, via a branch of the data bus 50, to all the calculation units 41 of the same column of calculation units 41.

[0090] As illustrated in [Fig.2], two control sub-blocks 61 located on two different processing layers 10-2, 10-3 can be connected by through-connections 11. This makes it possible to provide the pixel data to all the calculation units 41 of the processing processor 40 distributed over the different processing layers.

[0091] It should be noted that digital data at the output of the conversion block 30 can be representative of a single pixel 21 or of a group (a zone) of pixels 21 of the matrix 20 of the photosensitive sensor.

[0092] In the example considered, and as illustrated in [Fig.l], the conversion block 30 comprises five hundred and twelve pixel grouping blocks 32. A grouping block 32 is designated by the letter “B” (for “binning” in English) in [Fig.l]. Each pixel grouping block 32 is connected to eight ADCs 31 associated with eight successive columns of the pixel matrix 20. Each grouping block 32 is configured to provide digital data representative of a group of 8 x 8 pixels 21 of the pixel matrix 20. In such a case, digital data at the output of a grouping block 32 is representative of an area of ​​sixty-four (8x8 = 64) pixels of the photosensitive sensor. This digital data is for example a weighted average of the sixty-four pixels processed by the grouping block 32.This has the effect of reducing the overall image resolution by 8 x 8 (the resolution is reduced from 4096 x 3072 to 512 x 384) while keeping the visual information in the image. As the resolution is reduced, the intensive analysis processing requires less memory and fewer calculations, so processing time and power consumption are reduced. The grouping block 32 performs very simple online and on-the-fly processing that does not introduce latency. Every eight ADC clock cycles, a new digital data representative of a group of 8 x 8 pixels is produced.

[0093] Each control sub-block 61 simultaneously receives thirty-two digital data provided by thirty-two consecutive grouping blocks 32 (each digital data being representative of an area of ​​8 x 8 pixels). Each control sub-block 61 serializes these thirty-two digital data on the data bus 50 without loss of bandwidth (the operating frequency of a control sub-block 61 is therefore at least thirty-two times higher than that of an ADC). These thirty-two digital data are routed to at least one processing unit 41.

[0094] In the example considered, in which the processing processor 40 is organized into six lines of sixteen calculation units 41, the sixteen calculation units 41 of the same line each simultaneously receive thirty-two digital data items which represent a total of eight lines of pixels (i.e. 4096 x 8 pixels). After sixty-four ADC clock cycles, a line of calculation units 41 has received the digital data representing one-sixth of the pixels of an image (i.e. 512 lines of pixels, i.e. 4096 x 512 pixels). When they have received their digital data, the calculation units 41 of the same line can then begin their processing while other digital data relating to other lines of pixels are routed to other calculation units of another line.

[0095] It should be noted, however, that the use of the grouping blocks 32 is optional. Without the grouping blocks 32, a digital data item at the output of the conversion block 30 is representative of a single pixel 21 of the matrix 20.

[0096] In particular embodiments, the control block 60 may be configured to route the same digital data to at least two distinct calculation units 41 by transmitting said digital data only once on the data bus 50. It is thus possible to perform a redundancy function (for example when the same digital pixel data are transmitted to at least two distinct calculation units 41) or a thumbnail processing function with overlapping zones (for example when only part of the digital pixel data intended for a calculation unit 41 is also transmitted to another calculation unit 41). The fact of being able to transmit the same digital data to several distinct calculation units 41 in a single transmission on the data bus 50 avoids introducing latency into the processing.

[0097] As illustrated in [Fig.l], the three-dimensional integrated circuit 10 forming the imager may also optionally comprise an optimization processor 70 separate from the parallel processing processor 40. The optimization processor 70 is designated by “ISP” in the figures (for the English acronym “Image Signal Processor”, in French “image processor”).

[0098] The optimization processor 70 is for example adapted to execute an optimization or error correction algorithm, and to output a stream of images on a output interface (e.g. a MIPI interface).

[0099] In the example considered, and as illustrated in Figures 3 and 5, the optimization processor 70 is implemented on the processing layer 10-2. However, nothing would prevent the optimization processor 70 from being implemented on another processing layer, or from being implemented on several different processing layers.

[0100] Advantageously, the digital data output from the conversion block 30 are routed to the optimization processor 70 by a path distinct from the data bus 50, for example by another data bus distinct from the data bus 50 which feeds the shared memories 43 of the calculation units 41 of the parallel processing processor 40.

[0101] The use of two separate paths to route the digital data to the parallel processing processor 40 and to the optimization processor 70 allows for simultaneous image enhancement and intensive image analysis processing. Thanks to this decoupling, the image optimization and / or optical defect correction processing implemented by the optimization processor 70 does not add additional latency to the intensive image analysis processing implemented by the parallel processing processor 40.

[0102] It should be noted that the embodiment described above with reference to FIGS. 2 to 6 is a non-limiting example. In particular, the number of processing layers in the stack forming the three-dimensional integrated circuit 10 could be different (for example a single processing layer, or more than two processing layers). The same applies to the number of calculation units 41 of the parallel processing processor 40.

[0103] Figures 7 to 9 show another example embodiment of a smart imager according to the invention. In this example, the three-dimensional integrated circuit 10 comprises five layers: a sensor layer 10-1 and four processing layers 10-2 to 10-5. The sensor layer 10-1 is similar to that described above with reference to Figures 2 to 4. The processing layer 10-2 is similar to the processing layer 10-2 described above with reference to Figures 2, 3 and 5. The processing layers 10-3 and 10-4 are similar to the processing layer 10-3 described above with reference to Figures 2, 3 and 6.

[0104] The processing layer 10-5 is described in [Fig.9]. The processing layer 10-5 comprises a part of the parallel processing processor 40 as well as a part of the control block 60 and the data bus 50. In addition, the processing layer 10-5 comprises a mass memory 80 which can be used, for example, to store weights and parameters of one or more artificial intelligence algorithms intended to be executed by the parallel processing processor 40. This mass memory 80 can be produced using SRAM technology (an acronym for "Static Random Access Memory"). Access Memory") or with non-volatile technology to achieve more memory density. The 10-5 processing layer is connected to the 10-4 processing layer above it by through-hole connections 11 at the control sub-blocks 61, similarly to what was previously described with reference to [Fig. 2]. The 10-5 processing layer also occupies a silicon area of ​​approximately 11 mm2.

[0105] The three-dimensional integrated circuit 10 could also optionally comprise an approximate event detection preprocessor used to wake up the parallel processing processor 40 when an approximate event detection is detected. Such arrangements make it possible to place the parallel processing processor 40 in sleep mode as long as an approximate event detection is not detected by the preprocessor. The preprocessor may for example be implemented on the processing layer 10-2, and / or on one or more other processing layers. An additional pixel grouping step may possibly be envisaged to further reduce the resolution of the data provided to the preprocessor (for example a resolution of 108 x 76 pixels).

[0106] In the example illustrated in Figures 7 to 9, the thickness of each of the layers 10-2 to 10-4 is between 10 and 30 qm. The thickness of the layer 10-5 is of the order of 200 to 300 qm.

[0107] [Fig. 10] schematically represents an exemplary embodiment of a calculation unit 41 of the parallel processing processor 40. [Fig. 10] also illustrates how digital data output from the conversion block 30 can be routed to the shared memory of the calculation unit 41 with the data bus 50 under the control of a control sub-block 61 (ACS block).

[0108] In the example considered and illustrated in [Fig. 10], a multiplexer 45 makes it possible to select a type of data to be written in the shared memory 43. The multiplexer 48 makes it possible to select the address to be used for writing digital data, depending on whether the data to be written corresponds to pixel information coming from the data bus 50 or to information coming from the group of elementary processors 42. The arrow 50-1 represents digital data representative of one or more pixels transmitted on the data bus 50; the arrow 50-2 represents control or synchronization information transmitted by an ACS block on the data bus 50; the arrow 46 represents data coming from the group of elementary processors 42 to be written in the shared memory 43; the arrow 47 represents data read by the group of elementary processors 42.The calculation unit 41 comprises a filtering block 44 whose function is to detect whether or not a digital pixel data item transmitted on the data bus 50 must be processed by the calculation unit 41 and, if necessary, to determine a local address of the memory. 43 shared to which the data is to be written.

[0109] Different options are possible for routing the digital data output from the conversion block 30 to the different calculation units 41.

[0110] According to a first example, the routing of the data is carried out “point to point”, or “point to multipoint”. In this case, the control block 60 is configured to encapsulate a digital pixel data item in a message comprising at least one addressing identifier corresponding to at least one calculation unit 41 and to broadcast said message on the data bus 50 to several calculation units 41. The filtering block 44 is configured to detect, as a function of said at least one addressing identifier, whether or not the digital data item included in the message must be processed by the calculation unit 41.

[0111] According to a second example, the routing of the data is implemented by broadcasting. In this case, the filtering block 44 is configured to detect, as a function of a number of digital data received, whether or not a received digital data must be processed by the calculation unit 41. For example, the control block 60 is configured to emit a synchronization signal on the data bus 50 to indicate the start of the processing of a new image; the filtering block 44 knows that it will see a flow of digital data associated with this new image pass on the data bus; and the filtering block 44 also knows the ranks of the digital data that the calculation unit 41 to which it is attached must process in this flow.

[0112] The above description clearly illustrates that, through its various characteristics and their advantages, the present invention achieves the set objectives. In particular, the arrangement of the ADCs at the bottom of the column makes it possible to pool them efficiently on the columns of pixels. The surface area occupied by the ADCs is thus optimized and the reading circuit is simplified. The arrangement of the ADCs at the bottom of the column makes it possible to obtain a higher density of photodetectors in the pixel matrix, which makes it possible to improve the performance of the sensor (for example in terms of signal to noise or quantum efficiency).

[0113] The layer stacking structure also makes it possible to optimize performance (reduction in electrical consumption, high parallelization of tasks carried out by the circuits of the different layers).

[0114] The direct routing of the digital data from the conversion block 30 to the shared memories 43 of the calculation units 41 (without using an intermediate buffer memory) makes it possible to significantly limit the latency in the processing of the images, the size of the imager, and the power consumed by the imager.

[0115] The use of a parallel processing processor 40 makes it possible to respond to the constraints inherent in intensive image processing (constraints in terms of power consumption, thermal heating, surface area, and computing capacity). Advantageously, the digital data representative of the same line of pixels are all routed simultaneously to different computing units 41 of the parallel processing processor 40 in order to reduce latency in processing.

Claims

1. Claims Smart imager comprising a three-dimensional integrated circuit (10) comprising a stack of at least two layers (10-1, 10-2, 10-3) of integrated circuit, each layer being electrically connected to at least one other layer via through electrical connections (11), said layers comprising: - an upper layer (10-1), called a “sensor layer”, comprising a photosensitive sensor comprising a matrix (20) of pixels, each pixel (21) being configured to provide an analog signal representative of a received light intensity, - one or more lower layers (10-2, 10-3), called “treatment layers”, the three-dimensional integrated circuit (10) comprising: - a conversion block (30) implemented on the sensor layer (10-1) and / or on one or more processing layers (10-2, 10-3), the conversion block (30) comprising a plurality of analog-to-digital converters (31) each associated respectively with a column of the pixel matrix (20), the conversion block (30) being configured to transform the analog signals provided by the pixels (21) into digital data, - a parallel processing processor (40) implemented on one or more processing layers (10-2, 10-3), the parallel processing processor (40) comprising a plurality of calculation units (41), each calculation unit (41) comprising several elementary processors (42) and a memory (43) shared between the elementary processors (42) of the calculation unit (41), - a data bus (50) and a control block (60) implemented on the processing layer(s) (10-2, 10-3) on which the parallel processing processor (40) is implemented, and configured to route the digital data output from the conversion block (30) directly to the shared memories (43) of the calculation units (41), the digital data representative of the same line of pixels being all routed simultaneously to different computing units (41).

2. A smart imager according to claim 1 wherein the conversion block (30) is fully implemented on one or more processing layers (10-2, 10-3).

3. A smart imager according to claim 2 wherein the conversion block (30) is entirely implemented on a single processing layer (10-2).

4. A smart imager according to claim 3 wherein the conversion block (30) is entirely implemented on the processing layer (10-2) located immediately below the sensor layer (10-1).

5. A smart imager according to any one of claims 3 to 4 wherein the three-dimensional integrated circuit (10) comprises at least three layers (10-1, 10-2, 10-3), and the parallel processing processor (40) is implemented partly on the processing layer (10-2) which implements the conversion block (30) and partly on at least one other processing layer (10-3).

6. Intelligent imager according to any one of claims 1 to 5 wherein the conversion block (30) comprises pixel grouping blocks (32), each pixel grouping block (32) being connected to several analog-digital converters (31) associated with several successive columns of the pixel matrix, each pixel grouping block (32) being configured to provide digital data representative of a group of pixels (21) of the pixel matrix (20).

7. Intelligent imager according to any one of claims 1 to 6 wherein the control block (60) is configured to route the same digital data to at least two separate calculation units (41) by transmitting said digital data only once on the data bus (50).

8. An intelligent imager according to any one of claims 1 to 7 wherein the computing units (41) are arranged in a matrix manner in several rows and several columns, and the control block (60) comprises several control sub-blocks (61), each control sub-block (61) being connected to all the computing units (41) of a same column via a branch of the data bus (50).

9. Intelligent imager according to any one of claims 1 to 8 wherein the three-dimensional integrated circuit (10) further comprises an optimization processor (70) distinct from the parallel processing processor (40), implemented on one or more processing layers (10-2, 10-3), and wherein digital data output from the conversion block (30) are routed to the optimization processor (70) by a path distinct from the data bus (50).

10. Intelligent imager according to any one of claims 1 to 9 wherein the control block (60) is configured to encapsulate digital data in a message comprising at least one addressing identifier corresponding to at least one calculation unit (41), and to broadcast said message on the data bus (50) to several calculation units (41), and wherein each calculation unit (41) comprises a filtering block (44) configured to detect, as a function of said at least one addressing identifier, whether or not the digital data encapsulated in the message must be processed by the calculation unit (41).

11. Smart imager according to any one of claims 1 to 9 wherein each computing unit (41) comprises a filtering block (44) configured to detect, depending on a number of received digital data, whether or not a received digital data must be processed by the computing unit (41).

Citation Information

Patent Citations

  • Three-dimensional microelectronic circuit with optimised distribution of its digital and analogue functions

    EP3971979A1

  • Image sensor

    WO2014057106A1

  • Stacked backside-illuminated quanta image sensor with cluster-parallel readout

    WO2017161060A1

  • Programmable visual chip-based visual image processing system

    CN102665049B

  • High density parallel proximal image processing

    US20190297295A1