Intelligent image sensor

Abstract

An intelligent image sensor comprises an image sensor and a storage and calculation integrated system which are integrated in a bonding mode, the image sensor is configured to carry out image acquisition and processing, and image data obtained through processing are transmitted to the storage and calculation integrated system; and the storage and calculation integrated system is configured to perform image processing on the image data and store an image processing result. The intelligent image sensor provided by the embodiment of the invention can realize the functions of image recognition, image classification and the like quickly with low power consumption.

Description

Technical Field

[0001] The present disclosure relates to, but is not limited to, semiconductor technology, and more particularly, to an image sensor. Background Art

[0002] Current image sensors do not have the ability to process images. When faced with application scenarios such as image recognition and image classification, Figure 1 As shown, the image sensor 101 can often only transmit the collected image data to the processor 104 (such as a general-purpose central processing unit) through the I / O interface 102. When the processor 104 performs relevant calculations, the image data needs to be repeatedly transmitted between the memory 105 and the processor 104 under the control of the controller 103, which will lead to overall low efficiency and high energy consumption. Summary of the invention

[0003] The present disclosure provides an intelligent image sensor, including an image sensor and a storage and computing system integrated by bonding, wherein:

[0004] The image sensor is configured to collect and process images, and the processed image data is transmitted to the storage and computing integrated system;

[0005] The integrated storage and computing system is configured to perform image processing on the image data from the image sensor and store the result of the image processing.

[0006] The embodiment of the present disclosure integrates the image sensor and the storage and computing integrated system through bonding. The storage and computing integrated system itself provides computing and storage functions and has image processing capabilities. Therefore, image processing functions such as image recognition and classification can be implemented in the intelligent image sensor without transmitting to an external memory and processor. Compared with traditional image processing methods, the intelligent image sensor of the present embodiment can realize image recognition, image classification and other functions quickly and with low power consumption. The intelligent image sensor of the embodiment of the present disclosure can be widely used in various mobile devices.

[0007] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. Other advantages of the present disclosure can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings are used to provide an understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0009] Figure 1The architecture diagram of the system on which the traditional image sensor is based;

[0010] Figure 2 A module diagram of an intelligent image sensor according to an embodiment of the present disclosure;

[0011] Figure 3 A module diagram of a smart image sensor according to another embodiment of the present disclosure;

[0012] Figure 4 This is a module diagram of a smart image sensor according to another embodiment of the present disclosure;

[0013] Figure 5 A module diagram of a computing module, a storage module, and a read / write module according to an embodiment of the present disclosure;

[0014] Figure 6 This is a schematic diagram of the structure of an intelligent image sensor according to an embodiment of the present disclosure;

[0015] Figure 7 The figure is a schematic diagram of information transmission of the intelligent image sensor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is apparent to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0017] In the description of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment described as "exemplary" or "for example" in the present disclosure should not be interpreted as being more preferred or advantageous than other embodiments. "And / or" in this article is a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "Multiple" means two or more than two. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not limit them to be necessarily different.

[0018] In this article, "including any one or more of the following: option 1, option 2,..." or "including any one or more of option 1, option 2,..." means including any one of the listed options, or any combination of multiple options. For example, "including any one or more of the following: A, B" or "including any one or more of A and B" means including only A, or only B, or including A and B; for another example, "including any one or more of the following: A, B, C" or "including any one or more of A, B, C" means including only A, or only B, or only C, or including A and B, or including A and C, or including B and C, or including A, B and C. The same applies to more options.

[0019] When describing representative exemplary embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those of ordinary skill in the art can easily understand that these sequences can change and still remain within the spirit and scope of the disclosed embodiments.

[0020] Currently, commonly used artificial imaging hardware systems such as image sensors use charge-coupled device (CCD) arrays and complementary metal oxide semiconductor (CMOS) arrays. Such image sensors transmit image data serially to the memory and processing unit through a bus for interactive operations (i.e., von Neumann architecture). In recent years, the application of artificial intelligence (AI) has become more and more extensive, which has put forward higher requirements on the computing power and energy efficiency of the system. When current image sensors, processors and memories work together to perform complex AI tasks, their power consumption and latency have become issues that cannot be ignored, and new hardware architectures are needed to meet the needs of AI.

[0021] An embodiment of the present disclosure provides a smart image sensor, such as Figure 2 As shown, the image sensor 200 and the storage and computing integrated system 300 are included, and the image sensor 200 and the storage and computing integrated system 300 are integrated by bonding, wherein:

[0022] The image sensor 200 is configured to collect and process images, and the processed image data is transmitted to the storage and computing integrated system 300;

[0023] The storage and computing integrated system 300 is configured to perform image processing on the image data from the image sensor 200 and store the result of the image processing.

[0024] The intelligent image sensor of the disclosed embodiment integrates a storage and computing system on the basis of an image sensor. Compared with a traditional image sensor, the storage and computing system itself provides computing and storage functions, and therefore has image processing capabilities. Image processing functions such as image recognition and classification can be implemented in the intelligent image sensor without the need to transfer to an external memory and processor. Compared with traditional image processing methods, it is faster and consumes less energy. The image sensor of this embodiment is integrated with the storage and computing system by bonding, and the two are packaged together, with a simple structure and high density.

[0025] In an exemplary embodiment of the present disclosure, the image processing performed by the storage and computing integrated system includes any one or more of the following:

[0026] Image recognition;

[0027] Image classification;

[0028] Object detection.

[0029] The above-mentioned image recognition may be face recognition, object recognition, etc. Image classification may be to automatically classify the collected images into a set of predefined categories according to certain classification rules. Target detection is to identify and locate the target object in the image. There may be one or more target objects. The result of target detection may be the location and category information of the target object. However, this is only exemplary. The image processing disclosed in the present invention is not limited to this. It may also be image compression, image enhancement, image segmentation (such as semantic segmentation, instance segmentation), image transformation, image denoising or image rotation, etc.

[0030] In an exemplary embodiment of the present disclosure, Figure 2As shown, the intelligent image sensor includes, in addition to the image sensor 200 and the storage-computing integrated system 300, an I / O interface electrically connected to the storage-computing integrated system 300. In the illustrated example, the I / O interface includes a mobile industry processor interface (MIPI: Mobile Industry Processor Interface, abbreviated as MIPI) 400. The MIPI interface 400 is configured to implement information interaction between the storage-computing integrated system 300 and an external device, wherein the information interaction includes but is not limited to sending the image processing result obtained by the storage-computing integrated system 300 to an external device, receiving data and image processing instructions sent by an external device to the storage-computing integrated system 300, and receiving control instructions sent by an external device to the image sensor 200, wherein the control instructions sent to the image sensor may be, for example, enable / disable instructions, exposure time setting instructions, frame rate setting instructions, resolution setting instructions, pixel bit depth setting instructions, etc.

[0031] By using the MIPI interface, the smart image sensor of the embodiment of the present disclosure can be widely used in various mobile devices to quickly and low-power realize image acquisition and image processing functions such as image recognition and image classification. However, the smart image sensor of the embodiment of the present disclosure can also use other types of I / O interfaces or add other I / O interfaces, such as one or more of Ethernet interfaces, WLAN interfaces, and Bluetooth interfaces.

[0032] In an exemplary embodiment of the present disclosure, Figure 2 As shown, the image sensor 200 includes a pixel array 201, an analog signal processor 203 and an analog-to-digital converter (ADC) 205, wherein:

[0033] The pixel array 201 is configured to capture an image and obtain an analog signal of the captured image;

[0034] an analog signal processor 203, configured to process the analog signal; and

[0035] The analog-to-digital converter (ADC) 205 is configured to convert the analog signal processed by the analog signal processor 203 into a digital signal, and transmit the digital signal as the image data to the storage and computing system 300.

[0036] In this embodiment, the storage and computing integrated system 300 includes a control module 301, a read-write module 302, a computing module 303 and a storage module 305, wherein:

[0037] The computing module 303 is configured to perform image processing on the image data from the image sensor 200 to obtain a result of the image processing;

[0038] The storage module 305 is configured to store the image processing data, including the image data from the image sensor 200 and the image processing results. In addition, the storage module 305 can also store the intermediate data of the calculation, the data of the image processing model used for image processing, and the data used for training the image processing model, etc.;

[0039] The reading and writing module 302 is configured to read and write data from the storage module, select a computing module and a storage module for performing image processing tasks, and send a control signal to the selected computing module and storage module;

[0040] The control module 301 is configured to issue image processing tasks to the read-write module 302, and to send control signals to the calculation module 303 and the storage module 305 through the read-write module 302, so as to control the calculation module 303 to perform related operations of image processing, and control the storage module 305 to store the results of the image processing.

[0041] Based on the above structure, the image signal collected by the pixel array 201 is processed by the analog signal processor 203 (such as gain control, noise filtering, image enhancement, etc.), and then converted into a digital signal by the ADC 205 and transmitted to the storage and computing integrated system 300. In the storage and computing integrated system 300, the control module 301 controls the opening and closing of other modules through control signals. For example, through the read-write module 302, the storage module 305 and the calculation module 303 are controlled to calculate and store the image data respectively, so as to realize image processing such as image recognition and image classification. The image processing result can be sent to an external device through the MIPI interface 400. The read-write module 302 can write the data from the image sensor into the storage module 305 through addressing, read the data of the image processing result in the storage module 305 and transmit it to the external system, and can also select the calculation module and storage module that perform the image processing task, and transmit the corresponding control signal.

[0042] When the smart image sensor of this embodiment includes other systems, the MIPI interface can also be used for data interaction between the storage and computing system and the other systems.

[0043] In an exemplary embodiment of the present disclosure, the storage-computing integrated system can perform image processing based on the model, and the training of the model can also be completed in the storage-computing integrated system.

[0044] In this embodiment, the storage and computing integrated system is further configured to: receive image data collected by the image sensor or input by an external device, and save it as a training data set; train a pre-stored image processing model based on the training data set, and save the trained image processing model;

[0045] The storage and computing integrated system performs image processing on the image data and stores the results of the image processing, including: using the image data from the image sensor as input data of the trained image processing model, and saving the output data after operation of the trained image processing model as the result of the image processing.

[0046] The image processing model of this embodiment can be an image recognition model, an image classification model, etc. The image processing model used for training can be sent to the intelligent image sensor by an external device through an interface and stored in the storage module of the storage and computing integrated system. The storage module can also store training data sets, intermediate training data, trained image processing models, etc., and the computing module is also configured to perform calculations related to model training. The above-mentioned training data sets can be divided into training data sets and verification data sets, wherein the training data sets are used to train the image processing model, and the verification data sets are used to verify the performance of the trained image processing model. After the verification is passed, the trained image processing model is obtained. The control module can select the corresponding computing module and storage module through the read-write module to complete the training and verification of the model, and save the trained image processing model.

[0047] The following uses image recognition as an example to illustrate the working process of the intelligent image sensor according to an embodiment of the present disclosure:

[0048] There are two ways to perform image recognition. One is to collect a large number of similar images by an image sensor and store them in a storage and computing system for training. After the training is completed, the image is recognized. The specific process is: the pixel array collects a large number of similar image signals, which are preprocessed by an analog signal processor, and then the analog image signals are converted into image data in the form of digital signals via ADC. After the image data is transmitted to the storage and computing system, a training data set is formed and stored in the storage module. The computing module trains the pre-stored image recognition model based on the training data set. The image recognition model can be a convolutional neural network (CNN), a recurrent neural network (RNN), a densely connected network (DenseNet), a feedforward neural network (FNN), etc., but is not limited to this. The trained image recognition model is stored in the storage module. When performing image recognition, the pixel array collects a large image signal, which is preprocessed by an analog signal processor, and then the analog signal is processed into image data in the form of digital signals via ADC. After the image data is transmitted to the storage and computing system, it is recognized by the image recognition model in the chip, and the recognition result is stored in the storage module. Using the data collected by the internal image sensor as a training data set can make the trained image recognition module have good adaptability to the data generated after the image sensor is collected and processed, and can improve the accuracy of image recognition.

[0049] Another way is for an external device to directly input the training data set to the storage and computing system through the I / O interface. After the training data set is imported into the chip, the image recognition model is trained under the control of the control module. The trained image recognition model is stored in the storage module, and the subsequent recognition process is the same as the previous method. In addition to the image recognition module, other models for image processing such as image classification and target detection can also be trained and used through the above two methods.

[0050] If the image processing to be performed is image classification, there may be one or more data sets for classification. The image classification model is used to compare the image recognition results with the data sets used for classification to obtain the classification results. The image classification process can also be combined with the recognition process, using one model to complete both processes, depending on the complexity of the neural network and the size of the data set.

[0051] In this embodiment, the image processing model is trained in the storage-computing system. However, in another embodiment, the trained image processing model can be sent to the storage-computing system by an external device, and the storage-computing system performs image processing based on the sent image processing model. In other embodiments, the storage-computing system does not need to use a model when performing image processing.

[0052] In an example of this embodiment, the computing module 303 and the storage module 305 are implemented by a random access memory unit embedded with computing power, and data can be directly exchanged between the computing module 303 and the storage module 305. The random access memory can use a memristor device, such as a magnetic random access memory (MRAM: Magnetoresistive Random Access Memory), a resistive random access memory (RRAM: resistance random access memory), a phase-change memristor (PRAM: phase-change random-access memory), etc., see Figure 3 and Figure 4 By using the in-memory computing method, the computing module 303 and the storage module 303 are physically integrated (ie, physically integrated together), and direct data exchange can be performed between the two, thereby avoiding the decrease in efficiency and increase in energy consumption caused by data transmission between the CPU and the memory.

[0053] In another exemplary embodiment of the present disclosure, Figure 4 As shown, the storage and computing integrated system 300 includes, in addition to the control module 301, the computing module 303 and the storage module 305, a bus 304 connected to the control module 301, the computing module 303 and the storage module 305. The bus 304 is configured to transmit the image data output by the image sensor 200 to the storage module 305, and transmit the control signal sent by the control module 301 to the computing module 303 and the storage module 305. The control module 301 can also control the data stored in the storage module 305 to be transmitted to other systems via the MIPI interface 400 through the bus 304. However, in other embodiments, it is not necessary to adopt the form of a bus, and other forms of lines that can transmit data and instructions can also be used.

[0054] In this embodiment, the computing module 303 is an RRAM computing module, the storage module 305 is an RRAM storage module, and the RRAM computing module and the RRAM storage module are physically integrated, which can be implemented by using RRAM with in-memory computing function, and direct data exchange can be performed between the RRAM computing module and the RRAM storage module. The storage and computing integrated system 300 may also include other modules.

[0055] In another exemplary embodiment of the present disclosure, Figure 4 As shown, the computing module 303 and the storage module 305 in the storage-computing integrated system 300 are also implemented by random storage units with embedded computing capabilities, such as MRAM with in-memory computing functions. The computing module 303 is an MRAM computing module, and the storage module 305 is an MRAM storage module. The MRAM computing module and the MRAM storage module are physically integrated and can directly exchange data.

[0056] In this embodiment, the random storage unit embedded with computing power is divided into multiple areas, each of which includes at least one computing unit and one storage unit, the computing units in each area constitute a computing module 303, and the storage units in each area constitute a storage module 305. Each computing module 303 can be independently controlled to complete the assigned computing task, and each storage module 305 can be independently controlled to complete the assigned storage task, and any computing module 303 and any storage module 305 are electrically connected to form a mesh structure for data transmission. Under the control of the control unit 301, the read-write module 302 can select different combinations of computing modules 303 and storage modules 305 to complete different image processing tasks (including computing and storage), and adapt to different load conditions of the current computing modules 303 and storage modules 305.

[0057] like Figure 5 As shown, the read-write module 302 of this embodiment includes:

[0058] The monitoring unit 3023 is configured to detect the utilization of the computing module 303 and the storage module 305 in the storage-computing integrated system in real time;

[0059] The allocation unit 3025 is configured to perform dynamic task allocation to select the computing module 303 and the storage module 305 that execute the image processing task according to the image processing task issued by the control module 301 and the utilization rate of the detected computing module 303 and the storage module 305;

[0060] The read / write unit 3021 is configured to send control signals to the selected computing module 303 and storage module 305 through addressing.

[0061] by Figure 5 Taking the example that the memristor device shown includes M computing modules and N storage modules, M and N may be equal or unequal, that is, separate computing modules and storage modules may be set. The allocation unit may select m computing modules and n storage modules to perform image processing tasks, and m may be greater than n, or equal to n, or less than n. The selected computing modules and storage modules may all be located in the same area, or in different areas, or one part may be located in the same area and the other part may be located in different areas. This embodiment is not limited to this.

[0062] This embodiment integrates monitoring and dynamic allocation functions in the read-write module. The monitoring unit is used to perform real-time detection of the utilization of the computing modules and storage modules in the storage-computing integrated system, and the allocation unit can schedule the computing modules and storage modules through a dynamic task allocation algorithm based on the resources required for the image processing task. Each computing module and each storage module has an independent address, and the computing modules and storage modules located in different areas can be distinguished by site selection. Therefore, the read-write unit can send control signals to the computing modules and storage modules that perform image processing tasks through addressing. This embodiment can maintain load balance between computing modules and storage modules through detection and dynamic task allocation. Improve the overall computing efficiency and service life of the storage-computing integrated system.

[0063] In another embodiment, the computing module 303 and the storage module 305 may not directly exchange data, but may exchange data between the computing module 303 and the storage module 305 through the control of the control module 301 .

[0064] like Figure 6 As shown, it is a schematic diagram of the structure of an intelligent image sensor according to an embodiment of the present disclosure. The intelligent image sensor includes an interposer, and the silicon interposer 100 is used as an example in the figure but is not limited thereto. The pixel array 201, the analog signal processor 203 and the ADC 205 of the image sensor 200 are connected to the silicon interposer 100 by bonding, and the computing module 303, the storage module 305 and the read-write module 302 of the storage-computing integrated system 300 are connected to the silicon interposer 100 by bonding, and the control module 301 of the storage-computing integrated system 300 can be integrated on the silicon interposer. The image sensor 200 and the storage-computing integrated system 300 are connected as a whole and uniformly packaged through the silicon interposer 100. The silicon interposer 100 can be used as a carrier of these systems, and can also realize electrical connection between different systems, and can also integrate some functional modules such as control modules in these systems. For example, in another embodiment, a control module can be set in the image sensor 200, and the control module can be integrated in the silicon interposer 100 to control the pixel array, the analog signal processor and the ADC to be turned on or off, control data transmission, and perform parameter configuration.

[0065] Figure 6In the example shown, the analog signal processor 203 and ADC205 can be integrated on the same layer, the pixel array 201 is located on another layer, the layer where the pixel array 201 is located is bonded from top to bottom to the layer where the analog signal processor 203 and ADC205 are located, and the layer where the analog signal processor 203 and ADC205 are located is bonded from top to bottom to the silicon interposer 100. In the storage and computing integrated system 300, the computing module 303 and the storage module 305 are integrated on the same layer, the read-write module 302 is located on another layer, the layer where the computing module 303 and the storage module 305 are located is bonded from top to bottom to the layer where the read-write module 302 is located, and the layer where the read-write module 302 is located is bonded from top to bottom to the silicon interposer 100. However, this is only exemplary. Figure 6 The through silicon vias 11 can realize electrical connection between the layers, and the bonding structure 13 between the layers can use flip-chip bonding, wafer bonding, flip-chip bonding, laser bonding, etc.

[0066] The I / O interface in the smart image sensor of this embodiment can be formed at another position of the silicon interposer 100, and the smart image sensor can also include other systems, such as a clock system, a power supply system, etc. The power supply system can provide power for the entire smart image sensor, and the clock system provides a clock source input for other systems in the smart image sensor. This is merely exemplary. The power supply system and the clock system can be independently arranged on the silicon interposer 100, or they can be integrated together. The silicon interposer 100 of this embodiment can be made of a flexible material and packaged into a 3D stacked chip together with the various systems it carries, but the present disclosure is not limited to this. This embodiment can make full use of space and facilitate manufacturing through the reasonable distribution of the above-mentioned functions on different layers.

[0067] like Figure 7 As shown, there may be multiple information transmission paths between the smart image sensor of the above embodiment of the present disclosure and the external device, wherein:

[0068] Path 1: The image sensor 200 transmits the signal to the storage and computing integrated system 300, and then transmits it to the external device 500 via the I / O interface (such as the MIPI interface 400).

[0069] Path 2: Data is imported into the storage and computing integrated system 300 through the external device 500 for training and calculation.

[0070] Path 3: The storage and computing integrated system 300 sends a signal to an external control system to control the intelligent image sensor, such as mechanical control.

[0071] It will be appreciated by those skilled in the art that the functional modules / units in all or some of the systems and devices disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. An intelligent image sensor, characterized in that: It includes an image sensor and a storage and computing system integrated by bonding, wherein: The image sensor is configured to collect and process images, and the processed image data is transmitted to the storage and computing integrated system; The integrated storage and computing system is configured to perform image processing on the image data from the image sensor and store the result of the image processing.

2. The smart image sensor according to claim 1, wherein: The image processing performed by the storage and computing system includes any one or more of the following: Image recognition; Image classification; Object detection is the process of identifying and locating objects in an image.

3. The smart image sensor according to claim 1, wherein: The image sensor comprises: A pixel array is configured to capture an image and obtain an analog signal of the captured image; an analog signal processor, configured to process the analog signal; ADC is configured to convert the analog signal processed by the analog signal processor into a digital signal, and transmit the digital signal as the image data to the storage and computing integrated system.

4. The smart image sensor according to claim 1, wherein: The storage and computing integrated system comprises: a computing module configured to perform image processing on the image data from the image sensor to obtain a result of the image processing; A storage module, configured to store the image processed data, including the image data and the image processing result; A read / write module configured to read and write data from the storage module, select a computing module and a storage module for performing image processing tasks, and send a control signal to the selected computing module and storage module; The control module is configured to issue image processing tasks to the read-write module, and send the control signal to the selected computing module and storage module through the read-write module to control the selected computing module to perform image processing, and control the selected storage module to store the result of the image processing.

5. The smart image sensor according to claim 1 or 4, characterized in that: The storage and computing integrated system is further configured to: receive image data collected by the image sensor or input by an external device, and save it as a training data set; train a pre-stored image processing model based on the training data set, and save the trained image processing model; The storage and computing integrated system performs image processing on the image data from the image sensor and stores the results of the image processing, including: using the image data from the image sensor as input data of the trained image processing model, and saving the output data after calculation of the trained image processing model as the result of the image processing.

6. The smart image sensor according to claim 4, characterized in that: The computing module and the storage module are implemented by a random storage unit embedded with computing power, and the random storage unit embedded with computing power is divided into multiple areas, each area includes at least one computing unit and one storage unit; the computing units in each area constitute an independently controllable computing module, and the storage units in each area constitute an independently controllable storage module, and any computing module is electrically connected to any storage module, so that data can be directly exchanged; The read / write module is configured to select at least one of the computing modules and at least one of the storage modules according to the image processing task sent by the control module, and send the control signal to the selected computing module and storage module to execute the image processing task.

7. The smart image sensor according to claim 6, characterized in that: The read-write module comprises: A monitoring unit, configured to perform real-time detection of utilization rates of the computing module and the storage module in the storage-computing integrated system; an allocation unit configured to perform dynamic task allocation to select a computing module and a storage module for executing the image processing task according to the image processing task and the utilization rates of the detected computing modules and storage modules; A read / write unit, configured to send the control signal to the selected computing module and storage module through addressing; The selected computing modules and storage modules are all located in the same area, or are all located in different areas, or one part is located in the same area and the other part is located in different areas.

8. The smart image sensor according to claim 4, wherein: The intelligent image sensor also includes an intermediary layer, the pixel array, analog signal processor and ADC of the image sensor are connected to the intermediary layer by bonding, the computing module, storage module and read-write module of the storage-computing integrated system are connected to the intermediary layer by bonding, and the control module of the storage-computing integrated system is integrated on the intermediary layer; the intermediary layer connects the image sensor and the storage-computing integrated system as a whole.

9. The smart image sensor according to claim 8, characterized in that: The layer where the pixel array is located is bonded to the layer where the analog signal processor and the ADC are located from top to bottom, and the layer where the analog signal processor and the ADC are located is bonded to the intermediate layer from top to bottom; The layer where the computing module and the storage module are located is bonded to the layer where the read-write module is located from top to bottom, and the layer where the read-write module is located is bonded to the intermediate layer from top to bottom.

10. The smart image sensor according to claim 9, characterized in that: The intelligent image sensor also includes an I / O interface, which is configured to realize information interaction between the storage and computing system and an external device. The information interaction includes: sending the image processing result to an external device, receiving data and image processing instructions sent by an external device to the storage and computing system, and receiving control instructions sent by an external device to the image sensor.

Images