Solid-state imaging device, electronic device, and imaging system

Abstract

The present invention improves the accuracy of recognition processing used in an image sensor. A solid-state imaging device includes a pixel array, a converter, an image processing unit, a digital signal processing unit, and a control unit. The pixel array has a plurality of pixels that perform photoelectric conversion. The converter converts analog pixel signals output from the pixel array into digital image data. The image processing unit performs image processing on the digital image data. The digital signal processing unit performs recognition processing on the digital image data output by the image processing unit. The control unit performs optimization of at least one acquisition process among acquisition of analog pixel signals, acquisition of digital image data, and acquisition of the result of the recognition processing based on the result of the recognition processing.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a related application of U.S. Provisional Patent Application No. 62 / 967,869 filed on January 30, 2020, and claims priority to that provisional application, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a solid-state imaging device, an electronic device, and an imaging system. Background Art

[0004] In recent years, it has been required to perform various signal processing at high speed on image data captured by an image sensor. In addition, with the development of semiconductor process technology, a semiconductor device in which a plurality of chips (such as an image sensor chip, a memory chip, and a signal processing chip) are connected to a package through bumps, and a semiconductor device in which a bare chip including an image sensor disposed therein and a bare chip including a memory, a signal processing circuit, etc. disposed therein are laminated to the package has been proposed.

[0005] In the case where a semiconductor device combining an image sensor and a signal processing circuit (hereinafter, referred to as an imaging device) is installed in an electronic device such as a smart phone, the signal processing circuit in the imaging device performs various signal processing in response to a command from an application processor installed in the electronic device in many cases. For example, by performing neural network processing in the signal processing circuit and outputting the result, it is possible to improve the delay in transmitting a captured image to the outside and then processing it, and improve privacy and security by transmitting the captured image to the outside. Therefore, a technology for improving the recognition rate of a neural network used when performing estimation processing, etc. using a built-in signal processing circuit is required.

[0006] [Citation List]

[0007] [Patent Document]

[0008] [Patent Document 1]

[0009] WO 2018 / 051809 A1. Summary of the invention

[0010] [Technical issues]

[0011] Therefore, the present disclosure provides a solid-state imaging device, an electronic device, and an imaging system capable of improving the processing accuracy of a neural network used in an image sensor.

[0012] [Solution to the problem]

[0013] According to one embodiment, a solid-state imaging device includes: a pixel array having a plurality of pixels that perform photoelectric conversion; a converter that converts analog pixel signals output from the pixel array into digital image data; an image processing unit that performs image processing on the digital image data; a digital signal processing unit that performs recognition processing on the digital image data output by the image processing unit; and a control unit that performs optimization of at least one acquisition processing operation among acquisition of analog pixel signals, acquisition of digital image data, and acquisition of a result of the recognition processing based on a result of the recognition processing.

[0014] The control unit may feed back the result of the recognition process to perform optimization.

[0015] When the recognition rate of the recognition process decreases, the control unit may perform optimization regarding at least one acquisition process operation among acquisition of the analog pixel signal, acquisition of the digital image data, and acquisition of the result of the recognition process.

[0016] The control unit may control the exposure time of the pixels of the pixel array.

[0017] The control unit may optimize parameters related to image processing by the image processing unit.

[0018] The digital signal processing unit may perform recognition processing using a trained neural network model.

[0019] The control unit can retrain the neural network model.

[0020] The solid-state imaging device may further include a semiconductor device including: a first substrate on which the pixel array is arranged; and a second substrate laminated on the first substrate and on which the converter, the image processing unit, the digital signal processing unit and the control unit are arranged.

[0021] The first substrate and the second substrate may be bonded by any one of a CoC (Chip on Chip) method, a CoW (Chip on Wafer) method, and a WoW (Wafer on Wafer) method.

[0022] The solid-state imaging device may further include a selector for selecting an output from at least one of the converter, the image processing unit, and the digital signal processing unit, and outputting data to the outside.

[0023] When outputting image information, only information in a region of interest may be selected and output.

[0024] When outputting image information, different compression methods may be used to output information in a region of interest and information in other regions.

[0025] The information in the area of ​​interest and the information in other areas may be compressed and output at different compression rates.

[0026] When the recognition rate in the recognition process has decreased, the image information and the result of the recognition process may be output together.

[0027] According to one embodiment, an electronic device includes the above-described solid-state imaging device and an application processor on a substrate independent of the semiconductor device, wherein the application processor inputs / outputs data via a selector.

[0028] According to one embodiment, an imaging system includes: the above-mentioned solid-state imaging device; and a server connected to the solid-state imaging device via a network, wherein the output of the solid-state imaging device output converter and / or the output of the image processing unit, the output of the digital signal processing unit is associated with the server, and the server performs optimization of at least one acquisition processing operation among the acquisition of the analog pixel signal, the acquisition of digital image data, and the acquisition of the result of the recognition processing based on the received information, and deploys the optimized information to the solid-state imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a block diagram showing an overview of an electronic device including an imaging device according to an embodiment.

[0030] Figure 2 is a block diagram illustrating a process of an imaging device according to an embodiment.

[0031] Figure 3 is a block diagram illustrating a process of an imaging device according to an embodiment.

[0032] Figure 4 is a block diagram showing an overview of an imaging system according to an embodiment.

[0033] Figure 5 is a diagram illustrating an example of an acquired image according to an embodiment.

[0034] Figure 6 is a diagram showing a configuration example of a semiconductor substrate of an imaging device according to an embodiment. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the imaging device and the electronic device will be described with reference to the accompanying drawings. Although the main components of the imaging device and the electronic device will be mainly described below, the imaging device and the electronic device may include components and functions that are not shown or explained. The following description is not intended to exclude components and functions that are not shown or explained.

[0036] (First Embodiment)

[0037] Figure 1 1 is a block diagram showing a schematic configuration of an electronic device 2 including an imaging device 1 according to an embodiment. The electronic device 2 includes the imaging device 1 and an application processor (hereinafter referred to as AP 20). The electronic device 2 is a smartphone, a mobile phone, a tablet terminal, a PC, a digital camera, a digital video camera, etc. having an imaging function, and the specific mode of the device is not limited.

[0038] The imaging device 1 may be implemented as a semiconductor device. The semiconductor device may also be referred to as an image sensor or a solid-state imaging device. The imaging device 1 includes at least a CMOS image sensor (hereinafter referred to as CIS 10), an image processing unit (hereinafter referred to as ISP 11), a digital signal processing unit (hereinafter referred to as DSP 12), a control unit (hereinafter referred to as CPU 13), a storage unit 14, a shutter 15, and a selector (hereinafter referred to as SEL 16).

[0039] The CIS 10 is an image sensor having an optical system 100 , an imaging unit including a pixel array 102 , and an analog-to-digital conversion circuit (hereinafter referred to as ADC 104 ).

[0040] The optical system 100 includes, for example, a zoom lens, a single focus lens, an aperture, etc. The optical system 100 guides incident light to a pixel array 102 .

[0041] The pixel array 102 has a plurality of pixels arranged in a two-dimensional direction. Each pixel may be composed of a plurality of unit pixels of a plurality of colors such as RGB. Each unit pixel has a light receiving element such as a photodiode. The light receiving element photoelectrically converts the incident light and outputs an analog pixel signal. The light incident on the imaging unit is imaged on a light receiving surface arranged with a plurality of light receiving elements via the optical system 100, and each light receiving element accumulates charge according to the intensity of the incident light and outputs an analog pixel signal corresponding to the accumulated charge amount.

[0042] The ADC 104 converts the analog pixel signal output from the pixel array 102 into digital image data. Since the ADC 104 performs A / D conversion, the ISP 11, the DSP 12, the CPU 13, the storage unit 14, the shutter 15, and the SEL 16 process the digital image data at the subsequent stage of the ADC 104. A voltage generating circuit that generates a driving voltage for driving the imaging unit from a power supply voltage or the like supplied to the imaging device 1 may be provided inside the ADC 104 or separately from the ADC 104. In addition, a digital-to-analog conversion circuit (DAC) required to realize the A / D conversion of the ADC 104 may be provided separately.

[0043] The ISP 11 performs various image processing on the digital image data. The ISP 11 may perform image processing on the digital image data output from the CIS 10, or may perform image processing on the digital image data output from the CIS 10 and stored in the storage unit 14. The ISP 11 performs image processing in response to an external instruction. For example, the ISP 11 converts the digital image data so that the data is suitable for signal processing by the DSP 12. Hereinafter, the digital image data is simply referred to as an image, image data, etc.

[0044] The DSP 12 has a function of an information processing unit that performs at least one processing operation such as a predetermined recognition process and a detection process based on the image data processed by the ISP 11. That is, the DSP 12 performs recognition processing, etc. on the data processed by the ISP 11, etc. The DSP 12 may perform predetermined processing based on the result of performing the recognition processing, etc. For example, the DSP 12 may input the image data processed by the ISP 11 into a pre-trained learning neural network model to perform object recognition, etc. Then, the DSP 12 may perform processing for improving the accuracy of the neural network model based on the result of the recognition, etc. The neural network model is trained, for example, by deep learning, but is not limited thereto, and may be another model that can be retrained. When pre-trained data is used, for example, the trained parameters may be stored in the storage unit 14, and a neural network model may be formed based on the parameters.

[0045] The CPU 13 controls each unit in the imaging device 1 according to instructions from the AP 20 or the like. The CPU 13 can perform predetermined processing based on a program stored in the storage unit 14. The CPU 13 can be integrated with the DSP 12. That is, the CPU 13 can be formed into a configuration capable of realizing the processing of the above-mentioned DSP 12. The CPU 13 can set the imaging condition of the CIS 10 based on the recognition result of the DSP 12. In the present disclosure, as an example, the exposure time is controlled, but as will be described later, the present invention is not limited to this processing, and various types of processing that can be processed by the ISP 11 can be controlled.

[0046] The DSP 12 performs arithmetic processing using a computational model of machine learning by, for example, executing a program stored in the storage unit 14. The ISP 11 and the CPU 13 can also perform various operations by executing a program stored in the storage unit 14. Various information on the learned computational model, various information for various types of processing, and programs are pre-stored in the storage unit 14. The ISP 11, the DSP 12, and the CPU 13 read necessary information from the storage unit 14 and perform arithmetic processing.

[0047] The processing related to the machine learning model performed by the DSP 12 is, for example, a computational model of a deep neural network (hereinafter referred to as DNN) trained by deep learning as described above. The computational model can be designed based on parameters generated by inputting the output data of the ISP 11 as input and inputting learning data with labels associated with the input into the learned computational model.

[0048] The DSP 12 may perform predetermined recognition processing by arithmetic processing using the DNN. Here, the recognition processing is a process of automatically recognizing whether image data, which is output data of the ISP 11, contains characteristic image information. More specifically, the recognition processing is a process for performing arithmetic processing by giving input data to a calculation model formed by parameters generated by machine learning, and the input data is output data of the ISP 11.

[0049] In the process of performing arithmetic processing based on the learned calculation model stored in the storage unit 14, the DSP 12 may perform a product-sum operation between the dictionary coefficients stored in the storage unit 14 and the image data. The calculation result of the DSP 12 is stored in the storage unit 14 and output to the SEL 16. The result of the arithmetic processing using the calculation model by the DSP 12 may be image data or various information (metadata) obtained from the image data. The above-mentioned DSP 12 or CPU 13 may have a memory controller function for controlling writing and reading to the storage unit 14, or the memory controller may be provided separately from the DSP 12 and the CPU 13. The DSP 12 may perform detection processing such as motion detection processing, face detection processing, etc. The detection processing may be performed by the ISP 11 instead of the DSP 12. Alternatively, the ISP 11 and the DSP 12 may perform the detection processing in collaboration.

[0050] The storage unit 14 stores the digital pixel data output from the CIS 10, the programs executed by the ISP 11, the DSP 12, and the CPU 13, and various information related to the learned calculation model used by the DSP 12 for arithmetic processing. The storage unit 14 can store data related to the progress of the calculation processing of each of the above-mentioned units and the calculation results. The storage unit 14 is, for example, a readable and writable random access memory (RAM). By exchanging information about the calculation model in the storage unit 14, each of the above-mentioned units can perform various operations, and highly general and widely applicable processing can be achieved. When each of the above-mentioned units performs arithmetic processing based on the calculation model for a specific purpose, the storage unit 14 may include a ROM (read-only memory) as a part thereof.

[0051] The shutter 15 controls the exposure time in the CIS 10 according to the control from the CPU 13. For example, if the CPU 13 determines that it is too bright for the DSP 12 to recognize, the shutter 15 shortens the exposure time. On the contrary, when it is dark, the shutter 15 increases the exposure time. The shutter 15 may not be set outside the CIS 10, but may be set inside the CIS 10. The shutter 15 may be an analog shutter or a digital shutter.

[0052] The SEL 16 selects and outputs output data processed by the DSP 12 based on a selection control signal from the CPU 13. The data to be output may be the output of the DSP 12 as it is, or may be data stored in the storage unit 14 by the DSP 12. In some cases, the SEL 16 may select and output digital pixel data output by the ADC 104 based on a control signal from the CPU 13. The SEL 16 outputs data required by the AP 20 via an interface such as MIPI (Mobile Industry Processor Interface) or I2C (Inter-Integrated Circuit).

[0053] The AP 20 is a semiconductor device separate from the imaging device 1, and is mounted on the same base substrate as the imaging device 1 or on a base substrate different from the imaging device 1. The AP 20 includes a CPU different from the CPU 13 of the imaging device 1, and executes programs such as an operating system, various application software, etc. The AP 20 may be provided with a DSP different from the DSP 12 of the imaging device 1, and various types of signal processing may be performed by the DSP. The DSP in the AP 20 may be able to perform more advanced signal processing at a higher speed than the ISP 11, DSP 12, etc. in the imaging device 1.

[0054] In addition, the AP 20 may be provided with a function of performing image processing, signal processing, etc. of a graphics processing unit (GPU), a baseband processor, etc. The AP 20 may perform various types of processing on the image data and calculation results from the imaging device 1 as needed, and may control to display the image together with the recognition result on the display unit of the electronic device 2. The AP 20 may transmit data related to the processing result or the recognition result to an external cloud server via a predetermined wired or wireless network.

[0055] Note that as the predetermined network, various communication networks such as the Internet, a wired local area network (LAN), a wireless LAN, a mobile communication network, and near-field wireless communication such as Bluetooth (registered trademark) can be applied, for example. The destination of the image data and the calculation result is not limited to the cloud server, but may be various information processing devices having a communication function, such as an independent server, a file server, and a communication terminal such as a mobile phone.

[0056] Figure 1An example is shown in which the AP 20 sends an instruction to the imaging device 1. Hereinafter, an example in which the AP 20 sends an instruction to the imaging device 1 will be described, but the actual implementation is not limited to this form, and the following description will be interpreted as including a case in which a processor other than the AP 20 sends an instruction to the imaging device 1.

[0057] Figure 2 is a flowchart showing an imaging process according to an embodiment. The imaging device 1 performs the imaging process based on the flowchart. For example, the imaging device 1 is installed so as to capture a predetermined range of a factory line. This is shown as an example, and the imaging device can be applied without being limited to this example as long as the recognition process is performed. In addition, the imaging device may perform other inference processes instead of the recognition process.

[0058] First, the imaging device 1 performs preprocessing (S100). The preprocessing is, for example, a process of determining whether an object to be recognized is properly acquired as image data by capturing, and setting capturing conditions and the like.

[0059] Figure 3 is a flow chart illustrating this pre-processing according to an embodiment.

[0060] First, as pre-processing, the CIS 10 performs photographing and acquires an image ( S1000 ).

[0061] Next, the ISP 11 performs appropriate image processing on the acquired image (S1002). For example, the image processing includes various types of filtering processing, such as sharpening processing, resolution conversion, gain adjustment, dynamic range conversion processing, area cropping processing, color correction, color conversion, parameters related to normalization processing, etc. In addition, in order to convert the image into an image suitable for the recognition processing described later, image enlargement, reduction, rotation, and distortion correction processing may be performed. When the image to be acquired is a moving image, acquisition of optical flow or the like may be performed.

[0062] Next, the DSP 12 performs recognition processing (S1004) using the image subjected to predetermined image processing by the ISP 11. The recognition processing is a process for recognizing whether an operator is present at an appropriate position on the line, an object is passing through an appropriate position on the line, or the like.

[0063] Next, the CPU 13 determines the result of the recognition by the DSP 12 and determines whether the recognition rate is high enough (S1006). For example, the CPU 13 compares the recognition rate recognized by the DSP 12 with a predetermined value and determines whether the recognition rate is high enough. The recognition rate can be calculated based on the number of images that have been successfully recognized for more than a predetermined number of captured images. For example, the recognition rate can be calculated using frame images captured continuously for more than a predetermined number.

[0064] If the recognition rate is not high enough (S1006: No), the CPU 13 changes the image acquisition conditions (S1008). The image acquisition conditions may be image capture conditions, for example, conditions related to the exposure time of the shutter 15, or conditions related to image processing, such as filter coefficients in the ISP 11, for example, filter coefficients, gain adjustment conditions, etc. By changing the image acquisition conditions in this way, the imaging device 1 performs setting of parameters with a high recognition rate as preprocessing.

[0065] The image acquisition condition may be a combination of the above conditions, for example, a combination of parameters related to the CIS 10 , parameters related to the ISP 11 , and parameters related to the DSP 12 .

[0066] If the recognition rate is sufficiently high (S1006: Yes), the preprocessing may be terminated. Alternatively, determination of termination of the preprocessing may be performed.

[0067] Next, the CPU 13 determines whether to end the preprocessing (S1010). The determination of the preprocessing can obtain a recognition rate that is sufficiently higher than the predetermined recognition rate. In addition, as another condition, for example, the exposure time of the shutter 15 can be changed from the predetermined shortest exposure time to the longest exposure time, and then the recognition rate can be obtained. In this case, the CPU 13 can set the parameter with the highest recognition rate as the image acquisition condition. Although described as the exposure time, of course, other image acquisition conditions can be optimized in this way as described above.

[0068] If it is determined to end the preprocessing (S1010: YES), the CPU 13 ends the preprocessing.

[0069] On the other hand, if it is determined not to end the preprocessing (S1010: No), the CPU 13 may repeat the processing from the capture (S1000) in the CIS 10. As another example, when adjusting the filter of the image that has been acquired, the processing from the image processing (S1002) may be repeated. When the conditions for the recognition processing are changed in S1008, the processing from the recognition processing (S1004) may be repeated. When a combination of a plurality of conditions is used, all combinations of the plurality of conditions or a combination of some conditions appropriately selected from all combinations may be used.

[0070] return Figure 2 , the processing after the preprocessing will be described.

[0071] After completing the preprocessing, the CIS 10 starts capturing to perform the actual recognition processing (S102). The captured image may be, for example, a still image or a moving image. The capturing is performed using the capturing parameters set in the preprocessing.

[0072] Next, the ISP 11 performs predetermined image processing on the image captured by the CIS 10 (S104). The ISP 11 performs the above-mentioned filtering, gain adjustment, etc. on the image. If the parameters are changed by the preprocessing, the ISP 11 performs image processing based on the changed parameters.

[0073] Next, the DSP 12 performs recognition processing on the image processed by the ISP 11 (S106). The DSP 12 performs recognition processing such as whether the object has passed a predetermined position, whether the object is in a proper state, or whether the operator is working normally on the factory production line. In addition, facial recognition of the operator can be performed. For example, as described above, the imaging device 1 is fixedly set, and the situation in a predetermined area of ​​the image acquired from the setting is appropriately recognized.

[0074] All processing operations up to this point can be implemented in the imaging device 1. That is, the conditions for implementing appropriate recognition processing can be acquired without outputting data, etc. from the inside of the imaging device 1 to the outside of the AP 20, etc. via an interface. The conditions can be determined based on the environment by performing the preprocessing as described above. The recognition rate of the recognition processing in S106 may decrease due to changes in the environment. In this case, the processing of the preprocessing from S100 can be repeated.

[0075] For example, another preprocessing can be performed by monitoring the recognition rate and performing preprocessing when the recognition rate is lower than a predetermined value. In addition, as another example, the preprocessing can be repeated every predetermined time or every predetermined number of shots. In this way, the preprocessing can be repeated periodically.

[0076] As described above, according to the present embodiment, parameters and the like for realizing recognition processing suitable for the environment in the closed state of the imaging device 1 can be set. All the configurations of the imaging device 1 can be mounted on one chip or one laminated semiconductor device, as described later. If the CIS 10, ISP 11, DSP 12, CPU 13, storage unit 14, and SEL 16 are mounted on one such semiconductor device, parameters can be set without outputting information from the semiconductor device to the outside. Therefore, compared with when data is output to the outside, privacy and security can be protected at a higher speed, and the recognition rate can be improved while reducing the system cost.

[0077] In a factory production line, for example, when sunlight enters the factory, the recognition rate may be reduced due to sunlight under the same conditions. Even in this case, according to the present embodiment, the reduction in recognition rate can be suppressed. In addition, while maintaining the above advantages, the reduction in recognition rate due to changes in different environments (not limited to sunlight) can be avoided, and a system that is robust to the environment, etc. can be realized.

[0078] according to Figure 2, processing continues further.

[0079] After the recognition processing, the CPU 13 may further perform post-processing (S108). The post-processing does not have to be performed for each recognition processing operation, and may be performed in an appropriate span. For example, it may be performed every predetermined time or every predetermined number of shots. In addition, as another example, the post-processing may be performed sequentially. Post-processing is a process for improving the accuracy of the recognition processing in the DSP 12. The CPU 13 may retrain the neural network model used by the DSP 12, for example, by feeding back the recognition result of the DSP 12. By using the retrained neural network model, the recognition rate may be further improved. Similar to the above-mentioned pre-processing, the retraining may also be performed in the imaging device 1 (i.e., a semiconductor device).

[0080] Next, the SEL 16 outputs appropriate data (S110). Appropriate data is, for example, data captured by the CIS 10, data processed by the ISP 11, or data related to the recognition result of the DSP 12. For example, an image may not be output in a normal state, and an image may be output at a timing when an abnormality occurs in recognition. The recognition rate may be output all the time, or may be output only when an abnormality occurs. In this way, appropriate data is selected and output from the SEL 16 to the AP 20. The data to be output may be temporarily stored in the storage unit 14. For example, according to a request from the AP 20, the data stored in the storage unit 14 may be output together to a certain extent. In addition, in the SEL 16, data, requests, etc. may be input from the AP 20. For example, the input data, requests, etc. may be stored in the storage unit 14, or may be output to the CPU 13.

[0081] As described above, according to the present embodiment, the imaging device 1 can retrain the neural network model for improving the recognition process without outputting data to the outside. Similar to the preprocessing, this process can be implemented without a semiconductor device. Therefore, as described above, compared with when data is output to the outside, privacy and security can be protected at a higher speed, and recognition accuracy can be improved while reducing system costs.

[0082] For example, replacing an existing imaging device with a new imaging device generally results in a learning cost. In addition, the tuning cost for each installation location of the imaging device is high. Even in this case, as in the present embodiment, by optimizing the image acquisition conditions and the neural network model in an environment close to the imaging device 1, the cost can be reduced.

[0083] (Second Embodiment)

[0084] Figure 4is a schematic diagram showing an example of an imaging system according to a second embodiment. The imaging system 3 includes an electronic device 2 and a cloud 30. The cloud 30 is a broad concept, which may be a general cloud on the Internet, or may be a closed server, a group of servers, etc. in an intranet, for example. The electronic device 2 is connected to the cloud 30 via, for example, a wired or wireless network.

[0085] In the first embodiment described above, the preprocessing and retraining of the model are closed processes in one semiconductor device, but of course the retraining of the model can be realized externally. In this case, data can be sent from the electronic device 2 to a server in the cloud 30 at a predetermined timing. In this case, in order to protect privacy and improve security, the image data to be sent can be encrypted.

[0086] In the cloud 30 to which the data is transmitted, retraining can be performed using a CPU, a GPU, etc. with higher accuracy than the CPU 13, etc. provided in the imaging device 1. Therefore, the model can be retrained in the imaging device 1 while performing the recognition process, and the training can be performed in parallel on a more accurate server, etc.

[0087] In the present embodiment, for example, the parameters of the neural network model used in the DSP 12 may be transmitted together with the timing of data transmission from the electronic device 2 to the cloud 30. Then, the server or the like in the cloud 30 may retrain the received parameters using the received information. Afterwards, the server or the like may optimize the image acquisition in the electronic device 2 and improve the recognition accuracy through the neural network model by feeding back the parameters optimized for the electronic device 2 that sent the data. In this way, various parameters optimized based on the output from the imaging device 1 may be deployed from the server side to the imaging device 1.

[0088] For example, when the recognition rate decreases or an abnormal value appears in recognition, the electronic device 2 can only send the image data that causes such a problem. Then, the server or the like can analyze the data that causes the reduction in the recognition rate. The data can be used to optimize the parameters of the image acquisition conditions or the model used for recognition. In addition, not only the image data that causes such a problem but also the image data related to the front frame and the back frame can be sent together. Here, the front frame and the back frame are not limited to one frame before and after the main frame, and can include multiple frames.

[0089] The server or the like on the cloud 30 may receive data from a plurality of electronic devices 2. In this case, the recognition rate, image data, and identifier (ID) uniquely assigned to the electronic device 2 may be sent in association. In this case, the model of the electronic device 2 may also be sent. By sending the ID in this manner, one server or the like may process information from a plurality of electronic devices 2. In addition, by sending the model or the like, for electronic devices having the same model or devices having a similar imaging system, parameters or the like that have undergone the same processing may also be sent.

[0090] The information to be transmitted may not be the image data itself as described above, but may be appropriate information about the image. Figure 1 In the configuration shown in FIG. 1 , the imaging device 1 may include a motion detection circuit. The motion detection circuit is a circuit for obtaining how much motion or brightness change has occurred in the acquired image. The motion detection circuit can be provided in a semiconductor device like other components. The electronic device 2 may send information output from the motion detection circuit output from the imaging device 1 to a server, etc. The server, etc. may identify the cause of the reduced recognition rate by analyzing the motion detection result. The server, etc. may optimize the image acquisition conditions or the neural network model based on the analysis result.

[0091] The imaging device 1 may transmit information on the detection value used for exposure adjustment to a server, etc. The detection value is, for example, information on brightness changes.

[0092] The recognition results of the preceding and succeeding frames and the change in the recognition rate can be sent from the imaging device 1 to a server or the like together with the image data of the frames preceding and succeeding the image whose recognition rate has been reduced. In this case, parameters related to the image quality of the preceding and succeeding frames can be sent simultaneously. By sending this information, the parameters causing the reduction in the recognition rate on the server or the like can be analyzed, and the model can be further optimized.

[0093] The imaging device 1 can transmit not only the recognition result but also the data in the intermediate layer of the neural network model. The data in the intermediate layer can be, for example, data representing a feature quantity whose size has been compressed, and the optimal parameters can be acquired by analyzing such data on a server or the like. As another example, back propagation of errors can be performed from the data in the intermediate layer to optimize the encoder layer from the input layer to the intermediate layer in the neural network, or conversely, optimize the layer that realizes the recognition of the feature quantity from the intermediate layer to the output layer.

[0094] In the above description, parameters related to the internal state of the imaging device 1 and the like are output, but the present invention is not limited thereto. For example, data indicating the location, time, temperature, humidity, and other external environments where the imaging device 1 (i.e., the electronic device 2) is installed may be sent to a server, etc. By sending data related to the external environment in this manner, optimization based on the external environment may also be achieved.

[0095] In addition, by transmitting the position information, optimization can be performed for image acquisition conditions, etc. The position information may be, for example, GPS (Global Positioning System) information. In addition, information such as a capture angle of an object relative to the imaging device 1 may be output. The angle may be, for example, information acquired by a gyro sensor, an acceleration sensor, etc. further provided in the imaging device 1.

[0096] In addition, by sending time information, the change in recognition rate can be optimized according to time. In this case, the imaging device 1 can be configured to control parameters according to time even when normal capture is performed based on a program deployed from a server, etc. Of course, the control of the parameters can be performed based on other external environments such as temperature and humidity.

[0097] As described above, this process is expensive for the imaging device 1 to implement, and the imaging system 3 may perform the process on the cloud 30 .

[0098] As described above, according to the present embodiment, advanced retraining can be performed using a CPU or the like having higher performance than that of the imaging device 1. In addition, in this case, the server or the like may want to further improve the accuracy by analyzing the received inference results. For example, by using the data received by the server or the like as training data, a more accurate neural network model can be generated, and an image more suitable for recognition can be acquired by optimizing the image acquisition conditions.

[0099] (Third Embodiment)

[0100] Figure 5 is a diagram showing an example of an image acquired by the imaging device 1. The image Im is, for example, an image acquired by the imaging device 1 installed in a factory that produces bottle-shaped objects. In the factory, for example, an operator handles or visually inspects bottles flowing through a production line. The electronic device 2 is installed to image the factory production line to acquire images of the operator and the bottles.

[0101] As described above, the operator and the bottles flowing through the line are imaged in the image Im. As described in the above embodiment, for example, the imaging device 1 can output image information and recognition results in the DSP 12. However, if all images are output as low-compression, high-resolution images, there is a possibility that the bandwidth for transmission and the storage area for storing image information will be tight.

[0102] Therefore, in this embodiment, for example, in the case of sending image information for retraining, how to reduce the amount of data and maintain accuracy is described.

[0103] In the imaging device 1, for example, the SEL 16 may output a part of the image information to the outside. The image data output to the outside (for example, the AP 20) is sent to a server on the cloud 30. The imaging device 1 may crop and output only an area required for recognition.

[0104] For example, the imaging device 1 may output only the information in the region of interest (hereinafter referred to as ROI R1, etc.) that captures the operator. In this case, the information in the region other than ROI R1 may be deleted, and after performing appropriate image processing thereon, only ROI R1 may be cropped and transmitted. As another example, high-precision compression may be performed only within ROI R1. For example, the ISP 11 may change the data compression method in ROI R1 and the data compression method in other regions. ROI R1 may be compressed by a method that is capable of performing high-precision recovery but cannot perform data compression so much. Other regions may be compressed by a method that is difficult to restore with high precision but has a higher data compression rate than that in ROI R1.

[0105] ROI R1 may be compressed by the same data compression method using compression parameters that can be decompressed at high resolution, and other regions may be compressed by the same data compression method using compression parameters that do not provide high resolution but can further reduce the amount of data.

[0106] As another example, raw data may be output in ROI R1, while data compressed at a high compression rate may be transmitted in other regions.

[0107] The number of ROIs is not limited to one, and there may be multiple ROIs. Figure 5 As shown, there may be ROI R1 and ROI R2, and the data in these ROIs may be compressed by a method that can restore images with high precision different from the compression of other regions. In addition, the same precision does not need to be maintained between ROIs. That is, ROI R1 and ROI R2 can be compressed at different compression rates or by different compression methods. For example, the compression method used for ROI R2 may be able to restore images with higher precision than the compression method used for ROI R1. Of course, in this case, the regions outside ROI R1 and R2 may also be compressed by a method that further reduces the amount of data. In addition, the images in the regions outside ROI R1 and R2 may not be sent.

[0108] When information is transmitted in consecutive frames, information in the ROI may be transmitted in all frames, while information outside the ROI may be transmitted by sparsely spaced frames.

[0109] Then, a server in the cloud 30 or the like can use the image restored with high accuracy (e.g., a high-resolution image or raw data) to optimize image acquisition conditions or retrain a neural network model.

[0110] As described above, according to the present embodiment, images can be acquired under appropriate image acquisition conditions in the imaging device 1, and optimization of models and the like can be achieved, thereby further improving performance outside the imaging device 1. In addition, data can be sent and received without straining bandwidth while continuing recognition.

[0111] (Chip Structure of Imaging Device 1)

[0112] Next, we will describe Figure 1 The chip structure of the imaging device 1 in FIG. Figure 6 To show Figure 1 Schematic diagram of an example of a chip structure of the imaging device 1. Figure 6 The imaging device 1 in FIG. 1 is a laminated pair of a first substrate 40 and a second substrate 41. The first substrate 40 and the second substrate 41 are sometimes referred to as bare chips. Figure 6 In the example of FIG. 4 , the first substrate 40 and the second substrate 41 are rectangular, but specific shapes and sizes of the first substrate 40 and the second substrate 41 are arbitrary. The first substrate 40 and the second substrate 41 may have the same size or may be different sizes from each other.

[0113] Figure 1 The pixel array 102 shown is disposed on the first substrate 40. In addition, at least a portion of the optical system 100 of the CIS 10 may be mounted on the first substrate 40 in an on-chip manner. In addition, although not shown, the shutter 15 may be mounted on the first substrate 40. For example, the shutter may be disposed to cover the light receiving surface of the light receiving element of the first substrate 40 in the case of an optical shutter, and may be disposed to control the light receiving element of the pixel array 102 in the case of a digital shutter.

[0114] Figure 1 The ADC 104, ISP 11, DSP 12, CPU 13, storage unit 14, and SEL 16 shown are arranged on the second substrate 41. In addition, the second substrate 41 may be provided with components required for controlling the imaging device 1, such as an input / output interface and a power supply circuit (not shown).

[0115] The first substrate 40 and the second substrate 41 are configured as a semiconductor device that is bonded and laminated, for example, by a predetermined method. As a specific example of bonding, a so-called CoC (chip on chip) method can be adopted, in which the first substrate 40 and the second substrate 41 are cut from a wafer, divided into individual pieces, and then laminated on top of each other. Alternatively, a so-called CoW (chip on wafer) method can be adopted, in which one of the first substrate 40 and the second substrate 41 (for example, the first substrate 40) is cut from a wafer and divided into individual pieces, and the first substrate 40 is bonded to the second substrate 41 before dividing. Alternatively, a so-called WoW (wafer on wafer) method can be adopted, in which the first substrate 40 and the second substrate 41 are bonded together in the state of a wafer.

[0116] For example, vias, micro bumps, micro pads, plasma bonding, etc. may be used as a method for bonding the first substrate 40 and the second substrate 41. However, various other bonding methods may be used.

[0117] Figure 6 The arrangement of the components on the first substrate 40 and the second substrate 41 is given as an example, and is not limited thereto. Figure 6 At least one component on the second substrate 41 shown is disposed on the first substrate 40. Although a laminated structure is shown, the structure is not limited thereto, and a structure in which the above components are arranged on one semiconductor substrate may be used.

[0118] (Application to other sensors)

[0119] It should be noted that, although in the above-mentioned embodiment, an example is given of applying the technology according to the present disclosure to an imaging device 1 (image sensor) for acquiring a two-dimensional image, the application target of the technology according to the present disclosure is not limited to the imaging device. For example, the technology according to the present disclosure can be applied to various light receiving sensors, such as a time of flight (ToF) sensor, an infrared (IR) sensor, and a dynamic vision sensor (DVS). That is, by adopting a laminated chip structure of a light receiving sensor, it is possible to achieve a reduction in noise included in the sensor result, a reduction in the size of the sensor chip, and the like.

[0120] Accordingly, by performing the setting of image acquisition conditions and optimization of the neural network model by the above method, feedback processing can be performed in a state close to the imaging device 1 (i.e., after ensuring privacy and security). Depending on the situation, optimization can be achieved at low cost and high precision in a server on the cloud 30, etc.

[0121] The above-mentioned embodiments may have the following forms.

[0122] (1) A solid-state imaging device, comprising: a pixel array having a plurality of pixels that perform photoelectric conversion; a converter that converts analog pixel signals output from the pixel array into digital image data; an image processing unit that performs image processing on the digital image data; a digital signal processing unit that performs recognition processing on the digital image data output by the image processing unit; and a control unit that performs optimization of at least one acquisition processing operation among acquisition of analog pixel signals, acquisition of digital image data, and acquisition of a result of the recognition processing based on a result of the recognition processing.

[0123] (2) The solid-state imaging device according to (1), wherein the control unit feeds back a result of the recognition process to perform optimization.

[0124] (3) A solid-state imaging device according to (2), wherein, when the recognition rate of the recognition processing has decreased, the control unit performs optimization of at least one acquisition processing operation among acquisition of analog pixel signals, acquisition of digital image data, and acquisition of the result of the recognition processing.

[0125] (4) The solid-state imaging device according to (2) or (3), wherein the control unit controls an exposure time of pixels of the pixel array.

[0126] (5) The solid-state imaging device according to any one of (2) to (4), wherein the control unit optimizes parameters related to image processing by the image processing unit.

[0127] (6) The solid-state imaging device according to any one of (2) to (5), wherein the digital signal processing unit performs recognition processing using a trained neural network model.

[0128] (7) The solid-state imaging device according to (6), wherein the control unit retrains the neural network model.

[0129] (8) The solid-state imaging device according to any one of (1) to (7) further includes a semiconductor device, wherein the semiconductor device includes: a first substrate on which the pixel array is arranged; and a second substrate laminated on the first substrate, and the converter, the image processing unit, the digital signal processing unit and the control unit are arranged on the second substrate.

[0130] (9) The solid-state imaging device according to (8), wherein the first substrate and the second substrate are bonded by any one of a CoC (Chip on Chip) method, a CoW (Chip on Wafer) method, and a WoW (Wafer on Wafer) method.

[0131] (10) The solid-state imaging device according to (8) or (9), further including a selector for selecting an output from at least one of the converter, the image processing unit, and the digital signal processing unit and outputting data to the outside.

[0132] (11) The solid-state imaging device according to (10), wherein when the image information is output, only information in the region of interest is selected and output.

[0133] (12) The solid-state imaging device according to any one of (10), wherein when the image information is output, information in the region of interest and information in other regions are output using different compression methods.

[0134] (13) The solid-state imaging device according to (12), wherein information in the region of interest and information in other regions are compressed and output at different compression rates.

[0135] (14) The solid-state imaging device according to any one of (11) to (13), wherein, when a recognition rate in the recognition process has decreased, the image information is output together with a result of the recognition process.

[0136] (15) An electronic device comprising: the solid-state imaging device according to any one of (9) to (14); and an application processor on a substrate independent of the semiconductor device, wherein the application processor inputs / outputs data via the selector.

[0137] (16) An imaging system comprising: a solid-state imaging device according to any one of (1) to (13); and a server connected to the solid-state imaging device via a network, wherein the solid-state imaging device outputs the output of the converter and / or the output of the image processing unit in association with the output of the digital signal processing unit to the server, and the server optimizes at least one acquisition processing operation of acquiring analog pixel signals, acquiring digital image data, and acquiring results of recognition processing based on the received information, and deploys the optimized information to the solid-state imaging device.

[0138] Aspects of the present disclosure are not limited to the aforementioned embodiments, and include various modifications that can be implemented by those skilled in the art, and the effects of the present disclosure are not limited to the above details. In other words, various additions, modifications and partial deletions can be made without departing from the conceptual concept and gist of the present disclosure, and the conceptual concept and gist of the present disclosure can be derived from the details defined in the claims and their equivalents.

[0139] Furthermore, the present disclosure can also be applied to an apparatus that detects motion in the imaging device 1 and performs recognition processing, such as a surveillance camera, and the above-mentioned moving objects and medical fields.

[0140] [Reference Numbers List]

[0141] 1 Imaging device

[0142] 10 CIS

[0143] 100 Optical System

[0144] 102 pixel array

[0145] 104A DC

[0146] 11 ISP

[0147] 12 DSP

[0148] 13 CPU

[0149] 14 Storage Unit

[0150] 15 Shutter

[0151] 16 SEL

[0152] 2 Electronic devices

[0153] 20 AP

[0154] 3 Imaging system

[0155] 30 Cloud

[0156] 40 first substrate

[0157] 41 Second substrate.

Claims

1. A solid-state imaging device, comprising: a pixel array having a plurality of pixels that perform photoelectric conversion; A converter converts the analog pixel signal output from the pixel array into digital image data; an image processing unit, performing image processing on the digital image data; a digital signal processing unit, which performs recognition processing on the digital image data output by the image processing unit; as well as a control unit that performs optimization of at least one acquisition processing operation among acquisition of the analog pixel signal, acquisition of the digital image data, and acquisition of the result of the recognition processing based on the result of the recognition processing; Wherein, before performing the image processing and the recognition processing on the digital image data, preprocessing is performed on another digital image data different from the digital image data; Wherein, the preprocessing includes: acquiring the other digital image data; performing the image processing on the acquired another digital image data; performing the recognition processing on the other digital image data on which the image processing has been performed; when the recognition rate in the result of the recognition processing is higher than a set value, terminating the preprocessing; when the recognition rate in the result is lower than the set value, changing the image acquisition condition including the filter coefficient of the image processing unit, and when the filter coefficient is changed, repeating the processing starting from the image processing until a recognition rate higher than the set value is obtained; The solid-state imaging device acquires parameters suitable for the recognition process in the closed state of the solid-state imaging device through the preprocessing, without setting the parameters by outputting data to the outside of the solid-state imaging device.

2. The solid-state imaging device according to claim 1, wherein The control unit feeds back a result of the recognition processing of the digital image data to perform the optimization.

3. The solid-state imaging device according to claim 2, wherein: When a recognition rate of the recognition process of the digital image data decreases, the control unit performs optimization regarding the at least one acquisition process operation among acquisition of the analog pixel signal, acquisition of the digital image data, and acquisition of a result of the recognition process.

4. The solid-state imaging device according to claim 2, wherein: The control unit controls exposure time of the pixels of the pixel array.

5. The solid-state imaging device according to claim 2, wherein: The control unit optimizes parameters related to the image processing of the image processing unit.

6. The solid-state imaging device according to claim 2, wherein: The digital signal processing unit performs the recognition process using a trained neural network model.

7. The solid-state imaging device according to claim 6, wherein The control unit retrains the neural network model.

8. The solid-state imaging device according to claim 1, further comprising a semiconductor device, the semiconductor device comprising: a first substrate, the pixel array being arranged on the first substrate; as well as A second substrate is laminated on the first substrate, and the converter, the image processing unit, the digital signal processing unit, and the control unit are arranged on the second substrate.

9. The solid-state imaging device according to claim 8, wherein The first substrate and the second substrate are bonded by any one of a chip-on-chip method, a chip-on-wafer method, and a wafer-on-wafer method. 10 . The solid-state imaging device according to claim 8 , further comprising a selector for selecting an output from at least one of the converter, the image processing unit, and the digital signal processing unit and the output is output to the outside.

11. The solid-state imaging device according to claim 10, wherein When outputting image information, only the information in the region of interest is selected and output.

12. The solid-state imaging device according to claim 10, wherein When outputting image information, information in the region of interest and information in other regions are output using different compression methods.

13. The solid-state imaging device according to claim 12, wherein: The information in the region of interest and the information in the other regions are compressed and output at different compression rates.

14. The solid-state imaging device according to claim 11, wherein When the recognition rate in the recognition process of the digital image data decreases, the image information is output together with the result of the recognition process.

15. An electronic device, comprising: The solid-state imaging device according to claim 10; as well as an application processor on a substrate independent of the semiconductor device; wherein, The application processor inputs / outputs data via the selector.

16. An imaging system comprising: The solid-state imaging device according to claim 1; as well as A server is connected to the solid-state imaging device via a network; wherein, The solid-state imaging device outputs the output of the converter and / or the output of the image processing unit to the server in association with the output of the digital signal processing unit; and The server performs optimization on at least one acquisition processing operation among acquisition of the analog pixel signal, acquisition of the digital image data, and acquisition of the result of the recognition processing based on the received information, and deploys the optimized information to the solid-state imaging device.

Images