Solid-state image capturing device, information processing device, information processing system, information processing method, and program
By integrating the DNN processing unit and control unit in the solid-state image capture device and dynamically adjusting the DNN model, the flexibility and efficiency problems of the DNN model in the prior art in the image sensor are solved, and efficient image processing and private information protection are achieved.
Patent Information
- Application Number
- CN201980054380.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-31
- Filing Date
- 2019-08-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-08-28
AI Technical Summary
The prior art is difficult to effectively realize the flexibility and efficiency of deep neural network (DNN) models in image sensors, especially in image processing and private information protection, and it is difficult to dynamically adjust the DNN model to meet different image processing needs.
A solid-state image capture device is designed, including a DNN processing unit and a DNN control unit, and perform processing on the input image based on the DNN model, and dynamically adjust the DNN model according to the evaluation information to optimize the performance and efficiency of the image processing.
It realizes flexible adjustment of the DNN model, improves the accuracy and efficiency of image processing, reduces power consumption, and adapts to the needs of different image processing tasks.
Smart Images

Figure CN112585949B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a solid-state image capture device, an information processing device, an information processing system, an information processing method, and a computer program. Background Technology
[0002] Recently, various devices for implementing neural networks have been disclosed. For example, Patent Document 1 discloses a technique for searching for high-performance network structures based on the evaluation results of the acquired network structures.
[0003] Existing technical documents
[0004] Patent documents
[0005] Patent Document 1: Japanese Patent No. 2017-182710 Summary of the Invention
[0006] The problem to be solved by the present invention
[0007] Image sensors, including complementary metal-oxide-semiconductor (CMOS) and digital signal processors (DSP), are mounted on instruments such as digital cameras. Recently, for example, deep neural network (DNN) functionality is preferably implemented on image sensors, and advanced processing is performed to achieve diversification and acceleration of image processing, protection of private information, etc.
[0008] Therefore, this disclosure provides a solid-state image capture apparatus, an information processing apparatus, an information processing system, an information processing method, and a computer program capable of changing the DNN model to be executed.
[0009] Solution to the problem
[0010] To address the aforementioned issues, a solid-state image capture device includes: a DNN processing unit configured to perform DNN on an input image based on a DNN model; and a DNN control unit configured to receive control information generated from evaluation information based on the execution result of the DNN, and to modify the DNN model based on the control information. Attached Figure Description
[0011] Figure 1 This is a block diagram illustrating an exemplary configuration of a solid-state image capture system according to a first embodiment of the present disclosure.
[0012] Figure 2 This is a schematic diagram illustrating exemplary connection relationships of a solid-state image capture system according to a first embodiment of the present disclosure.
[0013] Figure 3 This is a schematic diagram illustrating an exemplary laminated structure of a solid-state image capture device according to a first embodiment of the present disclosure.
[0014] Figure 4 This is a diagram used to describe an exemplary object sensing process.
[0015] Figure 5 This is a sequence diagram illustrating an exemplary process of processing at a solid-state image capture system according to a first embodiment of the present disclosure.
[0016] Figure 6 This is a flowchart illustrating an exemplary process of learning processing at a solid-state image capture device according to a first embodiment of the present disclosure.
[0017] Figure 7 This is a flowchart illustrating an exemplary process of learning processing at a solid-state image capture device according to a first embodiment of the present disclosure.
[0018] Figure 8 This is a schematic diagram illustrating the processing of image data according to a first embodiment of the present disclosure.
[0019] Figure 9 This is a diagram illustrating an exemplary use case of a solid-state image capture system according to a first embodiment of the present disclosure.
[0020] Figure 10 This is a block diagram illustrating an exemplary configuration of a solid-state image capture system according to a modification of the first embodiment of the present disclosure.
[0021] Figure 11 This is a block diagram illustrating an exemplary configuration of a solid-state image capture system according to a second embodiment of the present disclosure.
[0022] Figure 12 This is a schematic diagram illustrating exemplary connection relationships of a solid-state image capture system according to a second embodiment of the present disclosure.
[0023] Figure 13 This is a schematic diagram illustrating exemplary connection relationships of a solid-state image capture system according to a second embodiment of the present disclosure.
[0024] Figure 14 This is a block diagram illustrating an exemplary configuration of a solid-state image capture system according to a third embodiment of the present disclosure.
[0025] Figure 15 This is a schematic diagram illustrating exemplary connection relationships of a solid-state image capture system according to a third embodiment of the present disclosure.
[0026] Figure 16 This is a hardware configuration diagram of an exemplary computer configured to implement the functions of the solid-state image capture device and information processing device of this disclosure.
[0027] Figure 17This is a diagram illustrating an exemplary schematic configuration for endoscopic surgery.
[0028] Figure 18 This is a block diagram illustrating an exemplary functional configuration of the camera and CCU.
[0029] Figure 19 This is a block diagram illustrating an exemplary schematic configuration of a diagnostic support system.
[0030] Figure 20 This is a block diagram illustrating an exemplary schematic configuration of a vehicle control system.
[0031] Figure 21 This is an illustrative diagram showing an exemplary installation location for the external information detection unit and the image capture unit. Detailed Implementation
[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The same parts in the following embodiments are indicated by the same reference numerals, and repeated descriptions thereof are omitted.
[0033] This disclosure will be described in the following order of description.
[0034] 1. First Embodiment
[0035] 1-1 Configuration of the solid-state image capture system according to the first embodiment
[0036] 1-2 Processing at the solid-state image capture system according to the first embodiment
[0037] 1-3 Use Cases of the Solid State Image Capture System According to the First Embodiment
[0038] 1-4 Modifications to the solid-state image capture system according to the first embodiment
[0039] 2. Second Embodiment
[0040] 2-1 Configuration of the solid-state image capture system according to the second embodiment
[0041] 2-2 Configuration of a modified example of the solid-state image capture system according to the second embodiment
[0042] 3. Third embodiment
[0043] 3-1 Configuration of the solid-state image capture system according to the third embodiment
[0044] 4. Hardware Configuration
[0045] 5. Exemplary applications of endoscopic surgery
[0046] 6. Exemplary Application 1
[0047] 7. Exemplary applications of moving objects
[0048] (1. First Embodiment)
[0049] [1-1. Configuration of the solid-state image capture system according to the first embodiment]
[0050] The following is for reference. Figure 1 and Figure 2 The configuration of a solid-state image capture system according to a first embodiment of the present disclosure is described. Figure 1 This is a block diagram illustrating an exemplary configuration of a solid-state image capture system according to a first embodiment of the present disclosure. Figure 2 This is a diagram illustrating the connection relationships of a solid-state image capture system according to a first embodiment of the present disclosure.
[0051] like Figure 1 As shown, the solid-state image capture system 1 includes a solid-state image capture device 100 and an information processing device 200.
[0052] like Figure 2 As shown, the solid-state image capture device 100 and the information processing device 200 are housed in the same housing 10. Specifically, the solid-state image capture device 100 and the information processing device 200 are housed as different chips in the same housing 10. The solid-state image capture device 100 and the information processing device 200 are each mounted as a system-on-a-chip (SoC), a multi-chip module (MCM), a system-in-package (SIP), a small outline package (SOP), etc. The solid-state image capture system 1 can be connected to an external device, for example, via an Internet communication network 300, to perform communication therebetween. In this case, the solid-state image capture system 1 can be connected to an external device, for example, via wireless communication, to perform communication therebetween. The solid-state image capture system 1 is suitable, for example, for a virtual personal assistant (VPA) or an in-vehicle camera.
[0053] like Figure 1 As shown, the solid-state image capture device 100 includes an image capture unit 110, an image capture processing unit 120, a DNN processing unit 130, a storage unit 140, an evaluation unit 150, a DNN control unit 160, a selector 170, a communication I / F 180, and a communication control unit 190.
[0054] Figure 3 This is a schematic diagram illustrating an exemplary laminated structure of the solid-state image capture device 100 according to the first embodiment. (As shown) Figure 3 As shown, the solid-state image capture device 100 has, for example, a laminated structure in which a first substrate 11 of rectangular shape and a second substrate 12 of rectangular shape are bonded to each other.
[0055] The first substrate 11 and the second substrate 12 can be bonded to each other, for example, through a so-called chip-on-a-chip (CoC) scheme, in which the first substrate 11 and the second substrate 12 are sliced into chips, and then the sliced first substrate 11 and the second substrate 12 are bonded to each other. Alternatively, the first substrate 11 and the second substrate 12 can be bonded to each other through a so-called chip-on-wafer (CoW) scheme, in which one of the first substrate 11 and the second substrate 12 (e.g., the first substrate 11) is diced into a chip, and then the sliced first substrate 11 is bonded to the second substrate 12, which has not yet been sliced (in other words, is in a wafer state). Alternatively, the first substrate 11 and the second substrate 12 can be bonded to each other in a wafer state through a so-called wafer-on-wafer (WoW) scheme.
[0056] The first substrate 11 and the second substrate 12 can be joined together, for example, by plasma bonding. However, this disclosure is not limited thereto, and various bonding methods can be used.
[0057] The first substrate 11 and the second substrate 12 may be the same or different in size. The first substrate 11 and the second substrate 12 are semiconductor substrates such as silicon substrates.
[0058] exist Figure 1 Among the components of the solid-state image capture device 100 shown, for example, the image capture unit 110 is disposed on the first substrate 11.
[0059] exist Figure 1 Among the components of the solid-state image capture device 100 shown, for example, an image capture processing unit 120, a DNN processing unit 130, a storage unit 140, an evaluation unit 150, a DNN control unit 160, a selector 170, a communication I / F 180, and a communication control unit 190 are disposed on the second substrate 12.
[0060] Therefore, the image capture units 110 of the solid-state image capture device 100 are stacked and mounted on components other than the image capture unit 110.
[0061] Refer again Figure 1 For example, image capture unit 110 includes: an optical system including a zoom lens, a focusing lens, and an aperture; and a pixel array unit, wherein each pixel in the pixel array unit is configured such that each pixel includes a light-receiving element, such as a photodiode, arranged in a two-dimensional matrix. Light incident from the outside is imaged through the optical system onto a light-receiving surface in the pixel array unit on which the light-receiving elements are arranged. Each pixel in the pixel array unit performs photoelectric conversion on the light incident on the light-receiving element and generates image data according to the amount of incident light. Image capture unit 110 outputs the thus captured image data to image capture processing unit 120.
[0062] For example, the image capture processing unit 120 converts image data into digital image data. For example, the image capture processing unit 120 performs "preprocessing," "data expansion," and "data normalization" on the converted digital image data. Preprocessing is a process performed on the image capture unit 110 during inference and learning, and includes processes such as dewarping, cropping, lens shading correction, reduction, and enlargement. Data expansion is a process performed on the image data during learning, and includes, for example, processes such as changing the aspect ratio of the image data, translating the image data, rotating the image data, inverting the image data, and geometrically deforming the image data. Data expansion includes, for example, processes such as changing the grayscale of the image data and changing the color. Data expansion also includes, for example, processes such as adding noise to the image data. Data normalization is a process performed on the image data during inference and learning, and includes, for example, processes such as setting the average pixel value of the image data to zero, setting the pixel dispersion to 1, and setting the correlation between components to 0 to whiten the image data. The image capture processing unit 120 outputs the digital image data, after various processing steps, to the DNN processing unit 130 and the selector 170.
[0063] The DNN processing unit 130 performs the process of recognizing objects included in the image data input from the image capture processing unit 120 by performing DNN on the image data based on, for example, a DNN model stored in the storage unit 140. Specifically, the DNN processing unit 130 performs the process of sensing objects included in the image data received from the image capture processing unit 120. The DNN processing unit 130 outputs the execution result of the DNN on the image data to the evaluation unit 150. The DNN processing unit 130 can be implemented, for example, by a DSP. In other words, the DNN processing unit 130 can be implemented by providing the DSP with the function of performing DNN. The solid-state image capture device 100 may include only one DNN processing unit 130, or it may include multiple DNN processing units 130.
[0064] The following is for reference. Figure 4 This describes an exemplary object sensing process performed by the DNN processing unit 130. Figure 4 This is a diagram used to describe an exemplary object sensing process performed by the DNN processing unit 130. Figure 4 The object sensing processing shown is merely exemplary and does not limit this disclosure.
[0065] like Figure 4 As shown, the DNN processing unit 130 performs, for example, image classification, object localization, object recognition, semantic segmentation, instance segmentation, and key point detection.
[0066] Image classification is the process of categorizing the types of objects included in an image dataset. For example, image classification identifies that an image dataset contains people and animals.
[0067] Object localization processing is the process of classifying the location and type of the main objects included in image data. For example, assignment processing designates the location of a person as the main object.
[0068] Object recognition processing is the process of classifying the location and type of multiple objects included in image data. For example, object recognition processing classifies people and animals and identifies their locations.
[0069] Semantic segmentation is the process of classifying the location and type of multiple objects included in image data and providing the same mask for objects of the same type. For example, semantic segmentation provides different types of masks for humans and animals. In this example, the same mask is provided for each animal.
[0070] Instance segmentation is the process of classifying the location and type of multiple objects included in image data and providing a mask for each object. For example, instance segmentation provides different types of masks for humans and animals.
[0071] Keypoint detection processing is the process of classifying the location and type of multiple objects included in image data and detecting the keypoints of each object. For example, keypoint detection processing detects the keypoints of a person.
[0072] Refer again Figure 1 Specifically, the DNN processing unit 130 performs object sensing processing by performing processes such as convolution, bias addition, activation calculation, and pooling on the image data received from the image capture processing unit 120.
[0073] Examples of activation functions used by the DNN processing unit 130 during activation computation include the identity function, sigmoid function, softmax function, step function, ReLU function, and Tanh function, but this disclosure is not limited thereto.
[0074] For example, the DNN processing unit 130 calculates the error and the value of the loss function during learning. The DNN processing unit 130 calculates the error of the target DNN model using methods such as gradient descent, stochastic gradient descent, Newton's method, quasi-Newton's method, or error backpropagation. The DNN processing unit 130 calculates the value of the loss function using methods such as least squares error or cross-entropy error.
[0075] For example, storage unit 140 stores at least one DNN model to be executed by DNN processing unit 130. Storage unit 140 may be implemented using semiconductor storage elements such as random access memory (RAM), read-only memory (ROM), or flash memory.
[0076] Examples of DNN models stored in storage unit 140 include classification, object sensing, human keypoint recognition, face detection, face recognition, image generation (e.g., generative adversarial networks (GANs), recurrent neural networks (e.g., recurrent neural networks (RNNs)), and denoising models. The above-described types of DNNs are merely exemplary and do not limit this disclosure.
[0077] Each DNN model stored in storage unit 140 is associated with identification information such as an ID. In other words, managing each DNN model using identification information and storing it in storage unit 140 allows the desired DNN model to be selected externally based on the identification information. Specifically, each DNN model stored in storage unit 140 can be read externally based on the identification information and written to by adding, changing, etc. The DNN models stored in storage unit 140 can be associated with the identification information of the solid-state image capture device 100. Therefore, when multiple solid-state image capture devices 100 are provided, it is possible to determine which device the DNN model is stored in. Storage unit 140 can store one or more inference DNN models and one or more learning DNN models. Alternatively, storage unit 140 can store both inference DNN models and one or more learning DNN models.
[0078] Each DNN model stored in storage unit 140 is distinguished by model information. Examples of model information include input image size, filter size, filter value, filter number, stride count, padding count, bias value, type of activation function, accuracy of the DNN, value of the loss function, batch size, type of normalization, and learning rate. The above-mentioned model information of the DNN model is merely exemplary and does not limit this disclosure.
[0079] The input image size refers to the size of the image data to which the DNN is applied, and is defined by the number of vertical pixels × the number of horizontal pixels × the number of channels. The number of channels refers to the number of input images.
[0080] The filter size is applied to the input image and is defined by the number of vertical pixels × the number of horizontal pixels × the number of channels. The filter value is the filter weight and is automatically set through learning. Features of the input image are extracted by applying the filter. In this example, performing a normal convolution on the input image generates a feature map smaller than the input image size. Performing a transposed convolution on the input image generates a feature map larger than the input image size.
[0081] Stride amount refers to the amount of movement of the filter when it is applied to the input image. Padding amount refers to the size of the region at the edges of the feature map after the filter is applied. The bias value is independent of the input image and is automatically set through learning.
[0082] For example, activation functions can be the identity function, sigmoid function, softmax function, step function, ReLU function, or Tanh function as described above.
[0083] The accuracy of the DNN refers to the accuracy of its recognition of the input image, and is evaluated by the evaluation unit 150 in this embodiment. The value of the loss function is calculated using least squares error, cross-entropy error, etc. Batch size refers to the number of data samples taken for the learning data. Learning rate refers to the learning count of the DNN model.
[0084] Storage unit 140 can also store, for example, learning data. For instance, the learning data could be training data used for calculating training error or test data used for calculating test error. Storage unit 140 can pre-store learning data, or it can store image data captured by image capture unit 110 in learning mode as learning data.
[0085] When storing image data, for example, captured by image capture unit 110, as learning data, storage unit 140 stores the image data as raw (RAW) data. In this case, before learning, image capture processing unit 120 provides "preprocessing," "data expansion," and "data normalization" to the image data captured by image capture unit 110. When storing image data instead of raw data in storage unit 140, storage unit 140 may also provide "preprocessing," "data expansion," and "data normalization" in advance.
[0086] Evaluation unit 150 generates evaluation information by evaluating the processing behavior of DNN processing unit 130, for example, based on the results of DNN execution by DNN processing unit 130. During inference, evaluation unit 150 generates evaluation information by measuring, for example, the power consumption used when DNN is executed by DNN processing unit 130 or the recognition accuracy of DNN. To determine the accuracy of DNN, the evaluation information preferably includes information related to the recognition accuracy of DNN. During learning, evaluation unit 150 generates evaluation information by measuring, for example, the recognition accuracy of DNN or the value of loss function. Evaluation unit 150 can generate evaluation information by evaluating the processing behavior of solid-state image capture device 100 including DNN processing unit 130. Evaluation unit 150 sends at least a portion of the evaluation information to information processing device 200. The evaluation information includes model information of DNN model, identification information of DNN model, identification information of solid-state image capture device 100, power consumption of each component, operating frequency of each component, frame rate, and the number of categories including the number of category classifications and the number of sensed objects (number of regions of interest (ROIs)). The information included in the assessment is exemplary only and does not limit this disclosure.
[0087] The DNN control unit 160 obtains control information from the control unit 230 of the information processing device 200. Based on the control information, the DNN control unit 160 modifies the parameters of the DNN model stored in the storage unit 140. Therefore, the DNN control unit 160 changes the DNN model stored in the storage unit 140 to a different DNN model. Specifically, for example, the DNN control unit 160 searches for and reads the DNN model based on the identification information of the DNN model and the identification information of the solid-state image capture device 100. Then, for example, the DNN control unit 160 writes additions, changes, etc., to the DNN model based on the control information. Additionally, for example, the DNN control unit 160 generates a new DNN model based on the control information and deletes the DNN model stored in the storage unit 140. For example, the DNN control unit 160 can send a notification that the DNN model modification is complete to the information processing device 200.
[0088] Specifically, the DNN control unit 160 performs normalization, drop-out, momentum, and weight initialization on the DNN model. Normalization is a process that constrains the degrees of freedom of the weights to prevent overlearning during learning. Drop-out is a process that selects units of the multilayer network with a predetermined probability and updates the weights during learning. Momentum is a process that adds a constant multiple of the previous correction amount when calculating the correction amount of the weight count to improve the convergence performance of the gradient descent method during learning. Weight initialization is a process that sets the initial values to random values given by a Gaussian distribution.
[0089] More specifically, the DNN control unit 160 preferably changes at least one of the input image size and the filter size of the DNN model stored in the storage unit 140. Therefore, DNN can be performed on various types of image data.
[0090] For example, the DNN control unit 160 controls the DNN processing unit 130 based on control information to change the number of classifications in the DNN model stored in the storage unit 140. For example, if the initially stored DNN model is based on the assumption that the DNN is performed for indoor image data and the number of classifications is relatively small, the DNN model can be changed to a DNN model that can be applied to outdoor images by increasing the number of classifications.
[0091] For example, the DNN control unit 160 adjusts the power consumption of the DNN model stored in the storage unit 140 based on control information. For instance, when the DNN processing unit 130 consumes excessive power while executing the DNN model, power consumption can be reduced by changing the DNN model to one with lower power consumption.
[0092] For example, the DNN control unit 160 modifies the recognition accuracy of the DNN model stored in the storage unit 140 based on control information. For instance, if the DNN model executed by the DNN processing unit 130 results in low recognition accuracy, the DNN model is changed to a DNN model with high recognition accuracy. Therefore, the accuracy of object recognition can be improved. When the recognition accuracy is higher than desired, the DNN model is changed to a DNN model with low recognition accuracy. Therefore, computational speed can be increased and power consumption reduced.
[0093] For example, the DNN control unit 160 can switch the executed DNN model to a DNN model stored in the storage unit 140 based on control information. Therefore, the DNN model stored in the storage unit 140 can be used without generating a new DNN model, and thus changes to the DNN model can be easily performed. Consequently, execution results from different DNNs can be easily obtained.
[0094] For example, selector 170 receives digital image data that has undergone various processing from image capture processing unit 120. For example, selector 170 receives DNN execution results from DNN processing unit 130. For example, selector 170 receives evaluation information from evaluation unit 150. For example, selector 170 selectively outputs data received from image capture processing unit 120, DNN processing unit 130, or evaluation unit 150 to communication I / F 180 based on control signals from control unit (not shown) of the selector.
[0095] The communication I / F 180 includes a transmitting unit 181 and a receiving unit 182. Data is transmitted from the solid-state image capture device 100 to the information processing device 200 via the transmitting unit 181. Data is input from the information processing device 200 to the solid-state image capture device 100 via the receiving unit 182.
[0096] The communication control unit 190 controls the communication I / F 180. Therefore, the communication I / F 180 sends data to and receives data from the information processing device 200. For example, the communication control unit 190 can encrypt data. When performing encryption, the communication control unit 190 can use a public key such as Data Encryption Standard (DES) or Advanced Encryption Standard (AES). The communication control unit can use encryption modes such as Electronic Codebook (ECB), Cipher Block Chaining (CBC), Cipher Feedback (CFB), Output Feedback (OFB), and Counter (CTR).
[0097] The information processing device 200 includes a communication I / F 210, a communication control unit 220, and a control unit 230. In a first embodiment, the information processing device 200 may be implemented, for example, by an application processor.
[0098] The communication I / F 210 includes a receiving unit 211 and a transmitting unit 212. Data is input from the solid-state image capture device 100 to the control unit 230 via the receiving unit 211. Data is transmitted from the information processing device 200 to the solid-state image capture device 100 via the transmitting unit 212.
[0099] The communication control unit 220 controls the communication I / F 210. Therefore, the communication I / F 210 sends data to and receives data from the solid-state image capture device 100. For example, the communication control unit 220 can encrypt data and transmit data with the solid-state image capture device 100. When performing encryption, the communication control unit 220 can use a public key such as DES or AES. The communication control unit 220 can use encryption modes such as ECB, CBC, CFB, OFB, and CTR.
[0100] For example, control unit 230 acquires evaluation information generated by evaluation unit 150 from solid-state image capture device 100 via communication I / F 210. Control unit 230 generates control information for modifying the DNN model based on the evaluation information. For example, control unit 230 sends the control information to solid-state image capture device 100 to cause solid-state image capture device 100 to execute another DNN model. Specifically, for example, control unit 230 sends the control information to solid-state image capture device 100 to change the number of classifications, recognition accuracy, and power consumption of the DNN model. Therefore, an optimal DNN can be executed based on the environment surrounding solid-state image capture device 100. The control information includes information for modification related to model information of the DNN model, identification information of the DNN model, identification information of solid-state image capture device 100, power consumption, operating frequency, frame rate, and number of classifications.
[0101] For example, the control unit 230 preferably acquires information related to the DNN model stored in the storage unit 140 of the solid-state image capture device 100. Therefore, the control unit 230 can generate control information based on the DNN model stored in the storage unit 140 of the solid-state image capture device 100. Thus, the solid-state image capture device 100 can effectively modify the DNN model.
[0102] [1-2. Processing at the solid-state image capture system according to the first embodiment]
[0103] The following is for reference. Figure 5 The processing at the solid-state image capture device 100 and the information processing device 200 is described. Figure 5 This is a sequence diagram showing the processing procedures at the solid-state image capture device 100 and the information processing device 200.
[0104] First, the solid-state image capture device 100 performs a DNN on the input image (step S101). Specifically, the solid-state image capture device 100 causes the DNN processing unit 130 to read the DNN model from the storage unit 140 and perform the DNN on the input image.
[0105] Subsequently, the solid-state image capture device 100 generates evaluation information for the DNN (step S102). Specifically, the solid-state image capture device 100 causes the evaluation unit 150 to generate evaluation information.
[0106] Subsequently, the solid-state image capture device 100 sends the evaluation information to the information processing device 200 (step S103). Specifically, the solid-state image capture device 100 causes the evaluation unit 150 to send the evaluation information to the information processing device 200 via the communication I / F 180.
[0107] Subsequently, the information processing device 200 generates control information (step S104). Specifically, the information processing device 200 causes the control unit 230 to generate control information based on the evaluation information.
[0108] Subsequently, the information processing device 200 sends control information to the solid-state image capture device 100 (step S105).
[0109] Subsequently, the solid-state image capture device 100 changes the DNN (step S106). Specifically, the solid-state image capture device 100 causes the DNN control unit 160 to change the DNN model stored in the storage unit 140.
[0110] The following is for reference. Figure 6 To describe the learning operations using DNN. Figure 6 This is a flowchart illustrating the process of using data pre-stored in a storage unit and learning processing using a DNN based on control signals.
[0111] First, the solid-state image capture device 100 sets the parameters of the DNN model (step S201). Specifically, based on the control signal from the information processing device 200, the solid-state image capture device 100 causes the DNN control unit 160 to control the DNN processing unit 130 to set the parameters of the DNN model stored in the storage unit 140.
[0112] Subsequently, the solid-state image capture device 100 executes the DNN (step S202). Specifically, the solid-state image capture device 100 causes the DNN processing unit 130 to execute the DNN based on the DNN model with parameters changed in step S201.
[0113] Subsequently, the solid-state image capture device 100 evaluates the DNN executed in step S202 (step S203). Specifically, the solid-state image capture device 100 causes the evaluation unit 150 to calculate the value of the loss function and the accuracy of the DNN.
[0114] Subsequently, the solid-state image capture device 100 performs learning until the value of the loss function and the accuracy of the DNN reach the expected value (step S204). Specifically, the solid-state image capture device 100 causes the DNN processing unit 130 to perform learning based on a control signal until the value of the loss function and the accuracy of the DNN reach the expected value.
[0115] Subsequently, the solid-state image capture device 100 stores the DNN model that has reached the desired value in the storage unit (step S205). Specifically, the solid-state image capture device 100 causes the DNN control unit 160 to control the DNN processing unit 130 to store the DNN model in the storage unit 140.
[0116] The following is for reference. Figure 7 and Figure 8 The learning process performed simultaneously with the sequential image capture execution in the image capture unit 110 is described. Figure 7 This is a flowchart illustrating the learning process performed simultaneously with sequential image capture in the image capture unit 110. Figure 8 This is a schematic diagram used to describe the processing of image data.
[0117] First, the solid-state image capture device 100 performs image capture of the optional object (step S301). Specifically, the solid-state image capture device 100 causes the image capture unit 110 to perform image capture of the optional object.
[0118] Subsequently, the solid-state image capture device 100 processes the image data obtained through image capture (step S302). Specifically, the solid-state image capture device 100 causes the image capture processing unit 120 to perform any one of preprocessing, data expansion, and data normalization on the image data.
[0119] Subsequently, the solid-state image capture device 100 stores the batch quantity in the storage unit 140 (step S303). Specifically, the solid-state image capture device 100 causes the DNN processing unit 130 to store the batch quantity in the storage unit 140.
[0120] Subsequently, the solid-state image capture device 100 sets the parameters of the DNN model (step S304). Specifically, the solid-state image capture device 100 causes the DNN control unit 160 to control the DNN processing unit 130 to set the parameters of the DNN model stored in the storage unit 140.
[0121] Subsequently, the solid-state image capture device 100 executes a DNN (step S305). Specifically, the solid-state image capture device 100 causes the DNN processing unit 130 to execute a DNN based on a DNN model with parameters changed in step S304. More specifically, the DNN processing unit 130 executes the DNN model for multiple image data and outputs a recognition result for each image data. In this process, the DNN processing unit 130 is set to an inference theory mode. For example, as Figure 8As shown, first image data ID1, second image data ID2 obtained by converting the aspect ratio of first image data ID1, and third image data ID3 obtained by adding noise to first image data ID1 are input to DNN processing unit 130. Then, DNN processing unit 130 outputs first recognition result RR1, second recognition result RR2, and third recognition result RR3 as recognition results for first image data ID1, second image data ID2, and third image data ID3, respectively. Facial positions are identified in each of the first recognition result RR1, second recognition result RR2, and third recognition result RR3. The facial positions identified in the first recognition result RR1, second recognition result RR2, and third recognition result RR3 are arranged to overlap each other as a fourth recognition result RR4.
[0122] Subsequently, the solid-state image capture device 100 evaluates the DNN executed in step S305 (step S306). Specifically, the solid-state image capture device 100 causes the evaluation unit 150 to calculate the value of the loss function and the accuracy of the DNN. In addition, the solid-state image capture device 100 causes the evaluation unit 150 to generate evaluation information based on the accuracy value of the DNN and the value of the loss function.
[0123] Subsequently, the solid-state image capture device 100 generates correct answer data based on the recognition results and evaluation information (step S307). Specifically, the solid-state image capture device 100 causes the evaluation unit 150 to generate correct answer data based on the first recognition result RR1, the second recognition result RR2, the third recognition result RR3, evaluation information, etc. For example, such as Figure 8 As shown, the evaluation unit 150 generates correct answer data by calculating the average of the first recognition result RR1, the second recognition result RR2, and the third recognition result RR3. In this example, the evaluation unit 150 generates correct answer data using an arithmetic mean, but it could also generate correct answer data using a weighted average. For example, the evaluation unit 150 can exclude recognition results with an accuracy equal to or lower than a predetermined threshold (recognition results with low accuracy) from the averaging process. When calculating the average of the recognition results, the evaluation unit 150 preferably corrects the coordinates (positional relationships) of each recognition result relative to the original image data. For example, for image data translated by the image capture processing unit 120, the recognition results obtained by the DNN processing unit 130 are also translated from the original image data. Therefore, the evaluation unit 150 preferably translates and corrects the recognition results to have the same coordinates as the original image data before averaging.
[0124] Subsequently, the solid-state image capture device 100 stores the correct answer data and image data in the storage unit 140 (step S308). Specifically, the solid-state image capture device 100 instructs the evaluation unit 150 to store the correct answer data and image data in the storage unit 140 as learning data LD1. Therefore, in this embodiment, the solid-state image capture device 100 can automatically generate teacher data based on multiple authentication results. In other words, the annotation process can be automated without manual operation.
[0125] Subsequently, the solid-state image capture device 100 performs learning until the value of the loss function and the accuracy of the DNN reach the desired values (step S309). Specifically, the solid-state image capture device 100 causes the DNN processing unit 130 to perform learning based on control signals until the value of the loss function and the accuracy of the DNN reach the desired values. More specifically, as Figure 8 As shown, the DNN control unit 160 controls the DNN processing unit 130 to perform the learning of the DNN model used in step S305 by using the learning data LD1 stored in the storage unit 140 in step S308. During this process, the DNN processing unit 130 is set to a learning mode. Note that the DNN processing unit 130 can perform DNN model learning by mixing the learning data LD1 with learning data pre-stored in the storage unit 140. In this case, preferably, a constant number of generated learning data included in the batch quantity stored in step S303 are mixed. The DNN processing unit 130 can perform learning using only the learning data LD1. As described above, in this embodiment, the annotation process is automated. Therefore, in this embodiment, the learning of the DNN model can be performed automatically without outputting image data from the solid-state image capture device 100 to the outside.
[0126] Subsequently, the solid-state image capture device 100 stores the learning results in the storage unit 140 (step S310). Specifically, the solid-state image capture device 100 causes the DNN control unit 160 to control the DNN processing unit 130 to store the learned DNN model in the storage unit 140.
[0127] Note that the evaluation unit 150 can generate learning data by re-performing the above process using the learned DNN model stored in the storage unit 140 in step S310. The solid-state image capture device 100 can perform DNN model learning not only using image data captured by the image capture unit 110 but also using the learning image data stored in the storage unit 140.
[0128] [1-3. Use Cases of the Solid-State Image Capture System According to the First Embodiment]
[0129] The following is for reference. Figure 9 To describe the use cases when changing the DNN model. Figure 9 It is a table listing the use cases when changing the DNN model.
[0130] Use case 1 describes the operation of the DNN processing unit 130 during inference. In this operation, the evaluation unit 150 of the solid-state image capture device 100 generates evaluation information for the executed DNN model. The control unit 230 of the information processing device 200 determines the target performance based on the evaluation information. In this case, the target performance is set as control information. In other words, in use case 1, the control information does not include the parameters of the DNN model. Then, the DNN control unit 160 of the solid-state image capture device 100 determines the parameters of the DNN model based on the control information and sets the parameters to the DNN model.
[0131] Use case 2 describes the operation of the DNN processing unit 130 during inference. In this operation, the evaluation unit 150 of the solid-state image capture device 100 generates evaluation information for the executed DNN model. The control unit 230 of the information processing device 200 determines the target performance based on the evaluation information and determines the parameters of the DNN model. In this case, the parameters of the DNN model are set as control information. In other words, in use case 2, the control information includes the target setting and the parameters of the DNN model. Then, the DNN control unit 160 of the solid-state image capture device 100 determines the parameters of the DNN model based on the control information and sets the parameters to the DNN model.
[0132] Use case 3 describes the operation of the DNN processing unit 130 during learning. In this operation, the evaluation unit 150 of the solid-state image capture device 100 generates evaluation information for the executed DNN model. The control unit 230 of the information processing device 200 determines the target performance based on the evaluation information. In this case, the target performance is set as control information. In other words, in use case 3, the control information does not include the parameters of the DNN model. Then, the DNN control unit 160 of the solid-state image capture device 100 determines the parameters of the DNN model based on the control information and sets the parameters to the DNN model.
[0133] Use case 4 describes the operation of the DNN processing unit 130 during learning. In this operation, the evaluation unit 150 of the solid-state image capture device 100 generates evaluation information for the executed DNN model. The control unit 230 of the information processing device 200 determines the target performance based on the evaluation information and determines the parameters of the DNN model. In this case, the parameters of the DNN model are set as control information. In other words, in use case 4, the control information includes the target setting and the parameters of the DNN model. Then, the DNN control unit 160 of the solid-state image capture device 100 determines the parameters of the DNN model based on the control information and sets the parameters to the DNN model.
[0134] [1-4. Modifications to the solid-state image capture system according to the first embodiment]
[0135] The following is for reference. Figure 10 The configuration of a solid-state image capture system according to a modified example of the first embodiment is described. Figure 10 This is a block diagram illustrating the configuration of a solid-state image capture system according to a modified example of the first embodiment.
[0136] like Figure 10 As shown, the solid-state image capture system 1A includes a solid-state image capture device 100A and an information processing device 200.
[0137] The solid-state image capture device 100A includes image capture units 110-1 and 110-2, image capture processing units 120-1 and 120-2, DNN processing units 130-1 and 130-2, storage units 140-1 and 140-2, evaluation units 150-1 and 150-2, DNN control units 160-1 and 160-2, selectors 170-1 and 170-2, multiplexer 171, demultiplexer 172, communication I / F 180, and communication control unit 190. Therefore, the solid-state image capture device 100A includes two image capture units, two image capture processing units, two DNN processing units, two storage units, two evaluation units, two DNN control units, and two selectors.
[0138] In this configuration, a solid-state image capture device 100A-1 comprises an image capture unit 110-1, an image capture processing unit 120-1, a DNN processing unit 130-1, a storage unit 140-1, an evaluation unit 150-1, a DNN control unit 160-1, and a selector 170-1. Similarly, a solid-state image capture device 100A-2 comprises an image capture unit 110-2, an image capture processing unit 120-2, a DNN processing unit 130-2, a storage unit 140-2, an evaluation unit 150-2, a DNN control unit 160-2, and a selector 170-2. Each component included in solid-state image capture devices 100A-1 and 100A-2 is identical to that in solid-state image capture device 100, and therefore its description is omitted.
[0139] The signals output from solid-state image capture devices 100A-1 and 100A-2 are input to multiplexer 171. Multiplexer 171 inputs the signals output from solid-state image capture devices 100A-1 and 100A-2 together to the transmission unit 181 of communication I / F 180. Therefore, the signals output from solid-state image capture devices 100A-1 and 100A-2 are transmitted to information processing device 200.
[0140] Control information output from the information processing device 200 is input to the demultiplexer 172 via the receiving unit 182 of the communication I / F 180. The demultiplexer 172 divides the input control information into two segments and inputs these two segments to the DNN control unit 160-1 and the DNN control unit 160-2, respectively. Therefore, the control information is input to the solid-state image capture device 100A-1 and the solid-state image capture device 100A-2.
[0141] exist Figure 10 In the present invention, each of the solid-state image capture devices 100A-1 and 100A-2 includes a storage unit, an evaluation unit, and a DNN control unit, but this configuration is merely exemplary and does not limit the present disclosure. The solid-state image capture devices 100A-1 and 100A-2 may share the storage unit, the evaluation unit, and the DNN control unit.
[0142] (2. Second Embodiment)
[0143] [2-1. Configuration of the solid-state image capture system according to the second embodiment]
[0144] The following is for reference. Figure 11 and Figure 12 The configuration of a solid-state image capture system according to a second embodiment of the present disclosure is described. Figure 11This is a block diagram illustrating an exemplary configuration of a solid-state image capture system according to a second embodiment of the present disclosure. Figure 12 This is a diagram illustrating the connection relationships of a solid-state image capture system according to a second embodiment of the present disclosure.
[0145] like Figure 11 As shown, the solid-state image capture system 1B includes a solid-state image capture device 100 and an information processing device 200A. The components and operations of each device included in the solid-state image capture system 1B are the same as those of the solid-state image capture system 1 according to the first embodiment, and therefore their description is omitted.
[0146] like Figure 12 As shown, the solid-state image capture device 100 and the information processing device 200A are connected to each other, for example, via an Internet communication network 300, to perform communication therebetween. In this case, the communication I / F 180 of the solid-state image capture device 100 and the communication I / F 210 of the information processing device 200A can be connected to each other via the Internet communication network 300 to perform communication therebetween. The communication I / F 180 of the solid-state image capture device 100 and the communication I / F 210 of the information processing device 200A can be connected to each other via wireless communication to perform communication therebetween. In the second embodiment, the information processing device 200A is, for example, a cloud server, which is connected to the solid-state image capture device 100 via the Internet communication network 300 and wireless communication to perform communication therebetween. The solid-state image capture system 1B is suitable, for example, for factory automation (FA) or surveillance cameras.
[0147] [2-2. Configuration of a modified example of the solid-state image capture system according to the second embodiment]
[0148] The following is for reference. Figure 13 The configuration of a solid-state image capture system according to a second embodiment of the present disclosure is described. Figure 13 This is a diagram illustrating a modified example of the connection relationship of a solid-state image capture system according to a second embodiment of the present disclosure.
[0149] The solid-state image capture system 1B-1 includes solid-state image capture devices 100-1, 100-2, ..., and 100-N (where N is an integer equal to or greater than 3), information processing devices 200A-1, 200A-2, ..., and 200-N. Specifically, in the solid-state image capture system 1B-1, each of the plurality of solid-state image capture devices is connected to each of the plurality of information processing devices via an Internet communication network 300 to perform communication therebetween. In the solid-state image capture system 1B-1, the number of solid-state image capture devices and the number of information processing devices may be equal to or different from each other.
[0150] Although the solid-state image capture system 1B-1 includes multiple solid-state image capture devices and multiple information processing devices, this configuration is merely exemplary and does not limit the present disclosure. The solid-state image capture system 1B-1 may, for example, include one solid-state image capture device and multiple information processing devices. Alternatively, the solid-state image capture system 1B-1 may, for example, include multiple solid-state image capture devices and one information processing device.
[0151] (3. Third embodiment)
[0152] [3-1. Configuration of the solid-state image capture system according to the third embodiment]
[0153] The following is for reference. Figure 14 and Figure 15 The configuration of a solid-state image capture system according to a third embodiment of the present disclosure is described. Figure 14 This is a block diagram illustrating an exemplary configuration of a solid-state image capture system according to a third embodiment of the present disclosure. Figure 15 This is a diagram illustrating the connection relationships of a solid-state image capture system according to a third embodiment of the present disclosure.
[0154] like Figure 14 As shown, the solid-state image capture system 1C includes a solid-state image capture device 100, an information processing device 200B, and a battery 400. The components of the solid-state image capture device 100 are the same as those of the solid-state image capture device 100 of the solid-state image capture system 1 according to the first embodiment, and therefore their description is omitted.
[0155] like Figure 15 As shown, the information processing device 200B is disposed inside the solid-state image capture device 100. Each of the solid-state image capture device 100 and the information processing device 200 is mounted as a SOC, MCM, SIP, SOP, etc. The battery 400 is connected to the solid-state image capture system 1C. The battery 400 supplies power to the solid-state image capture system 1C. The solid-state image capture system 1C may be suitable for, for example, an Internet of Things (IoT) camera.
[0156] Refer again Figure 14 The information acquisition device 200B includes a control unit 230. In a third embodiment, the evaluation unit 150, for example, directly outputs evaluation information to the control unit 230.
[0157] Control unit 230 receives evaluation information, for example, from evaluation unit 150. Control unit 230 is connected to battery 400, for example, and measures the remaining power of battery 400. In this case, control unit 230 generates control information for changing the DNN model, for example, based on the evaluation information and the remaining power of battery 400. Specifically, when the remaining power of battery 400 is low, control unit 230 generates control information for changing the DNN model to another DNN model with lower power consumption. Then, control unit 230 directly inputs the generated control information to DNN control unit 160. Therefore, DNN control unit 160 can control DNN processing unit 130 to perform changes to the DNN model based on the remaining power of battery 400.
[0158] (4. Hardware Configuration)
[0159] By, for example, having such Figure 16 The computer 100 shown in the configuration implements the solid-state image capture apparatus and information processing apparatus according to each of the above embodiments. An example of a solid-state image capture apparatus 100 according to the first embodiment is described below. Figure 16 This is a hardware configuration diagram illustrating an exemplary computer 1000 configured to perform the functions of a solid-state image capture device 100. The computer 1000 includes a CPU 1100, RAM 1200, ROM (Read-Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input-output interface 1600. The components of the computer 1000 are interconnected via a bus 1050.
[0160] The CPU 1100 operates based on computer programs stored in ROM 1300 or HDD 1400 and performs control of each component. For example, the CPU 1100 performs processing corresponding to various computer programs by loading computer programs stored in ROM 1300 or HDD 1400 into RAM 1200.
[0161] ROM 1300 stores boot programs such as the Basic Input / Output System (BIOS) executed by CPU 1100 when computer 1000 starts up, and computer programs that depend on the hardware of computer 1000.
[0162] HDD 1400 is a computer-readable recording medium in which a computer program executed by CPU 1100 and data used by the computer program are recorded in a non-transitory manner. Specifically, HDD 1400 is a recording medium that records a computer program according to the present disclosure, wherein the computer program is exemplary computer program data 1450.
[0163] Communication interface 1500 is an interface for connecting computer 1000 to an external network 1550 (e.g., the Internet). For example, through communication interface 1500, CPU 1100 receives data from another instrument, and data generated by CPU 1100 is sent to another instrument.
[0164] Input-output interface 1600 is an interface for connecting computer 1000 to input-output device 1650. For example, CPU 1100 receives data from input devices such as keyboards or mice through input-output interface 1600. CPU 1100 sends data to output devices such as displays, speakers, or printers through input-output interface 1600. Input-output interface 1600 can be used as a media interface for reading computer programs, etc., recorded on a predetermined recording medium. The medium is, for example, an optical recording medium such as a Digital Universal Disc (DVD) or Phase Change Rewritable Disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), magnetic tape, a magnetic recording medium, or a semiconductor memory.
[0165] For example, when the computer 1000 is used as the solid-state image capture device 100 according to the first embodiment, the CPU 1100 of the computer 1000 implements the function of each component included in the solid-state image capture device 100 by executing a computer program loaded on the RAM 1200. The HDD 1400 stores the computer program according to this disclosure therein. Note that the CPU 1100 reads and executes the computer program data 1450 from the HDD 1400, but these computer programs can be obtained from another device, for example, via an external network 1550.
[0166] (5. Exemplary applications of endoscopic surgery)
[0167] The technology disclosed herein (the Technology) is applicable to a variety of products. For example, the Technology disclosed herein can be applied to endoscopic surgery.
[0168] Figure 17 This is a diagram illustrating an exemplary schematic configuration of endoscopic surgery to which the technology of this disclosure (the technology) is applicable.
[0169] Figure 17 This illustrates a scenario where an operator (doctor) 11131 performs a medical procedure on a patient 11132 on bed 11133 using the endoscopic surgical system 11000. (Example) Figure 17 As shown, the endoscopic surgical system 11000 includes an endoscope 11100, other surgical instruments 11110 such as a pneumoperitoneum tube 11111 and an energy therapy device 11112, a support arm device 11120 supporting the endoscope 11100, and a trolley 11200 on which various devices for endoscopic medical procedures are mounted.
[0170] Endoscope 11100 includes a tube 11101 and a camera 11102. A portion of the tube 11101 extending a predetermined length from its front end is inserted into the body cavity of a patient 11132. The camera 11102 is connected to the bottom end of the tube 11101. In the illustrated example, endoscope 11100 is configured as a so-called rigid endoscope including a rigid tube 11101, but endoscope 11100 can also be configured as a so-called flexible endoscope including a flexible tube.
[0171] An opening for mounting the objective lens is provided at the front end of the endoscope tube 11101. The light source device 11203 is connected to the endoscope 11100, and the light generated by the light source device 11203 is guided to the front end of the endoscope tube 11101 via a light guide extending inside the endoscope tube 11101, and emitted through the objective lens toward the target to be observed in the body cavity of the patient 11132. Note that the endoscope 11100 can be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.
[0172] An optical system and an image capture element are housed inside the camera 11102, and reflected light (observation light) from the observed target is converged onto the image capture element by the optical system. The image capture element performs photoelectric conversion on the observation light, thereby generating an electrical signal corresponding to the observation light, in other words, an image signal corresponding to the observed image. The image signal is sent as raw data to the camera control unit (CCU) 11201.
[0173] The CCU 11201 includes a central processing unit (CPU) and a graphics processing unit (GPU), and controls the operation of the endoscope 11100 and the display device 11202 together. Additionally, the CCU 11201 receives image signals from the camera 11102 and provides various image processing techniques, such as image development processing (de-mosaic processing), to the image signals to display an image based on them.
[0174] The display device 11202 displays an image based on the image signal provided by the CCU 11201 for image processing, under the control of the CCU 11201.
[0175] The light source device 11203 includes a light source such as a light-emitting diode (LED) and supplies illumination light to the endoscope 11100 for image capture of surgical sites, etc.
[0176] Input device 11204 is an input interface for endoscopic surgical system 11000. Users can input various information and commands into endoscopic surgical system 11000 through input device 11204. For example, users can input commands to change the image capture conditions of endoscope 11100 (such as the type of illumination light, magnification, and focal length).
[0177] The treatment instrument control device 11205 controls the drive of the energy therapy instrument 11112 for use in tissue cauterization, slicing, vascular sealing, etc. The pneumoperitoneum device 11206 supplies gas into the body cavity through the pneumoperitoneum tube 11111, thereby inflating the patient's (11132) body cavity to obtain the field of vision of the endoscope 11100 and the operator's workspace. The recorder 11207 is a device capable of recording various information related to the medical procedure. The printer 11208 is a device capable of printing various information related to the medical procedure in various formats such as text, images, and graphics.
[0178] Note that the light source device 11203 supplying illumination light for capturing images of the surgical site to the endoscope 11100 can be implemented using a white light source configured as, for example, an LED, a laser beam source, or a combination thereof. When the white light source is configured as a combination of RGB laser beam sources, the output intensity and timing of each color (wavelength) can be controlled with high precision, thereby allowing adjustment of the white balance of the captured image at the light source device 11203. Furthermore, in this case, images corresponding to each of the RGB values can be captured in a time-division manner by illuminating the observation target with laser beams from each RGB laser beam source in a time-division manner and controlling the drive of the image capture element of the camera 11102 in synchronization with the illumination timing. Using this method, color images can be obtained without providing a color filter for the image capture element.
[0179] Furthermore, the drive of the light source device 11203 can be controlled so that the intensity of the output light changes at predetermined intervals. The drive of the image capture element of the camera 11102 is controlled synchronously with the timing of the light intensity change, thereby acquiring images in a time-division manner. Images can be synthesized to generate images with high dynamic range without the so-called underexposure or overexposure.
[0180] The light source device 11203 is capable of supplying light of a predetermined wavelength band for special light observation. In special light observation, for example, so-called narrowband light observation (narrowband imaging) is performed, in which predetermined tissues such as blood vessels on the mucosal surface are captured with high contrast by using light in a frequency band narrower than the illumination light used in normal observation (in other words, white light), taking advantage of the wavelength dependence of light absorption by body tissues. Alternatively, in special light observation, fluorescence observation can be performed, in which an image is obtained using fluorescence generated by irradiation with excitation light. In fluorescence observation, for example, body tissue is irradiated with excitation light to observe fluorescence from the body tissue (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue, and the body tissue is irradiated with excitation light corresponding to the fluorescence wavelength of the reagent to obtain a fluorescence image. The light source device 11203 is capable of supplying narrowband light and / or excitation light corresponding to this special light observation.
[0181] Figure 18 It is shown Figure 17 A block diagram illustrating an exemplary functional configuration of camera 11102 and CCU 11201 is shown.
[0182] Camera 11102 includes a lens unit 11401, an image capture unit 11402, a drive unit 11403, a communication unit 11404, and a camera control unit 11405. CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. Camera 11102 and CCU 11201 are connected to each other via a transmission cable 11400 to perform communication therebetween.
[0183] Lens unit 11401 is an optical system disposed at the portion connected to lens barrel 11101. Observation light obtained from the front end of lens barrel 11101 is guided to camera 11102 and incident on lens unit 11401. Lens unit 11401 is formed by combining multiple lenses, including zoom lenses and focusing lenses.
[0184] Image capture unit 11402 includes image capture elements. Image capture unit 11402 may include one image capture element (so-called single-plate configuration) or multiple image capture elements (so-called multi-plate configuration). When image capture unit 11402 has a multi-plate configuration, for example, image signals corresponding to RGB can be generated by the individual image capture elements, and these image signals can be synthesized to obtain a color image. Alternatively, image capture unit 11402 may include a pair of image capture elements for acquiring left-eye and right-eye image signals respectively for three-dimensional (3D) display. When performing 3D display, the operator 11131 can more accurately understand the depth of living tissue at the surgical site. Note that when image capture unit 11402 has a multi-plate configuration, multiple lens units 11401 are provided for each image capture element.
[0185] The image capture unit 11402 does not need to be located on the camera 11102. For example, the image capture unit 11402 can be located directly behind the objective lens inside the lens barrel 11101.
[0186] The drive unit 11403, implemented by an actuator and under the control of the camera control unit 11405, moves each of the zoom lens and focusing lens of the lens unit 11401 a predetermined distance along the optical axis. Therefore, the magnification and focus of the image captured by the image capture unit 11402 can be appropriately adjusted.
[0187] The communication unit 11404 is implemented by a communication device for communicating various information with the CCU 11201. The communication unit 11404 transmits the image signal acquired from the image capture unit 11402 as raw data to the CCU 11201 via the transmission cable 11400.
[0188] Communication unit 11404 receives control signals from CCU 11201 for controlling the drive of camera 11102 and supplies the control signals to camera control unit 11405. The control signals include image capture condition-related information such as the frame rate of a specified captured image, the exposure value when a specified image is captured, and / or the magnification and focus of a specified captured image.
[0189] Note that the aforementioned image capture conditions, such as frame rate, exposure value, magnification, and focus, can be appropriately specified by the user, or can be automatically set by the control unit 11413 of CCU 11201 based on the acquired image signal. In the latter case, endoscope 11100 has so-called automatic exposure (AE), automatic focus (AF), and automatic white balance (AWB) functions.
[0190] The camera control unit 11405 controls the driving of the camera 11102 based on the control signals received from the CCU 11201 via the communication unit 11404.
[0191] The communication unit 11411 is implemented by a communication device for communicating various information with the camera 11102. The communication unit 11411 receives image signals transmitted from the camera 11102 via a transmission cable 11400.
[0192] The communication unit 11411 sends control signals for controlling the camera 11102 to the camera 11102. Image signals and control signals can be sent via electrical communication, optical communication, etc.
[0193] The image processing unit 11412 performs various image processing operations on the image signal sent from the camera 11102 as raw data.
[0194] The control unit 11413 performs various controls related to image capture of surgical sites, etc., performed by the endoscope 11100, and the display of the captured images obtained by image capture of surgical sites, etc. For example, the control unit 11413 generates control signals for controlling the drive of the camera 11102.
[0195] Furthermore, the control unit 11413 uses the image signal processed by the image processing unit 11412 to display a captured image of the surgical site, etc., on the display device 11202. In this case, the control unit 11413 can identify various objects in the captured image using various image recognition technologies. For example, the control unit 11413 detects the edge shape, color, etc., of each object included in the captured image to identify, for example, surgical instruments such as surgical forceps, specific living sites, bleeding, and fog when using the energy therapy device 11112. When the display device 11202 displays the captured image, the control unit 11413 uses the recognition results to display various surgical support information on the image of the surgical site in an overlay manner. When the surgical support information is displayed and presented to the operator 11131 in an overlay manner, the burden on the operator 11131 can be reduced, and the operator 11131 can reliably perform medical surgery.
[0196] The transmission cable 11400 connecting the camera 11102 and the CCU 11201 is an electrical signal cable compatible with electrical signal communication, an optical fiber compatible with optical communication, or a composite cable thereof.
[0197] In the example shown, communication is performed in a wired manner using transmission cable 11400, but communication between camera 11102 and CCU 11201 can also be performed wirelessly.
[0198] The above description is an example of endoscopic surgery to which the technology of this disclosure applies. The technology of this disclosure is applicable to, for example, the endoscope 11100, the image capture unit 11402 of the camera 11102, and the image processing unit 11412 of the CCU 11201 described above. Specifically, the solid-state image capture device 100 according to this disclosure is applicable to the endoscope 11100, the image capture unit 11402 of the camera 11102, the image processing unit 11412 of the CCU 11201, and the like. When the technology of this disclosure is applied to these components, the performance of endoscopic surgery can be improved. For example, high-definition captured images can be obtained by using a solid-state image capture device 100 with an expanded dynamic range. Specifically, when capturing images inside and outside a living organism, objects in areas with large grayscale differences are easily identified. Furthermore, the solid-state image capture device can perform rapid operations, resulting in a reduction in the time spent on operations such as object sensing and control of the camera 11102.
[0199] (6. Exemplary Application 1)
[0200] The technology disclosed herein is applicable to a variety of products. For example, the technology disclosed herein can be applied to pathological diagnostic systems used by physicians and others to diagnose lesions by observing cells and tissues collected from patients, as well as diagnostic support systems (hereinafter referred to as diagnostic support systems). Diagnostic support systems may be whole slide imaging (WSI) systems that diagnose lesions or support diagnoses based on images acquired using digital pathology techniques.
[0201] Figure 19 This is a diagram illustrating an exemplary schematic configuration of a diagnostic support system 5500 applying the techniques of this disclosure. Figure 19 As shown, the diagnostic support system 5500 includes one or more pathology systems 5510. The diagnostic support system 5500 may also include a medical information system 5530 and a derivation device 5540.
[0202] Each of the one or more pathology systems 5510 is a system primarily for use by pathologists and installed in, for example, a laboratory or hospital. The pathology systems 5510 may be installed in different hospitals, and each pathology system 5510 is connected to the medical information system 5530 and the derivation device 5540 via various networks such as a wide area network (WAN) (including the Internet), a local area network (LAN), a public network, and a mobile communication network.
[0203] Each pathology system 5510 includes a microscope 5511, a server 5512, a display control device 5513, and a display device 5514.
[0204] Microscope 5511 has the functions of an optical microscope, performing image capture on an object placed on a glass slide and acquiring pathological images as digital images. The object of observation is, for example, tissue or cells collected from a patient, or it may be an organ, saliva, or blood.
[0205] Server 5512 stores and records the pathological images acquired by microscope 5511 in a storage unit (not shown). When a browsing request is received from display control device 5513, server 5512 searches for pathological images in the storage unit (not shown) and transmits the searched pathological images to display control device 5513.
[0206] Display control device 5513 transmits the request received from the user to view pathology images to server 5512. Then, display control device 5513 displays the pathology images received from server 5512 on display device 5514 using a liquid crystal display, electroluminescent (EL) display, or cathode ray tube (CRT) display. Note that display device 5514 can be compatible with 4K and 8K resolutions, and the number of display devices is not limited to one, but can be two or more.
[0207] When the object of observation is a solid substance such as an organ, it can be, for example, a stained section. For instance, this section can be produced by slicing a block cut from a sample such as an organ. During slicing, the block can be fixed with, for example, paraffin wax.
[0208] Staining of sections can be achieved through various staining methods, such as conventional staining like hematoxylin-eosin (HE) staining to visualize the morphology of the tissue, or immunohistochemical (IHC) staining to visualize the immune status of the tissue. A section can be stained using a variety of different reagents, or two or more sections (also known as adjacent sections) cut consecutively from the same piece can be stained using different reagents.
[0209] Microscope 5511 may include a low-resolution image capture unit for capturing images at low resolution and a high-resolution image capture unit for capturing images at high resolution. The low-resolution image capture unit and the high-resolution image capture unit may be different optical systems or they may be the same optical system. When the image capture units are the same optical system, the resolution of microscope 5511 can be changed according to the image capture target.
[0210] The microscope 5511 positions a slide containing the object of observation within the field of view of the stage. Microscope 5511 first acquires an entire image of the field of view using a low-resolution image capture unit, and designates the area of the object of observation from the acquired image. Subsequently, microscope 5511 divides the area containing the object of observation into multiple sub-regions of predetermined size, and sequentially captures images of each sub-region using a high-resolution image capture unit, thereby acquiring a high-resolution image of each sub-region. When switching target sub-regions, the stage, the image capture optics, or both can be moved. Each sub-region can overlap with adjacent sub-regions to prevent the creation of uncaptured areas due to accidental slide slippage. The entire image may include identifying information for associating the entire image with the patient. This identifying information may be, for example, a string or a QR code (registered trademark).
[0211] The high-resolution images acquired by microscope 5511 are input to server 5512. Server 5512 divides each high-resolution image into smaller local images (hereinafter referred to as tile images). For example, server 5512 divides a high-resolution image into 100 tile images, each consisting of 10 tiles × 10 tiles, in both the vertical and horizontal directions. When adjacent subdivided regions overlap, server 5512 can perform a stitching process on the adjacent high-resolution images using techniques such as template matching. In this case, server 5512 can generate tile images by dividing the overall high-resolution image joined together through the stitching process. However, the generation of tile images from high-resolution images can be performed before the aforementioned stitching process.
[0212] Server 5512 can generate smaller tile images by further subdividing each tile image. This generation of tile images can be repeated until a tile image of the size set to the minimum unit size is generated.
[0213] Once the smallest unit of tile image is generated in this manner, server 5512 performs a tile compositing process for each tile image, synthesizing a predetermined number of adjacent tile images to generate a single tile image. This tile compositing process can be repeated until a single tile image is finally generated. Through this process, a pyramid structure of tile image groups is generated, comprising one or more tile images at each level. In this pyramid structure, the number of pixels in a tile image at one level is equal to the number of pixels in a tile image at another level, but their resolutions differ from each other. For example, when synthesizing four 2×2 tile images to generate a single tile image at the upper level, the resolution of the upper-level tile image is half the resolution of the lower-level tile image used for synthesis.
[0214] When such a pyramid-structured group of tile images is created, the level of detail of the observed object displayed on the display device can be switched according to the hierarchy to which the target tile image belongs. For example, when using the bottom-level tile image, narrow areas of the observed object can be displayed in detail, while when using upper-level tile images, wide areas of the observed object can be displayed in a coarse manner.
[0215] The generated pyramid structure of tile images is stored in a storage unit (not shown), for example, along with identification information that uniquely identifies each tile image (referred to as tile identification information). When a request to acquire a tile image including the tile identification information is received from another device (e.g., display control device 5513 or derivation device 5540), server 5512 sends the tile image corresponding to the tile identification information to the other device.
[0216] Note that tile images as pathological images can be generated for each image capture condition, such as focal length or staining conditions. When generating tile images for each image capture condition, a specific pathological image can be displayed side-by-side with another pathological image that corresponds to an image capture condition different from the specific one and to the same region as the specific pathological image. Specific image capture conditions can be specified by the viewer. When the viewer specifies multiple image capture conditions, pathological images corresponding to the respective image capture conditions and the same region can be displayed side-by-side.
[0217] Server 5512 can store the pyramid-structured tile image group in another storage device besides server 5512, for example, in a cloud server. The partial or complete tile image generation process described above can be performed on a cloud server or similar location.
[0218] The display control device 5513 extracts the desired tile image from the pyramid-structured tile image group based on user input and outputs the tile image to the display device 5514. Through this process, the user can obtain the feeling of observing an object by changing the magnification of the observation. In other words, the display control device 5513 acts as a virtual microscope. The magnification of the virtual observation corresponds to the resolution in reality.
[0219] Note that high-resolution images can be captured using any method. High-resolution images can be obtained by capturing images of segmented regions while repeatedly stopping and moving the stage, or by capturing images of segmented regions while moving the stage at a predetermined speed. Processing to generate tile images from high-resolution images is not necessary, but the resolution of the overall high-resolution image stitched together can be gradually changed to generate images with progressively different resolutions. Similarly, in this case, a low-resolution image over a wide area can be progressively presented to the user, gradually progressing to a high-resolution image over a narrow area.
[0220] Medical information system 5530 is a so-called electronic medical record system, and it stores information such as patient identification information, patient disease information, examination and image information used for diagnosis, diagnostic results, and prescriptions. For example, pathological images obtained by capturing images of a patient's condition can be temporarily stored by server 5512 and then displayed on display device 5514 via display control device 5513. Pathologists using pathology system 5510 perform pathological diagnoses based on the pathological images displayed on display device 5514. The results of the pathological diagnoses performed by the pathologist are stored in medical information system 5530.
[0221] The derivation device 5540 can perform pathological image analysis. This analysis can utilize a learning model generated through machine learning. The derivation device 5540 can derive classification results for specific regions, tissue identification results, etc., as analysis results. Furthermore, the derivation device 5540 can derive identification results such as cell information, quantity, location, brightness information, and their score information. This information derived by the derivation device 5540 can be displayed as diagnostic support information on the display device 5514 of the pathology system 5510.
[0222] Note that the derivation device 5540 can be a server system consisting of one or more servers (including cloud servers). The derivation device 5540 can, for example, be incorporated into the display control device 5513 or server 5512 in the pathology system 5510. In other words, various analyses of pathological images can be performed in the pathology system 5510.
[0223] The technology of this disclosure is well adapted to the microscope 5511 among the aforementioned components. Specifically, the solid-state image acquisition device 100 according to this disclosure is well adapted to the microscope 5511. When the technology of this disclosure is applied to the microscope 5511, clearer pathological images can be obtained, thereby diagnosing lesions more accurately. In addition, pathological images can be obtained with lower latency, thereby performing diagnoses faster and more accurately.
[0224] (7. Exemplary application of moving objects)
[0225] The technology disclosed herein (the Technology) is applicable to a variety of products. For example, the Technology disclosed herein can be implemented as a device installed on any kind of mobile object such as automobiles, electric vehicles, hybrid electric vehicles, autonomous two-wheelers, bicycles, personal mobility devices, airplanes, drones, ships, or robots.
[0226] Figure 20 This is a block diagram illustrating an exemplary schematic configuration of a vehicle control system to which the technology of this disclosure is applicable, serving as an exemplary mobile object control system.
[0227] The vehicle control system 12000 includes multiple electronic control units interconnected via a communication network 12001. Figure 20 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. Additionally, a microcomputer 12051, a voice and image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown as the functional configuration of the integrated control unit 12050.
[0228] The drive system control unit 12010 controls the operation of devices related to the vehicle's drive system according to various computer programs. For example, the drive system control unit 12010 serves as a control device for generating drive power such as an internal combustion engine or drive motor for generating drive power for the vehicle, a drive power transmission mechanism for transmitting drive power to the wheels, a steering mechanism configured to adjust the vehicle's rudder angle, a braking device configured to generate braking force for the vehicle, and so on.
[0229] The body system control unit 12020 controls the operation of various devices installed on the vehicle body according to various computer programs. For example, the body system control unit 12020 acts as a control device for keyless entry systems, smart key systems, power windows, and various lights such as headlights, taillights, brake lights, indicator lights, and fog lights. In this case, the body system control unit 12020 can receive radio waves or various switching signals transmitted from a portable device used as a key. The body system control unit 12020 receives the input of radio waves or signals and controls the vehicle's door locking devices, power windows, lights, etc.
[0230] The external information detection unit 12030 detects external information about the vehicle on which the vehicle control system 12000 is installed. For example, the external information detection unit 12030 is connected to the image capture unit 12031. The external information detection unit 12030 causes the image capture unit 12031 to capture external images and receives the captured images. Based on the received images, the external information detection unit 12030 can perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc.
[0231] Image capture unit 12031 is a light sensor configured to receive light and output an electrical signal based on the amount of light received. Image capture unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by image capture unit 12031 can be visible light or invisible light such as infrared.
[0232] Internal information detection unit 12040 detects internal information. Internal information detection unit 12040 is connected to driver state detection unit 12041, for example, configured to detect driver state. Driver state detection unit 12041 includes, for example, a camera configured to capture images of the driver, and internal information detection unit 12040 can calculate the driver's fatigue or level of concentration, or determine whether the driver is dozing off, based on detection information input from driver state detection unit 12041.
[0233] The microcomputer 12051 can calculate control target values for the drive power generation device, steering mechanism, or braking device based on internal and external information acquired by the external information detection unit 12030 and the internal information detection unit 12040, and can output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control to realize the functions of an advanced driver assistance system (ADAS), including collision avoidance or impact reduction, following based on inter-vehicle distance, speed maintenance, collision warning, lane departure warning, etc.
[0234] The microcomputer 12051 can perform cooperative control, for example, autonomous driving for the vehicle to drive independently of the driver's operation, by controlling the drive power generation device, steering mechanism, braking device, etc., based on information about the vehicle's surroundings obtained by the external information detection unit 12030 and the internal information detection unit 12040.
[0235] The microcomputer 12051 can output control commands to the vehicle system control unit 12020 based on external information acquired by the external information detection unit 12030. For example, the microcomputer 12051 can control the headlights based on the position of the preceding vehicle or an approaching vehicle sensed by the external information detection unit 12030, and can perform cooperative control to achieve dimming such as switching from high beam to low beam.
[0236] The voice and image output unit 12052 sends an output signal of at least one of voice and image to an output device capable of visually or audibly notifying a person inside the vehicle or outside. Figure 20 In the example, audio speaker 12061, display unit 12062, and dashboard 12063 are exemplarily shown as output devices. Display unit 12062 may include, for example, at least one of an in-vehicle display and a head-up display.
[0237] Figure 21 This is a diagram showing an exemplary mounting location of the image capture unit 12031.
[0238] exist Figure 21 In the vehicle 12100, image units 12101, 12102, 12103, 12104, and 12105 are used as image capture unit 12031.
[0239] Image capture units 12101, 12102, 12103, 12104, and 12105 are disposed at locations such as the front nose, rearview mirror, rear bumper, rear door, and upper part of the windshield of vehicle 12100. Image capture unit 12101 at the front nose and image capture unit 12105 at the upper part of the windshield primarily acquire images of the front of vehicle 12100. Image capture units 12102 and 12103 at the rearview mirror primarily acquire images of the sides of vehicle 12100. Image capture unit 12104 at the rear bumper or rear door primarily acquires images of the rear of vehicle 12100. The front images acquired by image capture units 12101 and 12105 are mainly used to detect vehicles ahead, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.
[0240] Notice, Figure 21Exemplary image capture ranges of image capture units 12101 to 12104 are shown. Image capture range 12111 indicates the image capture range of image capture unit 12101 located at the front nose, image capture ranges 12112 and 12113 indicate the image capture ranges of image capture units 12102 and 12103 located at the respective rearview mirrors, and image capture range 12114 indicates the image capture range of image capture unit 12104 located at the rear bumper or rear door. For example, when viewing the vehicle 12100 from above, the image data captured by image capture units 12101 to 12104 can be combined to obtain a panoramic image.
[0241] At least one of the image capture units 12101 to 12104 may have the function of acquiring distance information. For example, at least one of the image capture units 12101 to 12104 may be a stereo camera made of a plurality of image capture elements, or may be an image capture element including pixels for phase difference detection.
[0242] For example, the microcomputer 12051 calculates the distance to each stereoscopic object in the image capture range 12111 to 12114 and the time change of that distance (relative to the speed of the vehicle 12100) based on distance information obtained from the image capture units 12101 to 12104. Specifically, it extracts the stereoscopic object that is closest to the vehicle 12100 on its travel path and is traveling at a predetermined speed (e.g., equal to or greater than 0 km / h) in a direction substantially the same as the direction of the vehicle 12100 as the preceding vehicle. Furthermore, the microcomputer 12051 pre-sets the distance to be maintained with the preceding vehicle, thereby performing automatic braking control (including follow-stop control), automatic acceleration control (including follow-start control), etc. In this way, cooperative control, for example, for autonomous driving, can be performed, in which the vehicle drives autonomously independent of the driver's operation.
[0243] For example, microcomputer 12051 can classify stereoscopic object data related to stereoscopic objects into two-wheeled vehicles, standard vehicles, large vehicles, pedestrians, utility poles, and other stereoscopic objects based on distance information obtained from image capture units 12101 to 12104, extract stereoscopic object data, and use the stereoscopic object data for automatic obstacle avoidance. For example, microcomputer 12051 identifies obstacles around vehicle 12100 as obstacles that are visually identifiable to the driver of vehicle 12100 or obstacles that are difficult to visually identify. Then, microcomputer 12051 determines a collision risk indicating the degree of danger of colliding with each obstacle, and if the collision risk is equal to or greater than a set value and a collision is likely to occur, microcomputer 12051 can perform collision avoidance operation support by outputting an alarm to the driver using audio speaker 12061 and display unit 12062 or by performing forced deceleration and avoidance steering through drive system control unit 12010.
[0244] At least one of the image capture units 12101 to 12104 may be an infrared camera configured to detect infrared light. For example, the microcomputer 12051 can identify a pedestrian by determining whether a pedestrian exists in the images captured by the image capture units 12101 to 12104. This pedestrian identification is performed, for example, by extracting feature points from the images captured by the image capture units 12101 to 12104 (which are infrared cameras), and by performing pattern matching processing on a series of feature points indicating the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the images captured by the image capture units 12101 to 12104 and identifies the pedestrian, the voice image output unit 12052 controls the display unit 12062 to display the identified pedestrian superimposed with a rectangular outline for emphasis. The voice image output unit 12052 can control the display unit 12062 to display an icon or similar icon indicating the pedestrian at a desired location.
[0245] The above description is an example of a vehicle control system to which the technology of this disclosure applies. The technology of this disclosure is applicable to the image capture unit 12031 in the above-described components. Specifically, the solid-state image capture device 100 according to this disclosure is applicable to the image capture unit 12031. When the technology of this disclosure is applied to the image capture unit 12031, the performance of the vehicle control system can be improved. For example, high-definition captured images can be obtained by using the solid-state image capture device 10 with an expanded dynamic range. Specifically, when capturing images inside and outside the vehicle, objects in areas with large grayscale differences are easily identified. In addition, the solid-state image capture device can perform fast operation, which reduces the time spent from object sensing to vehicle control operation.
[0246] Note that the effects described in this specification are merely exemplary and not limiting, but any other effects may be achieved.
[0247] Note that this technology can be configured as described below.
[0248] (1) A solid-state image capture device, comprising:
[0249] The DNN processing unit is configured to perform a DNN on the input image based on a DNN model; and
[0250] The DNN control unit is configured to receive control information generated from evaluation information based on the execution results of the DNN, and to modify the DNN model based on the control information.
[0251] (2) The solid-state image capture device according to (1), wherein the DNN control unit changes the parameters of the DNN model based on control information.
[0252] (3) The solid-state image capture device according to (2), wherein the DNN control unit changes at least one of the input image size and the filter size of the DNN model.
[0253] (4) The solid-state image capture device according to any one of (1) to (3) further includes a storage unit therein storing at least one DNN model executed by the DNN processing unit.
[0254] (5) The solid-state image capture device according to (4), wherein the storage unit stores learning data for the DNN processing unit therein.
[0255] (6) A solid-state image capture device according to (4) or (5), wherein the DNN control unit switches the executed DNN model to the DNN model stored in the storage unit based on control information.
[0256] (7) The solid-state image capture device according to any one of (1) to (6) further includes an evaluation unit configured to generate evaluation information by evaluating the execution results of the DNN.
[0257] (8) The solid-state image capture device according to (7), wherein the evaluation unit generates evaluation information by means of the evaluation processing situation during inference and learning of the DNN processing unit.
[0258] (9) A solid-state image capture device according to (7) or (8), wherein the evaluation unit generates correct answer data based on the recognition results and evaluation information of the DNN.
[0259] (10) The solid-state image capture device according to (9), wherein the DNN control unit changes the DNN model based on the correct answer data.
[0260] (11) A solid-state image capture device according to any one of (1) to (10), wherein the evaluation information includes information related to the recognition results of the DNN.
[0261] (12) An information processing apparatus, including a control unit configured to generate control information for changing a DNN model based on evaluation information of the execution result of a DNN model executed by an external device.
[0262] (13) The information processing apparatus according to (12), wherein the control unit acquires information related to the DNN model stored in an external device.
[0263] (14) An information processing system, comprising:
[0264] Solid-state image capture device; and
[0265] Information processing device
[0266] Solid-state image capture device, comprising:
[0267] The DNN processing unit is configured to perform a DNN on the input image based on a DNN model; and
[0268] The DNN control unit is configured to receive control information generated from evaluation information based on the execution results of the DNN, and to modify the DNN model based on this control information.
[0269] The information processing device includes a control unit configured to generate control information for modifying the DNN model based on the execution results of the DNN model.
[0270] (15) An information processing method, comprising:
[0271] The DNN model is used to execute a DNN on the input image; and
[0272] It receives evaluation information based on the execution results of the DNN and generates control information, and modifies the DNN model based on this control information.
[0273] (16) An information processing method, the method comprising generating control information for modifying a DNN model based on evaluation information of the execution results of a DNN model.
[0274] (17) A computer configured to function as a computer program that:
[0275] The DNN processing unit is configured to perform a DNN on the input image based on a DNN model; and
[0276] The DNN control unit is configured to receive control information generated from evaluation information based on the execution results of the DNN, and to modify the DNN model based on the control information.
[0277] (18) A computer program configured to use a computer as a control unit, the control unit being configured to generate control information for changing the DNN model based on evaluation information of the execution results of the DNN model.
[0278] Reference Symbol List
[0279] 1 Solid-state image capture system
[0280] 100 Solid-State Image Capture Device
[0281] 110 Image Capture Unit
[0282] 120 Image Capture Processing Units
[0283] 130 DNN processing units
[0284] 140 storage units
[0285] 150 evaluation units
[0286] 160 DNN control unit
[0287] 170 Selector
[0288] 171 Multiplexer
[0289] 172 Demultiplexer
[0290] 180, 210 Communication I / F
[0291] Transmitting units 181 and 212
[0292] Receiver units 182 and 211
[0293] 190, 220 Communication Control Unit
[0294] 200 Information Processing Devices
[0295] 230 Control Unit.
Claims
1. A solid-state image capture device comprising: A deep neural network processing unit, configured to execute a deep neural network on an input image based on a deep neural network model and output an execution result; An evaluation unit, which generates evaluation information based on the execution result of the deep neural network; as well as A deep neural network control unit is configured to receive control information generated based on the evaluation information, and change the number of classifications of the deep neural network model based on the control information, wherein the evaluation unit generates correct answer data by calculating the average value of a plurality of the execution results, and the correct answer data is stored as learning data in the solid-state image capture device for automatically performing learning in the solid-state image capture device.
2. The solid-state image capturing device according to claim 1, wherein: The deep neural network control unit further changes at least one of an input image size and a filter size of the deep neural network model.
3. The solid-state image capture device according to claim 1 further includes a storage unit that stores at least one of the deep neural network models executed by the deep neural network processing unit.
4. The solid-state image capturing device according to claim 3, wherein: The storage unit stores learning data for the deep neural network processing unit.
5. The solid-state image capturing device according to claim 3, wherein: The deep neural network control unit switches the executed deep neural network model to the deep neural network model stored in the storage unit based on the control information.
6. The solid-state image capturing device according to claim 1, wherein: The evaluation unit generates the evaluation information by evaluating the processing situations of the deep neural network processing unit during inference and learning.
7. The solid-state image capturing device according to claim 1, wherein: The deep neural network control unit changes the deep neural network model based on the correct answer data.
8. The solid-state image capturing device according to claim 1, wherein: The evaluation information includes information related to the recognition result of the deep neural network.
9. An information processing system comprising: a solid-state image capture device; as well as information processing device, The solid-state image capture device comprises: A deep neural network processing unit, configured to execute a deep neural network on an input image based on a deep neural network model and output an execution result; an evaluation unit, which generates evaluation information based on the execution result of the deep neural network; and a deep neural network control unit configured to receive control information generated based on evaluation information, and to change the number of classifications of the deep neural network model based on the control information, wherein the evaluation unit generates correct answer data by calculating an average value of a plurality of the execution results, the correct answer is stored as learning data in the solid-state image capture device for automatically performing learning in the solid-state image capture device, and The information processing device includes a control unit configured to generate the control information for changing the deep neural network model based on the evaluation information of the execution result of the deep neural network model.
10. An information processing method, comprising: Execute the deep neural network on the input image based on the deep neural network model and output the execution result; generating evaluation information based on the execution results of the deep neural network; and Receive control information generated based on the evaluation information, and change the number of classifications of the deep neural network model based on the control information, wherein the evaluation unit generates correct answer data by calculating the average value of multiple execution results, and the correct answer is stored as learning data in the solid-state image capture device for automatically performing learning in the solid-state image capture device.
11. A computer readable medium having stored thereon a computer program which, when executed, causes a computer to function as: A deep neural network processing unit, configured to execute a deep neural network on an input image based on a deep neural network model and output an execution result; An evaluation unit, which generates evaluation information based on the execution result of the deep neural network; as well as A deep neural network control unit is configured to receive control information generated based on the evaluation information, and change the number of classifications of the deep neural network model based on the control information, wherein the evaluation unit generates correct answer data by calculating the average value of a plurality of the execution results, and the correct answer is stored as learning data in a solid-state image capture device for automatically performing learning in the solid-state image capture device.
