Imaging apparatus and data output method

Abstract

The present disclosure relates to an imaging apparatus and a data output method in which the risk of privacy violation can be reduced. The imaging apparatus includes: an imaging unit in which a plurality of pixels are two-dimensionally arranged to capture an image; a signal processing unit that performs signal processing of the image data based on an output from the imaging unit; and an output unit which is not directly connected to the imaging unit and is capable of outputting a result of the signal processing only from the outside. The signal processing unit deletes the image data after the signal processing. The present disclosure can be applied to an image sensor having an AI function.

Description

Technical Field

[0001] The present disclosure relates to an imaging device and a data output method, and more particularly, to an imaging device and a data output method capable of reducing the risk of privacy invasion. Background Art

[0002] In Patent Document 1, an image sensor is disclosed in which an imaging unit that images an image and a signal processing unit that performs signal processing on image data based on an output of the imaging unit are arranged in a single chip. The image sensor disclosed in Patent Document 1 is configured to selectively output at least one of a signal processing result of the signal processing, intermediate data acquired during the signal processing, and image data to the outside.

[0003] [Citation List]

[0004] [Patent Document]

[0005] [Patent Document 1] WO 2018 / 051809 Summary of the invention

[0006] [Technical issues]

[0007] According to the image sensor of Patent Document 1, only the signal processing result can be output to the outside, so there is an advantage from the viewpoint of privacy protection. However, since it is not the case that image data cannot be output, there is a risk of privacy invasion.

[0008] The present disclosure takes such situations into consideration and can reduce the risk of privacy invasion.

[0009] [Solution to the problem]

[0010] An imaging device according to a first aspect of the present disclosure is an imaging device comprising: an imaging unit in which a plurality of pixels are two-dimensionally aligned, the imaging unit being configured to capture an image; a signal processing unit being configured to perform signal processing of image data based on an output of the imaging unit; and an output unit which is not directly connected to the imaging unit, the output unit being configured to be able to output only a signal processing result of the signal processing to the outside, wherein the signal processing unit deletes the image data after the signal processing.

[0011] An imaging device according to a second aspect of the present disclosure is an imaging device, comprising: an imaging unit in which a plurality of pixels are two-dimensionally aligned and configured to capture an image; a signal processing unit configured to perform signal processing of image data based on an output of the imaging unit; an output unit configured to be able to output a signal processing result of the signal processing to the outside; and an inspection processing unit configured to perform inspection processing before signal processing to check that the image data is not included in the output of the signal processing unit.

[0012] A data output method according to a first aspect of the present disclosure is a data output method, comprising deleting image data after signal processing using an imaging device, the imaging device comprising: an imaging unit in which a plurality of pixels are two-dimensionally aligned, the imaging unit being configured to capture an image; a signal processing unit being configured to perform the signal processing on the image data based on an output of the imaging unit; and an output unit, not directly connected to the imaging unit, being configured to be able to output only the signal processing result of the signal processing to the outside.

[0013] In a first aspect of the present invention, an imaging device comprises: an imaging unit in which a plurality of pixels are two-dimensionally aligned, the imaging unit being configured to capture an image; a signal processing unit being configured to perform signal processing of image data based on an output of the imaging unit; and an output unit which is not directly connected to the imaging unit and is configured to be able to output only a signal processing result of the signal processing to the outside, the image data being deleted after the signal processing.

[0014] In a second aspect of the present disclosure, an imaging device includes: an imaging unit in which a plurality of pixels are two-dimensionally aligned, the imaging unit being configured to capture an image; a signal processing unit being configured to perform signal processing of image data based on an output of the imaging unit; and an output unit being configured to be able to output a signal processing result of the signal processing to the outside, performing a check process before the signal processing to check that the image data is not included in the output of the signal processing unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram showing a configuration example of a conventional imaging device.

[0016] Figure 2 is a block diagram showing a configuration example of an imaging device to which the technology according to the present disclosure is applied.

[0017] Figure 3 is a perspective view showing an overview of an external configuration example of the imaging device.

[0018] Figure 4 is a block diagram showing a configuration example of an imaging device according to the first embodiment.

[0019] Figure 5 is a diagram illustrating an example of access restriction of a working memory.

[0020] Figure 6 is a flowchart for describing the operation of the DSP.

[0021] Figure 7 is a block diagram showing a first configuration example of an imaging device according to the second embodiment.

[0022] Figure 8 is a flowchart describing the operation of the network inspection block.

[0023] Fig. 9 is a diagram for describing the structure of an inspection network.

[0024] Fig.10 is a flowchart for describing the operation of the overall imaging device.

[0025] Fig.11 is a block diagram showing a second configuration example of the imaging device according to the second embodiment.

[0026] Fig.12 is a flowchart for describing the operation of the inspection dedicated processor.

[0027] Fig.13 is a block diagram showing a third configuration example of the imaging device according to the second embodiment.

[0028] Fig.14 is a flowchart for describing the operation of the inspection dedicated processor.

[0029] Fig.15 is a diagram for describing an application example of the technology according to the present disclosure.

[0030] Fig.16 is a diagram for describing an application example of the technology according to the present disclosure. DETAILED DESCRIPTION

[0031] Hereinafter, a mode for carrying out the present disclosure (hereinafter referred to as an embodiment) will be described. Note that the description will be given in the following order.

[0032] 1. Problems of conventional technology and overview of technology related to the present invention

[0033] 2. First Embodiment (Deleting Image Data After Recognition Processing)

[0034] 3. Second Embodiment (Inspection Image Data Is Not Included in the Output of the Recognition Processing Unit)

[0035] 4. Application Examples

[0036] <1. Problems of Conventional Technology and Overview of Technology Related to the Present Invention>

[0037] (Conventional imaging device)

[0038] Figure 1 is a block diagram showing a configuration example of a conventional imaging device.

[0039] Figure 1The imaging device IS shown in the figure is, for example, a complementary metal oxide semiconductor (CMOS) image sensor configured using one chip, receives incident light from an optical system not shown in the drawing, performs photoelectric conversion, and outputs image data corresponding to the incident light from the optical system.

[0040] Furthermore, the imaging device IS performs, for example, signal processing such as recognition processing of recognizing a predetermined recognition target or the like using output image data or the like, and outputs a signal processing result of the signal processing.

[0041] like Figure 1 As shown, the imaging device IS includes an imaging block 10, a signal processing block 20, a selector 30, and an external I / F 40. The imaging block 10 and the signal processing block 20 are electrically connected using a connection line, as are the signal processing block 20 and the selector 30. In addition, the imaging block 10 and the selector 30 are directly electrically connected using a connection line.

[0042] The imaging block 10 includes a pixel array 11 , a column analog-to-digital converter (ADC) 12 , and a control unit 13 .

[0043] The pixel array 11 is configured as an imaging unit in which a plurality of pixels are aligned two-dimensionally. The pixel array 11 is driven by a control unit 13 to capture an image. More specifically, the pixel array 11 in each pixel receives incident light from an optical system not shown in the drawings, performs photoelectric conversion, and outputs an analog image signal corresponding to the incident light.

[0044] The column ADC 12 reads an analog image signal output by the pixel array 11 and performs AD conversion according to the control of the control unit 13. The column ADC 12 outputs a digital image signal acquired by performing AD conversion of the analog image signal as image data (raw data).

[0045] The raw data output by the column ADC 12 is supplied to the signal processing block 20 and to the selector 30 .

[0046] The control unit 13 controls the operations of the pixel array 11 and the column ADC 12. More specifically, the control unit 13 controls the driving of each pixel of the pixel array 11 and the reading and AD conversion of an image signal using the column ADC 12.

[0047] The signal processing block 20 includes an image signal processing (ISP) unit 21 , a digital signal processor (DSP) 22 , a processing result I / F 23 , and an external I / F 24 .

[0048] The units configuring the signal processing block 20 are interconnected by a bus, and can exchange information as necessary.

[0049] Although not shown in the figure, a central processing unit (CPU) that performs various processes starting from control of the entire signal processing block 20 by executing a program stored in a memory not shown in the figure is arranged in the signal processing block 20.

[0050] The ISP unit 21 performs various image processing on the image data (raw data) from the imaging block 10. For example, the ISP unit performs high dynamic range (HDR) conversion processing, defect correction, development processing, and the like on the image data from the imaging block 10.

[0051] The DSP 22 functions as a signal processing unit that performs signal processing using image data after the image processing performed by the ISP unit 21 .

[0052] The processing result I / F 23 supplies the signal processing result using the image data performed by the DSP 22 and the result obtained by performing various image processing on the image data performed by the ISP unit 21 to the selector 30 .

[0053] The external I / F 24 outputs to the outside a signal processing result using the image data performed by the DSP 22. For example, the external I / F 24 is configured using a serial communication I / F such as a serial peripheral interface (SPI).

[0054] The selector 30 performs output control of selectively outputting image data (raw data) from the imaging block 10 and a signal processing result from the signal processing block 20 to the outside from the external I / F 40. In other words, the selector 30 selects image data from the imaging block 10, selects a signal processing result from the signal processing block 20, or both, and supplies them to the external I / F 40.

[0055] Intermediate data acquired during signal processing using the image data performed by the DSP 22 may be supplied to the selector 30 through the processing result I / F 23. In this case, the selector 30 performs output control of selectively outputting to the outside any one of the image data from the imaging block 10 and the signal processing result from the signal processing block 20 and the intermediate data acquired from the external I / F 40 during the signal processing. In the case where the intermediate data acquired during signal processing (e.g., recognition processing) is output to the outside from the external I / F 40, the intermediate data may be provided to a debugger of a program for executing the signal processing.

[0056] The external I / F 40 is an I / F that outputs the image data and the signal processing result supplied from the selector 30 to the outside. As the external I / F 40, for example, a parallel communication I / F having a relatively high speed such as a mobile industry processor interface (MIPI) may be employed.

[0057] In the external I / F 40, the image data from the imaging block 10 or the signal processing result according to the DSP 22 from the signal processing block 20 is output to the outside according to the output control of the selector 30. Therefore, for example, in the case where only the signal processing result according to the DSP 22 of the signal processing block 20 is required and the image data (captured image) is not required at the outside, only the signal processing result can be output, and thus the amount of data output to the outside from the external I / F 40 can be reduced. In addition, in the case where the image data (captured image) is not necessary, only the signal processing result according to the DSP 22 can also be output from the signal processing block 20 to the outside from the external I / F 24.

[0058] In this way, according to the imaging device IS, only the signal processing result can be output to the outside, so it can be considered that there is an advantage from the viewpoint of privacy protection. However, since it is impossible to output image data, there is a risk of privacy invasion.

[0059] (Imaging device to which the technology according to the present disclosure is applied)

[0060] Figure 2 is a block diagram showing a configuration example of an imaging device to which the technology according to the present disclosure is applied.

[0061] Moreover, in Figure 2 An imaging device 1 shown in FIG. 1 is, for example, a CMOS image sensor using a one-chip configuration, and receives incident light from an optical system not shown in the figure, performs photoelectric conversion, and outputs image data corresponding to the incident light from the optical system.

[0062] Furthermore, the imaging device 1 performs signal processing, for example, such as recognition processing of recognizing a predetermined recognition target using output image data, and outputs a signal processing result of the signal processing.

[0063] like Figure 2 As shown, the imaging device 1 includes an imaging block 10 and a signal processing block 20. In the imaging device 1, the same reference symbols are assigned to the components as in Figure 1 The same components as those included in the imaging device IS shown in FIG. 1 are shown in FIG. 1 , and descriptions thereof will be appropriately omitted.

[0064] Figure 2 The imaging device 1 shown in FIG. Figure 1 The imaging device IS shown in FIG. 1 is different in that it does not include a selector 30 or an external I / F 40. In other words, in FIG. Figure 2 The imaging device 1 shown in FIG. 1 does not have an external I / F directly connected to the pixel array 11 (imaging block 10 ) as an imaging unit.

[0065] also, Figure 2 The imaging device 1 shown in FIG. Figure 1 The imaging apparatus IS shown in FIG. 1 is different in that it includes a CPU 51 and an external I / F 52 inside the signal processing block 20 instead of the processing result I / F 23 and the external I / F 24 .

[0066] The CPU 51 performs various processes starting with control of the entire signal processing block 20 by executing a program stored in a memory not shown in the drawing.

[0067] with Figure 1 The external I / F 52 outputs the signal processing result of the image data according to the signal processing performed by the DSP 22 to the outside. Here, the external I / F 52 is not limited to a serial communication I / F such as SPI, and may be configured using a parallel communication I / F such as MIPI or any other communication I / F.

[0068] In other words, the external I / F 52 is configured as an output unit capable of outputting only the signal processing result of the signal processing to the outside, and is not directly connected to the pixel array 11 (imaging block 10 ) as an imaging unit.

[0069] Furthermore, the imaging device 1 may adopt a configuration in which the DSP 22 deletes the image data after signal processing or a configuration in which the CPU 51 checks that the image data is not included in the output of the DSP 22 before signal processing.

[0070] According to this, the imaging device 1 can realize a configuration in which image data cannot be output to the outside.

[0071] Figure 3 It shows Figure 2 2 is a perspective view of an overview of an external configuration example of the imaging device 1 shown in FIG.

[0072] For example, the imaging device 1 may be configured as a single-chip semiconductor device having a stacked structure in which a plurality of wafers are stacked.

[0073] More specifically, if Figure 3 As shown, the imaging device 1 is configured by stacking two dies including a die 61 and a die 62 .

[0074] exist Figure 3 , the pixel array 11 is mounted in a bare chip 61 on the upper side, and the column ADC 12, the control unit 13, the ISP unit 21, the DSP 22, the CPU 51, and the external I / F 52 are mounted in a bare chip 62 on the lower side.

[0075] The upper die 61 and the lower die 62 are electrically connected, for example, by forming a through hole that passes through the die 61 and reaches the die 62, by performing Cu-Cu bonding that directly connects the Cu wiring exposed on the lower surface side of the die 61 with the Cu wiring exposed on the upper surface side of the die 62, etc.

[0076] Here, as a system for performing AD conversion on an image signal output by the pixel array 11 , the column ADC 12 may employ, for example, a column-parallel AD system or an area AD system.

[0077] As a column parallel AD system, for example, an ADC is provided for each column of pixels configuring the pixel array 11, and the ADC of each column is responsible for AD conversion of the pixel signal of the column, thereby performing AD conversion of the image signal of the pixels of each column of a row in parallel. In the case of adopting the column parallel AD system, some column ADCs 12 that perform AD conversion of the column parallel AD system can be mounted in the bare chip 61 on the upper side.

[0078] In the area AD system, the pixels configuring the pixel array 11 are divided into a plurality of blocks, and an ADC is provided for each block. The ADC of each block is responsible for the AD conversion of the pixel signal of the block, and thus the AD conversion of the image signals of the pixels of the plurality of blocks is performed in parallel. In the area AD system, the AD conversion (readout and AD conversion) of the image signal can be performed using a block as the minimum unit only for necessary pixels among the pixels configuring the pixel array 11.

[0079] Furthermore, when the area of ​​the imaging device 1 is allowed to be increased, the imaging device 1 can be configured using one wafer.

[0080] In addition, Figure 3 In the example shown in , although the one-chip imaging device 1 is configured by stacking two bare chips 61 and 62 , the one-chip imaging device 1 may be configured by stacking three or more chips.

[0081] As the signal processing performed by the imaging device 1, that is, the signal processing of the DSP 22, for example, recognition processing of recognizing a predetermined recognition target from image data can be adopted.

[0082] In addition, the imaging device 1 may receive the output of a distance sensor such as a ToF (Time of Flight) sensor, which is arranged to have a predetermined positional relationship with the imaging device 1 using the external I / F 52. In this case, as the signal processing of the DSP 22, for example, a fusion process of acquiring a distance with high accuracy by integrating the output of the distance sensor and the captured image, such as a process of eliminating noise of a distance image acquired from the output of the distance sensor received from the external I / F 52 using the captured image, may be adopted.

[0083] Furthermore, the imaging device 1 may receive an image output by another image sensor provided to have a predetermined positional relationship with the imaging device 1 using the external I / F 52. In this case, as the signal processing of the DSP 22, for example, a self-positioning estimation process (SLAM (Simultaneous Localization and Mapping)) in which the image received by the external I / F 52 and the image captured by the imaging device 1 are used as a stereoscopic image may be adopted.

[0084] Hereinafter, an embodiment of an imaging device (image sensor) configured to perform recognition processing as signal processing of the DSP 22 will be described.

[0085] <2. First Embodiment>

[0086] (Configuration Example of Imaging Device)

[0087] Figure 4 is a block diagram showing a configuration example of an imaging device according to the first embodiment.

[0088] Figure 4 The imaging device 100 shown in FIG. 1 includes a pixel array 111 , a column ADC 112 , an ISP block 113 , a DSP 114 , a program memory 115 , a working memory 116 , a data transfer block 117 , and an output I / F 118 .

[0089] The pixel array 111, the column ADC 112 and the ISP block 113 have respective Figure 2 The pixel array 11, the column ADC 12, and the ISP unit 21 in the illustrated imaging device 1 have similar functions, and thus description thereof will be omitted.

[0090] The DSP 114 performs recognition processing of recognizing a predetermined recognition target of the image data as signal processing using the image data after the image processing according to the ISP block 113 .

[0091] In the program memory 115, a program for the DSP 114 to perform recognition processing is stored. The program memory 115 is configured using a ROM (Read Only Memory) and does not allow the program for performing recognition processing to be rewritten. The program is configured as a program in which processing details of signal processing using a neural network (for example, recognition processing using a DNN) are described to enable execution.

[0092] In other words, the DSP 114 may perform recognition processing using the DNN according to the program stored in the program memory 115 .

[0093] The working memory 116 is configured using a RAM (Random Access Memory) and temporarily stores data related to the recognition processing performed by the DSP 114. More specifically, in the working memory 116, image data used in the recognition processing and metadata of image data output as a recognition result of the recognition processing are written by the DSP 114. In addition, after the recognition processing using the DSP 114, the image data written in the working memory 116 is deleted by the DSP 114.

[0094] The data transfer block 117 is configured using, for example, a DMA (Direct Memory Access) controller, and controls data transfer from the work memory 116 to the output I / F 118. More specifically, the data transmission block 117 transmits only metadata (recognition result) written in the work memory 116 to the output I / F 118.

[0095] The output I / F 118 has a similar structure as included in Figure 2 The external I / F 52 in the imaging device 1 shown in FIG. 1 is a function of the external I / F 52 and is configured as an output unit which is not directly connected to the pixel array 11 and which can output metadata (recognition result) to the outside.

[0096] Figure 5 is a diagram showing an example of access restriction of the working memory 116 of the data transfer block 117 .

[0097] like Figure 5 As shown in A of FIG. 1 , access restriction of accessing only the storage area in which metadata (recognition result) is written in the storage area of ​​the working memory 116 can be applied to the data transfer block 117 .

[0098] In addition, if Figure 5 As shown in B, a read restriction that does not allow data to be read from a storage area in which image data is written in the storage area of ​​the working memory 116 may be applied to the data transfer block 117 .

[0099] According to the access restriction and the read restriction, even in a case where the image data written in the working memory 116 is not deleted due to a specific reason, a configuration in which only metadata (recognition result) can be output to the outside can be realized more reliably.

[0100] (DSP Operation)

[0101] Will refer to Figure 6 The flowchart shown in FIG. 1 describes the operation of DSP 114. Figure 6 processing.

[0102] In step S111 , the DSP 114 acquires the image data processed by the ISP block 113 .

[0103] In step S112 , the DSP 114 writes the acquired image data into the working memory 116 .

[0104] In step S113 , the DSP 114 performs DNN processing (recognition processing using DNN) on the image data written in the working memory 116 .

[0105] In step S114 , the DSP 114 deletes the image data written in the working memory 116 .

[0106] In step S115 , the DSP 114 outputs metadata as a recognition result of the recognition process to the work memory 116 .

[0107] According to the above configuration and processing, although the image data is written into the working memory 116 once for the recognition processing, the image data is deleted after the recognition processing, and only the metadata (recognition result) is output to the outside. As a result, the image data is not output to the outside, and the risk of privacy invasion can be reduced.

[0108] Furthermore, since the program for executing the recognition process is configured not to be rewritable, a structure in which image data is not output to the outside can be reliably realized.

[0109] <3. Second Embodiment>

[0110] In this embodiment, a configuration in which it is checked that image data is not included in the output of the DSP as the recognition processing unit will be described.

[0111] (First Configuration Example)

[0112] Figure 7 is a block diagram showing a first configuration example of an imaging device according to the second embodiment.

[0113] Figure 7 The imaging device 200 shown includes a pixel array 211, a column ADC 212, an ISP module 213, a DSP 214, a program memory 215, a working memory 216, and an output I / F 217. These components have substantially the same functions as those included in the Figure 4 The functions of the pixel array 111, the column ADC 112, the ISP block 113, the DSP 114, the program memory 115, the working memory 116, and the output I / F 118 in the imaging device are shown, and therefore a description thereof will be omitted.

[0114] However, with Figure 4Unlike the DSP 114 shown in FIG. 2 , the DSP 214 directly outputs the metadata as the recognition result of the recognition processing to the output I / F 217 instead of writing the metadata into the working memory 216.

[0115] also, Figure 4 The data transmission block 117 shown in FIG. 1 may be provided in the imaging device 200. In this case, the DSP 214 may write metadata as a recognition result of the recognition process in the working memory 216, and the data transmission block 117 may transmit the metadata written in the working memory 216 to the output I / F 217.

[0116] In addition, Figure 4 Unlike the program memory 115 shown in FIG. 1 , the program memory 215 is configured using not a ROM but a RAM, and a program for executing the recognition process loaded from the outside is written in the program memory 215 .

[0117] The imaging device 200 further includes an input I / F 218 , a network check block 219 , and an overall control CPU 220 .

[0118] The input I / F 218 receives an input of a program (hereinafter, also referred to as a DSP program) provided by a user of the imaging device 200, in which an AI model or the like operates in the DSP 214. In addition, the DSP program is considered to be a program in which processing details that enable execution of signal processing using a neural network (e.g., recognition processing using a DNN) are described.

[0119] The network inspection block 219 is configured using a predetermined processor or is implemented using predetermined software, and has a function of an inspection processing unit that performs inspection processing that inspects that image data is not included in an output of the DSP 214 before recognition processing using the DSP 214 .

[0120] More specifically, the network check block 219 loads the DSP program from the outside through the input I / F 218 and writes the DSP program into the program memory 215. The network check block 219 performs the above-mentioned check processing for the DSP program written into the program memory 215. In more detail, the network check block 219 checks that the image data is not included in the output of the DSP 214 by checking that the network structure of the neural network (DNN) is a structure that is not capable of outputting the image data, and outputs its check result to the overall control CPU 220.

[0121] The overall control CPU 220 executes a program stored in a memory not shown in the drawings, thereby performing various processes starting from the control of the overall imaging device 200. For example, the overall control CPU 220 controls the execution of the recognition process using the DSP 214 based on the inspection result from the network inspection block 219. In addition, the overall control CPU 220 functions as an alarm generation unit that generates alarm information based on the inspection result from the network inspection block 219.

[0122] (Operation of the network check block)

[0123] First, refer to Figure 8 The flowchart shown in describes the operation (inspection processing) of the network inspection block 219.

[0124] In step S211 , the network check block 219 loads the DSP program from the outside through the input I / F 218 .

[0125] In step S212 , the network inspection block 219 writes the DSP program loaded from the outside into the program memory 215 .

[0126] In step S213 , the network inspection block 219 inspects (checks) whether a network structure in which image data is not included in the output of the DNN is formed for the DSP program written into the program memory 215 .

[0127] like Fig. 9 As shown, for example, in the case where the network structure of the DNN is a structure in which the image data is output as it is or a structure in which the original input data (image data) can be restored from the network structure and the output result, the network check block 219 determines that there is a problem (NG) in the network structure of the DNN. On the other hand, in the case where the network structure of the DNN is a structure in which the original input data (image data) cannot be restored from the network structure and the output result (such as a structure with an irreversible activation function, a structure with a fully coupled layer, etc.), the network check block 219 determines that there is no problem (OK) in the network structure of the DNN.

[0128] In addition, in the DSP program, after checking that the network structure is a network structure of a DNN in which image data is not output in advance, changes other than weights (parameters) can be configured to be unchangeable by configuring the network structure of the DNN as a ROM. In addition, by configuring the weights (parameters) as a ROM as well, only fixed processing can be performed.

[0129] In step S214 , the network inspection block 219 outputs an inspection result indicating whether there is a problem in the network structure of the DNN to the overall control CPU 220 .

[0130] (Operation of the overall imaging device)

[0131] Next, we will refer to Fig.10 The flowchart shown in FIG. 2 describes the operation of the overall imaging device 200. For example, Fig.10 The processing is started according to the received instruction for the imaging device 200 to start imaging, etc.

[0132] In step S221, the column ADC 212 reads raw data from the pixel array 211 and outputs the raw data to the ISP block 213. The ISP block 213 performs various image processing on the raw data from the column ADC 212.

[0133] In step S222 , the overall control CPU 220 acquires the inspection result from the network inspection block 219 .

[0134] In step S223, the overall control CPU 220 determines whether there is a problem in the network structure of the DNN based on the inspection result from the network inspection block 219. In other words, it is determined whether the user can acquire image data from the output of the DSP 214.

[0135] In the case where it is determined in step S223 that there is no problem with the network structure of the DNN, in other words, in the case where it is determined that the user cannot obtain image data from the output of the DSP 214, the process proceeds to step S224. In step S224, the overall control CPU 220 sends an enable signal (ON) for executing DNN processing (recognition processing using DNN) to the DSP 214. Accordingly, the DSP 214 starts to execute recognition processing for image data after performing image processing using the ISP block 213.

[0136] On the other hand, in the case where it is determined in step S223 that there is a problem with the network structure of the DNN, in other words, in the case where it is determined that the user can obtain image data from the output of the DSP 214, the process proceeds to step S225. In step S225, the overall control CPU 220 sends a DNN process execution stop enable signal (OFF) to the DSP 214. The DSP 214 stops the execution of the recognition process of the image data after the image processing using the ISP block 213.

[0137] Furthermore, in step S226, the overall control CPU 220 generates alarm information and supplies the alarm information to the output I / F 217, whereby the output I / F 217 outputs an alarm message.

[0138] According to the above configuration and processing, it is checked that image data is not included in the output of the DSP 214, and the recognition processing is stopped if image data is included in the output of the DSP 214. As a result, image data is not output to the outside, and the risk of privacy invasion can be reduced.

[0139] (Second configuration example)

[0140] Fig.11 is a block diagram showing a second configuration example of the imaging device according to the second embodiment.

[0141] exist Fig.11 In the imaging device 200A shown in FIG. 1 , the same reference symbols will be assigned to those in FIG. Figure 7 Components are similar to those included in the imaging device 200 shown in , and description thereof will be omitted.

[0142] In other words, Fig.11 The imaging device 200A shown in FIG. 1 includes an inspection dedicated processor 231, an inspection dedicated program memory 232, and an inspection dedicated work memory 233, instead of the network inspection block 219 and the overall control CPU 220, and is similar to the one in FIG. Figure 7 The imaging device 200 shown in FIG. 2 is different.

[0143] The inspection dedicated processor 231 is configured using, for example, a dedicated CPU, and performs inspection processing that the inspection image data is not included in the output of the DSP 214 before the recognition processing using the DSP 214. In addition to the function of the inspection processing unit that performs the inspection processing, the inspection dedicated processor 231 may further have a function similar to Figure 7 The functions of the overall control CPU 220 shown in FIG.

[0144] In the inspection-dedicated program memory 232, a program for causing the inspection-dedicated processor 231 to execute the inspection process is stored. The inspection-dedicated program memory 232 is configured using a ROM and makes it impossible to rewrite the program for executing the inspection process.

[0145] The inspection-dedicated working memory 233 is configured using a RAM, and a program (DSP program) for executing recognition processing loaded from the outside is written in the inspection-dedicated working memory 233 .

[0146] (Check the operation of the dedicated processor)

[0147] Will refer to Fig.12 The flowchart shown describes the operation of the inspection dedicated processor 231.

[0148] In step S231 , the check dedicated processor 231 loads the DSP program from the outside through the input I / F 218 .

[0149] In step S232 , the inspection-dedicated processor 231 writes the DSP program loaded from the outside into the inspection-dedicated working memory 233 .

[0150] In step S233, the inspection dedicated processor 231 checks (inspects) whether the DSP program written in the inspection dedicated working memory 233 forms a network structure in which image data is not included in the output of the DNN.

[0151] In step S234, the dedicated processor 231 is checked to determine whether there is a problem in the network structure of the DNN.

[0152] In a case where it is determined in step S234 that there is no problem with the network structure of the DNN, the process proceeds to step S235, and the inspection dedicated processor 231 sends the DSP program written in the inspection dedicated working memory 233 to the program memory 215.

[0153] In step S235, the inspection dedicated processor 231 sends a DNN processing (recognition processing using DNN) execution enable signal (ON) to the DSP 214. According to this, the DSP 214 starts executing recognition processing for image data after performing image processing using the ISP block 213.

[0154] On the other hand, in the case where it is determined in step S234 that there is a problem in the network structure of the DNN, the process proceeds to step S237 to check that the dedicated processor 231 sends a DNN processing execution stop enable signal (OFF) to the DSP 214. The DSP 214 stops performing recognition processing on the image data after performing image processing using the ISP block 213.

[0155] Furthermore, in step S238 , the inspection dedicated processor 231 generates alarm information and supplies the alarm information to the output I / F 217 , and thus the output I / F 217 outputs the alarm message.

[0156] According to the above configuration and processing, it is checked that image data is not included in the output of the DSP 214, and the recognition processing is stopped if image data is included in the output of the DSP 214. As a result, image data is not output to the outside, and the risk of privacy invasion can be reduced.

[0157] Furthermore, since the program for executing the inspection process is configured not to be rewritable, a structure in which image data is not output from the DSP 214 can be reliably realized.

[0158] (Third configuration example)

[0159] Fig.13 is a block diagram showing a third configuration example of the imaging device according to the second embodiment.

[0160] exist Fig.13 In the imaging device 200B shown in FIG. 1 , the same reference symbols will be assigned to those in FIG. Fig.11Components are similar to those included in the imaging device 200A shown in FIG. 1 , and description thereof will be omitted.

[0161] In other words, Fig.13 The imaging device 200B shown in FIG. 1 does not have Fig.11 The imaging apparatus 200A shown in FIG. 1 has a different inspection-dedicated working memory 233 .

[0162] (Check the operation of the dedicated processor)

[0163] Will refer to Fig.14 The flowchart shown in describes the operation of the inspection dedicated processor 231.

[0164] In step S251, the check dedicated processor 251 loads the DSP program from the outside through the input I / F 218.

[0165] In step S252 , the inspection dedicated processor 231 directly writes the DSP program loaded from the outside into the program memory 215 .

[0166] In step S253 , the inspection dedicated processor 231 inspects (checks) whether a network structure in which image data is not included in the output of the DNN is formed for the DSP program written in the program memory 215 .

[0167] In step S254, the dedicated processor 231 is checked to determine whether there is a problem in the network structure of the DNN.

[0168] In the case where it is determined in step S254 that there is no problem with the network structure of the DNN, the process proceeds to step S255 to check that the dedicated processor 231 sends a DNN processing (recognition processing using DNN) execution enable signal (ON) to the DSP 214. Accordingly, the DSP 214 starts to execute recognition processing for image data after performing image processing using the ISP block 213.

[0169] On the other hand, in the case where it is determined in step S254 that there is a problem in the network structure of the DNN, the process proceeds to step S256 to check whether the dedicated processor 231 sends a DNN processing execution stop enable signal (OFF) to the DSP 214. The DSP 214 stops performing recognition processing on the image data after performing image processing using the ISP block 213.

[0170] Furthermore, in step S257, the inspection dedicated processor 231 generates alarm information and supplies the alarm information to the output I / F 217, and thus, the output I / F 217 outputs the alarm message.

[0171] According to the above configuration and processing, it is checked that image data is not included in the output of the DSP 214, and the recognition processing is stopped if image data is included in the output of the DSP 214. As a result, image data is not output to the outside, and the risk of privacy invasion can be reduced.

[0172] Furthermore, since the program for executing the inspection process is configured not to be rewritable, a structure in which image data is not output from the DSP 214 can be reliably realized.

[0173] In addition, Fig.11 Unlike the configuration of the imaging device 200A shown in , the DSP program loaded from the outside is directly written into the program memory 215 , and thus the network structure of the DNN can be inspected without arranging the inspection-dedicated working memory 233 .

[0174] The imaging device according to the above-mentioned second embodiment can be configured to be combined with the imaging device according to the first embodiment. In other words, after checking that the image data is not included in the output of the DSP 214, the image data is deleted after the recognition processing, and only the metadata (recognition result) can be configured to be output to the outside.

[0175] <4. Application Examples>

[0176] Hereinafter, application examples of the technology according to the present disclosure will be described.

[0177] (Surveillance Camera)

[0178] The technology according to the present disclosure may be applied to a surveillance camera for watching the elderly or people who need care.

[0179] More specifically, if Fig.15 As shown, in a case where the monitoring target appears to be quietly staying in the image data IMG11, metadata such as "no problem" is output as a recognition result of the recognition processing on the image data IMG11. In addition, in a case where the monitoring target appears to be taking medicine in the image data IMG12, metadata such as "taking medicine" is output as a recognition result of the recognition processing on the image data IMG12. In addition, in a case where the monitoring target appears to fall in the image data IMG13, metadata such as "the person has fallen" is output as a recognition result of the recognition processing on the image data IMG13.

[0180] In a surveillance camera in which a general image sensor is installed, image data is output to the outside for some reason, and thus there is a risk that a person's face, daily activities, etc. may be peeped into.

[0181] In contrast, according to the technology related to the present disclosure, since image data is not output to the outside, it is possible to implement the function of a surveillance camera while protecting privacy without peeking at a person's face, daily activities, etc.

[0182] (Consumer Behavior Analysis)

[0183] The technology related to the present disclosure can be applied to consumer behavior analysis in commercial facilities such as shopping malls.

[0184] More specifically, if Fig.16 As shown, in a case where a customer appears to be shopping in image data IMG21 and image data IMG22, metadata representing the customer's "gender" and "age", "time spent in the store", "purchased items", etc. are output as a recognition result of recognition processing on image data IMG21 and IMG22.

[0185] In this way, according to the technology related to the present disclosure, since image data is not output to the outside, only information such as "gender", "age", etc. useful for consumer behavior analysis can be extracted in a form that does not constitute personal information.

[0186] In addition, in an imaging device to which the technology related to the present disclosure is applied, in addition to performing recognition processing as a signal processing of a DSP, other signal processing such as the above-mentioned fusion processing and self-position estimation processing can also be performed. In addition, in this case, a structure that does not output image data to the outside can be realized, thereby reducing the risk of privacy invasion.

[0187] The advantageous effects described here are merely exemplary and not limiting, and other advantageous effects may be obtained.

[0188] Furthermore, an embodiment to which the technology according to the present disclosure is applied is not limited to the above-described embodiment, and various changes may be made without departing from the gist of the technology according to the present disclosure.

[0189] Furthermore, the present disclosure may be configured as follows. (1)

[0191] An imaging device includes: an imaging unit in which a plurality of pixels are aligned in two dimensions, the imaging unit being configured to capture an image; a signal processing unit being configured to perform signal processing of image data based on an output of the imaging unit; and an output unit not directly connected to the imaging unit, the output unit being configured to be able to output only a signal processing result of the signal processing to the outside, wherein the signal processing unit deletes the image data after the signal processing. (2)

[0193] The imaging device according to (1) further includes a first storage unit configured to temporarily write data related to the signal processing, wherein the signal processing unit deletes the image data written in the first storage unit for the signal processing after the signal processing. (3)

[0195] The imaging device according to (2), wherein the imaging device further includes a second storage unit storing a program for executing signal processing, and the second storage unit sets the program to be non-rewritable. (4)

[0197] The imaging device according to (3), wherein the program enables the signal processing using the neural network to be executed. (5)

[0199] The imaging device according to any one of (1) to (4), wherein the signal processing unit performs recognition processing on the image data as the signal processing and outputs metadata of the image data as a recognition result of the recognition processing. (6)

[0201] The imaging device according to any one of (1) to (5), wherein the imaging section and the signal processing section are arranged inside one chip. (7)

[0203] The imaging device according to (6), wherein the chip includes a first substrate and a second substrate bonded to each other, wherein the first substrate includes the imaging device, and wherein the second substrate includes the signal processing unit. (8)

[0205] An imaging device includes: an imaging unit in which a plurality of pixels are two-dimensionally aligned, the imaging unit being configured to capture an image; a signal processing unit being configured to perform signal processing of image data based on an output of the imaging unit; an output unit being configured to be able to output a signal processing result of the signal processing to the outside; and an inspection processing unit being configured to perform inspection processing to inspect that image data is not included in the output of the signal processing unit before the signal processing. (9)

[0207] The imaging device according to (8), wherein, in a case where it is determined that the user cannot acquire the image data from the output of the signal processing unit, the signal processing unit performs the signal processing. (10)

[0209] The imaging device according to (9), wherein a first program for executing the signal processing is loaded from the outside, and the inspection processing section executes the inspection processing of the first program. (11)

[0211] An imaging device according to (10), wherein the first program enables the signal processing using the neural network to be executed, and wherein the inspection processing unit inspects whether the network structure of the neural network is a structure that is not capable of outputting the image data. (12)

[0213] The imaging device according to (11), wherein, in a case where the network structure is a structure in which the image data cannot be restored from the network structure and the output result, the inspection processing unit instructs the signal processing unit to perform signal processing. (13)

[0215] The imaging device according to any one of (8) to (12), wherein the imaging device further has a storage unit storing a second program for executing the inspection process, the storage unit setting the second program to be non-rewritable. (14)

[0217] The imaging device according to any one of (8) to (13), wherein, in a case where it is determined that the user can acquire the image data from the output of the signal processing unit, the signal processing unit stops execution of the signal processing. (15)

[0219] The imaging device according to (14), further comprising a warning generating unit, which generates warning information when it is determined that the output unit of the signal processing unit contains the image data. (16)

[0221] The imaging device according to any one of (8) to (15), wherein the output unit is not directly connected to the imaging unit. (17)

[0223] The imaging device according to (8) to (16), wherein the imaging unit, the signal processing unit, and the inspection processing unit are arranged inside one chip. (18)

[0225] The imaging device according to (17), wherein the chip includes a first substrate and a second substrate bonded to each other, wherein the first substrate includes the imaging device, and wherein the second substrate includes a signal processing unit and an inspection processing unit. (19)

[0227] A data output method includes deleting image data after signal processing using an imaging device, the imaging device including: an imaging unit in which a plurality of pixels are two-dimensionally aligned, the imaging unit being configured to capture an image; a signal processing unit being configured to perform signal processing on the image data based on an output of the imaging unit; and an output unit not directly connected to the imaging unit and being configured to be able to output only a signal processing result of the signal processing to the outside. (20)

[0229] The data output method described in (19), wherein a check process is performed before the signal processing to check that the image data is not included in the output of the signal processing unit.

[0230] [Reference Numbers List]

[0231] 1 Imaging device

[0232] 10 Imaging Block

[0233] 11 pixel array 12 columns ADC

[0234] 13 Control unit

[0235] 20 Signal processing blocks

[0236] 21ISP unit 22DSP

[0237] 51CPU

[0238] 52 external I / F

[0239] 100 Imaging Devices

[0240] 111 pixel array 112 columns ADC

[0241] 113ISP block 114DSP

[0242] 115 Program Memory

[0243] 116 Working memory 117 Data transfer block 118 Output I / F

[0244] 200, 200A, 200B imaging device 211 pixel array 212 column ADC

[0245] 213ISP block 214DSP

[0246] 215 Program memory 216 Working memory 217 Output I / F

[0247] 218 Input I / F

[0248] 219 Network Check Block

[0249] 220 overall control CPU

[0250] 231 Check dedicated processor

[0251] 232 Check dedicated program memory

[0252] 233 Check dedicated work memory

Claims

1. An imaging device, comprising: an imaging unit, wherein a plurality of pixels are two-dimensionally aligned and configured to capture an image; a signal processing unit configured to perform signal processing on image data based on an output of the imaging unit; and an output unit, not directly connected to the imaging unit, the output unit being configured to be able to output only the signal processing result of the signal processing to the outside, The signal processing unit deletes the image data after the signal processing.

2. The imaging device according to claim 1, further comprising: a first storage unit configured to temporarily write data related to the signal processing, The signal processing unit deletes the image data written in the first storage unit for the signal processing after the signal processing.

3. The imaging device according to claim 2, further comprising: a second storage unit configured to store a program for executing the signal processing, The second storage unit sets the program to be non-rewritable.

4. The imaging device according to claim 3, wherein: The program enables the signal processing using the neural network to be performed.

5. The imaging device according to claim 1, wherein: The signal processing unit performs recognition processing on the image data as the signal processing and outputs metadata of the image data as a recognition result of the recognition processing.

6. The imaging device according to claim 1, wherein: The imaging unit and the signal processing unit are arranged inside one chip.

7. The imaging device according to claim 6, in, The chip includes a first substrate and a second substrate bonded to each other, Wherein, the first substrate includes the imaging unit, and Wherein, the second substrate includes the signal processing unit.

8. An imaging device comprising: an imaging unit, wherein a plurality of pixels are two-dimensionally aligned and configured to capture an image; a signal processing unit configured to perform signal processing on image data based on an output of the imaging unit; an output unit configured to output a signal processing result of the signal processing to the outside; and A check processing unit is configured to perform a check process of checking that the image data is not included in the output of the signal processing unit before the signal processing.

9. The imaging device according to claim 8, wherein: The signal processing unit performs the signal processing in a case where it is determined that the user cannot acquire the image data from the output of the signal processing unit.

10. The imaging device according to claim 9, in, A first program for executing the signal processing is loaded from the outside, and wherein the inspection processing unit executes the inspection processing for the first program.

11. The imaging device according to claim 10, in, The first program enables the signal processing using a neural network to be performed, and The inspection processing unit inspects whether the network structure of the neural network is a structure that is not capable of outputting the image data.

12. The imaging device according to claim 11, wherein: In a case where the network structure is a structure in which the image data cannot be restored from the network structure and the output result, the inspection processing unit instructs the signal processing unit to perform the signal processing.

13. The imaging device according to claim 8, further comprising: a storage unit configured to store a second program for executing the inspection process, The storage unit sets the second program to be non-rewritable.

14. The imaging device according to claim 8, wherein: In a case where it is determined that the user can acquire the image data from the output of the signal processing unit, the signal processing unit stops execution of the signal processing.

15. The imaging device according to claim 14, further comprising: An alarm generating unit is configured to generate alarm information if it is determined that the image data is included in the output of the signal processing unit.

16. The imaging device according to claim 8, wherein: The output unit is not directly connected to the imaging unit.

17. The imaging device according to claim 8, wherein: The imaging unit, the signal processing unit, and the inspection processing unit are arranged inside one chip.

18. The imaging device according to claim 17, in, The chip includes a first substrate and a second substrate bonded to each other, Wherein, the first substrate includes the imaging unit, and Wherein, the second substrate includes the signal processing unit and the inspection processing unit.

19. A data output method comprising deleting image data after signal processing using an imaging device, the imaging device comprising: an imaging unit, wherein a plurality of pixels are two-dimensionally aligned and configured to capture an image; a signal processing unit configured to perform the signal processing on the image data based on an output of the imaging unit; and The output unit, which is not directly connected to the imaging unit, is configured to be able to output only the signal processing result of the signal processing to the outside.

20. The data output method according to claim 19, wherein: A check process of checking that the image data is not included in an output of the signal processing unit is performed before the signal processing.

Images