Imaging device, imaging system, and imaging device control method

By integrating statistics calculation and ΔΣ type neurons within the imaging device, the issue of performance degradation due to sparsity in SNNs is addressed, enabling efficient feature detection and image capture with improved sparsity management and reduced power consumption.

WO2024247481A1PCT designated stage expired Publication Date: 2024-12-05SONY SEMICON SOLUTIONS CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/013809
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-29
Filing Date
2024-04-03
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Imaging devices using neural networks face performance degradation when improving sparsity, particularly with spiking neural networks (SNNs), which affects feature detection and image capture efficiency.

Method used

Incorporating a statistics calculation unit within the imaging device to generate and analyze output signals from intermediate layers, controlling frame periods and neuron thresholds, and utilizing a ΔΣ type neuron structure to manage sparsity and maintain performance by reducing unnecessary information transmission.

Benefits of technology

This approach effectively suppresses performance deterioration while enhancing sparsity, allowing for efficient feature detection and image capture with reduced power consumption and improved processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024013809_05122024_PF_FP_ABST
    Figure JP2024013809_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention suppresses performance deterioration when sparsity is improved in an imaging device that uses a neural network. The imaging device includes a predetermined number of output-side neurons and a statistic calculation unit. In this imaging device, each of the output-side neurons generates an output signal each time a predetermined unit time elapses on the basis of an input signal from an intermediate layer. The statistic calculation unit calculates a statistic of an output signal generated within a predetermined frame period longer than the unit time, for each output-side neuron.
Need to check novelty before this filing date? Find Prior Art

Description

Image capture device, image capture system, and image capture device control method

[0001] The present technology relates to an imaging device, and more particularly to an imaging device with a neural network structure, an imaging system, and a control method for the imaging device.

[0002] Conventionally, imaging devices that dynamically switch modes have been used in various fields such as crime prevention and traffic. For example, an imaging device has been proposed that captures a low-resolution image in a feature detection mode to detect features of a subject, and when a specific feature is detected, switches to an imaging mode to capture a high-resolution image (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-150814

[0004] In the above-mentioned conventional technology, a low-resolution image is captured until a specific feature is detected, thereby reducing power consumption compared to when a high-resolution image is always captured. However, when a neural network such as an SNN (Spiking Neural Network) is used for feature detection, there is a problem that the performance deteriorates as the sparsity of neurons is improved.

[0005] This technology was developed in light of these circumstances, and aims to suppress performance degradation when improving sparsity in imaging devices that use neural networks.

[0006] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof is an imaging device including a simplified detector including a predetermined number of output neurons that generate output signals every time a predetermined unit time elapses based on input signals from a hidden layer, and a statistics calculation unit that calculates statistics of the output signals generated within a predetermined frame period that is longer than the unit time for each of the output neurons, and a control method thereof. This brings about the effect of suppressing performance degradation when sparsity is improved.

[0007] In addition, in this first aspect, the apparatus may further include a function calculation unit that inputs each of the statistical quantities into a predetermined function and outputs a calculated value obtained by the input of the statistical quantity into a predetermined function, thereby providing the effect that the calculated value is output as the detection result.

[0008] In addition, in the first aspect, the image forming apparatus may further include a control unit that controls the length of the frame period in accordance with a detection result of the simple detector, thereby providing an effect that an appropriate frame period is set.

[0009] In addition, in this first aspect, the present invention may further include a control unit that controls a threshold value of a neuron in the simple detector in accordance with a detection result of the simple detector, thereby providing an effect of controlling the firing rate of the neuron.

[0010] In addition, in this first aspect, an image sensor that generates image data may be further provided, and the simple detector may detect a predetermined detection target from the image data by analysis. This brings about an effect of detecting the detection target.

[0011] In addition, in this first aspect, the image processing device may further include a detailed detector that detects a predetermined detection target from the image data by a more detailed analysis than the simple detector, an evaluator that generates an evaluation value from the image data by an analysis different from that of the detailed detector, and a rule selection unit that selects a rule to control the operation of the detailed detector and the evaluator based on the detection result of the simple detector, thereby providing the effect of selecting a rule based on the detection result.

[0012] In addition, in this first aspect, the simple detector and the detailed detector may be arranged within a neural network circuit, thereby providing an effect of reducing circuits outside the neural network circuit.

[0013] In addition, in this first aspect, the simple detector, the detailed detector, and the evaluator may be arranged within a neural network circuit, thereby providing an effect of reducing circuits outside the neural network circuit.

[0014] In addition, in this first aspect, an input control unit may be further provided that controls the operation of the image sensor based on the detection result of the simple detector, thereby reducing the amount of processing by the simple detector.

[0015] In addition, in this first aspect, the system may further include an environmental sensor that outputs a sensor signal different from the image data, and the rule selection unit may select the rule based on the sensor signal and a detection result of the simple detector. This brings about an effect of improving robustness.

[0016] In addition, in this first aspect, the simple detector may be disposed within an image sensor that generates image data, thereby providing an effect of reducing circuits outside the image sensor.

[0017] In this first aspect, the neurons in the simple detector may be spiking neurons, which has the effect of making it possible to make information sparsifiable in both time and space.

[0018] According to a second aspect of the present technology, there is provided an imaging device in which the following are arranged in at least one of a plurality of neurons in a neural network: a difference calculator that calculates the difference between an added value of a predetermined number of input signals and a feedback signal; an integrator that integrates the difference and outputs the integrated value; a limiting unit that outputs the integrated value as an output signal when the integrated value is outside a predetermined range; a comparator that compares the output signal with a predetermined value and supplies a comparison result; and a feedback unit that converts the comparison result into the feedback signal and feeds it back. This brings about an effect of suppressing performance degradation when sparsity is improved.

[0019] A third aspect of the present technology is an imaging system including a simple detector including a predetermined number of output neurons that generate output signals every time a predetermined unit time elapses based on input signals from a hidden layer, and a statistics calculation unit that calculates statistics of the output signals generated within a predetermined frame period that is longer than the unit time for each of the output neurons, and an analysis device that analyzes an image based on the detection results of the simple detector. This has the effect of suppressing performance degradation of the system when sparsity is improved.

[0020] 1 is a block diagram showing an example configuration of an imaging device according to a first embodiment of the present technology. FIG. 2 is a block diagram showing an example configuration of an imaging device in a case where a plurality of imaging units are arranged in the imaging device according to the first embodiment of the present technology. FIG. 3 is a block diagram showing an example configuration of an SNN circuit and a router according to the first embodiment of the present technology. FIG. 4 is a block diagram showing an example configuration of a core according to the first embodiment of the present technology. FIG. 5 is a diagram showing an example neural network structure of a simple detector according to the first embodiment of the present technology. FIG. 6 is a block diagram showing an example configuration of a ΔΣ neuron according to the first embodiment of the present technology. FIG. 7 is a diagram showing an example input signal and an output signal of a ΔΣ neuron according to the first embodiment of the present technology. FIG. 8 is a block diagram showing an example configuration of a ΔΣ neuron according to a first comparative example. FIG. 9 is a graph showing an example of an output signal and a waveform of a comparison result when a sine wave is input in the first comparative example and the first embodiment of the present technology. FIG. 10 is a graph showing an example of an output signal and a waveform of a comparison result when a sparse signal is input in the first comparative example and the first embodiment of the present technology. FIG. 11 is a graph showing an example of an output signal and a waveform of a comparison result when a stationary signal is input in the first comparative example and the first embodiment of the present technology. FIG. 1 is a block diagram showing an example configuration of a simple detector in a first embodiment of the present technology and a second comparative example. FIG. 2 is a diagram showing an example output value of an output-side neuron in the first embodiment of the present technology. FIG. 3 is a diagram showing an example output format in a second comparative example. FIG. 4 is a diagram showing an example output format in the first embodiment of the present technology. FIG. 5 is a flowchart showing an example operation of an analysis device in the first embodiment of the present technology. FIG. 6 is a block diagram showing an example configuration of an imaging unit in a first modified example of the first embodiment of the present technology. FIG. 7 is a block diagram showing an example configuration of an imaging device in a second embodiment of the present technology. FIG. 8 is a block diagram showing an example configuration of an imaging device in a third embodiment of the present technology. FIG. 9 is a flowchart showing an example operation of an analysis device in the third embodiment of the present technology. FIG. 10 is a block diagram showing an example configuration of an imaging device in a fourth embodiment of the present technology. FIG. 11 is a block diagram showing an example configuration of an imaging device in a fifth embodiment of the present technology.Fig. 10 is a diagram showing an example of a stacked structure of an image sensor according to a fifth embodiment of the present technology; Fig. 11 is a block diagram showing an example of a configuration of an imaging device according to a sixth embodiment of the present technology; Fig. 12 is a block diagram showing an example of a schematic configuration of a vehicle control system; Fig. 13 is an explanatory diagram showing an example of an installation position of an imaging unit;

[0021] Hereinafter, modes for implementing the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. First embodiment (an example in which a statistical quantity within a frame period is calculated) 2. Second embodiment (an example in which a simple detector, a detailed detector, and an evaluator that calculate a statistical quantity within a frame period are provided in an SNN circuit) 3. Third embodiment (an example in which a statistical quantity within a frame period is calculated and an input control unit controls an image sensor) 4. Fourth embodiment (an example in which a simple detector and a detailed detector that calculate a statistical quantity within a frame period are provided in an SNN circuit) 5. Fifth embodiment (an example in which a simple detector and a detailed detector that calculate a statistical quantity within a frame period are provided in an image sensor) 6. Sixth embodiment (an example in which a rule is selected based on the detection result of a simple detector that calculates a statistical quantity within a frame period and a sensor signal) 7. Application example to a moving body

[0022] 1 is a block diagram showing an example of the configuration of an imaging device 100 according to an embodiment of the present technology. The imaging device 100 is a device that captures image data, and includes an imaging unit 110 and an analysis device 130. The imaging unit 110 includes an image sensor 120 and a simple detector 200.

[0023] The image sensor 120 generates image data through photoelectric conversion. For example, the image sensor 120 determines whether the amount of light has changed for each pixel address and generates image data by arranging the determination results. Such a sensor is called an event-based vision sensor (EVS) or a dynamic vision sensor (DVS). By using an EVS, the input to the simple detector 200 becomes sparse, and even if the input undergoes a significant periodic change, the input can be further sparsed within the simple detector 200.

[0024] The image sensor 120 is not limited to an EVS, and can generate a grayscale signal for each pixel according to the amount of light and generate image data by arranging these grayscale signals. When generating a grayscale signal, the image sensor 120 generates an analog signal for each pixel and converts it into a digital signal using an AD (Analog to Digital) converter. Alternatively, the image sensor 120 includes a SPAD (Single-Photon Avalanche Diode) for each pixel and generates a digital signal by counting pulse signals from the SPAD.

[0025] Furthermore, the image sensor 120 switches the output destination of the image data between the simple detector 200 and the analysis device 130 based on the detection result of the simple detector 200 .

[0026] The image sensor 120 can also change the resolution of the image data generated based on the detection results of the simple detector 200. The resolution is changed by pixel addition, thinning readout, etc. In this case, for example, the image sensor 120 lowers the resolution of the image data sent to the simple detector 200 and increases the resolution of the image data sent to the analysis device 130.

[0027] The simple detector 200 detects a predetermined detection target from image data through analysis. The function of the simple detector 200 is implemented in hardware, for example, in an SNN circuit 500. The SNN circuit 500 is an example of a neural network circuit as defined in the claims.

[0028] In the simple detector 200, for example, a person or a specific preparatory movement is set as a detection target. The simple detector 200 detects the presence or absence of a detection target (such as a person) and supplies the detection result to the analysis device 130 and the image sensor 120. When the detection result is a predetermined value P, the image sensor 120 changes the output destination of the image data to the analysis device 130. Here, P is not limited to one, and can be any number P 0 and P 1 There may be two or more. 0 and P 1A detection object is assigned to each of the values. If the detection result is a predetermined value P, it indicates that the detection object corresponding to that value has been detected.

[0029] The analysis device 130 includes an input control unit 131 , a rule selection unit 132 , a detail detector 133 , and an evaluator 134 .

[0030] The input control unit 131 transmits and receives data between the outside of the analysis device 130 and the rule selection unit 132, the detail detector 133, and the evaluator 134. The input control unit 131 supplies the detection results from the simple detector 200 to the rule selection unit 132. The input control unit 131 also supplies image data from the image sensor 120 to at least one of the detail detector 133 and the evaluator 134. The input control unit 131 also supplies the detection results from the detail detector 133 to the rule selection unit 132, and transmits logs and alerts from the detail detector 133 and the evaluator 134 to an external server or the like.

[0031] The rule selection unit 132 selects a rule for controlling the operation of the detailed detector 133 and the evaluator 134 based on the detection result of the simple detector 200. For example, various rules are assumed, such as "activate the evaluator 134 when a specific detection target is detected by the detailed detector 133" or "output an alert to the outside when a predetermined evaluation value is obtained by the evaluator 134." The rule selection unit 132 controls the operation of the detailed detector 133 and the evaluator 134 according to the selected rule.

[0032] For example, in a certain rule, the rule selection unit 132 stops the operation of the detailed detector 133 and the evaluator 134 when the detection result of the simple detector 200 is not the predetermined value P. On the other hand, when the detection result of the simple detector 200 is the predetermined value P, the rule selection unit 132 causes the detailed detector 133 to start analysis while keeping the evaluator 134 stopped. Then, when the detection result of the detailed detector 133 is the predetermined value Q, the rule selection unit 132 causes the evaluator 134 to start analysis. Here, Q is not limited to one, and may be any number of Q 0 YaQ 1 If the detection result is a predetermined value Q, it indicates that the detection object corresponding to that value has been detected.

[0033] The detail detector 133 detects a predetermined detection target from image data by performing a more detailed analysis than the simple detector 200. For example, the presence or absence of a person, the number of people, the presence or absence of a specific motion, the location of a person in the image, etc. are set as detection targets. Furthermore, when a person is detected, the detail detector 133 can also detect whether or not the person is a specific individual. The detail detector 133 may perform these analyses on a single piece of image data, or on a video including multiple pieces of image data captured continuously.

[0034] The detail detector 133 outputs the detection result to the input control unit 131. Furthermore, when the detail detector 133 detects a specific individual, it can generate a log including the time of photographing and output the log to the input control unit 131.

[0035] The evaluator 134 generates an evaluation value from image data by an analysis different from that of the detail detector 133. For example, the evaluator 134 analyzes what actions one or more people are performing and what state they are in, and generates an evaluation value that indicates the results. The evaluator 134 may perform this analysis on a single piece of image data or on a moving image.

[0036] For example, in a certain rule, when the evaluation value is a predetermined value R, the evaluator 134 outputs a log including the shooting time and an alert to the input control unit 131. Here, R is not limited to one, and R 0 or R 1 The evaluation value being a predetermined value P indicates that an analysis object corresponding to that value has been detected.

[0037] As described above, a system in which the simple detector 200 continuously detects a detection target and operates the detailed detector 133 and the evaluator 134 when the detection target is detected is called an Always-ON system. The Always-ON system can reduce power consumption while maintaining performance compared to when the detailed detector 133 and the evaluator 134 are always operated.

[0038] In the figure, the imaging unit 110 and the analysis device 130 are arranged in one device, but this configuration is not limiting. They may also be distributed and arranged in multiple devices. Furthermore, a system in which the imaging unit 110 and the analysis device 130 are arranged is an example of an imaging system described in the claims.

[0039] In addition, although one image capturing unit 110 is arranged in the figure, the configuration is not limited to this. As shown in the example of Fig. 2, a plurality of image capturing units such as image capturing units 110-1 and 110-2 may be arranged.

[0040] As shown in FIG. 3, the simple detector 200 and the analysis device 130 may be arranged in one processor 600 .

[0041] 4 is a diagram showing an implementation example of an SNN circuit 500 according to the first embodiment of the present technology. The SNN circuit 500 includes, for example, an input / output interface 560 and a multi-core array 570.

[0042] The input / output interface 560 transmits and receives data between the outside and the multi-core array 570. The input / output interface 560 supplies image data from the image sensor 120 to the multi-core array 570, and supplies the detection results from the multi-core array 570 to the analysis device 130.

[0043] The multi-core array 570 has multiple cores 590 arranged in a two-dimensional lattice pattern. A router 580 is arranged adjacent to each core 590.

[0044] The router 580 controls the data path. This router 580 includes, for example, FIFO (First In, First Out) memories 581 to 585 and an arbiter 586. In the figure, "E" indicates the east direction of the router 580 in question, "S" indicates the south direction, "W" indicates the west direction, and "N" indicates the north direction. "L" indicates the direction toward the core 590 adjacent to the router 580.

[0045] The FIFO memory 581 holds data from the east direction in a first-in, first-out manner and outputs a request to the arbiter 586. The FIFO memory 582 holds data from the south direction in a first-in, first-out manner and outputs a request to the arbiter 586. The FIFO memory 583 holds data from the west direction in a first-in, first-out manner and outputs a request to the arbiter 586. The FIFO memory 584 holds data from the north direction in a first-in, first-out manner and outputs a request to the arbiter 586. The FIFO memory 585 holds data from the adjacent core 590 in a first-in, first-out manner and outputs a request to the arbiter 586.

[0046] The arbiter 586 arbitrates requests from the FIFO memories 581 to 585 and returns a response. When a response is received, the FIFO memory outputs data to one of the cores 590 adjacent to the east, west, north, or south via the arbiter 586.

[0047] 5 is a block diagram showing an example configuration of a core 590 according to the first embodiment of the present technology. The core 590 includes a core router 591, a neuron I / O (Input / Output) 592, a product-sum unit 593, a work memory 594, a membrane potential memory 595, and an LIF (Leaky Integrate and Fire) unit 596.

[0048] Core router 591 provides data from adjacent routers 580 to neuron I / O 592 and provides data from LIF unit 596 to adjacent routers 580 .

[0049] The multiply-accumulate unit 593 integrates data from the neuron I / O 592 using a work memory 594. The membrane potential memory 595 stores the membrane potential obtained by the integration. The LIF unit 596 determines whether the membrane potential exceeds a predetermined threshold and fires (in other words, whether a spike occurs), and supplies the result to the core router 591.

[0050] 6 is a diagram showing an example of a neural network structure of the simple detector 200 according to the first embodiment of the present technology. In the figure, "a" shows an example of the neural network structure, and "b" shows an example of a two-layer structure of the structure shown in "a". The circles in "b" in the figure represent neurons.

[0051] As shown in FIG. 1A, the neural network in the simple detector 200 is, for example, a convolutional neural network (CNN), which is composed of multiple layers such as convolutional layers 211 and 212 and a pooling layer 213. Each neuron in the first convolutional layer 211 receives as input pixel signals one or more pixels in the image data 700. Note that the neural network is not limited to a CNN.

[0052] As shown in b in the figure, for example, a predetermined number of ΔΣ neurons 300 are arranged in the convolution layer 212. In the figure, for simplicity, the channel direction is omitted and the representation is two-dimensional, but the same applies to three-dimensional representation.

[0053] As shown in FIG. 1B, a neural network is configured with a plurality of neurons, including one or more ΔΣ neurons 300. Spiking neurons in an SNN, for example, are used as these neurons. It is also possible to use general neurons in an ANN (Artificial Neural Network). Spiking neurons and general neurons can also be combined.

[0054] 7 is a block diagram showing an example of the configuration of a ΔΣ neuron 300 according to the first embodiment of the present technology. The ΔΣ neuron 300 includes a difference calculator 310, an integrator 320, a limiting unit 330, a comparator 340, and a feedback unit 350.

[0055] The difference calculator 310 calculates the difference B between the sum of a predetermined number of input signals and the feedback signal E from the feedback unit 350, and supplies the result to the integrator 320. For example, if the output of the previous neuron is 0, in 1 and in 2 and the weight is ω 0 , ω 1 and ω 2 As, in 0 ×ω 0 , in 1 ×ω 1 and in 2 ×ω 2 is input as an input signal.

[0056] The integrator 320 integrates the difference B and supplies the integral value C to the limiting section 330 .

[0057] If the integral value C is outside a predetermined limit range, the limiting unit 330 outputs the value as an output signal OUT to the subsequent pooling layer 213 and the comparator 340. For example, if the upper limit threshold of the limit range is set to Th 0 and the lower threshold is Th 1 The limiting unit 330 determines whether the integral value C is greater than or equal to Th 0 Above and Th 1 If the integral value C is less than or equal to Th, an output signal OUT of "0" is output. The output signal OUT being "0" means that no significant output signal is output and is limited. 0 or the integral value C is less than Th 1 If it is greater than the limiting unit 330, the limiting unit 330 outputs the integral value C as it is as the output signal OUT.

[0058] The comparator 340 compares the output signal OUT with a predetermined comparison threshold TH and supplies a comparison result D to the feedback section 350. For example, TH is set to "0", and the comparator 340 outputs "1" as the comparison result when the output signal OUT is greater than TH, and outputs "0" as the comparison result when the output signal OUT is equal to or less than TH.

[0059] The feedback section 350 converts the comparison result D into a feedback signal E and feeds it back to the difference calculator 310. For example, when the comparison result D is greater than "0", the feedback section 350 outputs a feedback signal E of "-1", and when the comparison result D is "0", the feedback section 350 outputs a feedback signal E of "1".

[0060] The above-mentioned ΔΣ neuron 300 calculates and outputs an output signal OUT at regular intervals in synchronization with a clock signal of a predetermined processing frequency. In the above configuration, the time derivative of the integral value C is expressed, for example, by the following equation: dC / dt=-(C-a) / T+Σ(ωj×inj)-Δ(out) ...Equation 1 In the above equation, t represents time, a represents a coefficient, T represents a time constant, and Δ() represents the comparison result of the comparator 340.

[0061] The value of the output signal OUT is expressed by the following equation: OUT=f(C) Equation 2 In the above equation, f( ) represents the activation function of the limiting section 330 .

[0062] Moreover, both the input signal and the output signal of the ΔΣ neuron 300 are digital signals. Note that the input signal may be an analog signal and the output signal may be a digital signal. In this case, a 1-bit DAC (Digital to Analog Converter) or the like is used as the feedback unit 350. Also, both the input signal and the output signal may be analog signals.

[0063] 8 is a diagram illustrating an example of an input signal and an output signal of the ΔΣ neuron 300 according to the first embodiment of the present technology. In the figure, a shows an example of the trajectory of the input signal over time, and b shows an example of the trajectory of the output signal.

[0064] As shown in a and b in the figure, an input signal is input for each unit time, and an output signal is calculated. These input and output signals are, for example, n-bit (n is an integer) spike signals. Such neurons are called spiking neurons. Note that the ΔΣ neuron 300 is not limited to a spiking neuron, and may be a general neuron whose input and output signals are not spike signals.

[0065] Here, a general ΔΣ neuron without the limiting unit 330 is assumed as a first comparative example.

[0066] 9 is a block diagram showing an example of the configuration of a ΔΣ neuron in a first comparative example. As shown in the figure, the first comparative example does not include a limiting unit 330, and the integral value is output as an output signal OUT and input to a comparator 340. In this first comparative example, the firing rate of the neuron is controlled by a comparison threshold TH of the comparator 340. Increasing the comparison threshold TH reduces the firing rate, thereby improving sparsity.

[0067] Here, in the above-mentioned Always ON system, by making the data, including the input from the image sensor 120, sparse as much as possible in both space and time, the Always ON system can be operated more efficiently.

[0068] Furthermore, while an ANN can only sparsify processing in the time direction by changing the processing frequency, an SNN can sparsify information in both time and space by preventing each neuron from transmitting unnecessary information. Unlike an ANN, an SNN can sparsify information spatially, so it is possible to achieve sparsity comparable to that of a slow ANN while maintaining the processing frequency.

[0069] Therefore, by using an SNN as the simple detector 200, it is possible to maintain the processing frequency while improving the processing efficiency.

[0070] However, there is a trade-off between sparsity and performance. In the first comparative example, increasing the comparison threshold TH of each neuron reduces the firing rate of the neuron, improving sparsity, but the amount of transmitted information becomes insufficient, resulting in increased errors in the analysis results.

[0071] In this first comparative example, the only threshold to be controlled is the comparison threshold TH, whereas in the ΔΣ neuron 300 provided with the limiting unit 330, two types of thresholds can be controlled: the comparison threshold TH and the thresholds of the limiting unit 330 (upper and lower thresholds). In this configuration, the sparsity can be improved by widening the limiting range from the lower threshold to the upper threshold. On the other hand, information loss can be minimized by lowering the comparison threshold TH. Therefore, by outputting the integral value C through the limiting unit 330 rather than directly, information loss when sparsity is increased can be minimized. This makes it possible to suppress performance degradation when sparsity is increased.

[0072] 10 is a graph showing an example of waveforms of an output signal and a comparison result when a sine wave is input to a neuron of the first comparative example and the first embodiment of the present technology. In the figure, "a" shows an example of waveforms of an input signal and an output signal when a sine wave is input to the neuron of the first comparative example, and "b" shows an example of waveforms of an input signal and a comparison result in that case.

[0073] Also, c in the same figure is an example of the waveforms of the input signal and output signal when a sine wave is input to a ΔΣ neuron 300 with a limiting section 330, and d in the same figure is an example of the waveforms of the input signal and the comparison result in that case.

[0074] The dotted lines a to d in the figure indicate the waveform of the input signal. The solid line a in the figure indicates the waveform of the output signal (i.e., the integral value), and the solid line b in the figure indicates the waveform of the comparison result. The solid line c in the figure indicates the waveform of the output signal from the limiting unit 330, and the solid line d in the figure indicates the waveform of the comparison result. The same applies to Figures 11 and 12.

[0075] 10A to 10D, in the configuration provided with the limiting unit 330, the compression rate is higher for information with periodicity such as a sine wave, and information loss is reduced compared to the first comparative example. Note that information loss increases for inputs that fluctuate in a complex manner.

[0076] 11 is a graph showing an example of waveforms of an output signal and a comparison result when a sparse signal is input in the first comparative example and the first embodiment of the present technology. In the figure, "a" shows an example of waveforms of an input signal and an output signal when a sparse signal is input to a neuron in the first comparative example, and "b" shows an example of waveforms of an input signal and a comparison result in that case.

[0077] Also, c in the same figure is an example of the waveforms of the input signal and output signal when a sparse signal is input to the ΔΣ neuron 300 with the limiting unit 330, and d in the same figure is an example of the waveforms of the input signal and the comparison result in that case.

[0078] As shown in the examples a to d in the figure, in the configuration provided with the limiting unit 330, the compression rate when a sparse signal is input is almost the same as in the first comparative example.

[0079] 12 is a graph showing an example of waveforms of an output signal and a comparison result when a steady signal is input in the first comparative example and the first embodiment of the present technology. In the figure, "a" shows an example of waveforms of an input signal and an output signal when a steady signal is input to the neuron of the first comparative example, and "b" shows an example of waveforms of an input signal and a comparison result in that case.

[0080] Also, c in the same figure is an example of the waveforms of the input signal and output signal when a steady signal is input to a ΔΣ neuron 300 with a limiting section 330, and d in the same figure is an example of the waveforms of the input signal and the comparison result in that case.

[0081] As shown in the examples a to d in the figure, in the configuration provided with the limiting section 330, the compression ratio when a stationary signal is input is almost the same as that of the first comparative example.

[0082] Next, the output format of the output layer in the neural network of the simple detector 200 will be described.

[0083] 13 is a block diagram showing an example of the configuration of the simple detector 200 in the first embodiment of the present technology and the second comparative example. In the drawing, "a" is a block diagram showing an example of the configuration of the simple detector 200 in the first embodiment.

[0084] The neural network in the simple detector 200 includes an input layer 210, an intermediate layer 220, and an output layer 230. The input layer 210 processes image data from the image sensor 120 and supplies an array of spike signals to the intermediate layer 220. The intermediate layer 220 processes the array of spike signals and supplies the array of spike signals to the output layer 230. The output layer 230 includes a predetermined number of output neurons 231, a statistical calculation unit 232, and a function calculation unit 233.

[0085] Each output neuron 231 generates an output signal every time a unit time elapses based on an input signal from the hidden layer, and supplies the output signal to the statistics calculation unit 232. If the number of output neurons 231 is M (M is an integer), an array of M output signals is output every unit time. The length of the unit time is, for example, 1 millisecond.

[0086] The statistical quantity calculation unit 232 calculates the statistical quantity of the output signal generated within a predetermined frame period longer than the unit time for each output neuron 231. The statistical quantities calculated include the maximum value, sum, and average value. The statistical quantity calculation unit 232 supplies the array of M statistical quantities as Logit to the function calculation unit 233.

[0087] The function calculation unit 233 inputs Logit into a predetermined function (such as a softmax function) and outputs the solution value (in other words, the calculated value) obtained as the detection result to the analysis device 130.

[0088] Here, a configuration in which the statistical quantity calculation unit 232 is not provided is assumed as a second comparative example. In the figure, "b" is a block diagram showing an example of the configuration of the output layer 230 in the second comparative example. In the second comparative example, the array of output signals from the output neuron 231 is input to the function calculation unit 233 as Logit.

[0089] In the figure, the function calculation unit 233 outputs the solution as the detection result, but this configuration is not limiting. As will be described later, the simple detector 200 can also output an array of statistics (Logit) to the analysis device 130 without calculating the solution.

[0090] 14 is a diagram showing an example of output values ​​of the output neuron 231 in the first embodiment of the present technology. A in the figure shows an example of output values ​​for each output neuron 231 at a certain time t. For example, eight output neurons 231 are arranged, and their respective identification numbers are "0" to "7." The circles in the figure indicate the output neurons 231. The output values ​​of these neurons are, for example, n-bit spike values.

[0091] 1B shows the output values ​​of each output neuron 231 in chronological order. As shown in FIG. 1B, each of the eight output neurons 231 outputs an output value (such as a spike value) per unit time. The numbers in parentheses in FIG. 1B indicate the identification numbers of the output neurons 231.

[0092] 15 is a diagram showing an example of an output format in the second comparative example. 9 At time t, an array of eight output values ​​(i.e., Logit) is input to the function calculation unit 233. The function calculation unit 233 outputs, as a solution, the identification number of the output neuron 231 corresponding to the largest output value among Logit. 10 Thereafter, the same process is repeatedly executed for each unit time.

[0093] In the second comparative example, if the sparsity of spike signals in an SNN is improved in the time direction, the amount of zero information increases when the function calculation unit 233 calculates a solution, which may prevent an appropriate solution from being obtained.

[0094] For example, at time t 9 From t 20 Assuming that "7" is the correct answer within the period up to time t 13 At time t, the function calculation unit 233 erroneously outputs "0". 17 , t 18 Ya, t 20 Even in this case, the output value of the output neuron 231 with the identification number "7" does not become maximum, and an incorrect solution is output.

[0095] To obtain an appropriate solution, the output layer 230 needs to accumulate non-zero information for a sufficiently long time, but this reduces the output frequency of the simple detector 200 and increases the delay. This is also undesirable because it delays or makes the evaluation by the downstream analysis device 130 inaccurate.

[0096] 16 is a diagram showing an example of an output format in the first embodiment of the present technology. As illustrated in the figure, in the first embodiment, a frame period longer than the unit time (e.g., 4× the unit time) is set. The statistics calculation unit 232 calculates statistics (e.g., maximum value) of the output signal generated within the frame period for each output neuron 231. For example, at time t 9 From t 12 During the frame period including t, the output neuron 231 with the identification number "7" outputs four output values ​​in sequence. If the maximum value of these is "8", the statistical quantity calculation unit 232 obtains "8" as the statistical quantity of that neuron. The statistical quantity calculation unit 232 also obtains statistical quantities for the other neurons, and supplies the array of the eight statistical quantities as Logit to the function calculation unit 233. At the next time t 13 From t 16 The same process is performed for the frame period including the Logit. The function calculation unit 233 outputs the identification number of the output neuron 231 corresponding to the largest output value of the Logit as a solution.

[0097] In the second comparative example, at time t 13 As shown in the figure, the output layer 230 calculates statistics to output an incorrect solution. 13 In the frame period including time t, the correct answer "7" can be output. 17 , t 18 , t 20 Similarly, the output layer 230 can output the correct solution.

[0098] As shown in the figure, by calculating statistics, it is possible to suppress the output of erroneous solutions when sparsity is improved in the time axis direction. Furthermore, since there is no need to accumulate non-zero information for a long period of time, it is possible to suppress a decrease in output frequency and an increase in delay compared to the second comparative example. This makes it possible to suppress a decrease in performance when sparsity is improved.

[0099] 17 is a flowchart showing an example of the operation of the analysis device 130 according to the first embodiment of the present technology. This operation is started, for example, when the simple detector 200 outputs a detection result.

[0100] The rule selection unit 132 determines whether the detection result of the simple detector 200 is a predetermined value P (in other words, whether a predetermined detection target has been detected) (step S901). If the detection result is not P (step S901: No), the rule selection unit 132 returns to step S901.

[0101] On the other hand, if the detection result is P (step S901: Yes), the rule selection unit 132 selects a rule based on the detection result, and, for example, activates the detail detector 133 based on the rule (step S902).

[0102] The rule selection unit 132 determines whether the detection result of the detail detector is a predetermined value Q (step S903). If the detection result is not Q (step S903: No), the rule selection unit 132 repeats step S901 and subsequent steps.

[0103] On the other hand, if the detection result is Q (step S903: Yes), the rule selection unit 132, for example, starts the evaluator 134 and determines whether the evaluation value is a predetermined value R (step S904). If the evaluation value is not R (step S904: No), the rule selection unit 132 repeats step S901 and subsequent steps.

[0104] On the other hand, if the evaluation value is R (step S904: Yes), the evaluator 134 records a log and outputs an alert (step S905), and the analysis device 130 ends the operation for analysis.

[0105] The rules shown in steps S903 to S905 above are examples of rules selected in step S902. The content of the rules is not limited to the examples shown in the figure. For example, the rule selection unit 132 may activate only one of the detail detector 133 and the evaluator 134 when the detection result is P. Alternatively, the evaluator 134 may only output an alert in step S905.

[0106] As described above, according to the first embodiment of the present technology, the statistical calculation unit 232 calculates the statistical quantities of the output signals generated within a frame period for each output neuron 231, thereby suppressing performance degradation due to erroneous solutions when sparsity is improved.

[0107] Furthermore, since a limiting unit 330 is provided within the ΔΣ neuron 300, which outputs the integral value as an output signal when the integral value is outside a predetermined range, it is possible to minimize information loss when improving sparsity.

[0108] In the first embodiment described above, the statistical quantity calculation unit 232 calculates the statistical quantities within a frame period, but the frame period can also be made variable. The imaging device 100 of this first embodiment differs from the first embodiment in that the length of the frame period is controlled in accordance with the output of the simple detector 200.

[0109] 18 is a block diagram showing an example configuration of the image capture unit 110 in a first modified example of the first embodiment of the present technology. The image capture unit 110 in this first modified example of the first embodiment differs from the first embodiment in that it further includes a control unit 510. The control unit 510 is provided in the SNN circuit 500, for example.

[0110] The control unit 510 controls the length of the frame period in accordance with the detection result of the simple detector 200 (i.e., the solution of the function calculation unit 233). For example, the control unit 510 shortens the frame period to be shorter than a predetermined value when the solution is smaller than a predetermined set value, and lengthens the frame period to be longer than the predetermined value when the solution is equal to or greater than the set value.

[0111] The simple detector 200 can also output Logit. In this case, for example, the control unit 510 finds a solution from Logit and controls the frame period in accordance with the solution.

[0112] By making the frame period (in other words, the frame rate) variable, an effect of essentially thinning out the output of the simple detector 200 can be obtained during time periods when the output of the simple detector 200 is sparse.

[0113] In addition, if the output frequency of the simple detector 200 is high, the control unit 510 can determine that important information is being received and increase the operating frequency of the analysis device 130, thereby making the entire system more efficient so that it can perform the necessary operations.

[0114] The control unit 510 can also control the threshold of the neuron in the simple detector 200 according to the detection result (solution or Logit) of the simple detector 200. For example, the control unit 510 increases the threshold in a time period when the solution is greater than a set value, and decreases the threshold in a time period when the solution is equal to or less than the set value, thereby reducing the output of the simple detector 200. In the case of the aforementioned ΔΣ neuron 300, the comparison threshold and the upper and lower thresholds of the limit range are each controlled.

[0115] Although the control of the frame period and the like is performed by the control unit 510 in the SNN circuit 500, the present invention is not limited to this configuration. The control of the frame period and the like can also be performed by an external device to the SNN circuit 500, for example, the input control unit 131.

[0116] In this way, according to the modified example of the first embodiment of the present technology, the control unit 510 controls the length of the frame period according to the detection result of the simple detector 200, and therefore, an appropriate frame rate can be set according to the detection result.

[0117] 2. Second Embodiment In the first embodiment described above, the simple detector 200 is provided within the SNN circuit 500, but it is also possible to provide a detailed detector 133 and an evaluator 134. The imaging device 100 in this second embodiment differs from the first embodiment in that the simple detector 200, the detailed detector 133, and the evaluator 134 are provided within the SNN circuit 500.

[0118] 19 is a block diagram showing an example configuration of an imaging device 100 according to the second embodiment of the present technology. The imaging device 100 according to the second embodiment includes an image sensor 120, a control CPU (Central Processing Unit) 140, and an SNN circuit 500.

[0119] The multi-core array 570 of the SNN circuit 500 is divided into multiple regions, and the functions of the simple detector 200, the detailed detector 133, and the evaluator 134 are assigned to these regions. Note that the router and input / output interface are omitted from the figure.

[0120] The control CPU 140 has the functions of the input control unit 131 and the rule selection unit 132 of the first embodiment. The control CPU 140 can also control setting values ​​related to the output of the image sensor 120 based on the output state of at least one of the simple detector 200, the detailed detector 133, and the evaluator 134. For example, setting values ​​related to the range of pixels to be output, pixel gain, and other characteristics are controlled.

[0121] As illustrated in the figure, by providing the simple detector 200, the detailed detector 133, and the evaluator 134 within the SNN circuit 500, there is no need to place the detailed detector 133 and the evaluator 134 outside the SNN circuit 500.

[0122] It should be noted that the modified example of the first embodiment can also be applied to the second embodiment.

[0123] Thus, according to the second embodiment of the present technology, the simple detector 200, the detailed detector 133, and the evaluator 134 are provided within the SNN circuit 500, so there is no need to place the detailed detector 133 and the evaluator 134 outside the SNN circuit 500.

[0124] 3. Third Embodiment In the first embodiment described above, the simple detector 200 calculates a solution from Logit and outputs it as a detection result, but it is also possible to output Logit as a detection result without calculating a solution. The imaging device 100 in this third embodiment differs from the first embodiment in that the simple detector 200 outputs Logit.

[0125] 20 is a block diagram showing an example configuration of an imaging device 100 according to a third embodiment of the present technology. The simple detector 200 according to the third embodiment differs from the first embodiment in that it does not calculate a solution but outputs Logit as a detection result to the input control unit 131.

[0126] Furthermore, the input control unit 131 calculates a solution from Logit, controls the image sensor 120 in accordance with the solution, and changes the output destination.

[0127] 21 is a flowchart showing an example of the operation of the analysis device 130 according to the third embodiment of the present technology. The operation according to the third embodiment differs from the first embodiment in that steps S910, S911, and S912 are executed instead of step S901.

[0128] The input control unit 131 calculates a solution from Logit (step S910) and determines whether the solution is a predetermined value P (step S911). If the solution is not P (step S911: No), the input control unit 131 returns to step S910.

[0129] On the other hand, if the solution is P (step S911: Yes), the input control unit 131 controls the output destination of the image sensor 120 to output image data to the detail detector 133 and the evaluator 134 in the analysis device 130 (step S912). After step S912, steps S902 and onward are executed.

[0130] As described above, the input control unit 131 calculates the solution from Logit, so that the simple detector 200 does not need to calculate the solution, and the amount of processing by the simple detector 200 can be reduced.

[0131] The second embodiment can be applied to the third embodiment.

[0132] As described above, according to the third embodiment of the present technology, the input control unit 131 calculates the solution from Logit, so that the simple detector 200 does not need to calculate the solution.

[0133] 4. Fourth Embodiment In the first embodiment described above, the simple detector 200 is provided in the SNN circuit 500, but it is also possible to provide a detailed detector 133. The imaging device 100 in this fourth embodiment differs from the first embodiment in that the simple detector 200 and the detailed detector 133 are provided in the SNN circuit 500.

[0134] 22 is a block diagram showing an example configuration of an imaging device 100 according to a fourth embodiment of the present technology. The imaging device 100 according to the fourth embodiment differs from the first embodiment in that a detail detector 133 is further provided in the SNN circuit 500.

[0135] As shown in the figure, by providing the simple detector 200 and the detailed detector 133 within the SNN circuit 500, it is no longer necessary to place the detailed detector 133 outside the SNN circuit 500.

[0136] It should be noted that the modified example of the first embodiment or the third embodiment can also be applied to the fourth embodiment.

[0137] As described above, according to the fourth embodiment of the present technology, since the simple detector 200 and the detailed detector 133 are provided within the SNN circuit 500, there is no need to place the detailed detector 133 outside the SNN circuit 500.

[0138] 5. Fifth Embodiment In the above-described first embodiment, the simple detector 200 is disposed outside the image sensor 120, but the simple detector 200 can also be disposed inside the image sensor 120. The imaging device 100 in this fifth embodiment differs from the first embodiment in that the simple detector 200 is disposed inside the image sensor 120.

[0139] 23 is a block diagram showing an example configuration of an imaging device 100 according to a fifth embodiment of the present technology. The imaging device 100 according to the fifth embodiment differs from the first embodiment in that a simple detector 200 is disposed in the image sensor 120.

[0140] The image sensor 120 includes, for example, a pixel array unit 121, a memory 122, a simple detector 200, and a controller 123. The simple detector 200 is implemented in an SNN circuit 500.

[0141] A plurality of pixels, each of which generates a pixel signal, are arranged in the pixel array section 121. The memory 122 stores various data such as image data. The controller 123 controls the circuits in the image sensor 120.

[0142] By providing the simple detector 200 inside the image sensor 120 , it is not necessary to dispose the simple detector 200 outside the image sensor 120 .

[0143] 24 is a diagram showing an example of a stacked structure of an image sensor 120 according to the fifth embodiment of the present technology. The image sensor 120 includes a logic chip 126 and a pixel chip 125 stacked on the logic chip 126. These chips are electrically connected by, for example, Cu-Cu bonding. Note that, in addition to Cu-Cu bonding, they can also be connected by vias or bumps.

[0144] The pixel array unit 121 is disposed on the pixel chip 125. The remaining circuits such as the simple detector 200 are disposed on the logic chip 126.

[0145] Although the circuits in image sensor 120 are arranged on pixel chip 125 and logic chip 126, these circuits may also be arranged on a single semiconductor chip.

[0146] Furthermore, the fifth embodiment can be applied to a modified example of the first embodiment, or to each of the second, third, and fourth embodiments.

[0147] As described above, according to the fifth embodiment of the present technology, since the simple detector 200 is provided inside the image sensor 120 , it is not necessary to dispose the simple detector 200 outside the image sensor 120 .

[0148] 6. Sixth Embodiment In the first embodiment described above, the rule selection unit 132 selects a rule based on the detection result, but with this configuration, there is a risk of selecting an incorrect rule due to erroneous detection by the simple detector 200. The imaging device 100 in this sixth embodiment differs from the first embodiment in that it selects a rule based on the sensor signal from the environmental sensor and the detection result.

[0149] 25 is a block diagram showing an example configuration of an imaging device 100 according to a sixth embodiment of the present technology. The imaging device 100 according to the sixth embodiment differs from the first embodiment in that it includes one or more environmental sensors 150.

[0150] The environmental sensor 150 generates a sensor signal different from the image data. For example, a temperature sensor, a humidity sensor, an illuminance sensor, a pressure sensor, an acoustic sensor, or a vibration sensor can be used as the environmental sensor 150. In addition, a proximity sensor, a contact sensor, or an inertial sensor attached to the measurement object or the periphery of the measurement object can also be used as the environmental sensor 150. The sensor signal of the environmental sensor 150 is supplied to the rule selection unit 132 via the input control unit 131.

[0151] The rule selection unit 132 selects a rule based on the sensor signal and detection result from the environmental sensor 150. For example, even if the simple detector 200 fails to detect a person approaching, the rule selection unit 132 can detect the person from sensor signals from an acoustic sensor, a vibration sensor, or the like and select an appropriate rule. This prevents the selection of an incorrect rule when the simple detector 200 makes a false detection or misses a detection, thereby making the system more robust.

[0152] Furthermore, the sixth embodiment can be applied to a modified example of the first embodiment, or to each of the second, third, fourth, and fifth embodiments.

[0153] As described above, according to the sixth embodiment of the present technology, the rule selection unit 132 selects a rule based on the sensor signal from the environmental sensor 150 and the detection result, thereby making the system more robust.

[0154] 7. Application Examples to Mobile Bodies The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0155] FIG. 26 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0156] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 26, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053.

[0157] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0158] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0159] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0160] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0161] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0162] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0163] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0164] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0165] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 26, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0166] FIG. 27 is a diagram showing an example of the installation position of the imaging unit 12031.

[0167] In FIG. 27, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0168] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0169] 27 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0170] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0171] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which runs autonomously without relying on driver operation.

[0172] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0173] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0174] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 of the above-described configuration. Specifically, the imaging device 100 of FIG. 1 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to suppress degradation in system performance when sparsity is improved.

[0175] Note that the above-described embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specifying the invention in the claims correspond to each other. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present technology having the same name correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist thereof.

[0176] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0177] The present technology may also be configured as follows: (1) An imaging device including a simple detector including a predetermined number of output-side neurons that generate output signals every time a predetermined unit time elapses based on input signals from a hidden layer, and a statistical calculation unit that calculates, for each of the output-side neurons, statistics of the output signals generated within a predetermined frame period that is longer than the unit time. (2) The imaging device according to (1), further including a function calculation unit that inputs each of the statistical quantities to a predetermined function and outputs a calculated value obtained. (3) The imaging device according to (3), further including a control unit that controls the length of the frame period in accordance with a detection result of the simple detector. (4) The imaging device according to (3), further including a control unit that controls a threshold value of a neuron in the simple detector in accordance with a detection result of the simple detector. (5) The imaging device according to any of (1) to (4), further including an image sensor that generates image data, wherein the simple detector detects a predetermined detection target from the image data by analysis. (6) The imaging device according to (5), further comprising: a detailed detector that detects a predetermined detection target from the image data by a more detailed analysis than the simple detector; an evaluator that generates an evaluation value from the image data by an analysis different from that of the detailed detector; and a rule selection unit that selects a rule for controlling the operation of the detailed detector and the evaluator based on the detection result of the simple detector. (7) The imaging device according to (6), wherein the simple detector and the detailed detector are arranged in a neural network circuit. (8) The imaging device according to (6), wherein the simple detector, the detailed detector, and the evaluator are arranged in a neural network circuit. (9) The imaging device according to any of (6) to (8), further comprising an input control unit that controls the operation of the image sensor based on the detection result of the simple detector. (10) The imaging device according to any of (6) to (9), further comprising: an environmental sensor that outputs a sensor signal different from the image data, and the rule selection unit selects the rule based on the sensor signal and the detection result of the simple detector. (11) The imaging device according to any one of (1) to (10), wherein the simple detector is disposed within an image sensor that generates image data.(12) The imaging device according to any one of (1) to (11), wherein the neuron in the simple detector is a spiking neuron. (13) An imaging device, in which the following are arranged in at least one of a plurality of neurons in a neural network: a difference calculator that calculates a difference between an added value of a predetermined number of input signals and a feedback signal, an integrator that integrates the difference and outputs the integrated value, a limiting unit that outputs the integrated value as an output signal if the integrated value is outside a predetermined range, a comparator that compares the output signal with a predetermined value and supplies the comparison result, and a feedback unit that converts the comparison result into the feedback signal and feeds it back. (14) An imaging system, comprising: a simple detector including a predetermined number of output-side neurons that generate output signals every time a predetermined unit time has elapsed based on input signals from a hidden layer, and a statistics calculator that calculates, for each output-side neuron, statistics of the output signals generated within a predetermined frame period that is longer than the unit time; and an analysis device that analyzes an image based on the detection results of the simple detector. (15) A control method for an imaging device, comprising the steps of: a predetermined number of output neurons generating output signals each time a unit time elapses based on input signals from an intermediate layer; and a step of calculating, for each output neuron, statistics of the output signals generated within a predetermined frame period longer than the unit time.

[0178] 100 Imaging device 110, 110-1, 110-2 Imaging unit 120 Image sensor 121 Pixel array unit 122 Memory 123 Controller 125 Pixel chip 126 Logic chip 130 Analysis device 131 Input control unit 132 Rule selection unit 133 Detailed detector 134 Evaluator 140 Control CPU 150 Environmental sensor 200 Simple detector 210 Input layer 211, 212 Convolution layer 213 Pooling layer 220 Hidden layer 230 Output layer 231 Output side neuron 232 Statistical quantity calculation unit 233 Function calculation unit 300 ΔΣ neuron 310 Difference calculation unit 320 Integrator 330 Limitation unit 340 Comparator 350 Feedback unit 500 SNN circuit 510 Control unit 560 Input / output interface 570 Multi-core array 580 Router 581 to 585 FIFO memory 586 Arbiter 590 Core 591 Core router 592 Neuron I / O 593 Product-accumulator unit 594 Work memory 595 Membrane potential memory 596 LIF unit 600 Processor 12031 Imaging unit

Claims

1. An imaging device equipped with a simplified detector including: a predetermined number of output neurons that generate an output signal every time a predetermined unit time elapses based on an input signal from an intermediate layer; and a statistical calculation unit that calculates, for each output neuron, the statistics of the output signals generated within a predetermined frame period that is longer than the unit time.

2. The imaging apparatus according to claim 1, further comprising a function calculation section which inputs each of said statistical quantities into a predetermined function and outputs a calculated value.

3. The imaging device according to claim 3, further comprising a control unit that controls the length of the frame period in accordance with a detection result of the simple detector.

4. The imaging device according to claim 3, further comprising a control unit that controls a threshold value of a neuron in the simple detector in accordance with a detection result of the simple detector.

5. The imaging device according to claim 1, further comprising an image sensor that generates image data, wherein the simple detector detects a predetermined detection target from the image data by analysis.

6. An imaging device as described in claim 5, further comprising: a detailed detector that detects a predetermined detection target from the image data by a more detailed analysis than that of the simple detector; an evaluator that generates an evaluation value from the image data by an analysis different from that of the detailed detector; and a rule selection unit that selects a rule for controlling the operation of the detailed detector and the evaluator based on the detection result of the simple detector.

7. The imaging device according to claim 6, wherein the simple detector and the detailed detector are arranged in a neural network circuit.

8. The imaging device according to claim 6, wherein the simple detector, the detailed detector and the evaluator are arranged in a neural network circuit.

9. The imaging device according to claim 6, further comprising an input control unit for controlling the operation of said image sensor based on the detection result of said simple detector.

10. The imaging device according to claim 6, further comprising an environmental sensor that outputs a sensor signal different from the image data, wherein the rule selection section selects the rule based on the sensor signal and the detection result of the simple detector.

11. The imaging device of claim 1, wherein the simplified detector is disposed within an image sensor that generates image data.

12. The imaging device according to claim 1, wherein the neurons in the simplified detector are spiking neurons.

13. An imaging device in which the following are arranged in at least one of multiple neurons in a neural network: a difference calculator that calculates the difference between the sum of a predetermined number of input signals and a feedback signal; an integrator that integrates said difference and outputs an integrated value; a limiting section that outputs said integrated value as an output signal if said integrated value is outside a predetermined range; a comparator that compares said output signal with a predetermined value and supplies a comparison result; and a feedback section that converts said comparison result into the feedback signal and feeds it back.

14. An imaging system comprising: a simple detector including a predetermined number of output neurons that generate an output signal every time a predetermined unit time elapses based on an input signal from an intermediate layer, and a statistical calculation unit that calculates, for each output neuron, the statistics of the output signals generated within a predetermined frame period that is longer than the unit time; and an analysis device that analyzes an image based on the detection results of the simple detector.

15. A method for controlling an imaging device comprising the steps of: a predetermined number of output neurons generating output signals each time a unit time elapses based on an input signal from an intermediate layer; and a step of calculating, for each output neuron, statistics of the output signals generated within a predetermined frame period that is longer than the unit time.

Citation Information

Patent Citations

  • Digital signal processing device and method

    JP1997307452A

  • Image processor and computer-readable storage medium thereof

    JP1999252450A

  • Method and apparatus for data refresh of analog non-volatile memory in deep learning neural networks

    JP2021517704A

  • Wheel deceleration-based antiskid brake controller with adaptive deceleration threshold

    US20030220731A1

  • Coping with feature error suppression: a mechanism to handle the concept drift

    US20220188410A1