Imaging device, imaging system, imaging method, and computer program

Abstract

Provided is an imaging device that performs a recognition process on a captured image and calculates a determination basis for the recognition process.　The imaging device comprises: an imaging unit having a pixel region in which a plurality of pixels are arrayed; a readout-unit control unit that controls a readout unit set as a part of the pixel region; a readout control unit that controls the readout of a pixel signal from the pixels included in the pixel region using the readout unit set by the readout-unit control unit; a recognition unit having a machine learning model learned on the basis of learning data; and a determination basis calculation unit that calculates the determination basis for the recognition process of the recognition unit. The recognition unit performs the recognition process for each readout unit, and the determination basis calculation unit calculates the determination basis.

Description

[Technical Field]

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an imaging device, an imaging system, an imaging method, and a computer program that are equipped with an image recognition function for captured images. [Background technology]

[0002] Currently, research into machine learning systems using deep learning is actively underway. For example, by applying deep learning to the image field, it is possible to achieve face detection and object recognition that surpass human capabilities. Furthermore, in response to the issue of the process by which machine learning systems arrive at their recognition results being treated as a black box, research is also being conducted to present the basis for machine learning systems' decisions (see, for example, Non-Patent Document 1).

[0003] For example, in image recognition processing, an analysis program has been proposed that uses the Grad-CAM method to generate a map showing the degree of attention given to each image part of the incorrect inference image that was focused on during inference, while modifying the incorrect inference image, which is the input image when an incorrect label is inferred, so as to maximize the score of the correct inference label, to generate a refined image (see Patent Document 1).

[0004] When applying image recognition technology to autonomous driving, it is necessary to present the driver with the basis for their decision in real time. However, there are limits to how quickly the basis for a decision can be calculated from video, and as the image quality of cameras increases, the processing load increases, making it increasingly difficult to present the basis for a decision in real time. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-197875 [Non-patent literature]

[0006] [Non-Patent Document 1] David Gunning, "Explainable Artificial Intelligence (XAI)," [online] DARPA, [Retrieved November 9, 2020], Internet (URL: https: / / www.darpa.mil / program / explainable-artificial-intelligence) [Non-patent document 2] R.Selvaraju et al,"Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization" [Non-patent document 3] Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization<https: / / arxiv.org / abs / 1610.02391> [Non-patent document 4] "Why Should I Trust You?": Explaining the Predictions of Any Classifier<https: / / arxiv.org / abs / 1602.04938> [Non-Patent Document 5] Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)<https: / / arxiv.org / pdf / 1711.11279.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] The object of the present disclosure is to provide an imaging device, an imaging system, an imaging method, and a computer program that perform recognition processing on captured images using a trained machine learning model and have the function of calculating the basis for judgment of the recognition processing. [Means for solving the problem]

[0008] The present disclosure has been made in consideration of the above problems, and a first aspect thereof is: an imaging unit having a pixel area in which a plurality of pixels are arranged; a read unit control unit that controls a read unit set as a part of the pixel area; a read control unit that controls reading of pixel signals from pixels included in the pixel area in read units set by the read unit control unit; a recognition unit having a machine learning model trained based on training data; a determination basis calculation unit that calculates a determination basis for the recognition process in the recognition unit; Equipped with the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. It is an imaging device.

[0009] The recognition unit learns learning data for each readout unit using a neural network model, and the judgment basis calculation unit infers a portion of the pixel area for each readout unit that affects each class in relation to the inference result of class classification in the neural network model.

[0010] The recognition unit performs machine learning processing using an RNN on pixel data of a plurality of read units in the same frame image, and performs the recognition processing based on the results of the machine learning processing.

[0011] Furthermore, a second aspect of the present disclosure is an imaging device including: an imaging unit having a pixel region in which a plurality of pixels are arranged; a read unit control unit that controls a read unit set as a part of the pixel region; and a read control unit that controls reading of pixel signals from pixels included in the pixel region in the read unit set by the read unit control unit; an information processing device including a recognition unit having a machine learning model trained based on training data, and a judgment basis calculation unit that calculates judgment basis for recognition processing in the recognition unit; Equipped with the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. An imaging system.

[0012] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize specific functions), and it does not matter whether each device or functional module is contained within a single housing.

[0013] Furthermore, a third aspect of the present disclosure is Each is executed by a processor, a readout unit control step of controlling a readout unit set as a part of a pixel area in which a plurality of pixels of the imaging unit are arranged; a read control step of controlling readout of pixel signals from pixels included in the pixel area in readout units set in the readout unit control step; a recognition step based on a machine learning model trained on training data; a determination basis calculation step of calculating a determination basis for the recognition processing in the recognition step; and In the recognition step, a recognition process is performed on pixel signals for each read unit, and in the determination basis calculation step, a determination basis for a result of the recognition process for each read unit is calculated. It is an imaging method.

[0014] Furthermore, a fourth aspect of the present disclosure is a read unit control unit that controls a read unit set as a part of a pixel region in which a plurality of pixels included in the imaging unit are arranged; a readout control unit that controls the reading of pixel signals from pixels included in the pixel area in readout units set by the readout unit control unit; a recognition unit having a machine learning model trained based on training data; a determination basis calculation unit that calculates a determination basis for the recognition process in the recognition step; Make the computer function as the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. It is a computer program written in a computer-readable format so as to

[0015] A computer program according to the fourth aspect of the present disclosure defines a computer program written in a computer-readable format to perform predetermined processing on a computer. In other words, by installing the computer program according to the fourth aspect of the present disclosure on a computer, a cooperative action is exerted on the computer, and the same effects as those of the imaging device according to the first aspect of the present disclosure can be obtained. [Effects of the Invention]

[0016] According to the present disclosure, it is possible to provide an imaging device, an imaging system, an imaging method, and a computer program that can quickly realize recognition processing of captured images using a trained machine learning model and calculation of the basis for that judgment.

[0017] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0018] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and accompanying drawings. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram showing an example of the functional configuration of an imaging device 100. As shown in FIG. [Figure 2] FIG. 2 is a diagram showing an example of hardware implementation of the image capture device 100. As shown in FIG. [Figure 3] FIG. 3 is a diagram showing another example of hardware implementation of the image capture device 100. In FIG. [Figure 4] FIG. 4 is a diagram showing a stacked image sensor 400 having a two-layer structure. [Figure 5] FIG. 5 is a diagram showing a stacked image sensor 500 having a three-layer structure. [Figure 6] FIG. 6 is a diagram showing an example of the configuration of the sensor unit 102. [Figure 7] Figure 7 shows the mechanism of image recognition processing using CNN. [Figure 8] FIG. 8 is a diagram showing the mechanism of image recognition processing for obtaining a recognition result from a part of an image to be recognized. [Figure 9] FIG. 9 is a diagram showing an example of discrimination processing (recognition processing) by DNN when time-series information is not used. [Figure 10] FIG. 10 is a diagram showing an example of discrimination processing (recognition processing) by DNN when time-series information is not used. [Figure 11] FIG. 11 is a diagram showing a first example of a classification process by a DNN when time-series information is used. [Figure 12] FIG. 12 is a diagram showing a first example of a classification process by a DNN when time-series information is used. [Figure 13] FIG. 13 is a diagram showing a second example of the classification process by the DNN when time-series information is used. [Figure 14] FIG. 14 is a second diagram illustrating the classification process by the DNN when time-series information is used. [Figure 15] FIG. 15 shows an example of the configuration of a DNN using a multi-layer convolutional neural network. [Figure 16] FIG. 16 is a diagram showing an example of a configuration for calculating the decision basis of a DNN that performs discrimination processing on time-series information. [Figure 17] FIG. 17 is a diagram for explaining an overview of the present disclosure. [Figure 18] FIG. 18 is a flowchart showing the processing procedure for executing each process of image recognition and calculation of the basis for judgment by the recognition processing unit 104. [Figure 19] FIG. 19 is a diagram showing an example of image data for one frame. [Figure 20] FIG. 20 is a diagram showing the flow of the recognition process executed on the image data shown in FIG. [Figure 21] FIG. 21 is a diagram showing an example of a functional configuration in the vicinity of the sensor control unit 103, the recognition processing unit 104, and the image processing unit 106. [Figure 22] FIG. 22 is a diagram showing an example of processing in the recognition processing unit 104. [Figure 23] FIG. 23 is a diagram for explaining the function of the recognition processing unit 104. As shown in FIG. [Figure 24] FIG. 24 is a diagram showing a frame read process (first embodiment). [Figure 25] FIG. 25 is a diagram showing an example of performing recognition processing on an image frame line by line. [Figure 26] FIG. 26 shows an example in which a valid recognition result is obtained during frame readout on a line-by-line basis, and the recognition process is terminated. [Figure 27] FIG. 27 shows an example in which a valid recognition result is obtained during frame readout on a line-by-line basis, and the recognition process is terminated. [Figure 28] FIG. 28 is a flowchart showing the procedure for the recognition and determination basis calculation process corresponding to the reading of pixel data in units of readout from a frame. [Figure 29]FIG. 29 is a diagram showing an example of a time chart of the control of the readout and recognition processing and the determination basis calculation (when a blank period blk is provided). [Figure 30] FIG. 30 is a diagram showing an example of a time chart of the control of the readout and recognition processing and the determination basis calculation (when a blank period blk is provided). [Figure 31] FIG. 31 is a diagram showing an example of a time chart of the control of the readout and recognition processing and the determination basis calculation (when the blank period blk is not provided). [Figure 32] FIG. 32 is a diagram showing a frame read process (first modified example). [Figure 33] FIG. 33 is a diagram showing a frame read process (second modified example). [Figure 34] FIG. 34 is a diagram showing a frame read process (third modified example). [Figure 35] FIG. 35 is a diagram showing an example of performing recognition processing on an image frame in area units of a predetermined size. [Figure 36] FIG. 36 shows an example in which a valid recognition result is obtained during frame readout in area units and the recognition process is terminated. [Figure 37] FIG. 37 shows an example in which a valid recognition result is obtained during frame readout in area units and the recognition process is terminated. [Figure 38] FIG. 38 is a diagram showing a frame read process (fourth modified example). [Figure 39] FIG. 39 is a diagram showing an example of performing recognition processing on an image frame in units of a predetermined sample area. [Figure 40] FIG. 40 is a diagram showing an example of a time chart (fourth modified example) of the control of the readout and recognition processing and the determination basis calculation. [Figure 41] FIG. 41 shows an example in which a valid recognition result is obtained in the middle of frame readout in units of a predetermined sample area, and the recognition process is terminated. [Figure 42]FIG. 42 shows an example in which a valid recognition result is obtained in the middle of frame readout in units of a predetermined sample area, and the recognition process is terminated. [Figure 43] FIG. 43 is a diagram showing a frame read process (fifth modified example). [Figure 44] FIG. 44 is a diagram showing a frame read process (sixth modified example). [Figure 45] FIG. 45 is a diagram showing an example of a pattern for performing frame reading and recognition processing (another example of the sixth modified example). [Figure 46] FIG. 46 is a diagram showing a frame read process (a first example of the seventh modified example). [Figure 47] FIG. 47 is a diagram showing a frame read process (a second example of the seventh modified example). [Figure 48] FIG. 48 is a diagram showing a frame read process (a third example of the seventh modified example). [Figure 49] FIG. 49 is a diagram showing a frame read process (fourth example of the seventh modified example). [Figure 50] FIG. 50 is a diagram for explaining the function of the recognition processing unit 104. [Figure 51] FIG. 51 is a flowchart showing the procedure for the recognition and determination basis calculation process corresponding to the reading of pixel data based on feature amounts. [Figure 52] FIG. 52 is a diagram illustrating a first processing procedure according to the eighth modified example. [Figure 53] FIG. 53 is a diagram illustrating a second processing procedure according to the eighth modified example. [Figure 54] FIG. 54 is a diagram showing an example of processing by the recognition processing unit 104 according to the eighth modified example. [Figure 55] FIG. 55 is a diagram for explaining the function of the recognition processing unit 104. [Figure 56] FIG. 56 is a diagram showing an example of the functional configuration of the read determination unit 2114 according to the second embodiment. [Figure 57]FIG. 57 is a diagram showing an example of a read unit pattern. [Figure 58] FIG. 58 is a diagram showing an example of a read order pattern. [Figure 59] FIG. 59 shows an example of a read order pattern. [Figure 60] FIG. 60 shows an example of a read order pattern. [Figure 61] FIG. 61 is a diagram for explaining a method for setting a readout area based on recognition information. [Figure 62] FIG. 62 is a diagram for explaining a method for setting a readout area based on recognition information. [Figure 63] FIG. 63 is a diagram for explaining a method for setting a readout area based on recognition information. [Figure 64] FIG. 64 is a diagram for explaining a method for setting a readout area based on recognition information. [Figure 65] FIG. 65 is a diagram for explaining a method for setting a readout area based on recognition information. [Figure 66] FIG. 66 is a diagram for explaining a method for setting a readout area based on recognition information. [Figure 67] FIG. 67 is a diagram showing the application fields of the present disclosure. [Figure 68] FIG. 68 is a diagram showing a schematic configuration example of a vehicle control system 6800. [Figure 69] FIG. 69 is a diagram showing an example of the installation position of the imaging unit 6830. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present disclosure will be described below in the following order with reference to the drawings.

[0021] A. Machine Learning Overview B. Configuration of the imaging device C. Overview of DNN D. Summary of the Disclosure E. Examples of the Disclosure F. Second Example G. Application Areas H. Application Examples

[0022] A. Machine Learning Overview In the field of images, well-known technologies that enable deep learning machine learning systems to visualize the basis for their recognition processing decisions include Grad-CAM (Gradient-weighted Class Activation Mapping), LIME (LOCAL Interpretable model-agnostic Explanations), and SHAP (SHapley Additive exPlanations), an advanced version of LIME.

[0023] However, currently, only still images can be displayed, making it difficult to display the basis for a decision quickly for video. For example, when deep learning is applied to image recognition from an in-vehicle camera in autonomous driving, the basis for that decision must be processed quickly and presented to the driver. However, there are limits to how quickly the basis for a decision can be calculated for video, and the increasing image quality of cameras increases the processing load, making it increasingly difficult to present the basis for a decision in real time.

[0024] This disclosure envisions that a small imaging device such as a digital camera is equipped with an image recognition function and a function for presenting the basis for judgment in image recognition. This disclosure aims to speed up the recognition process and realize real-time presentation of the basis for judgment by using a part of the pixel area of the imaging unit as a readout unit and performing image recognition processing and calculation processing of the basis for judgment for each readout unit.

[0025] B. Configuration of the imaging device The present disclosure can be applied to various devices that use machine learning models. FIG. 1 shows an example of the functional configuration of an imaging device 100 to which the present disclosure can be applied. The illustrated imaging device 100 includes an optical unit 101, a sensor unit 102, a sensor control unit 103, a recognition processing unit 104, a memory 105, an image processing unit 106, an output control unit 107, and a display unit 108. For example, a CMOS (Complementary Metal Oxide Semiconductor) can be used to integrate the sensor unit 102, the sensor control unit 103, the recognition processing unit 104, and the memory 105 into a CMOS image sensor. However, the imaging device 100 may be an infrared light sensor that captures images using infrared light, or another type of light sensor.

[0026] The optical unit 101 includes, for example, a plurality of optical lenses for focusing light from the subject onto the light-receiving surface of the sensor unit 102, an aperture mechanism for adjusting the size of an aperture for incident light, and a focus mechanism for adjusting the focus of light irradiated onto the light-receiving surface. The optical unit 101 may further include a shutter mechanism for adjusting the time for which light is irradiated onto the light-receiving surface. The aperture mechanism, focus mechanism, and shutter mechanism included in the optical unit are configured to be controlled by, for example, a sensor control unit 103. The optical unit 101 may be configured integrally with the imaging device 100 or may be configured separately from the imaging device 100.

[0027] The sensor unit 102 has a pixel array in which multiple pixels are arranged in a matrix. Each pixel includes a photoelectric conversion element, and the pixels arranged in a matrix form a light-receiving surface. The optical unit 101 forms an image of incident light on the light-receiving surface, and each pixel of the sensor unit 102 outputs a pixel signal corresponding to the irradiated light. The sensor unit 102 further includes a drive circuit for driving each pixel in the pixel array and a signal processing circuit for performing predetermined signal processing on signals read from each pixel and outputting the result as a pixel signal for each pixel. The sensor unit 102 outputs the pixel signals of each pixel in the pixel area as digital image data.

[0028] The sensor control unit 103 is configured by, for example, a microprocessor, controls the reading of pixel data from the sensor unit 102, and outputs image data based on each pixel signal read from each pixel. The pixel data output from the sensor control unit 103 is passed to the recognition processing unit 104 and the image processing unit 106.

[0029] The sensor control unit 103 also generates an imaging control signal for controlling imaging in the sensor unit 102 and supplies the signal to the sensor unit 102. The imaging control signal includes information indicating exposure and analog gain when imaging in the sensor unit 102. The imaging control signal further includes control signals for performing imaging operations of the sensor unit 102, such as a vertical synchronization signal and a horizontal synchronization signal.

[0030] The recognition processing unit 104 performs recognition processing (such as person detection, face identification, and image classification) of objects in an image using pixel data based on the pixel data passed from the sensor control unit 103. However, the recognition processing unit 104 may perform recognition processing using image data that has undergone image processing by the image processing unit 106. The recognition result by the recognition processing unit 104 is passed to the output control unit 107.

[0031] In this embodiment, the recognition processing unit 104 is configured using, for example, a DSP (Digital Signal Processor), and performs recognition processing using a machine learning model. Model parameters obtained by prior model training are stored in the memory 105, and the recognition processing unit 104 performs recognition processing using a trained model in which model parameters read from the memory 105 are set. Furthermore, if the model parameters used by the recognition processing unit 104 cannot ensure fairness in recognition results for pixel data or image data with a minority attribute, additional model training may be performed using Adversarial Examples generated from existing (or original) data with a minority attribute.

[0032] The image processing unit 106 processes the pixel data received from the sensor control unit 103 to obtain an image suitable for human viewing, and outputs, for example, image data consisting of a group of pixel data. For example, if a color filter is provided for each pixel in the sensor unit 102 and each pixel data has color information of R (red), G (green), or B (blue), the image processing unit 106 performs demosaic processing, white balance processing, and the like. The image processing unit 106 can also instruct the sensor control unit 103 to read pixel data required for image processing from the sensor unit 102. The image processing unit 106 passes the image data resulting from the pixel data processing to the output control unit 107. For example, the above-described functions of the image processing unit 106 are realized by an ISP (Image Signal Processor) executing a program stored in advance in a local memory (not shown).

[0033] The output control unit 107 is configured with, for example, a microprocessor. The output control unit 107 receives the recognition result of an object included in an image from the recognition processing unit 104 and the image data as the image processing result from the image processing unit 106, and outputs one or both of them to the outside of the imaging device 100. The output control unit 107 also outputs the image data to the display unit 108. The user can view the image displayed on the display unit 108. The display unit 108 may be built into the imaging device 100 or may be externally connected to the imaging device 100.

[0034] Fig. 2 shows an example of hardware implementation of the imaging device 100. In the example shown in Fig. 2, a sensor unit 102, a sensor control unit 103, a recognition processing unit 104, a memory 105, an image processing unit 106, and an output control unit 107 are mounted on a single chip 200. However, in Fig. 2, the memory 105 and the output control unit 107 are omitted from illustration to avoid confusion in the drawing.

[0035] 2, the recognition result by the recognition processing unit 104 is output to the outside of the chip 200 via the output control unit 107. In addition, the recognition processing unit 104 can acquire pixel data or image data to be used for recognition from the sensor control unit 103 via an interface inside the chip 200.

[0036] Fig. 3 shows another example of hardware implementation of the imaging device 100. In the example shown in Fig. 3, the sensor unit 102, sensor control unit 103, image processing unit 106, and output control unit 107 are mounted on a single chip 300, but the recognition processing unit 104 and memory 105 are arranged outside the chip 300. However, in Fig. 3 as well, the memory 105 and output control unit 107 are omitted from illustration to avoid confusion in the drawing.

[0037] 3, the recognition processing unit 104 acquires pixel data or image data to be used for recognition from the output control unit 107 via the inter-chip communication interface. The recognition processing unit 104 also directly outputs the recognition result to the outside. Of course, the recognition result by the recognition processing unit 104 can also be configured to be returned to the output control unit 107 in the chip 300 via the inter-chip communication interface and output from the output control unit 107 to the outside of the chip 300.

[0038] In the configuration example shown in Fig. 2, the recognition processing unit 104 and the sensor control unit 103 are both mounted on the same chip 200, so communication between the recognition processing unit 104 and the sensor control unit 103 can be performed at high speed via an interface within the chip 200. On the other hand, in the configuration example shown in Fig. 3, the recognition processing unit 104 is arranged outside the chip 200, so it is easy to replace the recognition processing unit 104. However, communication between the recognition processing unit 104 and the sensor control unit 103 must be performed via an interface between the chips, which makes communication slower.

[0039] 4 shows an example in which the semiconductor chips 200 (or 300) of the imaging device 100 are stacked in two layers to form a two-layer stacked image sensor 400. In the structure shown, a pixel section 411 is formed in a first-layer semiconductor chip 401, and a memory and logic section 412 is formed in a second-layer semiconductor chip 402.

[0040] The pixel unit 411 includes at least the pixel array in the sensor unit 102. The memory and logic unit 412 includes, for example, the sensor control unit 103, the recognition processing unit 104, the memory 105, the image processing unit 106, the output control unit 107, and an interface for communicating between the imaging device 100 and the outside. The memory and logic unit 412 also includes a part or all of a drive circuit that drives the pixel array in the sensor unit 102. Although not shown in FIG. 4, the memory and logic unit 412 may further include, for example, a memory that the image processing unit 106 uses to process image data.

[0041] As shown on the right side of FIG. 4, the imaging device 100 is configured as a single solid-state imaging element by bonding together a first layer semiconductor chip 401 and a second layer semiconductor chip 402 while making electrical contact with each other.

[0042] 5 shows an example in which the semiconductor chips 200 (or 300) of the imaging device 100 are stacked in three layers to form a three-layer stacked image sensor 500. In the structure shown in the figure, a pixel section 511 is formed in a first-layer semiconductor chip 501, a memory section 512 is formed in a second-layer semiconductor chip 502, and a logic section 513 is formed in a third-layer semiconductor chip 503.

[0043] The pixel unit 511 includes at least the pixel array in the sensor unit 102. The logic unit 513 includes, for example, the sensor control unit 103, the recognition processing unit 104, the image processing unit 106, the output control unit 107, and an interface for communicating between the imaging device 100 and the outside. The logic unit 513 also includes a part or all of a drive circuit that drives the pixel array in the sensor unit 102. The memory unit 512 may further include, in addition to the memory 105, a memory that the image processing unit 106 uses to process image data, for example.

[0044] As shown on the right side of Figure 5, the imaging device 100 is constructed as a single solid-state imaging element by bonding together a first layer semiconductor chip 501, a second layer semiconductor chip 502, and a third layer semiconductor chip 503 while maintaining electrical contact.

[0045] 6 shows an example of the configuration of the sensor unit 102. The sensor unit 102 shown in the figure includes a pixel array unit 601, a vertical scanning unit 602, an AD (Analog to Digital) conversion unit 603, a horizontal scanning unit 604, pixel signal lines 605, vertical signal lines VSL, a control unit 606, and a signal processing unit 607. Note that the control unit 606 and the signal processing unit 607 in FIG. 6 may be included in, for example, the sensor control unit 103 in FIG. 1.

[0046] The pixel array unit 601 is composed of a plurality of pixel circuits 610, each of which includes a photoelectric conversion element that performs photoelectric conversion on received light and a circuit that reads out electric charges from the photoelectric conversion element. The plurality of pixel circuits 610 are arranged in a matrix array in the horizontal direction (row direction) and the vertical direction (column direction). The arrangement of the pixel circuits 610 in the row direction forms a line. For example, when one frame of image is formed with 1920 pixels x 1080 lines, the pixel array unit 601 forms one frame of image using pixel signals read out from 1080 lines of lines each consisting of 1920 pixel circuits 610.

[0047] In the pixel array unit 601, pixel signal lines 605 are connected to the rows and columns of each pixel circuit 610, and vertical signal lines VSL are connected to the columns. The ends of each pixel signal line 605 that are not connected to the pixel array unit 601 are connected to a vertical scanning unit 602. Under the control of a control unit 606, the vertical scanning unit 602 transmits control signals such as drive pulses used to read pixel signals from pixels to the pixel array unit 601 via the pixel signal lines 605. The ends of the vertical signal lines VSL that are not connected to the pixel array unit 601 are connected to an AD conversion unit 603. The pixel signals read from the pixels are transmitted to the AD conversion unit 603 via the vertical scanning lines VSL.

[0048] A pixel signal is read from the pixel circuit 610 by transferring charges accumulated in a photoelectric conversion element due to exposure to a floating diffusion layer (FD) and converting the transferred charges into a voltage in the floating diffusion layer. The voltage converted from the charges in the floating diffusion layer is output to a vertical signal line VSL via an amplifier (not shown in FIG. 6).

[0049] The AD conversion unit 603 includes an AD converter 611 provided for each vertical signal line VSL, a reference signal generation unit 612, and a horizontal scanning unit 604. The AD converter 611 is a column AD converter that performs AD conversion processing for each column of the pixel array unit 601, and performs AD conversion processing on pixel signals supplied from the pixel circuits 610 via the vertical signal lines VSL to generate two digital values for correlated double sampling (CDS) processing that reduces noise and outputs them to the signal processing unit 607.

[0050] The reference signal generating unit 612 generates a ramp signal as a reference signal, which is used by each column AD converter 611 to convert a pixel signal into two digital values, based on a control signal from the control unit 606, and supplies the reference signal to each column AD converter 611. The ramp signal is a signal whose voltage level decreases at a constant slope over time, or a signal whose voltage level decreases in a step-like manner.

[0051] Within the AD converter 611, when a ramp signal is supplied, a counter starts counting in accordance with a clock signal, compares the voltage of the pixel signal supplied from the vertical signal line VSL with the voltage of the ramp signal, stops counting by the counter when the voltage of the ramp signal crosses the voltage of the pixel signal, and outputs a value corresponding to the count value at that time, thereby converting the pixel signal, which is an analog signal, into a digital value.

[0052] The signal processing unit 607 performs CDS processing based on the two digital values generated by the AD converter 611, generates a digital pixel signal (pixel data), and outputs it to the outside of the sensor control unit 103.

[0053] Under the control of the control unit 606, the horizontal scanning unit 604 performs a selection operation to select each AD converter 611 in a predetermined order, thereby causing each AD converter 611 to sequentially output the digital values temporarily held therein to the signal processing unit 607. The horizontal scanning unit 604 is configured using, for example, a shift register, an address decoder, etc.

[0054] Based on the imaging control signal supplied from the sensor control unit 103, the control unit 606 generates drive signals for controlling the driving of the vertical scanning unit 602, the AD conversion unit 603, the reference signal generation unit 612, the horizontal scanning unit 604, etc., and outputs the drive signals to each unit. For example, based on the vertical synchronization signal and the horizontal synchronization signal included in the imaging control signal, the control unit 606 generates control signals that the vertical scanning unit 602 supplies to each pixel circuit 610 via the pixel signal line 605, and supplies the control signals to the vertical scanning unit 602. The control unit 606 also passes information indicating an analog gain included in the imaging control signal to the AD conversion unit 603. Within the AD conversion unit 603, the gain of the pixel signal input to each AD converter 611 via the vertical signal line VSL is controlled based on the information indicating the analog gain.

[0055] Based on a control signal supplied from the control unit 606, the vertical scanning unit 602 supplies various signals including drive pulses to pixel signal lines 605 of a selected pixel row of the pixel array unit 601, to each pixel circuit 610 for each line, and causes each pixel circuit 610 to output a pixel signal to a vertical signal line VSL. The vertical scanning unit 602 is configured using, for example, a shift register, an address decoder, etc. Furthermore, the vertical scanning unit 602 controls the exposure of each pixel circuit 610 based on information indicating exposure supplied from the control unit 606.

[0056] The sensor unit 102 configured as shown in FIG. 6 is a column AD type image sensor in which AD converters 611 are arranged on a column-by-column basis.

[0057] The rolling shutter method and the global shutter method are available as imaging methods for capturing images using the pixel array unit 601. With the global shutter method, all pixels in the pixel array unit 601 are exposed simultaneously to read out pixel signals all at once. On the other hand, with the rolling shutter method, the pixel array unit 601 is exposed line by line from top to bottom to read out pixel signals.

[0058] C. Overview of DNN This section C provides an overview of recognition processing using DNNs (Deep Neural Networks) applicable to the present disclosure. In the present disclosure, it is assumed that recognition processing (hereinafter simply referred to as "image recognition processing") for image data is performed using a CNN (Convolutional Neural Network) and an RNN (Recurrent Neural Network) among DNNs.

[0059] C-1. Overview of CNN First, we will provide an overview of CNN. Image recognition processing using CNN generally involves performing image recognition processing based on image information, for example, pixels arranged in a matrix. Figure 7 shows a schematic diagram of how image recognition processing using CNN works. Processing is performed by a CNN 72 that has been trained in a predetermined manner on the entire pixel information 71 of an image 70 depicting a car, which is the object to be recognized. As a result, the "car" class is recognized as the recognition result 73.

[0060] It is also possible to perform CNN processing based on an image for each line, and obtain a recognition result from a portion of the image to be recognized. FIG. 8 schematically shows an image recognition processing mechanism for obtaining a recognition result from a portion of the image to be recognized. In FIG. 8, an image 80 is a partial acquisition of a "car," an object to be recognized, line by line. For example, pixel information 84a, 84b, and 84c for each line that forms pixel information 81 of this image 80 is sequentially processed by a CNN 82 that has been trained in a predetermined manner. Note that the "line" referred to here may not only refer to a single pixel line, but may also refer to a group of a predetermined number of pixel lines.

[0061] For example, it is assumed that the recognition result 83a obtained by the CNN 82 in the recognition process for pixel information 84a on the first line was not a valid recognition result. A valid recognition result here refers to, for example, a recognition result having a score indicating the reliability of the recognition result equal to or greater than a predetermined value. The CNN 82 updates its internal state 85 based on this recognition result 83a. Next, the CNN 82, whose internal state has been updated 85 using the previous recognition result 83a, performs recognition processing on pixel information 84b on the second line. In the example shown in FIG. 8 , this results in a recognition result 83b indicating that the object to be recognized is either a "car" or a "ship." Furthermore, the CNN 82 updates its internal information 85 based on this recognition result 83b. Next, the CNN 82, whose internal state has been updated 85 using the previous recognition result 83b, performs recognition processing on pixel information 84c on the third line. As a result, the object to be recognized is narrowed down to "car" out of "cars" and "ships."

[0062] The recognition process shown in Figure 8 updates the internal state of the CNN 82 using the result of the previous recognition process, and then the CNN 82 with its internal state updated performs recognition processing using pixel information of lines adjacent to the line on which the previous recognition process was performed. In other words, the recognition process shown in Figure 8 is performed on an image line by line while updating the internal state of the CNN based on the previous recognition result. Therefore, the recognition process shown in Figure 8 is a process that is performed recursively for each line, and can be considered to have a structure equivalent to an RNN.

[0063] C-2. Overview of RNN Next, we will provide a brief explanation of RNN. Figure 9 shows a schematic example of classification processing (recognition processing) by DNN when time-series information is not used. In this case, when one image is input to the DNN, classification processing is performed on the input image in the DNN, and the classification result is output.

[0064] Figure 10 shows in more detail the classification process shown in Figure 9. As shown in Figure 10, the DNN performs feature extraction processing and classification processing. In the DNN, first, feature amounts are extracted from the input image by feature extraction processing, and then classification processing is performed on the extracted feature amounts to obtain a classification result.

[0065] In contrast, Fig. 11 is a diagram schematically illustrating a first example of classification processing by DNN when time-series information is used. In the example shown in Fig. 11, classification processing by DNN is performed using a fixed number of past information in time series. That is, an image [T] at time T, an image [T-1] at time T-1 before time T, an image [T-2] at time T-2 before time T-1, ..., an image [TN] at time TN are input to the DNN. The DNN performs classification processing on each of the input images [T], [T-1], [T-2], ..., "TN", and obtains a classification result [T] at time T.

[0066] FIG. 12 shows the process shown in FIG. 11 in more detail. As shown in FIG. 12, the DNN performs the feature extraction process shown in FIG. 10 on each of the input images [T], [T-1], [T-2], ..., [TN] one-to-one to extract features corresponding to the images [T], [T-1], [T-2], ..., "TN." The DNN then integrates the features obtained based on these images [T], [T-1], [T-2], ..., "TN." Then, it performs a classification process on the integrated features to obtain a classification result [T] at time T. The methods shown in FIGS. 11 and 12 require multiple configurations for feature extraction, and additional configurations for feature extraction are required depending on the number of available past images, which may result in a large-scale DNN configuration.

[0067] A second example of the classification process by DNN when time-series information is used is shown in Fig. 13. In the example shown in Fig. 13, an image [T] at time T is input to a DNN whose internal state has been updated to the state at time T-1, and a classification result [T] at time T is obtained.

[0068] FIG. 14 shows the process shown in FIG. 13 in more detail. As shown in FIG. 14, the DNN performs the feature extraction process shown in FIG. 10 on the input image [T] at time T to extract features corresponding to image [T]. Within the DNN, the internal state is updated based on an image prior to time T, and features related to the updated internal state are stored. The features related to this stored internal information are integrated with the features in image [T], and classification processing is performed on the integrated features. The classification processing shown in FIGS. 13 and 14 is performed using a DNN whose internal state is updated using, for example, the previous classification result, and is a recursive process. A DNN that performs recursive processing in this way is called an RNN. Classification processing using an RNN is generally used for video recognition, and for example, classification accuracy can be improved by sequentially updating the internal state of the DNN using frame images that are updated in a time series.

[0069] The RNN is applied to, for example, an imaging device 100 using a rolling shutter system. In the rolling shutter system, pixel signals are read out line by line in sequence. The pixel signals read out for each line are then applied to the RNN as time-series information. This makes it possible to perform classification processing based on multiple lines with a smaller configuration than when a CNN is used (see FIG. 12). Of course, the RNN can also be applied to an imaging device 100 using a global shutter system. In this case, it is possible to regard adjacent lines as time-series information, for example.

[0070] C-3.Specific configuration of CNN Figure 15 shows an example of the configuration of a DNN using a multi-layer convolutional neural network (CNN). Generally, a CNN includes a feature extraction section that extracts features from an input image, and an image classification section that infers an output label (classification result) corresponding to the input image based on the extracted features. The former feature extraction section includes a "convolutional layer" that extracts edges and features by convolving the input image using a method of limiting connections between neurons and sharing weights, and a "pooling layer" that adds robustness to the features extracted by the convolutional layer by deleting information from positions that are not important for image classification.

[0071] In FIG. 15, reference numeral 1501 indicates an image that is input data to the CNN. Reference numerals 1502, 1504, and 1506 indicate outputs of the convolutional layer. Reference numerals 1503 and 1505 indicate outputs of the pooling layer. Reference numeral 1507 indicates the state in which the output 1506 of the convolutional layer is arranged one-dimensionally, reference numeral 1508 indicates a fully connected layer, and reference numeral 1509 indicates an output layer that is the inference result of class classification.

[0072] In the CNN 1500 shown in Fig. 15, the area enclosed by the box indicated by reference number 1520 is a feature extraction unit (e.g., equivalent to "feature extraction" in Fig. 10), which performs processing to acquire image features of an input image. The area enclosed by the box indicated by reference number 1530 is an image classification unit (e.g., equivalent to "classification" in Fig. 10), which identifies an output label based on the image features.

[0073] The stage of the inference process (the order of processing in each layer) is expressed as the output value in the lth layer, Y l Let Y be the processing in the lth layer. l =F l (Y l-1 ) and the first layer is Y 1 =F 1 (X), the final stage is Y=F 7 (Y 6 )

[0074] C-4. Overview of DNN decision-making criteria For example, algorithms such as Grad-CAM (Gradient-weighted Class Activation Mapping) (see, for example, Non-Patent Document 3), LIME (LOCAL Interpretable model-agnostic Explanations) (see, for example, Non-Patent Document 4), SHAP (SHapley Additive exPlanations), which is an advanced version of LIME, and TCAV (Testing with Concept Activation Vectors) (see, for example, Non-Patent Document 5) can be used to calculate the basis for decisions in DNNs (e.g., image classification or recognition).

[0075] Grad-CAM: Grad-CAM is an algorithm that estimates the locations in the input image data that contributed to the class classification by tracing the gradient backward from the label that is the classification result in the output layer (calculating the contribution of each feature map up to the class classification and backpropagating using the weights), and can visualize the locations that contributed to the class classification like a heat map. Alternatively, the position information of the pixels in the input image data can be retained until the final convolutional layer, and the influence of the position information on the final discrimination output can be obtained, so that the parts of the original input image that have a strong influence can be displayed as a heat map.

[0076] In the CNN 1500 shown in FIG. 15, when image recognition is performed on an input image and class c is output from the image classification unit 1530, a method for calculating the judgment basis based on the Grad-Cam algorithm (a method for generating a heat map) will be described.

[0077] Gradient y of class c c is the activation of the feature map A k Assuming that, the importance weights of the neurons are given as shown in the following equation (1).

[0078]

number

[0079] The final forward propagation output of the convolution layer is multiplied by the weight for each channel, and the Grad-Cam is calculated as shown in equation (2) below via the activation function ReLU.

[0080]

number

[0081] As shown in Figures 8 to 14, when the Grad-Cam algorithm is applied to a DNN that performs image recognition processing on a line-by-line basis, it is possible to calculate the basis for judgment for each line of the input image. For example, it is possible to display a heat map showing which part of the image [T] at time T was the basis for image classification.

[0082] Fig. 16 shows a schematic diagram of an example configuration in which the decision basis of a DNN that performs classification processing on time-series information is calculated based on the Grad-CAM algorithm. In the example shown in Fig. 16, an image [T] at time T is input to a DNN whose internal state has been updated to the state at time T-1, and a classification result [T] at time T is obtained. The decision basis calculation unit also calculates the basis location of the classification result [T] at time T within image "T" based on the Grad-Cam algorithm, and outputs a heat map.

[0083] LIME: LIME estimates that a specific input data item (feature) has "high importance in judgment" if its output reverses or fluctuates significantly when the input data item (feature) is changed. For example, to show the reason (basis) for the DNN's inference, it generates another locally approximating model (basis model). Then, when the DNN outputs a classification result, it can generate basis information using the basis model to generate a basis image.

[0084] TCAV: TCAV is an algorithm that calculates the importance of concepts (concepts that humans can easily understand) relative to the predictions of a trained model. For example, multiple pieces of input information are generated by duplicating or modifying the input information, and each piece of input information is input to a model (explained model) for which ground information is to be generated. The explained model then outputs multiple pieces of output information corresponding to each piece of input information. The ground model is then trained using pairs of each piece of input information and each corresponding piece of output information as training data, generating a ground model that locally approximates the target input information with another interpretable model. Once the DNN outputs the classification results, the ground information can be generated using the ground model, and a ground image can be generated in the same way.

[0085] D. Summary of the Disclosure In general, conventional image recognition functions require image processing for one to several frames of image data, which means that the basis for image recognition decisions can only be presented for one to several frames of image data, making them lacking in real-time performance. When applying image recognition technology to autonomous driving, there are limits to how quickly the basis for decisions can be presented to the driver.

[0086] In response to this, the present disclosure proposes an imaging device that performs image recognition processing on captured images at high speed and presents the basis for image recognition judgment in real time. The imaging device according to the present disclosure includes an imaging unit having a pixel area in which a plurality of pixels are arranged, a readout control unit that controls the reading of pixel signals from pixels included in the pixel area, a readout unit control unit that controls readout units set as part of the pixel area from which the readout control unit performs readout, a recognition unit that has learned learning data for each readout unit, and a judgment basis calculation unit that calculates the basis for judgment of the recognition processing in the recognition unit. The recognition unit performs recognition processing on pixel signals for each readout unit, and the judgment basis calculation unit calculates the basis for judgment of the results of the recognition processing for each readout unit.

[0087] Image recognition processing and calculation processing of judgment grounds for each read unit according to the present disclosure will be described with reference to FIG. 17 . Here, the target image is assumed to be an image of a car. Furthermore, memory 105 pre-stores a program or model parameters of a machine learning model that has been trained to be able to identify (classify) multiple types of objects, including cars, using predetermined learning data. The recognition processing unit 104 reads and executes this program or model parameters from memory 105, thereby enabling object identification processing of objects included in the captured image. The imaging device 100 captures images using a rolling shutter system, but even when capturing images using a global shutter system, image recognition processing and calculation processing of judgment grounds for each read unit are similarly possible. Furthermore, the recognition processing unit 104 is assumed to have both an image recognition processing function and a calculation function of judgment grounds for the recognition results.

[0088] First, the imaging device 100 starts capturing an image to be recognized (step S1701).

[0089] When imaging starts, the imaging device 1 sequentially reads out the frame line by line from the top end to the bottom end of the frame (step S1702).

[0090] When the lines are read up to a certain position, the recognition processing unit 104 identifies whether the subject is a "car" or a "ship" from the image based on the read lines (step S1703). For example, the objects "car" and "ship" have common features in their upper halves, so when the lines are read from the top and the feature is recognized, the recognized object can be identified as either a "car" or a "ship." At this time, the judgment basis calculation unit (see, for example, FIG. 16) calculates the basis for the DNN's identification of the object as either a "car" or a "ship" and displays the basis points in a heat map.

[0091] Here, as shown in step S1704a, by reading up to the line at or near the bottom of the frame, the entire object to be recognized appears, and the object identified as either a "car" or a "ship" in step S1702 is confirmed to be a "car." At this time, the judgment basis calculation unit calculates the basis for the DNN's identification of the object as a "car" and displays the location of that basis in a heat map.

[0092] Furthermore, as shown in step S1704b, by reading further lines from the line position read in step S1703, even when the line reaches the bottom of the "car," it is possible to identify the recognized object as a "car." For example, the lower half of a "car" and the lower half of a "ship" each have different characteristics. By reading the line up to the point where the difference in these characteristics becomes clear, it is possible to identify whether the object recognized in step S1703 is a "car" or a "ship." In the example shown in FIG. 17, the object is determined to be a "car" in step S1704b. At this time, the determination basis calculation unit calculates that the basis for the DNN's identification of the object as a "car" is the lower half of the "car," and displays the lower half of the "car" on the heat map.

[0093] Also, as shown in step S1704c, it is possible to jump from the line position in step S1703 to a line position that is likely to determine whether the object identified in step S1703 is a "car" or a "ship" and read it out further. By reading out the line to which this jump is made, it is possible to determine whether the object identified in step S1703 is a "car" or a "ship."

[0094] That is, when a candidate for a recognition result that satisfies a predetermined condition is obtained as a result of continuing line-by-line reading and recognition processing, the system jumps to the line position where a recognition result that satisfies the predetermined condition can be obtained, and performs line reading. Alternatively, when a candidate for a judgment basis that satisfies a predetermined condition is obtained as a result of continuing line-by-line reading and recognition processing, the system jumps to the line position where a judgment basis that satisfies the predetermined condition can be presented, and performs line reading.

[0095] The jump destination line position can be determined using a machine learning model that has been trained in advance based on predetermined learning data. Of course, the jump destination line position may be determined to be a predetermined number of lines (or a predetermined number of lines) ahead from the current line position. In this case, the judgment basis calculation unit calculates the basis for the DNN's identification of the object as a "car" based on the jump destination line position, and displays the location of that basis in a heat map.

[0096] If the object is confirmed in step S1704b or step S1704c, the recognition processing unit 104 further calculates the basis for the determination based on the Grad-CAM algorithm or the like, and then the image capturing device 100 can terminate the recognition process. This makes it possible to achieve higher speeds and power savings by reducing the amount of processing in the recognition process in the image capturing device 100, and also makes it possible to present the basis for the determination in real time.

[0097] The training data is data that holds multiple combinations of input signals and output signals for each readout unit. For example, in the task of identifying objects described above, a data set that combines input signals (line data, subsampled data, etc.) for each readout unit with object classes (human body / vehicle / non-object) and object coordinates (x, y, h, w) can be applied as training data. Alternatively, output signals may be generated from input signals only using self-supervised learning.

[0098] In the imaging device 100, the recognition processing unit 104 reads and executes a program or model parameters stored in the memory 105 as a machine learning model that has been pre-trained using the above-described training data, thereby functioning as a recognizer that uses DNN and further presents the basis for the recognizer's judgment.

[0099] FIG. 18 shows, in the form of a flowchart, the processing procedure for executing each process of image recognition and calculation of the basis for judgment by the recognition processing unit 104.

[0100] First, the DSP constituting the recognition processing unit 104 reads and executes the program or model parameters of the machine learning model from the memory 105 (step S1801). This allows the DSP to function as a recognizer using the trained machine learning model, and further enables calculation of the basis for image recognition.

[0101] Next, the recognition processing unit 104 instructs the sensor control unit 103 to start reading frames from the sensor unit 102 (step S1802). In this frame reading, for example, image data for one frame is read out sequentially in a predetermined reading unit (for example, in line units).

[0102] The recognition processing unit 104 checks whether image data for a predetermined number of lines in one frame has been read (step S1803). If it is determined that image data for a predetermined number of lines in one frame has been read (Yes in step S1803), the recognition processing unit 104 performs recognition processing on the read image data for the predetermined number of lines using a trained CNN (step S1804). That is, the recognition processing unit 104 performs recognition processing using a machine learning model, with image data for the predetermined number of lines as a unit region.

[0103] In image data recognition processing using CNN, for example, recognition or detection processes such as face detection, face authentication, gaze detection, facial expression recognition, face direction detection, object detection, object recognition, motion (animal object) detection, pet detection, scene recognition, state detection, and avoidance target recognition are performed. Face detection is the process of detecting the face of a person included in image data. Face authentication is a type of biometric authentication that authenticates whether the face of a person included in image data matches the face of a pre-registered person. Gaze detection is the process of detecting the gaze direction of a person included in image data. Facial expression recognition is the process of recognizing the facial expression of a person included in image data. Facial direction detection is the process of detecting the up-down direction of a person's face included in image data. Object detection is the process of detecting objects included in image data. Object recognition is the process of recognizing the identity of an object included in image data. Motion (animal object) detection is the process of detecting an animal included in image data. Pet detection is the process of detecting pets such as dogs and cats included in image data. Scene recognition is the process of recognizing the scene being photographed (the sea, mountains, etc.). State detection is a process of detecting the state of a subject such as a person included in image data (whether the state is normal or abnormal, etc.). Recognition of an object to be avoided is a process of recognizing an object to be avoided that exists ahead in the direction of travel when the vehicle itself is moving. The recognition process executed by the recognition processing unit 104 is not limited to the examples listed above.

[0104] Then, the recognition processing unit 104 determines whether the recognition process using CNN in step S1804 was successful (step S1805). Here, successful recognition means that a certain recognition result was obtained, such as a reliability level equal to or greater than a predetermined value, in the image recognition process such as the one exemplified above. On the other hand, unsuccessful recognition means that a sufficient detection result, recognition result, or authentication was not obtained, such as a reliability level not reaching a predetermined value, in the image recognition process such as the one exemplified above.

[0105] If the recognition processing unit 104 determines that the recognition processing using CNN has been successful (Yes in step S1805), it shifts the process to step S1809. On the other hand, if the recognition processing unit 104 determines that the recognition processing using CNN has failed (No in step S1805), it shifts the process to step S1806.

[0106] In step S1806, the recognition processing unit 104 waits until the next predetermined number of lines of image data is read from the sensor control unit 103 (No in step S1806). Then, when the next predetermined number of lines of image data (unit area) is read (Yes in step S1806), the recognition processing unit 104 executes recognition processing using an RNN on the read image data of the predetermined number of lines (step S1807). In the recognition processing using an RNN, for example, the results of machine learning processing using a CNN or an RNN that has been executed so far on image data of the same frame are also used (see, for example, FIGS. 13 and 14).

[0107] Then, the recognition processing unit 104 determines whether the recognition process using the RNN in step S1807 was successful (step S1808). Here, successful recognition means that a certain recognition result was obtained, such as a reliability level exceeding a predetermined value, in the image recognition process exemplified above. On the other hand, unsuccessful recognition means that a sufficient detection result, recognition result, or authentication was not obtained, such as a reliability level not reaching a predetermined value, in the image recognition process exemplified above.

[0108] If the recognition processing unit 104 determines that the recognition processing using the RNN has been successful (Yes in step S1808), it proceeds to step S1809.

[0109] In step S1809, the recognition processing unit 104 supplies the valid recognition result that was successful in step S1804 or step S1807 to, for example, the output control unit 107.

[0110] Next, in the recognition processing unit 104, the judgment basis calculation unit (see FIG. 16) calculates the judgment basis (step S1810) and supplies it to, for example, the output control unit 107. For example, when the judgment basis calculation unit uses the Grad-CAM algorithm, by tracing the gradient backward from the label that is the classification result in the output layer of the CNN or RNN that has succeeded in recognition (calculating the contribution of each feature map up to the classification and backpropagating using the weight), it is possible to identify the parts of the original input image that contributed to the classification and visualize them like a heat map.

[0111] The output control unit 107 outputs the recognition result output from the recognition processing unit 104 in step S1809 and the judgment basis calculated in step S1810 to the display unit 108 and displays them on the screen. For example, the original input image and the image recognition result are displayed on the screen of the display unit 108, and a heat map showing the result of the basis calculation is superimposed on the original input image. Furthermore, the output control unit 107 may store the recognition result output from the recognition processing unit 104 in step S1809 and the judgment basis calculated in step S1810 in the memory 105 in association with the original input image.

[0112] If the recognition processing unit 104 determines that the recognition processing using the RNN has failed (No in step S1808), the processing proceeds to step S1811. In step S1811, the recognition processing unit 104 checks whether or not reading of one frame of image data has been completed.

[0113] If it is determined that reading of one frame of image data has not been completed (No in step S1811), the process returns to step S1806, and the same process as above is repeatedly executed for the next predetermined number of lines of image data.

[0114] On the other hand, if it is determined that the reading of one frame of image data has been completed (Yes in step S1811), for example, the recognition processing unit 104 determines whether or not to end the series of processes shown in FIG. 18 (step S1812).

[0115] Here, the determination of whether to end the series of processes in step S1812 may be made based on, for example, whether an instruction to end has been input from outside the imaging device 100, or based on whether the series of processes for a predetermined number of frames of image data has been completed. Alternatively, the condition for ending the series of processes may be when a desired object has been recognized from a frame (or when it has been determined that the desired object cannot be recognized from a frame) and a basis for the determination (that the user can accept) has been presented.

[0116] 18 is not yet finished (No in step S1812), the recognition processing unit 104 returns the process to step S1802, reads the next frame, and repeats the same operations as above. On the other hand, when the recognition processing unit 104 determines that the process shown in Fig. 18 is finished (Yes in step S1812), the recognition processing unit 104 ends the entire process.

[0117] When performing consecutive recognition processes such as face detection, face authentication, gaze detection, facial expression recognition, face direction detection, object detection, object recognition, movement (moving object) detection, scene recognition, and state detection, if the immediately preceding recognition process fails, the next recognition process may be skipped. For example, when performing face authentication following face detection, if face detection fails, the next face authentication may be skipped.

[0118] Next, the specific operation of the recognition processing unit 104 will be described using an example in which face detection is performed using DNN.

[0119] Fig. 19 shows an example of image data for one frame. Fig. 20 shows the flow of recognition processing that the recognition processing unit 104 in the image capturing device 100 executes on the image data shown in Fig. 19.

[0120] When performing face detection using machine learning on image data such as that shown in FIG. 19, as shown in FIG. 20(a), image data for a predetermined number of lines is first input to the recognition processing unit 104 (corresponding to step S1803 in FIG. 18). The recognition processing unit 104 performs face detection by performing machine learning processing using CNN on the input image data for the predetermined number of lines (corresponding to step S1804 in FIG. 18). However, at the stage of FIG. 20(a), image data of the entire face has not yet been input, so the recognition processing unit 104 fails to detect the face (corresponding to No in step S1805 in FIG. 18).

[0121] 20(b), image data for the next predetermined number of lines is input to the recognition processing unit 104 (corresponding to step S1806 in FIG. 18). The recognition processing unit 104 uses the results of the face recognition processing using CNN on the image data for the predetermined number of lines input in FIG. 20(a), and executes face recognition processing using RNN on the newly input image data for the predetermined number of lines (corresponding to step S1807 in FIG. 18).

[0122] At the stage of Fig. 20(b), image data of the entire face has been input in addition to the pixel data for the predetermined number of lines input at the stage of Fig. 20(a). Therefore, at the stage of Fig. 20(b), the recognition processing unit 104 succeeds in face detection (corresponding to Yes in step S1808 in Fig. 18). Then, in this operation, the result of face recognition is output (corresponding to step S1809 in Fig. 18) without reading out the next and subsequent image data (image data in Figs. 20(c) to (f)).

[0123] In this way, by performing machine learning processing using DNN on image data for a predetermined number of lines at a time, it is possible to omit reading out image data and performing recognition processing on image data after successful face recognition. This makes it possible to complete detection and recognition processing in a short time, thereby reducing processing time and power consumption.

[0124] Furthermore, when calculating the basis for face recognition using the Grad-CAM algorithm, for example, by tracing the gradient backward from the label that is the classification result in the output layer of a neural network model that has succeeded in recognition (calculating the contribution of each feature map up to the classification and backpropagating using the weights), it is possible to identify the locations of the pixel data of some of the lines that contributed to the classification at the stage shown in Figure 20(b) and visualize them like a heat map. In other words, it is possible to speed up the recognition process and present the basis for judgment in real time.

[0125] The predetermined number of lines is determined by the size of the filter required by the learning model algorithm, and the minimum number is one line.

[0126] Furthermore, the image data read from the sensor unit 102 by the sensor control unit 103 may be image data thinned in at least one of the column direction and the row direction. In this case, for example, when image data is read every other row in the column direction, image data on the 2(N-1)th line (N is an integer equal to or greater than 1) is read.

[0127] Furthermore, if the filter required by the learning model algorithm is not in units of lines, but is a rectangular area in units of pixels, such as 1x1 pixels or 5x5 pixels, image data of a rectangular area corresponding to the shape and size of the filter may be input to the recognition processing unit 104 instead of image data with a predetermined number of lines as image data of a unit area on which the recognition processing unit 104 performs machine learning processing.

[0128] In addition, although a DNN consisting of CNN and RNN has been mentioned above as an example of a machine learning model that performs recognition processing, the present invention is not limited to these and machine learning models with other structures can also be used. Furthermore, although the Grad-Cam algorithm has been mainly mentioned as an example of calculating the decision basis in a DNN, the present invention is not limited to these and other algorithms can also be used to calculate the decision basis of a machine learning model.

[0129] E. Examples of the Disclosure E-1. Example (1) FIG. 21 shows a detailed example of the functional configuration of the imaging device 100 shown in FIG. 1, mainly in the vicinity of the sensor control unit 103, the recognition processing unit 104, and the image processing unit .

[0130] The sensor control unit 103 includes a readout unit 2101 and a readout control unit 2102. The recognition processing unit 104 includes a feature amount calculation unit 2111, a feature amount storage control unit 2112, a readout determination unit 2114, a recognition processing execution unit 2115, and a judgment basis calculation unit 2116. The feature amount storage control unit 2112 includes a feature amount storage unit 2113. The image processing unit 106 includes an image data storage control unit 2121, a readout determination unit 2123, and an image processing unit 2124. The image data storage control unit 2121 includes an image data storage unit 2122.

[0131] In the sensor control unit 103, the read control unit 2102 receives readout area information indicating a readout area to be read out in the recognition processing unit 104 from the readout determination unit 2114 in the recognition processing unit 104. The readout area information is, for example, the line numbers of one or more lines. However, the readout area information may be information specifying various patterns of readout areas, such as information indicating pixel positions within one line, or a combination of one or more line numbers and information indicating pixel positions of one or more pixels within a line. Note that the readout area is equivalent to the readout unit, but the readout area and the readout unit may be different.

[0132] Similarly, the read control unit 2102 receives read area information indicating the read area from which reading is to be performed in the image processing unit 106 from a read determination unit 2123 in the image processing unit 106 .

[0133] The read control unit 2102 passes read area information indicating the read area from which the input image is actually read, to the read unit 2101, based on the read determination units 2114 and 2123. For example, when a conflict occurs between the read area information received from the read determination unit 2114 and the read area information received from the read determination unit 2123, the read control unit 2102 arbitrates and adjusts the read area information to be passed to the read unit 2101, for example, so that both read areas are included or so that both read areas are a common area.

[0134] Furthermore, the read control unit 2102 can receive imaging control information (for example, exposure, analog gain, etc.) from the read determination unit 2114 or the read determination unit 2123. The read control unit 2102 passes the received imaging control information to the read unit 2101.

[0135] The readout unit 2101 reads pixel data from the sensor unit 102 in accordance with the readout region information passed from the readout control unit 2102. For example, the readout unit 2101 obtains a line number indicating the line to be read out and pixel position information indicating the position of the pixel to be read out on that line based on the readout region information, and passes the obtained line number and pixel position information to the sensor unit 102. The readout unit 2101 passes each piece of pixel data acquired from the sensor unit 102, together with the readout region information, to the recognition processing unit 104 and the image processing unit 106.

[0136] The readout unit 2101 also performs imaging control such as exposure and analog gain (AG) on the sensor unit 102 based on imaging control information received from the readout control unit 2102. Furthermore, the readout unit 2101 can generate a vertical synchronization signal and a horizontal synchronization signal and supply them to the sensor unit 102.

[0137] In the recognition processing unit 104, the read determination unit 2114 receives read information indicating the read area to be read next from the feature accumulation control unit 2112. The read determination unit 2114 generates read area information based on the received read information and passes it to the read control unit 2102.

[0138] Here, the read determination unit 2114 can use, for example, information in which read position information for reading pixel data of a predetermined read unit is added to the read unit as the read area indicated in the read area information. The read unit is a set of one or more pixels and serves as a unit of processing by the recognition processing unit 104 and the image processing unit 106. As an example, if the read unit is a line, a line number [L#x] indicating the position of the line is added as the read position information. Furthermore, if the read unit is a rectangular area including multiple pixels, information indicating the position of the rectangular area in the pixel array unit 601, for example, information indicating the position of the pixel in the upper left corner, is added as the read position information. The read determination unit 2114 is specified in advance as the read unit to be applied. However, the read determination unit 2114 can also determine the read unit in response to, for example, an instruction from outside the read determination unit 2114. Therefore, the read determination unit 2114 functions as a read unit control unit that controls the read unit.

[0139] The read determination unit 2114 can also determine the read area to be read next based on recognition information passed from the recognition process execution unit 2115 (described later) and generate read area information indicating the determined read area.

[0140] Similarly, in the image processing unit 106, the read determination unit 2123 receives read information indicating the read area to be read next, for example, from the image data storage control unit 2121. The read determination unit 2123 generates read area information based on the received read information and passes it to the read control unit 2102.

[0141] In the recognition processing unit 104, a feature calculation unit 2111 calculates feature amounts in the area indicated by the readout area information based on the pixel data and readout area information passed from the readout unit 2101. The feature calculation unit 2111 passes the calculated feature amounts to a feature accumulation control unit 2112.

[0142] Here, the feature amount calculation unit 2111 may calculate the feature amount based on past feature amounts passed from the feature amount accumulation control unit 2112, in addition to the pixel data passed from the readout unit 2101. Furthermore, the feature amount calculation unit 2111 may obtain information for setting exposure and analog gain from the readout unit 2101, for example, and further use this obtained information to calculate the feature amount.

[0143] In the recognition processing unit 104, the feature accumulation control unit 2112 accumulates the feature passed from the feature calculation unit 2111 in the feature accumulation unit 2113. Furthermore, when the feature is passed from the feature calculation unit 2111, the feature accumulation control unit 2112 generates read information indicating the read area from which the next read will be performed, and passes the read information to the read determination unit 2114.

[0144] Here, the feature accumulation control unit 2112 can integrate and accumulate already accumulated feature amounts and newly passed feature amounts. The feature accumulation control unit 2112 can also delete feature amounts that are no longer needed from the feature amounts accumulated in the feature accumulation unit 2113. Examples of feature amounts that are no longer needed include feature amounts related to the previous frame and feature amounts that have been calculated and already accumulated based on frame images of scenes different from the frame images for which new feature amounts have been calculated. The feature accumulation control unit 2112 can also initialize the feature accumulation unit 2113 by deleting all feature amounts accumulated therein as necessary.

[0145] Furthermore, the feature accumulation control unit 2112 generates a feature to be used in the recognition process by the recognition process execution unit 2115, based on the feature passed from the feature calculation unit 2111 and the feature accumulated in the feature accumulation unit 2113. The feature accumulation control unit 2112 passes the generated feature to the recognition process execution unit 2115.

[0146] The recognition process execution unit 2115 executes recognition processing based on the feature amounts passed from the feature amount accumulation control unit 2112. The recognition process execution unit 2115 performs object detection, face detection, and the like through recognition processing. The recognition process execution unit 2115 passes the recognition result obtained through the recognition processing to the output control unit 107. The recognition process execution unit 2115 can also pass recognition information including the recognition result generated through the recognition processing to the readout determination unit 2114. Note that the recognition process execution unit 2115 can receive feature amounts from the feature amount accumulation control unit 2112 and execute recognition processing based on, for example, a trigger generated by the trigger generation unit 2130.

[0147] The determination basis calculation unit 2116 calculates the basis for image recognition such as object detection and face detection in the recognition processing execution unit 2115. When the feature amount calculation unit 2111 and the recognition processing execution unit 2115 are configured using a neural network model, the determination basis calculation unit 2116 can estimate the parts of the original image that contributed to the recognition result by tracing the gradient backward from the label that is the identification result of class classification in the output layer using, for example, the Grad-Cam algorithm (calculating the contribution of each feature map up to class classification and backpropagating using the weights). Then, the determination basis calculation unit 2116 passes the calculated determination basis to the output control unit 107.

[0148] In the image processing unit 106, the image data storage control unit 2121 receives pixel data read from the read area and read area information corresponding to the image data from the reading unit 2101. The image data storage control unit 2121 associates the pixel data and the read area information and stores them in the image data storage unit 2122.

[0149] The image data storage control unit 2121 generates image data for image processing by the image processing unit 2124 based on the pixel data passed from the read unit 2101 and the image data stored in the image data storage unit 2122. The image data storage control unit 2121 passes the generated image data to the image processing unit 2124. The image data storage control unit 2121 can also pass the pixel data passed from the read unit 2101 to the image processing unit 2124 as is.

[0150] Furthermore, the image data storage control unit 2121 generates read information indicating the read area from which the next read will be performed based on the read area information passed from the reading unit 2101 , and passes the generated read information to the read determination unit 2123 .

[0151] Here, the image data storage control unit 2121 can integrate and store already stored image data and newly passed pixel data, for example, by averaging. The image data storage control unit 2121 can also delete image data that is no longer needed from the image data stored in the image data storage unit 2122. Possible examples of image data that is no longer needed include image data relating to the previous frame, and image data that has been calculated and already stored based on a frame image of a scene different from the frame image for which new image data was calculated. The image data storage control unit 2121 can also delete and initialize all image data stored in the image data storage unit 2122 as necessary.

[0152] Furthermore, the image data storage control unit 2121 can acquire information for setting exposure and analog gain from the reading unit 2101, and store image data corrected using the acquired information in the image data storage unit 2122.

[0153] The image processing unit 2124 performs predetermined image processing on the image data passed from the image data storage control unit 2121. For example, the image processing unit 2124 can perform predetermined image quality improvement processing on the image data. Furthermore, if the passed image data is image data in which data has been spatially reduced by line thinning or the like, the image processing unit 2124 can also fill in the thinned-out portions with image information by interpolation processing. The image processing unit 2124 passes the image data that has been subjected to image processing to the output control unit 107.

[0154] The image processing unit 2124 can receive image data from the image data storage control unit 2121 and perform image processing based on a trigger generated by the trigger generation unit 2130, for example.

[0155] The output control unit 107 outputs either or both of the recognition result passed from the recognition processing execution unit 2115 and the image data passed from the image processing unit 2124. The output control unit 107 may also output the recognition judgment basis passed from the judgment basis calculation unit 2116 together with the recognition result. The output control unit 107 outputs either or both of the recognition result and the image data in response to a trigger generated by the trigger generation unit 2130, for example.

[0156] The trigger generation unit 2130 generates a trigger to be passed to the recognition processing execution unit 2115, a trigger to be passed to the image processing unit 2124, and a trigger to be passed to the output control unit 107, based on information related to the recognition processing passed from the recognition processing unit 104 and information related to the image processing passed from the image processing unit 106. The trigger generation unit 2130 passes each of the generated triggers to the recognition processing execution unit 2115, the image processing unit 2124, and the output control unit 107, respectively, at predetermined timings.

[0157] 22 shows a more detailed example of processing in the recognition processing unit 104. In the figure, the readout area is assumed to be a line, and the readout unit 2101 reads pixel data line by line from the top to the bottom of the frame of the input image (from the sensor unit 1021). The line image data (line data) of the line L#x read out line by line by the readout unit 2101 is input to the feature amount calculation unit 2111.

[0158] The feature calculation unit 2111 executes a feature extraction process 2201 and an integration process 2203. The feature calculation unit 2111 performs the feature extraction process 2201 on the input line data to extract a feature 2202 from the line data. Here, the feature extraction process 2201 extracts the feature 2202 from the line data based on parameters obtained in advance by learning. The feature 2202 extracted by the feature extraction process 2201 is integrated with a feature (internal state) 2213 processed by the feature accumulation control unit 2112 by the integration process 2203. The integrated feature 2211 is passed to the feature accumulation control unit 2112.

[0159] The feature accumulation control unit 2112 executes internal state update processing 2212. The feature 2211 passed to the feature accumulation control unit 2112 is passed to the recognition processing execution unit 2115 and is subjected to internal state update processing 2212. The internal state update processing 2212 reduces the feature 2211 based on pre-learned parameters to update the internal state of the DNN, and generates feature (internal state) 2213 related to the updated internal state. This feature (internal state) 2213 is integrated with feature 2202 of the currently input line data by integration processing 2203. This processing by the feature accumulation control unit 2112 corresponds to processing using an RNN.

[0160] The recognition process execution unit 2115 executes recognition process 2221 on the feature 2211 passed from the feature accumulation control unit 2112 based on parameters previously learned using, for example, predetermined learning data, and outputs the recognition result.

[0161] As described above, in the recognition processing unit 104 according to the first embodiment, processing is performed based on pre-trained parameters in the feature extraction processing 2201, the integration processing 2203, the internal state update processing 2212, and the recognition processing 2221. The parameters are learned using, for example, training data based on an expected recognition target.

[0162] Furthermore, the determination basis calculation unit 2116 calculates the basis for recognition in the recognition processing execution unit 2115. When the feature calculation unit 2111 and the recognition processing execution unit 2115 are configured using a neural network model, the determination basis calculation unit 2116 uses, for example, the Grad-Cam algorithm to trace the gradient backward from the label that is the identification result of class classification in the output layer (calculating the contribution of each feature map up to class classification and backpropagating using the weights), thereby estimating the location within the image that has been read out so far that contributed to the recognition result, focusing on the feature 2202 extracted from the currently input line data by the feature extraction processing 2201 and the feature 2211 integrated with the feature (internal state) 2213 by the integration processing 2203. The determination basis calculation unit 2116 then passes the calculated determination basis to the output control unit 107.

[0163] The functions of the feature amount calculation unit 2111, feature amount storage control unit 2112, read determination unit 2114, recognition processing execution unit 2115, and determination basis calculation unit 2116 described above are realized, for example, by loading and executing a program stored in the memory 105 or the like into a DSP provided in the imaging device 100. Similarly, the functions of the image data storage control unit 2121, read determination unit 2123, and image processing unit 2124 described above are realized, for example, by loading and executing a program stored in the memory 105 or the like into an ISP provided in the imaging device 100. These programs may be stored in the memory 105 in advance, or may be supplied to the imaging device 100 from an external device and written into the memory 105.

[0164] Fig. 23 shows functional parts mainly related to the recognition processing unit 104 in Fig. 21. Of the functional configuration shown in Fig. 21, Fig. 23 omits the image processing unit 106, the output control unit 107, the trigger generation unit 2130, and the read control unit 2101 in the sensor control unit 103.

[0165] FIG. 24 illustrates the frame readout process in this embodiment. In this embodiment, the readout unit is a line, and pixel data is read out line by line for frame Fr(x). In the example shown in FIG. 24, in the m-th frame Fr(m), lines are read out line by line starting from line L#1 at the top of frame Fr(m), with lines L#2, L#3, ... being read out line by line. Then, when line readout in frame Fr(m) is completed, in the next (m+1)-th frame Fr(m+1), lines are read out line by line starting from line L#1 at the top in a similar manner.

[0166] Fig. 25 shows an outline of the recognition process in this embodiment. As shown in Fig. 25, recognition is performed by sequentially executing processing by CNN 82 and updating 85 of internal information on pixel information 84a, 84b, 84c of each line L#1, L#2, L#3, .... Since it is only necessary to input pixel information 84 for one line to CNN 82, it is possible to configure a recognizer 86 on an extremely small scale.

[0167] Furthermore, the judgment basis calculation unit 2116 uses, for example, the Grad-Cam algorithm to trace the gradient backward from the label that is the classification result in the output layer of the CNN 82 (calculating the contribution of each feature map up to the classification and backpropagating using the weights), thereby estimating the location on the current input line that contributed to the recognition result, and further estimating the location within the image that has been read so far that contributed to the recognition result, based on the updated internal information 85. The judgment basis calculation unit 2116 starts the calculation process of the judgment basis for the recognition result before the read process of the entire frame is completed, thereby shortening the time until the calculation result of the judgment basis is obtained and making it possible to present the recognition judgment basis in real time.

[0168] The recognizer 86 shown in FIG. 25 can be said to have a configuration as an RNN, since it executes processing by CNN 82 on image data input line by line to perform internal information update 85. By performing line-by-line recognition processing using an RNN, it may be possible to obtain valid recognition results without reading all lines included in a frame. When performing line-by-line recognition processing, the recognition processing unit 104 can terminate the recognition processing when a valid recognition result is obtained. Furthermore, the determination basis calculation unit 2116 can calculate the determination basis when the recognition processing unit 104 obtains a valid recognition result.

[0169] 26 and 27 show examples in which the recognition processing unit 104, which performs recognition processing on a line-by-line basis, obtains a valid recognition result in the middle of frame readout and ends the recognition processing.

[0170] 26 shows an example of line-by-line recognition processing when the handwritten number "8" is the recognition target. In the example shown in FIG. 26, the number "8" is recognized when approximately ¾ of the vertical range 2601 of the input frame 2600 is read. Therefore, the recognition processing unit 104 can output a valid recognition result indicating that the number "8" has been recognized when this range 2601 is read, and can end line reading and recognition processing for frame 2600. Furthermore, the determination basis calculation unit 2116 starts calculation processing of the determination basis when approximately ¾ of the vertical range 2601 is read and the number "8" is recognized. This shortens the time required to obtain the calculation result of the determination basis, making it possible to present the recognition determination basis in real time.

[0171] 27 shows an example of recognition processing in units of lines when a person is the recognition target. In the example shown in FIG. 27, a person 2702 is recognized when a range 2701 that is approximately half the vertical range is read in frame 2700. Therefore, the recognition processing unit 104 can output a valid recognition result indicating that the person 2702 has been recognized when this range 2701 is read, and can end line reading and recognition processing for frame 2700. Furthermore, the determination basis calculation unit 2116 starts calculation processing of the determination basis when the range 2701 that is approximately half the vertical range is read and the person 2702 is recognized, thereby shortening the time required to obtain the calculation result of the determination basis and making it possible to present the recognition determination basis in real time.

[0172] In this way, in this embodiment, if a valid recognition result is obtained during line readout for a frame, the line readout and recognition process can be terminated and the calculation process of the judgment basis can be started. This makes it possible to speed up and save power by reducing the processing amount in the recognition process, and also shorten the time required for the recognition process and presentation of the judgment basis.

[0173] 26 and 27 show an example in which line reading is performed from the top to the bottom of the frame, but the line reading method is not limited to this example. For example, line reading may be performed from the bottom to the top of the frame. Objects that are far from the image capture device 100 generally approach the vanishing point at the top of the frame, so by reading the line from the top to the bottom of the frame, they can be recognized earlier. On the other hand, objects that are closer to the image capture device 100 generally approach the bottom of the frame, away from the vanishing point, so by reading the line from the bottom to the top of the frame, they can be recognized earlier.

[0174] For example, consider a case where the imaging device 100 is installed for vehicle use to capture an image of the front. Since a near object (e.g., a vehicle or pedestrian ahead of the vehicle) is present in the lower portion of the captured image, it is more effective to perform line readout from the bottom edge of the frame toward the top edge. Furthermore, when an immediate stop is required in an ADAS (Advanced Driver-Assistance Systems), it is sufficient that at least one relevant object is recognized. Once one object is recognized, it is considered more effective to perform line readout again from the bottom edge of the frame. Furthermore, on expressways, for example, distant objects may be given priority. In this case, it is preferable to perform line readout from the top edge of the frame toward the bottom edge. That is, in the case of the imaging device 100 for vehicle use, the line readout direction and the line readout order may be switched depending on the driving situation, etc.

[0175] Furthermore, the frame readout unit may be the column direction of the row and column directions in the pixel array unit 601. For example, it is possible to use a plurality of pixels arranged in one column in the pixel array unit 601 as the readout unit. By applying a global shutter method as the imaging method, column readout, in which the column is the readout unit, is possible. The global shutter method makes it possible to switch between column readout and line readout. When readout is fixed to column readout, it is possible to use a rolling shutter method by rotating the pixel array unit 601 by 90°, for example.

[0176] For example, an object on the left side of the image capture device 100 can be recognized earlier and the basis for judgment can be presented in real time by sequentially reading out the frame from the left end using column readout. Similarly, an object on the right side of the image capture device 100 can be recognized earlier and the basis for judgment can be presented in real time by sequentially reading out the frame from the right end using column readout.

[0177] In an example where the image capture device 100 is used in a vehicle, for example, when the vehicle is turning, priority may be given to an object on the turning side. In such a case, it is preferable to perform column readout from the end of the turning side. The turning direction can be obtained, for example, based on steering information of the vehicle. However, this is not limiting, and for example, a sensor capable of detecting angular velocity in three directions can be provided in the image capture device 1, and the turning direction can be obtained based on the detection results of this sensor.

[0178] Fig. 28 shows in the form of a flowchart the procedure for the recognition and determination basis calculation process corresponding to the reading of pixel data in a read unit (for example, one line) from a frame according to the first embodiment. The illustrated processing procedure corresponds to the reading of pixel data in a read unit (for example, one line) from a frame, for example. However, Fig. 28 explains the procedure for the recognition and determination basis calculation process when the read unit is a line. For example, the read area information can use a line number indicating the line to be read.

[0179] First, the recognition processing unit 104 reads line data from a line indicated by the read line of the frame (step S2801). Specifically, the read determination unit 2114 passes the line number of the line to be read next to the sensor control unit 103. In the sensor control unit 103, the read unit 2101 reads pixel data of the line indicated by the line number from the sensor unit 102 as line data in accordance with the passed line number. The read unit 2101 passes the line data read from the sensor unit 102 to the feature amount calculation unit 2111. The read unit 2101 also passes read area information (e.g., line number) indicating the area from which pixel data has been read to the feature amount calculation unit 2111.

[0180] Next, the feature amount calculation unit 2111 calculates the feature amount of the image based on the line data passed from the reading unit 2101 (step S2802). Furthermore, the feature amount calculation unit 2111 acquires the feature amount accumulated in the feature amount accumulation unit 2113 from the feature amount accumulation control unit 2112 (step S2803), and integrates the feature amount calculated in step S2802 with the feature amount acquired from the feature amount accumulation control unit 2112 in step S2803 (step S2804). The integrated feature amount is passed to the feature amount accumulation control unit 2112. The feature amount accumulation control unit 2112 accumulates the integrated feature amount in the feature amount accumulation unit 2113 (step S2805).

[0181] Note that if the series of processes in steps S2801 to S2804 are for the first line of a frame and the feature amount storage unit 2113 has been initialized, for example, then steps S2803 and S2804 can be omitted. Furthermore, the process in step S2805 in this case is to store the line feature amount calculated based on the first line in the feature amount storage unit 2113.

[0182] The feature amount accumulation control unit 2112 also passes the integrated feature amount passed from the feature amount calculation unit 2111 to the recognition process execution unit 2115. The recognition process execution unit 2115 executes recognition process using the integrated feature amount passed from the feature amount accumulation control unit 2112 (step S2806). The recognition process execution unit 2115 outputs the recognition result of the recognition process to the output control unit 107 (step S2807).

[0183] The recognition process execution unit 2115 also outputs the recognition result from the recognition process to the determination basis calculation unit 2116. The determination basis calculation unit 2116 calculates the determination basis for the recognition result passed from the recognition process execution unit 2115 (step S2808). The determination basis calculation unit 2116 uses, for example, the Grad-Cam algorithm to estimate locations on the line data that contributed to the recognition result from the feature amounts of the line data calculated in step S2802, or estimates locations within the image range read so far that contributed to the recognition result based on the feature amounts integrated in step S2804. The determination basis calculation unit 2116 then outputs the calculated determination basis to the output control unit 107 (step S2809).

[0184] Next, in the recognition processing unit 104, the read determination unit 2114 determines the read line to be read next in accordance with the read information passed from the feature accumulation control unit 2112 (step S2810). For example, when the feature accumulation control unit 2112 receives read region information along with features from the feature calculation unit 2111, it determines the read line to be read next in accordance with, for example, a pre-specified read pattern (in line units in this example) based on this read region information. The processing from step S2801 is executed again for the read line determined in step S2810.

[0185] Next, an example of control of the readout and recognition processing according to the first embodiment will be described. Figures 29 and 30 show example time charts of the control of the readout and recognition processing and determination basis calculation according to the first embodiment. Figures 29 and 30 show example time charts in which a blank period blk in which no imaging operation is performed is provided within one imaging cycle (one frame cycle).

[0186] FIG. 29 shows an example of a time chart in which, for example, 1 / 2 of the imaging cycle is consecutively allocated to blank periods blk. In the figure, the imaging cycle is a frame cycle, for example, 1 / 30 [sec]. Frames are read out from the sensor unit 102 at this imaging cycle. The imaging time is the time required to capture images of all lines included in a frame. In the example shown in FIG. 29, a frame includes n lines, and imaging of n lines L#1 to L#n is completed in 1 / 60 [sec], which is 1 / 2 of the frame cycle of 1 / 30 [sec]. The time allocated to capturing images of one line is 1 / (60×n) [sec]. The blank period blk is the 1 / 30 [sec] period from the timing when the last line L#n in a frame is captured to the timing when the first line L#1 of the next frame is captured.

[0187] For example, when imaging of line L#1 is completed, imaging of the next line L#2 is started, and the recognition processing execution unit 2115 executes line recognition processing for that line L#1, and the determination basis calculation unit 2116 executes calculation processing of the determination basis for that line recognition. The recognition processing execution unit 2115 and the determination basis calculation unit 2116 finish their respective processes before imaging of the next line L#2 is started. When the recognition processing execution unit 2115 finishes the line recognition processing for line L#1, it outputs the recognition result of the recognition processing, and the determination basis calculation unit 2116 outputs the calculation result of the determination basis for that line recognition.

[0188] Similarly, for the next line L#2, when imaging of the line L#2 ends, imaging of the next line L#3 begins, and the recognition processing execution unit 2115 executes line recognition processing for the line L#1, and the judgment basis calculation unit 2116 executes calculation processing of the judgment basis for line recognition. Before imaging of the next line L#3 begins, the recognition processing execution unit 2115 and the judgment basis calculation unit 2116 finish their respective processes. In this way, imaging of the lines L#1, L#2, L#3, ..., L#m, ..., L#n is executed sequentially. Then, for each of the lines L#1, L#2, L#3, ..., L#m, ..., L#n, when imaging ends, imaging of the line following the line for which imaging has ended begins, and line recognition processing for the line for which imaging has ended and calculation processing of the judgment basis for the recognition result are executed.

[0189] In this way, by sequentially executing the recognition process and the calculation process of the basis for the judgment of the recognition result for each read unit (in this example, each line), the recognition result and the basis for the judgment can be obtained sequentially without inputting all the image data of the frame to the recognizer (recognition processing unit 104), thereby reducing the delay until the recognition result and the basis for the judgment are obtained. Furthermore, when a valid recognition result is obtained for a certain line, the recognition process can be terminated at that point, thereby shortening the time required for the recognition process and the calculation process of the basis for the judgment and saving power. Furthermore, by propagating and integrating information on the time axis for the recognition results of each line, it is possible to gradually improve the recognition accuracy.

[0190] In the example shown in FIG. 29, other processing that should be executed within a frame period (for example, image processing in the image processing unit 106 using the recognition result) can be executed during the blank period blk within the frame period.

[0191] Fig. 30 shows an example of a time chart in which a blank period blk is provided for each line imaged. In this figure, the frame period (image capture period) is 1 / 30 [sec] as in the example shown in Fig. 29, but the image capture time is the same 1 / 30 [sec] as the image capture period. Also, in the example shown in Fig. 30, in one frame period, images of n lines, line L#1 to line L#n, are captured at time intervals of 1 / (30×n) [sec], and the image capture time for one line is 1 / (60×n) [sec].

[0192] In this case, a blank period blk of 1 / (60×n) [sec] can be provided for each imaging of each line L#1 to L#n. During each blank period blk of each line L#1 to L#n, other processing to be performed on the imaging image of the corresponding line (e.g., image processing in the image processing unit 106 using the recognition result) can be performed. At this time, the time (approximately 1 / (30×n) [sec] in this example) until just before imaging of the line next to the target line is completed can be allocated to this other processing. In the example time chart shown in FIG. 30, the processing results of this other processing can be output for each line, making it possible to obtain the processing results of this other processing more quickly.

[0193] Fig. 31 shows another example of a time chart for the control of the readout, recognition processing, and judgment basis calculation according to the first embodiment. In the example time charts shown in Figs. 29 and 30, imaging of all lines L#1 to L#n included in a frame is completed in half the period of the frame cycle, and the remaining half of the period of the frame cycle is a blank period. In contrast, in the example time chart shown in Fig. 31, no blank period is provided within the frame cycle, and imaging of all lines L#1 to L#n included in a frame is performed using the entire period of the frame cycle.

[0194] Here, if the imaging time for one line is 1 / (60×n) [sec], the same as in Figures 29 and 30, and the number of lines included in a frame is n, the same as in Figures 29 and 30, the frame period, i.e., imaging period, is 1 / 60 [sec]. Therefore, in the example of the time chart shown in Figure 31, which does not have a blank period blk, the frame rate can be made faster than the examples in Figures 29 and 30 described above.

[0195] E-2. Variations In this section E-2, we will explain some variations of the embodiment described in section E-1. However, each variation can basically be realized using the recognition processing unit 104 having the functional configuration shown in Fig. 21.

[0196] E-2-1. Variation (1) In the first modified example, the recognition process and the calculation process of the basis for judgment are performed with the image data read out in units of a plurality of adjacent lines.

[0197] 32 illustrates a frame readout process according to the first modified example. As shown in the figure, in the first modified example, pixel data of a line group, each of which includes a plurality of adjacent lines, is read out sequentially for a frame Fr(m). In the recognition processing unit 104, the readout determination unit 2114 determines, for example, a line group Ls#x including a predetermined number of lines as a readout unit.

[0198] The read determination unit 2114 sends read area information, which is information indicating the read unit determined as the line group Ls#x and read position information for reading out pixel data of the read unit, to the read control unit 2102. The read control unit 2102 sends the read area information sent from the read determination unit 2114 to the read unit 2101. The read unit 2101 reads pixel data from the sensor unit 102 in accordance with the read area information sent from the read control unit 2102.

[0199] 32, in the mth frame Fr(m), line group Ls#x is read out line by line, starting from line group Ls#1 at the top of frame Fr(m), followed by line group Ls#2, Ls#3, ..., Ls#p, ... Once reading of line group Ls#x in frame Fr(m) is completed, in the next (m+1)th frame Fr(m+1), line group Ls#x is similarly read out line by line, starting from line group Ls#1 at the top, followed by line group Ls#2, Ls#3, ..., Ls#p, ...

[0200] In this way, by reading pixel data using a line group Ls#x including multiple lines as a read unit, it is possible to read one frame's worth of pixel data faster than when reading is performed line by line. Furthermore, the recognition processing unit 104 can use more pixel data in one recognition process, thereby improving the recognition response speed. Furthermore, since the number of reads per frame is reduced compared to reading per line, it is possible to suppress distortion of the captured frame image when the imaging method of the sensor unit 102 is the rolling shutter method.

[0201] 32, the line group Ls#x may be read from the bottom to the top of the frame, as in the embodiment described in Section E-1 above. Furthermore, if a valid recognition result is obtained during the reading of the line group Ls#x for a frame, the reading of the line group, the recognition process, and the calculation process of the basis for judgment can be terminated. This reduces the amount of processing in the recognition process, enabling faster speeds and power savings. Furthermore, by shortening the time required for the recognition process and the calculation process of the basis for judgment, the basis for recognition can be presented in real time.

[0202] E-2-2. Variation (2) Next, a second modified example will be described. In the second modified example, the read unit is a part of one line.

[0203] Fig. 33 illustrates a frame readout process according to the second modified example. In the example shown in Fig. 33, in line readout performed line-sequentially, a portion of each line (referred to as a partial line) is used as a readout unit, and pixel data is read out for each partial line Lp#x in each line for each frame Fr(m). In the recognition processing unit 104, the readout determination unit 2114 determines, for example, from among the pixels included in a line, a number of pixels that are adjacent to each other in sequence and whose number is less than the total number of pixels included in the line as a readout unit.

[0204] The read determination unit 2114 passes read area information, which is information indicating the read unit determined as the partial line Lp#x plus read position information for reading out pixel data of the partial line Lp#x, to the read control unit 2102. Here, the information indicating the read unit is composed of, for example, the position of the partial line Lp#x within one line and the number of pixels included in the partial line Lp#x. Furthermore, the read position information uses the line number including the partial line Lp#x to be read. The read control unit 2102 passes the read area information passed from the read determination unit 2114 to the read unit 2101. The read unit 2101 reads pixel data from the sensor unit 102 in accordance with the read area information passed from the read control unit 2102.

[0205] 33, in the mth frame Fr(m), partial lines Lp#2, Lp#3, ..., and each partial line Lp#x included in each line are read out line by line, starting from partial line Lp#1 included in the top line of frame Fr(m). After reading out the line group in frame Fr(m) is completed, in the next (m+1)th frame Fr(m+1), partial lines Lp#x are read out line by line, starting from partial line Lp#1 included in the top line.

[0206] In this way, in line readout, by limiting the pixels to be read out to those included in a portion of the line, it is possible to transfer pixel data in a narrower bandwidth than when pixel data is read out from the entire line. By using the readout method according to the modification shown in Figure 33, the amount of pixel data transferred is reduced compared to when pixel data is read out from the entire line, making it possible to save power.

[0207] In the second modified example shown in Fig. 33, the partial lines may be read from the bottom to the top of the frame, as in the embodiment described in Section E-1 above. Furthermore, if a valid recognition result is obtained during the reading of the partial lines for a frame, the reading of the partial line Lp#x, the recognition process, and the calculation process of the judgment basis can be terminated. This reduces the amount of processing in the recognition process, thereby enabling faster speeds and power savings. Furthermore, by shortening the time required for the recognition process and the calculation process of the judgment basis, the judgment basis can be presented in real time.

[0208] E-2-3. Variation (3) Next, a third modified example will be described. In the third modified example, the read unit is an area of a predetermined size within a frame. Fig. 34 illustrates a frame read process according to the third modified example.

[0209] In the example shown in Fig. 34, an area Ar#xy of a predetermined size containing a plurality of pixels adjacent in both the line and vertical directions within a frame is used as a readout unit, and for frame Fr(m), the area Ar#xy is read out sequentially, for example, in the line direction, and further, the sequential readout of this area Ar#xy in the line direction is repeated sequentially in the vertical direction. In the recognition processing unit 104, the readout determination unit 2114 determines the area Ar#xy defined by, for example, the size in the line direction (number of pixels) and the size in the vertical direction (number of lines) as the readout unit. In the example shown in Fig. 33, the pixel readout position in the line direction on each line was fixed, whereas in the example shown in Fig. 34, the pixel readout position in the line direction moves for each line.

[0210] The read determination unit 2114 passes readout area information, which is information indicating the readout unit determined as the area Ar#xy, plus readout position information for reading out pixel data of the area Ar#xy, to the readout control unit 2102. Here, the information indicating the readout unit is composed of, for example, the above-mentioned line size (number of pixels) and vertical size (number of lines). Furthermore, the readout position information uses the position of a specific pixel included in the area Ar#xy to be read out, for example the pixel position of the pixel in the upper left corner of the area Ar#xy. The readout control unit 2102 passes the readout area information passed from the readout determination unit 2114 to the readout unit 2101. The readout unit 2101 reads pixel data from the sensor unit 102 in accordance with the readout area information passed from the readout control unit 2102.

[0211] 34, in the mth frame Fr(m), areas Ar#x and Ar#y are read out in the line direction starting from area Ar#1-1 located in the upper left corner of frame Fr(m), followed by areas Ar#2-1, Ar#3-1, ... In frame Fr(m), when reading out has been performed up to the right end in the line direction, the vertical read position is moved, and areas Ar#x and Ar#y are read out again starting from the left end of frame Fr(m) in the line direction in order, starting from area Ar#1-2, Ar#2-2, Ar#3-2, ...

[0212] Fig. 35 is a schematic diagram outlining the recognition process when image data is read out for each area of a predetermined size from a frame as shown in Fig. 34. As shown in Fig. 35, recognition is performed by sequentially executing processing by CNN 82 and updating 85 of internal information on pixel information 84a, 84b, 84c of each area Ar#1-1, Ar#2-1, Ar#3-1, .... It is only necessary to input pixel information 84 for one area to CNN 82, making it possible to configure a recognizer 86 on an extremely small scale.

[0213] Furthermore, the judgment basis calculation unit 2116 uses, for example, the Grad-Cam algorithm to trace the gradient backward from the label that is the classification result in the output layer of the CNN 82 (calculating the contribution of each feature map up to the classification and backpropagating using the weights), thereby estimating the locations in the current input area that contributed to the recognition result, and further estimating the locations in the image within the range that has been read so far that contributed to the recognition result, based on the updated internal information 85. The judgment basis calculation unit 2116 starts the calculation process of the judgment basis for the recognition result before the read process of the entire frame is completed, thereby shortening the time until the calculation result of the judgment basis is obtained and making it possible to present the recognition judgment basis in real time.

[0214] The recognizer 86 shown in FIG. 35 can be said to have a configuration as an RNN, since it processes sequentially input information using a CNN 82 and updates internal information 85. By performing recognition processing for each area using an RNN, it may be possible to obtain valid recognition results without reading all areas included in a frame. When performing recognition processing for each area, the recognition processing unit 104 can end the recognition processing when a valid recognition result is obtained. Furthermore, the judgment basis calculation unit 2116 can calculate the judgment basis when the recognition processing unit 104 obtains a valid recognition result.

[0215] 36 and 37 show examples in which the recognition processing unit 104 terminates the recognition processing midway through frame readout when the readout unit is area Ar#xy.

[0216] FIG. 36 shows an example of area-based recognition processing when the handwritten number "8" is the recognition target. In the example shown in FIG. 36, the number "8" is recognized at position P1 when approximately two-thirds of the range 3601 of the input frame 3600 is read out. Therefore, the recognition processing unit 104 can output a valid recognition result indicating that the number "8" has been recognized when this range 3601 is read out, and terminate area readout and recognition processing for frame 3600. Furthermore, the determination basis calculation unit 2116 starts calculation processing of the determination basis when approximately two-thirds of the range 3601 is read out and the number "8" is recognized. This shortens the time required to obtain the calculation result of the determination basis, making it possible to present the recognition determination basis in real time.

[0217] 37 shows an example of recognition processing in area units when a person is the recognition target. In the example shown in FIG. 37, a person 3702 is recognized at position P2 when approximately half a range 3701 in the vertical direction is read out in frame 3700. Therefore, the recognition processing unit 104 can output a valid recognition result indicating that person 3702 has been recognized when this range 3701 is read out, and can end area readout and recognition processing for frame 3700. Furthermore, the determination basis calculation unit 2116 starts calculation processing of the determination basis when approximately half a range 2701 in the vertical direction is read out and person 3702 is recognized, thereby shortening the time required to obtain the calculation result of the determination basis and making it possible to present the recognition determination basis in real time.

[0218] In this way, in the third modification, if a valid recognition result is obtained during area readout for a frame, the area readout and recognition process can be terminated and the calculation process of the judgment basis can be started. This enables faster processing and power saving by reducing the processing amount in the recognition process, and also shortens the time required for the recognition process and presentation of the judgment basis. Furthermore, according to the third modification shown in Figures 34 to 37, redundant reading is reduced compared to the embodiment in which reading is performed across the entire width in the line direction, making it possible to further shorten the time required for the recognition process.

[0219] In the third modified example, the area Ar#xy is read out from the left end to the right end in the line direction and from the top end to the bottom end of the frame in the vertical direction, but this is not limited to this example. For example, the line direction reading may be performed from the right end to the left end, and the vertical direction reading may be performed from the bottom end to the top end of the frame.

[0220] E-2-4. Variation (4) Next, a fourth modified example will be described. In the fourth modified example, the read unit is a pattern consisting of a plurality of pixels including non-adjacent pixels. Figure 38 illustrates a frame read process according to the fourth modified example.

[0221] 38, a pattern Pφ#xy consisting of a plurality of pixels arranged discretely and periodically in both the line direction and the vertical direction is set as a readout unit. Specifically, the pattern Pφ#xy is made up of six pixels arranged periodically: three pixels arranged at a predetermined interval in the line direction and three pixels arranged at a predetermined interval in the vertical direction so that their positions in the line direction correspond to those of the three pixels. In the recognition processing unit 104, the readout determination unit 2114 determines the plurality of pixels arranged according to this pattern Pφ#xy as a readout unit.

[0222] Although the above description has been given of the pattern Pφ#xy being composed of a plurality of discrete pixels, the pattern serving as a readout unit is not limited to this example. For example, the pattern Pφ#xy may be composed of a plurality of pixel groups each including a plurality of adjacent pixels arranged discretely. In the example shown in FIG. 38, the pattern Pφ#xy is composed of a plurality of pixel groups each consisting of four adjacent pixels (2 pixels x 2 pixels) arranged discretely and periodically. In the example shown in FIG. 38, each pattern Pφ#xy is composed of a total of six pixel groups, with three pixel groups arranged discretely and periodically in the line direction and two pixel groups arranged discretely and periodically in the vertical direction.

[0223] The read determination unit 2114 passes readout area information, which is information indicating the readout unit determined as the pattern Pφ#xy and readout position information for reading out the pattern Pφ#xy, to the readout control unit 2102. Here, the information indicating the readout unit may be configured, for example, by information indicating the positional relationship between a predetermined pixel among the pixels constituting the pattern Pφ#xy (for example, the pixel in the upper left corner among the pixels constituting the pattern Pφ#xy) and each of the other pixels constituting the pattern Pφ#xy. Furthermore, the readout position information may be information indicating the position of a predetermined pixel included in the pattern Pφ#xy to be read out (information indicating the position within the line and the line number). The readout control unit 2102 passes the readout area information passed from the readout determination unit 2114 to the readout unit 2101. The readout unit 2101 reads pixel data from the sensor unit 10 in accordance with the readout area information passed from the readout control unit 2102.

[0224] 38, in the mth frame Fr(m), for example, pattern Pφ#1-1, whose upper left corner pixel is located at the upper left corner of frame Fr(m), is read out by sequentially shifting the position, for example, by one pixel in the line direction, and patterns Pφ#2-1, Pφ#3-1, ..., and each pattern Pφ#xy is read out. For example, when the right end of pattern Pφ#xy reaches the right end of frame Fr(m), the position is shifted vertically by one pixel (one line) from the left end of frame Fr(m), and patterns Pφ#1-2, Pφ#2-2, Pφ#3-2, ..., and each pattern Pφ#xy is read out in the same manner.

[0225] Because the pattern Pφ#xy is composed of periodically arranged pixels, the operation of moving the pattern Pφ#xy by one pixel can be considered an operation of shifting the phase of the pattern Pφ#xy. That is, in the fourth modification, each pattern P#xy is read out while shifting the phase of the pattern Pφ#xy in the line direction by Δφ. The pattern Pφ#xy is moved in the vertical direction by shifting the phase Δφ' in the vertical direction relative to the position of the first pattern Pφ#1-y in the line direction, for example.

[0226] FIG. 39 shows a schematic diagram of a recognition process applicable to the fourth modification. FIG. 39 shows an example in which a pattern Pφ#z is formed by four pixels spaced one pixel apart in the horizontal (line) and vertical directions. As shown in FIGS. 39(a) to 39(d), patterns Pφ#1, Pφ#2, Pφ#3, and Pφ#4, each consisting of four pixels shifted in phase by one pixel in the horizontal and vertical directions, enable all 16 pixels included in a 4×4 pixel area to be read out without overlap. The four pixels read out according to patterns Pφ#1, Pφ#2, Pφ#3, and Pφ#4 are subsamples Sub#1, Sub#2, Sub#3, and Sub#4, extracted from the 16 pixels included in the 4×4 pixel sample area without overlapping each other.

[0227] 39(a) to 39(d), the recognition process is performed by executing the process by the CNN 82 and the internal information update 85 for each of the subsamples Sub#1, Sub#2, Sub#3, and Sub#4. Therefore, it is only necessary to input four pieces of pixel data to the CNN 82 for one recognition process, and the recognizer 86 can be configured on an extremely small scale.

[0228] Furthermore, the judgment basis calculation unit 2116 uses, for example, the Grad-Cam algorithm to trace the gradient backward from the label that is the classification result in the output layer of the CNN 82 (calculating the contribution of each feature map up to the classification and backpropagating using the weights), thereby estimating the locations on the current input pattern that contributed to the recognition result, and further estimating the locations of the images within the range that have been read so far that contributed to the recognition result, based on the updated internal information 85. The judgment basis calculation unit 2116 starts the calculation process of the judgment basis for the recognition result before the readout process for the entire frame is completed, thereby shortening the time required to obtain the calculation result of the judgment basis and making it possible to present the recognition judgment basis in real time.

[0229] FIG. 40 shows an example time chart of the readout and control of the recognition process and determination basis calculation according to the fourth modified example. In the figure, the imaging period is the frame period, which is 1 / 30 [sec] in the example shown in FIG. 40. Frames are read out from the sensor unit 102 at this frame period. The imaging time is the time required to capture images of all subsamples Sub#1 to Sub#4 included in a frame, which is 1 / 30 [sec] in the example shown in FIG. 40, the same as the imaging period. Note that imaging of one subsample Sub#x is called subsample imaging.

[0230] In the fourth modification, the imaging time is divided into four periods, and subsample imaging of each of the subsamples Sub#1, Sub#2, Sub#3, and Sub#4 is performed in each period. Specifically, the sensor control unit 103 performs subsample imaging using the subsample Sub#1 over the entire frame in the first period of the first to fourth periods into which the imaging time is divided. The sensor control unit 103 extracts the subsample Sub#1 while moving, for example, a 4 pixel x 4 pixel sample area in the line direction so that the sample areas do not overlap each other. The sensor control unit 103 repeatedly performs the operation of extracting the subsample Sub#1 while moving the sample area in the line direction in the vertical direction.

[0231] When extraction of one frame's worth of subsamples Sub#1 is completed, the recognition processing unit 104 inputs the extracted one frame's worth of subsamples Sub#1, for example, into the recognizer 86 for each subsample Sub#1, and performs recognition processing. The recognition processing unit 104 outputs the recognition result after completing recognition processing for one frame. Alternatively, the recognition processing unit 104 may output the recognition result if a valid recognition result is obtained during recognition processing for one frame, and terminate recognition processing for that subsample Sub#1. Furthermore, the determination basis calculation unit 2116 performs calculation processing of the determination basis for recognition for that subsample Sub#1.

[0232] Thereafter, in the second, third, and fourth periods, subsample imaging is similarly performed over the entire frame using subsamples Sub#2, Sub#3, and Sub#4, respectively. If a valid recognition result is obtained during the recognition process for one frame, the recognition processing unit 104 outputs the recognition result, and the determination basis calculation unit 2116 calculates the determination basis for the recognition of the subsamples in the recognition processing unit 104.

[0233] The frame readout process according to the fourth modified example when the readout unit is a sample area will be specifically described with reference to FIGS.

[0234] Fig. 41 shows an example of recognition processing in sample area units when the handwritten number "8" is the recognition target. Specifically, this shows an example of recognition processing when three numbers "8" of different sizes are included in one frame. Each frame 4100 shown in Fig. 41(a), (b), and (c) includes three objects 4101, 4102, and 4103 depicting the number "8" of different sizes. However, of the three objects 4101, 4102, and 4103 included in frame 4100, object 4101 is the largest and object 4103 is the smallest.

[0235] In FIG. 41(a), subsample Sub#1 is extracted from the sample area indicated by reference number 4111. By extracting subsample Sub#1 from each sample area 4110 included in frame 4100, pixels are read out from frame 4100 in a grid pattern at intervals of every other pixel in both the horizontal and vertical directions, as shown in FIG. 41(a). In the example shown in FIG. 41(a), the recognizer 86 can recognize only object 4101, which is the largest of objects 4101, 4102, and 4103, based on the pixel data of the pixels read out in this grid pattern. It is also possible to present the basis for determining the recognition of the largest object 4101.

[0236] After the extraction of subsample Sub#1 from frame 4100 is completed, the extraction of subsample Sub#2, indicated by reference numeral 4112, is performed. Subsample Sub#2 is composed of pixels that are shifted by one pixel horizontally and one pixel vertically within the sample area relative to subsample Sub#1. Since the recognizer 86 has a structure equivalent to an RNN and its internal state is updated based on the recognition result of subsample Sub#1, the recognition result resulting from the extraction of subsample Sub#2 is affected by the recognition process of subsample Sub#1. The recognition process resulting from the extraction of subsample Sub#2 can be considered to be performed based on pixel data of pixels read in a checkerboard pattern, as shown in FIG. 41(b). Therefore, the state in which subsample Sub#2 is further extracted, as shown in FIG. 41(b), has improved pixel data-based resolution compared to the recognition process performed when only subsample Sub#1 is extracted, as shown in FIG. 41(a), enabling more accurate recognition. In the example shown in FIG. 41(b), the recognizer 86 can also recognize object 4102, which is the next largest object after object 4101. It is also possible to present the basis for determining recognition of object 4102, which is the next largest object after object 4101.

[0237] Fig. 41(c) shows a state in which extraction of all subsamples Sub#1 to Sub#4 has been completed in frame 4100. In the state shown in Fig. 41(c), all pixels included in frame 4100 have been read out, and in addition to objects 4101 and 4102 recognized in the extraction of subsamples Sub#1 and Sub#2, the smallest object 4103 has been recognized. It is also possible to present the basis for determining the recognition of all objects 4101, 4102, and 4103.

[0238] 42 shows an example of recognition processing in sample area units when a person is the recognition target, and illustrates an example of recognition processing when one frame contains images of three people of different sizes due to different distances from the image capture device 100. Each frame 4200 shown in FIGS. 42(a), (b), and (c) contains three objects 4201, 4202, and 4203, each made up of an image of a person of different sizes depending on the distance from the image capture device 100. However, of the three objects 4201, 4202, and 4203 included in frame 4200, object 4201 made up of an image of a person closest to the image capture device 100 is the largest, and object 4203 made up of an image of a person farthest from the image capture device 100 is the smallest.

[0239] 42(a), similarly to the example shown in FIG. 41(a), subsample Sub#1 is extracted, and the recognizer 86 executes recognition processing to recognize object 4201, which is the largest of objects 4201, 4202, and 4203. It is also possible to present the basis for determining the recognition of the largest object 4201.

[0240] In FIG. 42(b), subsample #2 is extracted, similar to the example shown in FIG. 41(b). Subsample #2 is composed of pixels that are shifted by one pixel horizontally and one pixel vertically within the sample area relative to subsample Sub#1. The recognizer 86 has a structure equivalent to an RNN, and its internal state is updated based on the recognition result of subsample Sub#1. Therefore, the recognition result obtained by extracting subsample Sub#2 is affected by the recognition process of subsample Sub#1. As a result, compared to the case shown in FIG. 42(a) where recognition is performed with only subsample Sub#1 extracted, the resolution based on pixel data is improved, enabling more accurate recognition. In the example shown in FIG. 42(b), the recognizer 86 can further recognize object 4202, which is the second largest object after object 4201. It is also possible to present the basis for determining the recognition of object 4202, which is the second largest object after object 4201.

[0241] Similarly to the example shown in FIG. 41(c), FIG. 42(c) shows a state where extraction of all subsamples Sub#1 to Sub#4 has been completed. In the state shown in FIG. 42(c), all pixels contained in frame 4200 have been read out, and in addition to objects 4201 and 4202 recognized by the extraction of subsamples Sub#1 and Sub#2, the smallest object 4203 has also been recognized. In this way, by repeating the extraction and recognition process of subsamples Sub#1, Sub#2, ..., it becomes possible to recognize people who are successively farther away. It is also possible to present the basis for determining the recognition of all objects 4201, 4202, and 4203.

[0242] 41 and 42, frame reading, recognition processing, and determination basis calculation processing can be controlled according to the time allocable to recognition processing. As an example, if the time allocable to recognition processing is short, frame reading and recognition processing can be terminated when extraction of subsample Sub#1 in frame 4100 is completed and object 4101 is recognized, and determination basis for the recognition results up to that point can be calculated and presented. On the other hand, if the time allocable to recognition processing is long, frame reading and recognition processing can be continued until extraction of all subsamples Sub#1 to Sub#4 is completed.

[0243] Alternatively, the recognition processing unit 104 may control frame readout, recognition processing, and calculation processing of the basis for judgment according to the reliability (score) of the recognition result. For example, in Fig. 42(b), if the score for the recognition result based on the extraction and recognition processing of subsample Sub#2 is equal to or greater than a predetermined value, the recognition processing unit 104 may terminate the recognition processing and execute calculation processing of the basis for judgment for the recognition result at that point, but may not execute extraction of the next subsample Sub#3.

[0244] In this way, in the fourth variant, the recognition process can be terminated when a predetermined recognition result is obtained, which enables faster processing and power saving by reducing the amount of processing in the recognition processing unit 104, and also makes it possible to shorten the time required for the recognition process and presenting the basis for judgment.

[0245] In addition, in the fourth modification, the recognition response speed for large objects in a frame can be increased, making it possible to increase the frame rate. In addition, in accordance with the increased recognition response speed, the time required to present the basis for the recognition result can be shortened.

[0246] E-2-5. Variation (5) Next, a fifth modified example will be described. In the fifth modified example, the read unit is a pattern in which multiple pixels, including non-adjacent pixels, are randomly arranged. Figure 43 illustrates a frame read process according to the fifth modified example.

[0247] 43, for example, a pattern Rd#m_x configured with a plurality of pixels arranged discretely and non-periodically within a frame Fr(m) is used as a readout unit. That is, the readout unit according to the fifth modification is the entire frame.

[0248] In the fifth modification, referring to FIG. 40 described above, one frame period is divided into multiple periods, and a pattern is switched for each period. In the example shown in FIG. 43, in the first period into which the frame period of the mth frame Fr(m) is divided, the recognition processing unit 104 reads pixels according to a pattern Rd#m_1 consisting of multiple pixels that are discretely and non-periodically arranged in the frame Fr(m) and performs recognition processing. As an example, if the total number of pixels included in the frame Fr(m) is s and the number of divisions of the frame period is D, the recognition processing unit 104 selects (s / D) pixels that are discretely and non-periodically arranged in the frame Fr(m) to form the pattern Rd#m_1. In addition, the determination basis calculation unit 2116 calculates the determination basis for the recognition result at this point.

[0249] In the next period into which the frame cycle is divided, the recognition processing unit 104 reads pixels according to pattern Rd#m_2 in frame Fr(m), which selects pixels different from pattern Rd#m_1, and performs recognition processing. Also, the judgment basis calculation unit 2116 calculates the judgment basis for the recognition results up to this point.

[0250] Similarly, in the next (m+1)th frame Fr(m+1), in the first period into which the frame period of frame Fr(m+1) is divided, the recognition processing unit 104 reads pixels according to a pattern Rd#(m+1)_1 consisting of multiple pixels that are discretely and non-periodically arranged in frame Fr(m+1) and performs recognition processing. Also, the judgment basis calculation unit 2116 calculates the judgment basis for the recognition results up to this point.

[0251] In the next period, the recognition processing unit 104 reads pixels according to pattern Rd#(m+1)_2, which selects pixels different from those in pattern Rd#(m+1)_1, and performs recognition processing. The determination basis calculation unit 2116 also calculates the determination basis for the recognition results up to this point.

[0252] In the recognition processing unit 104, for example, in the first period into which the frame period of frame Fr(m) is divided, the read determination unit 2114 selects a predetermined number of pixels from all pixels included in frame Fr(m) based on a pseudo-random number to determine pattern Rd#m_1 as a read unit. In the next period, for example, the read determination unit 2114 selects a predetermined number of pixels from all pixels included in frame Fr(m) excluding the pixels selected in pattern Rd#m_1 based on a pseudo-random number to determine pattern Rd#m_2 as a read unit. However, the recognition processing unit 104 may again select a predetermined number of pixels from all pixels included in frame Fr(m) based on a pseudo-random number to determine pattern Rd#m_2 as a read unit.

[0253] The read determination unit 2114 passes read area information, which is information indicating the read unit determined as pattern Rd#m_x and read position information for reading out pixel data of the pattern Rd#m_x, to the read control unit 2102. The read control unit 2102 passes the read area information passed from the read determination unit 2114 to the read unit 2101. The read unit 2101 reads pixel data from the sensor unit 102 in accordance with the read area information passed from the read control unit 2102.

[0254] Here, the information indicating the read unit may be configured, for example, by position information within the frame Fr(m) of each pixel included in the pattern Rd#m_1 (for example, information indicating the line number and pixel position within the line). In this case, the read unit targets the entire frame Fr(m), so the read position information can be omitted. Information indicating the position of a specific pixel within the frame Fr(m) may also be used as the read position information.

[0255] In this manner, in the fifth modified example, the frame readout process is performed using a pattern Rd#m_x consisting of a plurality of pixels that are discretely and non-periodically arranged from all pixels of the frame Fr(m). Therefore, compared to when a periodic pattern is used, it is possible to reduce sampling artifacts. For example, the frame readout process according to the fifth modified example can suppress false detection or non-detection of temporal periodic patterns (e.g., flicker) in the recognition process. Furthermore, the frame readout process can also suppress false detection or non-detection of spatial periodic patterns (e.g., fences or mesh-like structures) in the recognition process.

[0256] Furthermore, with this frame readout process, the amount of pixel data available for recognition processing increases over time. This allows for faster recognition response speed for large objects in frame Fr(m), and a higher frame rate. Furthermore, in line with this faster recognition response speed, the time required to present the basis for the recognition result can be shortened.

[0257] In the above description, the recognition processing unit 104 generates each pattern Rd#m_x each time, but the pattern generation method is not limited to this example. For example, each pattern Rd#m_x may be generated in advance and stored in a memory, and the read determination unit 2114 may read out and use one of the stored patterns Rd#m_x from the memory.

[0258] E-2-6. Variation (6) Next, a sixth modified example will be described. In the sixth modified example, the configuration of the readout unit is changed depending on the result of the recognition process. Fig. 44 illustrates the frame readout process according to the sixth modified example. Here, the description will be given taking as an example the readout unit based on a pattern consisting of a plurality of pixels including non-adjacent pixels, as described with reference to Fig. 38.

[0259] 44, in the mth frame Fr(m), similar to the pattern Pφ#xy shown in FIG. 38, the read determination unit 2114 generates a pattern Pt#xy consisting of a plurality of pixels arranged discretely and periodically in both the line direction and the vertical direction, and sets this as an initial read unit. The read determination unit 2114 passes read area information, in which read position information for reading out the pattern Pt#xy is added to information indicating the read unit determined as the pattern Pt#xy, to the read control unit 2102. The read control unit 2102 passes the read area information passed from the read determination unit 2114 to the read unit 2101. The read unit 2101 reads pixel data from the sensor unit 10 in accordance with the read area information passed from the read control unit 2102.

[0260] As shown in FIG. 44, in frame Fr(m), the recognition processing unit 104 reads out patterns Pt#1-1, Pt#2-1, Pt#3-1 in order, while shifting their position horizontally from the left end by a phase Δφ, performs recognition processing, and calculates the basis for judgment. When the right end of pattern Pt#xy reaches the right end of frame Fr(m), it again reads out patterns Pt#1-2, ... from the right end of frame Fr(m), while shifting their position vertically by a phase Δφ' and moving their position horizontally by a phase Δφ, and performs recognition processing and judges the recognition error.

[0261] The recognition processing unit 104 generates a new pattern Pt'#xy in accordance with the recognition result for frame Fr(m). As an example, it is assumed that the recognition processing unit 104 recognizes a target object (e.g., a person) in the center of frame Fr(m) during the recognition process for frame Fr(m). In the recognition processing unit 104, the read determination unit 2114 generates a pattern Pt'#xy in which reading is concentrated on pixels in the center of frame Fr(m) as a new read unit in accordance with this recognition result.

[0262] The read determination unit 2114 can generate the pattern Pt'#x-1 using fewer pixels than the pattern Pt#xy, and can make the pixel arrangement of the pattern Pt'#xy denser than the pixel arrangement in the pattern Pt#xy.

[0263] The read determination unit 2114 passes readout area information, which is information indicating the read unit determined as the pattern Pt'#xy and readout position information for reading out the pattern Pt'#xy, to the readout control unit 2102. The readout determination unit 2114 then applies this pattern Pt'#xy to the next frame Fr(m+1). The readout control unit 2102 passes the readout area information passed from the read determination unit 2114 to the readout unit 2101. The readout unit 2101 reads pixel data from the sensor unit 102 in accordance with the readout area information passed from the readout control unit 2102.

[0264] 44, in frame Fr(m+1), the recognition processing unit 104 first reads out pattern Pt'#1-1 at the center of frame Fr(m+1), performs recognition processing, and calculates the basis for judgment, then moves the position horizontally, for example, by a phase Δφ, and performs readout using pattern Pt'#2-1, performs recognition processing, and calculates the basis for judgment.Furthermore, the recognition processing unit 104 moves the position of pattern Pt'#1-1 vertically by a phase Δφ', and then reads out patterns Pt'#1-2 and Pt'#2-2 while further moving the position horizontally by a phase Δφ.

[0265] In the sixth modification, a pattern Pt'#xy to be used for reading pixels in the next frame Fr(m+1) is generated based on the recognition result in frame Fr(m) based on the pattern Pt#xy, which is the initial pattern. This enables more accurate recognition processing. Furthermore, by using a new pattern Pt'#xy generated according to the result of the recognition processing to narrow down the recognition processing to the part where the object is recognized, it is possible to reduce the amount of processing in the recognition processing unit 104, save power, and improve the frame rate. Furthermore, it is possible to calculate the basis for judgment of the recognition result and present it in real time.

[0266] Here, another example of the sixth modified example will be described. Fig. 45 illustrates a frame readout process according to this other example of the sixth modified example. In Fig. 45, pattern Cc#x is an annular readout unit whose radius changes over time.

[0267] In the example shown in Figure 45, in the first period into which the frame period of frame Fr(m) is divided, frame readout is performed using pattern Cc#1 with a small radius, in the next period frame readout is performed using pattern Cc#2 with a larger radius than pattern Cc#1, and in the period after that frame readout is performed using pattern Cc#3 with an even larger radius than pattern Cc#2.

[0268] For example, as shown in Fig. 44, in frame Fr(m), the recognition processing unit 104 reads out patterns Pt#1-1, Pt#2-1, and Pt#3-1 in this order while moving the position horizontally from the left edge by a phase Δφ, and performs recognition processing and calculation processing of the judgment basis. Then, when the right edge of pattern Pt#xy reaches the right edge of frame Fr(m), the recognition processing unit 104 reads out patterns Pt#1-2, ... again from the right edge of frame Fr(m) while moving the position vertically by a phase Δφ' and moving the position horizontally by a phase Δφ, and performs recognition processing and calculation processing of the judgment basis.

[0269] Here, the recognition processing unit 104 generates a new pattern Cc#1 having a ring shape according to the recognition result or the basis for judgment in frame Fr(m). As an example, it is assumed that the recognition processing unit 104 recognizes a target object (e.g., a person) in the center of frame Fr(m) during the recognition process on frame Fr(m) and calculates the basis for judgment. In the recognition processing unit 104, the read determination unit 2114 generates patterns Cc#1, Cc#2, ... as shown in FIG. 45 according to the recognition result or the basis for judgment, reads out the patterns Cc#1, Cc#2, ..., and performs the recognition process and the calculation process for the basis for judgment.

[0270] In Figure 45, the radius of pattern Cc#m is increased over time, but the method of generating a read pattern having a circular shape is not limited to this example, and the radius of pattern Cc#m may be decreased over time.

[0271] As yet another example of the sixth modification, the density of pixels in the readout pattern may be changed. In addition, in the pattern Cc#m shown in Fig. 45, the size is changed from the center to the periphery of the annular shape or from the periphery to the center, but the method for generating an annular readout pattern is not limited to this example.

[0272] E-2-7. Variation (7) Next, a seventh modified example will be described. In the above-described embodiment of the present disclosure and the first to fourth modified examples, the line, area, and pattern for reading out pixels are moved according to the order of coordinates within a frame (line number, order of pixels within a line, etc.). In contrast, in the seventh modified example, the line, area, and pattern for reading out pixels are set so that the pixels within a frame can be read out more uniformly in a short time.

[0273] Fig. 46 illustrates a first example of a frame readout process according to the seventh modification. For ease of explanation, Fig. 46 assumes that frame Fr(m) includes eight lines: lines L#1, L#2, L#3, L#4, L#5, L#6, L#7, and L#8.

[0274] The readout process shown in Fig. 46(a) corresponds to the readout process shown in Fig. 24, and reads pixel data for frame Fr(m) line by line in the order of lines L#1, L#2, ..., L#8. In the example shown in Fig. 46(a), there is a large delay from when readout of frame Fr(m) starts until the pixel data for the bottom part of frame Fr(m) is obtained.

[0275] On the other hand, Fig. 46(b) shows an example of a readout process according to a first example of the seventh modified example. In the example shown in Fig. 46(b), the readout unit is a line, as in Fig. 46(a) described above, but in frame Fr(m), for each odd-numbered line and each even-numbered line, two lines are paired together, with the inter-line distance being half the number of lines in frame Fr(m). Of these pairs, the pairs with odd-numbered line numbers are read out sequentially, and then the pairs with even-numbered line numbers are read out sequentially.

[0276] 46(b), in the first half of the frame period, for example, among the lines L#x included in frame Fr(m), the read order of odd-numbered lines L#1, L#3, L#5, and L#7 is reversed, and lines L#1, L#5, L#3, L#7 are read in this order. Similarly, in the second half of the frame period, among the lines L#x included in frame Fr(m), the read order of even-numbered lines L#2, L#4, L#6, and L#8 is reversed, and lines L#2, L#6, L#4, L#8 are read in this order. Such control of the read order of lines L#x can be realized by the read determination unit 2114 sequentially setting the read position information.

[0277] By determining the read order of each line in a frame as shown in Figure 46(b), it is possible to reduce the delay from when readout of frame Fr(m) begins until pixel data at the bottom of frame Fr(m) is obtained, compared to the example shown in Figure 46(a). Also, in the seventh modification, it is possible to increase the frame rate by speeding up the recognition response speed for large objects within a frame. Furthermore, in line with the increased recognition response speed, it is possible to reduce the time until the basis for determining the recognition result is presented.

[0278] 46(b) is an example, and the readout region can be set so as to facilitate recognition of an intended object. For example, the readout determination unit 2114 can set a region in the frame where recognition processing is to be preferentially executed based on external information provided from outside the imaging device 100, and determine readout position information so as to preferentially execute readout of the readout region for this region. The readout determination unit 2114 can also set a region in the frame where recognition processing is to be preferentially executed according to a scene captured in the frame.

[0279] Furthermore, in the first example of the seventh modified example, similarly to the above-described embodiment of the present disclosure, if a valid recognition result is obtained during the reading of each line for a frame, the line reading and recognition process can be terminated. This enables faster and more energy-efficient recognition by reducing the amount of processing in the recognition process, and also shortens the time required for the recognition process and the time required for presenting the basis for the judgment.

[0280] Next, a second example of the seventh modified example will be described. In the first example of the seventh modified example described above, one line is used as a readout unit, but this is not limited to this example. In the second example of the seventh modified example, two non-adjacent lines are used as a readout unit.

[0281] Fig. 47 illustrates a frame readout process according to a second example of the seventh modified example. In the example shown in Fig. 47, in the frame Fr(m) described in Fig. 46(b), each line with an odd line number is paired with each line with an even line number, and two lines separated by a distance of 1 / 2 the number of lines are used as readout units. Specifically, the pair of lines L#1 and L#5, the pair of lines L#3 and L#7, the pair of lines L#2 and L#6, and the pair of lines L#4 and L#8 are each used as readout units. Among these pairs, the pairs with odd line numbers are read out sequentially, followed by the pairs with even line numbers.

[0282] In the second example of the seventh variant, the reading unit includes two lines, so it is possible to reduce the time required for the recognition process and the time required to present the basis for judgment compared to the seventh variant described above.

[0283] Next, a third example of the seventh modified example will be described. In this third example, when the read unit according to the third modified example of the embodiment of the present disclosure (see FIG. 34) is an area of a predetermined size within a frame, a read area for reading the read unit is set so that pixels within the frame can be read more uniformly in a short time.

[0284] FIG. 48 illustrates a frame readout process according to a third example of the seventh modified example. In the example shown in FIG. 48, the positions of each area Ar#xy shown in FIG. 34 are discretely specified in the frame Fr(m), and the frame Fr(m) is readout. As an example, after area Ar#1-1 in the upper left corner of frame Fr(m) is readout and a recognition process and a calculation process of the judgment basis are performed, area Ar#3-1 in frame Fr(m) that includes the same line as area Ar#1-1 and is located in the center of the frame Fr(m) in the line direction is readout and a recognition process and a calculation process of the judgment basis are performed. Next, area Ar#1-3 in the upper left corner of the lower half of the frame Fr(m) is readout and a recognition process and a calculation process of the judgment basis are performed. Finally, area Ar#3-3 in frame Fr(m) that includes the same line as area Ar#1-3 and is located in the center of the frame Fr(m) in the line direction is readout and a recognition process and a calculation process of the judgment basis are performed. Similarly, areas Ar#2-2 and Ar#4-2, and areas Ar#2-4 and Ar#4-4 are subjected to reading, recognition processing, and calculation processing of the basis for judgment.

[0285] By determining the read order in this manner, it is possible to reduce the delay from when readout of frame Fr(m) starts from the left edge of frame Fr(m) until pixel data on the bottom and right edge of frame Fr(m) is obtained, compared to the example shown in FIG. 34. Furthermore, in this third example, the recognition response speed for large objects within a frame can be increased, making it possible to increase the frame rate. Furthermore, in line with the increased recognition response speed, it is possible to reduce the time until the basis for determining the recognition result is presented.

[0286] Also in this third example, similar to the above-described embodiment of the present disclosure, if a valid recognition result is obtained during the reading of each area Ar#xy for a frame, the reading of area Ar#xy, the recognition process, and the calculation process of the judgment basis can be terminated. This enables faster and more energy-efficient recognition by reducing the amount of processing in the recognition process, and also shortens the time required for the recognition process and the time required for presenting the judgment basis.

[0287] Next, a fourth example of the seventh modified example will be described. In this fourth example, a readout area for reading out the readout unit is set so that the pixels in the frame can be read out more uniformly in a short time in the example of the fourth modified example of the embodiment of the present disclosure (see FIG. 38) in which the readout unit is a pattern made up of a plurality of pixels that are discretely and periodically arranged in both the line direction and the vertical direction.

[0288] Fig. 49 illustrates a frame readout process according to a fourth example of the seventh modified example of the second embodiment. In the example shown in Fig. 49, the pattern Pφ#z has the same configuration as the pattern Pφ#xy shown in Fig. 38, and the position of the pattern Pφ#z is discretely specified in the frame Fr(m) to read out the frame Fr(m).

[0289] As an example, the recognition processing unit 104 starts from the upper left corner of frame Fr(m) and reads pattern Pφ#1 located at the upper left corner, performs recognition processing, and calculates the basis for judgment. Next, it reads pattern Pφ#2, which is shifted by half the spacing between pixels in pattern Pφ#1 in both the line and vertical directions, and performs recognition processing and calculates the basis for judgment. Next, it reads pattern Pφ#3, which is shifted by half the spacing in the line direction relative to the position of pattern Pφ#1, and performs recognition processing and calculates the basis for judgment. Next, it reads pattern Pφ#4, which is shifted by half the spacing in the vertical direction relative to the position of pattern Pφ#1, and performs recognition processing and calculates the basis for judgment. These readouts, recognition processing, and calculations of the basis for judgment for patterns Pφ#1 to Pφ#4 are repeatedly performed while shifting the position of pattern Pφ#1, for example, by one pixel in the line direction, and then by further shifting it by one pixel in the vertical direction.

[0290] By determining the read order in this manner, it is possible to reduce the delay from when readout of frame Fr(m) starts from the left edge of frame Fr(m) until pixel data on the bottom and right edge of frame Fr(m) is obtained, compared to the example shown in FIG. 38. Furthermore, in this fourth example, the recognition response speed for large objects within a frame can be increased, making it possible to increase the frame rate. Furthermore, in line with the increased recognition response speed, it is possible to reduce the time until the basis for determining the recognition result is presented.

[0291] Also, in this fourth example, similar to the above-described embodiment of the present disclosure, if a valid recognition result is obtained during the reading of each pattern Pφ#z for a frame, the reading of the pattern Pφ#z and the recognition process can be terminated and the calculation process of the judgment basis can be started. This makes it possible to speed up and save power by reducing the processing load in the recognition process, and also shorten the time required for the recognition process and the presentation of the judgment basis.

[0292] E-2-8. Variation (8) Next, an eighth modification will be described. In the eighth modification of the embodiment of the present disclosure, a readout region to be read out next is determined based on the feature amount generated in the feature amount accumulation control unit 2112.

[0293] 50 shows an example of the functional configuration of the recognition processing unit 104 according to the eighth modified example. The feature amount accumulation control unit 2112 integrates the feature amount passed from the feature amount calculation unit 2111 with the feature amount accumulated in the feature amount accumulation unit 2113, and passes the integrated feature amount together with readout information to the readout determination unit 2114. The readout determination unit 2114 generates readout region information based on the feature amount and readout information passed from the feature amount accumulation control unit 2112. The readout determination unit 2114 passes the generated readout region information to the readout unit 2101 in the sensor control unit 103.

[0294] Fig. 51 shows in the form of a flowchart the procedure for the recognition and determination basis calculation process corresponding to the readout of pixel data in a readout unit based on image feature amounts, according to the eighth modified example. The illustrated processing procedure corresponds to the readout of pixel data in a readout unit (for example, one line) from a frame, for example. However, Fig. 51 explains the procedure for the recognition and determination basis calculation process when the readout unit is a line. For example, the readout area information can use a line number indicating the line to be read.

[0295] First, the recognition processing unit 104 reads line data from a line indicated by the read line of the frame (step S5101). Specifically, the read determination unit 2114 passes the line number of the line to be read next to the sensor control unit 103. In the sensor control unit 103, the read unit 2101 reads pixel data of the line indicated by the line number from the sensor unit 102 as line data in accordance with the passed line number. The read unit 2101 passes the line data read from the sensor unit 102 to the feature amount calculation unit 2111. The read unit 2101 also passes read area information (e.g., line number) indicating the area from which pixel data has been read to the feature amount calculation unit 2111.

[0296] Next, the feature amount calculation unit 2111 calculates the feature amount of the image based on the line data passed from the reading unit 2101 (step S5102). Furthermore, the feature amount calculation unit 2111 acquires the feature amount accumulated in the feature amount accumulation unit 2113 from the feature amount accumulation control unit 2112 (step S5103), and integrates the feature amount calculated in step S5102 with the feature amount acquired from the feature amount accumulation control unit 2112 in step S5103 (step S5104). The integrated feature amount is passed to the feature amount accumulation control unit 2112. The feature amount accumulation control unit 2112 accumulates the integrated feature amount in the feature amount accumulation unit 2113 (step S5105).

[0297] Note that if the series of processes in steps S5101 to S5104 are processes for the first line of a frame and the feature amount storage unit 2113 has been initialized, for example, then steps S5103 and S5104 can be omitted. Furthermore, the process in step S5105 in this case is to store the line feature amount calculated based on the first line in the feature amount storage unit 2113.

[0298] The feature amount accumulation control unit 2112 also passes the integrated feature amount passed from the feature amount calculation unit 2111 to the recognition process execution unit 2115. The recognition process execution unit 2115 executes recognition process using the integrated feature amount passed from the feature amount accumulation control unit 2112 (step S5106). The recognition process execution unit 2115 outputs the recognition result of the recognition process to the output control unit 107 (step S5107).

[0299] The recognition process execution unit 2115 also outputs the recognition result from the recognition process to the determination basis calculation unit 2116. The determination basis calculation unit 2116 calculates the determination basis for the recognition result passed from the recognition process execution unit 2115 (step S5108). The determination basis calculation unit 2116 uses, for example, the Grad-Cam algorithm to estimate locations on the line data that contributed to the recognition result from the feature amounts of the line data calculated in step S5102, or estimates locations within the image range that has been read so far that contributed to the recognition result based on the feature amounts integrated in step S5104. Then, the determination basis calculation unit 2116 outputs the calculated determination basis to the output control unit 107 (step S5109).

[0300] Next, in the recognition processing unit 104, the read determination unit 2114 determines the read line to be read next according to the integrated feature and read information passed from the feature accumulation control unit 2112 (step S5110). For example, when the feature accumulation control unit 2112 receives the integrated feature and read area information from the feature calculation unit 2111, it determines the read line to be read next according to the read pattern (line by line in this example) corresponding to the integrated feature. The processing from step S5101 is executed again for the read line determined in step S5110.

[0301] (First process) Next, a first process according to the eighth modified example will be described with reference to Fig. 52. Fig. 52 illustrates the first process procedure according to the eighth modified example. However, the read unit is a line, and imaging is performed using a rolling shutter method. In addition, the memory 105 pre-stores a learning model trained using predetermined learning data to perform image recognition processing such as identifying numbers as a program, and the recognition processing unit 104 is capable of identifying numbers by reading and executing this program from the memory 105.

[0302] First, the imaging device 100 starts capturing an image of a target image (handwritten numeral "8") to be recognized (step S5201).

[0303] When imaging starts, the sensor control unit 103 sequentially reads out the frame line by line from the top to the bottom in accordance with the readout area information passed from the recognition processing unit 104 (step S5202).

[0304] When the lines have been read up to a certain position, the recognition processing unit 104 identifies the number "8" or "9" from the image formed by the read lines (step S5203). In the recognition processing unit 104, the read determination unit 2114 generates read area information specifying a line L#m that is predicted to enable identification of whether the object identified in step S5203 is the number "8" or "9" based on the integrated feature amount passed from the feature amount accumulation control unit 2112, and passes this information to the read unit 2101. Then, the recognition processing unit 104 executes recognition processing and calculation processing of the determination basis based on the pixel data of the line L#m read by the read unit 2101 (step S5204).

[0305] If the object is confirmed in step S5204, the recognition processing unit 104 can further calculate the basis for the determination based on the Grad-CAM algorithm or the like, and then terminate the recognition processing. This makes it possible to achieve higher speeds and power savings by reducing the processing amount in the recognition processing in the image capture device 100, and also makes it possible to present the basis for the determination in real time.

[0306] (Second process) Next, a second process according to the eighth modified example will be described. Fig. 53 illustrates the second process according to the eighth modified example. As in the first process, a program of a learning model trained to identify numbers is stored in advance in memory 105, and the recognition processing unit 104 is capable of identifying numbers by reading and executing this program from memory 105. The second process shown in Fig. 53 uses the handwritten number "8" as the target image, as in the process shown in Fig. 52, but reads frames line by line while thinning out lines according to the feature amount of the image.

[0307] First, the imaging device 100 starts capturing an image of a target image (handwritten numeral "8") to be recognized (step S5301).

[0308] When imaging begins, the sensor control unit 103 reads the frame line by line while thinning out the lines from the top to the bottom of the frame in accordance with the readout area information passed from the recognition processing unit 104 (step S5302). In the example shown in Fig. 53, the sensor control unit 103 first reads line L#1 at the top of the frame in accordance with the readout area information, and then reads line L#p obtained by thinning out a predetermined number of lines. The recognition processing unit 104 performs recognition processing and calculation processing of the basis for judgment on the line data of lines L#1 and L#p each time it is read.

[0309] Further, reading is performed line by line by thinning, and the recognition processing unit 104 performs recognition processing on the line data read from line L#q, and it is assumed that the number "8" or "0" is recognized as a result (step S5303).

[0310] Here, the read determination unit 2114 generates read area information that specifies a line L#r that is predicted to enable identification of whether the object identified in step S5303 is the number "8" or "0," based on the integrated feature passed from the feature accumulation control unit 2112, and passes this information to the read unit 2101. In this case, the position of line L#r may be on the upper or lower end side of the frame relative to line L#q.

[0311] The recognition processing unit 104 executes recognition processing and calculation processing of the basis for judgment based on the pixel data read out from the line L#r by the readout unit 2101 (step S5304).

[0312] In the second process illustrated in Figure 53, lines of a frame are read out while thinning out lines according to the features of the input image, which makes it possible to further shorten the recognition process and save power, as well as present the basis for judgment in real time.

[0313] Fig. 54 shows in more detail an example of processing by the recognition processing unit 104 according to the eighth modified example. In this figure, a read determination unit 2114 is added to the configuration shown in Fig. 22. A feature amount 2213 relating to the internal state updated by the internal state update process 2212 is input to the read determination unit 2114.

[0314] The read determination unit 2114 generates read area information (e.g., a line number) indicating the read area to be read next based on the input feature amount 2213 related to the internal state, and outputs the information to the read unit 2101. The read determination unit 2114 executes a program of a learning model that has been trained in advance to determine the next read area. The learning model is trained using training data based on, for example, expected read patterns and recognition targets.

[0315] Note that, in the imaging device 100 according to the above-described embodiment of the present disclosure and its first to eighth modified examples, the recognition processing unit 104 performs recognition processing for each readout unit, but the present disclosure is not limited to this example. For example, it may be possible to switch between recognition processing for each readout unit and normal recognition processing (reading the entire frame and performing recognition processing based on pixel data of pixels read from the entire frame). That is, the normal recognition processing is performed based on the pixels of the entire frame, and therefore it is possible to obtain more accurate recognition results, while the recognition processing for each readout unit is possible to perform high-speed and power-saving recognition processing and present the basis for judgment in real time.

[0316] For example, while performing the recognition process for each read unit, a normal recognition process may be started at regular intervals to ensure high recognition accuracy.Furthermore, while performing the recognition process for each read unit, a normal recognition process may be started when a predetermined event such as an emergency occurs to improve the stability of recognition.

[0317] When switching from the recognition process for each read unit to the normal recognition process, there is a problem that the normal recognition process is less rapid than the recognition process for each read unit. Therefore, in the normal recognition process, the operating clock of the device (the processor that executes the recognition process (the program of the learned learning model)) may be switched to a higher speed mode.

[0318] Furthermore, there is a problem that the reliability of the recognition process for each read unit is low. Therefore, when the reliability of the recognition process for each read unit decreases or when the basis for the judgment presented for the recognition result is incomprehensible, it is possible to switch to the normal recognition process. After that, when the reliability of the recognition process is restored to a high level, it is possible to return to the recognition process for each read unit.

[0319] F. Second Example Next, a second embodiment of the present disclosure will be described. In the second embodiment, when reading out a frame, parameters such as a readout unit, a readout order within a frame based on the readout unit, and a readout area are adaptively set.

[0320] 55 shows an example of the functional configuration of the recognition processing unit 104 according to the second embodiment. The recognition processing unit 104 shown in the figure additionally includes an external information acquisition unit 5501. The read determination unit 2114 receives pixel data from the read unit 2101 and recognition information from the recognition processing execution unit 2115.

[0321] The external information acquisition unit 5501 acquires external information generated outside the image capture device 100 and passes the acquired external information to the readout determination unit 2114. For example, the external information acquisition unit 5501 is configured with an interface that transmits and receives signals in a predetermined format. For example, if the image capture device 100 is for vehicle use, the external information can include vehicle information and surrounding environment information. Examples of the vehicle information include steering information and speed information. Examples of the environment information include ambient brightness. In the following, unless otherwise specified, the image capture device 100 is used for vehicle use, and the external information is vehicle information acquired from the vehicle in which the image capture device 100 is installed.

[0322] 56 shows a detailed functional configuration example of the read determination unit 2114 according to the second embodiment. In the figure, the read determination unit 2114 includes a read unit pattern selection unit 5610, a read order pattern selection unit 5620, and a read determination processing unit 5630. The read unit pattern selection unit 5610 includes a read unit pattern DB (database) 5611 in which a plurality of different read patterns are stored in advance. Furthermore, the read order pattern selection unit 5620 includes a read order pattern DB 5621 in which a plurality of different read order patterns are stored in advance.

[0323] The read determination unit 2114 sets priorities for each read unit pattern stored in the read unit pattern DB5611 and each read order pattern stored in the read order pattern DB5621 based on at least one of the passed recognition information, pixel data, vehicle information and environmental information, and clarity of the judgment basis.

[0324] The read unit pattern selection unit 5610 selects the read unit pattern set to the highest priority from among the read unit patterns stored in the read unit pattern DB 5611. The read unit pattern selection unit 5610 passes the read unit pattern selected from the read unit pattern DB 5611 to the read determination processing unit 5630. Similarly, the read order pattern selection unit 5620 selects the read order pattern set to the highest priority from among the read order patterns stored in the read order pattern DB 5621. The read order pattern selection unit 5620 passes the read order pattern selected from the read order pattern DB 5621 to the read determination processing unit 5630.

[0325] The read determination processing unit 5630 determines the read area to be read next from the frame based on the read information passed from the feature accumulation control unit 121, the read unit pattern passed from the read unit pattern selection unit 5610, and the read order pattern passed from the read order pattern selection unit 5620, and passes read area information indicating the determined read area to the reading unit 2101.

[0326] Next, a method for setting a read unit pattern and a read order pattern by the read determination unit 2114 shown in FIG. 56 will be described.

[0327] Fig. 57 shows an example of a read unit pattern applicable to the second embodiment. In the example shown in the figure, five read unit patterns 5701, 5702, 5703, 5704, and 5705 are shown. Each of the read unit patterns 5701 to 5705 is stored in advance in the read unit pattern DB 5611 shown in Fig. 56.

[0328] Read unit pattern 5701 is a read unit pattern in which a line is used as a read unit and readout is performed line by line in frame 5700. Read unit pattern 5702 is a read pattern in which an area of a predetermined size is used as a readout unit in frame 5700 and readout is performed area by area in frame 5700.

[0329] Read unit pattern 5703 is a read unit pattern that uses a pixel set of multiple periodically arranged pixels, including non-adjacent pixels, as a read unit, and performs readout for each of these multiple pixels in frame 5700. Read unit pattern 5704 is a read unit pattern that uses a plurality of discretely and non-periodically arranged pixels (random pattern) as a readout unit, and performs readout while updating the random pattern in frame 5700. These read unit patterns 5703 and 5704 make it possible to sample pixels more uniformly from frame 5700.

[0330] Furthermore, the read unit pattern 5705 is a read unit pattern that is adaptively generated based on the recognition information.

[0331] Note that the read unit applicable to the read unit pattern according to the second embodiment is not limited to the example shown in Fig. 57. For example, each read unit described in the first embodiment and each modified example thereof can be applied as the read unit pattern according to the second embodiment.

[0332] Figures 58 to 60 show examples of readout order patterns applicable to the second embodiment. Figure 58 shows an example of a readout order pattern when the readout unit is a line, Figure 59 shows an example of a readout order pattern when the readout unit is an area, and Figure 60 shows an example of a readout order pattern when the readout unit is the above-mentioned pixel set. In each of Figures 58 to 60, readout order patterns 5801, 5901, and 6001 on the left side show examples of readout order patterns in which readout is performed sequentially in line order or pixel order, respectively.

[0333] A readout order pattern 5801 in Fig. 58 is an example in which readout is performed line-sequentially from the top to the bottom of a frame 5800. A readout order pattern 5901 in Fig. 59 and a readout order pattern 6001 in Fig. 60 are examples in which readout is performed area-by-area or pixel set-by-pixel set along the line direction from the upper left corner of each of frames 5900 and 6000, respectively, and this line-direction readout is repeated in the vertical direction of each of frames 5900 and 6000. These readout order patterns 5801, 5901, and 6001 are called forward readout order patterns.

[0334] On the other hand, readout order pattern 5802 in Fig. 58 is an example in which readout is performed line-sequentially from the bottom edge toward the top edge of frame 5800. Readout order patterns 5902 and 6002 in Fig. 59 and 60 are examples in which readout is performed area-by-area or pixel set-by-pixel set along the line direction from the bottom right corner of frame 5900 and frame 6000, respectively, and this line-direction readout is repeated in the vertical direction of frame 5700. These readout order patterns 5802, 5902, and 6002 are called reverse readout order patterns.

[0335] Furthermore, readout order pattern 5803 in Figure 58 is an example in which reading is performed from the top to the bottom of frame 5800 while thinning out lines. Readout order patterns 5903 and 6003 in Figures 59 and 60 are examples in which areas are read out at discrete positions and in discrete readout orders within frames 5900 and 6000, respectively. In readout order pattern 5903, when a readout unit consists of, for example, four pixels, each pixel is read out in the order indicated by the arrows in each figure. In readout order pattern 6003, for example, as in the area indicated by reference number 6004 in Figure 60, each pixel is read out while the pixel serving as the reference for the pattern is moved to a discrete position in an order different from the order of pixel positions in the line and column directions.

[0336] The read order patterns 5801 to 5803, read order patterns 5901 to 5903, and read order patterns 6001 to 6003 described with reference to FIGS. 58 to 60 are stored in advance in the read order pattern DB 5621 shown in FIG.

[0337] How to set the read unit pattern: An example of a method for setting a read unit pattern according to the second embodiment will be specifically described with reference to FIGS.

[0338] First, a method for setting a read unit pattern based on image information (pixel data) will be described. The read determination unit 2114 detects noise contained in the pixel data passed from the read unit 2101. Here, a group of pixels arranged closely together has higher resistance to noise than a group of individual pixels arranged scatteredly. Therefore, when the pixel data passed from the read unit 2101 contains a predetermined level of noise or more, the read determination unit 2114 sets the priority of read unit pattern 5701 or 5702, of the read unit patterns 5701 to 5705 stored in the read unit pattern DB 5611, to be higher than the priority of the other read unit patterns.

[0339] Next, a method for setting a readout unit pattern based on the recognition information will be described. A first setting method is applied when many objects larger than a predetermined size are recognized in frame 5700 based on the recognition information passed from the recognition process execution unit 2115. In this case, the read determination unit 2114 sets the priority of readout unit pattern 5703 or 5704, among readout unit patterns 5701 to 5705 stored in readout unit pattern DB 5611, to be higher than the priority of the other readout unit patterns. This is because uniform sampling of the entire frame 5700 makes it possible to further improve the speed of reporting.

[0340] The second setting method is applied, for example, when flicker is detected in an image captured based on pixel data. In this case, the read determination unit 2114 sets the priority of the read unit pattern 5704, among the read unit patterns 5701 to 5705 stored in the read unit pattern DB 5611, to be higher than the priorities of the other read unit patterns. This is because, for flicker, artifacts caused by flicker can be suppressed by sampling the entire frame 5700 using a random pattern.

[0341] The third setting method is applied when a configuration that is considered to be capable of executing recognition processing more efficiently is generated in the case where the configuration of the readout unit is adaptively changed based on the recognition information. In this case, the read determination unit 2114 sets the priority of the readout unit pattern 5705, among the readout unit patterns 5701 to 5705 stored in the readout unit pattern DB 5611, to be higher than the priorities of the other readout unit patterns.

[0342] Next, a description will be given of a method for setting a read unit pattern based on external information acquired by the external information acquisition unit 5501. A first setting method is applied when the vehicle on which the imaging device 1 is mounted turns to the left or right based on the external information. In this case, the read determination unit 2114 sets the priority of the read unit pattern 5701 or 5702, among the read unit patterns 5701 to 5705 stored in the read unit pattern DB 5611, to be higher than the priority of the other read unit patterns.

[0343] Here, in this first setting method, the read determination unit 2114 sets the read unit pattern 5701 to be the column direction among the row and column directions in the pixel array unit 601, and to read out sequentially in the column direction of the frame 5700. Also, the read unit pattern 5702 sets the area to be read out along the column direction, and to repeat this in the line direction.

[0344] When the vehicle turns left, the read determination unit 2114 sets the read determination processing unit 5630 to start column-sequential readout or readout along the column direction of the area from the left end side of the frame 5700. On the other hand, when the vehicle turns right, the read determination processing unit 2114 sets the read determination processing unit 5630 to start column-sequential readout or readout along the column direction of the area from the right end side of the frame 5700.

[0345] When the vehicle equipped with the imaging device 100 is moving straight, the read determination unit 2114 performs normal line-by-line reading or reading along the line direction of the area, and when the vehicle turns left or right, it can initialize the features stored in the feature storage unit 2113, for example, and resume the read process by reading column-by-column as described above or reading the area along the column direction.

[0346] The second method of setting the readout unit pattern based on external information is applied, for example, when the vehicle equipped with the imaging device 100 is traveling on a highway. In this case, the readout determination unit 2114 sets the priority of the readout unit pattern 5701 or 5702, among the readout unit patterns 5701 to 5705 stored in the readout unit pattern DB 5611, to be higher than the priority of the other readout unit patterns. When traveling on a highway, it is considered important to recognize objects that are distant small objects. Therefore, by reading frames 5700 in order from the top, it is possible to further improve the speed at which distant small objects are detected.

[0347] How to set the reading order pattern: An example of a method for setting a readout order pattern according to the second embodiment will be specifically described with reference to FIGS.

[0348] First, a method for setting a readout order pattern based on image information (pixel data) will be described. The readout determination unit 2114 detects noise contained in the pixel data passed from the readout unit 2101. Here, the smaller the change in the area to be recognized, the less the recognition process is affected by noise, making the recognition process easier. Therefore, when the pixel data passed from the readout unit 2101 contains a predetermined amount of noise or more, the readout determination unit 2114 sets the priority of any one of readout order patterns 5801, 5901, and 6001, among readout order patterns 5801-5803, 5901-5903, and 6001-6003 stored in the readout order pattern DB 5621, to be higher than the priority of the other readout order patterns. However, the priority of any one of readout order patterns 5802, 5902, and 6002 may be set to be higher than the priority of the other readout order patterns.

[0349] The priority of which of the read order patterns 5801, 5901 and 6001 and the read order patterns 5802, 5902 and 6002 is set higher can be determined based on, for example, which of the read unit patterns 5701 to 5705 is set higher in the read unit pattern selection unit 5610, and whether the readout is performed from the upper end or the lower end of the frame 5700.

[0350] Next, a method for setting a read order pattern based on the recognition information will be described. When a large number of objects larger than a predetermined size are recognized in frame 5700 based on the recognition information passed from the recognition process execution unit 2115, the read determination unit 2114 sets the priority of any of read order patterns 5803, 5903, and 6003, among read order patterns 5801 to 5803, 5901 to 5903, and 6001 to 6003 stored in read order pattern DB 5621, to be higher than the priority of the other read order patterns. This is because uniform sampling can improve the speed of reporting rather than sequentially reading out the entire frames 5800, 5900, and 6000.

[0351] Next, a method for setting a readout order pattern based on external information will be described. A first setting method is an example in which the vehicle on which the imaging device 1 is mounted turns to the left or right based on external information. In this case, the read determination unit 2114 sets the priority of one of the readout order patterns 5801, 5901, and 6001 among the readout order patterns 5801 to 5803, 5901 to 5903, and 6001 to 6003 stored in the readout order pattern DB 5621 higher than the priority of the other readout unit patterns.

[0352] Here, in this first setting method, the read determination unit 2114 sets the read order pattern 5801 so that the column direction of the row and column directions in the pixel array unit 601 is the read unit, and readout is performed column by column in the line direction of the frame 5700. Also, for the read order pattern 5901, the setting is such that an area is read out along the column direction, and this is repeated in the line direction. Furthermore, for the read order pattern 6001, the setting is such that a pixel set is read out along the column direction, and this is repeated in the line direction.

[0353] When the vehicle turns left, the read determination unit 2114 sets the read determination processing unit 5630 to perform column-sequential readout or readout along the column direction of the area starting from the left end side of the frame 5700. On the other hand, when the vehicle turns right, the read determination processing unit 2114 sets the read determination processing unit 5630 to perform column-sequential readout or readout along the column direction of the area starting from the right end side of the frame 5700.

[0354] When the vehicle is moving straight, the read determination unit 2114 performs reading in the usual line unit or along the line direction of the area, and when the vehicle turns left or right, it can initialize the features accumulated in the feature accumulation unit 2113, for example, and resume the read process by reading in the column order as described above, or by reading the area along the column direction.

[0355] The second setting method of the readout order pattern based on external information is applied when the vehicle equipped with the imaging device 1 is traveling on a highway based on the external information. In this case, the readout determination unit 2114 sets the priority of the readout order patterns 5801, 5901, and 6001 among the readout order patterns 5801-5803, 5901-5903, and 6001-6003 stored in the readout order pattern DB 5621 higher than the priority of the other readout order patterns. When the vehicle is traveling on a highway, it is considered important to recognize objects that are distant small objects. Therefore, by reading out the frames 5800, 5900, and 6000 in order from the top end, it is possible to improve the speed of detecting objects that are distant small objects.

[0356] Here, as described above, when the priorities of the read unit patterns and the read order patterns are set based on a plurality of different information (image information, recognition information, external information), there is a possibility that different read unit patterns or different read order patterns may collide with each other. In order to avoid this collision, for example, it is conceivable to make the priorities set based on the respective information different in advance.

[0357] F-1. First Modification of the Second Embodiment Next, a first modified example of the second embodiment will be described. In the first modified example of the second embodiment, a readout region is adaptively set when performing frame readout. The first modified example of the second embodiment is realized using the recognition processing unit 104 shown in FIG. 55.

[0358] A method for adaptively setting a readout area according to a first modified example of the second embodiment will be described below, assuming that the imaging device 100 is used as an in-vehicle device.

[0359] F-1-1. Example of setting the read area based on recognition information First, a first setting method for adaptively setting a readout area based on recognition information will be described. In the first setting method, the read determination unit 2114 adaptively sets an area within a frame using an area or class detected by the recognition process of the recognition process execution unit 2115, and limits the readout area to be read next. This first setting method will be described with reference to Figs. 61 and 62.

[0360] 61, for a frame 6100, line reading is performed line by line, with the read unit being a line, in line order and with line thinning. In the example shown in Fig. 61, the recognition process execution unit 2115 executes recognition process on the entire frame 6100 based on pixel data read by line reading. As a result, the recognition process execution unit 2115 detects a specific object (a person in the example shown in Fig. 61) in an area 6101 within the frame 6100. The recognition process execution unit 2115 passes recognition information indicating this recognition result to the read determination unit 2114.

[0361] The read determination unit 2114 determines the read area to be read next based on the recognition information passed from the recognition process execution unit 2115. For example, the read determination unit 2114 determines the area including the recognized area 6101 and the periphery of the area 6101 as the read area to be read next. The read determination unit 2114 passes read area information indicating the read area based on the area 6102 to the read unit 2101.

[0362] The read unit 2101 performs frame readout without, for example, line thinning in accordance with the readout region information passed from the read determination unit 2114, and passes the readout pixel data to the recognition processing unit 104. FIG. 62 shows an example of an image readout in accordance with the readout region. In the example shown in FIG. 62, in frame 6200, which is the frame next to frame 6100, pixel data of region 6102 indicated in the readout region information is acquired, and the area outside region 6102 is ignored. In the recognition processing unit 104, the recognition processing execution unit 2115 performs recognition processing on region 6102. As a result, the recognition processing execution unit 2115 recognizes that the person detected in region 6101 is a pedestrian. Furthermore, the determination basis calculation unit 2116 calculates the basis for recognizing a pedestrian in region 6102.

[0363] Furthermore, in this first determination method, it is also possible to limit the readout region to be read next depending on the type of recognized object. For example, if the object recognized in frame 6100 is a traffic light, the readout determination unit 2114 can limit the readout region to be read in the next frame 6200 to the lamp portion of the traffic light. Furthermore, if the object recognized in frame 6100 is a traffic light, the readout determination unit 2114 can change the frame readout method to a readout method that reduces the influence of flicker, and perform reading in the next frame 6200. As an example of a readout method that reduces the influence of flicker, the pattern Rd#m_x according to the fifth modified example of the second embodiment described above can be applied.

[0364] Next, a second setting method for adaptively setting a readout area based on recognition information will be described. In the second setting method, the readout determination unit 2114 limits the readout area to be read next by using recognition information during the recognition process in the recognition process execution unit 2115. This second setting method will be specifically described with reference to Figs. 63 and 64.

[0365] 63 and 64, the object to be recognized is assumed to be a vehicle license plate. Fig. 63 shows an example in which an object indicating a bus vehicle is recognized in an area 6301 during recognition processing in response to frame readout of a frame 6300. That is, the readout unit 2101 reads out the area 6301, and the recognition processing execution unit 2115 can perform recognition processing limited to the area 6301.

[0366] Here, if the recognition processing execution unit 2115 recognizes that the object is a bus vehicle in area 6301 during the recognition processing, it can predict the position of the license plate of the bus vehicle based on the content recognized from area 6301. The read determination unit 2114 determines the read area to be read next based on the predicted position of the license plate, and passes read area information indicating the determined read area to the read unit 2101.

[0367] The reading unit 2101 reads, for example, the next frame 6400 of frame 6300 in accordance with the read area information passed from the read determination unit 2114, and passes the read pixel data to the recognition processing unit 104. FIG. 64 shows an example of an image read in accordance with the read area. In the example shown in FIG. 64, pixel data of an area 6401 including the predicted position of the license plate indicated in the read area information is acquired in frame 6400. In the recognition processing unit 104, the recognition processing execution unit 2115 performs recognition processing on the area 6401. As a result, recognition processing is performed on the license plate as an object included in the area 6401, and it is possible to acquire, for example, the vehicle number of a bus detected in the recognition processing on the area 6301. Furthermore, the determination basis calculation unit 2116 calculates the basis for recognizing the vehicle number of the bus for the area 6401.

[0368] In this second setting method, the read area of the next frame 6400 is determined in the middle of the recognition processing of the entire target object in response to the reading of frame 6300 by the recognition processing execution unit 2115, thereby enabling high-precision recognition processing to be performed more quickly.

[0369] In the recognition process for frame 6300 shown in Fig. 63, if the reliability indicated in the recognition information passed from the recognition process execution unit 2115 during recognition is equal to or higher than a predetermined level, the read determination unit 2114 determines area 6401 as the next read area to be read, and performs the reading shown in Fig. 64. In this case, if the reliability indicated in the recognition information is lower than a predetermined level, the recognition process is performed for the entire object in frame 6300. In addition, the judgment basis calculation unit 2116 calculates the judgment basis for the entire object in frame 6300.

[0370] Next, a third setting method for adaptively setting a readout area based on recognition information will be described. In the third setting method, the read determination unit 2114 limits the readout area to be read next based on the reliability of the recognition process in the recognition process execution unit 2115 or the clarity of the calculated judgment basis. This third setting method will be specifically described using Figures 65 and 66.

[0371] 65, line reading is performed line by line, with the read unit being a line, sequentially and by line thinning, for a frame 6500. In the example shown in Fig. 65, the recognition process execution unit 2115 executes recognition process based on pixel data read by line reading for the entire frame 6500, and detects a specific object (a person in this example) in an area 6501 within the frame 6500. The recognition process execution unit 2115 passes recognition information indicating this recognition result to the read determination unit 2114.

[0372] If the reliability indicated in the recognition information passed from the recognition process execution unit 2115 is equal to or higher than a predetermined level, the read determination unit 2114 generates read region information indicating that reading of the frame next to frame 6500 is not to be performed. The read determination unit 2114 passes the generated read region information to the read unit 2101.

[0373] On the other hand, if the reliability indicated in the recognition information passed from the recognition process execution unit 2115 is less than a predetermined value or the basis for the determination calculated by the determination basis calculation unit 2116 is unclear, the read determination unit 2114 generates readout region information to perform reading of the frame next to frame 6500. For example, the read determination unit 2114 generates readout region information that specifies, as a readout region, an area corresponding to an area 6501 in which a specific object (person) is detected in frame 6500. The read determination unit 2114 passes the generated readout region information to the readout unit 2101.

[0374] The read unit 2101 reads the frame next to frame 6500 in accordance with the read region information passed from the read determination unit 2114. Here, the read determination unit 2114 can add to the read region information an instruction to read out the region corresponding to region 6501 in the frame next to frame 6500 without thinning out. The read unit 2101 reads out the frame next to frame 6500 in accordance with this read region information, and passes the read pixel data to the recognition processing unit 104.

[0375] FIG. 66 shows an example of an image read out in accordance with the readout region information. For example, in frame 6600, which is the frame next to frame 6500, pixel data is acquired for region 6601, which corresponds to region 6501 indicated in the readout region information. For example, the pixel data of frame 6500 can be used as is for the portion of frame 6600 other than region 6601 without reading it. In the recognition processing unit 104, the recognition processing execution unit 2115 performs recognition processing on region 6601. This makes it possible to recognize with a higher degree of reliability that a person detected in region 6101 is a pedestrian. Furthermore, the determination basis calculation unit 2116 can quickly calculate the basis for recognizing a pedestrian for region 6601.

[0376] F-1-2. Example of setting the readout area based on external information Next, a first setting method will be described, in which the readout area is adaptively set based on external information. In the first setting method, the readout determination unit 2114 adaptively sets an area within the frame based on vehicle information passed from the external information acquisition unit 5501, and limits the readout area to be read next. This makes it possible to perform recognition processing suited to the vehicle's driving.

[0377] For example, the read determination unit 2114 acquires the inclination of the vehicle based on the vehicle information and determines the read area according to the acquired inclination. As an example, when the read determination unit 2114 acquires, based on the vehicle information, that the vehicle has run up on a bump or the like and the front side is raised, it corrects the read area toward the upper end of the frame. Furthermore, when the read determination unit 2114 acquires, based on the vehicle information, that the vehicle is turning, it determines the read area to be an unobserved area in the turning direction (for example, the area on the left end if the vehicle is turning left).

[0378] Next, a second setting method will be described, in which the readout area is adaptively set based on external information. In the second setting method, map information that can sequentially reflect the current location is used as the external information. In this case, the readout determination unit 2114 generates readout area information that instructs, for example, to increase the frequency of frame readout when the current location is in an area where caution is required for vehicle travel (for example, around a school or nursery school). This makes it possible to prevent accidents caused by children running out into the road.

[0379] Next, a third setting method will be described, which adaptively sets the readout region based on external information. In the third setting method, detection information from another sensor is used as the external information. An example of the other sensor is a LiDAR (Laser Imaging Detection and Ranging) sensor. The readout determination unit 2114 generates readout region information that skips reading of a region where the reliability of the detection information from the other sensor is equal to or higher than a predetermined level. This enables power saving and speeding up of frame readout and recognition processing.

[0380] G. Application Areas The present disclosure can be applied primarily to an imaging device 100 that senses visible light, but can also be applied to devices that sense various types of light, such as infrared light, ultraviolet light, and X-rays. Therefore, the technology disclosed herein can be applied to a variety of fields to achieve faster recognition processing, lower power consumption, and real-time presentation of the basis for the recognition results. Figure 67 summarizes the fields to which the technology disclosed herein can be applied.

[0381] (1) Viewing: A device that takes images for viewing, such as a digital camera or a mobile device with a camera function. (2) Transportation: Devices used for traffic purposes, such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognizing the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. (3) Home appliances: A device used in home appliances such as TVs, refrigerators, air conditioners, and robots to capture user gestures and operate the appliances according to those gestures. (4) Medical and Healthcare: Devices used for medical and healthcare purposes, such as endoscopes and devices that take blood vessel images using infrared light. (5)Security: Devices used for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. (6) Beauty: Cosmetic devices such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp. (7) Sports: Sports equipment, such as action cameras and wearable cameras. (8) Agriculture: Agricultural equipment, such as cameras for monitoring fields and crop conditions. (9) Production, manufacturing and service industries: Equipment used in the production, manufacturing, and service industries, such as cameras or robots, for monitoring the status of production, manufacturing, processing, or service provision.

[0382] H. Application Examples The technology disclosed herein can be applied to imaging devices mounted on various moving bodies such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, and robots.

[0383] FIG. 68 shows a schematic configuration example of a vehicle control system 6800, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0384] The vehicle control system 6800 includes a plurality of electronic control units connected via a communication network 6820. In the example shown in Fig. 68, the vehicle control system 6800 includes a drive system control unit 6821, a body system control unit 6822, an outside-vehicle information detection unit 6823, an inside-vehicle information detection unit 6824, and an integrated control unit 6810. Also shown as functional components of the integrated control unit 6810 are a microcomputer 6801, an audio / video output unit 6802, and an in-vehicle network I / F (interface) 6803.

[0385] The drivetrain control unit 6821 controls the operation of devices related to the vehicle's drivetrain in accordance with various programs. The vehicle's drivetrain includes, for example, a driveforce generating device such as an internal combustion engine or a drive motor for generating vehicle driveforce, a driveforce transmission mechanism for transmitting driveforce to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating vehicle braking force. The drivetrain control unit 6821 functions as a control device for these devices.

[0386] The body system control unit 6822 controls the operation of various devices equipped in the vehicle body in accordance with various programs. The vehicle body is equipped with, for example, a keyless entry system, a smart key system, a power window device, and various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. The body system control unit 6822 may also function as a control device for these devices equipped in the vehicle body. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches may be input to the body system control unit 6822. The body system control unit 6822 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0387] The outside-vehicle information detection unit 6823 detects information outside the vehicle equipped with the vehicle control system 6800. For example, an imaging unit 6830 is connected to the outside-vehicle information detection unit 6823. The outside-vehicle information detection unit 6823 causes the imaging unit 6830 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 6823 may perform object detection processing or distance detection processing for people, vehicles, obstacles, signs, road markings, etc., based on the images received from the imaging unit 6830. The outside-vehicle information detection unit 6823, for example, performs image processing on the received images, and performs object detection processing or distance detection processing based on the results of the image processing.

[0388] The outside vehicle information detection unit 6823 performs object detection processing using a learning model program that has been trained in advance to detect objects in images. The outside vehicle information detection unit 6823 may also perform real-time calculation and presentation of the basis for judgment of the object detection results.

[0389] The imaging unit 6830 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 6830 can output the electrical signal as an image, or can output it as distance measurement information. The light received by the imaging unit 6830 may be visible light or invisible light such as infrared light. In the vehicle control system 6800, it is assumed that the imaging unit 6830 is installed in several locations on the vehicle body. The installation positions of the imaging unit 6830 will be described later.

[0390] The interior information detection unit 6824 detects information inside the vehicle. To the interior information detection unit 6824, for example, a driver state detection unit 6840 that detects the state of the driver is connected. The driver state detection unit 6840 includes, for example, a camera that captures an image of the driver, and the interior information detection unit 6824 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 6840, or may determine whether the driver is dozing off. In addition, the driver state detection unit 6840 may further include a biosensor that detects bioinformation such as the driver's brain waves, pulse, body temperature, and exhaled breath.

[0391] The microcomputer 6801 can calculate control target values for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 6823 or the inside-vehicle information detection unit 6824, and output control commands to the drivetrain control unit 6821. For example, the microcomputer 6801 can perform cooperative control aimed at realizing the functions of an Advanced Driver Assistance System (ADAS), including avoiding or mitigating collisions between vehicles, following based on the distance between vehicles, maintaining vehicle speed, warning of vehicle collisions, or warning of vehicle lane departure.

[0392] In addition, the microcomputer 6801 can perform cooperative control for the purpose of automatic driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 6823 or the inside vehicle information detection unit 6824.

[0393] Furthermore, the microcomputer 6801 can output a control command to the body system control unit 6822 based on information about the outside of the vehicle acquired by the outside-of-vehicle information detection unit 6823. For example, the microcomputer 6801 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-of-vehicle information detection unit 6823, and perform cooperative control such as switching from high beams to low beams for the purpose of preventing glare.

[0394] The audio / video output unit 6802 transmits at least one output signal of audio and / or video to an output device capable of visually or audibly notifying information to passengers in the vehicle or to the outside of the vehicle. In the system configuration example shown in Fig. 68, an audio speaker 6811, a display unit 6812, and an instrument panel 6813 are provided as output devices. The display unit 6812 may include, for example, at least one of an on-board display and a head-up display.

[0395] Fig. 69 is a diagram showing an example of the installation position of the imaging unit 6830. In the example shown in Fig. 69, a vehicle 6900 has imaging units 6901, 6902, 6903, 6904, and 6905 as the imaging unit 6830.

[0396] The imaging units 6901, 6902, 6903, 6904, and 6905 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 6900. The imaging unit 6901 provided at the front nose and the imaging unit 6905 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 6900. The imaging units 6902 and 6903 provided at the left and right side mirrors mainly acquire images of the left and right sides of the vehicle 6900, respectively. The imaging unit 6904 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 6900. The forward images acquired by the imaging units 6901 and 6905 are mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, and road markings.

[0397] 69 also illustrates the imaging ranges of the imaging units 6901 to 6904. Imaging range 6911 indicates the imaging range of imaging unit 6901 provided on the front nose, imaging ranges 6912 and 6913 indicate the imaging ranges of imaging units 6902 and 6903 provided on the side mirrors, respectively, and imaging range 6914 indicates the imaging range of imaging unit 6904 provided on the rear bumper or back door. For example, by overlaying the image data captured by imaging units 6901 to 6904, a bird's-eye view image of vehicle 6900 viewed from above can be obtained.

[0398] At least one of the imaging units 6901 to 6904 may have a function of acquiring distance information. For example, at least one of the imaging units 6901 to 6904 may be a stereo camera made up of multiple imaging elements, or an imaging element having pixels for detecting a phase difference.

[0399] For example, the microcomputer 6801 can extract, as a preceding vehicle, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 6900 and traveling in approximately the same direction as the vehicle 6900 at a predetermined speed (for example, 0 km / h or higher) by calculating the distance to each three-dimensional object within the imaging ranges 6911-6914 and the change in this distance over time (the relative speed with respect to the vehicle 6900) based on distance information obtained from the imaging units 6901-6904. Furthermore, the microcomputer 6801 can set a vehicle-to-vehicle distance that should be maintained in advance in front of the preceding vehicle, and instruct the body system control unit 6822 to perform automatic braking control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, the vehicle control system 6800 can perform cooperative control aimed at automatic driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0400] For example, the microcomputer 6801 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, standard vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 6901 to 6904, and can use the data for automatic obstacle avoidance. For example, the microcomputer 6801 distinguishes obstacles around the vehicle 6900 into obstacles that are visible to the driver of the vehicle 6900 and obstacles that are difficult to see. The microcomputer 6801 then determines the collision risk, which indicates the degree of risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 6801 can provide driving assistance for avoiding a collision with the obstacle by outputting an alarm to the driver via the audio speaker 6811 or the display unit 6812, or by performing forced deceleration or avoidance steering via the drivetrain control unit 6821.

[0401] At least one of the image capturing units 6901-6904 may be an infrared camera that detects infrared rays. For example, the microcomputer 6801 can recognize a pedestrian by determining whether or not a pedestrian is present in the images captured by the image capturing units 6901-6904. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 6901-6904 as infrared cameras and performing pattern matching processing on a series of feature points that indicate the outline of an object to determine whether or not the object is a pedestrian. When the microcomputer 6801 determines that a pedestrian is present in the images captured by the image capturing units 6901-6904 and recognizes the pedestrian, the audio / video output unit 6802 controls the display unit 6812 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / video output unit 6802 may also control the display unit 6812 to display an icon indicating the pedestrian at a desired position. [Industrial Applicability]

[0402] Although the present disclosure has been described in detail above with reference to specific embodiments, it is obvious that those skilled in the art can make modifications or substitutions to the embodiments without departing from the spirit and scope of the present disclosure.

[0403] This specification has mainly described embodiments in which the present disclosure is applied to an imaging device that senses visible light, but the gist of the present disclosure is not limited thereto. Furthermore, the present disclosure can also be applied to devices that sense various types of light, such as infrared light, ultraviolet light, and X-rays, to achieve faster speeds and lower power consumption by reducing the amount of processing in recognition processing, and to present the basis for judgment regarding the recognition results in real time. Furthermore, the technology disclosed herein can be applied to various fields to achieve faster recognition processing, lower power consumption, and real-time presentation of the basis for judgment regarding the recognition results.

[0404] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0405] The present disclosure may also be configured as follows.

[0406] (1) an imaging unit having a pixel area in which a plurality of pixels are arranged; a read unit control unit that controls a read unit set as a part of the pixel area; a read control unit that controls reading of pixel signals from pixels included in the pixel area in read units set by the read unit control unit; a recognition unit having a machine learning model trained based on training data; a determination basis calculation unit that calculates a determination basis for the recognition process in the recognition unit; Equipped with the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. Imaging device.

[0407] (2) the recognition unit learns the learning data for each read unit using a neural network model; the determination basis calculation unit infers a portion of a pixel region for each readout unit that affects each class with respect to an inference result of class classification in the neural network model; The imaging device according to (1) above.

[0408] (3) The recognition unit executes a machine learning process using an RNN on pixel data of the plurality of read units in the same frame image, and executes the recognition process based on the results of the machine learning process. The imaging device according to any one of (1) and (2) above.

[0409] (4) The read unit control unit instructs the read control unit to end the read operation when the recognition unit outputs the recognition result that satisfies a predetermined condition, or when the judgment basis calculation unit calculates a judgment basis that satisfies a predetermined condition for the recognition result. The imaging device according to any one of (1) to (3) above.

[0410] (5) When the recognition unit outputs a candidate for the recognition result that satisfies a predetermined condition, or when the judgment basis calculation unit can calculate a candidate for the judgment basis that satisfies a predetermined condition, the read unit control unit instructs the read control unit to read the read unit at a position where the recognition result that satisfies the predetermined condition or the judgment basis that satisfies the predetermined condition is expected to be obtained or the read unit is expected to present the judgment basis that satisfies the predetermined condition. The imaging device according to any one of (1) to (4) above.

[0411] (6) The read unit control unit instructs the read control unit to thin out the pixels included in the pixel area in the read unit and read out the pixel signals, and when the recognition unit outputs the candidate, instructs the read control unit to read out a read unit from among the thinned read units that is expected to produce a recognition result or judgment basis that satisfies a predetermined condition. The imaging device according to (5) above.

[0412] (7) The read unit control unit controls the read unit based on at least one of pixel information based on the pixel signal, recognition information output from the recognition unit, judgment basis calculated by the judgment basis calculation unit, and external information acquired from the outside. The imaging device according to (1) above.

[0413] (8) The read unit control unit sets a line consisting of a plurality of the pixels aligned in one row of the array as the read unit. The imaging device according to (1) above.

[0414] (9) The read unit control unit sets a pattern consisting of a plurality of the pixels including the pixels that are not adjacent to each other as the read unit. The imaging device according to (1) above.

[0415] (10) The readout unit control unit forms the pattern by arranging the plurality of pixels according to a predetermined rule. The imaging device according to (9) above.

[0416] (11) The read unit control unit sets a priority for each of the plurality of read units based on at least one of pixel information based on the pixel signal, recognition information output from the recognition unit, a basis calculated by the determination basis calculation unit, and external information acquired from the outside. The imaging device according to (1) above.

[0417] (12) An imaging device including: an imaging unit having a pixel region in which a plurality of pixels are arranged; a read unit control unit that controls a read unit set as part of the pixel region; and a read control unit that controls reading of pixel signals from pixels included in the pixel region in the read unit set by the read unit control unit. an information processing device including a recognition unit having a machine learning model trained based on training data, and a judgment basis calculation unit that calculates judgment basis for recognition processing in the recognition unit; Equipped with the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. Imaging system.

[0418] (13) each executed by a processor; a readout unit control step of controlling a readout unit set as a part of a pixel area in which a plurality of pixels of the imaging unit are arranged; a read control step of controlling readout of pixel signals from pixels included in the pixel area in readout units set in the readout unit control step; a recognition step based on a machine learning model trained on training data; a determination basis calculation step of calculating a determination basis for the recognition processing in the recognition step; and In the recognition step, a recognition process is performed on pixel signals for each read unit, and in the determination basis calculation step, a determination basis for a result of the recognition process for each read unit is calculated. Imaging method.

[0419] (14) a read unit control unit that controls a read unit set as a part of a pixel area in which a plurality of pixels of the imaging unit are arranged; a readout control unit that controls the reading of pixel signals from pixels included in the pixel area in readout units set by the readout unit control unit; a recognition unit having a machine learning model trained based on training data; a determination basis calculation unit that calculates a determination basis for the recognition process in the recognition step; Make the computer function as the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. A computer program written in a computer-readable form so as to [Explanation of symbols]

[0420] 100...imaging device, 101...optical section, 102...sensor section 103...sensor control unit, 104...recognition processing unit, 105...memory 106... image processing unit, 107... output control unit, 108... display unit 601: pixel array section, 602: vertical scanning section, 603: AD conversion section 604: horizontal scanning unit, 605: pixel signal line, 606: control unit 607: signal processing unit, 610: pixel circuit, 611: AD converter 612...Reference signal generation section 6800...Vehicle control system, 6801...Microcomputer 6802...Audio and video output unit, 6803...In-vehicle network interface 6810: Integrated control unit, 6811: Audio speaker 6812...Display unit, 6813...Instrument panel 6820...Communication network, 6821...Drive system control unit 6822...Body control unit, 6823...External information detection unit 6824...In-house information detection unit, 6830...Imaging unit 6840...Driver state detection unit

Claims

1. an imaging unit having a pixel area in which a plurality of pixels are arranged; a read unit control unit that controls a read unit set as a part of the pixel area; a read control unit that controls reading of pixel signals from pixels included in the pixel area in read units set by the read unit control unit; a recognition unit having a machine learning model trained based on training data; a determination basis calculation unit that calculates a determination basis for the recognition process in the recognition unit; Equipped with the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. Imaging device.

2. the recognition unit learns learning data for each read unit using a neural network model; the determination basis calculation unit infers a portion of a pixel region for each readout unit that affects each class with respect to an inference result of class classification in the neural network model; The imaging device according to claim 1 .

3. the recognition unit executes a machine learning process using an RNN on pixel data of the plurality of read units in the same frame image, and executes the recognition process based on a result of the machine learning process. The imaging device according to claim 1 .

4. the read unit control unit instructs the read control unit to end the read when the recognition unit outputs the recognition result that satisfies a predetermined condition, or when the judgment basis calculation unit calculates a judgment basis that satisfies a predetermined condition for the recognition result. The imaging device according to claim 1 .

5. When the recognition unit outputs a candidate for the recognition result that satisfies a predetermined condition, or when the judgment basis calculation unit can calculate a candidate for the judgment basis that satisfies a predetermined condition, the read unit control unit instructs the read control unit to read the read unit at a position where the recognition result that satisfies the predetermined condition is expected to be obtained or the judgment basis that satisfies the predetermined condition is expected to be presented. The imaging device according to claim 1 .

6. the read unit control unit instructs the read control unit to thin out the pixels included in the pixel area in the read unit and read out the pixel signals, and when the recognition unit outputs the candidate, instructs the read control unit to read out a read unit that is expected to produce a recognition result or judgment basis that satisfies a predetermined condition from among the thinned read units. The imaging device according to claim 5 .

7. the read unit control unit controls the read unit based on at least one of pixel information based on the pixel signal, recognition information output from the recognition unit, judgment basis calculated by the judgment basis calculation unit, and external information acquired from the outside. The imaging device according to claim 1 .

8. the read unit control unit sets a line consisting of a plurality of the pixels aligned in one row of the array as the read unit; The imaging device according to claim 1 .

9. the read unit control unit sets a pattern consisting of a plurality of the pixels including the pixels that are not adjacent to each other as the read unit; The imaging device according to claim 1 .

10. the readout unit control unit arranges the plurality of pixels according to a predetermined rule to form the pattern; The imaging device according to claim 9 .

11. the read unit control unit sets a priority for each of the plurality of read units based on at least one of pixel information based on the pixel signal, recognition information output from the recognition unit, a basis calculated by the determination basis calculation unit, and external information acquired from the outside. The imaging device according to claim 1 .

12. an imaging device including: an imaging unit having a pixel region in which a plurality of pixels are arranged; a read unit control unit that controls a read unit set as a part of the pixel region; and a read control unit that controls reading of pixel signals from pixels included in the pixel region in the read unit set by the read unit control unit; an information processing device including a recognition unit having a machine learning model trained based on training data, and a judgment basis calculation unit that calculates judgment basis for recognition processing in the recognition unit; Equipped with the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. Imaging system.

13. Each is executed by a processor, a readout unit control step of controlling a readout unit set as a part of a pixel area in which a plurality of pixels of the imaging unit are arranged; a read control step of controlling readout of pixel signals from pixels included in the pixel area in readout units set in the readout unit control step; a recognition step based on a machine learning model trained on training data; a determination basis calculation step of calculating a determination basis for the recognition processing in the recognition step; and In the recognition step, a recognition process is performed on pixel signals for each read unit, and in the determination basis calculation step, a determination basis for a result of the recognition process for each read unit is calculated. Imaging method.

14. a read unit control unit that controls a read unit set as a part of a pixel region in which a plurality of pixels included in the imaging unit are arranged; a readout control unit that controls the reading of pixel signals from pixels included in the pixel area in readout units set by the readout unit control unit; a recognition unit having a machine learning model trained based on training data; a determination basis calculation unit that calculates a determination basis for the recognition process in the recognition step; Make the computer function as the recognition unit performs recognition processing on pixel signals for each read unit, and the determination basis calculation unit calculates a determination basis for a result of the recognition processing for each read unit. A computer program written in a computer-readable form so as to

Images