Information processing device and information processing method, computer program, and sensor device

Abstract

Provided is an information processing technique for analyzing the cause of a result of recognition using a machine learning model.　This information processing device comprises: a recognition processing unit that performs object recognition processing by use of a learned machine learning model with respect to sensor information from a sensor unit; a cause analyzing unit that analyzes the cause of a result of recognition by the recognition processing unit, on the basis of the sensor information from the sensor unit as well as the recognition result from the recognition processing unit; and a control unit that controls an output of the sensor unit. The sensor unit is an image sensor, and the cause analyzing unit determines the cause of degradation of a recognition characteristic of the recognition processing unit on the basis of image data for cause analysis having a low resolution and a great bit length.

Description

Technical Field

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing apparatus and an information processing method for analyzing recognition processing using a machine learning model, a computer program, and a sensor device.

Background Art

[0002] In recent years, with the improvement in performance of imaging devices such as small cameras mounted on digital still cameras, digital video cameras, multifunctional mobile phones (smartphones), etc., imaging devices equipped with an image recognition function for recognizing a predetermined object included in a captured image have been developed. For example, there has been proposed an imaging device that reads pixel signals in a read unit set as a part of the pixel region of an imaging element, and performs recognition processing on the pixel signals for each read unit by a recognition unit that has learned teacher data for each read unit, thereby reducing the recognition processing time and power consumption (see Patent Document 1).

[0003] In image recognition processing, for example, among DNNs (Deep Neural Networks), machine learning models such as CNNs (Convolutional Neural Networks) and RNNs (Recurrent Neural Networks) are generally being used more and more.

[0004] Here, there are cases where sufficient recognition performance cannot be achieved, such as when the recognition rate of image recognition processing is low or the reliability of recognition is low, and the causes include the performance of the recognition algorithm and the performance of the sensor. In order to clarify the influence of the recognition algorithm on the recognition performance, for example, XAI (Explainable Artificial Intelligence) technology has been developed. On the other hand, when sufficient recognition performance cannot be realized due to the performance of the sensor, it is difficult to clarify which specific characteristics of the sensor are the cause.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Patent No. 6635221 [Overview of the project] [Problems that the invention aims to solve]

[0006] The purpose of this disclosure is to provide an information processing device and information processing method, a computer program, and a sensor device for analyzing the causes of recognition results using a machine learning model. [Means for solving the problem]

[0007] This disclosure has been made in consideration of the above issues, and its first aspect is: A recognition processing unit performs object recognition processing using a pre-trained machine learning model on sensor information from the sensor unit, A cause analysis unit analyzes the cause of the recognition result by the recognition processing unit based on the sensor information from the sensor unit and the recognition result of the recognition processing unit, A control unit that controls the output of the sensor unit, It is an information processing device equipped with the following features.

[0008] The information processing device relating to the first aspect further includes a trigger generation unit that generates a trigger for output control of the sensor unit to the control unit. The trigger generation unit generates the trigger based on at least one of the following: the recognition result or recognition reliability of the recognition processing unit, the cause analysis result of the cause analysis unit, or external information provided from outside the information processing device.

[0009] The control unit controls the output of the sensor unit based on at least one of the recognition results of the recognition processing unit or the analysis results of the cause analysis unit. If the sensor unit is an image sensor, the control unit controls the spatial arrangement of the image for analysis by the cause analysis unit. The control unit also controls the adjustment target for the sensor output for analysis by the cause analysis unit from among the multiple characteristics of the sensor unit. Then, based on the analysis results of the cause analysis unit, the control unit controls the setup of the sensor unit for acquiring sensor information for normal recognition processing by the recognition processing unit.

[0010] Furthermore, the second aspect of this disclosure is, A recognition processing step involves using a pre-trained machine learning model to perform object recognition processing on sensor information from the sensor unit, A cause analysis step in which the cause of the recognition result by the recognition processing unit is analyzed based on the sensor information from the sensor unit and the recognition result of the recognition processing unit, A control step for controlling the output of the sensor unit, This is an information processing method that possesses the following properties.

[0011] Furthermore, the third aspect of this disclosure is, A recognition processing unit performs object recognition processing using a pre-trained machine learning model on sensor information from the sensor unit. A cause analysis unit analyzes the cause of the recognition result by the recognition processing unit based on the sensor information from the sensor unit and the recognition result of the recognition processing unit. A control unit that controls the output of the sensor unit, It is a computer program written in a computer-readable format to make a computer function.

[0012] The computer program relating to the third aspect of this disclosure defines a computer program written in a computer-readable format to perform a predetermined process on a computer. In other words, by installing the computer program relating to the third aspect of this disclosure on a computer, collaborative effects can be achieved on the computer, and the same effects as those of the information processing device relating to the first aspect of this disclosure can be obtained.

[0013] Furthermore, the fourth aspect of this disclosure is, Sensor unit, A recognition processing unit performs object recognition processing using a trained machine learning model on sensor information from the aforementioned sensor unit. A cause analysis unit analyzes the cause of the recognition result by the recognition processing unit based on sensor information from the image sensor and the recognition result of the recognition processing unit, A control unit that controls the output of the image sensor, It is equipped with, The sensor device is characterized in that the sensor unit, the recognition processing unit, the cause analysis unit, and the control unit are integrated within the same semiconductor package. [Effects of the Invention]

[0014] According to this disclosure, it is possible to provide an information processing device and information processing method, a computer program, and a sensor device that adaptively input data for analysis and analyze the causes of recognition results using a machine learning model.

[0015] The effects described herein are merely illustrative, and the effects brought about by this disclosure are not limited to those described herein. Furthermore, this disclosure may produce additional effects beyond those described above.

[0016] Further purposes, features, and advantages of this disclosure will become apparent through a more detailed description based on the embodiments and accompanying drawings described below. [Brief explanation of the drawing]

[0017] [Figure 1] FIG. 1 is a diagram showing a functional configuration example of the imaging device 100. [Figure 2] FIG. 2 is a diagram showing a hardware implementation example of the imaging device 100. [Figure 3] FIG. 3 is a diagram showing another hardware implementation example of the imaging device 100. [Figure 4] FIG. 4 is a diagram showing an example of a stacked image sensor 400 having a two-layer structure. [Figure 5] FIG. 5 is a diagram showing an example of a stacked image sensor 500 having a three-layer structure. [Figure 6] FIG. 6 is a diagram showing a configuration example of the sensor unit 102. [Figure 7] FIG. 7 is a diagram for explaining a mechanism for switching the image output mode. [Figure 8] FIG. 8 is a diagram illustrating an image with high resolution and low bit length. [Figure 9] FIG. 9 is a diagram illustrating an image with low resolution and high bit length. [Figure 10] FIG. 10 is a diagram showing signal values of each pixel on corresponding horizontal scanning lines of an image with high resolution and low bit length and an image with low resolution and high bit length. [Figure 11] FIG. 11 is a diagram showing signal values of each pixel on corresponding horizontal scanning lines of an image with high resolution and low bit length and an image with low resolution and high bit length. [Figure 12] FIG. 12 is a diagram showing a result of linearly interpolating signal values of an image with low bit length. [Figure 13] FIG. 13 is a diagram showing a spatial arrangement example of the analysis output. [Figure 14] FIG. 14 is a diagram showing a spatial arrangement example of the analysis output. [Figure 15] FIG. 15 is a diagram showing a spatial arrangement example of the analysis output. [Figure 16] FIG. 16 is a diagram showing a spatial arrangement example of the analysis output. [Figure 17]Figure 17 shows an example of switching the spatial arrangement of analytical outputs. [Figure 18] Figure 18 shows an example of switching the spatial arrangement of analytical outputs. [Figure 19] Figure 19 shows an example of switching the spatial arrangement of the analytical output. [Figure 20] Figure 20 shows an example of the functional configuration of the imaging device 100 for performing cause analysis of recognition results. [Figure 21] Figure 21 is a flowchart showing the processing procedure for performing normal recognition processing in the imaging device 100 shown in Figure 20. [Figure 22] Figure 22 is a flowchart showing the processing procedure for outputting image data for analysis, which is performed in the imaging device 100 shown in Figure 20. [Figure 23] Figure 23 is a flowchart showing the processing procedure for analyzing the cause of the recognition result, which is performed in the imaging device 100 shown in Figure 20. [Figure 24] Figure 24 is a diagram illustrating the field of application of this disclosure. [Figure 25] Figure 25 shows a schematic example of the configuration of the vehicle control system 2500. [Figure 26] Figure 26 shows an example of the installation position of the imaging unit 2530. [Modes for carrying out the invention]

[0018] The present disclosure will be described below in the following order, with reference to the drawings.

[0019] A. Configuration of the imaging device B. Summary of this Disclosure C. About causal analysis C-1. Root cause analysis from an information quantity perspective C-2. Root cause analysis from the perspective of the recognition system D. Variations of sensor output D-1. Spatial arrangement of output for analysis D-2. Target of adjustment for analytical output D-3. Combinations of analytical outputs D-4. Control trigger for analytical output D-5. Control timing of analytical output D-5-1.1 Switching of analysis output at frame intervals D-5-2.1 Switching to analytical output for frames shorter than 1 frame E. Functional configuration E-1. About causal analysis E-2. Regarding the generation of control information E-3. About control triggers E-4. Operation of the imaging device E-4-1. Normal Recognition Processing Operation E-4-2. Output operation of data for analysis E-4-3. Cause analysis processing of recognition results E-4-4. How to output image data for analysis F. Application Fields G. Application Examples

[0020] A. Configuration of the imaging device Figure 1 shows an example of the functional configuration of an imaging device 100 to which this disclosure can be applied. The illustrated imaging device 100 includes an optical unit 101, a sensor unit 102, a sensor control unit 103, a recognition processing unit 104, a memory 105, an image processing unit 106, an output control unit 107, and a display unit 108. For example, a CMOS (Complementary Metal Oxide Semiconductor) can be used to form a CMOS image sensor by integrating the sensor unit 102, the sensor control unit 103, the recognition processing unit 104, and the memory 105. However, the imaging device 100 may also be an infrared light sensor that performs imaging using infrared light, or other types of optical sensors.

[0021] The optical unit 101 includes, for example, multiple optical lenses for focusing light from a subject onto the light-receiving surface of the sensor unit 102, an aperture mechanism for adjusting the size of the aperture for incident light, and a focus mechanism for adjusting the focus of the light illuminating the light-receiving surface. The optical unit 101 may further include a shutter mechanism for adjusting the time that light illuminates the light-receiving surface. The aperture mechanism, focus mechanism, and shutter mechanism of the optical unit are configured to be controlled, for example, by the sensor control unit 103. The optical unit 101 may be configured integrally with the imaging device 100 or separately from the imaging device 100.

[0022] The sensor unit 102 includes a pixel array in which multiple pixels are arranged in a matrix. Each pixel contains a photoelectric conversion element, and the pixels arranged in the matrix form a light-receiving surface. The optical unit 101 forms an image of incident light on the light-receiving surface, and each pixel of the sensor unit 102 outputs a pixel signal corresponding to the irradiated light. The sensor unit 102 further includes a drive circuit for driving each pixel in the pixel array, and a signal processing circuit that performs predetermined signal processing on the signals read from each pixel and outputs them as pixel signals for each pixel. The sensor unit 102 outputs the pixel signals of each pixel within the pixel area as digital image data.

[0023] The sensor control unit 103 is configured, for example, by a microprocessor, and controls the reading of pixel data from the sensor unit 102, and outputs image data based on the pixel signals read from each pixel. The pixel data output from the sensor control unit 103 is passed to the recognition processing unit 104 and the image processing unit 106.

[0024] Furthermore, the sensor control unit 103 generates and supplies imaging control signals to the sensor unit 102 to control the sensor characteristics (resolution, line length, frame rate, shutter speed / exposure, etc.) of the sensor unit 102. The imaging control signals include information indicating the exposure and analog gain during imaging by the sensor unit 102. The imaging control signals further include control signals for performing the imaging operation of the sensor unit 102, such as vertical synchronization signals and horizontal synchronization signals.

[0025] The recognition processing unit 104 performs recognition processing of objects in the image (person detection, face recognition, image classification, etc.) based on the pixel data passed from the sensor control unit 103. However, the recognition processing unit 104 may also perform recognition processing using image data after image processing by the image processing unit 106. The recognition result from the recognition processing unit 104 is passed to the output control unit 107.

[0026] In this embodiment, the recognition processing unit 104 is configured using, for example, a DSP (Digital Signal Processor) and performs recognition processing using a machine learning model. Model parameters obtained through prior model training are stored in memory 105, and the recognition processing unit 104 performs recognition processing using the model with the model parameters read from memory 105. Specifically, the machine learning model consists of a DNN such as a CNN or an RNN.

[0027] The image processing unit 106 processes the pixel data received from the sensor control unit 103 to obtain an image suitable for human viewing, and outputs image data consisting of, for example, a set of pixel data. For example, if each pixel in the sensor unit 102 is provided with a color filter, and each pixel data has color information of either R (red), G (green), or B (blue), the image processing unit 106 performs demosaicing, white balance processing, etc. The image processing unit 106 can also instruct the sensor control unit 103 to read the pixel data necessary for image processing from the sensor unit 102. The image processing unit 106 passes the processed image data to the output control unit 107. For example, the above functions of the image processing unit 106 are realized by the ISP (Image Signal Processor) executing a program that is pre-stored in local memory (not shown).

[0028] The output control unit 107 is composed of, for example, a microprocessor. The output control unit 107 receives the recognition results of objects contained in the image from the recognition processing unit 104, and the image data as an image processing result from the image processing unit 106, and outputs one or both of these to the outside of the imaging device 100. The output control unit 107 also outputs the image data to the display unit 108. The user can view the image displayed on the display unit 108. The display unit 108 may be built into the imaging device 100 or may be externally connected to the imaging device 100.

[0029] Figure 2 shows an example of hardware implementation of the imaging device 100. In the example shown in Figure 2, the sensor unit 102, sensor control unit 103, recognition processing unit 104, memory 105, image processing unit 106, and output control unit 107 are mounted on a single chip 200. However, to prevent confusion in the drawing, the memory 105 and output control unit 107 are omitted from the illustration in Figure 2.

[0030] In the configuration example shown in Figure 2, the recognition result from the recognition processing unit 104 is output to the outside of the chip 200 via the output control unit 107. The recognition processing unit 104 can also acquire pixel data or image data for use in recognition from the sensor control unit 103 via an interface inside the chip 200.

[0031] Figure 3 shows another hardware implementation example of the imaging device 100. In the example shown in Figure 3, the sensor unit 102, sensor control unit 103, image processing unit 106, and output control unit 107 are mounted on a single chip 300, but the recognition processing unit 104 and memory 105 are located outside the chip 300. However, in Figure 3 as well, the memory 105 and output control unit 107 are omitted from the illustration to prevent confusion in the drawing.

[0032] In the configuration example shown in Figure 3, the recognition processing unit 104 acquires pixel data or image data for recognition from the output control unit 107 via the inter-chip communication interface. The recognition processing unit 104 also directly outputs the recognition result to the outside. Of course, it is also possible to configure the system so that the recognition result from the recognition processing unit 104 is returned to the output control unit 107 in the chip 300 via the inter-chip communication interface, and the output control unit 107 outputs it to the outside of the chip 300.

[0033] In the configuration example shown in Figure 2, since both the recognition processing unit 104 and the sensor control unit 103 are mounted on the same chip 200, communication between the recognition processing unit 104 and the sensor control unit 103 can be performed at high speed via an interface within the chip 200. On the other hand, in the configuration example shown in Figure 3, since the recognition processing unit 104 is located outside the chip 300, it is easy to replace the recognition processing unit 104. However, communication between the recognition processing unit 104 and the sensor control unit 103 must be performed via an interface between chips, resulting in slower speeds.

[0034] Figure 4 shows an example in which the imaging device 100 is formed as a stacked CMOS image sensor 400 with a two-layer structure in which the semiconductor chips 200 (or 300) of the imaging device are stacked in two layers. In the illustrated structure, the pixel section 411 is formed on the first layer semiconductor chip 401, and the memory and logic section 412 are formed on the second layer semiconductor chip 402.

[0035] The pixel unit 411 includes at least the pixel array in the sensor unit 102. The memory and logic unit 412 includes, for example, the sensor control unit 103, the recognition processing unit 104, the memory 105, the image processing unit 106, the output control unit 107, and an interface for communication between the imaging device 100 and the outside. The memory and logic unit 412 further includes some or all of the drive circuit that drives the pixel array in the sensor unit 102. Although not shown in Figure 4, the memory and logic unit 412 may further include, for example, memory used by the image processing unit 106 for processing image data.

[0036] As shown on the right side of Figure 4, the imaging device 100 is configured as a single solid-state image sensor by bonding the first layer semiconductor chip 401 and the second layer semiconductor chip 402 together while making electrical contact.

[0037] Figure 5 shows an example of a stacked CMOS image sensor 500 formed as a three-layer stacked CMOS image sensor 500, in which the semiconductor chips 200 (or 300) of the imaging device 100 are stacked in three layers. In the illustrated structure, the pixel section 511 is formed on the first layer semiconductor chip 501, the memory section 512 is formed on the second layer semiconductor chip 502, and the logic section 513 is formed on the third layer semiconductor chip 503.

[0038] The pixel unit 511 includes at least the pixel array in the sensor unit 102. The logic unit 513 includes, for example, the sensor control unit 103, the recognition processing unit 104, the image processing unit 106, the output control unit 107, and an interface for communication between the imaging device 100 and the outside. The logic unit 513 further includes some or all of the drive circuit for driving the pixel array in the sensor unit 102. In addition to the memory 105, the memory unit 512 may further include, for example, memory used by the image processing unit 106 for processing image data.

[0039] As shown on the right side of Figure 5, the imaging device 100 is configured as a single solid-state image sensor by bonding together the first layer semiconductor chip 501, the second layer semiconductor chip 502, and the third semiconductor chip 503 while electrically contacting them.

[0040] Figure 6 shows an example configuration of the sensor unit 102. The illustrated sensor unit 102 includes a pixel array unit 601, a vertical scanning unit 602, an AD (Analog to Digital) conversion unit 603, a horizontal scanning unit 604, a pixel signal line 605, a vertical signal line VSL, a control unit 606, and a signal processing unit 607. Note that the control unit 606 and signal processing unit 607 in Figure 6 may be included in the sensor control unit 103 in Figure 1, for example.

[0041] The pixel array unit 601 is composed of a plurality of pixel circuits 610, each containing a photoelectric conversion element that performs photoelectric conversion on received light and a circuit that reads out the charge from the photoelectric conversion element. The plurality of pixel circuits 610 are arranged in a matrix in the horizontal (row) and vertical (column) directions. The row-direction arrangement of the pixel circuits 610 forms a line. For example, when one frame of image is formed with 1920 pixels × 1080 lines, the pixel array unit 601 forms one frame of image by reading out pixel signals for 1080 lines, each line consisting of 1920 pixel circuits 610.

[0042] In the pixel array unit 601, a pixel signal line 605 is connected to each row and column of each pixel circuit 610, and a vertical signal line VSL is connected to each column. The ends of each pixel signal 605 that are not connected to the pixel array unit 601 are connected to the vertical scanning unit 602. The vertical scanning unit 602 transmits control signals, such as drive pulses for reading pixel signals from pixels, to the pixel array unit 601 via the pixel signal line 605, according to the control of the control unit 606. The ends of the vertical signal line VSL that are not connected to the pixel array unit 601 are connected to the AD conversion unit 603. The pixel signals read from the pixels are transmitted to the AD conversion unit 603 via the vertical scanning line VSL.

[0043] The pixel signal from the pixel circuit 610 is read out by transferring the charge accumulated in the photoelectric conversion element due to exposure to the floating diffusion layer (FD), and then converting the transferred charge into a voltage in the floating diffusion layer. The voltage converted from the charge in the floating diffusion layer is output to the vertical signal line VSL via an amplifier.

[0044] The AD conversion unit 603 includes an AD converter 611 provided for each vertical signal line VSL, a reference signal generation unit 612, and a horizontal scanning unit 604. The AD converter 611 is a column AD converter that performs AD conversion processing on each column of the pixel array unit 601. It performs AD conversion processing on the pixel signal supplied from the pixel circuit 610 via the vertical signal line VSL to generate two digital values ​​for correlated double sampling (CDS) processing to reduce noise, and outputs them to the signal processing unit 607.

[0045] The reference signal generation unit 612 generates a ramp signal as a reference signal, which each column AD converter 611 uses to convert a pixel signal into two digital values, based on a control signal from the control unit 606, and supplies it to each column AD converter 611. The ramp signal is a signal in which the voltage level decreases at a constant rate with respect to time, or a signal in which the voltage level decreases in a stepwise manner.

[0046] Within the AD converter 611, when a ramp signal is supplied, a counter starts counting according to the clock signal. The voltage of the pixel signal supplied from the vertical signal line VSL is compared with the voltage of the ramp signal. When the voltage of the ramp signal crosses the voltage of the pixel signal, the counter stops counting and outputs a value corresponding to the count value at that time, thereby converting the analog pixel signal into a digital value.

[0047] The signal processing unit 607 performs CDS processing based on the two digital values ​​generated by the AD converter 611 to generate a pixel signal (pixel data) of the digital signal and outputs it to the outside of the sensor control unit 103.

[0048] The horizontal scanning unit 604, under the control of the control unit 606, performs a selection operation to select each AD converter 611 in a predetermined order, thereby sequentially outputting the digital values ​​temporarily held by each AD converter 611 to the signal processing unit 607. The horizontal scanning unit 604 is configured using, for example, a shift register or an address decoder.

[0049] The control unit 606 generates drive signals to control the operation of the vertical scanning unit 602, the AD conversion unit 603, the reference signal generation unit 612, and the horizontal scanning unit 604, etc., based on the imaging control signal supplied from the sensor control unit 103, and outputs them to each unit. For example, based on the vertical synchronization signal and horizontal synchronization signal included in the imaging control signal, the control unit 606 generates a control signal for the vertical scanning unit 602 to supply to each pixel circuit 610 via the pixel signal line 605, and supplies it to the vertical scanning unit 602. The control unit 606 also passes information indicating the analog gain included in the imaging control signal to the AD conversion unit 603. Within the AD conversion unit 603, the gain of the pixel signal input to each AD converter 611 via the vertical signal line VSL is controlled based on this information indicating the analog gain.

[0050] The vertical scanning unit 602, based on control signals supplied from the control unit 606, supplies various signals, including drive pulses, to the pixel signal lines 605 of the selected pixel row in the pixel array unit 601, line by line, to each pixel circuit 610, causing each pixel circuit 610 to output a pixel signal to the vertical signal line VSL. The vertical scanning unit 602 is configured using, for example, a shift register or an address decoder. The vertical scanning unit 602 also controls the exposure in each pixel circuit 610 based on exposure information supplied from the control unit 606.

[0051] As shown in Figure 6, the sensor unit 102 is a column AD type image sensor in which each AD converter 611 is arranged in a row.

[0052] Two imaging methods can be used when imaging with the pixel array unit 601: the rolling shutter method and the global shutter method. In the global shutter method, all pixels of the pixel array unit 601 are exposed simultaneously and the pixel signals are read out all at once. On the other hand, in the rolling shutter method, the pixel array unit 601 is exposed line by line from top to bottom and the pixel signals are read out sequentially.

[0053] B. Summary of this Disclosure As shown in Figure 1 recognitionIn an imaging device 100 equipped with a function, the recognition performance of the recognition processing unit 104, which outputs through the output control unit 107, may be insufficient (for example, the recognition rate or recognition reliability may be low). Two possible causes for this are the performance of the recognition algorithm and the performance of the sensor. When sufficient recognition performance cannot be achieved due to the latter, the performance of the sensor, it is difficult to determine which specific characteristics of the sensor are causing the problem.

[0054] When we talk about sensor performance, it includes factors such as sensor resolution, bit length (the number of grayscale levels for each pixel), frame rate, and dynamic range. It is difficult to identify from captured images that the reason for the inability to improve recognition performance is due to sensor performance, and even more difficult to determine which of the above sensor performance factors is the cause.

[0055] If the recognition processing unit 104 can acquire the sensor information necessary to achieve sufficient recognition performance (i.e., image data with high resolution, high bit length, high bit rate, and high dynamic range), then a decrease in recognition performance due to the performance of the sensor unit 102 will not occur. Furthermore, the sensor information may include not only the image but also metadata related to resolution, bit rate, dynamic range, shutter speed, analog gain, etc. However, due to hardware limitations of the sensor unit 102, it is not practical to acquire sensor information including such metadata. Specifically, it is difficult for the sensor unit 102 to capture high-resolution and high-bit-length images, and it can only acquire either high-resolution, low-bit-length images with suppressed bit length, or low-resolution, high-bit-length images with suppressed resolution.

[0056] Therefore, while ideally the cause of the degradation in recognition performance in the recognition processing unit 104 should be analyzed using high-resolution and high-bit-length images, in this disclosure, high-resolution and low-bit-length images are used for normal recognition processing, while low-resolution and high-bit-length images are used for analysis when the cause of the degradation in recognition performance is to be analyzed more strictly or in detail.

[0057] In other words, in the imaging device 100 to which this disclosure is applied, the sensor unit 102 has two image output modes: outputting a high-resolution, low-bit-length image during normal recognition processing (including normal image output), and outputting a low-resolution, high-bit-length image during cause analysis.

[0058] Two possible triggers for switching image output modes are listed below.

[0059] (1) The high-resolution, low-bit-length images output during normal recognition processing were analyzed to determine whether there was insufficient gradation for recognition processing, and it was determined that there was insufficient gradation. (2) The high-resolution, low-bit-length images output during normal recognition processing cannot be recognized or the reliability of the recognition is low.

[0060] When either of the triggers described in (1) or (2) above occurs, the image output mode of the sensor unit 102 is switched from normal recognition mode (high resolution and low bit length) to cause analysis mode (low resolution and high bit length). By using the low-resolution and high-bit-length image, it is possible to determine whether the inability to recognize or the low reliability of recognition is due to the performance of the sensor unit 102, and which characteristic of the sensor unit 102 is the cause.

[0061] Therefore, according to this disclosure, it is possible to identify which characteristic of the sensor unit 102 (resolution, bit length, etc.) is causing the inability to recognize an object from the image data acquired from the sensor unit 102 or the low reliability of the recognition. Furthermore, based on the analysis results, the setup of the sensor unit 102 can be dynamically changed to improve the recognition performance.

[0062] C. About causal analysis Figure 7 illustrates the mechanism for switching the image output mode in the imaging device 100.

[0063] Figure 7(a) shows a high-resolution, high-bit-length image. Using this high-resolution, high-bit-length image, the recognition processing unit 104 can simultaneously perform both normal recognition processing and analysis processing to determine the cause of the inability to perform recognition or the low reliability of recognition. However, due to hardware limitations of the sensor unit 102, it is difficult to output high-resolution, high-bit-length images.

[0064] In contrast, Figure 7(b) shows a high-resolution, low-bit-length image, and Figure 7(c) shows a low-resolution, high-bit-length image. The sensor unit 102 can output both these high-resolution, low-bit-length images and low-resolution, high-bit-length images without being subject to hardware limitations. However, while the high-resolution, low-bit-length image shown in Figure 7(b) can be used for normal recognition processing, the lack of gradation makes it impossible to identify the cause of recognition failure or low recognition reliability. On the other hand, while the low-resolution, high-bit-length image shown in Figure 7(c) has sufficient gradation, making it possible to identify the cause of recognition failure or low recognition reliability, the lack of resolution (i.e., small image size) makes it impossible to perform normal recognition processing.

[0065] Here, let's compare the low-bit length image shown in Figure 7(b) with the high-bit length image shown in Figure 7(c). In the low-bit length image, the amount of information per pixel is small, so the difference from the high-bit length image is large. As a result, objects with few grayscale levels become invisible in the low-bit length image, but can be found in the high-bit length image with many grayscale levels.

[0066] Figures 8 and 9 illustrate high-resolution and low-bit-length images, and low-resolution and high-bit-length images, respectively, of the same object. Here, high-resolution and low-bit-length images refer to, for example, HD (High-Zigzag Definition) resolution images with 1280 x 720 pixels (720p) and 2 bits (4 levels of grayscale). Low-resolution and high-bit-length images refer to, for example, QQVGA (Quarter Quater Video Graphic Array) resolution images with 160 x 120 pixels and 8 bits (256 levels of grayscale).

[0067] Figures 8 and 9 both show images of two pedestrians. It is assumed that the pedestrian on the left was captured with a high signal value, while the pedestrian on the right was only captured with a low signal value. Referring to Figure 8 first, one pedestrian on the left can be observed, but the pedestrian on the right is obscured due to the lack of detail and cannot be observed. Referring to Figure 9 next, although it is difficult to distinguish whether or not they are pedestrians due to the low resolution, it can be observed that two objects are present in the image.

[0068] In short, high-bit-length images contain a large amount of information per pixel. Therefore, even if an object cannot be recognized or its recognition reliability is low in a low-bit-length image, it may be possible to recognize the object in a high-bit-length image.

[0069] C-1. Root cause analysis from an information quantity perspective Figure 10 plots the signal values ​​of each pixel on the corresponding horizontal scan line for the high-resolution and low-bit-length images shown in Figure 8, and the low-resolution and high-bit-length images shown in Figure 9. The horizontal axis represents the x-coordinate of the image frame, and the vertical axis represents the signal value. Figure 10 also shows the signal values ​​of each pixel on the horizontal scan line passing through two objects (i.e., two pedestrians).

[0070] The signal values ​​of each pixel in high-resolution and low-bit-length images are plotted as gray dots. Similarly, the signal values ​​of each pixel in low-resolution and high-bit-length images are plotted as black dots. Furthermore, the true signal values ​​along the horizontal line are shown as solid lines in Figure 10. High-resolution and low-bit-length images are plotted densely along the horizontal axis due to their high resolution, but the signal values ​​are plotted discretely along the vertical axis due to their low gradation. Conversely, low-resolution and high-bit-length images are plotted discretely along the horizontal axis due to their low resolution, but the signal values ​​are plotted at fine intervals due to their high gradation.

[0071] In Figure 10, referring to the true values ​​drawn with black lines, a high peak on the left and a low peak on the right are observed, which correspond to two objects (two pedestrians) contained in the image. The area enclosed by the frame indicated by reference number 1100 in Figure 11 indicates the area where the object on the right (pedestrian) is located. Referring to the area within frame 1100, the signal value of each pixel in the high-resolution, low-bit-length image is truncated, resulting in a signal level of 0, and therefore it cannot be observed. On the other hand, in the case of the low-resolution, high-bit-length image, the signal value of each pixel is plotted at approximately the same signal level as the true value. Therefore, it can be seen that the object on the right (pedestrian) can also be observed in the low-resolution, high-bit-length image.

[0072] As shown in Figure 10, high-resolution and low-bit-length images are plotted discretely along the vertical axis due to their low bit depth. Therefore, Figure 12 shows the gradation of the high-resolution and low-bit-length images linearly interpolated, as indicated by reference number 1200. By calculating the difference between the linear interpolation of the low-bit-length image and the gradation of the high-bit-length image, it is possible to check whether there is information that was not visible in the low-bit-length image.

[0073] Therefore, if a high-resolution, low-bit-length image like the one shown in Figure 8 is used as the image for normal recognition processing, and a low-resolution, high-bit-length image like the one shown in Figure 9 is used as the image for cause analysis, the difference in the amount of grayscale information between the two images can be used to analyze the reason why an object could not be recognized in the image used for normal recognition processing.

[0074] C-2. Root cause analysis from the perspective of the recognition system Figures 8 and 9 show examples of high-resolution and low-bit-length images and low-resolution and high-bit-length images, respectively, of the same object. When the recognition processing unit 104 processes each image, it obtains a recognition result that there is one pedestrian from the high-resolution and low-bit-length image shown in Figure 8, while it obtains a recognition result that there are two objects (which cannot be recognized as pedestrians) from the low-resolution and high-bit-length image shown in Figure 9.

[0075] Thus, when high-resolution, low-bit-length images like those shown in Figure 8 are used as images for normal recognition processing, and low-resolution, high-bit-length images like those shown in Figure 9 are used as images for cause analysis, if inconsistencies occur in the recognition results of each image, it can be analyzed that there is information that was not visible in the low-bit-length image, i.e., insufficient gradation is the cause of the inability to recognize the object.

[0076] Here, let's compare the information-based cause analysis described in section C-1 with the recognition-based cause analysis described in this section C-2. The information-based cause analysis is affected by noise, while the recognition-based cause analysis has the advantage of reducing the impact of noise. However, the recognition-based cause analysis has the challenge of increasing computational complexity because it requires processing both the image used for normal recognition and the image used for cause analysis separately.

[0077] D. Variations of sensor output Figure 7 shows an example where the sensor unit 102 outputs high-resolution and low-bit-length images for normal recognition processing, as well as low-resolution and high-bit-length images for cause analysis. However, the image output for cause analysis is not limited to this. Section D describes variations in sensor output for cause analysis.

[0078] D-1. Spatial arrangement of output for analysis In the example shown in Figure 7, the sensor unit 102 outputs images for normal recognition processing and images for cause analysis in a time-division manner. The method for outputting the images for cause analysis is not limited to this. For example, the images for cause analysis may be spatially arranged within the images for normal recognition processing. In such a case, normal recognition processing and analysis processing of the recognition results can be performed simultaneously.

[0079] Figure 13 shows an example where analysis images are placed line by line on top of a regular recognition image. Figure 14 shows an example where small rectangular analysis image blocks are arranged in a grid pattern on top of a regular recognition image. By evenly distributing the analysis images line by line or in a grid pattern, it is possible to efficiently find areas within the image frame that cause a decrease in recognition performance (unrecognizable or with low recognition reliability).

[0080] Figure 15 also shows an example where small rectangular image blocks for analysis are placed in an arbitrary pattern on a regular image used for recognition processing. For example, by using the recognition results to place the image blocks for analysis in a pattern corresponding to the size and shape of the recognized object, the recognized object can be analyzed intensively. Although not shown in the illustration, it is also possible to randomly place the image blocks for analysis on a regular image used for recognition processing.

[0081] Figure 16 also shows an example of dynamically generating a pattern consisting of a collection of small image blocks for analysis on top of the image used for normal recognition processing. For example, by dynamically generating an analysis pattern around the recognized object using the recognition results, the recognized object can be analyzed intensively.

[0082] D-2. Target of adjustment for analytical output In the explanations so far, we have described an example in which images for normal recognition processing and images for analysis are acquired from the images captured by the sensor unit 102, mainly by adjusting the resolution and bit length. That is, we have described an example in which the image for normal recognition processing is a high-resolution, low-bit-length image, while the image for analysis is a low-resolution, high-bit-length image. However, this is just one example, and it is possible to acquire image outputs for analysis by adjusting various characteristics of the sensor unit 102.

[0083] Basically, the characteristics of the sensor unit 102 as shown in (1) to (4) below can be adjusted to obtain an image output for analysis. Of course, other characteristics of the sensor unit 102 may also be used as the target for adjusting the analysis output.

[0084] (1) Resolution (2) bit length (3) Frame rate (4) Shutter speed / exposure

[0085] D-3. Combinations of analytical outputs For example, as illustrated in Figures 13 to 16, for an image region for analysis spatially arranged on an image used for normal recognition processing, one or more of the characteristics listed in (1) to (4) above are adjusted to produce an image for analysis. As described above, the adjustment target for the analysis output may be a combination of resolution and bit length, or a combination of two or more other characteristics may be used to produce the analysis output.

[0086] Section D-1 above shows several examples of spatial arrangement of analysis output, but multiple spatial arrangements can be combined and switched frame by frame. Figure 17 shows an example in which, in one frame, the spatial arrangement is such that the images for analysis are placed on a line-by-line basis, and in the next frame, it switches to a spatial arrangement in which the image blocks for analysis are placed in a grid.

[0087] Furthermore, multiple spatial arrangements can be combined within a single frame, and the spatial arrangement can be switched within the frame. Figure 18 shows an example where the spatial arrangement places the images for analysis line by line up to a certain point in the frame, and then switches to a spatial arrangement where the image blocks for analysis are arranged in a grid from that point onward. Figure 19 shows an example of a spatial arrangement where the images for analysis are placed line by line, but the spacing between the lines is adaptively changed. As an application, it is also possible to go back to a certain line, readjust the object to be adjusted, and then output the image for analysis.

[0088] In all the spatial arrangement examples shown in Figures 13 to 19, blocks of images for analysis, such as lines and blocks, are discretely arranged. Different adjustment targets for analysis output may be assigned to each line or block. For example, the combination of resolution and bit length may be adjusted up to a certain line in a frame, but from that point onward, the frame rate may be switched to the adjustment target.

[0089] Alternatively, the adjustment target for the image used for analysis can be switched for each frame.

[0090] D-4. Control trigger for analytical output For example, you could use any of the following (1) to (3) as a trigger to control the image output for analysis.

[0091] (1) Recognition result or confidence level of recognition (2)Cause analysis results (3) External information

[0092] Specifically, when the recognition processing unit 104 fails to recognize an object that should be present in the input image, or when the reliability of object recognition is low, it is used as a trigger to output an analytical image in order to analyze the cause. Also, when the cause analysis unit 2003 outputs an analysis result indicating that the cause of the decrease in recognition reliability lies in the performance of the sensor unit 102, it is used as a trigger to output an analytical image. Furthermore, external information that triggers the output of an analytical image includes the surrounding environment of the imaging device 100 (for example, environmental information around the vehicle on which the imaging device 100 is mounted) and instructions from the user for cause analysis.

[0093] Then, in response to any of the above triggers occurring, one of the following controls (1) to (4) is performed.

[0094] (1) In response to a trigger, start or stop the output of images for analysis. (2) Change the spatial arrangement of the images for analysis in response to the trigger. (3) Depending on the trigger, change the target of the image adjustment for analysis. (4) Change the combination of images for analysis depending on the trigger.

[0095] D-5. Control timing of analytical output As explained in section D-3 above, the analysis output may be switched at intervals of one frame, or it may be switched at intervals of less than one frame.

[0096] D-5-1.1 Switching of analysis output at frame intervals As illustrated in Figure 17, the spatial arrangement of the analytical output may be switched between frames. Alternatively, the adjustment target may be switched while the spatial arrangement of the analytical output remains the same between frames. Furthermore, the combination of the spatial arrangement of the analytical output and the adjustment target may be switched between frames.

[0097] D-5-2.1 Switching to analytical output for frames shorter than 1 frame As shown in Figures 18 and 19, the spatial arrangement of the analytical output may be switched within a frame. Alternatively, the adjustment target may be switched while keeping the spatial arrangement of the analytical output the same within a frame. Furthermore, the combination of the spatial arrangement of the analytical output and the adjustment target may be switched between frames.

[0098] E. Functional configuration Figure 20 schematically shows an example of the functional configuration of the imaging device 100, which is configured to analyze the causes of the degradation in recognition performance in the recognition processing unit 104. As already explained in Section A above, the recognition processing unit 104 performs recognition processing on images captured by the sensor unit 102 using a machine learning model composed of DNNs such as CNNs and RNNs. Furthermore, the degradation in recognition performance referred to here specifically includes the inability to recognize objects that should be present in the captured image, and low reliability of recognition.

[0099] The imaging device 100 shown in Figure 20 includes a recognition data acquisition unit 2001, an analysis data acquisition unit 2002, a sensor control unit 103, a recognition processing unit 104, a cause analysis unit 2003, a control information generation unit 2004, and a trigger generation unit 2005. Although the imaging device 100 basically has the functional configuration shown in Figure 1, for convenience, the sensor unit 102, memory 105, image processing unit 106, output control unit 107, and display unit 108 are omitted from the illustration.

[0100] The recognition data acquisition unit 2001 acquires image data from the sensor unit 102 (not shown in Figure 20) that the recognition processing unit 104 uses for normal recognition processing. The analysis data acquisition unit 2002 also acquires image data from the sensor unit 102 (not shown in Figure 20) that the cause analysis unit 2003 uses to analyze the cause of the deterioration in recognition performance in the recognition processing unit 104.

[0101] The sensor control unit 103 controls the sensor characteristics (resolution, line length, frame rate, shutter speed / exposure, etc.) of the sensor unit 102 based on the control information supplied from the control information generation unit 2004. Specifically, when the recognition data acquisition unit 2001 attempts to acquire image data from the sensor unit 102, the sensor control unit 103 controls the sensor characteristics of the sensor unit 102 based on the recognition control information supplied from the control information generation unit 2004. When the analysis data acquisition unit 2002 attempts to acquire image data from the sensor unit 102, the sensor control unit 103 controls the sensor characteristics of the sensor unit 102 based on the analysis control information supplied from the control information generation unit 2004.

[0102] While an entire frame may consist of an image for analysis, typically, an analysis image consisting of a pattern of lines or small pixel blocks is placed within a frame (see, for example, Figures 13 to 19). Therefore, the sensor control unit 103 controls the sensor unit 102 to place an analysis image with adjusted sensor characteristics in a predetermined area within a frame, consisting of a pattern of lines or pixel blocks, based on the spatial arrangement specified by the analysis control information supplied from the control information generation unit 2004.

[0103] The recognition processing unit 104 receives recognition image data acquired from the sensor unit 102 by the recognition data acquisition unit 2001 and performs recognition processing of objects in the image (person detection, face recognition, image classification, etc.). As already explained in section A above, the recognition processing unit 104 performs recognition processing using a machine learning model composed of DNNs such as CNNs and RNNs.

[0104] The cause analysis unit 2003 uses the recognition image data acquired from the sensor unit 102 by the recognition data acquisition unit 2001 and the analysis image data acquired from the sensor unit 102 by the analysis data acquisition unit 2002 to perform a cause analysis of the deterioration in recognition performance in the recognition processing unit 104. For example, the cause analysis unit 2003 performs a cause analysis from the information quantity perspective as described in section C-1 above, and a cause analysis from the recognition device perspective as described in section C-2 above.

[0105] The control information generation unit 2004 further comprises an analysis control information generation unit 2006 and a recognition control information generation unit 2009.

[0106] The recognition control information generation unit 2009 generates control information for the sensor unit 102 so that the recognition data acquisition unit 2001 can acquire image data for normal recognition processing (for example, high-resolution and low-bit length images) from the sensor unit 102, and supplies this information to the sensor control unit 103. Basically, the recognition control information generation unit 2009 sets up the control information for normal recognition processing based on the analysis results by the cause analysis unit 2003. That is, if the analysis results indicate that the cause of the decrease in recognition reliability in the recognition processing unit 104 is due to the performance of the sensor unit 102, the recognition control information generation unit 2009 searches for more appropriate control information so that the analysis results indicate that the performance of the sensor unit 102 is no longer the cause of the decrease in recognition reliability.

[0107] Furthermore, the analysis control information generation unit 2006 generates control information for the sensor unit 102, which enables the analysis data acquisition unit 2002 to acquire image data for analysis (for example, low-resolution and high-bit-length images) from the sensor unit 102, and supplies this information to the sensor control unit 103.

[0108] The image data for analysis is basically arranged in a specific area consisting of a predetermined line or pixel block pattern within a single frame. Furthermore, the image data for analysis is an image that has been adjusted using at least one or more combinations of the sensor characteristics of the sensor unit 102 as the adjustment target. The analysis control information generation unit 2006 further includes a spatial arrangement setting unit 2007 for setting the spatial arrangement of the image data for analysis, and an adjustment target setting unit 2008 for setting the adjustment target of the image data for analysis. The unit generates analysis control information including the spatial arrangement and adjustment target set by these setting units 2007 and 2008 and supplies it to the sensor control unit 103.

[0109] The trigger generation unit 2005 generates a control trigger for the control information generation unit 2004. The trigger generation unit 2005 generates a trigger based on either the recognition result or recognition reliability of the recognition processing unit 104, the analysis result by the cause analysis unit 2003, or external information supplied from outside the imaging device 100, and supplies it to the control information generation unit 2004. Then, the analysis control information generation unit 2006 generates or stops analysis control information, sets or changes the spatial arrangement of analysis image data by the spatial arrangement setting unit 2007, or sets or changes the adjustment target of the analysis image data by the adjustment target setting unit 2008, in response to the trigger supplied by the trigger generation unit 2005.

[0110] Alternatively, the sensor unit 102 may be configured as a single CMOS image sensor by including the recognition data acquisition unit 2001, the analysis data acquisition unit 2002, and the sensor control unit 103. Or, all the functional components shown in Figure 20 may be included and configured as a single CMOS image sensor.

[0111] E-1. About causal analysis The cause analysis unit 2003 analyzes the cause of the recognition result in the recognition processing unit 104 based on the recognition data acquired by the recognition data acquisition unit 2001 and the analysis data acquired by the analysis data acquisition unit 2002.

[0112] As explained in section C above, the cause analysis unit 2003 may perform at least one of the following: cause analysis from an information quantity perspective or cause analysis from a recognition perspective.

[0113] The cause analysis unit 2003, in its cause analysis from an information quantity perspective, focuses on the difference in information quantity between the recognition data and the analysis data to analyze the cause of the recognition result in the recognition processing unit 104. For example, if a high-resolution, low-bit-length image is used as recognition data and a low-resolution, high-bit-length image is used as analysis data, it is possible to check whether there is information that was not visible in the low-bit-length image by calculating the difference between the linear interpolation of the grayscale of the low-bit-length image and the high-bit-length image (see, for example, Figure 12). Then, by utilizing the difference in the amount of grayscale information between the two images used for recognition and analysis, it is possible to analyze that the reason why an object could not be recognized in the image used for normal recognition processing is due to the bit length.

[0114] Furthermore, in the cause analysis from the perspective of the recognition unit, the cause analysis unit 2003 performs recognition processing on both the recognition data and the analysis data, and analyzes the cause of the recognition result in the recognition processing unit 104 by focusing on whether or not the recognition results for each data are consistent. For example, if high-resolution and low-bit-length images are used as recognition data and low-resolution and high-bit-length images are used as analysis data, if there is inconsistency in the recognition results for each image, it can be analyzed that there is information that was not visible in the low-bit-length image, that is, insufficient gradation is the cause of the inability to recognize the object.

[0115] While information-based causal analysis is affected by noise, cognitive-based causal analysis has the advantage of reducing the impact of noise. However, cognitive-based causal analysis has the challenge of increasing computational complexity because it requires processing both the image used for normal recognition and the image used for causal analysis separately.

[0116] E-2. Regarding the generation of control information As described above, within the control information generation unit 2004, the recognition control information generation unit 2009 generates control information for the sensor unit 102 to acquire image data for normal recognition processing (for example, high-resolution and low-bit-length images), while the analysis control information generation unit 2006 generates control information for the sensor unit 102 to acquire image data for analysis (for example, low-resolution and high-bit-length images).

[0117] The analysis control information generation unit 2006 generates control information for spatially arranging images for cause analysis within the image used for normal recognition processing. The spatial arrangement setting unit 2007 sets the spatial arrangement of the images for analysis based on the analysis results of the cause analysis unit 2003. For example, the spatial arrangement setting unit 2007 can set various spatial arrangements on the image used for normal recognition processing, such as arranging the images for analysis in line units, arranging small image blocks for analysis in a grid, arranging small image blocks for analysis in an arbitrary pattern, or dynamically generating a pattern consisting of a collection of small image blocks for analysis (see, for example, Figures 13 to 19). Basically, the spatial arrangement setting unit 2007 sets the spatial arrangement of image data for analysis based on the recognition results of the recognition processing unit 104 so that the area around where an object or other object has been recognized can be analyzed intensively.

[0118] Furthermore, the analysis control information generation unit 2006 generates control information for controlling the adjustment targets of the image data for analysis. The adjustment target setting unit 2008 sets the adjustment targets when acquiring images for analysis from the sensor unit 102, based on the analysis results of the cause analysis unit 2003. If the sensor unit 102 is an image sensor, it has characteristics such as resolution, bit length, frame rate, and shutter speed / exposure. The adjustment target setting unit 2008 sets one or more combinations of these image sensor characteristics as the adjustment targets.

[0119] The analysis control information generation unit 2006 then combines the spatial arrangement and the adjustment target to generate analysis control information and supplies it to the sensor control unit 103.

[0120] For example, the analysis control information generation unit 2006 generates analysis control information that instructs the image sensor to adjust one or more combinations of the characteristics of the sensor unit 102, such as resolution, bit length, frame rate, and shutter speed / exposure, for an image region for analysis (see, for example, Figures 13 to 16) spatially arranged on an image for normal recognition processing.

[0121] Furthermore, the analysis control information generation unit 2006 may generate control information that switches the spatial arrangement of the images for analysis every frame (see, for example, Figure 17), or control information that switches the spatial arrangement of the images for analysis within a single frame (see, for example, Figures 18 and 19).

[0122] Meanwhile, the recognition control information generation unit 2009 generates control information for the sensor unit 102 so that the recognition data acquisition unit 2001 can acquire image data for normal recognition processing (for example, high-resolution and low-bit length images) from the sensor unit 102, and supplies this information to the sensor control unit 103.

[0123] Basically, the recognition control information generation unit 2009 sets up control information for normal recognition processing based on the analysis results from the cause analysis unit 2003. That is, if the analysis results indicate that the cause of the decrease in recognition reliability in the recognition processing unit 104 is due to the performance of the sensor unit 102, the recognition control information generation unit 2009 searches for more appropriate control information so that the analysis results indicate that the performance of the sensor unit 102 is no longer the cause of the decrease in recognition reliability.

[0124] E-3. About control triggers The trigger generation unit 2005 generates a trigger based on one of the following: the recognition result or recognition reliability of the recognition processing unit 104, the analysis result of the cause analysis unit 2003, or external information supplied from outside the imaging device 100, and supplies it to the control information generation unit 2004. Specifically, when the recognition reliability of the recognition processing unit 104 is low, when the cause analysis unit 2003 outputs an analysis result indicating that the cause of the decrease in recognition reliability is the performance of the sensor unit 102, or when external information that serves as a trigger is input, the trigger generation unit 2005 generates a trigger and supplies it to the control information generation unit 2004. External information that serves as a trigger for outputting an image for analysis includes the surrounding environment of the imaging device 100 (for example, environmental information around the vehicle on which the imaging device 100 is mounted) and instructions for cause analysis input from the user.

[0125] The analysis control information generation unit 2006 within the control information generation unit 2004 performs one of the following controls, for example (1) to (4), in response to a trigger supplied from the trigger generation unit 2005.

[0126] (1) In response to a trigger, start or stop the output of images for analysis. (2) Change the spatial arrangement of the images for analysis in response to the trigger. (3) Depending on the trigger, change the target of the image adjustment for analysis. (4) Change the combination of images for analysis depending on the trigger.

[0127] E-4. Operation of the imaging device Section E-4 describes each operation performed in the imaging device 100, which is equipped with a function to analyze the cause of the recognition result, as shown in Figure 20.

[0128] E-4-1. Normal Recognition Processing Operation Figure 21 shows a flowchart illustrating the processing procedure for performing normal recognition processing in the imaging device 100 shown in Figure 20.

[0129] In carrying out this processing procedure, the sensor control unit 103 sets the sensor unit 102 to the characteristics for normal recognition processing (resolution, bit length, frame rate, shutter speed / exposure, etc.) based on the control information for normal recognition processing generated by the recognition control information generation unit 2009.

[0130] Then, the recognition data acquisition unit 2001 acquires image data from the sensor unit 102 (not shown in Figure 20) that the recognition processing unit 104 uses for normal recognition processing (step S2101). The recognition processing unit 104 receives the recognition image data acquired from the sensor unit 102 by the recognition data acquisition unit 2001, performs recognition processing of objects in the image (person detection, face recognition, image classification, etc.) (step S2102), and outputs the recognition result (step S2103).

[0131] The recognition result output by the recognition processing unit 104 includes information on the object recognized from the input image, as well as information on the recognition confidence level. The trigger generation unit 2005 receives the recognition result and checks whether the recognition confidence level is low or not (step S2104).

[0132] If the recognition confidence level is not low (No. in step S2104), the process returns to step S2101 and repeatedly executes the normal recognition process consisting of steps S2101 to S2103 until the normal recognition process is completed.

[0133] On the other hand, if the recognition confidence level is low (Yes in step S2104), the trigger generation unit 2005 generates a trigger to start an analysis of the cause of the decreased recognition confidence level (step S2105). As a result, the imaging device 100 interrupts the normal recognition process and moves to a processing operation to analyze the cause of the decreased recognition confidence level.

[0134] E-4-2. Output operation of data for analysis For example, when the trigger generation unit 2005 generates a trigger to start an analysis of the cause of the decrease in recognition reliability, the imaging device 100 starts the output processing of data for analysis. Figure 22 shows the processing procedure for outputting image data for analysis, which is performed in imaging device 100, in flowchart form.

[0135] Within the analysis control information generation unit 2006, the spatial arrangement setting unit 2007 sets the spatial arrangement of the image data for analysis based on the cause analysis results from the cause analysis unit 2003 (step S2201). In addition, the adjustment target setting unit 2008 sets the characteristics to be adjusted from among the multiple characteristics of the sensor unit 102 when outputting the image data for analysis, based on the cause analysis results from the cause analysis unit 2003 (step S2202).

[0136] Then, the control information generation unit 2004 generates control information for analysis for the sensor unit 102 based on the spatial arrangement of the image data for analysis set by the spatial arrangement setting unit 2007 and the adjustment target set by the adjustment target setting unit 2008, and outputs it to the sensor control unit 103 (step S2203).

[0137] The sensor control unit 103 controls the sensor unit 102 to perform imaging with analytical characteristics (resolution, bit length, frame rate, shutter speed / exposure, etc.) based on the analytical control information generated by the analytical control information generation unit 2009 (step S2204).

[0138] As a result of performing these processing steps, the data acquisition unit 2002 for analysis can acquire image data from the sensor unit 102 that the cause analysis unit 2003 will use to analyze the cause of the deterioration in recognition performance in the recognition processing unit 104. Then, the cause analysis processing of the recognition result, which will be explained in the following section E-4-3, is started.

[0139] E-4-3. Cause analysis processing of recognition results As described above, in response to the trigger generation unit 2005 generating a trigger, the imaging device 100 starts outputting data for analysis and processing to analyze the cause of the decreased recognition confidence. Figure 23 shows the processing procedure for analyzing the cause of the recognition result, which is performed in the imaging device 100, in flowchart form.

[0140] In carrying out this processing procedure, the sensor control unit 103 is assumed to set the sensor unit 102 to analytical characteristics (resolution, bit length, frame rate, shutter speed / exposure, etc.) based on the analytical control information generated by the analytical control information generation unit 2006. Furthermore, in the following, it is assumed that the analytical image, consisting of line units, a grid, or an arbitrary pattern, is spatially arranged so that the image for normal recognition processing and the analytical image are output simultaneously from the sensor unit 102.

[0141] The recognition data acquisition unit 2001 acquires image data from the sensor unit 102 that the recognition processing unit 104 uses for normal recognition processing (step S2301). The analysis data acquisition unit 2002 also acquires image data from the sensor unit 102 that the cause analysis unit 2003 uses for cause analysis of the recognition result by the recognition processing unit 104 (step S2302).

[0142] The recognition processing unit 104 uses the recognition image data acquired in step S2301 to perform recognition processing on objects within the image (person detection, face recognition, image classification, etc.) (step S2303) and outputs the recognition result (step S2304).

[0143] The recognition result output by the recognition processing unit 104 includes information on the object recognized from the input image, as well as information on the recognition confidence level. The trigger generation unit 2005 receives the recognition result and checks whether the recognition confidence level is low or not (step S2305).

[0144] If the recognition confidence level is not low (No. in step S2305), the trigger generation unit 2005 generates a trigger to terminate the analysis of the cause of the decreased recognition confidence level (step S2306). As a result, the imaging device 100 interrupts this cause analysis process and proceeds to the normal recognition process shown in Figure 21.

[0145] On the other hand, if the recognition confidence level is low (Yes in step S2305), the cause analysis unit 2003 uses the recognition image data acquired by the recognition data acquisition unit 2001 from the sensor unit 102 and the analysis image data acquired by the analysis data acquisition unit 2002 from the sensor unit 102 to perform a cause analysis of the current recognition result or recognition confidence level in the recognition processing unit 104 (step S2307).

[0146] If the cause analysis unit 2003 is able to determine the cause of the current recognition result or recognition reliability in the recognition processing unit 104 (Yes in step S2308), the recognition control information generation unit 2009 within the control information generation unit 2004 sets up the control information for normal recognition processing based on the cause analysis result. That is, the recognition control information generation unit 2009 modifies the control information for normal recognition processing to eliminate the cause of the decrease in recognition reliability (step S2310). Next, the trigger generation unit 2005 generates a trigger to terminate the analysis of the cause of the decrease in recognition reliability (step S2310). As a result, the imaging device 100 interrupts this cause analysis process and moves to the normal recognition process shown in Figure 21.

[0147] On the other hand, if the cause analysis unit 2003 is unable to determine the cause of the current recognition result or recognition reliability in the recognition processing unit 104 (No. in step S2308), the cause analysis process continues within the imaging device 100.

[0148] In this case, within the analysis control information generation unit 2006, the spatial arrangement setting unit 2007 sets the spatial arrangement of the image data for analysis based on the cause analysis results from the cause analysis unit 2003 (step S2311). In addition, the adjustment target setting unit 2008 sets the characteristics of the sensor unit 102 that are to be adjusted when outputting the image data for analysis, based on the cause analysis results from the cause analysis unit 2003 (step S2312).

[0149] Then, the control information generation unit 2004 generates control information for analysis for the sensor unit 102 based on the spatial arrangement of the image data for analysis set by the spatial arrangement setting unit 2007 and the adjustment target set by the adjustment target setting unit 2008, and outputs it to the sensor control unit 103 (step S2313).

[0150] The sensor control unit 103 controls the sensor unit 102 to perform imaging with analytical characteristics (resolution, bit length, frame rate, shutter speed / exposure, etc.) based on the analytical control information generated by the analytical control information generation unit 2009 (step S2314).

[0151] As a result of performing these processing steps, the data acquisition unit for analysis 2002 is able to acquire image data from the sensor unit 102 that the cause analysis unit 2003 will use to analyze the cause of the deterioration in recognition performance in the recognition processing unit 104. Therefore, the process returns to step S2301 and continues the cause analysis process.

[0152] E-4-4. How to output image data for analysis There are two methods for outputting image data for analysis: outputting it simultaneously with the normal recognition image data (see, for example, Figures 13 to 19), and outputting only the image data for analysis based on a trigger.

[0153] The former method, which outputs image data for analysis simultaneously with the image data used for normal recognition, has the advantage of allowing cause analysis to be performed without any time lag between the normal recognition process and the output. However, since image data for analysis is always output, there is a drawback in that the amount of information in the normal recognition image data is reduced accordingly.

[0154] On the other hand, the latter method, which outputs image data for analysis based on a predetermined trigger, has the challenge of a time lag between normal recognition processing and causal analysis. However, it has the advantage that the information in the normal recognition image data is hardly reduced because the image data for analysis is output only when necessary.

[0155] F. Application Fields This disclosure can be applied primarily to imaging devices 100 that sense visible light, but it can also be applied to devices that sense various types of light, such as infrared light, ultraviolet light, and X-rays. Therefore, the technology described in this disclosure can be applied to various fields to analyze the causes of recognition results and recognition reliability, and based on the analysis results, the control information of the sensor unit 102 can be set up to suit the recognition process. Figure 24 summarizes the fields to which the technology described in this disclosure can be applied.

[0156] (1) Viewing: A device for capturing images intended for viewing, such as a digital camera or a portable device with a camera function. (2) Transportation: Devices used for traffic purposes, such as on-board sensors that photograph the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognizing the driver's condition; surveillance cameras that monitor moving vehicles and roads; and distance measuring sensors that measure distances between vehicles. (3) Home appliances: A device used in home appliances such as TVs, refrigerators, air conditioners, and robots to capture user gestures and perform device operations according to those gestures. (4) Medical and healthcare: Devices used for medical and healthcare purposes, such as endoscopes and devices that perform angiography using infrared light reception. (5) Security: Security devices such as surveillance cameras for crime prevention and cameras for person recognition. (6) Beauty: Devices used for cosmetic purposes, such as skin measuring devices for photographing skin and microscopes for photographing the scalp. (7) Sports: Devices used for sports, such as action cameras and wearable cameras. (8) Agriculture: Agricultural equipment, such as cameras used to monitor the condition of fields and crops. (9) Production, manufacturing, and service industries: Devices used in production, manufacturing, or service industries, such as cameras or robots, for monitoring the status of production, manufacturing, processing, or provision of services.

[0157] G. Application Examples The technology disclosed herein can be applied to imaging devices mounted on various mobile devices such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, and robots.

[0158] Figure 25 shows a schematic configuration example of a vehicle control system 2500, which is an example of a mobile control system to which the technology described herein may be applied.

[0159] The vehicle control system 2500 comprises multiple electronic control units connected via a communication network 2520. In the example shown in Figure 25, the vehicle control system 2500 includes a drive system control unit 2521, a body system control unit 2522, an external information detection unit 2523, an internal information detection unit 2524, and an integrated control unit 2510. The functional configuration of the integrated control unit 2510 is also shown, consisting of a microcomputer 2501, an audio / image output unit 2502, and an in-vehicle network interface 2503.

[0160] The drivetrain control unit 2521 controls the operation of devices related to the vehicle's drivetrain according to various programs. The vehicle's drivetrain includes, for example, a drivetrain generating device for generating vehicle driving force, such as an internal combustion engine or a drive motor; a drivetrain transmission mechanism for transmitting driving force to the wheels; a steering mechanism for adjusting the vehicle's steering angle; and a braking device for generating vehicle braking force. The drivetrain control unit 2521 functions as a control device for these devices.

[0161] The body system control unit 2522 controls the operation of various devices mounted on the vehicle body according to various programs. The vehicle body is equipped with, for example, a keyless entry system, a smart key system, and a power window system, as well as various lamps such as headlights, reverse lights, brake lights, turn signals, or fog lights. Alternatively, the body system control unit 2522 functions as a control device for these devices mounted on the vehicle body. In this case, the body system control unit 2522 may receive radio waves transmitted from a portable device that replaces a key, or signals from various switches. The body system control unit 2522 receives these radio waves or signals and controls the vehicle's door lock system, power window system, lamps, etc.

[0162] The external information detection unit 2523 detects information from outside the vehicle equipped with the vehicle control system 2500. For example, an imaging unit 2530 is connected to the external information detection unit 2523. The external information detection unit 2523 causes the imaging unit 2530 to capture images of the outside of the vehicle and receives the captured images. Based on the images received from the imaging unit 2530, the external information detection unit 2523 may perform object detection processing such as detecting people, vehicles, obstacles, signs or road markings, or distance detection processing. For example, the external information detection unit 2523 performs image processing on the received images and performs object detection processing or distance detection processing based on the results of the image processing.

[0163] The external information detection unit 2523 performs object detection processing using a pre-trained model program designed to detect objects in images. Furthermore, if the reliability of object detection is low, the external information detection unit 2523 may analyze the cause and set up the control information for the imaging unit 2530 based on the analysis results.

[0164] The imaging unit 2530 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 2530 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 2530 may be visible light or invisible light such as infrared light. The vehicle control system 2500 assumes that the imaging unit 2530 is installed in several locations on the vehicle body. The installation locations of the imaging unit 2530 will be described later.

[0165] The in-vehicle information detection unit 2524 detects information inside the vehicle. The in-vehicle information detection unit 2524 is connected to, for example, a driver status detection unit 2540 that detects the driver's state. The driver status detection unit 2540 may include, for example, a camera that images the driver, and the in-vehicle information detection unit 2524 may calculate the driver's level of fatigue or concentration, or determine whether the driver is drowsy, based on the detection information input from the driver status detection unit 2540. The driver status detection unit 2540 may also further include a biosensor that detects biological information such as the driver's brain waves, pulse, body temperature, and breath.

[0166] The microcomputer 2501 can calculate control target values ​​for the drive force generator, steering mechanism, or braking system based on information inside and outside the vehicle acquired by the external information detection unit 2523 or the internal information detection unit 2524, and output control commands to the drive system control unit 2521. For example, the microcomputer 2501 can perform cooperative control aimed at realizing ADAS (Advanced Driver Assistance System) functions, including collision avoidance or impact mitigation, following driving based on distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning.

[0167] Furthermore, the microcomputer 2501 can perform cooperative control for purposes such as autonomous driving, where the vehicle drives autonomously without driver intervention, by controlling the drive force generating device, steering mechanism, or braking device, etc., based on information about the vehicle's surroundings acquired by the external information detection unit 2523 or the internal information detection unit 2524.

[0168] Furthermore, the microcomputer 2501 can output control commands to the body system control unit 2522 based on external information acquired by the external information detection unit 2523. For example, the microcomputer 2501 can control the headlights according to the position of a preceding or oncoming vehicle detected by the external information detection unit 2523, and perform coordinated control such as switching from high beams to low beams for the purpose of reducing glare.

[0169] The audio-image output unit 2502 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying information to the vehicle's occupants or to those outside the vehicle. In the system configuration example shown in Figure 25, the output devices include an audio speaker 2511, a display unit 2512, and an instrument panel 2513. The display unit 2512 may include, for example, at least one of an onboard display and a head-up display.

[0170] Figure 26 shows an example of the installation location of the imaging unit 2530. In the example shown in Figure 26, the vehicle 2600 has imaging units 2601, 2602, 2603, 2604, and 2605 as the imaging unit 2530.

[0171] The imaging units 2601, 2602, 2603, 2604, and 2605 are installed, for example, on the front nose, side mirrors, rear bumper, back door, and above the windshield inside the vehicle 2600. The imaging unit 2601 installed on the front nose and the imaging unit 2605 installed above the windshield inside the vehicle mainly acquire images of the area in front of the vehicle 2600. The imaging units 2602 and 2603 installed on the left and right side mirrors mainly acquire images of the left and right sides of the vehicle 2600, respectively. The imaging unit 2604 installed on the rear bumper or back door mainly acquires images of the area behind the vehicle 2600. The forward images acquired by the imaging units 2601 and 2605 are mainly used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, and road markings.

[0172] Figure 26 also illustrates the imaging ranges of each imaging unit 2601 to 2604. Imaging range 2611 indicates the imaging range of imaging unit 2601 located on the front nose, imaging ranges 2612 and 2613 indicate the imaging ranges of imaging units 2602 and 2603 located on the side mirrors, respectively, and imaging range 2614 indicates the imaging range of imaging unit 2604 located on the rear bumper or back door. For example, by superimposing the image data captured by imaging units 2601 to 2604, an overhead view image of the vehicle 2600 can be obtained.

[0173] At least one of the imaging units 2601 to 2604 may be equipped with a function for acquiring distance information. For example, at least one of the imaging units 2601 to 2604 may be a stereo camera consisting of multiple image sensors, or an image sensor having pixels for phase difference detection.

[0174] For example, the microcomputer 2501, based on distance information obtained from imaging units 2601 to 2604, can determine the distance to each object within the imaging range 2611 to 2614 and the temporal change of this distance (relative speed to the vehicle 2600). In particular, it can extract the closest object on the vehicle 2600's path that is traveling in approximately the same direction as the vehicle 2600 at a predetermined speed (e.g., 0 km / h or more) as the preceding vehicle. Furthermore, the microcomputer 2501 can set a predetermined distance to be maintained in front of the preceding vehicle and instruct the body system control unit 2522 to perform automatic braking control (including follow-and-stop control) and automatic acceleration control (including follow-and-start control). In this way, the vehicle control system 2500 can perform cooperative control aimed at autonomous driving, such as autonomous driving that does not depend on the driver's operation.

[0175] For example, the microcomputer 2501 can classify and extract three-dimensional object data based on distance information obtained from imaging units 2601 to 2604, classifying it into motorcycles, passenger cars, heavy vehicles, pedestrians, utility poles, and other three-dimensional objects, which can then be used for automatic obstacle avoidance. For instance, the microcomputer 2501 identifies obstacles around the vehicle 2600 into those visible to the driver of the vehicle 2600 and those that are difficult to see. The microcomputer 2501 then determines the collision risk, which indicates the degree of risk of collision with each obstacle. If the collision risk is above a set value and there is a possibility of collision, the microcomputer 2501 can provide driving assistance to avoid collisions with obstacles by outputting a warning to the driver via the audio speaker 2511 or display unit 2512, or by performing forced deceleration or evasive steering via the drive system control unit 2521.

[0176] At least one of the imaging units 2601 to 2604 may be an infrared camera that detects infrared light. For example, the microcomputer 2501 can recognize pedestrians by determining whether or not pedestrians are present in the images captured by the imaging units 2601 to 2604. Such pedestrian recognition is performed, for example, by a procedure to extract feature points from the images captured by the imaging units 2601 to 2604 as infrared cameras, and a procedure to perform pattern matching on a series of feature points that indicate the contour of an object to determine whether or not it is a pedestrian. When the microcomputer 2501 determines that a pedestrian is present in the images captured by the imaging units 2601 to 2604 and recognizes the pedestrian, the audio-image output unit 2502 controls the display unit 2512 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio-image output unit 2502 may also control the display unit 2512 to display an icon indicating a pedestrian or the like at a desired position. [Industrial applicability]

[0177] The present disclosure has been described in detail above with reference to specific embodiments. However, it will be obvious that those skilled in the art can modify or substitute these embodiments without departing from the gist of the present disclosure.

[0178] This specification has primarily described embodiments in which the disclosure is applied to imaging devices that sense visible light, but the gist of this disclosure is not limited thereto. Furthermore, the disclosure can also be similarly applied to devices that sense various types of light, such as infrared light, ultraviolet light, and X-rays, to analyze the limitations of recognition performance caused by sensor performance and to achieve higher recognition performance by adjusting the sensor characteristics. In addition, the technology relating to this disclosure can be applied to various fields to achieve higher recognition performance by analyzing the limitations of recognition performance caused by sensor performance.

[0179] In short, this disclosure has been explained in the form of examples, and the contents of this specification should not be interpreted restrictively. The claims should be considered in order to determine the gist of this disclosure.

[0180] Furthermore, this disclosure may also take the following form.

[0181] (1) A recognition processing unit that performs object recognition processing on sensor information from the sensor unit using a trained machine learning model, A cause analysis unit analyzes the cause of the recognition result by the recognition processing unit based on the sensor information from the sensor unit and the recognition result of the recognition processing unit, A control unit that controls the output of the sensor unit, An information processing device equipped with the following.

[0182] (2) The control unit controls the output of the sensor unit based on at least one of the recognition result of the recognition processing unit or the analysis result of the cause analysis unit. The information processing device described in (1) above.

[0183] (3) The sensor unit has a first characteristic and a second characteristic, The control unit controls whether to output to the sensor unit first sensor information that enhances the performance of the first characteristic and diminishes the performance of the second characteristic, or second sensor information that diminishes the performance of the first characteristic and enhances the performance of the second characteristic. The information processing device described in either (1) or (2) above.

[0184] (4) The sensor unit is an image sensor, The control unit controls whether to output high-resolution, low-bit-length image data for normal recognition processing, or low-resolution, high-bit-length image data for cause analysis, to the image sensor. An information processing device as described in any of (1) to (3) above.

[0185] (5) The cause analysis unit identifies the cause of the deterioration in the recognition characteristics of the recognition processing unit based on low-resolution and high-bit-length image data for cause analysis. The information processing device described in (4) above.

[0186] (6) The system further comprises a trigger generation unit that generates a trigger for output control of the sensor unit to the control unit, An information processing device as described in any of (1) to (5) above.

[0187] (7) The trigger generation unit generates the trigger based on at least one of the following: the recognition result or recognition confidence level of the recognition processing unit, the cause analysis result of the cause analysis unit, or external information provided from outside the information processing device. The information processing device described in (6) above.

[0188] (8) The control unit controls the spatial arrangement of the sensor outputs for analysis by the cause analysis unit. An information processing device as described in any of (1) to (7) above.

[0189] (9) The sensor unit is an image sensor, The control unit controls the spatial arrangement of images for analysis by the cause analysis unit. The information processing device described in (8) above.

[0190] (10) The control unit controls the arrangement of the analysis images on the normal recognition processing images in line units. The information processing device described in (9) above.

[0191] (11) The control unit controls the arrangement of the blocks of the analysis image in a grid pattern on the image for normal recognition processing. The information processing device described in (9) above.

[0192] (12) The control unit controls the arrangement of blocks of the analysis image in a predetermined pattern on the image for normal recognition processing. The information processing device described in (9) above.

[0193] (13) The control unit dynamically generates a pattern of blocks of the image for analysis on the image for normal recognition processing based on the recognition result of the recognition processing unit. The information processing device described in (9) above.

[0194] (14) The control unit controls the adjustment target for the sensor output for analysis by the cause analysis unit, among the multiple characteristics of the sensor unit. An information processing device as described in any of (1) through (13) above.

[0195] (15) The sensor unit is an image sensor, The control unit adjusts at least one or more combinations of the resolution, bit length, frame rate, or shutter speed of the image sensor. The information processing device described in (14) above.

[0196] (16) The control unit controls the setup of the sensor unit for acquiring sensor information for normal recognition processing by the recognition processing unit based on the analysis results of the cause analysis unit. An information processing device as described in any of (1) to (15) above.

[0197] (16-1) The sensor unit is an image sensor, The control unit sets at least one or more combinations of the resolution, bit length, frame rate, or shutter speed of the image sensor for acquiring an image for normal recognition processing by the recognition processing unit. The information processing device described in (16) above.

[0198] (17) The sensor unit is an image sensor, The control unit switches the sensor output for analysis by the cause analysis unit for each frame captured by the image sensor or within each frame. An information processing device as described in any of (1) to (12) above.

[0199] (18) A recognition processing step in which a trained machine learning model is used to perform object recognition processing on sensor information from the sensor unit, A cause analysis step in which the cause of the recognition result by the recognition processing unit is analyzed based on the sensor information from the sensor unit and the recognition result of the recognition processing unit, A control step for controlling the output of the sensor unit, An information processing method having

[0200] (19) A recognition processing unit that performs object recognition processing on sensor information from the sensor unit using a trained machine learning model. A cause analysis unit analyzes the cause of the recognition result by the recognition processing unit based on the sensor information from the sensor unit and the recognition result of the recognition processing unit. A control unit that controls the output of the sensor unit, A computer program written in a computer-readable format to enable a computer to function.

[0201] (20) Sensor unit, A recognition processing unit performs object recognition processing using a trained machine learning model on sensor information from the aforementioned sensor unit. A cause analysis unit analyzes the cause of the recognition result by the recognition processing unit based on sensor information from the image sensor and the recognition result of the recognition processing unit, A control unit that controls the output of the image sensor, It is equipped with, A sensor device characterized in that the sensor unit, the recognition processing unit, the cause analysis unit, and the control unit are integrated within the same semiconductor package. [Explanation of Symbols]

[0202] 100...Imaging device, 101...Optical unit, 102...Sensor unit 103...Sensor control unit, 104...Recognition processing unit, 105...Memory 106...Image processing unit, 107...Output control unit, 108...Display unit 601...Pixel array section, 602...Vertical scanning section, 603...AD conversion section 604...Horizontal scanning unit, 605...Pixel signal line, 606...Control unit 607...Signal processing unit, 610...Pixel circuit, 611...AD converter 612...Reference signal generation section 2001…Data acquisition unit for recognition, 2002…Data acquisition unit for analysis 2003…Cause Analysis Unit, 2004…Control Information Generation Unit 2005…Trigger generation unit, 2006…Control information generation unit for analysis 2007...Spatial arrangement setting unit, 2008...Adjustment target setting unit 2009... Recognition control information generation unit 2500…Vehicle control system, 2501…Microcomputer 2502...Audio and video output unit, 2503...In-vehicle network interface 2510...Integrated control unit, 2511...Audio speaker 2512…Display unit, 2513…Instrument panel 2520...Communication network, 2521...Drive system control unit 2522...Body system control unit, 2523...External vehicle information detection unit 2524...Internal information detection unit, 2530...Imaging unit 2540... Driver status detection unit

Claims

1. A control unit that controls the switching of the characteristics of the sensor unit according to the type of sensor information output from the sensor unit, A recognition processing unit performs object recognition processing using a trained machine learning model on recognition sensor information from the aforementioned sensor unit, A cause analysis unit analyzes the cause of the decrease in recognition performance by the recognition processing unit based on the analysis sensor information from the sensor unit, the recognition sensor information, and the recognition result of the recognition processing unit for the recognition sensor information. An information processing device equipped with the following.

2. The sensor unit has a plurality of characteristics, The control unit controls the switching of each characteristic of the sensor unit depending on whether the sensor unit outputs the analysis sensor information or the recognition sensor information. The information processing apparatus according to claim 1.

3. The aforementioned sensor unit has a first characteristic and a second characteristic, The control unit controls whether to output to the sensor unit first sensor information that enhances the performance of the first characteristic and reduces the performance of the second characteristic, or second sensor information that reduces the performance of the first characteristic and enhances the performance of the second characteristic. The information processing apparatus according to claim 1.

4. The aforementioned sensor unit is an image sensor, The control unit controls whether to output high-resolution, low-bit-length image data for normal recognition processing, or low-resolution, high-bit-length image data for cause analysis, to the image sensor. The information processing apparatus according to claim 1.

5. The cause analysis unit identifies the cause of the deterioration in the recognition performance of the recognition processing unit based on low-resolution, high-bit-length image data for cause analysis. The information processing apparatus according to claim 4.

6. The system further comprises a trigger generation unit that generates a trigger for generating control information to control the characteristics of the sensor unit, The information processing apparatus according to claim 1.

7. The trigger generation unit generates a trigger for generating or stopping the control information for acquiring the analysis sensor information, or for setting or changing the characteristics of the sensor unit for outputting the analysis sensor information, based on at least one of the recognition result or recognition reliability of the recognition processing unit, the cause analysis result of the cause analysis unit, or external information provided from outside the information processing device. The information processing apparatus according to claim 6.

8. The aforementioned sensor unit is an image sensor, When the cause analysis unit performs analysis, the control unit controls the characteristics of the image sensor regarding the position on the image for analysis by the cause analysis unit within one frame of the image captured by the image sensor, on the image for recognition processing by the recognition processing unit. The information processing apparatus according to claim 1.

9. The image used for recognition processing by the recognition processing unit is image data output from the image sensor at high resolution and low bit length, while the image used for analysis by the cause analysis unit is image data output from the image sensor at low resolution and high bit length. The information processing apparatus according to claim 8.

10. When the control unit performs analysis by the cause analysis unit, it controls the characteristics of the image sensor so that the images for analysis by the cause analysis unit are arranged line by line on the images for recognition processing by the recognition processing unit within one frame of the image captured by the image sensor. The information processing apparatus according to claim 9.

11. When the cause analysis unit performs analysis, the control unit controls the characteristics of the image sensor so that blocks of images for analysis by the cause analysis unit are arranged in a grid pattern on the image for recognition processing by the recognition processing unit within one frame of the image captured by the image sensor. The information processing apparatus according to claim 9.

12. When the cause analysis unit performs analysis, the control unit controls the characteristics of the image sensor so that, within one frame of the image captured by the image sensor, blocks of images for analysis by the cause analysis unit are arranged in a predetermined pattern on the image for recognition processing by the recognition processing unit. The information processing apparatus according to claim 9.

13. When the cause analysis unit performs analysis, the control unit dynamically generates a pattern of blocks of the image for analysis by the cause analysis unit on the image for recognition processing by the recognition processing unit, based on the recognition result of the recognition processing unit. The information processing apparatus according to claim 12.

14. The control unit controls, based on whether or not to perform analysis by the cause analysis unit, to switch the characteristics among the multiple characteristics of the sensor unit that are to be adjusted for the sensor output for analysis by the cause analysis unit. The information processing apparatus according to claim 1.

15. The aforementioned sensor unit is an image sensor, The control unit selects at least one or more combinations of characteristics of the image sensor, such as resolution, bit length, frame rate, or shutter speed, as the target of adjustment. The information processing apparatus according to claim 14.

16. The control unit controls the setup of the sensor unit for acquiring sensor information for recognition processing by the recognition processing unit when the cause analysis unit does not perform analysis, based on the analysis results of the cause analysis unit. The information processing apparatus according to claim 1.

17. The aforementioned sensor unit is an image sensor, The control unit switches the sensor output for analysis by the cause analysis unit for each frame captured by the image sensor or within each frame. The information processing apparatus according to claim 1.

18. A control step in which the control unit controls switching the characteristics of the sensor unit according to the type of sensor information output from the sensor unit, The recognition processing step involves the recognition processing unit performing object recognition processing on the recognition sensor information from the sensor unit using a trained machine learning model, The cause analysis unit performs a cause analysis step in which it analyzes the cause of the deterioration in recognition performance by the recognition processing unit based on the analysis sensor information from the sensor unit, the recognition sensor information, and the recognition result of the recognition processing unit for the recognition sensor information. An information processing method having

19. A control unit that controls the switching of the characteristics of the sensor unit according to the type of sensor information output from the sensor unit, A recognition processing unit performs object recognition processing using a trained machine learning model on the recognition sensor information from the aforementioned sensor unit. A cause analysis unit analyzes the cause of the decrease in recognition performance by the recognition processing unit based on the analysis sensor information from the sensor unit, the recognition sensor information, and the recognition result of the recognition processing unit for the recognition sensor information. A computer program written in a computer-readable format to enable a computer to function.

20. A sensor unit consisting of an image sensor, A control unit that controls the switching of the characteristics of the sensor unit according to the type of sensor information output from the sensor unit, A recognition processing unit performs object recognition processing using a trained machine learning model on recognition sensor information from the aforementioned sensor unit, A cause analysis unit analyzes the cause of the decrease in recognition performance by the recognition processing unit based on the analysis sensor information from the sensor unit, the recognition sensor information, and the recognition result of the recognition processing unit for the recognition sensor information. It is equipped with, A sensor device characterized in that the sensor unit, the recognition processing unit, the cause analysis unit, and the control unit are integrated within the same semiconductor package.

Images