Information processing apparatus, information processing system, information processing method, and information processing program

Abstract

[Problem] One aspect of the present disclosure addresses the problem of providing an information processing device, an information processing system, an information processing method, and an information processing program capable of suppressing a deterioration in the recognition rate of recognition processing, even if recognition processing is performed using data from an area constituting a portion of image data. [Solution] This information processing device is provided with: a reading unit which sets read pixels as a portion of a pixel area in which a plurality of pixels are arranged in a two-dimensional array, and controls reading of pixel signals from the pixels contained in the pixel area; and a setting unit which sets the read pixels on the basis of color filter arrangement information relating to the pixel area.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and an information processing program.

Background Art

[0002] In recent years, with the improvement in performance of imaging devices such as small cameras mounted on digital still cameras, digital video cameras, multifunctional mobile phones (smartphones), etc., information processing apparatuses equipped with an image recognition function for recognizing a predetermined object included in an imaged image have been developed. Further, a color filter such as a Bayer array is arranged on the sensor of the imaging device. However, when performing recognition processing using data in a partial region of image data, the position and number of filters included in each region are different, and there is a risk that the recognition rate of the recognition processing will decrease.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] One aspect of the present disclosure provides an information processing apparatus, an information processing system, an information processing method, and an information processing program capable of suppressing a decrease in the recognition rate of recognition processing even when performing recognition processing using data in a partial region of image data.

Means for Solving the Problems

[0005] In order to solve the above problems, in the present disclosure, readout pixels are set as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and a readout unit that controls the readout of pixel signals from the pixels included in the pixel region, A setting unit that sets the read pixels based on the color filter array information of the pixel region; An information processing apparatus including the above is provided.

[0006] The setting unit may set the read pixels based on external information.

[0007] The external information may be at least any one of a recognition result, map information, vehicle information, and external sensor information.

[0008] The setting unit may set priorities for at least two of the recognition result, map information, vehicle information, and external sensor information, and set the read pixels based on the set multiple priorities.

[0009] At least some of the plurality of pixels are provided with a polarizing filter, and the setting unit may set the priority of the pixels provided with the polarizing filter based on at least any one of the recognition result, map information, vehicle information, and external sensor information.

[0010] At least some of the plurality of pixels are provided with a spectroscopic filter, The setting unit may set the priority of the pixels provided with the spectroscopic filter based on at least any one of the recognition result, map information, vehicle information, and external sensor information.

[0011] At least some of the plurality of pixels are provided with an infrared filter, The setting unit may set the priority of the pixels provided with the infrared filter based on at least any one of the recognition result, map information, vehicle information, and external sensor information.

[0012] A sensor unit in which a plurality of pixels are arranged in a two-dimensional array; A sensor control unit that controls the sensor unit; A recognition processing unit, and an information processing system including the above, The sensor control unit A readout unit is provided that sets readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controls the readout of pixel signals from the pixels included in the pixel region. The recognition processing unit includes a setting unit that sets the readout pixels based on the color filter array information of the pixel region. An information processing system is provided.

[0013] To solve the above problems, one aspect of the present disclosure includes a readout step of setting readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controlling the readout of pixel signals from the pixels included in the pixel region. A setting step of setting the readout pixels based on the color filter array information of the pixel region. An information processing method is provided.

[0014] To solve the above problems, one aspect of the present disclosure includes a readout step of setting readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controlling the readout of pixel signals from the pixels included in the pixel region. A setting step of setting the readout pixels based on the color filter array information of the pixel region. A program is provided that causes a computer to execute the above steps.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 6A

Figure 6B

Figure 6C

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 8C

Figure 9A

Figure 9B

Figure 10

Figure 11

Figure 12A

Figure 12B

Figure 13A

Figure 13B

Figure 14A

Figure 14B

Figure 15A

Figure 15B

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of an information processing apparatus, an information processing system, an information processing method, and an information processing program will be described with reference to the drawings. Hereinafter, the description will focus on the main components of the information processing apparatus, the information processing system, the information processing method, and the information processing program. However, the information processing apparatus, the information processing system, the information processing method, and the information processing program may have components and functions that are not illustrated or described. The following description does not exclude components and functions that are not illustrated or described.

[0017] [1. Configuration examples according to each embodiment of the present disclosure] An overall configuration example of the information processing system according to each embodiment will be schematically described. FIG. 1 is a block diagram showing a configuration of an example of the information processing system 1. In FIG. 1, the information processing system 1 includes a sensor unit 10, a sensor control unit 11, a recognition processing unit 12, a memory 13, a visual recognition processing unit 14, and an output control unit 15. These units are, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor (CIS) integrally formed using CMOS. Note that the information processing system 1 is not limited to this example, and may be another type of optical sensor such as an infrared light sensor that performs imaging using infrared light. Further, the sensor control unit 11, the recognition processing unit 12, the memory 13, the visual recognition processing unit 14, and the output control unit 15 constitute the information processing apparatus 1a.

[0018] The sensor unit 10 outputs a pixel signal corresponding to the light irradiated on the light receiving surface via the optical unit 30. More specifically, the sensor unit 10 has a pixel array in which pixels including at least one photoelectric conversion element are arranged in a matrix. The light receiving surface is formed by each pixel arranged in a matrix in the pixel array. The sensor unit 10 further includes a drive circuit for driving each pixel included in the pixel array, and a signal processing circuit that performs predetermined signal processing on the signal read from each pixel and outputs it as a pixel signal of each pixel. The sensor unit 10 outputs the pixel signals of the respective pixels included in the pixel region as digital-form image data.

[0019] Hereinafter, in the pixel array of the sensor unit 10, the area where effective pixels for generating pixel signals are arranged is referred to as a frame. Frame image data is formed from pixel data based on each pixel signal output from each pixel included in the frame. Also, each row in the pixel array of the sensor unit 10 is respectively referred to as a line, and line image data is formed from pixel data based on the pixel signals output from each pixel included in the line. Further, the operation of the sensor unit 10 to output a pixel signal corresponding to the light irradiated on the light receiving surface is referred to as imaging. The sensor unit 10 is controlled for the exposure during imaging and the gain (analog gain) for the pixel signal in accordance with the imaging control signal supplied from the sensor control unit 11 described later.

[0020] The sensor control unit 11 is constituted by, for example, a microprocessor, controls the reading of pixel data from the sensor unit 10, and outputs pixel data based on each pixel signal read from each pixel included in the frame. The pixel data output from the sensor control unit 11 is supplied to the recognition processing unit 12 and the visual recognition processing unit 14.

[0021] Also, the sensor control unit 11 generates an imaging control signal for controlling imaging in the sensor unit 10. The sensor control unit 11 generates the imaging control signal in accordance with instructions from, for example, the recognition processing unit 12 and the visual recognition processing unit 14 described later. The imaging control signal includes information indicating the exposure and the analog gain during imaging in the sensor unit 10 described above. The imaging control signal further includes control signals (vertical synchronization signal, horizontal synchronization signal, etc.) used for the sensor unit 10 to perform the imaging operation. The sensor control unit 11 supplies the generated imaging control signal to the sensor unit 10.

[0022] The optical unit 30 is for irradiating the light from the subject onto the light-receiving surface of the sensor unit 10, and is disposed, for example, at a position corresponding to the sensor unit 10. The optical unit 30 includes, for example, a plurality of lenses, a diaphragm mechanism for adjusting the size of the aperture with respect to the incident light, and a focus mechanism for adjusting the focus of the light irradiated onto the light-receiving surface. The optical unit 30 may further include a shutter mechanism (mechanical shutter) for adjusting the time during which the light-receiving surface is irradiated with light. The diaphragm mechanism, focus mechanism, and shutter mechanism included in the optical unit 30 can be controlled, for example, by the sensor control unit 11. Not limited to this, the diaphragm and focus in the optical unit 30 can also be controlled from outside the information processing system 1. Further, it is also possible to integrally configure the optical unit 30 with the information processing system 1.

[0023] The recognition processing unit 12 performs recognition processing of an object included in the image formed by the pixel data based on the pixel data supplied from the sensor control unit 11. In the present disclosure, for example, a DSP (Digital Signal Processor) reads out and executes a program that has been pre-learned with teacher data and stored as a learning model in the memory 13, thereby performing recognition processing using a DNN (Deep Neural Network), and the recognition processing unit 12 as a machine learning unit is configured. The recognition processing unit 12 can instruct the sensor control unit 11 to read out the pixel data necessary for the recognition processing from the sensor unit 10. The recognition result by the recognition processing unit 12 is supplied to the output control unit 15.

[0024] The visual recognition processing unit 14 executes processing for obtaining an image suitable for human visual recognition on the pixel data supplied from the sensor control unit 11, and outputs, for example, image data composed of a set of pixel data. For example, the visual recognition processing unit 14 is configured by an ISP (Image Signal Processor) reading out and executing a program pre-stored in a memory (not shown).

[0025] For example, if each pixel included in the sensor unit 10 is provided with a color filter and the pixel data has color information of R (red), G (green), and B (blue), the demosaicing process, white balance process, etc. can be executed. Further, the visual recognition processing unit 14 can instruct the sensor control unit 11 to read out the pixel data necessary for the visual recognition processing from the sensor unit 10. The image data obtained by subjecting the pixel data to image processing by the visual recognition processing unit 14 is supplied to the output control unit 15.

[0026] The output control unit 15 is constituted by, for example, a microprocessor, and outputs one or both of the recognition result supplied from the recognition processing unit 12 and the image data supplied as the visual recognition result from the visual recognition processing unit 14 to the outside of the information processing system 1. The output control unit 15 can output the image data to, for example, a display unit 31 having a display device. Thereby, the user can visually recognize the image data displayed by the display unit 31. Note that the display unit 31 may be built in the information processing system 1 or may be an external configuration of the information processing system 1.

[0027] FIGS. 2A and 2B are schematic diagrams showing examples of the hardware configuration of the information processing system 1 according to each embodiment. FIG. 2A shows an example in which the sensor unit 10, the sensor control unit 11, the recognition processing unit 12, the memory 13, the visual recognition processing unit 14, and the output control unit 15 among the configurations shown in FIG. 1 are mounted on one chip 2. In FIG. 2A, the memory 13 and the output control unit 15 are omitted to avoid complexity.

[0028] In the configuration shown in FIG. 2A, the recognition result by the recognition processing unit 12 is output to the outside of the chip 2 via an output control unit 15 (not shown). Also, in the configuration of FIG. 2A, the recognition processing unit 12 can acquire the pixel data for recognition from the sensor control unit 11 via the interface inside the chip 2.

[0029] Figure 2B shows an example in which, for one chip 2, the sensor unit 10, the sensor control unit 11, the visual recognition processing unit 14, and the output control unit 15 of the configuration shown in FIG. 1 are mounted, and the recognition processing unit 12 and the memory 13 (not shown) are placed outside the chip 2. Also in FIG. 2B, similar to FIG. 2A described above, the memory 13 and the output control unit 15 are omitted to avoid complexity.

[0030] In the configuration of this FIG. 2B, the recognition processing unit 12 acquires pixel data for recognition via an interface for inter-chip communication. Also, in FIG. 2B, it is shown that the recognition result by the recognition processing unit 12 is directly output to the outside from the recognition processing unit 12, but this is not limited to this example. That is, in the configuration of FIG. 2B, the recognition processing unit 12 may return the recognition result to the chip 2 and cause it to be output from an output control unit (not shown) mounted on the chip 2.

[0031] In the configuration shown in FIG. 2A, the recognition processing unit 12 is mounted on the chip 2 together with the sensor control unit 11, and communication between the recognition processing unit 12 and the sensor control unit 11 can be executed at high speed by an interface inside the chip 2. On the other hand, in the configuration shown in FIG. 2A, the recognition processing unit 12 cannot be replaced, and it is difficult to change the recognition processing. In contrast, in the configuration shown in FIG. 2B, since the recognition processing unit 12 is provided outside the chip 2, communication between the recognition processing unit 12 and the sensor control unit 11 needs to be performed via an inter-chip interface. Therefore, communication between the recognition processing unit 12 and the sensor control unit 11 becomes slower compared to the configuration of FIG. 2A, and there is a possibility of a delay in control. On the other hand, the recognition processing unit 12 can be easily replaced, and various recognition processes can be realized.

[0032] Hereinafter, unless otherwise specified, the information processing system 1 adopts the configuration in which the sensor unit 10, the sensor control unit 11, the recognition processing unit 12, the memory 13, the visual recognition processing unit 14, and the output control unit 15 are mounted on one chip 2 as shown in FIG. 2A.

[0033] In the configuration shown in FIG. 2A described above, the information processing system 1 can be formed on one substrate. Not limited to this, the information processing system 1 may be a stacked CIS in which a plurality of semiconductor chips are stacked and integrally formed.

[0034] As an example, the information processing system 1 can be formed by a two-layer structure in which semiconductor chips are stacked in two layers. FIG. 3A is a diagram showing an example in which the information processing system 1 according to each embodiment is formed by a stacked CIS having a two-layer structure. In the structure of FIG. 3A, the pixel portion 20a is formed on the semiconductor chip of the first layer, and the memory + logic portion 20b is formed on the semiconductor chip of the second layer. The pixel portion 20a includes at least the pixel array in the sensor portion 10. The memory + logic portion 20b includes, for example, the sensor control portion 11, the recognition processing portion 12, the memory 13, the visual recognition processing portion 14, and the output control portion 15, and an interface for communicating between the information processing system 1 and the outside. The memory + logic portion 20b further includes a part or all of a drive circuit that drives the pixel array in the sensor portion 10. Although not shown, the memory + logic portion 20b can further include, for example, a memory used by the visual recognition processing portion 14 for processing image data.

[0035] As shown on the right side of FIG. 3A, the information processing system 1 is configured as one solid-state imaging device by bonding the semiconductor chip of the first layer and the semiconductor chip of the second layer while bringing them into electrical contact.

[0036] As another example, the information processing system 1 can be formed by a three-layer structure in which semiconductor chips are stacked in three layers. FIG. 3B is a diagram showing an example in which the information processing system 1 according to each embodiment is formed by a stacked-type CIS having a three-layer structure. In the structure of FIG. 3B, a pixel portion 20a is formed on the semiconductor chip of the first layer, a memory portion 20c is formed on the semiconductor chip of the second layer, and a logic portion 20b is formed on the semiconductor chip of the third layer. In this case, the logic portion 20b includes, for example, a sensor control portion 11, a recognition processing portion 12, a visual recognition processing portion 14, and an output control portion 15, and an interface for communicating between the information processing system 1 and the outside. Further, the memory portion 20c can include a memory 13 and a memory used by, for example, the visual recognition processing portion 14 for processing image data. The memory 13 may be included in the logic portion 20b.

[0037] As shown on the right side of FIG. 3B, the information processing system 1 is configured as one solid-state imaging device by bonding the semiconductor chip of the first layer, the semiconductor chip of the second layer, and the semiconductor chip of the third layer while bringing them into electrical contact with each other.

[0038] FIG. 4 is a block diagram showing a configuration of an example of the sensor portion 10 applicable to each embodiment. In FIG. 4, the sensor portion 10 includes a pixel array portion 101, a vertical scanning portion 102, an AD (Analog to Digital) conversion portion 103, pixel signal lines 106, vertical signal lines VSL, a control portion 1100, and a signal processing portion 1101. In FIG. 4, the control portion 1100 and the signal processing portion 1101 can also be included in, for example, the sensor control portion 11 shown in FIG. 1.

[0039] The pixel array unit 101 includes a plurality of pixel circuits 100 each including a photoelectric conversion element such as a photodiode that performs photoelectric conversion on the received light, and a circuit that reads out charges from the photoelectric conversion element. In the pixel array unit 101, the plurality of pixel circuits 100 are arranged in a matrix in the horizontal direction (row direction) and the vertical direction (column direction). In the pixel array unit 101, the arrangement of the pixel circuits 100 in the row direction is called a line. For example, when an image of one frame is formed with 1920 pixels × 1080 lines, the pixel array unit 101 includes at least 1080 lines each including at least 1920 pixel circuits 100. An image (image data) of one frame is formed by pixel signals read out from the pixel circuits 100 included in the frame.

[0040] Hereinafter, the operation of reading out pixel signals from each pixel circuit 100 included in a frame in the sensor unit 10 will be described as appropriately reading out pixels from the frame, etc. Also, the operation of reading out pixel signals from each pixel circuit 100 included in the lines included in a frame will be described as appropriately reading out the lines, etc.

[0041] Also, in the pixel array unit 101, pixel signal lines 106 are connected to each row and column of each pixel circuit 100, and vertical signal lines VSL are connected to each column. The end of the pixel signal line 106 that is not connected to the pixel array unit 101 is connected to the vertical scanning unit 102. The vertical scanning unit 102 transmits control signals such as drive pulses when reading out pixel signals from the pixels to the pixel array unit 101 via the pixel signal lines 106 in accordance with the control of a control unit 1100 described later. The end of the vertical signal line VSL that is not connected to the pixel array unit 101 is connected to the AD conversion unit 103. The pixel signals read out from the pixels are transmitted to the AD conversion unit 103 via the vertical signal lines VSL.

[0042] The readout control of the pixel signal from the pixel circuit 100 will be schematically described. The readout of the pixel signal from the pixel circuit 100 is performed by transferring the charge accumulated in the photoelectric conversion element due to exposure to the floating diffusion layer (FD; Floating Diffusion) and converting the charge transferred in the floating diffusion layer into a voltage. The voltage at which the charge is converted in the floating diffusion layer is output to the vertical signal line VSL via an amplifier.

[0043] More specifically, in the pixel circuit 100, during exposure, the photoelectric conversion element and the floating diffusion layer are set to an off (open) state, and the photoelectric conversion element accumulates the charge generated in response to the incident light by photoelectric conversion. After the exposure ends, the floating diffusion layer and the vertical signal line VSL are connected according to the selection signal supplied via the pixel signal line 106. Further, according to the reset pulse supplied via the pixel signal line 106, the floating diffusion layer is connected to the power supply voltage VDD or the supply line of the black level voltage for a short period to reset the floating diffusion layer. The voltage of the reset level of the floating diffusion layer (referred to as voltage A) is output to the vertical signal line VSL. Thereafter, the transfer pulse supplied via the pixel signal line 106 sets the photoelectric conversion element and the floating diffusion layer to an on (closed) state to transfer the charge accumulated in the photoelectric conversion element to the floating diffusion layer. A voltage (referred to as voltage B) corresponding to the charge amount of the floating diffusion layer is output to the vertical signal line VSL.

[0044] The AD conversion unit 103 includes an AD converter 107 provided for each vertical signal line VSL, a reference signal generation unit 104, and a horizontal scanning unit 105. The AD converter 107 is a column AD converter that performs AD conversion processing for each column of the pixel array unit 101. The AD converter 107 performs AD conversion processing on the pixel signal supplied from the pixel circuit 100 via the vertical signal line VSL and generates two digital values (values corresponding to voltage A and voltage B respectively) for correlated double sampling (CDS: Correlated Double Sampling) processing for noise reduction.

[0045] The AD converter 107 supplies the two generated digital values to the signal processing unit 1101. The signal processing unit 1101 performs CDS processing based on the two digital values supplied from the AD converter 107, and generates a pixel signal (pixel data) by a digital signal. The pixel data generated by the signal processing unit 1101 is output to the outside of the sensor unit 10.

[0046] The reference signal generation unit 104 generates, as a reference signal, a ramp signal used by each AD converter 107 to convert a pixel signal into two digital values based on a control signal input from the control unit 1100. The ramp signal is a signal whose level (voltage value) decreases at a constant slope with respect to time, or a signal whose level decreases stepwise. The reference signal generation unit 104 supplies the generated ramp signal to each AD converter 107. The reference signal generation unit 104 is configured by using, for example, a DAC (Digital to Analog Converter) or the like.

[0047] When a ramp signal whose voltage drops stepwise according to a predetermined slope is supplied from the reference signal generation unit 104, the counter starts counting according to the clock signal. The comparator compares the voltage of the pixel signal supplied from the vertical signal line VSL with the voltage of the ramp signal, and stops the counting by the counter at the timing when the voltage of the ramp signal crosses the voltage of the pixel signal. The AD converter 107 converts the pixel signal by an analog signal into a digital value by outputting a value corresponding to the count value of the time when the counting is stopped.

[0048] The AD converter 107 supplies the two generated digital values to the signal processing unit 1101. The signal processing unit 1101 performs CDS processing based on the two digital values supplied from the AD converter 107, and generates a pixel signal (pixel data) by a digital signal. The pixel signal by the digital signal generated by the signal processing unit 1101 is output to the outside of the sensor unit 10.

[0049] Under the control of the control unit 1100, the horizontal scanning unit 105 performs a selection scan to select each AD converter 107 in a predetermined order, thereby sequentially outputting to the signal processing unit 1101 each digital value temporarily held by each AD converter 107. The horizontal scanning unit 105 is configured using, for example, a shift register, an address decoder, or the like.

[0050] The control unit 1100 performs drive control of the vertical scanning unit 102, the AD conversion unit 103, the reference signal generation unit 104, the horizontal scanning unit 105, etc. according to the imaging control signal supplied from the sensor control unit 11. The control unit 1100 generates various drive signals that serve as the basis for the operations of the vertical scanning unit 102, the AD conversion unit 103, the reference signal generation unit 104, and the horizontal scanning unit 105. For example, the control unit 1100 generates a control signal for the vertical scanning unit 102 to supply to each pixel circuit 100 via the pixel signal line 106 based on the vertical synchronization signal or the external trigger signal included in the imaging control signal and the horizontal synchronization signal. The control unit 1100 supplies the generated control signal to the vertical scanning unit 102.

[0051] Also, the control unit 1100 outputs, for example, information indicating an analog gain included in the imaging control signal supplied from the sensor control unit 11 to the AD conversion unit 103. The AD conversion unit 103 controls the gain of the pixel signal input to each AD converter 107 included in the AD conversion unit 103 via the vertical signal line VSL according to the information indicating this analog gain.

[0052] Based on the control signal supplied from the control unit 1100, the vertical scanning unit 102 supplies various signals including drive pulses to the pixel signal lines 106 of the selected pixel rows of the pixel array unit 101 to each pixel circuit 100 line by line, and causes each pixel circuit 100 to output a pixel signal to the vertical signal line VSL. The vertical scanning unit 102 is configured using, for example, a shift register, an address decoder, or the like. Also, the vertical scanning unit 102 controls the exposure in each pixel circuit 100 according to the information indicating the exposure supplied from the control unit 1100.

[0053] The sensor unit 10 configured as described above is a column AD type CMOS (Complementary Metal Oxide Semiconductor) image sensor in which the AD converters 107 are arranged for each column.

[0054] [2. Examples of Existing Technologies Applicable to the Present Disclosure] Prior to the description of each embodiment according to the present disclosure, for the sake of easy understanding, the existing technologies applicable to the present disclosure will be briefly described.

[0055] (2-1. Overview of Rolling Shutter) As imaging methods when imaging is performed by the pixel array unit 101, a rolling shutter (RS) method and a global shutter (GS) method are known. First, the rolling shutter method will be briefly described. FIGS. 5A, 5B, and 5C are schematic diagrams for explaining the rolling shutter method. In the rolling shutter method, as shown in FIG. 5A, imaging is performed in order by line from, for example, the line 201 at the upper end of the frame 200.

[0056] Note that, in the above description, "imaging" has been described as referring to the operation of the sensor unit 10 outputting a pixel signal corresponding to the light irradiated on the light receiving surface. More specifically, "imaging" refers to a series of operations from performing exposure in a pixel to transferring a pixel signal based on the charge accumulated by exposure in a photoelectric conversion element included in the pixel to the sensor control unit 11. Also, as described above, a frame refers to a region in the pixel array unit 101 where the effective pixel circuits 100 for generating pixel signals are arranged.

[0057] For example, in the configuration of FIG. 4, exposure is simultaneously executed in each pixel circuit 100 included in one line. After the end of the exposure, the pixel signals based on the charge accumulated by the exposure are simultaneously transferred from each pixel circuit 100 included in the line through the respective vertical signal lines VSL corresponding to each pixel circuit 100. By sequentially executing this operation in units of lines, imaging by the rolling shutter can be realized.

[0058] FIG. 5B schematically shows an example of the relationship between imaging and time in the rolling shutter method. In FIG. 5B, the vertical axis represents the line position and the horizontal axis represents time. In the rolling shutter method, since the exposure for each line is performed sequentially in line order, as shown in FIG. 5B, the exposure timing for each line will shift in order according to the position of the line. Therefore, for example, when the horizontal positional relationship between the information processing system 1 and the subject changes rapidly, as illustrated in FIG. 5C, distortion occurs in the image of the captured frame 200. In the example of FIG. 5C, the image 202 corresponding to the frame 200 is an image tilted at an angle according to the speed and direction of the change in the horizontal positional relationship between the information processing system 1 and the subject.

[0059] In the rolling shutter method, it is also possible to perform imaging while thinning out lines. FIGS. 6A, 6B, and 6C are schematic diagrams for explaining line thinning in the rolling shutter method. As shown in FIG. 6A, similar to the example of FIG. 5A described above, imaging is performed in line units from the line 201 at the upper end of the frame 200 toward the lower end of the frame 200. At this time, imaging is performed while skipping lines every predetermined number.

[0060] Here, for the sake of explanation, it is assumed that imaging is performed every other line by thinning out one line. That is, after imaging the nth line, imaging of the (n + 2)th line is performed. At this time, it is assumed that the time from imaging of the nth line to imaging of the (n + 2)th line is equal to the time from imaging of the nth line to imaging of the (n + 1)th line when no thinning is performed.

[0061] FIG. 6B schematically shows an example of the relationship between imaging and time when line skipping is performed in the rolling shutter method. In FIG. 6B, the vertical axis represents the line position, and the horizontal axis represents time. In FIG. 6B, exposure A corresponds to the exposure in FIG. 5B where no line skipping is performed, and exposure B shows the exposure when one-line skipping is performed. As shown in exposure B, by performing line skipping, the deviation in the exposure timing at the same line position can be reduced compared to the case where no line skipping is performed. Therefore, as exemplified as image 203 in FIG. 6C, the distortion in the tilt direction generated in the image of the captured frame 200 becomes smaller compared to the case where no line skipping is performed as shown in FIG. 5C. On the other hand, when line skipping is performed, the resolution of the image becomes lower compared to the case where no line skipping is performed.

[0062] In the above description, an example of sequentially imaging lines from the upper end to the lower end of frame 200 in the rolling shutter method has been described, but this is not limited to this example. FIGS. 7A and 7B are diagrams schematically showing examples of other imaging methods in the rolling shutter method. For example, as shown in FIG. 7A, in the rolling shutter method, sequential imaging of lines can be performed from the lower end to the upper end of frame 200. In this case, compared to the case of sequentially imaging lines from the upper end to the lower end of frame 200, the horizontal direction of the distortion of image 202 is reversed.

[0063] Also, for example, by setting the range of the vertical signal line VSL that transfers the pixel signal, it is possible to selectively read out a part of the lines. Furthermore, by setting the lines for imaging and the vertical signal line VSL that transfers the pixel signal respectively, it is also possible to set the lines for starting and ending imaging to be other than the upper end and the lower end of frame 200. FIG. 7B schematically shows an example in which a rectangular region 205 whose width and height are less than the width and height of frame 200 respectively is used as the imaging range. In the example of FIG. 7B, imaging is performed sequentially from line 204 at the upper end of region 205 toward the lower end of region 205.

[0064] (2-2. Overview of Global Shutter) Next, as an imaging method when imaging is performed by the pixel array unit 101, the global shutter (GS) method will be schematically described. FIGS. 8A, 8B, and 8C are schematic diagrams for explaining the global shutter method. In the global shutter method, as shown in FIG. 8A, exposure is simultaneously performed in the pixel circuits 100 included in the frame 200.

[0065] When realizing the global shutter method in the configuration of FIG. 4, as an example, it is conceivable to adopt a configuration in which a capacitor is further provided between the photoelectric conversion element and the FD in each pixel circuit 100. Then, a first switch is provided between the photoelectric conversion element and the capacitor, and a second switch is provided between the capacitor and the floating diffusion layer, respectively, and the opening and closing of each of these first and second switches are controlled by pulses supplied via the pixel signal line 106.

[0066] In such a configuration, during the exposure period, in the pixel circuits 100 included in the frame 200, the first and second switches are each opened, and at the end of the exposure, the first switch is changed from open to closed to transfer the charge from the photoelectric conversion element to the capacitor. Thereafter, regarding the capacitor as a photoelectric conversion element, the charge is read out from the capacitor in the same sequence as the readout operation described in the rolling shutter method. As a result, simultaneous exposure is possible in the pixel circuits 100 included in the frame 200.

[0067] FIG. 8B schematically shows an example of the relationship between imaging and time in the global shutter method. In FIG. 8B, the vertical axis represents the line position, and the horizontal axis represents time. In the global shutter method, since exposure is simultaneously performed in the pixel circuits 100 included in the frame 200, as shown in FIG. 8B, the exposure timing can be made the same for each line. Therefore, for example, even when the horizontal positional relationship between the information processing system 1 and the subject changes at high speed, as illustrated in FIG. 8C, no distortion corresponding to the change occurs in the image 206 of the captured frame 200.

[0068] In the global shutter method, the simultaneity of the exposure timing in the pixel circuit 100 included in the frame 200 can be ensured. Therefore, by controlling the timing of each pulse supplied by the pixel signal lines 106 of each line and the timing of transfer by each vertical signal line VSL, sampling (reading out of pixel signals) in various patterns can be realized.

[0069] FIGS. 9A and 9B are diagrams schematically showing examples of sampling patterns that can be realized in the global shutter method. FIG. 9A is an example of extracting samples 208 for reading pixel signals in a checkerboard pattern from each pixel circuit 100 arranged in a matrix included in the frame 200. FIG. 9B is an example of extracting samples 208 for reading pixel signals in a grid pattern from each of the pixel circuits 100. Also, in the global shutter method as well, imaging can be performed line by line in the same manner as the rolling shutter method described above.

[0070] (2-3.DNN) Next, the recognition processing using DNN (Deep Neural Network) applicable to each embodiment will be schematically described. In each embodiment, among DNNs, CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) are used to perform recognition processing on image data. Hereinafter, the "recognition processing on image data" will be appropriately referred to as "image recognition processing" and the like.

[0071] (2-3-1.Outline of CNN) First, CNN will be schematically described. Image recognition processing by CNN generally performs image recognition processing based on image information composed of pixels arranged, for example, in a matrix. FIG. 10 is a diagram for schematically explaining image recognition processing by CNN. Processing by a predetermined CNN 52 is performed on the entire pixel information 51 of an image 50 depicting the number "8", which is an object to be recognized. As a result of this, the number "8" is recognized as the recognition result 53.

[0072] On the other hand, it is also possible to perform processing by a CNN based on the image for each line and obtain a recognition result from a part of the image to be recognized. FIG. 11 is a diagram for schematically explaining the image recognition processing for obtaining a recognition result from a part of the image to be recognized. In FIG. 11, the image 50' is obtained by partially acquiring the number "8", which is the object to be recognized, in line units. For example, the pixel information 54a, 54b, and 54c for each line forming the pixel information 51' of this image 50' are sequentially subjected to processing by the CNN 52' learned in advance.

[0073] For example, it is assumed that the recognition result 53a obtained by the recognition processing by the CNN 52' for the pixel information 54a of the first line is not a valid recognition result. Here, a valid recognition result refers to, for example, a recognition result in which the score indicating the reliability for the recognized result is equal to or higher than a predetermined value. Note that the reliability according to the present embodiment means an evaluation value indicating how much the recognition result [T] output by the DNN can be trusted. For example, the range of the reliability is in the range of 0.0 to 1.0, and the closer the numerical value is to 1.0, the less there are other competing candidates having a score similar to the recognition result [T]. On the other hand, the closer it is to 0, the more other competing candidates having a score similar to the recognition result [T] appear.

[0074] The CNN 52' updates the internal state 55 based on this recognition result 53a. Next, for the pixel information 54b of the second line, recognition processing is performed by the CNN 52' in which the internal state 55 has been updated by the previous recognition result 53a. In FIG. 11, as a result, a recognition result 53b indicating that the number to be recognized is either "8" or "9" is obtained. Further, based on this recognition result 53b, the internal information of the CNN 52' is updated 55. Next, for the pixel information 54c of the third line, recognition processing is performed by the CNN 52' in which the internal state 55 has been updated by the previous recognition result 53b. In FIG. 11, as a result, the number to be recognized is narrowed down to "8" among "8" or "9".

[0075] Here, the recognition process shown in this Figure 11 updates the internal state of the CNN using the result of the previous recognition process, and the CNN with the updated internal state performs the recognition process using the pixel information of the lines adjacent to the line where the previous recognition process was performed. That is, the recognition process shown in this Figure 11 is executed while sequentially updating the internal state of the CNN based on the previous recognition result for each line of the image. Therefore, the recognition process shown in Figure 11 is a process that is recursively executed line by line, and it can be considered to have a structure corresponding to an RNN.

[0076] (2-3-2. Outline of RNN) Next, the RNN will be briefly described. Figures 12A and 12B are diagrams schematically showing an example of discrimination processing (recognition processing) by a DNN when time-series information is not used. In this case, as shown in Figure 12A, one image is input to the DNN. In the DNN, discrimination processing is performed on the input image, and a discrimination result is output.

[0077] Figure 12B is a diagram for explaining the processing of Figure 12A in more detail. As shown in Figure 12B, the DNN executes feature extraction processing and discrimination processing. In the DNN, feature amounts are extracted from the input image by feature extraction processing. Also, in the DNN, discrimination processing is executed on the extracted feature amounts to obtain a discrimination result.

[0078] Figures 13A and 13B are diagrams schematically showing a first example of discrimination processing by a DNN when time-series information is used. In the examples of Figures 13A and 13B, discrimination processing by the DNN is performed using a fixed number of past information in the time series. In the example of Figure 13A, the image [T] at time T, the image [T-1] at time T-1 before time T, and the image [T-2] at time T-2 before time T-1 are input to the DNN. In the DNN, discrimination processing is executed on each of the input images [T], [T-1], and [T-2], and a discrimination result [T] at time T is obtained. A confidence level is assigned to the discrimination result [T].

[0079] FIG. 13B is a diagram for explaining the process of FIG. 13A in more detail. As shown in FIG. 13B, in the DNN, for each of the input images [T], [T-1], and [T-2], the feature extraction process described with reference to FIG. 12B above is executed one-to-one to extract feature amounts corresponding to the images [T], [T-1], and [T-2], respectively. In the DNN, each feature amount obtained based on these images [T], [T-1], and [T-2] is integrated, and an identification process is executed on the integrated feature amount to obtain the identification result [T] at time T. A confidence level is assigned to the identification result [T].

[0080] In the methods of FIGS. 13A and 13B, a plurality of configurations for performing feature amount extraction are required, and depending on the number of past images that can be used, a configuration for performing feature amount extraction is required, and there is a risk that the configuration of the DNN will become large-scale.

[0081] FIGS. 14A and 14B are diagrams schematically showing a second example of the identification process by the DNN when time-series information is used. In the example of FIG. 14A, an image [T] at time T is input to the DNN whose internal state has been updated to the state at time T-1, and the identification result [T] at time T is obtained. A confidence level is assigned to the identification result [T].

[0082] FIG. 14B is a diagram for explaining the process of FIG. 14A in more detail. As shown in FIG. 14B, in the DNN, the feature extraction process described with reference to FIG. 12B above is executed on the input image [T] at time T to extract the feature amount corresponding to the image [T]. In the DNN, the internal state is updated by an image before time T, and the feature amount related to the updated internal state is stored. The feature amount related to this stored internal information and the feature amount in the image [T] are integrated, and an identification process is executed on the integrated feature amount.

[0083] The identification process shown in FIGS. 14A and 14B is executed using a DNN whose internal state is updated using, for example, the immediately previous identification result, and is a recursive process. In this way, a DNN that performs a recursive process is called an RNN (Recurrent Neural Network). The identification process by an RNN is generally used for moving image recognition, etc., and for example, by sequentially updating the internal state of the DNN with frame images updated in time series, it is possible to improve the identification accuracy.

[0084] In the present disclosure, an RNN is applied to the structure of a rolling shutter system. That is, in the rolling shutter system, pixel signals are read out sequentially by line. Therefore, the pixel signals read out sequentially by line are applied to the RNN as information in time series. As a result, it becomes possible to execute an identification process based on a plurality of lines with a smaller configuration compared to the case of using a CNN (see FIG. 13B). Not limited to this, an RNN can also be applied to the structure of a global shutter system. In this case, for example, it is conceivable to regard adjacent lines as information in time series.

[0085] (2-4. Regarding driving speed) Next, the relationship between the driving speed of the frame and the readout amount of the pixel signal will be described with reference to FIGS. 15A and 15B. FIG. 15A is a diagram showing an example of reading all the lines in the image. Here, it is assumed that the resolution of the image to be the recognition process is horizontal 640 pixels × vertical 480 pixels (480 lines). In this case, by driving at a driving speed of 14400 [lines / second], an output at 30 [fps (frame per second)] becomes possible.

[0086] Next, consider thinning out lines and performing imaging. For example, as shown in FIG. 15B, assume that imaging is performed by skipping every other line, i.e., performing 1 / 2 thinning readout. As a first example of 1 / 2 thinning, when driving at a driving speed of 14,400 [lines / second] as described above, since the number of lines read from the image is halved, the resolution decreases, but output at 60 [fps], which is twice the speed when no thinning is performed, becomes possible, and the frame rate can be improved. As a second example of 1 / 2 thinning, when driving at a driving speed of 7,200 [fps], which is half of that in the first example, the frame rate becomes 30 [fps] as in the case of no thinning, but power consumption can be reduced.

[0087] When reading out the lines of the image, whether to perform no thinning, increase the driving speed by performing thinning, or perform thinning and adjust the driving speed can be selected according to, for example, the purpose of the recognition process based on the read pixel signals.

[0088] FIG. 16 is a schematic diagram for schematically explaining the recognition process according to the present embodiment of the present disclosure. In FIG. 16, in step S1, the information processing system 1 (see FIG. 1) according to the present embodiment starts imaging a target image to be recognized.

[0089] Note that the target image is, for example, an image in which the number '8' is handwritten. Also, in the memory 13, a learning model that has been learned to be able to identify numbers by predetermined teacher data is pre-stored as a program, and the recognition processing unit 12 can identify the numbers included in the image by reading out and executing this program from the memory 13. Further, assume that the information processing system 1 performs imaging by the rolling shutter method. Note that even when the information processing system 1 performs imaging by the global shutter method, the following processing is applicable in the same manner as in the case of the rolling shutter method.

[0090] When the imaging is started, in step S2, the information processing system 1 sequentially reads out the frames line by line from the upper end side to the lower end side of the frame.

[0091] When the line is read up to a certain position, the recognition processing unit 12 identifies the number "8" or "9" from the image formed by the read line (step S3). For example, since the numbers "8" and "9" include a common feature portion in the upper half portion, when the lines are read in order from the top and the feature portion is recognized, it can be identified that the recognized object is either the number "8" or "9".

[0092] Here, as shown in step S4a, by reading up to the line at the lower end of the frame or a line near the lower end, the entire appearance of the recognized object appears, and it is determined that the object identified as either the number "8" or "9" in step S2 is the number "8".

[0093] On the other hand, steps S4b and S4c are processes related to the present disclosure.

[0094] As shown in step S4b, even while advancing the line reading from the line position where the reading was performed in step S3 and reaching the lower end of the number "8", it is possible to identify that the recognized object is the number "8". For example, the lower half of the number "8" and the lower half of the number "9" have different features. By reading the line up to the portion where this difference in features becomes clear, it becomes possible to identify whether the object recognized in step S3 is the number "8" or "9". In the example of FIG. 16, in step S4b, it is determined that the object is the number "8".

[0095] Also, as shown in step S4c, it is also conceivable to jump from the line position in step S3 to a line position where it seems possible to distinguish whether the object identified in step S3 is the number "8" or "9" by further reading in the state of step S3. By reading the line at this jump destination, it is possible to determine whether the object identified in step S3 is the number "8" or "9". Note that the line position of the jump destination can be determined based on a learning model pre-learned based on predetermined teacher data.

[0096] Here, when the object is determined in step S4b or step S4c described above, the information processing system 1 can end the recognition process. Thereby, it becomes possible to shorten the recognition process time and save power in the information processing system 1.

[0097] Note that the teacher data is data that holds a plurality of combinations of input signals and output signals for each reading unit. As an example, in the task of identifying the above-described numbers, data for each reading unit (line data, subsampled data, etc.) can be applied as the input signal, and data indicating the "correct number" can be applied as the output signal. As another example, for example, in the task of detecting an object, data for each reading unit (line data, subsampled data, etc.) can be applied as the input signal, and an object class (human body / vehicle / non-object), object coordinates (x, y, h, w), etc. can be applied as the output signal. Also, the output signal may be generated from only the input signal using self-supervised learning.

[0098] (First Embodiment) FIG. 17 is a diagram showing problems in recognition processing when color filters are arranged in a Bayer array in the pixels of the sensor unit 10. Here, the case of reading line data will be described. When reading the line data L170, the image data mainly consists of R pixels (pixels having a red filter) and G pixels (pixels having a green filter). Also, when reading the line data L172, the image data mainly consists of G pixels and B pixels (pixels having a blue filter). Thus, the color information that can be obtained varies depending on the line. For this reason, in the case of skip reading or random reading, there is a possibility that color information cannot be stably obtained, and the accuracy may decrease in recognition tasks that require color.

[0099] FIG. 18 is a diagram showing problems in recognition processing by subsampling when color filters are arranged in a Bayer array. (a) to (d) respectively show the subsampling positions R230, G230, R232, G232, G234, B230 of the image data corresponding to times t1 to t6. As shown in these figures, when sampling and reading on a pixel-by-pixel basis, the color information that can be obtained by sampling varies. For this reason, in the case of skip reading or random reading, there is a possibility that color information cannot be stably obtained. For this reason, the accuracy may decrease in recognition tasks that require color. Note that the higher the sampling period can be adjusted, the more possible it is to drive the sensor unit 10 at high speed and with low power consumption. On the other hand, if the cycle pattern of sampling is made complex, the circuit scale of the sensor unit 10 will increase or the design difficulty will increase.

[0100] FIG. 19 is a diagram schematically showing the case of sampling and reading in block units. As shown in FIG. 19, if sampling and reading are performed in units of block B190, it is possible to obtain all types of color information. On the other hand, depending on how the cycle pattern can be supported, the circuit scale of the sensor unit 10 will increase or the design difficulty will increase.

[0101] FIG. 20 is a functional block diagram of an example for explaining the functions of the sensor control unit 11 and the recognition processing unit 12 according to the present embodiment. In FIG. 20, the sensor control unit 11 includes a reading unit 110. FIG. 20 further shows an accumulation unit 123b that stores the pixel array information of the pixel array unit 101. The pixel array information includes information on the filters arranged in the pixels constituting the sensor unit 10. For example, the pixel array information includes information on the types of filters such as R (red filter), G (green filter), B (blue filter), W (white filter), spectral filters, and infrared filters. The pixel array information also includes information on so-called special pixels such as the polarization direction in the case of having a polarization filter.

[0102] The recognition processing unit 12 includes a feature amount calculation unit 120, a feature amount accumulation control unit 121, a reading area determination unit 123, and a recognition processing execution unit 124. The reading area determination unit 123 further includes a read pixel type determination unit 123a.

[0103] In the sensor control unit 11, the reading unit 110 reads image data from the pixel array unit 101 based on the reading area information. This reading area information is supplied from the recognition processing unit 12. The reading area information is information indicating the reading area for reading from the sensor unit 10. That is, the reading area information is, for example, the line number of one or more lines. Not limited to this, the reading area information may be information indicating the pixel positions within one line. Also, as the reading area information, by combining one or more line numbers and information indicating the pixel positions of one or more pixels within the line, it is possible to specify reading areas of various patterns. Note that the reading area is equivalent to the reading unit. Not limited to this, the reading area and the reading unit may be different.

[0104] In addition, the reading unit 110 can receive information indicating exposure or analog gain from the recognition processing unit 12 or the visual recognition processing unit 14 (see FIG. 1). This reading unit 110 reads out pixel data from the sensor unit 10 according to the read area information input from the read area determination unit 123. For example, the reading unit 110 obtains a line number indicating the line to be read and pixel position information indicating the position of the pixel to be read on the line based on the read area information, and outputs the obtained line number and pixel position information to the sensor unit 10.

[0105] Also, the reading unit 110 sets the exposure or analog gain (AG) for the sensor unit 10 according to the supplied information indicating the exposure or analog gain. Further, the reading unit 110 can generate a vertical synchronization signal and a horizontal synchronization signal and supply them to the sensor unit 10.

[0106] In the recognition processing unit 12, the read pixel type determination unit 123a receives read information indicating the read area to be read next from the feature amount accumulation control unit 121. The pixel type determination unit 123a generates read area information based on the received read information based on the information of the pixel array and outputs it to the reading unit 110. The pixel type determination unit 123a changes the type of pixel according to vehicle information, map information, and information of external sensors. For example, since G pixels with a G filter arranged have high sensitivity and less noise, G pixels are preferentially selected in normal recognition processing. On the other hand, when the color information increases in the scenery, R, G, and B pixels are selected. In the present embodiment, a pixel with an R (red filter) arranged is called an R pixel, a pixel with a G (green filter) arranged is called a G pixel, a pixel with a B (blue filter) arranged is called a B pixel, a pixel with a W (white filter) arranged is called a W pixel, a polarized pixel with a polarization filter arranged, a pixel with a spectroscopic filter arranged is called a spectroscopic pixel, and a pixel with an infrared filter arranged is called an infrared pixel.

[0107] Here, as the read area indicated in the read area information, for example, information with read position information for reading pixel data of the read unit added thereto for each predetermined read unit can be used. The read unit is a set of one or more pixels and serves as the unit of processing by the recognition processing unit 12 and the visual recognition processing unit 14. As an example, if the read unit is a line, a line number [L#x] indicating the position of the line is added as the read position information. Also, if the read unit is a rectangular area including a plurality of pixels, information indicating the position in the pixel array unit 101 of the rectangular area, for example, information indicating the position of the upper left corner pixel, is added as the read position information. The pixel type determination unit 123a has a read unit applied thereto specified in advance. Also, in the global shutter method, when the read pixel type determination unit 123a reads sub-pixels, it is possible to include the position information of the sub-pixels in the read area. Not limited to this, the pixel type determination unit 123a can also determine the read unit, for example, in response to an instruction from outside the read area determination unit 123. Therefore, the read area determination unit 123 functions as a read unit control unit that controls the read unit.

[0108] Note that the read area determination unit 123 can also determine the read area for the next read based on the recognition information supplied from the recognition processing execution unit 124 described later, and generate read area information indicating the determined read area.

[0109] In the recognition processing unit 12, the feature amount calculation unit 120 calculates the feature amount in the area indicated in the read area information based on the pixel data supplied from the read unit 110 and the read area information. The feature amount calculation unit 120 outputs the calculated feature amount to the feature amount accumulation control unit 121.

[0110] The feature amount calculation unit 120 may calculate the feature amount based on the pixel data supplied from the read unit 110 and the past feature amount supplied from the feature amount accumulation control unit 121. Not limited to this, the feature amount calculation unit 120 may, for example, acquire information for setting exposure and analog gain from the read unit 110, and further calculate the feature amount using the acquired information.

[0111] In the recognition processing unit 12, the feature amount accumulation control unit 121 accumulates the feature amounts supplied from the feature amount calculation unit 120 in the feature amount accumulation unit 122. Further, when a feature amount is supplied from the feature amount calculation unit 120, the feature amount accumulation control unit 121 generates readout information indicating a readout area for the next readout, and outputs the readout information to the readout area determination unit 123.

[0112] Here, the feature amount accumulation control unit 121 can integrate and accumulate the already accumulated feature amounts and the newly supplied feature amounts. Further, the feature amount accumulation control unit 121 can delete the unnecessary feature amounts among the feature amounts accumulated in the feature amount accumulation unit 122. Examples of the unnecessary feature amounts include the feature amounts related to the previous frame, and the feature amounts that have been calculated and already accumulated based on the frame images of a scene different from the frame image of the frame in which the new feature amounts are calculated. Further, the feature amount accumulation control unit 121 can also delete and initialize all the feature amounts accumulated in the feature amount accumulation unit 122 as necessary.

[0113] Further, the feature amount accumulation control unit 121 generates the feature amounts for the recognition processing execution unit 124 to use in the recognition processing based on the feature amounts supplied from the feature amount calculation unit 120 and the feature amounts accumulated in the feature amount accumulation unit 122. The feature amount accumulation control unit 121 outputs the generated feature amounts to the recognition processing execution unit 124.

[0114] The recognition processing execution unit 124 executes the recognition processing based on the feature amounts supplied from the feature amount accumulation control unit 121. The recognition processing execution unit 124 performs object detection, face detection, etc. by the recognition processing. The recognition processing execution unit 124 outputs the recognition results obtained by the recognition processing to the output control unit 15 and the confidence calculation unit 125. The recognition results include the information of the detection scores.

[0115] The recognition processing execution unit 124 can also output recognition information including the recognition result generated by the recognition processing to the read area determination unit 123. Note that the recognition processing execution unit 124 can receive feature amounts from the feature amount accumulation control unit 121 and execute recognition processing based on a trigger generated by, for example, a trigger generation unit (not shown).

[0116] FIG. 21 is a diagram showing an example of a pixel array unit 101 in which R, G, and B pixels are arranged in a row in a direction orthogonal to the read direction of line data. As shown in FIG. 21, it is possible to acquire all types of color information regardless of which line the data is read from, such as line data L190, L192. Therefore, in the recognition processing using line data, color information can be stably acquired, so that a decrease in the accuracy of a recognition task that requires color can be suppressed.

[0117] FIG. 22 is a diagram showing an example in which G pixels are arranged at intervals in a row in a direction orthogonal to the read direction of line data, and columns in which R and B pixels are alternately arranged in a row are arranged therebetween. As shown in FIG. 22, in the columns in which R and B pixels are alternately arranged in a row, the arrangement order of the R and B pixels is different for each column. It is possible to acquire all types of color information regardless of which line the data is read from, such as line data L200, L202. Therefore, in the recognition processing using line data, color information can be stably acquired, so that a decrease in the accuracy of a recognition task that requires color can be suppressed. Further, by arranging in such a manner, it is also possible to improve the spatial frequency of the R and B pixels in the column direction as compared with the example of FIG. 21.

[0118] FIG. 23 is a diagram showing an example in which R, B, and G pixels are alternately arranged in a single row in a direction orthogonal to the readout direction of line data, and the arrangement order of the R, B, and G pixels is different for each column. No matter which line the data is read from, such as line data L230, L232, etc., it is possible to acquire all types of color information. Therefore, in the recognition process using line data, color information can be stably acquired, so that a decrease in the accuracy of recognition tasks that require color can be suppressed. Also, by arranging the pixels in this way, it is possible to increase the spatial frequency of the G pixels in the column direction and decrease the spatial frequency of the G pixels in the row direction compared to the example of FIG. 21. It is also possible to increase the spatial frequency of the R and B pixels in the column direction and decrease the spatial frequency of the R and B pixels in the row direction.

[0119] FIG. 24 is a diagram showing an example in which R, B, and G pixels are alternately arranged in a single row in the readout direction of line data. No matter which line the data is read from, such as line data L240, L242, L244, etc., it is possible to acquire a single color information.

[0120] FIG. 25 is a diagram showing another readout method in FIG. 24. Diagram (a) shows the readout range L250 at time t1, and diagram (b) shows the readout range L252 at time t2. In this way, for example, by such an arrangement, it is possible to preferentially read G pixels. The pixel type determination unit 123a can apply a readout method such as continuously reading G pixels until color-based recognition is required. Since G pixels have high sensitivity and low noise, the recognition rate is improved.

[0121] FIG. 26 is a diagram showing an example when a plurality of rows are read simultaneously with respect to FIG. 24. Diagram (a) shows the readout range L260 at time t1, and diagram (b) shows the readout range L262 at time t2. In this way, when the pixel type determination unit 123a wants to perform stable recognition, it can set a readout area that reads out several lines at a time instead of in line units.

[0122] FIG. 27 is a diagram showing an example in which R, B, G, and W pixels are alternately arranged in a single row in the readout direction of line data. The (a) figure shows the readout range L270 at time t1, and the (b) figure shows the readout range L272 at time t2. In this way, the pixel type determination unit 123a can also set readout that prioritizes W pixels. Since W pixels have high sensitivity and little noise, it is possible to improve the recognition rate by a readout method such as continuously reading W pixels until color-based recognition is required.

[0123] FIG. 28 is a diagram showing an example in which R, B, and G pixels are alternately arranged in two columns each in a direction orthogonal to the readout direction of line data. The (a) figure shows the subsampling position S240 at time t1, and the (b) figure shows the subsampling position S242 at time t2. In this way, all types of color information can be acquired for data at any periodic timing. Therefore, since color information can be stably acquired, it is possible to suppress a decrease in the accuracy of recognition tasks that require color.

[0124] FIG. 29 is a diagram showing an example in which R, B, G, and spectroscopic pixels are alternately arranged in a single row in the readout direction of line data. Line S290 indicates the row in which the spectroscopic pixels are arranged. The numerical values within line S290 are numbers corresponding to spectroscopic characteristics. For example, an example is shown in which, as the peak of the transmission wavelength in the range from 400 to 720 nanometers increases from 1 to 6, it shifts toward the longer wavelength side. That is, the peak of the transmission wavelength of the spectroscopic filter corresponding to the spectroscopic pixel indicated by 1 is the shortest wavelength and is on the 400-nanometer side. On the other hand, the peak of the transmission wavelength of the spectroscopic filter corresponding to the spectroscopic pixel indicated by 6 is the longest wavelength and is on the 720-nanometer wavelength side.

[0125] The pixel type determination unit 123a normally reads visible light pixels R, B, and G, but can change the priority so as to read line L290 of the spectroscopic image (line S290) when it is desired to identify the material of an object. For example, in an emergency stop / avoidance situation, when it is impossible to distinguish between a person and a poster, the recognition rate can be improved by using spectroscopic information for identification.

[0126] FIG. 30 is a diagram showing an example in which a polarizing filter is further stacked on the arrangement example of FIG. 21. The right diagram shows an arrangement example of the polarizing pixels in area A300. In this way, for example, the polarization filters may be arranged in 16 pixels out of 64 pixels. This is just an example, and the arrangement example of the polarization filters is not limited to this. For example, each polarization filter may be arranged dispersedly. For example, the number of pixels in the pixel array unit 101 is, for example, 1260×1260, etc., and 64 pixels are arranged periodically in the 1260×1260 pixel arrangement.

[0127] The pixel type determination unit 123a normally reads the visible light pixels R, B, and G, but when it is desired to remove reflection and perform recognition, the priority is changed so as to read the polarization image (area A300). For example, when recognizing the direction of the face of the driver of an oncoming vehicle that cannot be observed due to reflection on the front glass, etc., the priority is changed so as to read the polarization image (area A300). Thereby, the reflection component can be suppressed and the decrease in the recognition rate can be suppressed.

[0128] FIG. 31 is a diagram showing an example in which a polarizing filter equivalent to area A300 is stacked on each of the arrangement examples of FIGS. 22, 23, 24, and 27. In this way, the arrangement of the polarizing filter and each color filter may be changed.

[0129] FIG. 32 is a diagram showing an example in which a polarizing filter is further arranged in the arrangement example of FIG. 28. It is an example in which each polarizing filter is stacked for each polarization area A320 every 2×2 pixels. Thereby, it is possible to acquire all types of color information for data at any periodic timing, and when it is desired to remove reflection and perform recognition, it is possible to change the priority so as to read the polarization image (area A320).

[0130] FIG. 33 is a diagram showing an example in which a polarizing filter is further arranged in the arrangement example of FIG. 29. It is an example in which each polarizing filter is stacked for each polarization area A300 every 4×4 pixels. Accordingly, the pixel type determination unit 123a reads the visible light pixels R, B, and G during normal times, but can change the priority to read the spectroscopic image (line S290) when it is desired to identify the material of an object. Also, when it is desired to perform recognition after removing reflection, the priority can be changed to read the polarization image (area A300).

[0131] FIG. 34 is a diagram showing an example in which far-infrared pixels H having an infrared filter are arranged in a part of the arrangement example of FIG. 27. In FIG. 34, the far-infrared pixels H are arranged alternately with the R pixels in the column L340 of the R pixels. Accordingly, the pixel type determination unit 123a reads the visible light pixels during normal times, but changes the priority to read the far-infrared pixels H when it is desired to perform recognition in a dark place. For example, it is effective when recognizing whether there are people or living things in a dark area where there are no streetlights and no front light is shining.

[0132] FIG. 35 is a diagram showing an example in which a polarization filter is further arranged in a part of the arrangement example of FIG. 24. In FIG. 35, a polarization filter is further arranged in the area A300. Accordingly, the pixel type determination unit 123a reads the visible light pixels during normal times, but changes the priority to read the far-infrared pixels H when it is desired to perform recognition in a dark place. For example, it is effective when recognizing whether there are people or living things in a dark area where there are no streetlights and no front light is shining. Also, the pixel type determination unit 123a reads the visible light pixels R, B, and G during normal times, but changes the priority to read the polarization image (area A300) when it is desired to perform recognition after removing reflection. For example, the priority is changed to read the polarization image (area A300) when recognizing the direction of the face of the driver of an oncoming vehicle that cannot be observed due to reflection on the front glass. Thereby, the reflection component can be suppressed and a decrease in the recognition rate can be suppressed.

[0133] FIG. 36 is a diagram showing an example in which a part of the W pixel column in FIG. 35 is replaced with spectroscopic pixels. In FIG. 36, a polarizing filter is further arranged in area A300. Thereby, the pixel type determination unit 123a normally reads visible light pixels, but when it is desired to recognize in a dark place, the priority is changed so as to read the far-infrared pixel H. For example, it is effective when recognizing whether there are people or living things in a dark area where there are no street lights and no front light hits. Further, the pixel type determination unit 123a normally reads visible light pixels R, B, and G, but when it is desired to remove reflection and recognize, the priority is changed so as to read the polarized image (area A300). For example, when recognizing the direction of the face of the driver of an oncoming vehicle that cannot be observed due to reflection on the front glass, the priority is changed so as to read the polarized image (area A300). Thereby, the reflection component can be suppressed and the decrease in the recognition rate can be suppressed. Furthermore, the pixel type determination unit 123a normally reads visible light pixels R, B, and G, but when it is desired to identify the material of an object, it is possible to change the priority so as to read the spectroscopic image (line S290). For example, in an emergency stop / avoidance situation, when it is impossible to distinguish between a person and a poster, the recognition rate can be improved by using spectroscopic information for identification.

[0134] FIG. 37 is a flowchart showing the processing flow of the pixel type determination unit 123a. First, the pixel type determination unit 123a sets a priority A for each type of pixel indicating which type of pixel to prioritize based on the recognition result of the recognition processing unit 12 using the pixel array information (step S100). For example, normally, the priority of the G pixel is set highest. On the other hand, when the recognition rate of the G pixel is decreasing, the priorities of the R pixel and the B pixel are also set to be increased.

[0135] Next, the pixel type determination unit 123a sets a priority B for each type of pixel based on the map information using the pixel array information (step S102). For example, if the position of the vehicle in the map information is a tunnel, the priorities of the infrared pixel H and the W pixel are set high. On the other hand, if it is flat ground without obstacles, the priority of the G pixel is set highest. Also, if the position of the vehicle is in a busy street, the priority of the spectroscopic pixel is increased so that the difference between a person and a fixed object becomes clearer.

[0136] Next, the pixel type determination unit 123a sets a priority C for each type of pixel based on the vehicle information using the pixel array information (step S104). For example, the vehicle information includes information such as speed and traveling direction. When traveling at high speed, the pixel type determination unit 123a determines that, for example, it is traveling in a place where there are no people, and sets the priority of, for example, G pixels to be the highest. Conversely, when traveling at low speed, the priority of spectral pixels is increased so that the difference between a person and a stationary object becomes clearer. Also, when the direction of the vehicle is the direction in which the sun shines into the windshield, the priority of polarization pixels is increased.

[0137] Next, the pixel type determination unit 123a sets a priority D for each type of pixel based on the information of the external sensor using the pixel array information (step S106). For example, the information of the external sensor includes information such as illuminance, temperature, and humidity. When the illuminance is high, the pixel type determination unit 123a changes the priority of polarization pixels. On the other hand, when the illuminance is low, the priorities of infrared pixels H and W pixels are increased.

[0138] Next, the pixel type determination unit 123a integrates the priorities A, B, C, and D (step S108), determines the next readout area according to the readout pattern from the pixel type with the highest priority, and ends the process.

[0139] As described above, according to the present embodiment, the readout unit 110 sets readout pixels as a part of the pixel region of the pixel array unit 101 in which a plurality of pixels are arranged in a two-dimensional array, and controls the readout of pixel signals from the pixels included in the pixel region. At this time, the pixel type determination unit 123a sets the readout area, that is, the readout pixels, based on the color filter array information of the pixel region included in the pixel array information. Thereby, pixels with color filters arranged according to the situation can be selected. Thereby, a decrease in the recognition rate of the recognition processing execution unit 124 can be suppressed.

[0140] (Second Embodiment)

[0141] (2-1. Application Examples of the Technology of the Present Disclosure) Next, as a second type of embodiment, application examples of the information processing apparatus 1a according to the first or second embodiment according to the present disclosure will be described. FIG. 38 is a diagram showing a usage example of using the information processing apparatus 1a according to the first embodiment. In the following, when there is no particular need for distinction, the description will be made by representing it with the information processing apparatus 1a.

[0142] The above-described information processing apparatus 1a can be used, for example, in various cases where it senses light such as visible light, infrared light, ultraviolet light, X-rays, etc. and performs recognition processing based on the sensing results as follows.

[0143] · Devices for taking pictures of images for viewing, such as digital cameras and mobile devices with camera functions. · In-vehicle sensors for taking pictures of the front, rear, surroundings, inside the vehicle, etc. of an automobile for safe driving such as automatic stop and recognition of the driver's state, surveillance cameras for monitoring moving vehicles and roads, ranging sensors for performing ranging between vehicles, etc., devices for use in traffic. · Devices for use in home appliances such as TVs, refrigerators, air conditioners, etc. that take pictures of the user's gestures and perform device operations according to the gestures. · Devices for use in medical and healthcare, such as endoscopes and devices for performing blood vessel imaging by receiving infrared light. · Devices for use in security, such as surveillance cameras for crime prevention purposes and cameras for person authentication purposes. · Devices for use in beauty, such as skin measuring devices for taking pictures of the skin and microscopes for taking pictures of the scalp. · Devices for use in sports, such as action cameras and wearable cameras for sports applications. · Devices for use in agriculture, such as cameras for monitoring the state of fields and crops.

[0144] (2-2. Application Examples to Mobile Bodies) The technology according to the present disclosure (this technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.

[0145] FIG. 39 is a block diagram showing a schematic configuration example of a vehicle control system which is an example of a movement control system to which the technology according to the present disclosure can be applied.

[0146] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in FIG. 41, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside vehicle information detection unit 12030, an inside vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

[0147] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 functions as a control device such as a driving force generation device for generating a driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0148] The body system control unit 12020 controls the operations of various devices equipped on the vehicle according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as a headlamp, a backlamp, a brake lamp, a turn signal, or a fog lamp. In this case, radio waves transmitted from a portable device that substitutes for a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these inputs of radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0149] The vehicle exterior information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing, such as for a person, a vehicle, an obstacle, a sign, or characters on the road surface, based on the received image.

[0150] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of received light. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. Also, the light received by the imaging unit 12031 may be visible light or non-visible light such as infrared light.

[0151] The vehicle interior information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the vehicle interior information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the vehicle interior information detection unit 12040 may calculate the degree of driver fatigue or concentration based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0152] Based on the information inside and outside the vehicle acquired by the out-vehicle information detection unit 12030 or the in-vehicle information detection unit 12040, the microcomputer 12051 can calculate the control target values of the driving force generating device, the steering mechanism, or the braking device, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing functions of an ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation of the vehicle, following driving based on the inter-vehicle distance, constant vehicle speed driving, collision warning of the vehicle, or lane departure warning of the vehicle.

[0153] In addition, based on the information around the vehicle acquired by the out-vehicle information detection unit 12030 or the in-vehicle information detection unit 12040, the microcomputer 12051 can perform cooperative control aimed at autonomous driving, etc., which runs autonomously regardless of the driver's operation, by controlling the driving force generating device, the steering mechanism, or the braking device, etc.

[0154] Also, based on the out-vehicle information acquired by the out-vehicle information detection unit 12030, the microcomputer 12051 can output a control command to the body system control unit 12020. For example, the microcomputer 12051 can perform cooperative control aimed at anti-glare, such as controlling the headlamp according to the position of the preceding vehicle or oncoming vehicle detected by the out-vehicle information detection unit 12030 and switching the high beam to the low beam.

[0155] The audio-visual output unit 12052 transmits at least one output signal of audio and image to an output device capable of notifying information visually or aurally to the vehicle occupants or outside the vehicle. In the example of FIG. 36, as the output device, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are illustrated. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0156] FIG. 40 is a diagram showing an example of the installation position of the imaging unit 12031.

[0157] In FIG. 40, the vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.

[0158] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the front glass in the vehicle interior of the vehicle 12100, for example. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the front glass in the vehicle interior mainly acquire images in front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images on the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires images behind the vehicle 12100. The front images acquired by the imaging units 12101 and 12105 are mainly used for detecting a preceding vehicle or detecting pedestrians, obstacles, traffic lights, traffic signs, or lanes.

[0159] Note that FIG. 40 shows an example of the imaging ranges of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided at the front nose, and the imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively. The imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or the back door. For example, by overlapping the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 seen from above can be obtained.

[0160] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0161] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 determines the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the temporal change of this distance (relative speed with respect to the vehicle 12100). Thus, it can extract, as the preceding vehicle, the closest three-dimensional object on the traveling path of the vehicle 12100 that is traveling in substantially the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or more). Further, the microcomputer 12051 can set the inter-vehicle distance to be secured in advance in front of the preceding vehicle and perform automatic braking control (including follow-up stop control) and automatic acceleration control (including follow-up start control), etc. In this way, cooperative control for the purpose of automatic driving, etc., which autonomously travels without relying on the driver's operation, can be performed.

[0162] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, and can use it for automatic avoidance of obstacles. For example, the microcomputer 12051 discriminates obstacles around the vehicle 12100 into obstacles visible to the driver of the vehicle 12100 and obstacles difficult to visually recognize. Then, the microcomputer 12051 determines a collision risk indicating the degree of risk of collision with each obstacle, and when the collision risk is equal to or higher than a set value and there is a possibility of collision, it can output an alarm to the driver via the audio speaker 12061 or the display unit 12062, or perform forced deceleration or avoidance steering via the drive system control unit 12010 to provide driving support for collision avoidance.

[0163] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian exists in the captured images of the imaging units 12101 to 12104. Such recognition of a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras and a procedure of performing pattern matching processing on a series of feature points indicating the outline of an object to determine whether it is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio-visual output unit 12052 controls the display unit 12062 to superimpose and display a rectangular outline for emphasizing the recognized pedestrian. Further, the audio-visual output unit 12052 may control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0164] As described above, an example of a vehicle control system to which the technology according to the present disclosure can be applied has been described. The technology according to the present disclosure can be applied to the imaging unit 12031 and the vehicle exterior information detection unit 12030 among the configurations described above. Specifically, for example, the sensor unit 10 of the information processing system 1 is applied to the imaging unit 12031, and the recognition processing unit 12 is applied to the vehicle exterior information detection unit 12030. The recognition result output from the recognition processing unit 12 is passed to the integrated control unit 12050 via, for example, the communication network 12001.

[0165] In this way, by applying the technology according to the present disclosure to the imaging unit 12031 and the vehicle exterior information detection unit 12030, it is possible to perform recognition of a short-distance object and recognition of a long-distance object respectively, and it is possible to perform recognition of a short-distance object with high simultaneity, so that more reliable driving support becomes possible.

[0166] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0167] Note that the present technology can be configured as follows.

[0168] (1) A readout pixel is set as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and a readout unit that controls the readout of pixel signals from the pixels included in the pixel region. A setting unit that sets the readout pixel based on the color filter array information of the pixel region. An information processing apparatus comprising the same.

[0169] (2) The information processing apparatus according to (1), wherein the setting unit sets a readout pixel based on external information.

[0170] (3) The information processing apparatus according to (1), wherein the external information is at least any one of a recognition result, map information, vehicle information, and external sensor information.

[0171] (4) The information processing apparatus according to (1), wherein the setting unit sets priorities for at least two pieces of information among a recognition result, map information, vehicle information, and external sensor information, and sets the readout pixel based on the set plurality of priorities.

[0172] (5) At least some of the plurality of pixels are provided with a polarizing filter. The information processing apparatus according to (4), wherein the setting unit sets a priority for a pixel provided with a polarizing filter based on at least any one of a recognition result, map information, vehicle information, and external sensor information.

[0173] (6) At least some of the plurality of pixels are provided with a spectroscopic filter. The information processing apparatus according to (5), wherein the setting unit sets a priority for a pixel provided with the spectroscopic filter based on at least any one of a recognition result, map information, vehicle information, and external sensor information.

[0174] (7) At least some of the plurality of pixels are provided with an infrared filter. The setting unit is the information processing apparatus according to (5), which sets the priority of the pixels where the infrared filter is arranged based on at least any one of the recognition result, map information, vehicle information, and external sensor information.

[0175] (8) An information processing system including a sensor unit in which a plurality of pixels are arranged in a two-dimensional array, a sensor control unit that controls the sensor unit, and a recognition processing unit, wherein the sensor control unit has a reading unit that sets reading pixels as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controls the reading of pixel signals from the pixels included in the pixel region, and the recognition processing unit has a setting unit that sets the reading pixels based on the color filter array information of the pixel region. An information processing system.

[0176] (9) A reading step of setting reading pixels as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controlling the reading of pixel signals from the pixels included in the pixel region, and a setting step of setting the reading pixels based on the color filter array information of the pixel region. An information processing method.

[0177] (10) A reading step of setting reading pixels as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controlling the reading of pixel signals from the pixels included in the pixel region, and a setting step of setting the reading pixels based on the color filter array information of the pixel region, A program for causing a computer to execute.

Description of Reference Numerals

[0178] 1: Information processing system, 2: Information processing apparatus, 10: Sensor unit, 12: Recognition processing unit, 110: Reading unit, 120: Feature amount calculation unit, 123a: Reading pixel type determination unit (setting unit), 124: Recognition processing execution unit.

Claims

1. A readout unit that sets readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array and controls the readout of pixel signals from the pixels included in the pixel region; A setting unit that sets the readout pixels based on the color filter array information of the pixel region; comprising; At least some of the plurality of pixels are provided with polarization filters, The setting unit sets the priority of the pixels provided with the polarization filters based on at least one of a recognition result, map information, vehicle information, and external sensor information. An information processing apparatus.

2. The external sensor information includes at least one of illuminance, temperature, and humidity. The information processing apparatus according to claim 1.

3. When the illuminance is high, the setting unit increases the priority of the pixels provided with the polarization filters. The information processing apparatus according to claim 2.

4. A readout unit that sets readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array and controls the readout of pixel signals from the pixels included in the pixel region; A setting unit that sets the readout pixels based on the color filter array information of the pixel region; comprising; At least some of the plurality of pixels are provided with spectral filters, The setting unit sets the priority of the pixels provided with the spectral filters based on at least one of a recognition result, map information, vehicle information, and external sensor information. An information processing apparatus.

5. having a recognition processing execution unit that performs recognition processing on the image data read from the readout pixels, When the object cannot be identified by the recognition processing, the setting unit increases the priority of the pixels provided with the spectral filters. The information processing apparatus according to claim 4.

6. The vehicle information includes the traveling direction or speed of the vehicle, The setting unit increases the priority of the pixels with the spectroscopic filter disposed thereon when the vehicle is traveling at a low speed. The information processing apparatus according to claim 5.

7. A reading unit that sets read pixels as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array and controls the reading of pixel signals from the pixels included in the pixel region; A setting unit that sets the read pixels based on the color filter array information of the pixel region; comprising At least some of the plurality of pixels have an infrared filter disposed thereon, The setting unit sets the priority of the pixels with the infrared filter disposed thereon based on at least one of a recognition result, map information, vehicle information, and external sensor information. An information processing apparatus.

8. having a recognition processing execution unit that performs recognition processing on the image data read from the read pixels, The setting unit increases the priority of the pixels with the infrared filter disposed thereon when it is desired to perform recognition in a dark place in the recognition processing. The information processing apparatus according to claim 7.

9. The setting unit increases the priority of the pixels with the infrared filter disposed thereon when the position of the vehicle is in a tunnel. The information processing apparatus according to claim 7.

10. The setting unit sets priorities for at least two pieces of information among a recognition result, map information, vehicle information, and external sensor information, and sets the read pixels based on the set multiple priorities. The information processing apparatus according to any one of claims 1 to 9.

11. A sensor unit in which a plurality of pixels are arranged in a two-dimensional array; A sensor control unit that controls the sensor unit; A recognition processing unit, and an information processing system comprising: The sensor control unit A readout pixel is set as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and a readout unit is provided for controlling the readout of pixel signals from the pixels included in the pixel region. The recognition processing unit A setting unit that sets the readout pixel based on the color filter array information of the pixel region is provided, At least some of the plurality of pixels are provided with polarization filters. The setting unit sets the priority of the pixels provided with the polarization filters based on at least one of the recognition result, map information, vehicle information, and external sensor information. An information processing system.

12. A sensor unit in which a plurality of pixels are arranged in a two-dimensional array, A sensor control unit that controls the sensor unit, An information processing system including a recognition processing unit, The sensor control unit A readout pixel is set as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and a readout unit is provided for controlling the readout of pixel signals from the pixels included in the pixel region. The recognition processing unit A setting unit that sets the readout pixel based on the color filter array information of the pixel region is provided, At least some of the plurality of pixels are provided with spectroscopic filters. The setting unit sets the priority of the pixels provided with the spectroscopic filters based on at least one of the recognition result, map information, vehicle information, and external sensor information. An information processing system.

13. A sensor unit in which a plurality of pixels are arranged in a two-dimensional array, A sensor control unit that controls the sensor unit, An information processing system including a recognition processing unit, The sensor control unit A readout unit is provided that sets readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controls the readout of pixel signals from the pixels included in the pixel region. The recognition processing unit has a setting unit that sets the readout pixels based on the color filter array information of the pixel region. and at least some of the plurality of pixels are provided with infrared filters. The setting unit sets the priority of the pixels provided with the infrared filters based on at least one of the recognition result, map information, vehicle information, and external sensor information. An information processing system.

14. The setting unit sets priorities for at least two of the recognition result, map information, vehicle information, and external sensor information, and sets the readout pixels based on the set multiple priorities. The information processing system according to any one of claims 11 to 13.

15. A readout step of setting readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controlling the readout of pixel signals from the pixels included in the pixel region, and a setting step of setting the readout pixels based on the color filter array information of the pixel region, are provided, at least some of the plurality of pixels are provided with polarization filters, and the setting step sets the priority of the pixels provided with the polarization filters based on at least one of the recognition result, map information, vehicle information, and external sensor information. An information processing method.

16. A readout step of setting readout pixels as part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array, and controlling the readout of pixel signals from the pixels included in the pixel region, and a setting step of setting the readout pixels based on the color filter array information of the pixel region, are provided, At least one of the plurality of pixels is provided with a spectroscopic filter. The setting step is an information processing method for setting the priority of the pixel provided with the spectroscopic filter based on at least one of the recognition result, map information, vehicle information, and external sensor information.

17. A reading step of setting read pixels as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array and controlling the reading of pixel signals from the pixels included in the pixel region; A setting step of setting the read pixels based on the color filter array information of the pixel region; Comprising: At least one of the plurality of pixels is provided with an infrared filter. The setting step is an information processing method for setting the priority of the pixel provided with the infrared filter based on at least one of the recognition result, map information, vehicle information, and external sensor information.

18. A reading step of setting read pixels as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array and controlling the reading of pixel signals from the pixels included in the pixel region; A setting step of setting the read pixels based on the color filter array information of the pixel region; Comprising: At least one of the plurality of pixels is provided with a polarizing filter. The setting step is a program for causing a computer to execute an information processing method for setting the priority of the pixel provided with the polarizing filter based on at least one of the recognition result, map information, vehicle information, and external sensor information.

19. A reading step of setting read pixels as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array and controlling the reading of pixel signals from the pixels included in the pixel region; A setting step of setting the read pixels based on the color filter array information of the pixel region; Comprising: At least one of the plurality of pixels is provided with a spectroscopic filter, The setting step is a program that causes a computer to execute an information processing method for setting the priority of the pixel provided with the spectroscopic filter based on at least one of the recognition result, map information, vehicle information, and external sensor information.

20. A reading step of setting a read pixel as a part of a pixel region in which a plurality of pixels are arranged in a two-dimensional array and controlling the reading of pixel signals from the pixels included in the pixel region; A setting step of setting the read pixel based on the color filter array information of the pixel region; comprising At least one of the plurality of pixels is provided with an infrared filter, The setting step is a program that causes a computer to execute an information processing method for setting the priority of the pixel provided with the infrared filter based on at least one of the recognition result, map information, vehicle information, and external sensor information.

Images