Light detection device and information processing system

Abstract

[Problem] The present disclosure provides a light detection device and an information processing system that are capable of changing the resolution of a plurality of regions of a captured image while suppressing an increase in power consumption. [Solution] The light detection device is provided with: a pixel unit that sequentially acquires a captured image at different timings; a region control unit that identifies a region of interest in accordance with the captured image, sets the resolution of a first image corresponding to the region of interest to a first resolution, and sets the resolution of a second image corresponding to regions other than the region of interest to a second resolution that is lower than the first resolution; and an output circuit that outputs the first image and the second image at different frame rates in accordance with the control by the region control unit.

Description

Photodetector and information processing system

[0001] The present disclosure relates to a light detection device and an information processing system.

[0002] A technique is known in which a captured image is processed based on a region of interest and output as a moving image on a display unit. In capturing such a moving image, the image may be divided into multiple regions and captured at different resolutions for each region (see Patent Document 1).

[0003] International Publication No. 2019 / 171522

[0004] However, if an image is divided into a plurality of regions and imaged with different resolutions for each region, power consumption may increase.

[0005] Therefore, the present disclosure provides a light detection device and an information processing system that are capable of changing the resolution of multiple regions of a captured image while suppressing an increase in power consumption.

[0006] In order to solve the above problems, according to the present disclosure, there is provided a photodetection device including: a pixel unit that captures images successively at different timings; an area control unit that identifies an area of ​​interest according to the captured images, and sets the resolution of a first image corresponding to the area of ​​interest to a first resolution and the resolution of a second image corresponding to an area other than the area of ​​interest to a second resolution lower than the first resolution; and an output circuit that outputs the first image and the second image at different frame rates according to the control of the area control unit.

[0007] The region control unit may output the first image at a first frame rate and the second image at a second frame rate, and the first frame rate may be lower than the second frame rate.

[0008] The region control unit may output the first image at a first frame rate and the second image at a second frame rate, and the first frame rate may be higher than the second frame rate.

[0009] The region control unit may output the second image including the first image at the first frame rate, and may output the second image including the first image and the second image not including the first image, in sequence, at the second frame rate.

[0010] The region control unit may sequentially output the second image including the first image and an image of only the first image at the first frame rate, and may output the second image including the first image at the second frame rate.

[0011] The pixel unit is configured with a plurality of pixels arranged in a matrix, each pixel including a photodiode that performs photoelectric conversion, and the region control unit may add the analog signals when reading out the analog signals from the pixels corresponding to the second image.

[0012] The pixel unit may further include a readout circuit that selects each pixel arranged in the row direction of the pixel unit for each row and outputs an analog signal to a signal line for each column connected to each pixel arranged in the column direction of the pixel unit, and an analog-to-digital conversion unit that converts the analog signal of the signal line for each column into a digital signal.

[0013] When the region control unit causes the readout circuit to read out analog signals from pixels corresponding to the second image, the region control unit may simultaneously select multiple rows and cause the readout circuit to output analog signals from pixels arranged in the multiple rows.

[0014] When reading out analog signals from pixels corresponding to the first image, the region control unit may cause the readout circuit to select rows that are less than the plurality of rows, including a single row, and output analog signals from first pixels arranged in the selected rows.

[0015] The region control unit may cause the readout circuit to read out an analog signal simultaneously with the first pixel when a second pixel corresponding to the second image is arranged in the selected row.

[0016] The pixel may further include a signal processing circuit capable of adding the digital signals, and when the second pixel is arranged in the selected row, the signal processing circuit may add the digital signals based on the analog signals read out from the second pixel so as to achieve the second resolution.

[0017] The output circuit may output the first image and the second image in a predetermined image data structure.

[0018] When the first image is included within the area of ​​the second image, the output circuit may output the image in a first image data structure in which second image data corresponding to the second image and first image data corresponding to the first image are included in the same row.

[0019] When the first image is included within the area of ​​the second image, the output circuit may add first image data corresponding to the first image and output the image data converted to the second resolution and the second image data corresponding to the second image in a second image data structure contained within the same row.

[0020] When the first image is included within the region of the second image, the output circuit may output a third image data structure including only the first image, and the second image data structure.

[0021] The output circuit may include information regarding the timing at which reading of each row is started in the first to third image data structures.

[0022] The second image has a first peripheral area image corresponding to a first peripheral area surrounding the attention area and a second peripheral area image corresponding to a second peripheral area surrounding the first peripheral area, and the area control unit may set the resolution of the second peripheral area image to be lower than the resolution of the first peripheral area.

[0023] In order to solve the above problems, according to the present disclosure, there may be provided an information processing system including: a gaze point detection unit that detects a person's gaze point; and a light detection device, wherein the light detection device has: a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of interest based on the gaze point, and sets a resolution of a first image corresponding to the region of interest to a first resolution, and a resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; and an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit.

[0024] The image processing device may further include a display unit that outputs the first image at a first frame rate and the second image at a second frame rate, wherein the display unit sets the first frame rate to be lower than the second frame rate.

[0025] The image processing device may further include a display unit that outputs the first image at a first frame rate and the second image at a second frame rate, wherein the display unit sets the first frame rate higher than the second frame rate.

[0026] The image capturing apparatus may further include a display unit that displays the captured image.

[0027] In order to solve the above problems, according to the present disclosure, there is provided an information processing system having: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of interest based on the gaze point, and sets a resolution of a first image corresponding to the region of interest to a first resolution and a resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates according to control of the region control unit; and a display unit that displays the region of interest based on the first image and displays a region other than the region of interest based on the second image.

[0028] In order to solve the above problems, according to the present disclosure, there is provided an information processing system having: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of attention based on the gaze point, and sets a resolution of a first image corresponding to the region of attention to a first resolution and a resolution of a second image corresponding to a region other than the region of attention to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates in accordance with control of the region control unit; and a signal processing unit that processes the first image at a first frequency and the second image at a second frequency.

[0029] In order to solve the above problems, according to the present disclosure, there is provided an information processing system having: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of attention based on the gaze point, and sets a resolution of a first image corresponding to the region of attention to a first resolution and a resolution of a second image corresponding to a region other than the region of attention to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates in accordance with control by the region control unit; and a display unit that generates and displays an image to be presented to a user based on the image data output at the different frame rates.

[0030] 17A, 17B, and 17C are diagrams illustrating an example of the appearance of an information processing system according to the present disclosure. 17A is a diagram illustrating a modified example of the appearance of an information processing system according to the present disclosure. 17B is a block diagram illustrating a schematic configuration of an information processing system according to the present disclosure. 17B is a diagram illustrating an example of the configuration of a gaze point detection unit. 17C is a diagram illustrating an example of the hardware configuration of a gaze point detection unit and an imaging unit. 17D is a diagram illustrating an example of the configuration of an imaging unit. 17E is a diagram illustrating an example of a gaze region and a peripheral region. 17F is a diagram illustrating an example of different frame rates for the gaze region and the peripheral region. 17G is a diagram illustrating an example of setting the frame rate of the peripheral region to 1 / N times that of the gaze region. 17H is a diagram illustrating a readout sequence for each frame. 17H is a diagram illustrating a sequence corresponding to FIG. 7. 17H is a diagram illustrating a sequence corresponding to FIG. 8. 17H is a diagram illustrating an example of the data structure of image data of each zone transmitted from the photodetector according to the present disclosure. 17H is a diagram illustrating an example of a data structure when image data of each zone is combined according to FIG. 7. 17H is a diagram illustrating an example of a data structure when image data of each zone is combined according to FIG. 8. 17H is a block diagram illustrating a schematic configuration of an information processing system including a photodetector according to the present disclosure. 17A, 17B, and 17C are diagrams illustrating examples of zone constraints that are OK and NG. 21 。 FIG. 22 is a sequence diagram showing the procedure for transmitting and receiving image data between a photodetector and a signal processing device when a zone setting ID is assigned by the photodetector. 23 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device of FIG. 18. 24 is a diagram showing an example of the data structure of image data transmitted by the photodetector to the signal processing device. 25 is a sequence diagram showing the procedure for transmitting and receiving image data between a photodetector and a signal processing device when a zone setting ID is assigned by the signal processing device. 26 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device of FIG. 21. 27 is a diagram showing the data structure of image data of each zone transmitted from the photodetector to the signal processing device. 28 is a diagram showing zone setting information. 29 is a diagram showing the time lag between when the gaze detection device detects the gaze position and when the photodetector reads out pixel signals from the zone corresponding to the gaze position. 29 is a diagram explaining the gaze position and coordinate designation of each zone in two VST sensors for the left and right eyes. 29 is a diagram showing data transmitted and received between two VST sensors constituting the gaze detection device. 29 is a block diagram showing a case where two images generated by the VST sensors are concatenated and transmitted to an AP as a single image. 10 is a diagram showing the data structure of image data transmitted from the CIS1(L) to the AP. FIG. 11 is a diagram showing exposure and readout timings of the photodetector.FIG. 1 is a block diagram showing a schematic configuration of an information processing system in which a plurality of photodetectors are connected to a single lane and communicate with a signal processing device.

[0031] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0032] (First embodiment) Fig. 1A is a diagram illustrating an example of the appearance of an information processing system 1001 of the present disclosure, Fig. 1B is a diagram illustrating a modified example of the appearance of the information processing system 1001 of the present disclosure, and Fig. 2 is a block diagram illustrating a schematic configuration of the information processing system 1001 of the present disclosure. As shown in Fig. 1A, the information processing system 1001 according to this embodiment is configured as a head-mounted display (HMD). An example of the appearance of the head-mounted display (HMD) of this embodiment will be described with reference to Fig. 1A.

[0033] In this example, the HMD 1001 is composed of an output mechanism unit 1011 and a wearing mechanism unit 1012. The wearing mechanism unit 1012 includes a wearing band 1013 that wraps around the head when worn by the user, thereby securing the device in place. Note that the band does not have to wrap around the head as long as it is secured to the head.

[0034] The output mechanism unit 1011 includes a housing 1014 shaped to cover the left and right eyes when the user wears the HMD 1001, and includes a display panel inside that faces the eyes when worn. The housing 1014 may further include a lens that is positioned between the display panel (display unit 2005 ( FIG. 2 )) and the user's eyes when the HMD 1001 is worn, and that expands the user's field of view. The display panel may be divided into left and right halves, and each region may display a stereo image corresponding to the parallax between the eyes, and stereoscopic vision may be realized by such a display.

[0035] The HMD 1001 may further include speakers or earphones at positions corresponding to the user's ears when worn. In this example, the HMD 1001 includes a camera 1015 on the front surface of the housing 1014, which captures video of the surrounding real space in a field of view corresponding to the user's line of sight. In this specification, the camera 1015 may be referred to as a light detection unit 1015.

[0036] The camera 1015 is, for example, a light detection unit, and includes a light detection device such as an image sensor or a distance measurement sensor, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complemented Metal Oxide Semiconductor) sensor, and an optical system such as an imaging lens. For example, in FIG. 1A , the camera 1015 is configured as a stereo camera that captures images of the space in front of the user from left and right viewpoints corresponding to the user's left and right eyes. Note that the camera 1015 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may be used. For hand tracking applications, the camera 1015 may be configured to capture images of the space below the information processing system. For eye tracking or face tracking applications, the camera 1015 may be configured to capture images of the user's eyes or face.

[0037] The HMD 1001 also includes a sensor 2008 ( FIG. 2 ). The sensor group may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0038] The HMD 1001 may be connected to other processing devices by wireless communication, or may be connected by wire using a USB (Universal Serial Bus) or the like.

[0039] In this case, the HMD 1001 may be configured to run an online application such as a game in which multiple users can participate via a network. In this case, the HMD 1001 performs predetermined processing on the image captured by the camera 1015, and generates and displays a display image within the field of view of the camera 1015.

[0040] The content of the displayed image is not particularly limited, and may vary depending on the functions the user desires from the system and the content of the application that has been started.

[0041] For example, the HMD 1001 may perform some processing on the image captured by the camera 1015, or may superimpose a virtual object that interacts with the image of a real object. Alternatively, the HMD 1001 may render a virtual world in a field of view corresponding to the user's field of view based on the captured image or measurements by a motion sensor included in the sensor group of the HMD 1001.

[0042] Representative examples of these modes include virtual reality (VR), augmented reality (AR), and mixed reality (MR). In addition, a see-through mode (VST: Video See Through) in which the real world can be seen through the screen of the HMD 1001 may be realized by directly displaying an image captured by the camera (light detection unit) 1015 as a display image.

[0043] An information processing system 1101 according to this embodiment shown in FIG. 1B is configured as a glasses-type HMD.

[0044] The HMD main body 1111 is worn on the user's head when in use. The HMD main body 1111 has a front part 1112, a right temple part 1113 provided on the right side of the front part 1112, a left temple part 1114 provided on the left side of the front part 1112, and a glass part 1115 attached to the underside of the front part 1112. Note that although the glass is integrated in FIG. 1B , it may have two separate glasses, one for each eye, or may be configured to cover only one eye.

[0045] The display unit 1103 is a see-through type display unit and is provided on the surface of the glass unit 1115. The display unit 1103 performs AR display of a virtual object in accordance with the control of the processing circuit (control unit) 2001. Note that the display unit 1103 may be a non-see-through type display unit. In this case, AR display is performed by displaying on the display unit 2005 an image in which the virtual object is superimposed on an image currently captured by the camera 1104.

[0046] The camera 1104 is, for example, a light detection unit, and includes a light detection device such as an image sensor, such as a CCD sensor or a CMOS sensor, or a distance measurement sensor, and an optical system, such as an imaging lens. The camera 1104 is provided facing outward on the outer surface of the front unit 1112, captures an image of an object in real space, and outputs the image information obtained by the capture to the processing circuit (control unit) 2001. In FIG. 1B , for example, two cameras 1104 are provided laterally at a predetermined interval on the front unit 1112. Note that the camera (light detection unit) 1015 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may be used. In hand tracking applications, the camera 1104 may be provided to capture an image of the space below the information processing system. In eye tracking or face tracking applications, the camera 1104 may be provided to capture an image of the user's eyes or face.

[0047] The glasses-type HMD 1101 also includes a sensor 2008 ( FIG. 2 ). The sensor unit may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0048] The glasses-type HMD 1101 may also include a communication IF 2009 ( FIG. 2 ), which communicates with a smartphone or an external device other than a smartphone (for example, a personal computer (PC) or a server device on a network) via a wired or wireless connection.

[0049] <Example of Hardware Configuration of HMD> Next, an example of the hardware configuration of the information processing system (HMD 1001 or glasses-type HMD 1101) will be described with reference to Fig. 2. As shown in Fig. 2, the hardware of the information processing system is composed of a CPU 2001, a memory 2002, a camera 2003, a display unit 2005, an input unit 2006, an output unit 2007, a sensor 2008, a communication interface (IF) 2009, an external network 2010, and a secondary storage device 2011, which are mutually connected via a bus 2012 and can transmit and receive data and programs.

[0050] The processing circuit 2001 operates based on programs stored in the memory 2002 or the secondary storage device 2011 and controls the overall operation of the information processing systems 1001 and 1101. The processing circuit is, for example, a processor, which reads and executes each program from the memory 2002 to realize the corresponding function of each program. The processor may include, for example, any one or more of a multi-core processor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or an equivalent discrete or integrated logic circuit. The processing circuit may be realized as multiple chips.

[0051] Memory 2020 may include any type of memory for storing data and executable software instructions, implemented, for example, by random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or semiconductor memory devices such as flash memory, hard disk, optical disk, etc.

[0052] 1A and the camera 1104 in FIG. 1B, and includes a light detection device such as an image sensor or a distance measurement sensor, such as a CCD sensor or a CMOS sensor, and an optical system such as an imaging lens. The camera 2003 has a light detection device 2004.

[0053] The display unit 2005 is a display panel provided inside the housing, and is made up of a display device such as an LCD (Liquid Crystal Display) or an organic EL (ElectroLuminescence).

[0054] The input unit 2006, which is not shown in Figures 1A and 1B, is composed of input devices such as a keyboard, mouse, touch panel, microphone, and controller through which the user inputs operation commands, and supplies various input signals to the processing circuit (control unit) 2001.

[0055] The output unit 2007 is composed of an audio output device such as a speaker, a force feedback device, an odor feedback device, etc., and is controlled by the processing circuit 2001 to output the processing results as sound, force feedback, or odor.

[0056] The sensor 2008 may include at least one of various sensors, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor, for detecting the movement, posture, and position of the HMDs 1001 and 1101. The sensor 2008 may also include a biosensor that senses a person's biometric information and a pressure sensor that detects input.

[0057] The communication interface 2009 is an interface for connecting the information processing systems 1001 and 1101 to an external network 2010. For example, the processing circuit 2001 receives data from other devices and transmits data generated by the processing circuit 1100 to other devices via the communication interface 2009.

[0058] The above describes an example of an information processing system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the light detection device 2004 in the above-described configuration.

[0059] The photodetector 2004 according to the present disclosure realizes the camera 1015 in FIG. 1, and is composed of a CCD sensor or a CMOS sensor, and outputs the captured image to the signal processing device 111 included in the processing circuit 2001.

[0060] Based on the gaze point information supplied from the gaze point detection unit described later, the light detection device 2004 considers the gaze point area, which is the range in the vicinity of the gaze point position in the captured image, to be the area of ​​interest that the user is paying attention to, and changes the resolution between the area of ​​interest and areas other than the area of ​​interest.

[0061] 1, the light detection device 2004 will be described as being configured by a stereo camera that captures images of the space in front from left and right viewpoints corresponding to the left and right eyes of the user of the HMD 1001. Unless otherwise specified, the description will proceed assuming that the camera 1015 and the light detection device 2004 basically have the same configuration.

[0062] The optical detection device 2004 of this embodiment is capable of outputting images of a first region including the gaze region (e.g., a Z1 image of the Z1 zone), a second region arranged to surround at least a portion of the first region and having a different resolution from the first region (e.g., a Z2 image of the Z2 zone), and a third region arranged to surround at least a portion of the second region and having a different resolution from the second region (e.g., a Z3 image of the Z3 zone) at different frame rates.

[0063] In addition, in the display unit 2005 of FIG. 2 , the region of interest is displayed based on the Z1 image output from the photodetector 2004, and the region other than the region of interest is displayed based on at least one of the Z2 and Z3 images output from the photodetector 2004. This reduces the display processing load in the information processing systems 1001 and 1101, thereby reducing the power consumption of the information processing systems 1001 and 1101. Also, the display unit 2005 can display images based on image data output from the photodetector 2004 at different frame rates. Furthermore, in the signal processing device 111 included in the processing circuit 2001 of the information processing systems 1001 and 1101, the Z1 image is processed at a first frequency and the Z2 image is processed at a second frequency. In this way, by processing the Z1 image corresponding to the region of interest at the first frequency and the Z2 and Z3 images corresponding to the region other than the region of interest at a second frequency lower than the first frequency, the image processing load can be reduced, thereby reducing the power consumption of the information processing system.

[0064] <Configuration Example of Gaze Point Detection Unit> The following mainly describes an example using the HMD 1001 of FIG. 1A, but the present disclosure is also applicable to the HMD 1011 of FIG. 1B. FIGS. 3 and 4 are diagrams illustrating the configuration of the HMD 1001 according to the present disclosure. The gaze point detection unit 59 shown in FIGS. 3 and 4 is provided in at least one of the processing circuit 2001 and the sensor 2008 of FIG. 2. FIG. 3 is a diagram illustrating an example of the configuration of the gaze point detection unit 59 of the HMD 1001. FIG. 4 is a block diagram illustrating the connection relationship between the gaze point detection unit 59, an application processor (hereinafter, AP) 59AP, and a light detection device 2004 according to an embodiment. In this specification, the light detection device 2004 according to an embodiment may be referred to as an imaging unit 60.

[0065] The bottom part of Fig. 3 shows a schematic view of the HMD 1001 and the head F of the user wearing it, as seen from above. The HMD 1001 displays a display image 61P as shown in the top part, which the user views with the left and right eyes EL and ER. The display image 61P is composed of images 61PL and 61PR viewed by the left and right eyes EL and ER, respectively. In this example, the gaze point detection unit 59 includes infrared LEDs 59STL and 59STR, eye image sensors 59SRL and 59SRR each consisting of an infrared camera or a PSD (Position Sensitive Detector) sensor, and an image analysis device 59A.

[0066] The infrared LEDs 59STL and 59STR irradiate the user's left and right eyes EL and ER with infrared rays, respectively. The eye image sensors 59SRL and 59SRR capture images of the user's left and right eyes EL and ER, respectively, and supply the image data to the image analysis device 59A.

[0067] The image analyzer 59A identifies the position of the pupil and the position of the infrared light reflected by the cornea from the captured images of the left and right eyes, and identifies the user's line of sight from this positional relationship. This method, known as the corneal reflex method, has been put to practical use in the field of line of sight detection technology.

[0068] The method for detecting the gaze is not limited to this, and any common method may be used, such as capturing images of the left and right eyes with a visible light camera and identifying the gaze from the relative positions of the inner corners of the eyes and the irises.

[0069] The image analysis device 59A determines the intersection RPL, RPR between the line of sight detected by the eyeball image sensors 59SRL, 59SRR and the display unit 61 (display panel) on which the display image 61P is displayed as the user's point of gaze, detects its position coordinates, and supplies them to the image capture unit 60.

[0070] FIG. 4 shows an example of the hardware configuration of the gaze point detection unit 59 and the spatial imaging unit 35.

[0071] As shown in Figure 4, the gaze point detection unit 59 includes an eyeball image sensor 59S (59STL, 59STR) that images the eyeball, and an application processor 59AP that essentially functions as an image analysis device 59A to acquire and detect the gaze point from the captured image of the eyeball.

[0072] Based on the gaze point information supplied from the application processor 59AP, the imaging unit 60 sets different resolutions and frame rates in areas other than the area near the gaze point (gape point area), i.e., areas other than the attention area, and captures images.

[0073] In this configuration, the MIPI (Mobile Industry Processor Interface) architecture can be used for data transmission of captured images from the eye image sensor 59S (59STL, 59STR) to the application processor 59AP.

[0074] In addition, the i2C (Inter-Integrated Circuit) or i3C architecture can be used to transmit information relating to the point of interest from the application processor 59AP to the imaging unit 60.

[0075] The gaze point detection unit 59 may have an integrated configuration in which the eye image sensor 59S (59STL, 59STR) and the application processor 59AP are stacked.

[0076] <Configuration Example of Imaging Unit> Next, a configuration example of the imaging unit 60 and a gaze area will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing a configuration example of the imaging unit 60. Fig. 6 is a diagram showing examples of the gaze area and peripheral areas. Hereinafter, the gaze area will be referred to as zone Z1 or first area, the peripheral area of ​​the gaze area will be referred to as zone Z2 or second area, and the peripheral area of ​​zone Z2 or second area will be referred to as zone Z3 or third area.

[0077] 5 , the imaging unit 60 is capable of hardware binning, which combines 4 or 16 pixels into one pixel, and software binning, and includes a pixel unit 81, a vertical scanning circuit 82, an analog-to-digital converter (ADC) group 83, a horizontal transfer scanning circuit 84, a timing control circuit 85, a horizontal transfer line 86, an amplifier circuit 87, a signal processing circuit 88, an output circuit 89, and a region control unit 90.

[0078] The pixel section 81 has a configuration in which pixels, each including a photodiode that performs photoelectric conversion, are arranged in a matrix. Each pixel holds a signal charge corresponding to the amount of incident light. A vertical scanning circuit 82 drives the pixels row by row by supplying a drive pulse to each pixel via pixel drive wiring (not shown). As a result, analog signals from the pixels in each row are supplied to an ADC group 83 via vertical signal lines provided for each column.

[0079] More specifically, in hardware binning processing that combines four pixels into one pixel, the pixels are driven in units of two rows, and the signals are supplied to the ADC group 83 via vertical signal lines provided for each column. Furthermore, in hardware binning processing that combines 16 pixels into one pixel, the pixels are driven in units of four rows, and the signals are supplied to the ADC group 83 via vertical signal lines provided for each column. In other words, in hardware binning processing, analog binning processing, which reads analog signals from multiple rows simultaneously, is used to speed up readout. Note that the analog binning processing is not limited to this method, and other methods may also be used. For example, in hardware binning processing that combines four or 16 pixels into one pixel, a process of thinning out the readout pixels may be performed.

[0080] In software binning, pixels are driven row by row, and after analog-to-digital conversion, pixel addition is performed to combine 4 or 16 pixels into 1 pixel. A detailed example of the binning process will be described later with reference to FIGS. 7 to 10. The binning process is not limited to this method, and other methods may also be used.

[0081] The ADC group 83 has a configuration in which an ADC is arranged for each pixel column as an analog-to-digital conversion circuit, and the ADC is composed of a comparator 101 that compares a reference voltage generated by a reference voltage output circuit 100 with an analog signal obtained from a pixel for each row line via a vertical signal line, a counter 102 that counts the comparison time, and a latch 103 that holds the count result. Note that the ADC group 83 according to this embodiment corresponds to an analog-to-digital conversion unit.

[0082] The analog signal read out by the vertical signal line is compared with a reference voltage having a slope waveform by a comparator 101, and when the two match, a count value is obtained, thereby converting the signal into a digital signal. The output of each latch 103 is connected to a horizontal transfer line 86.

[0083] The horizontal transfer scanning circuit 84 is composed of, for example, a shift register, and by sequentially outputting horizontal scanning pulses, inputs one row of digital signals stored in the latch 103 to an amplifier circuit 87 and a signal processing circuit 88 via a horizontal transfer line 86.

[0084] The timing control circuit 85 generates and outputs clock signals and control signals that serve as the basis for the operations of the vertical scanning circuit 82, the ADC group 83, and the horizontal transfer scanning circuit 84, based on the vertical synchronization signal, horizontal synchronization signal, and master clock.

[0085] The signal processing circuit 88 performs software binning processing on each digital signal to be subjected to software binning processing, and also reconstructs each digital signal and the binned digital signal by associating them with the positions of the pixels arranged in a matrix in the pixel section 81. Thereafter, the signal processing circuit 88 performs predetermined processing such as demosaic processing and gamma correction to generate captured image data.

[0086] The output circuit 89 appropriately buffers the captured image data generated by the signal processing circuit 88 and outputs it row by row.

[0087] As shown in FIG. 6 , the region control unit 90 sets zones Z1, Z2, and Z3 based on the gaze point information supplied from the gaze point detection unit 59. For example, zone Z1 is a region where image data is generated in units of one pixel, zone Z2 is a region where image data is generated by grouping four pixels (2×2) into one pixel, and zone Z3 is a region where image data is generated by grouping 16 pixels (4×4) into one pixel. As described above, in zone Z1, for example, exposure and pixel signal readout are performed for each pixel. In zone Z2, binning processing is performed in units of multiple pixels, and pixel signals are read out for each four pixels. In zone Z3, binning processing is performed in units of multiple pixels, and pixel signals are read out for each 16 pixels.

[0088] The region control unit 90 also changes the frame rates of zone Z1, zone Z2, and zone Z3. For example, assume that zone Z1 is 400 x 400 pixels, zone Z2 is 400 x 400 pixels, and zone Z3 is 1000 x 1000 pixels. If the frame rates of zone Z1, zone Z2, and zone Z3 are 120 fps (frames per second), the output data rate of zone Z1 is 400 x 400 x 8 bit x 120 fps = 18 MB / s. Similarly, the output data rate of zone Z2 is 400 x 400 x 8 bit x 120 fps = 18 MB / s, and the output data rate of zone Z3 is 1000 x 1000 x 8 bit x 120 fps = 114 MB / s. The total output data rate is 150 MB / s.

[0089] On the other hand, if the frame rate of zone Z1 is set to 60 fps and the frame rates of zones Z2 and Z3 are set to 120 fps, the total amount of output data is reduced from 150 MB / s to 141 MB / s. Furthermore, if the frame rate of zone Z3 is set to 60 fps and the frame rates of peripheral area zones Z1 and Z2 are set to 120 fps, the total amount of output data is reduced from 150 MB / s to 93 MB / s. Furthermore, if the frame rate of zones Z2 and Z3 is set to 60 fps and the frame rate of peripheral area zone Z1 is set to 120 fps, the total amount of output data is reduced from 150 MB / s to 84 MB / s. In this way, by reducing the frame rate of any of zones Z1, Z2, and Z3, it is possible to further reduce power consumption and data bandwidth.

[0090] With this configuration, in the image captured by the imaging unit 60, the resolution of pixels in areas other than the focus area (focus area) can be reduced based on the focus information from the focus detection unit 59, thereby reducing the output time of the captured image data. Furthermore, by reducing the frame rate of any of zones Z1, Z2, and Z3 according to visual characteristics, the total amount of output data can be reduced. As a result, it is possible to suppress delays in the series of processes from imaging to display.

[0091] In the above, an example has been described in which an integral type configuration is adopted as the ADC constituting the ADC group 83, but other configurations may also be used, such as a successive approximation type ADC.

[0092] Here, a detailed example of control by the area control unit 90 will be described using Figures 7 to 11 while referring to Figure 6. For simplicity of explanation, the following describes the cases of zones Z1 and Z2, but similar control processing is also possible for zone Z3.

[0093] 7 is a diagram showing an example in which the frame rate of zone Z1 is set to 1 / N times the frame rate of zone Z2. In this embodiment, the ratio of the area with the higher frame rate to the area with the lower frame rate is 1:N, where N is a natural number. In the example of FIG. 7, N is 2. The center of zone Z1 is the gaze point RP.

[0094] Line LZ2 is the timeline of zone Z2, and line LZ1 is the timeline of zone Z1. (m+(n-1)) ｔｈ The frame indicates the frame order, and the image data of zone Z2 and the image data of zone Z1 are output at the timing of the intersections n2 and n1 of the line LZ2 and the line LZ1, respectively. n and m are natural numbers, and n ranges from 1 to 4 in FIG. 7. The display unit 61 displays (m+(n-1)) ｔｈ In other words, the display unit 61 displays the frame rate of zone Z1 at 1 / N times the frame rate of zone Z2.

[0095] As shown in FIG. 7, the gaze point RP is set to each (m+n) ｔｈ The image data of zone Z1 is output once every N frames. ｔｈ frame, (m+3) ｔｈ The resolution of zone Z2 in a frame is, for example, one-fourth that of zone Z1, and the readout speed is also increased. Therefore, the output data is reduced compared to when image data of zone Z1 is output for all frames. As a result, it is possible to suppress delays in the series of processes from image capture to display.

[0096] 8 is a diagram showing an example in which the frame rate of zone Z2 is set to 1 / N times the frame rate of zone Z1. As described above, in this embodiment, the ratio of the area with the higher frame rate to the area with the lower frame rate is set to 1:N, where N is a natural number. In the example of FIG. 8, N is 2. The center of zone Z1 is the gaze point RP.

[0097] Line LZ2 is the timeline of zone Z2, and line LZ1 is the timeline of zone Z1. (m+(n-1)) ｔｈ The frame indicates the frame order, and the image data of zone Z2 and the image data of zone Z1 are output at the timing of the intersections n2 and n1 of the line LZ2 and the line LZ1, respectively. n and m are natural numbers, and n ranges from 1 to 4 in FIG. 8. The display unit 61 displays (m+(n-1)) ｔｈ In other words, the frame rate of zone Z2 is displayed on the display unit 61 as 1 / N times the frame rate of zone Z1.

[0098] As shown in FIG. 8, the gaze point RP is set to each (m+n) ｔｈ The image data for zone Z1 varies for each frame, and the image data for zone Z1 is output for every frame. On the other hand, the image data for zone Z2 is output once every N frames. Therefore, the amount of output data is reduced compared to when the image data for zone Z1 and the image data for zone Z2 are output for every frame. As a result, it is possible to suppress delays in the series of processes from imaging to display. Furthermore, (m+1) ｔｈ frame, (m+3) ｔｈ The image data of zone Z2 in the frame may not be output, or the image data of zone Z2 in the series of previous frames may be processed and output.

[0099] 9 is a diagram showing a read sequence for each frame, and shows an example in which image data for zone Z1 and image data for zone Z2 are read out for each frame.

[0100] The position setting line ZL indicates the position of zone Z1. At timing Lb, the position of zone Z1 moves from position A to position B, and at timing LC, the position of zone Z1 moves from position B to position C. The synchronization signal XVS is a signal that indicates the frame readout timing output by the region control unit 90. The readout of each frame is switched by the synchronization signal XVS. The vertical axis of the sequence indicates the row of the pixel unit 81, and the horizontal axis indicates time.

[0101] The line SH of the previous frame indicates the reset timing of the pixels in each row of the pixel section 81 and indicates the start timing of charge accumulation for each pixel. The line RD of the current frame indicates the readout timing of the pixels in each row of the pixel section 81 and indicates the start timing of charge readout for each pixel. The line SH of the current frame also indicates the reset timing of the pixels relative to the readout timing of the pixels in the next frame.

[0102] The time interval between line SH of the previous frame and line RD of the current frame is the same. In other words, line SH and line RD are parallel. Zone Z2, indicated by the dashed line, on line RD is read out in units of two rows of the pixel section 81, while zone Z1, indicated by the solid line, is read out in units of one row of the pixel section 81. Therefore, the slope of the row with respect to time indicated by the dashed line is twice the slope of the row with respect to time indicated by the solid line. In other words, the readout speed of zone Z2 when reading out the same number of rows is twice the readout speed of zone Z1.

[0103] As described above, the scanning speed of each pixel row differs between zone Z1 and zone Z2. In zone Z1, pixel signals from all pixels must be read out, so the scanning speed of each pixel row by the vertical scanning circuit 82 cannot be made very fast. In contrast, in zone Z2, the scanning speed can be made faster than that of zone Z1 because it is only necessary to read pixel signals from the number of pixels thinned out by the binning process. By combining zones Z1 and Z2 to read pixel signals, the time required to read all pixel signals in each frame can be reduced, and as a result, the vertical blanking period VBLK of each frame can be lengthened. During the vertical blanking period VBLK, the operation of the peripheral circuits of the pixel section 81 can be stopped, so by lengthening the vertical blanking period VBLK, power consumption can be reduced.

[0104] 6 again, there is an area where zone Z2 and zone Z1 are set on the same row. In such a case, the row-by-row readout of zone Z1 is also performed on zone Z2. For this reason, the signal processing circuit 88 performs a software binning process on the pixel signals of zone Z2, which have been read out row by row, after analog-to-digital conversion, to combine the signals of four pixels into one pixel.

[0105] Fig. 10 is a diagram showing a sequence corresponding to Fig. 7. That is, this is an example in which the frame rate of zone Z1 is set to 1 / N times the frame rate of zone Z2.

[0106] The processing designation line PC indicates the readout region. "Z1 full RD yes" is a frame in which zone Z1 and zone Z2 are read out. "Z1 full RD no" is a frame in which binning processing is performed with all pixels as zone Z2. As described above, the position setting line ZL indicates the position of zone Z1, and the synchronization signal XVS is a signal indicating the frame readout timing output by the region control unit 90. The vertical axis of the sequence indicates the row of the pixel unit 81, and the horizontal axis indicates time.

[0107] The line SH indicates the timing for resetting the pixels in each row of the pixel section 81 and indicates the timing for starting charge accumulation for each pixel. The line RD indicates the timing for reading out the pixel signals from each pixel in each row of the pixel section 81 and indicates the timing for starting charge readout for each pixel.

[0108] In a frame with "Z1 full RD", the time difference between each line of the line SH of the previous frame and each line of the line RD of the current frame is the same as in Fig. 9. On the other hand, the time difference between each line of the line SH of the current frame occurring in a frame with "Z1 full RD" and each line of the line RD of the next frame is the same.

[0109] Similarly, in a frame with "Z1 full RD not present," the time difference between each row of line SH in the previous frame and line RD in the current frame is the same. In a frame with "Z1 full RD not present," all pixels are read out as zone Z2, so the slope of line RD in the current frame is twice the slope of zone Z1 of line SH in the current frame. Therefore, the vertical blanking period VBLK in a frame with "Z1 full RD not present" is longer than the vertical blanking period VBLK in a frame with "Z1 full RD present." As can be seen from these, as the number of frames with "Z1 full RD not present" increases, the power consumption of the pixel unit 81 is reduced.

[0110] Fig. 11 is a diagram showing a sequence corresponding to Fig. 8. That is, this is an example in which the frame rate of zone Z2 is set to 1 / N times the frame rate of zone Z1.

[0111] The processing designation line PC indicates the readout area. "Z2 RD None" is a frame in which only zone Z1 is read out. The other frames are frames in which zones Z1 and Z2 are read out. As described above, the position setting line ZL indicates the position of zone Z1, and the synchronization signal XVS is a signal indicating the frame readout timing output by the area control unit 90. The vertical axis of the sequence indicates the row of the pixel unit 81, and the horizontal axis indicates time.

[0112] The line SH indicates the timing for resetting the pixels in each row of the pixel section 81 and indicates the timing for starting charge accumulation for each pixel. The line RD indicates the timing for reading out the pixels in each row of the pixel section 81 and indicates the timing for starting charge readout for each pixel.

[0113] In a frame with "Z2 RD not present," the time difference between each row of line SH of the previous frame and line RD of the current frame is the same. In this case, data is actually read only from the solid-line area of ​​line RD of the current frame. Therefore, the vertical blanking period VBLK in a frame with "Z2 RD not present" is longer than the vertical blanking period VBLK in other frames. As can be seen from this, as the number of frames with "Z2 RD not present" increases, the power consumption of the pixel unit 81 is reduced.

[0114] 12 is a diagram showing an example of the data structure of image data for each zone transmitted from the photodetector corresponding to FIG. 7. It is a diagram showing an example of the data structure output by the output circuit 89. This example data structure corresponds to the sequence of FIG. 7. In other words, this is an example in which the frame rate of zone Z1 is 1 / N times the frame rate of zone Z2. FIG. 12 shows an example in which the data for zone Z1 and zone Z2 are output as two frames. Frame FZ1 is an example of the data structure for zone Z1, and frame FZ2 is an example of the data structure for zone Z2.

[0115] The image data in each of zones Z1 to Z3 includes a frame start signal (FS), virtual channel information (VC), embedded data (EBD), a packet footer (PF), a data type (DT), extra data (EXD), and effective pixel data (EF).

[0116] The zone setting information is included in, for example, the embedded data (EBD). As described above, the zone setting information for each of zones Z1 to Z3 includes information about the positions and sizes of all of zones Z1 to Z3. The spare data (EXD) includes information about the timing at which reading of each row started (see line RD in FIG. 10), etc. In addition, different Virtual Channel IDs are embedded in the virtual channel information (VC) of frame FZ1 and the virtual channel information (VC) of frame FZ2.

[0117] In the effective pixel data (EF) of the first frame FZ2, data for the area of ​​zone Z1 included in zone Z2 is generated by performing software binning on the data for the area of ​​zone Z1 using the signal processing circuit 88. In this way, when zone Z1 is included within the area of ​​zone Z2, the output circuit 89 adds the first image data corresponding to zone Z1 and outputs image data with the same second resolution as zone Z2 and the second image data corresponding to zone Z2 in an image data structure contained within the same row.

[0118] The effective pixel data (EF) of the second frame FZ2 includes effective pixel data obtained by taking all pixels as zone Z2, so that the size of the data structure of the first frame FZ2 is equal to the size of the data structure of the second frame FZ2.

[0119] FIG. 13 corresponds to FIG. 7 and shows an example of a data structure when image data for each zone is combined by the signal processing circuit 88. It also shows an example of a data structure output by the output circuit 89. This example data structure corresponds to the sequence in FIG. 7. That is, this is an example in which the frame rate of zone Z1 is 1 / N times the frame rate of zone Z2. FIG. 13 shows an example in which the data for zones Z1 and Z2 is output as one frame. Frame FZS is an example of a data structure when the data for zones Z1 and Z2 is one frame, and frame FZ2 is an example of the data structure for zone Z2.

[0120] Because the effective pixel data (EF) of frame FZS must be matched to the resolution of zone Z1, when averaging adjacent n x n pixels, n identical data pieces are consecutive in the horizontal direction in zone Z2. Similarly, n identical data pieces are consecutive in the vertical direction. In this way, when zone Z1 is included in zone Z2, the output circuit 89 outputs image data in a structure in which the second image data corresponding to zone Z2 and the first image data corresponding to zone Z1 are included in the same row.

[0121] Fig. 14 is a diagram showing an example of a data structure corresponding to Fig. 8. That is, this is an example in which the frame rate of zone Z2 is 1 / N times the frame rate of zone Z1. Fig. 14 shows an example in which the data of zone Z1 and zone Z2 is output as two frames. As in Fig. 12, frame FZ1 is an example of the data structure of zone Z1, and frame FZ2 is an example of the data structure of zone Z2.

[0122] 15 is a diagram showing an example of a data structure when image data from each zone is combined by the signal processing circuit 88. It is a diagram showing an example of a data structure corresponding to FIG. 8. That is, this is an example in which the frame rate of zone Z2 is 1 / N times the frame rate of zone Z1. As with FIG. 13, this is an example in which the data from zones Z1 and Z2 is output as one frame. Frame FZS is an example of a data structure when the data from zones Z1 and Z2 is one frame, and frame FZ1 is an example of the data structure of zone Z1.

[0123] As described above, according to this embodiment, in an image captured by the imaging unit 60, the area control unit 90 lowers the resolution of pixels in zones Z2 and Z3 other than zone Z1 based on the gaze point information from the gaze point detection unit 59, thereby making it possible to reduce the output time of captured image data. Furthermore, by reducing the frame rate of any of zones Z1, Z2, and Z3 in accordance with visual characteristics, it is possible to reduce the power consumption and data bandwidth of the photodetector 1 without the user perceiving any impact.

[0124] (General Configuration of Information Processing System) The light detection device 1 according to the present disclosure can be incorporated into an information processing system having a gaze detection function.

[0125] 16 is a block diagram showing a schematic configuration of an information processing system 1001 including the light detection device 1 according to the present disclosure. The information processing system 1001 in FIG. 16 includes the light detection device 1, a signal processing device 111, and a gaze detection device 112.

[0126] Specifically, the signal processing device 111 is configured by an ISP (Image Signal Processor) or an AP (Application Processor).

[0127] The photodetector 1 has a configuration similar to that shown in Fig. 5. Fig. 16 shows a simplified internal configuration of the photodetector 1 of Fig. 5, and includes at least a pixel unit 81, a signal processing circuit 88, and an interface circuit (IF) 113. The interface circuit (IF) 113 of Fig. 16 corresponds to the output circuit 89 of Fig. 5.

[0128] The signal processing device 111 includes a frame buffer 114 , a first processing unit (IFE) 115 , and a second processing unit (IPE) 116 .

[0129] The frame buffer 114 stores image data for each frame output from the photodetector 1. As will be described later, the photodetector 1 generates image data for each of a plurality of zones. Thus, the frame buffer 114 stores image data for each of the plurality of zones.

[0130] The first processing unit 115 performs various image processing on each image data to generate a luminance signal and a color difference signal (YUV signal).

[0131] The second processing unit 116 performs various image processing and image synthesis processing based on a plurality of luminance signals and color difference signals corresponding to a plurality of zones, and generates a synthesized luminance signal and color difference signal (YUV signal).

[0132] The gaze detection device 112 detects the gaze of the user and transmits information about the gaze position (hereinafter also referred to as the gaze position) to the signal processing device 111. More specifically, the signal processing device 111 calculates zone information based on the gaze detection information detected by the gaze detection device 112 and transmits the calculated zone information to the light detection device 1.

[0133] The photodetector 1 generates image data by changing the resolution for each zone based on the zone information transmitted from the signal processor 111 .

[0134] (Zone Setting Restrictions) The light detection device 1 according to the present disclosure sets a plurality of zones in the pixel unit 81 based on the gaze position detected by the gaze detection device 112. In doing so, the light detection device 1 needs to comply with the zone setting restrictions.

[0135] More specifically, in the photodetector 1 according to the present disclosure, a region including the line of sight detected by the line of sight detection device 112 at its center position is defined as zone Z1, and a region including zone Z1 is defined as zone Z2. Zone Z1 is a region where photoelectrically converted pixel signals are output pixel by pixel. Zone Z2 is a region where binning processing is performed, in which pixel signals from multiple pixels are combined and output as a pixel signal for a single pixel. For example, if zone Z2 is subjected to 2×2 pixel binning processing, zone Z2 has a size that is a multiple of four of zone Z1.

[0136] The photodetector 1 according to the present disclosure may have three or more zones. For example, if a zone Z3 is provided that encompasses zone Z2, binning processing is performed in zone Z3 using pixel regions larger in size than zone Z2. For example, if binning processing is performed on 4×4 pixels in zone Z3, zone Z3 has a size that is a multiple of 16 of zone Z1.

[0137] There are constraints that zone Z2 must contain the entire area of ​​zone Z1, and zone Z3 must contain the entire area of ​​zone Z2. For example, if all sides of zone Z1 are inside all sides of zone Z2 as in Fig. 17A, or if at least some sides of zone Z1 are in contact with any side of zone Z2 as in Fig. 17B, the zone setting is determined to be valid (OK), but if at least some sides of zone Z1 are located outside any side of zone Z2 as in Fig. 17C, the zone setting is determined to be invalid (NG: error).

[0138] The light detection device 1 according to the present disclosure needs to set multiple zones based on the gaze position so as to satisfy the above-mentioned zone setting constraints. Because the gaze position detected by the gaze detection device 112 may change for each frame, the light detection device 1 may change the positions and sizes of the multiple zones for each frame based on the zone information from the signal processing device 111.

[0139] (Zone Setting ID) The photodetector 1 transmits image data for each zone to the signal processing device 111 using, for example, a virtual channel compliant with the MIPI (Mobile Industry Processor Interface) standard. At this time, the signal processing device 111 needs to know which zone of which frame the image data transmitted from the photodetector 1 includes. Therefore, the photodetector 1 according to the present disclosure is provided with a zone setting ID. The zone setting ID is information that identifies the frame to which the read image data belongs. The photodetector 1 according to the present disclosure adds the zone setting ID to the image data and transmits it to the signal processing device 111.

[0140] The zone setting ID may be assigned by the photodetector 1 or by the signal processing device 111 .

[0141] 18 is a sequence diagram showing the procedure for transmitting and receiving image data between the photodetector 1 and the signal processing device 111 when the photodetector 1 assigns a zone setting ID. When the signal processing device 111 receives gaze detection information from the gaze detection device 112 (step S1), it calculates zone information relating to the position and size of a zone Z1 that includes the gaze position at its center (step S2). The zone information is stored, for example, in a zone setting register, which will be described later.

[0142] The signal processing device 111 transmits the calculated zone information to the photodetector 1 (step S3). The photodetector 1 assigns a zone setting ID corresponding to the zone information (step S4). The zone setting ID is associated with a frame from which pixel signals are read out.

[0143] The photodetector 1 transmits the zone setting ID to the signal processing device 111 (step S5). The photodetector 1 sets zones Z1 to Z3 in the pixel unit 81, performs an exposure operation, and then reads out pixel signals. For example, in zone Z1, pixel signals are read out for each pixel, whereas in zones Z2 and Z3, pixel signals are read out by performing binning processing for each set of pixels (step S6). After the photodetector 1 sets the zones, a time delay of N frames (N is an integer greater than or equal to 1) may occur before the image signal reflecting the zone setting is read out.

[0144] The photodetector 1 transmits a packet in which the corresponding zone setting ID is added to the image data for each frame to the signal processor 111 via a virtual channel conforming to the MIPI standard, for example (step S7).

[0145] The signal processing device 111 compares the zone setting ID attached to the received image data with the zone setting ID received in step S5 to identify which frame the received image data belongs to (step S8).

[0146] Fig. 19 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device 111 of Fig. 18. Fig. 19 shows an example of storing setting information for zones Z1 to Z3 in the zone setting register. The zone setting register stores the two-dimensional coordinates of the first pixel and the last pixel in each of zones Z1 to Z3.

[0147] Fig. 20 is a diagram showing an example of the data structure of image data that the photodetector 1 transmits to the signal processing device 111 in step S7 of Fig. 18. The image data in Fig. 20 is data in which the pixel values ​​(16-bit data) of each pixel are arranged in the order of zones Z3, Z2, and Z1, and this data is followed by 16-bit data representing the zone setting ID.

[0148] 21 is a sequence diagram showing a procedure for transmitting and receiving image data between the photodetector 1 and the signal processing device 111 when the signal processing device 111 assigns a zone setting ID. When the signal processing device 111 receives gaze detection information from the gaze detection device 112 (step S11), it calculates zone information relating to the position and size of a zone Z1 that includes the gaze position at its center (step S12) and assigns a zone setting ID (step S13). This sequence differs from the sequence diagram of FIG. 18 in that the signal processing device 111 assigns a zone setting ID.

[0149] The signal processing device 111 adds the zone setting ID to the calculated zone information and transmits it to the photodetector 1 (step S14).

[0150] The photodetector 1 sets, for example, zones Z1 to Z3 in the pixel section 81, performs an exposure operation, and then reads out pixel signals from each zone. For example, in zone Z1, pixel signals are read out for each pixel, whereas in zones Z2 and Z3, pixel signals are read out by performing binning processing for each set of pixels (step S15).

[0151] The photodetector 1 transmits a packet in which the corresponding zone setting ID is added to the image data for each frame to the signal processor 111 via a virtual channel conforming to the MIPI standard, for example (step S16).

[0152] The signal processing device 111 compares the zone setting ID attached to the received image data with the zone setting ID assigned by itself in step S13 to identify which frame the received image data is from (step S17).

[0153] Fig. 22 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device 111 of Fig. 21. The zone setting register of Fig. 22 contains information on the zone setting ID numbered in step S13. The other information is the same as that of Fig. 20.

[0154] The data structure of the image data that the photodetector 1 transmits to the signal processing device 111 in step S16 is the same as that shown in FIG. 20, and therefore a description thereof will be omitted.

[0155] 18 to 22 show an example in which a zone setting ID is added when image data is transmitted from the photodetector 1 to the signal processing device 111. However, it is preferable to add information about the position and size of each zone (hereinafter, "zone setting information") to the image data in addition to the zone setting ID and transmit the image data from the photodetector 1 to the signal processing device 111. The reason for this is that the photodetector 1 can change the position and size of each zone for each frame. Therefore, unless the position and size of each zone are notified to the signal processing device 111, the signal processing device 111 cannot grasp the position and size of each zone in each frame, resulting in poor utilization of the storage capacity of the frame buffer 114. Therefore, the photodetector 1 according to the present disclosure adds zone setting information for all zones to the header of image data for each zone transmitted to the signal processing device 111. This allows the signal processing device 111 to grasp the position and size of all zones regardless of which zone's image data it receives, thereby ensuring the storage capacity of the frame buffer 114 for storing the received image data for each zone.

[0156] Fig. 23 is a diagram showing the data structure of image data for each zone transmitted from the photodetector 1 to the signal processing device 111. Fig. 23 shows an example of setting zones Z1 to Z3 in the pixel unit 81. Fig. 23 shows the data structures of zones Z3, Z2, and Z1 in order from left to right. The image data for each of zones Z1 to Z3 includes a frame start signal (FS), virtual channel information (VC), embedded data (EBD), a packet footer (PF), a data type (DT), and valid pixel data.

[0157] The zone setting information is included in, for example, the embedded data (EBD). As described above, the zone setting information for each of the zones Z1 to Z3 includes information about the positions and sizes of all of the zones Z1 to Z3.

[0158] The zone setting information needs to be updated whenever the position and / or size of a zone changes, and since the position and / or size of a zone may change from frame to frame, the zone setting information may also need to be updated from frame to frame.

[0159] However, if there is no notification of zone information update from the signal processing device 111, the photodetector 1 continues to use the zone setting information of the immediately preceding frame. Since the gaze detection device 112 and the photodetector 1 usually operate asynchronously, the timing at which the gaze detection device 112 detects gaze movement does not necessarily coincide with the frame switching timing in the photodetector 1. In general, the gaze detection cycle by the gaze detection device 112 is often shorter than the frame cycle of the photodetector 1.

[0160] The above-described zone setting information may include only gaze coordinates instead of all coordinates of each zone. The left side of Fig. 24 shows an example in which the coordinates of all pixels in each zone are included in the zone setting information. The right side of Fig. 24 shows an example in which the sizes of zones Z1 and Z2 are registered in advance, and the center positions of each zone Z1 and Z2 are set as gaze positions, and only the coordinates of the gaze positions are included in the zone setting information. In the example on the right side of Fig. 24, the sizes of zones Z1 and Z2 cannot be changed, but the coordinate information for zones Z1 and Z2 included in the zone setting information can be significantly reduced.

[0161] 16, the line-of-sight information detected by the line-of-sight detection device 112 is input to the signal processing device 111. However, the line-of-sight information detected by the line-of-sight detection device 112 may be input directly to the light detection device 1.

[0162] The gaze detection device 112 may output gaze coordinates obtained by normalizing the coordinates of the pixel unit 81. Alternatively, the gaze detection device 112 may convert the physical gaze coordinates detected by the gaze detection device 112 into the coordinates of the pixel unit 81 and output the converted coordinates.

[0163] (Prediction of Gaze Coordinates) Gaze information detected by the gaze detection device 112 is input to the photodetector 1 via the signal processing device 111. The photodetector 1 performs exposure operations and pixel signal readout operations based on the gaze position detected by the gaze detection device 112, so there is a time lag between when the gaze detection device 112 detects the gaze position and when the photodetector 1 reads out pixel signals from the zone corresponding to the gaze position. Furthermore, the period in which the gaze detection device 112 detects the gaze is different from the frame period for exposure and pixel signal readout by the photodetector 1, and the two operate asynchronously.

[0164] FIG. 25 is a diagram showing the time lag between when the gaze detection device 112 detects the gaze position and when the photodetector 1 reads out pixel signals from a zone corresponding to the gaze position. The vertical lines in FIG. 25 represent the gaze detection cycle and the frame switching timing of the VST sensor. That is, by the time the photodetector 1 reads out pixel signals, the gaze may have moved to another position. Therefore, the gaze movement during the time lag shown in FIG. 25 may be predicted, and pixel signals may be read out from a zone corresponding to the predicted gaze position. Any algorithm may be used to predict the gaze position; for example, the gaze position may be predicted by linearly interpolating the gaze positions in two or more past frames.

[0165] (Gaze position in three-dimensional coordinates) Some gaze detection devices 112 output the gaze (gaze) position of a person as coordinates in three-dimensional space (three-dimensional coordinates). In addition, the light detection device 1 according to the present disclosure may include a VST (Video See Through) sensor that generates a captured image for the left eye and a VST sensor that generates a captured image for the right eye.

[0166] FIG. 26 is a diagram illustrating the gaze position and coordinate designation of each zone in two VST sensors for the left and right eyes. The photodetector 1 projects three-dimensional coordinates 118 on a display plane 117 virtually arranged in three-dimensional space corresponding to the gaze (gaze) position output from the gaze detection device 112 onto an image plane 119 of the VST sensor for the left eye and an image plane 120 of the VST sensor for the right eye, respectively, to calculate coordinates corresponding to the gaze position on each image plane. Note that the virtual display plane 117 in FIG. 26 does not necessarily have to coincide with the physical display plane. The coordinates of zones Z1 to Z3 are designated on the image plane for the left eye and the image plane for the right eye, respectively.

[0167] Even when a VST sensor is used, as shown in Fig. 25, a time lag occurs between when the gaze position is detected by gaze detection device 112 and when image data for each zone is read out on each image plane 119, 120. Therefore, as shown in Fig. 25, the movement of the gaze during the time lag may be predicted, and pixel signals may be read out from the zone corresponding to the predicted gaze position.

[0168] Fig. 27 is a diagram showing data transmitted and received between two VST sensors (hereinafter referred to as CIS1(L) and CIS2(R)) that constitute the gaze detection device 112. Fig. 27 shows an example in which the signal processing device 111 is an AP (Application Processor). Fig. 27 shows an example in which CIS1(L) is the primary (main sensor) and CIS2(R) is the secondary (sub-sensor), but CIS2(R) may be the primary and CIS1(L) may be the secondary.

[0169] The AP 111 transmits zone information about the zone Z1, including the gaze position, to the CIS1(L) for the left eye via, for example, an I2C (Internal Integrated Circuit) or an I3C (Improved Inter Integrated Circuit). The zone information includes the position and size (x, y, w, h) of the zone Z1 and offset information (xoffset, yoffset) of the zone Z1 in the CIS2(R). x and y are the two-dimensional coordinates of the image plane of the CIS1(L), w is the width of the zone Z1, and h is the height of the zone Z1.

[0170] The CIS1(L) generates zone setting information including the position and size (x, y, w, h) of the zone Z1. The zone setting information includes offset information (xoffset, yoffset) of the zone Z2 transmitted from the AP 111.

[0171] CIS1(L) transmits the generated zone setting information to CIS2(R) for the right eye via, for example, I2C / I3C or SPI (Serial Peripheral Interface). CIS2(R) generates zone setting information including the position and size (x+xoffset, y+yoffset, w, h) of zone Z2. Note that since there is almost no deviation between the left and right lines of sight in the vertical direction of the image plane, yoffset may be omitted.

[0172] CIS1(L) and CIS2(R) each transmit image data to AP111 independently via, for example, MIPI.

[0173] There is a risk of a difference in the timing at which the zone setting information is reflected in CIS1(L) and CIS2(R). Therefore, reflection timing information may be added to the zone setting information transmitted from CIS1(L) to CIS2(R) to synchronize the setting reflection timing between CIS1(L) and CIS2(R). The reflection timing information is information indicating that the zone setting information will be reflected N frames later, assuming that CIS1(L) and CIS2(R) are synchronized. This prevents a problem of the settings being reflected in different frames between the left-eye image and the right-eye image due to a difference in the setting timing of the zone setting information between CIS1(L) and CIS2(R).

[0174] The image data generated by CIS1(L) and the image data generated by CIS2(R) may be concatenated and transmitted to AP 111 as a single image. FIG. 28 is a block diagram showing a case where two images generated by CIS1(L) and CIS2(R) are concatenated and transmitted to AP 111 as a single image. In the example of FIG. 28 , the image data output from CIS2(R) is transmitted to CIS1(L) by, for example, MIPI. CIS1(L) concatenates the two image data and transmits the resulting image data to AP 111 via, for example, MIPI.

[0175] 29 is a diagram showing the data structure of image data transmitted by CIS1(L) to AP 111. The image data includes a frame start (FS), a frame header (FH), image data generated by CIS1(L) and image data generated by CIS(R), a frame footer (FF), and a frame end (FE).

[0176] In Figure 29, the image data generated by CIS1 (L) is placed on the higher bit side of the bit string, and the image data generated by CIS2 (R) is placed on the lower bit side of the bit string, but they may also be placed as separate bit strings.

[0177] In the case of Figure 29, AP 111 can acquire two pieces of image data almost simultaneously, so there is no need to synchronize the reception timing of the two pieces of image data or adjust the difference in reception timing, as in the case of Figure 28, and the processing of AP 111 can be simplified.

[0178] (Additional Information on Zone Setting ID) A frame number may be assigned to the frame currently being captured. The frame number is managed using a string of multiple bits. The photodetector 1 according to the present disclosure may use the lowest N bits of the bit string representing the frame number as a zone setting ID and transmit it to the signal processing device 111.

[0179] Fig. 30 is a diagram showing exposure and readout timing of the photodetector 1. The photodetector 1 transmits the above-mentioned zone setting ID to the signal processing device 111 (e.g., AP 111) using an existing communication interface. Fig. 30 shows an example in which the lowest two bits of the frame number are used as the zone setting ID using, for example, a GPIO (General Purpose Input / Output) which is an existing communication interface. "LL", "LH", "HL", and "HH" in Fig. 30 indicate examples in which the lowest two bits are "00", "01", "10", and "11", respectively.

[0180] 30, image data exposed within a frame period from time t1 to t2 has the value "0" of the two lowest bits of the frame number added to the image data via the MIPI interface during the period from time t3 to t4, and is transmitted to the AP 111. In this case, the two lowest bits "00" have been transmitted to the AP 111 in advance via GPIO, and therefore the AP 111 understands that the data is exposed within the frame period from time t1 to t2, since the value of the two lowest bits added to the image data is "0".

[0181] The GPIO is a communication interface that is prepared in advance to enable the photodetector 1 and the AP 111 to send and receive various information. In this way, the photodetector 1 can send, for example, the two least significant bits of an existing frame number to the AP 111 using an existing communication interface, so that the zone setting ID can be set and notified without adding new hardware. Note that, although the two least significant bits are shown here as an example, the number of bits is not important.

[0182] (Connecting Multiple CISs to a Single Lane) A plurality of photodetector devices 1 according to the present disclosure can be connected to a single lane to communicate with the signal processing device 111 .

[0183] 31 is a block diagram showing a schematic configuration of an information processing system 1001 in which a plurality of photodetectors 1 are connected to a single lane 122 and communicate with a signal processing device 111. In FIG. 31, an example in which the signal processing device 111 is an AP 111 is shown.

[0184] In the information processing system 1001 of Fig. 31 , each photodetector 1 connected to a single lane 122 can transmit and receive various information to and from the AP 111, separately from the other photodetector devices 1. In the single lane 122, packets containing image data from multiple photodetector devices 1 are combined, for example, in a time-division manner and transmitted. In addition, in the single lane 122, packets containing various information to be sent from the AP 111 to multiple photodetector devices 1 are transmitted. The header of each packet includes information identifying the destination photodetector 1 or AP 111.

[0185] The present technology can be configured as follows:

[0186] (1) A photodetection device comprising: a pixel unit that captures images successively at different timings; a region control unit that identifies a region of interest according to the captured images, sets a first resolution to a first image corresponding to the region of interest, and sets a second resolution that is lower than the first resolution to a second image corresponding to a region other than the region of interest; and an output circuit that outputs the first image and the second image at different frame rates in accordance with the control of the region control unit.

[0187] (2) The photodetector according to (1), wherein the region control unit outputs the first image at a first frame rate and the second image at a second frame rate, and the first frame rate is lower than the second frame rate.

[0188] (3) The photodetector according to (2), wherein the region control unit outputs the first image at a first frame rate and the second image at a second frame rate, and the first frame rate is higher than the second frame rate.

[0189] (4) The photodetector according to (2), wherein the region control unit outputs the second image including the first image at the first frame rate, and outputs the second image including the first image and the second image not including the first image, in sequence, at the second frame rate.

[0190] (5) The photodetector according to (3), wherein the region control unit sequentially outputs the second image including the first image and an image of only the first image at the first frame rate, and outputs the second image including the first image at the second frame rate.

[0191] (6) The photodetector according to any one of (1) to (5), wherein the pixel unit is configured with a plurality of pixels arranged in a matrix, each pixel including a photodiode that performs photoelectric conversion, and the region control unit adds and reads out analog signals when reading out analog signals from pixels corresponding to the second image.

[0192] (7) The photodetector according to (6), further comprising: a readout circuit that selects, for each row, pixels arranged in the row direction of the pixel unit and outputs an analog signal to a signal line for each column connected to each pixel arranged in the column direction of the pixel unit; and an analog-to-digital conversion unit that converts the analog signal of the signal line for each column into a digital signal.

[0193] (8) The photodetection device according to (7), wherein the region control unit, when reading out analog signals from pixels corresponding to the second image, causes the readout circuit to simultaneously select multiple rows and output analog signals from pixels arranged in the multiple rows.

[0194] (9) The photodetection device according to (8), wherein when the region control unit reads out analog signals from pixels corresponding to the first image, the region control unit causes the readout circuit to select rows that are less than the plurality of rows, including a single row, and output analog signals from first pixels arranged in the selected rows.

[0195] (10) The photodetection device according to (9), wherein the region control unit causes the readout circuit to read out an analog signal simultaneously with the first pixel when a second pixel corresponding to the second image is arranged in the selected row.

[0196] (11) The photodetection device according to (10), further comprising a signal processing circuit capable of adding the digital signals, wherein the signal processing circuit adds the digital signals based on analog signals read out from the second pixels when the second pixels are arranged in the selected row so as to achieve the second resolution.

[0197] (12) The light detection device according to any one of (1) to (10), wherein the output circuit outputs the first image and the second image in a predetermined image data structure.

[0198] (13) The photodetector according to (12), wherein, when the first image is included in the area of ​​the second image, the output circuit outputs the first image data in a first image data structure in which second image data corresponding to the second image and first image data corresponding to the first image are included in the same row.

[0199] (14) The photodetector device described in (12), wherein, when the first image is included within the area of ​​the second image, the output circuit adds first image data corresponding to the first image and outputs the image data converted to the second resolution and the second image data corresponding to the second image in a second image data structure included in the same row.

[0200] (15) The photodetector according to (13), wherein the output circuit outputs a third image data structure including only the first image and a second image data structure when the first image is included in an area of ​​the second image.

[0201] (16) The photodetector according to (15), wherein the output circuit includes information on the timing at which reading of each row is started in the first to third image data structures.

[0202] (17) The optical detection device described in (1), wherein the second image has a first peripheral area image corresponding to a first peripheral area surrounding the attention area and a second peripheral area image corresponding to a second peripheral area surrounding the first peripheral area, and the area control unit reduces the resolution of the second peripheral area image to be lower than the resolution of the first peripheral area.

[0203] (18) An information processing system including: a gaze point detection unit that detects a person's gaze point; and a light detection device, wherein the light detection device has: a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of interest based on the gaze point, sets a resolution of a first image corresponding to the region of interest to a first resolution, and sets a resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; and an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit.

[0204] (19) The information processing system according to (18), further comprising a display unit that outputs the first image at a first frame rate and the second image at a second frame rate, wherein the display unit sets the first frame rate to be lower than the second frame rate.

[0205] (20) The information processing system according to (18), further comprising a display unit that outputs the first image at a first frame rate and the second image at a second frame rate, wherein the display unit sets the first frame rate to be higher than the second frame rate.

[0206] (21) An information processing system comprising: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of interest based on the gaze point, sets a resolution of a first image corresponding to the region of interest to a first resolution, and sets a resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit; and a display unit that displays the region of interest based on the first image and the region other than the region of interest based on the second image.

[0207] (22) An information processing system comprising: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of interest based on the gaze point, sets a resolution of a first image corresponding to the region of interest to a first resolution, and sets a resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit; and a signal processing unit that processes the first image at a first frequency and processes the second image at a second frequency.

[0208] (23) An information processing system having: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures images at different timings; a region control unit that identifies a region of interest based on the gaze point, sets a resolution of a first image corresponding to the region of interest to a first resolution, and sets a resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit; and a display unit that generates and displays an image to be presented to a user based on the image data output at the different frame rates.

[0209] (24) The information processing system according to (18), further comprising a display unit that displays the first image and the second image at different frame rates.

[0210] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0211] 11: Information processing system, 60: Photodetector (imaging unit), 61: Display unit, 83: Analog-to-digital converter group, 88: Signal processing circuit, 89: Output circuit, 90: Area control unit.

Claims

1. A photodetection device comprising: a pixel unit that successively captures imaging images at different timings; a region control unit that identifies a region of interest according to the imaging image, sets a resolution of a first image corresponding to the region of interest as a first resolution, and sets a resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; and an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit.

2. The photodetection device according to claim 1, wherein the region control unit outputs the first image at a first frame rate and outputs the second image at a second frame rate, and the first frame rate is smaller than the second frame rate.

3. The photodetection device according to claim 1, wherein the region control unit outputs the first image at a first frame rate and outputs the second image at a second frame rate, and the first frame rate is larger than the second frame rate.

4. The photodetection device according to claim 2, wherein the region control unit outputs the second image including the first image at the first frame rate, and outputs, at the second frame rate, the second image including the first image and the second image not including the first image in sequence.

5. The photodetection device according to claim 3, wherein the region control unit outputs, at the first frame rate, the second image including the first image and the image of only the first image in sequence, and outputs the second image including the first image at the second frame rate.

6. The pixel unit is configured by a plurality of pixels in which pixels including photodiodes that perform photoelectric conversion are arranged in a matrix, and the region control unit adds and reads out an analog signal when reading out the analog signal from the pixels corresponding to the second image. The photodetection device according to claim 1.

7. The photodetection device according to claim 6, further comprising: a readout circuit that selects each pixel arranged in the row direction of the pixel unit for each row and outputs an analog signal to a signal line for each column connected to each pixel arranged in the column direction of the pixel unit; and an analog-to-digital conversion unit that converts the analog signal of the signal line for each column into a digital signal.

8. The light detection device according to claim 7, wherein when the area control unit reads an analog signal from the pixels corresponding to the second image in the readout circuit, the area control unit selects a plurality of rows simultaneously and outputs an analog signal from the pixels arranged in the plurality of rows.

9. The light detection device according to claim 8, wherein when the area control unit reads an analog signal from the pixels corresponding to the first image in the readout circuit, the area control unit selects a selection row that is less than the plurality of rows and includes a single row, and outputs an analog signal from the first pixels arranged in the selection row.

10. The light detection device according to claim 9, wherein when the second pixels corresponding to the second image are arranged in the selection row, the area control unit causes the readout circuit to read an analog signal simultaneously with the first pixels.

11. The light detection device according to claim 10, further comprising a signal processing circuit capable of adding the digital signals, wherein when the second pixels are arranged in the selection row, the signal processing circuit adds the digital signals based on the analog signals read from the second pixels so as to have the second resolution.

12. The light detection device according to claim 2, wherein the output circuit outputs the first image and the second image in a predetermined image data structure.

13. The light detection device according to claim 12, wherein when the first image is included in the area of the second image, the output circuit outputs the first image data structure in which the second image data corresponding to the second image and the first image data corresponding to the first image are included in the same row.

14. The light detection device according to claim 12, wherein when the first image is included in the area of the second image, the output circuit adds the first image data corresponding to the first image, and outputs the image data having the second resolution and the second image data corresponding to the second image in the second image data structure included in the same row.

15. The light detection device according to claim 14, wherein when the first image is included in the area of the second image, the output circuit outputs a third image data structure including only the first image and a second image data structure.

16. The light detection device according to claim 15, wherein the output circuit includes information regarding the timing at which the reading of each row is started in the first to third image data structures.

17. The second image has a first peripheral region image corresponding to a first peripheral region around the region of interest and a second peripheral region image corresponding to a second peripheral region around the first peripheral region, and the region control unit lowers the resolution of the second peripheral region image below the resolution of the first peripheral region. The light detection device according to claim 1.

18. An information processing system comprising a fixation point detection unit that detects a person's fixation point and a light detection device, wherein the light detection device includes a pixel unit that continuously captures imaging images at different timings, a region control unit that identifies a region of interest based on the fixation point, sets the resolution of a first image corresponding to the region of interest as a first resolution, and sets the resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution, and an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit.

19. The information processing system according to claim 18, further comprising a display unit that outputs the first image at a first frame rate and outputs the second image at a second frame rate, and the display unit makes the first frame rate smaller than the second frame rate.

20. The information processing system according to claim 18, further comprising a display unit that outputs the first image at a first frame rate and outputs the second image at a second frame rate, and the display unit makes the first frame rate larger than the second frame rate.

21. An information processing system comprising a fixation point detection unit that detects a person's fixation point, a pixel unit that continuously captures imaging images at different timings, a region control unit that identifies a region of interest based on the fixation point, sets the resolution of a first image corresponding to the region of interest as a first resolution, and sets the resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution, an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit, and a display unit in which the region of interest is displayed based on the first image and the region other than the region of interest is displayed based on the second image.

22. An information processing system comprising: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures imaging images at different timings; a region control unit that specifies a region of interest based on the gaze point, sets the resolution of a first image corresponding to the region of interest as a first resolution, and sets the resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit; and a signal processing unit that processes the first image at a first frequency and processes the second image at a second frequency.

23. An information processing system comprising: a gaze point detection unit that detects a person's gaze point; a pixel unit that continuously captures imaging images at different timings; a region control unit that specifies a region of interest based on the gaze point, sets the resolution of a first image corresponding to the region of interest as a first resolution, and sets the resolution of a second image corresponding to a region other than the region of interest to a second resolution lower than the first resolution; an output circuit that outputs the first image and the second image at different frame rates according to the control of the region control unit; and a display unit that generates and displays an image to be presented to a user based on the image data output at the different frame rates.

Images