Light detection device and information processing system

Abstract

[Problem] To provide a light detection device capable of reducing power consumption without making a frame rate extremely slow even when a line-of-sight position is changed. [Solution] This light detection device is provided with: a pixel unit having a plurality of pixels each including a photoelectric conversion element; and a control unit for alternatively selecting a first mode and a second mode between a first region comprising two or more pixels including a gaze position among the plurality of pixels and a second region surrounding at least a portion of the first region, the first mode making a scanning speed for reading out a pixel signal subjected to photoelectric conversion different for each region, the second mode making the scanning speed the same.

Description

Photodetector and information processing system

[0001] The present disclosure relates to a light detection device and an information processing system.

[0002] BACKGROUND ART In wearable displays such as head-mounted displays, a technology is known that captures an image of real space in a field of view that corresponds to the user's line of sight, and processes the captured image according to the real space to realize augmented reality or virtual reality.

[0003] For example, a technique is known in which the resolution of a region of interest including the gaze position of a user wearing a head-mounted display is made higher than that of other regions (see Patent Document 1).

[0004] International Publication No. 2019 / 171522

[0005] However, when changing the resolution of the area of ​​interest and other areas as in Patent Document 1, the area of ​​interest must be changed every time the gaze position changes, which takes time to change the area of ​​interest and resolution, and there is a risk that the captured image cannot be displayed in real time.

[0006] Furthermore, in Patent Document 1, although the resolution is changed between the region of interest and other regions, consideration is not given sufficiently to power consumption.

[0007] Therefore, the present disclosure provides a light detection device and an information processing system that can prevent the frame rate from slowing down excessively even when the gaze position changes, and that can reduce power consumption.

[0008] In order to solve the above problems, according to the present disclosure, there is provided a photodetection device comprising: a pixel section having a plurality of pixels each including a photoelectric conversion element; and a control section that alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for each region, the first region being made up of two or more pixels among the plurality of pixels that include a gaze position, and a second region surrounding at least a part of the first region, and a second mode in which the scanning speed is made the same for each region.

[0009] The control unit may update at least one of the position and the size of the first area based on the input update information of the gaze position.

[0010] The control unit may alternatively select either the first mode or the second mode on a frame-by-frame basis, and may disable circuits that do not require operation by making the vertical blanking period longer when the first mode is selected than when the second mode is selected.

[0011] When the first mode is selected, the control unit may make the vertical blanking period longer than when the second mode is selected, in which the scanning speed for scanning pixel rows including the first region and the scanning speed for scanning pixel rows including the second region are made equal by making the scanning speed for scanning pixel rows including the first region and the scanning speed for scanning pixel rows including the second region faster than the scanning speed for scanning pixel rows including the first region.

[0012] When the first mode is selected, the control unit may allow at least one of the position or size of the first area to be changed until the pixel shutter operation timing at the position of the first area, and when the second mode is selected, may allow at least one of the position or size of the first area to be changed until the timing predicted from the position of the first area in the immediately preceding frame.

[0013] When the first mode or the second mode is selected, the control unit may start reading out pixel signals from the first region or the second region from the beginning of a frame.

[0014] If the control unit is reading out the pixel signals when instructed to switch from the first mode to the second mode, the control unit may stop reading out and start exposure operation in the second mode, and may make the interval between frames at which exposure operation in the second mode is started shorter than the interval between frames thereafter.

[0015] If the control unit is reading out the pixel signals when instructed to switch from the first mode to the second mode, the control unit may start exposure operation in the second mode after completing reading out one frame, and may keep the frame intervals the same regardless of whether or not switching from the first mode to the second mode is performed.

[0016] If the control unit is reading out the pixel signals when instructed to switch from the second mode to the first mode, the control unit may stop reading out and start exposure operation in the first mode, and may make the interval between frames at which exposure operation in the first mode is started shorter than the interval between frames thereafter.

[0017] If the control unit is reading out the pixel signals when instructed to switch from the second mode to the first mode, the control unit may start the exposure operation in the first mode after the reading out of one frame has been completed, with a delay so as not to interfere with the reading out of one frame, and may keep the frame intervals the same regardless of whether or not the control unit switches from the second mode to the first mode.

[0018] The control unit may set an exposure period in the first mode for a frame in which the second mode is switched to the first mode to be shorter than that for subsequent frames. When the exposure period is shortened, a signal amount for subsequent frames decreases, and therefore, a gain may be applied to compensate for this decrease.

[0019] If the control unit is reading out pixel signals when instructed to switch from the second mode to the first mode, the control unit may wait one frame after the reading out is completed, and then start reading out pixel signals in the second mode.

[0020] The control unit may perform a rolling shutter operation.

[0021] The control unit may perform a global shutter operation.

[0022] When the control unit performs the global shutter operation and selects the first mode, the control unit may allow at least one of the position and size of the first area to be changed until a timing at which a pixel signal at a position in the first area is read out.

[0023] The gaze position update information may be input in synchronization with frame switching timing.

[0024] When the gaze position update information is input multiple times within one frame period, the control unit may update at least one of the position and size of the first region based on the most recently input update information, and read out the pixel signals of the first region. The control unit may predictively update the start position of the first region based on the signals input multiple times.

[0025] The image processing device may further include a signal processing circuit that generates image data on a frame-by-frame basis based on the pixel signals of each of the plurality of pixels, and the signal processing circuit may include, in the image data, information that identifies the frame and information that specifies the position and size of the first region.

[0026] Regardless of whether the first mode or the second mode is selected, the control unit may perform a binning process on the second region, thereby making the resolution of the second region lower than the resolution of the first region.

[0027] The control unit may further include a third region surrounding at least a portion of the second region, and when the first mode is selected, the control unit may make the scanning speeds of the first region, the second region, and the third region different from one another, and when the second mode is selected, the control unit may make the scanning speeds of the first region, the second region, and the third region the same.

[0028] According to the present disclosure, there is provided an information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a control unit that alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for each region, the first region consisting of two or more pixels including the gaze position among the plurality of pixels, and a second region surrounding at least a part of the first region, and a second mode in which the scanning speed is made the same; an input unit that accepts user requests; and a processing circuit that transmits a signal to the control unit to make the selection based on the user request.

[0029] According to the present disclosure, there is provided an information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a control unit that alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for each region, in a first region consisting of two or more pixels including the gaze position among the plurality of pixels, and a second region surrounding at least a part of the first region, and a second mode in which the scanning speed is made the same; and a processing circuit that, in the first mode, processes the pixel signals read out at different scanning speeds in the first region and the second region to generate an image for display.

[0030] According to the present disclosure, there is provided an information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a control unit that alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for each region, and a second mode in which the scanning speed is made the same for a first region consisting of two or more pixels that include the gaze position among the plurality of pixels, and a second region that surrounds at least a part of the first region; and a processing circuit that transmits a signal to make the selection to the control unit based on gaze detection information detected by the gaze point detection unit.

[0031] 1 is a diagram illustrating an example of the appearance of an information processing system according to the present disclosure. A diagram illustrating a modified example of the appearance of the information processing system according to the present disclosure. A block diagram illustrating a schematic configuration of an information processing system according to the present disclosure. A diagram illustrating an example of the configuration of a gaze point detection unit of an HMD. A diagram illustrating an example of the configuration of the hardware of a gaze point detection unit and a spatial imaging unit. A block diagram illustrating a detailed configuration of an imaging unit. A diagram illustrating an example of a plurality of zones set by an area control unit. A block diagram illustrating functions of an HMD. A diagram illustrating a first drive method. A diagram illustrating a second drive method. A timing diagram illustrating a first example of a switching procedure from the first drive method to the second drive method. A timing diagram illustrating a second example of a switching procedure from the first drive method to the second drive method. A timing diagram illustrating a first example of a switching procedure from the second drive method to the first drive method. A timing diagram illustrating a second example of a switching procedure from the second drive method to the first drive method. A timing diagram illustrating a third example of a switching procedure from the second drive method to the first drive method. A diagram illustrating shutter operations and readout operations of a photodetector according to a third embodiment. A timing diagram illustrating shutter operations and readout operations of pixel signals in a photodetector according to a fourth embodiment. 21A, 21B, and 21C are diagrams showing examples of zone constraints that are OK and NG. A sequence diagram showing the procedure for transmitting and receiving image data between the photodetector and a signal processing device when the photodetector assigns a zone setting ID. A diagram showing an example of setting information in a zone setting register provided inside the signal processing device of FIG. 22. A diagram showing an example of the data structure of image data transmitted from the photodetector to the signal processing device. A sequence diagram showing the procedure for transmitting and receiving image data between the photodetector and a signal processing device when the signal processing device assigns a zone setting ID. A diagram showing an example of setting information in a zone setting register provided inside the signal processing device of FIG. 25. A diagram showing the data structure of image data of each zone transmitted from the photodetector to the signal processing device. A diagram showing zone setting information. A diagram showing the time lag between when the gaze detection device detects the gaze position and when the photodetector reads out pixel signals from the zone corresponding to the gaze position. 10A and 10B are diagrams for explaining coordinate designation of gaze positions and each zone in two VST sensors for the left and right eyes;A diagram showing data transmitted and received between two VST sensors that constitute a gaze detection device. A block diagram when two images generated by the VST sensors are joined together and transmitted as a single image to an AP. A diagram showing the data structure of image data transmitted by CIS1(L) to an AP. A diagram showing exposure and readout timing of a photodetector. A block diagram showing a schematic configuration of a photodetection system in which multiple photodetectors are connected to a single lane and communicate with a signal processing device. A block diagram showing an example of a schematic configuration of an in-vivo information acquisition system.

[0032] Hereinafter, embodiments of a light detection device and an information processing system according to the present disclosure will be described with reference to the drawings. The following description will focus on the main components of the light detection device and the information processing system according to the present disclosure, but the light detection device may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0033] Fig. 1A is a diagram illustrating an example of the appearance of an information processing system 1001 of the present disclosure, Fig. 1B is a diagram illustrating a modified example of the appearance of the information processing system 1001 of the present disclosure, and Fig. 2 is a block diagram illustrating a schematic configuration of the information processing system 1001 of the present disclosure. As shown in Fig. 1A, the information processing system 1001 according to this embodiment is configured as a head-mounted display (HMD). An example of the appearance of the head-mounted display (HMD) of this embodiment will be described with reference to Fig. 1A.

[0034] In this example, the HMD 1001 is composed of an output mechanism unit 1011 and a wearing mechanism unit 1012. The wearing mechanism unit 1012 includes a wearing band 1013 that wraps around the head when worn by the user, thereby securing the device in place. Note that the band does not have to wrap around the head as long as it is secured to the head.

[0035] The output mechanism unit 1011 includes a housing 1014 shaped to cover the left and right eyes when the user wears the HMD 1001, and includes a display panel inside that faces the eyes when worn. The housing 1014 may further include a lens that is positioned between the display panel (display unit 2005 ( FIG. 2 )) and the user's eyes when the HMD 1001 is worn, and that expands the user's field of view. The display panel may be divided into left and right halves, and each region may display a stereo image corresponding to the parallax between the eyes, and stereoscopic vision may be realized by such a display.

[0036] The HMD 1001 may further include speakers or earphones at positions corresponding to the user's ears when worn. In this example, the HMD 1001 includes a camera 1015 on the front surface of the housing 1014, which captures video of the surrounding real space in a field of view corresponding to the user's line of sight. In this specification, the camera 1015 may be referred to as a light detection unit 1015.

[0037] The camera 1015 is, for example, a light detection unit, and includes a light detection device such as an image sensor or a distance measurement sensor, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and an optical system such as an imaging lens. For example, in FIG. 1A , the camera 1015 is configured as a stereo camera that captures images of the space in front of the user from left and right viewpoints corresponding to the user's left and right eyes. Note that the camera 1015 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may be used. For hand tracking applications, the camera 1015 may be configured to capture images of the space below the information processing system. For eye tracking or face tracking applications, the camera 1015 may be configured to capture images of the user's eyes or face.

[0038] The HMD 1001 also includes a sensor 2008 (FIG. 2). The sensor group may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0039] The HMD 1001 may be connected to other processing devices via wireless communication, or may be connected via a wired connection such as a USB (Universal Serial Bus).

[0040] In this case, the HMD 1001 may be configured to run an online application such as a game in which multiple users can participate via a network. In this case, the HMD 1001 performs predetermined processing on the image captured by the camera 1015, and generates and displays a display image within the field of view of the camera 1015.

[0041] The content of the displayed image is not particularly limited, and may vary depending on the functions the user desires from the system and the content of the application that has been started.

[0042] For example, the HMD 1001 may perform some processing on the image captured by the camera 1015, or may superimpose a virtual object that interacts with the image of a real object. Alternatively, the HMD 1001 may render a virtual world in a field of view corresponding to the user's field of view based on the captured image or measurements by a motion sensor included in the sensor group of the HMD 1001.

[0043] Representative examples of these modes include virtual reality (VR), augmented reality (AR), and mixed reality (MR). In addition, a see-through mode (VST: Video See-Through) in which the real world can be seen through the screen of the HMD 1001 may be realized by directly displaying the image captured by the light detection unit 1015 as the display image.

[0044] An information processing system 1101 according to this embodiment shown in FIG. 1B is configured as a glasses-type HMD.

[0045] The HMD main body 1111 is worn on the user's head when in use. The HMD main body 1111 has a front part 1112, a right temple part 1113 provided on the right side of the front part 1112, a left temple part 1114 provided on the left side of the front part 1112, and a glass part 1115 attached to the underside of the front part 1112. Note that although the glass is integrated in FIG. 1B , it may have two separate glasses, one for each eye, or may be configured to cover only one eye.

[0046] The display unit 1103 is a see-through type display unit and is provided on the surface of the glass unit 1115. The display unit 1103 performs AR display of a virtual object in accordance with the control of the processing circuit (control unit) 2001. Note that the display unit 1103 may be a non-see-through type display unit. In this case, AR display is performed by displaying on the display unit 2005 an image in which the virtual object is superimposed on an image currently captured by the light detection unit 1104.

[0047] The camera 1104 is, for example, a light detection unit, and includes a light detection device such as an image sensor, such as a CCD sensor or a CMOS sensor, or a distance measurement sensor, and an optical system, such as an imaging lens. The camera 1104 is provided facing outward on the outer surface of the front unit 1112, captures an image of an object in real space, and outputs the image information obtained by the capture to the processing circuit (control unit) 2001. In FIG. 1B , for example, two cameras 1104 are provided laterally at a predetermined interval on the front unit 1112. Note that the camera (light detection unit) 1015 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may be used. In hand tracking applications, the camera 1104 may be provided to capture an image of the space below the information processing system. In eye tracking or face tracking applications, the camera 1104 may be provided to capture an image of the user's eyes or face.

[0048] The glasses-type HMD 1101 also includes a sensor 2008 ( FIG. 2 ). The sensor unit may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0049] The glasses-type HMD 1101 may also include a communication IF 2009 (FIG. 2), which communicates with a smartphone or an external device other than a smartphone (for example, a personal computer (PC) or a server device on a network) via a wired or wireless connection.

[0050] <Example of Hardware Configuration of HMD> Next, an example of the hardware configuration of the information processing system (HMD 1001 or glasses-type HMD 1101) will be described with reference to Fig. 2. As shown in Fig. 2, the hardware of the information processing system is composed of a CPU 2001, a memory 2002, a camera 2003, a display unit 2005, an input unit 2006, an output unit 2007, a sensor 2008, a communication interface (IF) 2009, an external network 2010, and a secondary storage device 2011, which are mutually connected via a bus 2012 and can transmit and receive data and programs.

[0051] The processing circuit 2001 operates based on programs stored in the memory 2002 or the secondary storage device 2011 and controls the overall operation of the information processing systems 1001 and 1101. The processing circuit is, for example, a processor, which reads and executes each program from the memory 2002 to realize the corresponding function of each program. The processor may include, for example, any one or more of a multi-core processor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or an equivalent discrete or integrated logic circuit. The processing circuit may be realized as multiple chips.

[0052] Memory 2020 may include any type of memory for storing data and executable software instructions, implemented, for example, by random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or semiconductor memory devices such as flash memory, hard disk, optical disk, etc.

[0053] 1A and the camera 1104 in FIG. 1B, and includes a light detection device such as an image sensor or a distance measurement sensor, such as a CCD sensor or a CMOS sensor, and an optical system such as an imaging lens. The camera 2003 has a light detection device 2004.

[0054] The display unit 2005 is a display panel provided inside the housing, and is made up of a display device such as an LCD (Liquid Crystal Display) or an organic EL (ElectroLuminescence).

[0055] Although not shown in Figures 1A and 1B, the input unit 2006 is composed of input devices such as a keyboard, mouse, touch panel, microphone, and controller through which the user inputs operation commands, and supplies various input signals to the control unit 2001.

[0056] The output unit 2007 is composed of an audio output device such as a speaker, a force feedback device, an odor feedback device, etc., and is controlled by the processing circuit 2001 to output the processing results as sound, force feedback, or odor.

[0057] The sensor 2008 may include at least one of various sensors, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor, for detecting the movement, posture, and position of the HMDs 1001 and 1101. The sensor 2008 may also include a biosensor that senses a person's biometric information and a pressure sensor that detects input.

[0058] The communication interface 2009 is an interface for connecting the information processing systems 1001 and 1101 to an external network 2010. For example, the processing circuit 2001 receives data from other devices and transmits data generated by the processing circuit 1100 to other devices via the communication interface 2009.

[0059] The above describes an example of an information processing system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the light detection device 2004 among the above-described configurations. The light detection device 2004 according to the present disclosure realizes the camera 1015 in FIG. 1 , is composed of a CCD sensor or a CMOS sensor, and outputs captured images to an image analysis unit and a display image generation unit (described later) in the processing circuit 2001.

[0060] Based on the gaze point information supplied from the gaze point detection unit described later, the light detection device 2004 considers the gaze point area, which is the range in the vicinity of the gaze point position in the captured image, to be the area of ​​interest that the user is paying attention to, and changes the resolution between the area of ​​interest and areas other than the area of ​​interest.

[0061] 1, the light detection device 2004 will be described as being configured by a stereo camera that captures images of the space in front from left and right viewpoints corresponding to the left and right eyes of the user of the HMD 1001. Unless otherwise specified, the description will proceed assuming that the camera 1015 and the light detection device 2004 basically have the same configuration.

[0062] The photodetection device 2004 of this embodiment alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for each region, and a second mode in which the scanning speed is made the same for a first region consisting of two or more pixels including the gaze position among multiple pixels, and a second region surrounding at least a part of the first region.

[0063] The processing circuit 2001 can transmit a signal for switching between the first mode and the second mode to, for example, the area control unit 90 based on a user request received from the input unit.

[0064] In addition, in the first mode, the processing circuit 2001 in FIG. 2 can process pixel signals read out at different scanning speeds in the first and second regions to generate an image for display.

[0065] Furthermore, based on the gaze detection information detected by the gaze point detection unit in FIG. 2, the processing circuit 2001 can transmit a signal for switching between the first mode and the second mode to, for example, the area control unit 90.

[0066] As described above, the photodetector 2004 can alternatively select between a first mode in which the scanning speed is varied for each region, and a second mode in which the scanning speed is the same, thereby enabling it to respond to cases in which the user's line of sight moves quickly, and displaying images that do not feel strange to the user.

[0067] <Configuration Example of Gaze Point Detection Unit> The following mainly describes an example using the HMD 1001 of FIG. 1A, but the present disclosure is also applicable to the HMD 1011 of FIG. 1B. FIGS. 3 and 4 are diagrams illustrating the configuration of the HMD 1001 according to the present disclosure. The gaze point detection unit 59 shown in FIGS. 3 and 4 is provided in at least one of the processing circuit 2001 and the sensor 2008 of FIG. 2. FIG. 3 is a diagram illustrating an example of the configuration of the gaze point detection unit 59 of the HMD 1001. FIG. 4 is a block diagram illustrating the connection relationship between the gaze point detection unit 59, an application processor (hereinafter, AP) 59AP, and a light detection device 2004 according to an embodiment. In this specification, the light detection device 2004 according to an embodiment may be referred to as an imaging unit 60.

[0068] The lower part of the figure shows a schematic view of the HMD 1001 and the head F of the user wearing it, as seen from above. The HMD 1001 displays a display image 61P, as shown in the upper part, which the user views with the left and right eyes EL and ER. The display image 61P is composed of images 61PL and 61PR viewed by the left and right eyes EL and ER, respectively. In this example, the gaze point detection unit 59 includes infrared LEDs 59STL and 59STR, eye image sensors 59SRL and 59SRR, which are infrared cameras or PSD (Position Sensitive Detector) sensors, and an image analysis device 59A. In this specification, the gaze point detection unit 59 may be referred to as a gaze detection device.

[0069] The infrared LEDs 59STL and 59STR irradiate the user's left and right eyes EL and ER with infrared rays, respectively. The eye image sensors 59SRL and 59SRR capture images of the user's left and right eyes EL and ER, respectively, and supply the image data to the image analysis device 59A.

[0070] The image analyzer 59A identifies the position of the pupil and the position of the infrared light reflected by the cornea from the captured images of the left and right eyes, and identifies the user's line of sight from this positional relationship. This method, known as the corneal reflex method, has been put to practical use in the field of line of sight detection technology.

[0071] The method for detecting the gaze is not limited to this, and any common method may be used, such as capturing images of the left and right eyes with a visible light camera and identifying the gaze from the relative positions of the inner corners of the eyes and the irises.

[0072] The image analysis device 59A determines the intersection RPL, RPR between the line of sight detected by the eyeball image sensors 59SRL, 59SRR and the display unit 61 (display panel) on which the display image 61P is displayed as the user's point of gaze, detects its position coordinates, and supplies them to the image capture unit 60.

[0073] As shown in Figure 4, the gaze point detection unit 59 includes an eyeball image sensor 59S (59STL, 59STR) that images the eyeball, and an application processor 59AP that essentially functions as an image analysis device 59A, acquiring the gaze point from the captured image of the eyeball and detecting the gaze point.

[0074] Based on the gaze point information supplied from the application processor 59AP, the imaging unit 60 sets the resolution of the area other than the area near the gaze point (gape point area), i.e., the area other than the attention area, to be lower, and captures the image.

[0075] In this configuration, the architecture of the Mobile Industry Processor Interface (MIPI) can be used for data transmission of captured images from the eye image sensor 59S (59STL, 59STR) to the application processor 59AP.

[0076] In addition, the application processor 59AP can use an i2C (Inter-Integrated Circuit) or i3C architecture to transmit information relating to the point of interest to the imaging unit 60 .

[0077] The gaze point detection unit 59 may be configured such that the eye image sensor 59S (59STL, 59STR) and the application processor 59AP are stacked and integrated.

[0078] <Configuration Example of Imaging Unit> Next, a configuration example of the imaging unit 60 will be described. Fig. 5 is a block diagram showing a detailed configuration of the imaging unit 60. The imaging unit 60 includes a pixel unit 81, a vertical scanning circuit 82, an analog-digital converter (ADC) group 83, a horizontal transfer scanning circuit 84, a timing control circuit 85, a horizontal transfer line 86, an amplifier circuit 87, a signal processing circuit 88, an output circuit 89, a region control unit 90, and a reference voltage generator 91.

[0079] The pixel section 81 has a configuration in which pixels, each including a photodiode that performs photoelectric conversion, are arranged in a matrix. Each pixel holds a signal charge corresponding to the amount of incident light. A vertical scanning circuit 82 drives the pixels row by row by supplying a drive pulse to each pixel via pixel drive wiring (not shown). As a result, analog signals from the pixels in each row are supplied to an ADC group 83 via vertical signal lines provided for each column.

[0080] The ADC group 83 is an analog-to-digital conversion circuit configured to have an ADC arranged for each pixel column, the ADC comprising a comparator 101 that compares a reference voltage generated by a reference voltage output unit 100 with an analog signal obtained from a pixel via a vertical signal line for each row line, a counter 102 that counts the comparison time, and a latch 103 that holds the count result.

[0081] The analog signal read out by the vertical signal line is compared with a reference voltage having a slope waveform by a comparator 101, and when the two match, a count value is obtained, thereby converting the signal into a digital signal. The output of each latch 103 is connected to a horizontal transfer line 86.

[0082] The horizontal transfer scanning circuit 84 is composed of, for example, a shift register, and by sequentially outputting horizontal scanning pulses, inputs one row of digital signals stored in the latch 103 to an amplifier circuit 87 and a signal processing circuit 88 via a horizontal transfer line 86.

[0083] The timing control circuit 85 generates and outputs clock signals and control signals that serve as the basis for the operations of the vertical scanning circuit 82, the ADC group 83, and the horizontal transfer scanning circuit 84, based on the vertical synchronization signal, horizontal synchronization signal, and master clock.

[0084] The signal processing circuit 88 serves as a readout processing circuit and performs predetermined processing such as defect correction, demosaic processing, and gamma correction on the input digital signals of each pixel to generate captured image data.

[0085] The output circuit 89 appropriately buffers the captured image data generated by the signal processing circuit 88 and outputs it row by row.

[0086] The region control unit 90 inputs timing information for setting multiple zones in the pixel unit 81 to the timing control circuit 85 based on the gaze point information supplied from the gaze point detection unit 59. FIG. 6 is a diagram showing an example of multiple zones set by the region control unit 90. FIG. 6 shows an example in which the region control unit 90 sets zones Z1 and Z2. There is no limit to the number of zones set by the region control unit 90, but the following mainly describes an example in which the region control unit 90 sets zones Z1 and Z2. Zone Z1 is a pixel region that includes the gaze point position and is a high-resolution region that generates pixel signals in 1×1 pixel units. Zone Z2 is a pixel region that surrounds at least a portion of zone Z1 and is a region that performs binning processing to generate pixel signals in 2×2 pixel units, for example. Zone Z2 is a pixel region with a lower resolution than zone Z1.

[0087] More specifically, the area control unit 90 identifies the pixel position of the coordinates corresponding to the gaze point in the image captured by the imaging unit 60 based on the gaze point information supplied from the gaze point detection unit 59, and identifies a rectangular gaze point area having a predetermined range in the horizontal and vertical directions based on the pixel position corresponding to the identified gaze point as the attention area.

[0088] In the above, an example has been described in which an integral type configuration is adopted as the ADC constituting the ADC group 83, but other configurations may also be used, such as a successive approximation type ADC.

[0089] <Functions Realized by the HMD of FIG. 1> Next, functions realized by the HMD 1001 of FIG. 2 will be described. FIG. 7 is a block diagram illustrating the functions of the HMD 1001. As shown in FIG. 7, the HMD 1001 functionally operates the gaze point detection unit 59, the imaging unit 60, the image analysis unit 71, the display image generation unit 72, and the display unit 61 as shown in FIG. 3 to capture images of real space, analyze the captured images, and process the images based on the analysis results to generate and display a display image. The gaze point detection unit 59, the image analysis unit 71, and the display image generation unit 72 of FIG. 7 are provided in at least one of the processing circuit 2001 or the sensor 2008 of FIG. 2.

[0090] That is, the gaze point detection unit 59 detects the gaze point of the user wearing the HMD 1001 and outputs the detection result to the imaging unit 60 as gaze point information.

[0091] Based on the gaze point information, the imaging unit 60 regards the gaze point area as the area of ​​interest that the user is paying attention to, images the real space with high resolution set for the pixels in the area of ​​interest and low resolution set for the pixels in areas other than the area of ​​interest (gape point area), and outputs the captured image to the image analysis unit 71 and the display image generation unit 72.

[0092] The image analysis unit 71 performs analysis processing, such as face detection, face recognition, gesture recognition, visual tracking, position detection, or acquiring the posture of the HMD 1001 using Visual SLAM (Simultaneous Localization and Mapping), based on the image captured by the imaging unit 60, and outputs the analysis results to the display image generation unit 72.

[0093] Based on the analysis results supplied from the image analysis unit 71, the display image generation unit 72 performs predetermined processing on the image captured by the imaging unit 60 to generate a display image, and outputs the generated image to the display unit 61 for display.

[0094] By functioning as described above, based on the gaze point information detected by the gaze point detection unit 59, the imaging unit 60 regards the gaze point area as the area of ​​interest that the user is paying attention to and sets it to high resolution, and captures an image with areas other than the area of ​​interest set to low resolution. This effectively reduces the amount of data in the image captured by the imaging unit 60, thereby reducing the processing load in the analysis process in the image analysis unit 71 and the display image generation process in the display image generation unit 72. As a result, it is possible to suppress delays related to the series of processes.

[0095] The photodetector 2004 according to the present disclosure is applicable to the image capturing unit 60 in Fig. 5. The photodetector 2004 according to the present disclosure is also applicable to an event-based vision sensor (EVS) that detects events. Representative embodiments of the photodetector 2004 according to the present disclosure will be described below.

[0096] First Embodiment The photodetector 2004 according to the first embodiment includes a control unit that selects either a first driving method or a second driving method to expose and read each pixel in the pixel unit 81. The first driving method is a driving method that emphasizes reducing power consumption. The second driving method is a driving method that emphasizes tracking the user's gaze. In this specification, the first driving method may be referred to as the first mode, and the second driving method may be referred to as the second mode. The control unit is, for example, the region control unit 90 in the analog-to-digital converter 83 in FIG. 5. The region control unit 90 sets multiple regions (e.g., a first region and a second region) in the pixel unit 81 based on the user's gaze position. The first region is composed of two or more pixels that include the gaze position. The second region surrounds at least a portion of the first region. In this specification, the first region may be referred to as zone Z1, and the second region may be referred to as zone Z2.

[0097] The region control unit 90 updates at least one of the position and size of the zone Z1 based on the gaze position update information input. The region control unit 90 alternatively selects either the first drive method or the second drive method on a frame-by-frame basis, and when the first drive method is selected, makes the vertical blanking period VBLK longer than when the second drive method is selected. The first drive method and the second drive method will be described below in order.

[0098] <First Drive Method> In the first drive method, the scanning speed for reading out photoelectrically converted pixel signals is made different between zones Z1 and Z2. FIG. 8 is a diagram illustrating the first drive method. The horizontal axis of FIG. 8 represents time, and the vertical axis represents the row direction of the pixel section 81. FIG. 8 illustrates the shutter operation and readout operation of each pixel row for two frames. Times t3, t11, and t16 represent frame switching timings, and times t3 to t11 and times t11 to t16 each represent a period of one frame. The period of one frame is always the same.

[0099] The first shutter operation in Fig. 8 starts at time t0. The shutter operation is performed sequentially for each pixel row. During the period from time t0 to t1, the shutter operation for zone Z2 is performed. In zone Z2, binning processing is performed in units of, for example, 2 x 2 pixels.

[0100] Subsequently, from time t1, a shutter operation is performed in zone Z1. Exposure is performed during the period from the shutter operation to the readout operation. In zone Z1, exposure and readout are performed for each pixel.

[0101] A shutter operation for zone Z2 is performed between time t2 and time t4. A shutter operation for one frame is performed between time t0 and t4. Between time t3 and t7, pixel signals for one frame for which a shutter operation was performed between time t0 and t4 are sequentially read out. More specifically, between time t3 and t5, pixel signals for zone Z2 for which a shutter operation was performed between time t0 and t1 are read out. Between time t5 and t6, pixel signals for zone Z1 for which a shutter operation was performed between time t1 and t2 are read out. Between time t6 and t7, pixel signals for zone Z2 for which a shutter operation was performed between time t2 and t4 are read out. The period from the time the shutter operation was performed until the pixel signals were read out is the exposure period, and this exposure period is common to all pixels.

[0102] In the first driving method, the scanning speed of each pixel row differs between zone Z1 and zone Z2. In zone Z1, pixel signals from all pixels must be read out, so the scanning speed of each pixel row by the vertical scanning circuit 82 cannot be very fast. In contrast, in zone Z2, the scanning speed can be faster than that of zone Z1 because it is only necessary to read pixel signals from the number of pixels thinned out by binning. By combining zones Z1 and Z2 to read pixel signals, the time required to read all pixel signals in each frame can be reduced, and as a result, the vertical blanking period of each frame can be extended. During the vertical blanking period, the operation of the peripheral circuits of the pixel section 81 can be stopped, so extending the vertical blanking period reduces power consumption.

[0103] In the first embodiment, a zone change can be accepted within the period from time t1 to t9. Hereinafter, times t1 to t9 will be referred to as the zone change acceptance period. A zone change is instructed, for example, by an application processor (hereinafter, AP) when a gaze detection device (not shown, for example, gaze point detection unit 59 in FIG. 2) detects a movement of the gaze. Time t9 is the time when the second shutter operation for zone Z1 starts.

[0104] For the second shutter operation, it is necessary to predict the start time of the shutter operation for zone Z1. Time t9 is predicted from the position of zone Z1 for the first shutter operation. Alternatively, time t9 may be predicted from the movement of the line of sight in the past few frames.

[0105] The second shutter operation of zone Z2 is performed from time t8 to t9, the second shutter operation of zone Z1 is performed from time t9 to t10, and the second shutter operation of zone Z2 is performed from time t10 to t12. From time t11 to t13, pixel signals of zone Z2, which underwent the shutter operation from time t8 to t9, are read out. From time t13 to t14, pixel signals of zone Z1, which underwent the shutter operation from time t9 to t10, are read out. From time t14 to t15, pixel signals of zone Z2, which underwent the shutter operation from time t10 to t12, are read out.

[0106] The zone acceptance period from time t1 to t9 is a zone setting period when the second pixel signal readout is performed from time t11.

[0107] 8 illustrates a zone change acceptance period in a comparative example. In this comparative example, zone setting is performed on a frame-by-frame basis. The zone change acceptance period in this comparative example is set to an earlier period (from just before time t1 to time t3) than the zone change acceptance period in the first embodiment. In this comparative example, if a zone change instruction is not received from the AP within this period, the position and size of the previous zone Z1 remain unchanged, and a second pixel signal readout is performed from time t11.

[0108] As shown in FIG. 8, in the first embodiment, compared to the comparative example, the zone setting can be changed up until immediately before the second readout of pixel signals is started, improving the ability to follow the user's line of sight.

[0109] 8 , the scanning speed of each pixel row in zone Z2 is made faster than the scanning speed of each pixel row in zone Z1, so the blanking period from time t7 when the first readout operation ends to time t11 when the next frame starts is longer. During the vertical blanking period VBLK, the operation of each circuit in the photodetector 2004 can be stopped, and therefore the longer the vertical blanking period VBLK, the more the power consumption of the photodetector 2004 can be reduced.

[0110] In this way, the first driving method can perform the shutter operation of each pixel and the readout operation of the pixel signal with low power consumption.

[0111] <Second Drive Method> In the second drive method, the scanning speed of zones Z1 and Z2 is the same. FIG. 9 is a diagram explaining the second drive method. In the second drive method, the scanning speed of each pixel row in zones Z1 and Z2 is the same. Times t2, t8, and t13 are frame switching timings. The first shutter operation of zones Z1 and Z2 is performed from time t1 to t4. From time t2 to t3, pixel signals from zone Z2, which underwent shutter operation from time t1 to t4, are read out. From time t3 to t5, pixel signals from zone Z1, which underwent shutter operation from time t1 to t4, are read out. From time t5 to t7, pixel signals from zone Z2, which underwent shutter operation from time t1 to t4, are read out.

[0112] A second shutter operation of zones Z1 and Z2 is performed from time t6 to t10. From time t8 to t9, pixel signals of zone Z2, which underwent shutter operation from time t6 to t10, are read out. From time t9 to t11, pixel signals of zone Z1, which underwent shutter operation from time t6 to t10, are read out. From time t11 to t12, pixel signals of zone Z2, which underwent shutter operation from time t6 to t10, are read out.

[0113] Times t3 to t9 are the zone change acceptance period in the second driving method. If a zone change instruction arrives from, for example, the AP during this zone change acceptance period, it can be reflected in the second readout operation. Pixel signals from zone Z1 are read out starting at time t9. The start time t9 of the second pixel readout from zone Z1 is predicted from the position and size of zone Z1 in the previous frame.

[0114] 9 illustrates a zone change acceptance period according to a comparative example. In this comparative example, zones are set on a frame-by-frame basis. Therefore, the zone change acceptance period according to this comparative example is set from time t2 to time t8, for example.

[0115] In the second driving method, the zone change is accepted up until the timing when the second readout of pixel signals from zone Z1 starts, resulting in better line-of-sight tracking. Furthermore, like the first driving method, the second driving method performs binning processing in zone Z2, which reduces the total number of pixel signals read out and reduces power consumption. On the other hand, because the scanning speed of zones Z1 and Z2 is the same, the vertical blanking period VBLK (times t7 to t8) is shorter than in the first driving method, and power saving during the vertical blanking period VBLK is not possible.

[0116] As described above, in the first embodiment, when reading out pixel signals from multiple zones set in the pixel unit 81, the pixel control unit 90 alternatively selects between a first drive method that reads pixel signals with an emphasis on reducing power consumption and a second drive method that reads pixel signals with an emphasis on line-of-sight tracking. This allows pixel signals from multiple zones to be read out with low power consumption. Alternatively, it allows each pixel signal to be read out with good line-of-sight tracking. Furthermore, when the first drive method is selected, the region control unit 90 allows changes to at least one of the position and size of zone Z1 until the timing at which a pixel shutter operation is performed at the position of zone Z1. When the second drive method is selected, the region control unit 90 allows changes to at least one of the position and size of zone Z1 until the timing predicted from the position of zone Z1 in the immediately preceding frame. The longer the allowable period, the better the line-of-sight tracking of the user's line-of-sight.

[0117] Second Embodiment A photodetector 2004 according to a second embodiment is characterized in that it can switch between a first drive method and a second drive method.

[0118] <Switching from the first driving method to the second driving method> Fig. 10 is a timing diagram showing a first example of the procedure for switching from the first driving method to the second driving method. In Fig. 10 and subsequent figures, the first driving method is referred to as DV1 and the second driving method is referred to as DV2.

[0119] Between times t1 and t2, shuttering of zones Z1 and Z2 in the first drive method DV1 is performed sequentially for each pixel row. Time t3 is the frame switching timing, and readout of each pixel signal that underwent shuttering between times t1 and t2 begins. When an instruction to switch from the first drive method DV1 to the second drive method DV2 is issued immediately after time t3, readout of pixel signals is stopped, and shuttering of zones Z1 and Z2 in the second drive method DV2 begins (times t3 to t5). Time t4 is the frame switching timing, and pixel signals in zones Z1 and Z2 that underwent shuttering between times t3 and t5 are read out sequentially from time t4 (times t4 to t6). The period between times t3 and t4 is shorter than the frame period from time t4 onwards.

[0120] 10 , if the region control unit 90 is in the process of reading out pixel signals when it receives an instruction to switch from the first drive method DV1 to the second drive method DV2, it stops the readout and starts an exposure operation using the second drive method DV2, and it also makes the interval between frames at which it starts exposure operation using the second drive method DV2 shorter than the intervals between subsequent frames. The frame period of the frame at which it receives an instruction to switch from the first drive method DV1 to the second drive method DV2 is shorter than the frame periods of the other frames, and the frame intervals become uneven, but it is possible to quickly switch to a new drive method mid-frame.

[0121] 11 is a timing diagram showing a second example of the procedure for switching from the first drive method DV1 to the second drive method DV2. In this second example, when a command to switch from the first drive method DV1 to the second drive method DV2 is issued at time t3, shuttering of each pixel row in zones Z1 and Z2 is performed sequentially from time t5 to t7. Then, from time t6 to t9, the pixel signals for which shuttering was performed at times t5 to t7 are read out sequentially.

[0122] In this way, in the second example shown in Figure 11, if the area control unit 90 is in the process of reading out pixel signals when it receives an instruction to switch from the first drive method DV1 to the second drive method DV2, it starts exposure operation using the second drive method DV2 after reading out one frame is completed, and keeps the frame intervals the same regardless of whether or not it switches from the first drive method DV1 to the second drive method DV2.

[0123] In the first example of Fig. 10, an instruction to switch from the first drive method DV1 to the second drive method DV2 can be accepted at any time, whereas in the second example of Fig. 11, an instruction to switch from the first drive method DV1 to the second drive method DV2 is accepted only up to the frame switching timing. For example, if an instruction to switch from the first drive method DV1 to the second drive method DV2 is issued between times t5 and t8, this is reflected in the shutter operation between times t10 and t13.

[0124] Both the first driving scheme DV1 and the second driving scheme DV2 can accommodate zone changes.

[0125] FIG. 12 is a timing diagram showing a first example of the procedure for switching from the second drive method DV2 to the first drive method DV1. Times t2, t5, t8, t11, and t13 are frame switching timings. From times t1 to t3, shuttering of each pixel row in zones Z1 and Z2 is performed sequentially using the second drive method DV2. From time t2, pixel signals are read out from each pixel row that underwent shuttering from times t1 to t3. When an instruction to switch from the second drive method DV2 to the first drive method DV1 is issued around time t3, pixel signal readout is halted, and instead, shuttering of each pixel row in zones Z1 and Z2 using the first drive method DV1 is initiated (times t3 to t4). At time t5, which is the frame switching timing, pixel signals are read out from each pixel row that underwent shuttering from times t3 to t4 (times t5 to t6).

[0126] Thus, in the first example shown in Figure 12, the frame period of the frame for which switching from the second driving method DV2 to the first driving method DV1 is instructed is shorter than the frame periods of the other frames, resulting in uneven frame intervals.

[0127] 12, an instruction to switch from the first driving method DV1 to the second driving method DV2 can be accepted at any time. The zone change acceptance period is until the start of readout of zone Z1 in the second driving method DV2.

[0128] FIG. 13 is a timing diagram showing a second example of the procedure for switching from the second drive method DV2 to the first drive method DV1. Times t2, t7, t10, t13, and t15 are frame switching timings. From times t1 to t3, shutter operations are performed sequentially for each pixel row in zones Z1 and Z2 using the second drive method DV2. From time t2 onward, pixel signals are read out from each pixel row. When an instruction to switch from the second drive method DV2 to the first drive method DV1 is issued at time t4 while pixel signals are being read out, pixel signal readout is continued until completion (times t2 to t6). Furthermore, from time t5, shutter operations are performed sequentially for each pixel row using the first drive method DV1. By starting the shutter operation at time t5, which is later than time t4 when the drive method switch instruction was issued, interference between pixel signal readout using the first drive method DV1 and shutter operation using the second drive method DV2 is prevented. By delaying the shutter operation, the accumulation time of charges due to photoelectric conversion of each pixel becomes shorter in this frame, but this can be dealt with by gain correction, so no practical problem occurs.

[0129] In the second example shown in FIG. 13, both the instruction to switch from the second drive method DV2 to the first drive method DV1 and the instruction to change zones can be accepted between the timing at which the shutter operation starts in one frame and the timing at which the shutter operation starts in the next frame.

[0130] In the second example of FIG. 13, the frame period can be kept constant even when the driving method is switched, as compared with the first example of FIG.

[0131] 14 is a timing chart showing a third example of the procedure for switching from the second driving method DV2 to the first driving method DV1. Times t2, t5, t8, t11, and t13 are frame switching timings.

[0132] The pixel signals of each pixel row that underwent shutter operation from time t1 to t2 are read out sequentially from time t2 to t4. When an instruction to switch from the second drive method DV2 to the first drive method DV1 is issued around time t2, there is a one-frame wait, and then the shutter operation of each pixel row using the first drive method DV1 is performed sequentially from time t6 to t7. Thereafter, the pixel signals that underwent shutter operation from time t6 to t7 are read out from time t8 to t9.

[0133] In this specification, waiting for one frame when switching the drive method is referred to as inserting a dummy frame. In the third example of Fig. 14, inserting a dummy frame delays the transmission of pixel signals by one frame, but the frame period can always be maintained constant.

[0134] In the third example shown in FIG. 14, both the instruction to switch from the second drive method DV2 to the first drive method DV1 and the instruction to change zones can be accepted between the timing at which the shutter operation starts in one frame and the timing at which the shutter operation starts in the next frame.

[0135] In this way, in the second embodiment, it is possible to switch between the first drive method DV1 and the second drive method DV2 as needed. Multiple options are provided for switching the drive method, which allows the drive method to be switched quickly or with low power consumption in response to a user request.

[0136] Third Embodiment The photodetector 2004 according to the first and second embodiments performs a rolling shutter operation, whereas the photodetector 2004 according to the third embodiment performs a global shutter operation.

[0137] The rolling shutter operation is a process in which the exposure operation is performed sequentially for each pixel row, whereas the global shutter operation is a process in which all pixel rows are exposed simultaneously.

[0138] FIG. 15 is a diagram showing the shutter operation and readout operation of the photodetector 2004 according to the third embodiment. In FIG. 15, times t2 and t8 are frame switching timings. At time t1, charge is transferred from the photodiode to memory for all pixels simultaneously. From times t2 to t3, pixel signals from each pixel row in zone Z2 are read out sequentially. At time t4, a shutter operation for the next frame is performed for all pixels simultaneously. From times t3 to t5, pixel signals from each pixel row in zone Z1 are read out sequentially. From times t5 to t6, pixel signals from each pixel row in zone Z2 are read out sequentially. Thereafter, at time t7, charge is transferred from the photodiode to memory for all pixels simultaneously, and from times t8 to t12, the same processing as that from times t2 to t6 is performed.

[0139] The zone setting acceptance period for frames after time t8 is from time t3 to t9. Because pixel signal readout begins at time t8, which is before time t9, if a zone setting instruction is not input at time t8, the zone setting is predicted based on the zone setting of the immediately preceding frame or the line of sight movement over the immediately preceding few frames, and pixel signal readout begins from time t8.

[0140] In this way, in the third embodiment, even when a global shutter operation is performed, the zone setting can be changed up until immediately before the start of pixel signal readout, and a captured image can be generated with good tracking ability to the line of sight.

[0141] Fourth Embodiment In the fourth embodiment, gaze detection information detected by a gaze detection device (for example, the gaze point detection unit 59 in FIG. 2) is synchronized with frame switching timing.

[0142] FIG. 16 is a timing diagram of the shutter operation and pixel signal readout operation in the photodetector 2004 according to the fourth embodiment. In FIG. 16, times t2, t6, and t9 are frame switching timings. FIG. 16 shows an example in which the scanning speed of each pixel row in zones Z1 and Z2 is the same. The shutter operation of all pixel rows is performed at times t1 to t3. Gaze detection information is input in synchronization with time t2, which is the frame switching timing. The gaze detection information may be input directly from the gaze detection device, or may be input via an AP to which the gaze detection device is connected. The gaze detection information includes information indicating the user's most recent gaze position.

[0143] The area control unit 90 sets the zones based on the gaze detection information, and then reads out pixel signals from times t2 to t5.

[0144] A shutter operation for the next frame is performed from time t4 to t7. Gaze detection information is input in synchronization with time t6, which is the frame switching timing. The region control unit 90 sets zones based on the gaze detection information, and then reads pixel signals from time t6 to t8.

[0145] FIG. 17 is a timing diagram according to a modification of FIG. 16 . In FIG. 17 , times t2, t7, and t11 are frame switching timings. In FIG. 17 , gaze detection information is input multiple times within each frame period. For example, the zone change acceptance period within the frame period from times t7 to t11 is from times t3 to t8. The region control unit 90 sets the zone based on the gaze detection information input immediately before time t8. When starting to read out pixel signals in the frame from times t7 to t11, t8 is predicted based on information from the gaze input device, and pixel signal readout is started.

[0146] In this way, in the fourth embodiment, when gaze detection information is input more frequently than the frame rate of the light detection device 2004, the zone setting can be quickly changed based on the gaze detection information input just before the frame cycle switching timing, thereby further improving gaze tracking ability.

[0147] Fifth Embodiment In the fifth embodiment, a distinctive data structure is used when image data is transmitted from the photodetector 2004 to a downstream AP or the like.

[0148] FIG. 18 is a diagram illustrating the fifth embodiment. The timing diagram on the left side of FIG. 18 is the same as that in FIG. 9. Data read during the frame period from time t8 to t13 accepts changes to the zone setting up until time t9, so the zone setting information is unknown at time t8 when reading starts. Therefore, when image data corresponding to pixel signals from time t9 onwards is output, zone setting information is added.

[0149] In the example of Fig. 18, the embedded data EBD output after one frame of image data has been output contains zone setting information. The zone setting information includes information indicating the frame in which the zone was generated, and information specifying the position and size of each zone.

[0150] In this way, in the fifth embodiment, zone setting information is added to the image data sent from the photodetector 2004 to a downstream AP or the like, so that the AP or the like can identify the frame in which the received image data was generated and the position and size of each zone.

[0151] Sixth Embodiment In the first to fifth embodiments, examples were shown in which the pixel unit 81 had two zones Z1 and Z2, but there is no limit to the number of zones. Fig. 19 shows an example in which the pixel unit 81 has three zones Z1 to Z3. Zone 3 is an area that surrounds at least a part of zone Z2. Zone 3 undergoes a different binning process from zone Z2. For example, if zone Z2 undergoes binning process in 2x2 pixel units, zone 3 undergoes binning process in 4x pixel units.

[0152] Even if the number of zones set in the pixel section 81 is three or more, the first and second driving methods described above can be applied.

[0153] (Schematic Configuration of Light Detection System) The light detection device 2004 according to the present disclosure can be incorporated into an information processing system 1001 having a gaze detection function.

[0154] Fig. 20 is a block diagram showing a schematic configuration of an information processing system 1001 including a light detection device 2004 according to the present disclosure. The information processing system 1001 in Fig. 20 includes the light detection device 2004, a signal processing device 111, and a gaze detection device 112. The signal processing device 111 and the gaze detection device 112 in Fig. 20 are provided in at least one of the processing circuit 2001, the camera 2003, and the sensor 2008 in Fig. 2.

[0155] Specifically, the signal processing device 111 is configured by an ISP (Image Signal Processor) or an AP (Application Processor).

[0156] The photodetector 2004 has a configuration similar to that shown in Fig. 5. Fig. 20 shows a simplified internal configuration of the photodetector 2004 of Fig. 5, and includes at least a pixel unit 81, a signal processing circuit 88, and an interface circuit (IF) 113. The interface circuit (IF) 113 of Fig. 20 corresponds to the output circuit 89 of Fig. 5.

[0157] The signal processing device 111 includes a frame buffer 114 , a first processing unit (IFE) 115 , and a second processing unit (IPE) 116 .

[0158] The frame buffer 114 stores the image data for each frame output from the photodetector 2004. As will be described later, the photodetector 2004 generates image data for each of a plurality of zones. Thus, the frame buffer 114 stores the image data for each of the plurality of zones.

[0159] The first processing unit 115 performs various image processing on each image data to generate a luminance signal and a color difference signal (YUV signal).

[0160] The second processing unit 116 performs various image processing and image synthesis processing based on a plurality of luminance signals and color difference signals corresponding to a plurality of zones, and generates a synthesized luminance signal and color difference signal (YUV signal).

[0161] The gaze detection device 112 detects the gaze of the user and transmits information about the gaze position (hereinafter also referred to as the gaze position) to the signal processing device 111. More specifically, the signal processing device 111 calculates zone information based on the gaze detection information detected by the gaze detection device 112 and transmits the calculated zone information to the light detection device 2004.

[0162] The photodetector 2004 generates image data by changing the resolution for each zone based on the zone information transmitted from the signal processor 111 .

[0163] (Zone Setting Restrictions) The light detection device 2004 according to the present disclosure sets a plurality of zones in the pixel unit 81 based on the gaze position detected by the gaze detection device 112. In doing so, the light detection device 2004 needs to comply with the zone setting restrictions.

[0164] More specifically, in the photodetector 2004 according to the present disclosure, a region including the line of sight detected by the line of sight detection device 112 at its center position is defined as zone Z1, and a region including zone Z1 is defined as zone Z2. Zone Z1 is a region where photoelectrically converted pixel signals are output pixel by pixel. Zone Z2 is a region where binning processing is performed, in which pixel signals from multiple pixels are combined and output as a pixel signal for a single pixel. For example, if zone Z2 is subjected to 2×2 pixel binning processing, zone Z2 has a size that is a multiple of four of zone Z1.

[0165] The photodetector 2004 according to the present disclosure may have three or more zones. For example, if a zone Z3 is provided that encompasses zone Z2, binning processing is performed in zone Z3 using pixel regions larger in size than zone Z2. For example, if zone Z3 is subjected to 4x4 pixel binning processing, zone Z3 has a size that is a multiple of 16 of zone Z1.

[0166] There is a constraint that zone Z2 must contain the entire area of ​​zone Z1, and zone Z3 must contain the entire area of ​​zone Z2. For example, if all sides of zone Z1 are inside all sides of zone Z2 as in Fig. 21A, or if at least some sides of zone Z1 are in contact with any side of zone Z2 as in Fig. 21B, the zone setting is determined to be valid (OK), but if at least some sides of zone Z1 are located outside any side of zone Z2 as in Fig. 21C, the zone setting is determined to be invalid (NG: error).

[0167] The light detection device 2004 according to the present disclosure needs to set multiple zones based on the gaze position so as to satisfy the above-mentioned zone setting constraints. Since the gaze position detected by the gaze detection device 112 may change for each frame, the light detection device 2004 may change the positions and sizes of the multiple zones for each frame based on the zone information from the signal processing device 111.

[0168] (Zone Setting ID) The photodetector 2004 transmits image data for each zone to the signal processing device 111 using, for example, a virtual channel compliant with the MIPI (Mobile Industry Processor Interface) standard. At this time, the signal processing device 111 needs to know which zone of which frame the image data transmitted from the photodetector 2004 includes. Therefore, the photodetector 2004 according to the present disclosure is provided with a zone setting ID. The zone setting ID is information that identifies the frame to which the read image data belongs. The photodetector 2004 according to the present disclosure adds the zone setting ID to the image data and transmits it to the signal processing device 111.

[0169] The zone setting ID may be assigned by the photodetector 2004 or by the signal processing device 111 .

[0170] 22 is a sequence diagram showing the procedure for transmitting and receiving image data between the photodetector 2004 and the signal processing device 111 when the photodetector 2004 assigns a zone setting ID. When the signal processing device 111 receives gaze detection information from the gaze detection device 112 (step S1), it calculates zone information relating to the position and size of a zone Z1 that includes the gaze position at its center (step S2). The zone information is stored, for example, in a zone setting register, which will be described later.

[0171] The signal processing device 111 transmits the calculated zone information to the photodetector 2004 (step S3). The photodetector 2004 assigns a zone setting ID corresponding to the zone information (step S4). The zone setting ID is associated with a frame from which pixel signals are read out.

[0172] The photodetector 2004 transmits the zone setting ID to the signal processing device 111 (step S5). The photodetector 2004 sets zones Z1 to Z3 in the pixel unit 81, performs an exposure operation, and then reads out pixel signals. For example, in zone Z1, pixel signals are read out for each pixel, whereas in zones Z2 and Z3, pixel signals are read out by performing binning processing for each set of pixels (step S6). After the photodetector 2004 sets the zones, a time delay of N frames (N is an integer equal to or greater than 1) may occur before the photodetector 2004 reads out an image signal reflecting the zone setting.

[0173] The photodetector 2004 transmits a packet in which the corresponding zone setting ID is added to the image data for each frame to the signal processor 111 via a virtual channel conforming to the MIPI standard, for example (step S7).

[0174] The signal processing device 111 compares the zone setting ID attached to the received image data with the zone setting ID received in step S5 to identify which frame the received image data belongs to (step S8).

[0175] Fig. 23 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device 111 of Fig. 22. Fig. 23 shows an example of storing setting information for zones Z1 to Z3 in the zone setting register. The zone setting register stores the two-dimensional coordinates of the first pixel and the last pixel in each of zones Z1 to Z3.

[0176] Fig. 24 is a diagram showing an example of the data structure of image data that the photodetector 2004 transmits to the signal processor 111 in step S7 of Fig. 22. The image data in Fig. 24 is data in which the pixel values ​​(16-bit data) of each pixel are arranged in the order of zones Z3, Z2, and Z1, and this data is followed by 16-bit data representing the zone setting ID.

[0177] 25 is a sequence diagram showing the procedure for transmitting and receiving image data between the light detection device 2004 and the signal processing device 111 when the signal processing device 111 assigns a zone setting ID. When the signal processing device 111 receives gaze detection information from the gaze detection device 112 (step S11), it calculates zone information relating to the position and size of a zone Z1 that includes the gaze position at its center (step S12) and assigns a zone setting ID (step S13). This sequence differs from the sequence diagram of FIG. 22 in that the signal processing device 111 assigns a zone setting ID.

[0178] The signal processing device 111 adds the zone setting ID to the calculated zone information and transmits it to the photodetector 2004 (step S14).

[0179] The photodetector 2004 sets, for example, zones Z1 to Z3 in the pixel section 81, performs an exposure operation, and then reads out pixel signals from each zone. For example, in zone Z1, pixel signals are read out for each pixel, whereas in zones Z2 and Z3, pixel signals are read out by performing binning processing for each set of pixels (step S15).

[0180] The photodetector 2004 transmits a packet in which the corresponding zone setting ID is added to the image data for each frame to the signal processor 111 via a virtual channel conforming to the MIPI standard, for example (step S16).

[0181] The signal processing device 111 compares the zone setting ID attached to the received image data with the zone setting ID assigned by itself in step S13 to identify which frame the received image data is from (step S17).

[0182] Fig. 26 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device 111 of Fig. 25. The zone setting register of Fig. 26 contains information on the zone setting ID numbered in step S13. The other information is the same as that of Fig. 24.

[0183] The data structure of the image data that the photodetector 2004 transmits to the signal processing device 111 in step S16 is the same as that shown in FIG. 24, and therefore will not be described here.

[0184] 22 to 26 show an example in which a zone setting ID is added when image data is transmitted from the photodetector 2004 to the signal processing device 111. However, it is preferable to add information about the position and size of each zone (hereinafter, "zone setting information") to the image data in addition to the zone setting ID and transmit the image data from the photodetector 2004 to the signal processing device 111. The reason for this is that the photodetector 2004 can change the position and size of each zone for each frame. Therefore, unless the photodetector 2004 notifies the signal processing device 111 of the position and size of each zone, the signal processing device 111 cannot grasp the position and size of each zone in each frame, resulting in poor utilization of the storage capacity of the frame buffer 114. Therefore, the photodetector 2004 according to the present disclosure adds zone setting information for all zones to the header of image data for each zone transmitted to the signal processing device 111. This allows the signal processing device 111 to grasp the position and size of all zones regardless of which zone's image data it receives, thereby ensuring the storage capacity of the frame buffer 114 for storing the received image data for each zone.

[0185] Fig. 27 is a diagram showing the data structure of image data for each zone transmitted from the photodetector 2004 to the signal processor 111. Fig. 27 shows an example of setting zones Z1 to Z3 in the pixel section 81. Fig. 27 shows the data structures of zones Z3, Z2, and Z1, from left to right. The image data for each of zones Z1 to Z3 includes a frame start signal (FS), virtual channel information (VC), embedded data (EBD), a packet footer (PF), a data type (DT), and valid pixel data.

[0186] The zone setting information is included in, for example, the embedded data (EBD). As described above, the zone setting information for each of the zones Z1 to Z3 includes information about the positions and sizes of all of the zones Z1 to Z3.

[0187] The zone setting information needs to be updated whenever the position and / or size of a zone changes, and since the position and / or size of a zone may change from frame to frame, the zone setting information may also need to be updated from frame to frame.

[0188] However, the photodetector 2004 continues to use the zone setting information of the immediately preceding frame if there is no notification of zone information update from the signal processing device 111. Since the gaze detection device 112 and the photodetector 2004 usually operate asynchronously, the timing at which the gaze detection device 112 detects gaze movement does not necessarily coincide with the frame switching timing of the photodetector 2004. In general, the gaze detection cycle by the gaze detection device 112 is often shorter than the frame cycle of the photodetector 2004.

[0189] The above-described zone setting information may include only gaze coordinates instead of all coordinates of each zone. The left side of Fig. 28 shows an example in which the coordinates of all pixels in each zone are included in the zone setting information. The right side of Fig. 28 shows an example in which the sizes of zones Z1 and Z2 are registered in advance, and the center positions of each zone Z1 and Z2 are set as gaze positions, and only the coordinates of the gaze positions are included in the zone setting information. In the example on the right side of Fig. 28, the sizes of zones Z1 and Z2 cannot be changed, but the coordinate information for zones Z1 and Z2 included in the zone setting information can be significantly reduced.

[0190] Furthermore, the sizes of zones Z1 and Z2 may be fixed to a predetermined size. The coordinates of zones Z1 and Z2 may be set so that the center positions of zones Z1 and Z2 are the gaze coordinates. In Fig. 20, the gaze information detected by the gaze detection device 112 is input to the signal processing device 111, but the gaze information detected by the gaze detection device 112 may also be input directly to the light detection device 2004.

[0191] The gaze detection device 112 may output gaze coordinates obtained by normalizing the coordinates of the pixel unit 81. Alternatively, the gaze detection device 112 may convert the physical gaze coordinates detected by the gaze detection device 112 into the coordinates of the pixel unit 81 and output the converted coordinates.

[0192] (Prediction of Gaze Coordinates) Gaze information detected by the gaze detection device 112 is input to the photodetector 2004 via the signal processing device 111. The photodetector 2004 performs exposure operations and pixel signal readout operations based on the gaze position detected by the gaze detection device 112, so there is a time lag between when the gaze detection device 112 detects the gaze position and when the photodetector 2004 reads out pixel signals from the zone corresponding to the gaze position. Furthermore, the period in which the gaze detection device 112 detects the gaze is different from the frame period for exposure and pixel signal readout by the photodetector 2004, and the two operate asynchronously.

[0193] FIG. 29 is a diagram showing the time lag between when the gaze detection device 112 detects the gaze position and when the photodetector 2004 reads out pixel signals from a zone corresponding to the gaze position. The vertical lines in FIG. 29 represent the gaze detection cycle and the frame switching timing of the VST sensor. That is, by the time the photodetector 2004 reads out pixel signals, the gaze may have moved to another position. Therefore, the gaze movement during the time lag shown in FIG. 29 may be predicted, and pixel signals may be read out from a zone corresponding to the predicted gaze position. Any algorithm may be used to predict the gaze position; for example, the gaze position may be predicted by linearly interpolating the gaze positions in two or more past frames.

[0194] (Gaze Position in Three-Dimensional Coordinates) Some gaze detection devices 112 output the gaze (gaze) position of a person as coordinates in three-dimensional space (three-dimensional coordinates). In addition, the light detection device 2004 according to the present disclosure may include a VST (Video See-Through) sensor that generates a captured image for the left eye and a VST sensor that generates a captured image for the right eye.

[0195] Figure 30 is a diagram illustrating the gaze position and coordinate designation of each zone in two VST sensors for the left and right eyes. The photodetector 2004 projects three-dimensional coordinates 118 on a display plane 117 virtually arranged in three-dimensional space corresponding to the gaze (gaze) position output from the gaze detection device 112 onto an image plane 119 of the VST sensor for the left eye and an image plane 120 of the VST sensor for the right eye, respectively, to calculate coordinates corresponding to the gaze position on each image plane. Note that the virtual display plane 117 in Figure 30 does not necessarily have to coincide with the physical display plane. The coordinates of zones Z1 to Z3 are designated on the image plane for the left eye and the image plane for the right eye, respectively.

[0196] Even when a VST sensor is used, as shown in Fig. 29, a time lag occurs between when the gaze position is detected by gaze detection device 112 and when image data for each zone is read out on each image plane 119, 120. Therefore, as shown in Fig. 29, the movement of the gaze during the time lag may be predicted, and pixel signals may be read out from the zone corresponding to the predicted gaze position.

[0197] Fig. 31 is a diagram showing data transmitted and received between two VST sensors (hereinafter referred to as CIS1(L) and CIS2(R)) that constitute the gaze detection device 112. Fig. 31 shows an example in which the signal processing device 111 is an AP (Application Processor). Fig. 31 shows an example in which CIS1(L) is the primary (main sensor) and CIS2(R) is the secondary (sub-sensor), but CIS2(R) may be the primary and CIS1(L) may be the secondary.

[0198] The AP 111 transmits zone information about the zone Z1, including the gaze position, to the CIS1(L) for the left eye via, for example, an internal integrated circuit (I2C) or an improved inter-integrated circuit (I3C). The zone information includes the position and size (x, y, w, h) of the zone Z1 and offset information (xoffset, yoffset) of the zone Z1 in the CIS2(R). x and y are the two-dimensional coordinates of the image plane of the CIS1(L), w is the width of the zone Z1, and h is the height of the zone Z1.

[0199] CIS1(L) generates zone setting information including the position and size (x, y, w, h) of zone Z1. The zone setting information includes offset information (xoffset, yoffset) of zone Z2 transmitted from AP 111.

[0200] CIS1(L) transmits the generated zone setting information to CIS2(R) for the right eye via, for example, I2C / I3C or SPI (Serial Peripheral Interface). CIS2(R) generates zone setting information including the position and size of zone Z2 (x+xoffset, y+yoffset, w, h). Note that since there is almost no deviation between the left and right lines of sight in the vertical direction of the image plane, yoffset may be omitted.

[0201] CIS1(L) and CIS2(R) each transmit image data to AP111 independently via, for example, MIPI.

[0202] There is a risk of a difference in the timing at which the zone setting information is reflected in CIS1(L) and CIS2(R). Therefore, reflection timing information may be added to the zone setting information transmitted from CIS1(L) to CIS2(R) to synchronize the setting reflection timing between CIS1(L) and CIS2(R). The reflection timing information is information indicating that the zone setting information will be reflected N frames later, assuming that CIS1(L) and CIS2(R) are synchronized. This prevents a problem of the settings being reflected in different frames between the left-eye image and the right-eye image due to a difference in the setting timing of the zone setting information between CIS1(L) and CIS2(R).

[0203] The image data generated by CIS1(L) and the image data generated by CIS2(R) may be concatenated and transmitted to AP 111 as a single image. Fig. 32 is a block diagram showing a case where two images generated by CIS1(L) and CIS2(R) are concatenated and transmitted to AP 111 as a single image. In the example of Fig. 32, the image data output from CIS2(R) is transmitted to CIS1(L) by, for example, MIPI. CIS1(L) concatenates the two image data and transmits the resulting image data to AP 111 via, for example, MIPI.

[0204] 33 is a diagram showing the data structure of image data that CIS1(L) transmits to AP 111. The image data includes a frame start (FS), a frame header (FH), image data generated by CIS1(L) and image data generated by CIS(R), a frame footer (FF), and a frame end (FE).

[0205] In Figure 33, the image data generated by CIS1 (L) is placed on the higher bit side of the bit string, and the image data generated by CIS2 (R) is placed on the lower bit side of the bit string, but they may also be placed as separate bit strings.

[0206] In the case of Figure 33, AP 111 can acquire two pieces of image data almost simultaneously, so there is no need to synchronize the reception timing of the two pieces of image data or adjust the difference in reception timing, as in the case of Figure 32, and the processing of AP 111 can be simplified.

[0207] (Additional Information on Zone Setting ID) A frame number may be assigned to the frame currently being captured. The frame number is managed using a string of multiple bits. The photodetector 2004 according to the present disclosure may use the lowest N bits of the bit string representing the frame number as a zone setting ID and transmit it to the signal processing device 111.

[0208] Fig. 34 is a diagram showing the exposure and readout timing of the photodetector 2004. The photodetector 2004 transmits the above-mentioned zone setting ID to the signal processing device 111 (e.g., AP 111) using an existing communication interface. Fig. 34 shows an example in which the lowest two bits of the frame number are used as the zone setting ID using, for example, a GPIO (General Purpose Input / Output), which is an existing communication interface. "LL", "LH", "HL", and "HH" in Fig. 34 indicate examples in which the lowest two bits are "00", "01", "10", and "11", respectively.

[0209] 34, image data exposed within a frame period from time t1 to t2 has the value "0" of the lowest two bits of the frame number added to the image data via the MIPI interface during the period from time t3 to t4, and is transmitted to the AP 111. In this case, the lowest two bits "00" have been transmitted to the AP 111 in advance via GPIO, and therefore the AP 111 understands that the value of the lowest two bits added to the image data is "0," and therefore that the data was exposed within the frame period from time t1 to t2.

[0210] The GPIO is a communication interface that is prepared in advance to enable the photodetector 2004 and the AP 111 to send and receive various information. In this way, the photodetector 2004 can send, for example, the lowest two bits of an existing frame number to the AP 111 using an existing communication interface, so that the zone setting ID can be set and notified without adding new hardware. Note that, although the lowest two bits are shown here as an example, the number of bits is not important.

[0211] (Connecting Multiple CISs to a Single Lane) A plurality of photodetector devices 2004 according to the present disclosure can be connected to a single lane to communicate with the signal processing device 111.

[0212] 35 is a block diagram showing a schematic configuration of an information processing system 1001 in which a plurality of photodetectors 2004 are connected to a single lane 122 and communicate with a signal processing device 111. In FIG. 35, an example in which the signal processing device 111 is an AP 111 is shown.

[0213] In the information processing system 1001 in Fig. 35 , each photodetector 2004 connected to a single lane 122 can transmit and receive various types of information to and from the AP 111, separately from the other photodetectors 2004. In the single lane 122, packets containing image data from the multiple photodetectors 2004 are combined, for example, in a time-division manner and transmitted. In addition, in the single lane 122, packets containing various types of information to be sent from the AP 111 to the multiple photodetectors 2004 are transmitted. The header of each packet contains information identifying the destination photodetector 2004 or AP 111.

[0214] <Application Example to Intra-Vivo Information Acquisition System> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

[0215] FIG. 36 is a block diagram showing an example of a schematic configuration of a system for acquiring in-vivo information of a patient using a capsule endoscope to which the technology according to the present disclosure (the present technology) can be applied.

[0216] The in-vivo information acquisition system 10001 includes a capsule endoscope 10100 and an external control device 10200 .

[0217] The capsule endoscope 10100 is swallowed by a patient during an examination. The capsule endoscope 10100 has an imaging function and a wireless communication function, and moves through the inside of organs such as the stomach and intestines by peristaltic movement or the like until it is naturally expelled from the patient, sequentially capturing images of the inside of the organs (hereinafter also referred to as in-vivo images) at predetermined intervals, and sequentially wirelessly transmitting information about the in-vivo images to an external control device 10200 outside the body.

[0218] The external control device 10200 comprehensively controls the operation of the in-vivo information acquisition system 10001. The external control device 10200 also receives information about the in-vivo images transmitted from the capsule endoscope 10100, and generates image data for displaying the in-vivo images on a display device (not shown) based on the received information about the in-vivo images.

[0219] In this way, the in-vivo information acquisition system 10001 can obtain in-vivo images of the state inside the patient's body at any time from the time the capsule endoscope 10100 is swallowed until it is expelled.

[0220] The configurations and functions of the capsule endoscope 10100 and the external control device 10200 will be described in more detail.

[0221] The capsule endoscope 10100 has a capsule-shaped housing 10101, which houses a light source unit 10111, an imaging unit 10112, an image processing unit 10113, a wireless communication unit 10114, a power supply unit 10115, a power supply unit 10116, and a control unit 10117.

[0222] The light source unit 10111 is composed of a light source such as an LED (light emitting diode), and irradiates the imaging field of the imaging unit 10112 with light.

[0223] The imaging unit 10112 is composed of an imaging element and an optical system consisting of multiple lenses provided in front of the imaging element. Reflected light (hereinafter referred to as observation light) of light irradiated onto the body tissue to be observed is collected by the optical system and incident on the imaging element. In the imaging unit 10112, the imaging element photoelectrically converts the incident observation light, generating an image signal corresponding to the observation light. The image signal generated by the imaging unit 10112 is provided to the image processing unit 10113.

[0224] The image processing unit 10113 is configured with processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and performs various signal processing on the image signal generated by the imaging unit 10112. The image processing unit 10113 provides the image signal that has been subjected to the signal processing to the wireless communication unit 10114 as RAW data.

[0225] The wireless communication unit 10114 performs predetermined processing such as modulation on the image signal that has been subjected to signal processing by the image processing unit 10113, and transmits the image signal to the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 also receives a control signal related to drive control of the capsule endoscope 10100 from the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 provides the control signal received from the external control device 10200 to the control unit 10117.

[0226] The power supply unit 10115 is composed of an antenna coil for receiving power, a power regeneration circuit that regenerates power from the current generated in the antenna coil, a boost circuit, etc. The power supply unit 10115 generates power using the principle of so-called contactless charging.

[0227] The power supply unit 10116 is configured by a secondary battery and stores the power generated by the power supply unit 10115. In Fig. 36, to avoid cluttering the drawing, arrows and the like indicating the destinations of the power supply unit 10116 are omitted, but the power stored in the power supply unit 10116 is supplied to the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the control unit 10117 and can be used to drive these units.

[0228] The control unit 10117 is composed of a processor such as a CPU, and appropriately controls the operation of the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the power supply unit 10115 in accordance with control signals transmitted from the external control device 10200.

[0229] The external control device 10200 is configured with a processor such as a CPU or a GPU, or a microcomputer or control board equipped with a processor and a storage element such as a memory. The external control device 10200 controls the operation of the capsule endoscope 10100 by transmitting a control signal to the control unit 10117 of the capsule endoscope 10100 via the antenna 10200A. In the capsule endoscope 10100, for example, the light irradiation conditions of the light source unit 10111 for the observation object can be changed by the control signal from the external control device 10200. Furthermore, the imaging conditions (e.g., the frame rate and exposure value of the imaging unit 10112) can be changed by the control signal from the external control device 10200. Furthermore, the control signal from the external control device 10200 can change the content of processing in the image processing unit 10113 and the conditions for transmitting image signals from the wireless communication unit 10114 (e.g., the transmission interval, the number of transmitted images, etc.).

[0230] The external control device 10200 also performs various image processing on the image signal transmitted from the capsule endoscope 10100 to generate image data for displaying the captured in-vivo image on a display device. The image processing can include various signal processing such as development processing (demosaic processing), image quality improvement processing (band enhancement processing, super-resolution processing, NR (Noise Reduction) processing, and / or image stabilization processing), and / or enlargement processing (electronic zoom processing). The external control device 10200 controls the driving of the display device to display the captured in-vivo image based on the generated image data. Alternatively, the external control device 10200 may record the generated image data in a recording device (not shown) or print it out on a printing device (not shown).

[0231] The above describes an example of an in-vivo information acquisition system to which the technology disclosed herein can be applied. The technology disclosed herein can be applied to the imaging unit 10112 of the above-described configuration. By applying the technology disclosed herein to the imaging unit 10112, clearer images of the surgical site can be obtained, thereby improving the accuracy of the examination.

[0232] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor is considered to be circuitry or processing circuitry, including transistors and other circuits. A processor may also be a programmed processor that executes a program stored in a memory.

[0233] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0234] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0235] The present technology may be configured as follows: (1) A photodetector including: a pixel unit having a plurality of pixels each including a photoelectric conversion element; and a control unit that alternatively selects between a first mode in which a scanning speed for reading out photoelectrically converted pixel signals is different for a first region consisting of two or more pixels including a gaze position among the plurality of pixels, and a second mode in which the scanning speed is the same for a second region surrounding at least a part of the first region. (2) The photodetector described in (1), in which the control unit updates at least one of a position or a size of the first region based on input update information of the gaze position. (3) The photodetector described in (1) or (2), in which the control unit alternatively selects between the first mode and the second mode on a frame-by-frame basis, and when the first mode is selected, makes a vertical blanking period longer than when the second mode is selected, thereby stopping circuits that do not require operation. (4) The photodetector according to (3), wherein, when the first mode is selected, the control unit increases the scanning speed when scanning pixel rows that do not include the first region but include the second region faster than the scanning speed when scanning pixel rows that include the first region, thereby making the vertical blanking period longer than when the second mode is selected in which the scanning speed for the pixel rows that include the first region and the scanning speed for the pixel rows that include the second region are made equal. (5) The photodetector according to any one of (1) to (4), wherein, when the first mode is selected, the control unit allows at least one of the position and size of the first region to be changed until a pixel shutter operation timing for the position of the first region, and when the second mode is selected, allows at least one of the position and size of the first region to be changed until a timing predicted from the position of the first region in the immediately preceding frame. (6) The photodetector according to any one of (1) to (5), wherein, when the first mode or the second mode is selected, the control unit starts readout of pixel signals from the first region or the second region from the beginning of a frame.(7) The photodetector according to any one of (1) to (6), wherein, if readout of the pixel signals is being performed when the control unit is instructed to switch from the first mode to the second mode, the control unit stops readout and starts exposure operation in the second mode, and sets the interval between frames at which the exposure operation in the second mode is started shorter than the interval between frames thereafter. (8) The photodetector according to any one of (1) to (6), wherein, if readout of the pixel signals is being performed when the control unit is instructed to switch from the first mode to the second mode, the control unit starts exposure operation in the second mode after readout of one frame is completed, and sets the interval between frames to be the same regardless of whether switching from the first mode to the second mode is performed. (9) The photodetector according to any one of (1) to (6), wherein, if readout of the pixel signals is being performed when the control unit is instructed to switch from the second mode to the first mode, the control unit stops readout and starts exposure operation in the first mode, and sets the interval between frames at which the exposure operation in the first mode is started shorter than the interval between frames thereafter. (10) The photodetector according to any one of (1) to (6), wherein, if the control unit is reading out the pixel signals when the instruction to switch from the second mode to the first mode is given, the control unit starts an exposure operation in the first mode after the completion of readout for one frame, with a delay so as not to interfere with the readout for one frame, and keeps the frame intervals the same regardless of whether the switching from the second mode to the first mode is performed. (11) The photodetector according to (10), wherein the control unit makes the exposure period in the first mode shorter in a frame in which the second mode is switched to the first mode than in subsequent frames. (12) The photodetector according to any one of (1) to (6), wherein, if the control unit is reading out the pixel signals when the instruction to switch from the second mode to the first mode is given, the control unit starts reading out the pixel signals in the second mode with an interval of one frame after the readout is completed. (13) The photodetector according to any one of (1) to (12), wherein the control unit performs a rolling shutter operation.(14) The photodetector according to any one of (1) to (12), wherein the control unit performs a global shutter operation. (15) The photodetector according to (14), wherein, when the control unit performs the global shutter operation and selects the first mode, the control unit allows at least one of the position and size of the first region to be changed until a timing at which pixel signals at the position of the first region are read out. (16) The photodetector according to any one of (1) to (15), wherein the gaze position update information is input in synchronization with a frame switching timing. (17) The photodetector according to any one of (1) to (15), wherein, when the gaze position update information is input multiple times within a period of one frame, the control unit updates at least one of the position and size of the first region based on the update information input last, and reads out pixel signals from the first region. (18) The photodetector according to any one of (1) to (17), further comprising: a signal processing circuit that generates image data on a frame-by-frame basis based on the pixel signals of each of the plurality of pixels, wherein the signal processing circuit includes, in the image data, information for identifying a frame and information for specifying the position and size of the first region. (19) The photodetector according to any one of (1) to (18), wherein, regardless of whether the first mode or the second mode is selected, the control unit performs a binning process on the second region to make the resolution of the second region lower than the resolution of the first region. (20) The photodetector according to any one of (1) to (19), further comprising: a third region that surrounds at least a part of the second region, wherein, when the first mode is selected, the control unit makes the scanning speeds of the first region, the second region, and the third region different, and when the second mode is selected, makes the scanning speeds of the first region, the second region, and the third region the same.(21) An information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a control unit that alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for each region, the first region consisting of two or more pixels including the gaze position among the plurality of pixels, and a second region surrounding at least a part of the first region, and a second mode in which the scanning speed is made the same; an input unit that accepts user requests; and a processing circuit that transmits a signal to the control unit to make the selection based on the user request. (22) An information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a control unit that alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for a first region consisting of two or more pixels including the gaze position among the plurality of pixels, and a second region surrounding at least a part of the first region, and a second mode in which the scanning speed is made the same for each region; and a processing circuit that, in the first mode, processes the pixel signals read out at different scanning speeds in the first region and the second region to generate an image for display. (23) An information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a control unit that alternatively selects between a first mode in which the scanning speed for reading out photoelectrically converted pixel signals is made different for each region, in a first region consisting of two or more pixels including the gaze position among the plurality of pixels, and a second region surrounding at least a part of the first region, and a second mode in which the scanning speed is made the same; and a processing circuit that transmits a signal to the control unit to make the selection based on gaze detection information detected by the gaze point detection unit.

[0236] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0237] 1 Light detection device, 31 Output mechanism unit, 32 Wearing mechanism unit, 33 Wearing band, 34 Housing, 35 Spatial imaging unit, 51 Control unit, 52 Input unit, 53 Output unit, 54 Memory unit, 55 Communication unit, 56 Drive, 57 Removable storage medium, 58 Bus, 59 Gazing point detection unit, 59A Image analysis device, 59AP Application processor, 59S Eyeball image sensor, 59SRL Eyeball image sensor, 59SRR Eyeball image sensor, 59STL Eyeball image sensor, 59STR Infrared, 59STR Eyeball image sensor, 60 Imaging unit, 61 Display unit, 61P Display image, 61PL Image, 61PR Image, 71 Image analysis unit, 72 Display image generation unit, 81 Pixel unit, 82 Vertical scanning circuit, 83 Analog-to-digital converter, 84 Horizontal transfer scanning circuit, 85 timing control circuit, 86 horizontal transfer line, 87 amplifier circuit, 88 signal processing circuit, 89 output circuit, 90 area control unit, 100 reference voltage output unit, 101 comparator, 102 counter, 103 latch, 110 light detection system, 111 signal processing device, 112 line of sight detection device, 113 interface circuit (IF), 114 frame buffer, 115 first processing unit, 116 second processing unit, 117 three-dimensional space, 118 three-dimensional coordinates, 119 image plane, 120 image plane, 122 single lane, 10001 in-vivo information acquisition system, 10100 capsule endoscope, 10101 housing, 10111 light source unit, 10112 imaging unit, 10113 image processing unit, 10114 wireless communication unit, 10114A antenna, 10115 Power supply unit, 10116 power supply unit, 10117 control unit, 10200 external control device, 10200A antenna

Claims

1. A photodetection device comprising: a pixel portion having a plurality of pixels each including a photoelectric conversion element; a first region composed of two or more pixels including a fixation position among the plurality of pixels; and a second region surrounding at least a part of the first region, and a control unit that alternatively selects a first mode in which a scanning speed for reading a pixel signal that has been photoelectrically converted is made different for each region and a second mode in which the scanning speed is made the same.

2. The photodetection device according to claim 1, wherein the control unit updates at least one of the position or size of the first region based on the input update information of the fixation position.

3. The photodetection device according to claim 1, wherein the control unit alternatively selects the first mode or the second mode in units of frames, and when the first mode is selected, the vertical blanking period is made longer than when the second mode is selected to stop an unnecessary circuit.

4. The photodetection device according to claim 3, wherein when the first mode is selected, the control unit makes the scanning speed when scanning a pixel row including the second region without including the first region faster than the scanning speed when scanning a pixel row including the first region, so that the vertical blanking period is made longer than when the second mode is selected in which the scanning speed when scanning a pixel row including the first region and the scanning speed when scanning a pixel row including the second region are made the same.

5. The photodetection device according to claim 1, wherein when the first mode is selected, the control unit allows a change in at least one of the position or size of the first region until the pixel shutter operation timing at the position of the first region, and when the second mode is selected, the control unit allows a change in at least one of the position or size of the first region until the timing predicted from the position of the first region in the previous frame.

6. The photodetection device according to claim 1, wherein when the first mode or the second mode is selected, the control unit starts reading the pixel signal of the first region or the second region from the head of the frame.

7. The photodetection device according to claim 1, wherein when the control unit is instructed to switch from the first mode to the second mode and the pixel signal is being read, the control unit stops the reading and starts an exposure operation in the second mode, and makes the interval of the frames in which the exposure operation in the second mode is started shorter than the interval of the subsequent frames.

8. When the control unit is reading the pixel signal when instructed to switch from the first mode to the second mode, if the reading for one frame is completed, the exposure operation in the second mode is started, and regardless of whether or not to switch from the first mode to the second mode, the frame intervals are made the same. The photodetection device according to claim 1.

9. When the control unit is reading the pixel signal when instructed to switch from the second mode to the first mode, the reading is aborted and the exposure operation in the first mode is started, and the frame interval of the frame in which the exposure operation in the first mode is started is made shorter than the frame intervals of the subsequent frames. The photodetection device according to claim 1.

10. When the control unit is reading the pixel signal when instructed to switch from the second mode to the first mode, after the reading for one frame is completed, the exposure operation in the first mode is delayed and started so as not to interfere with the reading for one frame, and regardless of whether or not to switch from the second mode to the first mode, the frame intervals are made the same. The photodetection device according to claim 1.

11. In the frame in which the control unit switches from the second mode to the first mode, the exposure period in the first mode is made shorter than that of the subsequent frames. The photodetection device according to claim 10.

12. When the control unit is reading the pixel signal when instructed to switch from the second mode to the first mode, after the reading is completed, after skipping one frame, the reading of the pixel signal in the second mode is started. The photodetection device according to claim 1.

13. The control unit performs a rolling shutter operation. The photodetection device according to claim 1.

14. The control unit performs a global shutter operation. The photodetection device according to claim 1.

15. When the control unit performs the global shutter operation and selects the first mode, at least one of the position or size of the first region is allowed to be changed until the timing of reading the pixel signal at the position of the first region. The photodetection device according to claim 14.

16. The update information of the fixation position is input in synchronization with the frame switching timing. The photodetection device according to claim 1.

17. When the update information of the fixation position is input multiple times within one frame period, the control unit updates at least one of the position or size of the first region based on the last input update information, and reads out the pixel signal of the first region. The light detection device according to claim 1.

18. A signal processing circuit that generates image data in frame units based on the pixel signals in each of the plurality of pixels is provided. The signal processing circuit includes information for identifying a frame and information for specifying the position and size of the first region in the image data. The light detection device according to claim 1.

19. Even when either the first mode or the second mode is selected, the control unit performs binning processing on the second region to make the resolution of the second region lower than the resolution of the first region. The light detection device according to claim 1.

20. A third region surrounding at least a part of the second region is provided. When the first mode is selected, the control unit makes the scanning speeds of the first region, the second region, and the third region different from each other. When the second mode is selected, the control unit makes the scanning speeds of the first region, the second region, and the third region the same. The light detection device according to claim 1.

21. A fixation point detection unit that detects a person's fixation position, a pixel unit having a plurality of pixels each including a photoelectric conversion element, a first region composed of two or more pixels including the fixation position among the plurality of pixels, and a second region surrounding at least a part of the first region. A control unit that selectively selects a first mode in which the scanning speed for reading out the pixel signal that has been photoelectrically converted is different for each region, and a second mode in which the scanning speed is the same, an input unit that receives a user's request, and a processing circuit that transmits a signal for making the selection based on the user's request to the control unit. An information processing system.

22. An information processing system comprising: a fixation point detection unit that detects a person's fixation position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a first region composed of two or more pixels including the fixation position among the plurality of pixels, and a second region surrounding at least a part of the first region, a control unit that alternatively selects a first mode in which scanning speeds for reading out pixel signals photoelectrically converted are made different for each region and a second mode in which the scanning speeds are made the same; and a processing circuit that, in the first mode, processes pixel signals read out at different scanning speeds in the first region and the second region to generate a display image.

23. An information processing system comprising: a fixation point detection unit that detects a person's fixation position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a first region composed of two or more pixels including the fixation position among the plurality of pixels, and a second region surrounding at least a part of the first region, a control unit that alternatively selects a first mode in which scanning speeds for reading out pixel signals photoelectrically converted are made different for each region and a second mode in which the scanning speeds are made the same; and a processing circuit that transmits a signal for performing the selection to the control unit based on the line-of-sight detection information detected by the fixation point detection unit.

Images