Light detection device and information processing system

Abstract

[Problem] To provide a light detection device that prevents occurrence of a sense of incongruity at a boundary region. [Solution] A light detection device according to the present invention comprises: a pixel unit that includes a plurality of pixels each including a photoelectric conversion element; and a control unit that controls pixel signals of respective pixels in a boundary region between a first region, which is constituted by 2 or more pixels including a gaze position among the plurality of pixels, and a second region, which is disposed so as to surround at least part of the first region and has a resolution different from that of the first region, on the basis of pixel signals of surrounding pixels.

Description

Photodetector and information processing system

[0001] The present disclosure relates to a light detection device and an information processing system.

[0002] BACKGROUND ART In wearable displays such as head-mounted displays, a technology is known that captures an image of real space in a field of view that corresponds to the user's line of sight, and processes the captured image according to the real space to realize augmented reality or virtual reality.

[0003] For example, a technique is known in which the resolution of a region of interest including the gaze position of the eyes of a user wearing a head-mounted display is made higher than that of other regions (see Patent Document 1).

[0004] International Publication No. 2019 / 171522

[0005] Since the resolution differs between the region of interest and other regions, the resolution changes discontinuously at the boundary between the regions, resulting in an unnatural captured image. In particular, the greater the difference in resolution between the region of interest and other regions, the greater the degree of unnaturalness may be.

[0006] Therefore, the present disclosure provides a light detection device and an information processing system that prevent the feeling of incongruity from occurring in the boundary area.

[0007] In order to solve the above problems, according to the present disclosure, there is provided a photodetection device comprising: a pixel section having a plurality of pixels each including a photoelectric conversion element; and a control section that controls, based on pixel signals of surrounding pixels, a pixel signal of each pixel in a boundary region between a first region consisting of two or more pixels among the plurality of pixels that include a gaze position, and a second region that is arranged to surround at least a portion of the first region and has a different resolution from the first region.

[0008] The control unit may generate pixel signals for pixels in the boundary region closer to the first region that are closer to the pixel signals of the pixels in the first region, and may generate pixel signals for pixels closer to the second region that are closer to the pixel signals of the pixels in the second region.

[0009] The control unit may control the pixel signal of each pixel in the boundary region by a weighted average of the pixel signals of one or more adjacent pixels in a first direction and the pixel signals of one or more adjacent pixels in a second direction that intersects the first direction.

[0010] The control unit may calculate the weighted average after weighting the pixel signals of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in a first direction, and the pixel signals of one or more adjacent pixels in a second direction that intersects the first direction with weights that correspond to the pixel positions of each pixel in the boundary region.

[0011] The boundary region may have a first boundary region and a second boundary region arranged on both ends of the first region in the first direction, and a third boundary region and a fourth boundary region arranged on both ends of the first region in the second direction, and the control unit may control a pixel signal for each pixel in the first to fourth boundary regions using the corresponding weight and surrounding pixels.

[0012] The weighted averages of the first boundary area and the second boundary area may be calculated by analog arithmetic processing, and the weighted averages of the third boundary area and the fourth boundary area may be calculated by digital signal processing.

[0013] The weighted averages of the first boundary area and the second boundary area may be calculated by digital signal processing, and the weighted averages of the third boundary area and the fourth boundary area may be calculated by analog arithmetic processing.

[0014] The control unit may have a signal processing circuit that performs digital signal processing to calculate a weighted average of the pixel signals of each pixel in the boundary region between the pixel signals of one or more pixels adjacent in the first direction and the pixel signals of one or more pixels adjacent in the second direction.

[0015] The control unit may include: an analog arithmetic processing circuit that calculates a weighted average between the pixel signal of each pixel in the boundary region and pixel signals of one or more pixels adjacent to the pixel signal in the first direction; and a digital signal processing unit that calculates a weighted average between the pixel signal of each pixel weighted averaged by the analog arithmetic processing circuit and pixel signals of one or more pixels adjacent to the pixel signal in the second direction weighted averaged by the analog arithmetic processing circuit.

[0016] The analog arithmetic processing circuit and the digital signal processing unit may calculate the weighted average after weighting the pixel signals of each pixel in the boundary region, the pixel signals of one or more pixels adjacent in a first direction, and the pixel signals of one or more pixels adjacent in a second direction intersecting the first direction with weights corresponding to the pixel positions of each pixel in the boundary region.

[0017] The weighted average may be calculated for some pixel regions in the boundary region by the analog arithmetic processing circuit, and for pixel regions other than the some pixel regions by the digital signal processing unit.

[0018] The control unit may include an analog arithmetic processing circuit that calculates a weighted average of the pixel signals of each pixel in the boundary region between the pixel signals of one or more pixels adjacent to the pixel signals in the first direction and the pixel signals of one or more pixels adjacent to the pixel signals in the second direction.

[0019] The analog arithmetic processing circuit may calculate the weighted average after weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more pixels adjacent in the first direction, and the pixel signals of one or more pixels adjacent in the second direction with weights according to the pixel position of each pixel in the boundary region.

[0020] the analog arithmetic processing circuit includes a plurality of signal lines transmitting pixel signals of a plurality of pixels arranged in the second direction, and the analog arithmetic processing circuit includes: a plurality of weighted averaging circuits provided for each of the plurality of signal lines and calculating the weighted average; and a plurality of switch control circuits provided corresponding to the plurality of weighted averaging circuits, wherein each of the plurality of weighted averaging circuits includes: a plurality of first capacitors provided for each corresponding signal line in the boundary region, each having a different capacitance; a plurality of second capacitors provided corresponding to a plurality of pixels adjacent in the first direction and the second direction to a pixel connected to the corresponding signal line, and having the same capacitance; a plurality of first switches that switch whether to charge the plurality of first capacitors with charge corresponding to the pixel signal of the one pixel; and a second switch that switches whether to combine the accumulated charges of the plurality of second capacitors and the accumulated charges of the plurality of first capacitors, and each of the plurality of switch control circuits controls switching of the plurality of first switches and the second switches in the corresponding weighted averaging circuit, and the weighted average calculated by each of the plurality of weighted averaging circuits may have a nonlinear relationship with the corresponding weight.

[0021] the analog arithmetic processing circuit includes a plurality of signal lines that transmit pixel signals of a plurality of pixels arranged in the second direction, the analog arithmetic processing circuit including: a plurality of weighted average circuits that are provided for each of the plurality of signal lines and calculate the weighted average; and a plurality of switch control circuits that are provided corresponding to the plurality of weighted average circuits, each of the plurality of weighted average circuits including a plurality of sub-weighted average circuits that calculate a weighted average between one pixel connected to a corresponding signal line and a plurality of different pixels arranged around the pixel, each of the plurality of sub-weighted average circuits including a plurality of capacitors and a plurality of switches that switch whether or not to store charge in the plurality of capacitors according to the pixel value of the one pixel or the pixel value of another pixel arranged around the one pixel, and each of the plurality of switch control circuits controls switching of the plurality of switches in the corresponding plurality of sub-weighted average circuits, and there may be a linear relationship between a weighted average obtained by combining the weighted averages calculated by each of the plurality of sub-weighted average circuits and the corresponding weight.

[0022] The boundary region may have a first boundary region and a second boundary region arranged at both ends in the first direction, and a third boundary region and a fourth boundary region arranged at both ends in the second direction, and an arithmetic average of the first boundary region, the second boundary region, the third boundary region, and the fourth boundary region may be calculated by the analog arithmetic processing circuit.

[0023] The pixel array may include an aperture pixel region having the plurality of pixels, an optical black region arranged to surround the aperture pixel region, and a process dummy region arranged to surround the optical black region, and the plurality of weighted average circuits and the switch control circuit may be arranged to correspond to only at least a portion of the aperture pixel region.

[0024] The weighted average circuits and the switch control circuit may be arranged to correspond to only a part of the aperture pixel region including a center position of the aperture pixel region.

[0025] The boundary region may have a first boundary region and a second boundary region, and the control unit may control the pixel signals of each pixel in the first boundary region between the first region and the second region, and in the second boundary region between the second region and a third region that is arranged to surround at least a portion of the second region and has a resolution different from that of the first region and the second region, based on the pixel signals of surrounding pixels.

[0026] According to the present disclosure, there is provided an information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit that has a plurality of pixels, each of which includes a photoelectric conversion element; a control unit that controls pixel signals of each pixel in a boundary region between a first region consisting of two or more pixels among the plurality of pixels that include the gaze position and a second region that is arranged to surround at least a portion of the first region and has a different resolution from the first region, based on pixel signals of surrounding pixels; and a display unit that displays an image generated based on the pixel signals of each pixel in the first region, the second region, and the boundary region.

[0027] According to the present disclosure, there is provided an information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit that has a plurality of pixels each including a photoelectric conversion element; a control unit that controls pixel signals of each pixel in a boundary region between a first region consisting of two or more pixels among the plurality of pixels that include the gaze position and a second region that is arranged to surround at least a portion of the first region and has a different resolution from the first region, based on pixel signals of surrounding pixels; and a processing circuit that generates pixel signals of each pixel in the boundary region based on pixel signals of surrounding pixels in the boundary region.

[0028] According to the present disclosure, there is provided an information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit that has a plurality of pixels, each of which includes a photoelectric conversion element; a control unit that controls pixel signals of each pixel in a boundary region between a first region consisting of two or more pixels among the plurality of pixels that include the gaze position and a second region that is arranged to surround at least a portion of the first region and has a different resolution from the first region, based on pixel signals of surrounding pixels; and a display unit that combines and displays a first image based on the pixel signals of each pixel in the first region, a second image based on the pixel signals of each pixel in the second region, and a third image based on the pixel signals of each pixel in the boundary region.

[0029] 10 is a diagram illustrating an example of the appearance of an information processing system according to the present disclosure. FIG. 11 is a diagram illustrating a modified example of the appearance of the information processing system according to the present disclosure. FIG. 12 is a block diagram illustrating a schematic configuration of an information processing system according to the present disclosure. FIG. 13 is a diagram illustrating an example of the configuration of a gaze point detection unit of an HMD. FIG. 14 is a diagram illustrating an example of the configuration of the gaze point detection unit and a spatial imaging unit as hardware. FIG. 15 is a block diagram illustrating a detailed configuration of an imaging unit. FIG. 16 is a diagram illustrating an example of a plurality of zones set by a region control unit. FIG. 17 is a block diagram illustrating the function of an HMD. FIG. 18 is a diagram illustrating a boundary region. FIG. 19 is a block diagram illustrating a part of the internal configuration of the analog-digital converter of FIG. 5. FIG. 19 is a circuit diagram of an analog-digital converter of a photodetector according to a second embodiment. FIG. 19 is a circuit diagram illustrating an enlarged view of a part of FIG. 20. FIG. 21 is a diagram illustrating a correspondence relationship between a mixing ratio of a two-pixel weighted average circuit and the load capacitance of a vertical signal line. FIG. 22 is a circuit diagram illustrating an analog-digital converter of a photodetector according to a third embodiment. FIG. 23 is a circuit diagram illustrating an enlarged view of a part of FIG. 21. FIG. 22 is a diagram illustrating a correspondence relationship between a mixing ratio of a four-pixel weighted average circuit and the load capacitance of a vertical signal line. FIG. 23 is a diagram illustrating a relationship between a mixing ratio and the load capacitance of a vertical signal line for which a weighted average is calculated. FIG. 24 is a circuit diagram of a main part of an analog-digital converter of a photodetector according to a modified example of the third embodiment. 29A, 29B, and 29C are diagrams showing examples of zone constraints (OK and NG), 30A, 30B, and 30C, respectively.

[0043] FIG. 29A is a diagram showing the correspondence relationship between the mixing ratio and the load capacitance of a vertical signal line.

[0044] FIG. 29B is a diagram plotting the relationship between the mixing ratio and the load capacitance of a vertical signal line for which the weighted average of FIG. 18 is calculated.

[0045] FIG. 30B is a diagram showing an example of providing three zones.

[0046] FIG. 30C is a diagram explaining a switch control circuit.

[0047] FIG. 30D is a diagram showing an example of weighted average processing for first to fourth boundary regions.

[0048] FIG. 30E is a diagram showing the values ​​of the mixing ratio in the boundary region.

[0049] FIG. 30F is a diagram showing the connection relationship between the switch control circuit and the first to third 4-pixel weighted average circuits in the 4-pixel weighted average circuit ... block diagram showing a schematic configuration of a photodetection system including a photodetection device according to the present disclosure.

[0049] FIGS. 29A, 29B, and 29C are diagrams showing examples of OK and NG zone constraints.

[0049] FIG. 30F is a sequence diagram showing the procedure for transmitting and receiving image data between a photodetection device and a signal processing device when the photodetection device assigns a zone setting ID.

[0049] FIG. 30H is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device of FIG. 30. 10 is a sequence diagram showing an example of the data structure of image data transmitted from a photodetector to a signal processing device, and FIG. 11 is a sequence diagram showing a procedure for transmitting and receiving image data between a photodetector and a signal processing device when the signal processing device assigns a zone setting ID.34 is a diagram showing an example of setting information of a zone setting register provided inside the signal processing device of FIG. 33. A diagram showing the data structure of image data of each zone transmitted from a photodetector to a signal processing device. A diagram showing zone setting information. A diagram showing the time lag from when the gaze detection device detects the gaze position until the photodetector reads out a pixel signal from the zone corresponding to the gaze position. A diagram explaining the gaze position and coordinate designation of each zone in two VST sensors for the left eye and the right eye. A diagram showing data transmitted and received between two VST sensors constituting the gaze detection device. A block diagram when two images generated by the VST sensors are concatenated and transmitted as a single image to an AP. A diagram showing the data structure of image data transmitted by CIS1(L) to an AP. A diagram showing exposure and readout timing of a photodetector. A block diagram showing a schematic configuration of a photodetection system in which multiple photodetectors are connected to a single lane and communicate with a signal processing device. A block diagram showing an example of a schematic configuration of an in-vivo information acquisition system.

[0030] Hereinafter, embodiments of a light detection device and an information processing system according to the present disclosure will be described with reference to the drawings. The following description will focus on the main components of the light detection device and the information processing system according to the present disclosure, but the light detection device may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0031] Fig. 1A is a diagram illustrating an example of the appearance of an information processing system 1001 of the present disclosure, Fig. 1B is a diagram illustrating a modified example of the appearance of the information processing system 1001 of the present disclosure, and Fig. 2 is a block diagram illustrating a schematic configuration of the information processing system 1001 of the present disclosure. As shown in Fig. 1A, the information processing system 1001 according to this embodiment is configured as a head-mounted display (HMD). An example of the appearance of the head-mounted display (HMD) of this embodiment will be described with reference to Fig. 1A.

[0032] In this example, the HMD 1001 is composed of an output mechanism unit 1011 and a wearing mechanism unit 1012. The wearing mechanism unit 1012 includes a wearing band 1013 that wraps around the head when worn by the user, thereby securing the device in place. Note that the band does not have to wrap around the head as long as it is secured to the head.

[0033] The output mechanism unit 1011 includes a housing 1014 shaped to cover the left and right eyes when the user wears the HMD 1001, and includes a display panel inside that faces the eyes when worn. The housing 1014 may further include a lens that is positioned between the display panel (display unit 2005 ( FIG. 2 )) and the user's eyes when the HMD 1001 is worn, and that expands the user's field of view. The display panel may be divided into left and right halves, and each region may display a stereo image corresponding to the parallax between the eyes, and stereoscopic vision may be realized by such a display.

[0034] The HMD 1001 may further include speakers or earphones at positions corresponding to the user's ears when worn. In this example, the HMD 1001 includes a camera 1015 on the front surface of the housing 1014, which captures video of the surrounding real space in a field of view corresponding to the user's line of sight. In this specification, the camera 1015 may be referred to as a light detection unit 1015.

[0035] The camera 1015 is, for example, a light detection unit, and includes a light detection device such as an image sensor or a distance measurement sensor, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and an optical system such as an imaging lens. For example, in FIG. 1A , the camera 1015 is configured as a stereo camera that captures images of the space in front of the user from left and right viewpoints corresponding to the user's left and right eyes. Note that the camera 1015 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may be used. For hand tracking applications, the camera 1015 may be configured to capture images of the space below the information processing system. For eye tracking or face tracking applications, the camera 1015 may be configured to capture images of the user's eyes or face.

[0036] The HMD 1001 also includes a sensor 2008 (FIG. 2). The sensor group may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0037] The HMD 1001 may be connected to other processing devices via wireless communication, or may be connected via a wired connection such as a USB (Universal Serial Bus).

[0038] In this case, the HMD 1001 may be configured to run an online application such as a game in which multiple users can participate via a network. In this case, the HMD 1001 performs predetermined processing on the image captured by the camera 1015, and generates and displays a display image within the field of view of the camera 1015.

[0039] The content of the displayed image is not particularly limited, and may vary depending on the functions the user desires from the system and the content of the application that has been started.

[0040] For example, the HMD 1001 may perform some processing on the image captured by the camera 1015, or may superimpose a virtual object that interacts with the image of a real object. Alternatively, the HMD 1001 may render a virtual world in a field of view corresponding to the user's field of view based on the captured image or measurements by a motion sensor included in the sensor group of the HMD 1001.

[0041] Representative examples of these modes include virtual reality (VR), augmented reality (AR), and mixed reality (MR). In addition, a see-through mode (VST: Video See-Through) in which the real world can be seen through the screen of the HMD 1001 may be realized by directly displaying an image captured by the camera (light detection unit) 1015 as a display image.

[0042] An information processing system 1101 according to this embodiment shown in FIG. 1B is configured as a glasses-type HMD.

[0043] The HMD main body 1111 is worn on the user's head when in use. The HMD main body 1111 has a front part 1112, a right temple part 1113 provided on the right side of the front part 1112, a left temple part 1114 provided on the left side of the front part 1112, and a glass part 1115 attached to the underside of the front part 1112. Note that although the glass is integrated in FIG. 1B , it may have two separate glasses, one for each eye, or may be configured to cover only one eye.

[0044] The display unit 1103 is a see-through type display unit and is provided on the surface of the glass unit 1115. The display unit 1103 performs AR display of a virtual object in accordance with the control of the processing circuit (control unit) 2001. Note that the display unit 1103 may be a non-see-through type display unit. In this case, AR display is performed by displaying on the display unit 2005 an image in which the virtual object is superimposed on an image currently captured by the camera 1104.

[0045] The camera 1104 is, for example, a light detection unit, and includes a light detection device such as an image sensor, such as a CCD sensor or a CMOS sensor, or a distance measurement sensor, and an optical system, such as an imaging lens. The camera 1104 is provided facing outward on the outer surface of the front unit 1112, captures an image of an object in real space, and outputs the image information obtained by the capture to the processing circuit (control unit) 2001. In FIG. 1B , for example, two cameras 1104 are provided laterally at a predetermined interval on the front unit 1112. Note that the camera (light detection unit) 1015 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may be used. In hand tracking applications, the camera 1104 may be provided to capture an image of the space below the information processing system. In eye tracking or face tracking applications, the camera 1104 may be provided to capture an image of the user's eyes or face.

[0046] The glasses-type HMD 1101 also includes a sensor 2008 ( FIG. 2 ). The sensor unit may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0047] The glasses-type HMD 1101 may also include a communication IF 2009 (FIG. 2), which communicates with a smartphone or an external device other than a smartphone (for example, a personal computer (PC) or a server device on a network) via a wired or wireless connection.

[0048] <Hardware Configuration Example of HMD> Next, an example of the hardware configuration of the information processing system (HMD 1001 or glasses-type HMD 1101) will be described with reference to Fig. 2. As shown in Fig. 2, the hardware of the information processing system is composed of a CPU 2001, a memory 2002, a camera 2003, a display unit 2005, an input unit 2006, an output unit 2007, a sensor 2008, a communication interface (IF) 2009, an external network 2010, and a secondary storage device 2011, which are mutually connected via a bus 2012 and can transmit and receive data and programs.

[0049] The processing circuit 2001 operates based on programs stored in the memory 2002 or the secondary storage device 2011 and controls the overall operation of the information processing systems 1001 and 1101. The processing circuit is, for example, a processor, which reads and executes each program from the memory 2002 to realize the corresponding function of each program. The processor may include, for example, any one or more of a multi-core processor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or an equivalent discrete or integrated logic circuit. The processing circuit may be realized as multiple chips.

[0050] Memory 2020 may include any type of memory for storing data and executable software instructions, implemented, for example, by random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or semiconductor memory devices such as flash memory, hard disk, optical disk, etc.

[0051] 1A and the camera 1104 in FIG. 1B, and includes a light detection device such as an image sensor or a distance measurement sensor, such as a CCD sensor or a CMOS sensor, and an optical system such as an imaging lens. The camera 2003 has a light detection device 2004.

[0052] The display unit 2005 is a display panel provided inside the housing, and is made up of a display device such as an LCD (Liquid Crystal Display) or an organic EL (ElectroLuminescence).

[0053] The input unit 2006, which is not shown in Figures 1A and 1B, is composed of input devices such as a keyboard, mouse, touch panel, microphone, and controller through which the user inputs operation commands, and supplies various input signals to the processing circuit (control unit) 2001.

[0054] The output unit 2007 is composed of an audio output device such as a speaker, a force feedback device, an odor feedback device, etc., and is controlled by the processing circuit 2001 to output the processing results as sound, force feedback, or odor.

[0055] The sensor 2008 may include at least one of various sensors, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor, for detecting the movement, posture, and position of the HMDs 1001 and 1101. The sensor 2008 may also include a biosensor that senses a person's biometric information and a pressure sensor that detects input.

[0056] The communication interface 2009 is an interface for connecting the information processing systems 1001 and 1101 to an external network 2010. For example, the processing circuit 2001 receives data from other devices and transmits data generated by the processing circuit 1100 to other devices via the communication interface 2009.

[0057] The above describes an example of an information processing system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the light detection device 2004 in the above-described configuration.

[0058] The photodetection device 2004 according to the present disclosure realizes the camera 1015 in FIG. 1 and is composed of a CCD sensor or a CMOS sensor, and outputs the captured image to an image analysis unit and a display image generation unit (described later) within the processing circuit 2001.

[0059] Based on the gaze point information supplied from the gaze point detection unit described later, the light detection device 2004 considers the gaze point area, which is the range in the vicinity of the gaze point position in the captured image, to be the area of ​​interest that the user is paying attention to, and changes the resolution between the area of ​​interest and areas other than the area of ​​interest.

[0060] 1, the light detection device 2004 will be described as being configured by a stereo camera that captures images of the space in front from left and right viewpoints corresponding to the left and right eyes of the user of the HMD 1001. Unless otherwise specified, the description will proceed assuming that the camera 1015 and the light detection device 2004 basically have the same configuration.

[0061] The photodetection device 2004 of this embodiment controls the pixel signals of each pixel in a boundary region between a first region (e.g., zone Z1) consisting of two or more pixels including the gaze position among multiple pixels, and a second region (e.g., zone Z2) arranged to surround at least a portion of the first region and having a different resolution from the first region, based on the pixel signals of surrounding pixels.

[0062] The display unit 2005 in FIG. 2 can display an image generated based on pixel signals of the pixels in the first region, the second region, and the boundary region.

[0063] Furthermore, the processing circuit 2001 in FIG. 2 can generate pixel signals for each pixel in the boundary area based on pixel signals of pixels surrounding the boundary area described above.

[0064] In addition, the display unit 2005 in Figure 2 can combine and display a first image based on the pixel signals of each pixel in the first region described above, a second image based on the pixel signals of each pixel in the second region, and a third image based on the pixel signals of each pixel in the boundary region.

[0065] As a result, it is possible to display an image that does not seem strange to the user, while keeping the resolution of the image output from the photodetector device 2004 lowered in areas other than the region of interest including the gaze position in order to reduce power consumption.

[0066] <Configuration Example of Gaze Point Detection Unit> The following mainly describes an example using the HMD 1001 of FIG. 1A, but the present disclosure is also applicable to the HMD 1011 of FIG. 1B. FIGS. 3 and 4 are diagrams illustrating the configuration of the HMD 1001 according to the present disclosure. The gaze point detection unit 59 shown in FIGS. 3 and 4 is provided in at least one of the processing circuit 2001 and the sensor 2008 of FIG. 2. FIG. 3 is a diagram illustrating an example of the configuration of the gaze point detection unit 59 of the HMD 1001. FIG. 4 is a block diagram illustrating the connection relationship between the gaze point detection unit 59, an application processor (hereinafter, AP) 59AP, and a light detection device 2004 according to an embodiment. In this specification, the light detection device 2004 according to an embodiment may be referred to as an imaging unit 60.

[0067] The lower part of the figure shows a schematic view of the HMD 1001 and the head F of the user wearing it, as seen from above. The HMD 1001 displays a display image 61P, as shown in the upper part, which the user views with the left and right eyes EL and ER. The display image 61P is composed of images 61PL and 61PR viewed by the left and right eyes EL and ER, respectively. In this example, the gaze point detection unit 59 includes infrared LEDs 59STL and 59STR, eye image sensors 59SRL and 59SRR, which are infrared cameras or PSD (Position Sensitive Detector) sensors, and an image analysis device 59A. In this specification, the gaze point detection unit 59 may be referred to as a gaze detection device.

[0068] The infrared LEDs 59STL and 59STR irradiate the user's left and right eyes EL and ER with infrared rays, respectively. The eye image sensors 59SRL and 59SRR capture images of the user's left and right eyes EL and ER, respectively, and supply the image data to the image analysis device 59A.

[0069] The image analyzer 59A identifies the position of the pupil and the position of the infrared light reflected by the cornea from the captured images of the left and right eyes, and identifies the user's line of sight from this positional relationship. This method, known as the corneal reflex method, has been put to practical use in the field of line of sight detection technology.

[0070] The method for detecting the gaze is not limited to this, and any common method may be used, such as capturing images of the left and right eyes with a visible light camera and identifying the gaze from the relative positions of the inner corners of the eyes and the irises.

[0071] The image analysis device 59A determines the intersection RPL, RPR between the line of sight detected by the eyeball image sensors 59SRL, 59SRR and the display unit 61 (display panel) on which the display image 61P is displayed as the user's point of gaze, detects its position coordinates, and supplies them to the image capture unit 60.

[0072] As shown in Figure 4, the gaze point detection unit 59 includes an eyeball image sensor 59S (59STL, 59STR) that captures an image of the eyeball, and an application processor 59AP that essentially functions as an image analysis device 59A, acquiring the gaze point from the captured image of the eyeball and detecting the gaze point.

[0073] Based on the gaze point information supplied from the application processor 59AP, the imaging unit 60 sets the resolution of the area other than the area near the gaze point (gape point area), i.e., the area other than the attention area, to be lower, and captures the image.

[0074] In this configuration, the architecture of the Mobile Industry Processor Interface (MIPI) can be used for data transmission of captured images from the eye image sensor 59S (59STL, 59STR) to the application processor 59AP.

[0075] In addition, the application processor 59AP can use an i2C (Inter-Integrated Circuit) or i3C architecture to transmit information relating to the point of interest to the imaging unit 60 .

[0076] The gaze point detection unit 59 may be configured such that the eye image sensor 59S (59STL, 59STR) and the application processor 59AP are stacked and integrated.

[0077] <Configuration Example of Imaging Unit> Next, a description will be given of a configuration example of the imaging unit 60. Fig. 5 is a block diagram showing a detailed configuration of the imaging unit 60. The imaging unit 60 includes a pixel unit 81, a vertical scanning circuit 82, an analog-digital converter (ADC) group 83, a horizontal transfer scanning circuit 84, a timing control circuit 85, a horizontal transfer line 86, an amplifier circuit 87, a signal processing circuit 88, an output circuit 89, and a region control unit 90.

[0078] The pixel section 81 has a configuration in which pixels, each including a photodiode that performs photoelectric conversion, are arranged in a matrix. Each pixel holds a signal charge corresponding to the amount of incident light. A vertical scanning circuit 82 drives the pixels row by row by supplying a drive pulse to each pixel via pixel drive wiring (not shown). As a result, analog signals from the pixels in each row are supplied to an ADC group 83 via vertical signal lines provided for each column.

[0079] The ADC group 83 is an analog-to-digital conversion circuit configured to have an ADC arranged for each pixel column, the ADC comprising a comparator 101 that compares a reference voltage generated by a reference voltage output unit 100 with an analog signal obtained from a pixel via a vertical signal line for each row line, a counter 102 that counts the comparison time, and a latch 103 that holds the count result.

[0080] The analog signal read out by the vertical signal line is compared with a reference voltage having a slope waveform by a comparator 101, and when the two match, a count value is obtained, thereby converting the signal into a digital signal. The output of each latch 103 is connected to a horizontal transfer line 86.

[0081] The horizontal transfer scanning circuit 84 is composed of, for example, a shift register, and by sequentially outputting horizontal scanning pulses, inputs one row of digital signals stored in the latch 103 to an amplifier circuit 87 and a signal processing circuit 88 via a horizontal transfer line 86.

[0082] The timing control circuit 85 generates and outputs clock signals and control signals that serve as the basis for the operations of the vertical scanning circuit 82, the ADC group 83, and the horizontal transfer scanning circuit 84, based on the vertical synchronization signal, horizontal synchronization signal, and master clock.

[0083] The signal processing circuit 88 serves as a readout processing circuit and performs predetermined processing such as defect correction, demosaic processing, and gamma correction on the input digital signals of each pixel to generate captured image data.

[0084] The output circuit 89 appropriately buffers the captured image data generated by the signal processing circuit 88 and outputs it row by row.

[0085] The region control unit 90 sets multiple zones in the pixel unit 81 based on the gaze point information supplied from the gaze point detection unit 59. FIG. 6 is a diagram showing an example of multiple zones set by the region control unit 90. FIG. 6 shows an example in which the region control unit 90 sets zones Z1 and Z2. There is no limit to the number of zones set by the region control unit 90, but the following mainly describes an example in which the region control unit 90 sets zones Z1 and Z2. Zone Z1 is a pixel region that includes the gaze point position and is a high-resolution region that generates pixel signals in 1×1 pixel units. Zone Z2 is a pixel region that surrounds at least a portion of zone Z1 and is a region that performs binning processing to generate pixel signals in 2×2 pixel units, for example. Zone Z2 is a pixel region with a lower resolution than zone Z1.

[0086] More specifically, the area control unit 90 identifies the pixel position of the coordinates corresponding to the gaze point in the image captured by the imaging unit 60 based on the gaze point information supplied from the gaze point detection unit 59, and identifies a rectangular gaze point area having a predetermined range in the horizontal and vertical directions based on the pixel position corresponding to the identified gaze point as the attention area.

[0087] In the above, an example has been described in which an integral type configuration is adopted as the ADC constituting the ADC group 83, but other configurations may also be used, such as a successive approximation type ADC.

[0088] <Functions Realized by the HMD of FIG. 1> Next, functions realized by the HMD 1001 of FIG. 2 will be described. FIG. 7 is a block diagram illustrating the functions of the HMD 1001. As shown in FIG. 7, the HMD 1001 functionally operates the gaze point detection unit 59, the imaging unit 60, the image analysis unit 71, the display image generation unit 72, and the display unit 61 as shown in FIG. 3 to capture images of real space, analyze the captured images, and process the images based on the analysis results to generate and display a display image. The gaze point detection unit 59, the image analysis unit 71, and the display image generation unit 72 of FIG. 7 are provided in at least one of the processing circuit 2001 or the sensor 2008 of FIG. 2.

[0089] That is, the gaze point detection unit 59 detects the gaze point of the user wearing the HMD 1001 and outputs the detection result to the imaging unit 60 as gaze point information.

[0090] Based on the gaze point information, the imaging unit 60 regards the gaze point area as the area of ​​interest that the user is paying attention to, images the real space with high resolution set for the pixels in the area of ​​interest and low resolution set for the pixels in areas other than the area of ​​interest (gape point area), and outputs the captured image to the image analysis unit 71 and the display image generation unit 72.

[0091] The image analysis unit 71 performs analysis processing, such as face detection, face recognition, gesture recognition, visual tracking, position detection, or acquiring the posture of the HMD 1001 using Visual SLAM (Simultaneous Localization and Mapping), based on the image captured by the imaging unit 60, and outputs the analysis results to the display image generation unit 72.

[0092] Based on the analysis results supplied from the image analysis unit 71, the display image generation unit 72 performs predetermined processing on the image captured by the imaging unit 60 to generate a display image, and outputs the image to the display unit 61 for display.

[0093] By functioning as described above, based on the gaze point information detected by the gaze point detection unit 59, the imaging unit 60 regards the gaze point area as the area of ​​interest that the user is paying attention to and sets it to high resolution, and captures an image with areas other than the area of ​​interest set to low resolution. This effectively reduces the amount of data in the image captured by the imaging unit 60, thereby reducing the processing load in the analysis process in the image analysis unit 71 and the display image generation process in the display image generation unit 72. As a result, it is possible to suppress delays related to the series of processes.

[0094] The photodetector 2004 according to the present disclosure is applicable to the image capturing unit 60 in Fig. 5. The photodetector 2004 according to the present disclosure is also applicable to an event-based vision sensor (EVS) that detects events. Representative embodiments of the photodetector 2004 according to the present disclosure will be described below.

[0095] First Embodiment A photodetector 2004 according to a first embodiment has a pixel unit 81 provided with a focus region and other regions, and generates pixel signals by photoelectric conversion with different resolutions between the focus region and other regions. Hereinafter, the focus region will be referred to as zone Z1 or first region, and the region other than the focus region will be referred to as zone Z2 or second region.

[0096] The photodetector 2004 according to the first embodiment includes a control unit that controls pixel signals of each pixel in a boundary region between a first region including two or more pixels including a gaze position among a plurality of pixels each including a photoelectric conversion element, and a second region that is arranged to surround at least a part of the first region and has a different resolution from the first region, based on pixel signals of surrounding pixels.

[0097] The control unit is, for example, the analog-to-digital converter 83 or the signal processing circuit 88 in FIG. 5 . For pixels in the boundary region closer to the first region, the control unit generates pixel signals that are closer to the pixel signals of the pixels in the first region, and for pixels closer to the second region, generates pixel signals that are closer to the pixel signals of the pixels in the second region. The control unit controls the pixel signals of each pixel in the boundary region by calculating a weighted average of pixel signals of one or more pixels adjacent to the pixel in a first direction (e.g., row direction) X and pixel signals of one or more pixels adjacent to the pixel in a second direction (e.g., column direction) Y intersecting the first direction X. The control unit calculates the weighted average by weighting the pixel signals of each pixel in the boundary region, the pixel signals of one or more pixels adjacent to the pixel in the first direction X, and the pixel signals of one or more pixels adjacent to the pixel in the second direction Y intersecting the first direction X with weights corresponding to the pixel positions of each pixel in the boundary region.

[0098] In the following, the first region will be mainly referred to as zone Z1, and the second region will be mainly referred to as zone Z2, and a specific description will be given.

[0099] Zone Z1 is an area with a higher resolution than zone Z2. In zone Z1, for example, exposure and pixel signal readout are performed for each pixel. In zone Z2, binning processing is performed in units of multiple pixels, and pixel signals are readout for each multiple pixels.

[0100] A boundary region is disposed between zone Z1 and zone Z2. Fig. 8 is a diagram illustrating the boundary region BR. Fig. 8 shows a portion of a pixel region in a pixel unit 81. Fig. 8 shows an example in which a pixel signal is output for each pixel in zone Z1, and in zone Z2, binning processing is performed on a 2 x 2 pixel basis as shown in the dashed-dotted line frame to output one pixel signal. Fig. 8 shows an example in which a boundary region BR of four pixels is provided between zones Z1 and Z2, but the number of pixels in the boundary region BR is arbitrary.

[0101] The dashed-line frame in FIG. 8 represents a 2×2 pixel region in the boundary region BR, and the four pixel values ​​within the dashed-line frame are (x1, x2, x3, x4). In this embodiment, the multiple pixels in the boundary region BR are divided into 2×2 pixel regions, each of which is the same size as the zone Z2, and the pixel values ​​(x1, x2, x3, x4) of each pixel in each pixel region are weighted and averaged using a mixing ratio α to generate a pixel value (x'1, x'2, x'3, x'4). The mixing ratio α is a coefficient that changes depending on the distance between zones Z1 and Z2. In this specification, the mixing ratio α is sometimes referred to as a weight. For example, the mixing ratio α decreases as the pixel is closer to zone Z1 and increases as the pixel is closer to zone Z2.

[0102] FIG. 9 is a block diagram showing a portion of the internal configuration of the analog-digital converter 83 shown in FIG. 5 . In FIG. 9 , the analog-digital converter 83 is referred to as a column ADC (Analog Digital Converter). As shown in FIG. 9 , a plurality of vertical signal lines VSL extend in a second direction (column direction) Y and are arranged at predetermined intervals in a first direction (row direction) X. A plurality of pixels arranged in the second direction Y are connected to each of the vertical signal lines VSL. Pixel signals output from two adjacent pixels in the second direction Y are input to the analog-digital converter 83 via a common vertical signal line VSL. The analog-digital converter 83 includes a comparator 101 for each vertical signal line. Each comparator 101 compares the pixel signals output from the pixels via the vertical signal lines VSL with a reference voltage and outputs the comparison result.

[0103] The output voltage of the comparator 101 is input to a digital signal generation circuit 11. The digital signal generation circuit 11 is composed of, for example, a counter 102, a latch 103, and an amplifier circuit 87, as shown in FIG. 5 . The digital pixel signal output from the digital signal generation circuit 11 is input to a signal processing circuit 88.

[0104] The signal processing circuit 88 performs digital signal processing to calculate a weighted average based on a plurality of digital pixel signals corresponding to a plurality of pixel signals output from a plurality of pixels included in the boundary region BR.

[0105] More specifically, the signal processing circuit 88 divides the pixels included in the boundary region BR into 2x2 pixel units and calculates a weighted average for each pixel in the 2x2 pixel unit using the mixture ratio α. Of the pixel values ​​(x1, x2, x3, x4) of the 2x2 pixels, the weighted average x'1 of the pixel value x1 is expressed as follows (1):

[0106] The other pixel values ​​(x2, x3, x4) of the 2×2 pixels are calculated in the same manner as in equation (1).

[0107] The signal processing circuit 88 divides each pixel in the boundary region BR into pixel regions of the same size as the binning process for zone Z2, and converts the pixel value of each pixel into a weighted average with the pixel values ​​of other pixels in the pixel region. In calculating the weighted average, the mixing ratio α is taken into account, so that the influence of the pixel values ​​of zone Z1 is increased for pixels closer to zone Z1, and the influence of the pixel values ​​of zone Z2 is increased for pixels closer to zone Z2. This allows the pixel values ​​of each pixel in the boundary region BR to change seamlessly between zones Z1 and Z2.

[0108] When the signal processing circuit 88 calculates the weighted average of each pixel in units of 2 × 2 pixels within the boundary region BR, the hardware configuration can be the same as that shown in Fig. 5, and the weighted average of each pixel can be calculated simply by changing the processing content in the signal processing circuit 88. Therefore, the existing analog-to-digital converter 83 can be used as is.

[0109] FIG. 9 shows an example in which a weighted average is calculated by digital signal processing using the pixel values ​​of three pixels surrounding the pixel for which the weighted average is to be calculated. However, the weighted average may also be calculated using the pixel values ​​of two or more pixels adjacent in the row direction and two or more pixels adjacent in the column direction to the pixel for which the weighted average is to be calculated.

[0110] As described above, in the first embodiment, the boundary region BR between zones Z1 and Z2 is divided into pixel regions of the same size as the binning processing of zone Z2, and the pixel values ​​of each pixel in each pixel region are converted into weighted averages by calculating a weighted average between the pixel values ​​of each pixel and the pixel values ​​of other pixels in the pixel region. This allows the pixel values ​​of each pixel in the boundary region BR closer to zone Z1 to approach the pixel values ​​of zone Z1, and closer to zone Z2 to approach the pixel values ​​of zone Z2, thereby eliminating any sense of incongruity in the boundary region BR between zones Z1 and Z2. Furthermore, in the first embodiment, the weighted average is calculated using digital signal processing by the signal processing circuit 88, eliminating the need to change the hardware configuration of the analog-to-digital converter 83, making the first embodiment easier to apply.

[0111] Second Embodiment In a second embodiment, the weighted average of pixels adjacent to each other in a first direction (for example, the row direction) X is calculated by a hardware circuit.

[0112] FIG. 10 is a circuit diagram of an analog-to-digital converter 83 of a photodetector 2004 according to the second embodiment, and FIG. 11 is an enlarged circuit diagram of a part of FIG.

[0113] The analog-to-digital converter 83 according to the second embodiment includes a two-pixel weighted average circuit 12 for each of a plurality of vertical signal lines VSL. The two-pixel weighted average circuit 12 calculates a weighted average between two pixels adjacent to each other in a first direction (e.g., the row direction) X. As shown in detail in FIG. 11 , the two-pixel weighted average circuit 12 includes four capacitors C1, C2, C3, and C4, each with a different capacitance, and a plurality of switches SW1 to SW6. The capacitance ratios of the four capacitors C1 to C4 are, for example, 8:4:2:1, and differ by multiples of two. One end of the capacitor C1, which has the largest capacitance, is connected to the vertical signal line VSL for which the weighted average is to be calculated. Switches SW1, SW3, and SW5 are connected between one end of each of the other three capacitors C2 to C4 and the vertical signal line VSL for which the weighted average is to be calculated, respectively, and switches SW2, SW4, and SW6 are connected between one end of each of these capacitors C2 to C4 and the adjacent vertical signal line VSL, respectively. The other ends of the four capacitors C1 to C4 are all connected to a first input terminal of a comparator 101. The voltage at the first input terminal is a voltage level weighted-averaged by the four capacitors C1 to C4 and the four switches. The multiple switches are switched based on a mixing ratio α.

[0114] A reference voltage output from a reference voltage output unit 100 made up of a DAC is supplied to a second input terminal of the comparator 101 .

[0115] The output voltage of the comparator 101 is input to a digital signal generation circuit 11. The digital signal generation circuit 11 is composed of, for example, a counter 102, a latch 103, and an amplifier circuit 87, as shown in FIG. 5 . The digital pixel signal output from the digital signal generation circuit 11 is input to a signal processing circuit 88.

[0116] The signal processing circuit 88 calculates a weighted average between two pixels adjacent to each other in the second direction (column direction) Y, thereby calculating a weighted average in units of 2×2 pixels.

[0117] In this way, in the second embodiment, the weighted average of each pixel in a 2×2 pixel unit is calculated using the two-pixel weighted average circuit 12 in the analog-to-digital converter 83 and the signal processing circuit 88 .

[0118] FIG. 12 is a diagram showing the correspondence relationship between the mixture ratio of the two-pixel weighted average circuit 12 and the load capacitance of the vertical signal line.

[0119] The two-pixel weighted average circuit 12 calculates a weighted average x'k expressed by the following equation (2) and inputs it to the first input terminal of the comparator 101. xk is the pixel value of the vertical signal line VSL for which the weighted average is to be calculated, and xm is the pixel value of the adjacent vertical signal line VSL.

[0120] 12 shows the correspondence relationship between α / 2 in equation (2), the load capacitance of the vertical signal line VSL for pixel value xk, and the load capacitance of the vertical signal line VSL for pixel value xm. As shown in Fig. 12, the load capacitances of the two vertical signal lines VSL can be switched in multiple ways depending on the value of the mixing ratio α, and the pixel values ​​of the vertical signal line VSL for which the weighted average is to be calculated can be weighted by dividing the capacitance of these load capacitances.

[0121] The signal processing circuit 88 calculates the weighted average x″k expressed by the following equation (3) through digital signal processing. In equation (3), two weighted averages adjacent to each other in the second direction (column direction) Y are calculated.

[0122] By increasing the number of capacitors in the two-pixel weighted average circuit 12, the weighted average can be adjusted more precisely according to the mixture ratio α. For example, to adjust the weighted average in 2n ways, (n+1) capacitors and 2n switches are required. Note that the capacitance ratio of the multiple capacitors does not necessarily have to be a multiple of 2.

[0123] Although Figure 10 shows an example in which the two-pixel weighted average circuit 12 calculates a weighted average between two pixels adjacent in the row direction, the two-pixel weighted average circuit 12 may calculate a weighted average between two pixels adjacent in the column direction, and the downstream signal processing circuit 88 may calculate a weighted average between two pixels adjacent in the row direction.

[0124] FIG. 10 shows an example in which a weighted average is calculated by digital signal processing using the pixel values ​​of three pixels surrounding the pixel for which the weighted average is to be calculated. However, the weighted average may also be calculated using the pixel values ​​of two or more pixels adjacent in the row direction and two or more pixels adjacent in the column direction to the pixel for which the weighted average is to be calculated.

[0125] As described above, in the second embodiment, among the pixel regions that are units for calculating the weighted average in the boundary region BR, the weighted average of pixels adjacent to each other in the first direction (e.g., the row direction) X is calculated by the two-pixel weighted average circuit 12 in the analog-to-digital converter 83, and the weighted average of pixels adjacent to each other in the second direction Y (e.g., the column direction) is calculated by digital signal processing in the downstream signal processing circuit 88. This reduces the processing load on the signal processing circuit 88, and speeds up processing because part of the process for calculating the weighted average is performed by a hardware circuit. Note that it may also be possible to speed up processing by combining analog arithmetic processing and digital arithmetic processing.

[0126] Third Embodiment In a third embodiment, a weighted average is calculated by a hardware circuit in both the first direction (for example, row direction) X and the second direction Y (for example, column direction).

[0127] FIG. 13 is a circuit diagram of an analog-to-digital converter 83 of a photodetector 2004 according to a third embodiment, and FIG. 14 is an enlarged circuit diagram of a portion of FIG. 13. The analog-to-digital converter 83 according to the third embodiment has a four-pixel weighted average circuit 13 for each of a plurality of vertical signal lines VSL. As shown in detail in FIG. 14, the four-pixel weighted average circuit 13 has seven capacitors C1 to C7 and five switches SW1 to SW5. The capacitor C1 and the switch SW1 are connected in series, the capacitor C2 and the switch SW2 are connected in series, the capacitor C3 and the switch SW3 are connected in series, and the capacitor C4 and the switch SW4 are connected in series. One end of each of the switches SW1 to SW4 is connected to the vertical signal line VSL for which the weighted average is to be calculated. The other end of each of the capacitors C1 to C4 is connected to a first input terminal of the corresponding comparator 101.

[0128] One end of each of the capacitors C5 to C7 is connected to a different vertical signal line VSL in the pixel area, and the other end of each of the capacitors C5 to C7 is connected to one end of a switch SW5. The other end of the switch SW5 is connected to a first input terminal of a corresponding comparator 101.

[0129] The capacitance ratio of the capacitors C1 to C4 is, for example, 8:4:2:1. The capacitances of the capacitors C4 to C7 are the same.

[0130] The switches SW1 to SW5 are controlled to switch on or off according to the mixing ratio α. More specifically, a switch control circuit (not shown in FIG. 13 ) controls the switching of the switches SW1 to SW5 according to the mixing ratio α. The mixing ratio α varies depending on the position of the pixel for which the weighted average is to be calculated. As a result, if the pixel for which the weighted average is to be calculated is close to zone 1, the 4-pixel weighted average circuit 13 calculates the weighted average with a greater influence from the pixel values ​​of the pixels in zone 1, and if the pixel for which the weighted average is to be calculated is close to zone 2, the 4-pixel weighted average circuit 13 calculates the weighted average with a greater influence from the pixel values ​​of the pixels in zone 2.

[0131] In Figure 14, the pixel value of the pixel connected to the vertical signal line VSL for which the weighted average is to be calculated is x1, the pixel values ​​of the three pixels on the three surrounding vertical signal lines VSL are x2, x3, and x4, and the weighted average input to the first input terminal of the comparator 101 is x'1.

[0132] The four-pixel weighted average circuit 13 calculates the weighted average x'k expressed by the following equation (4).

[0133] 15 is a diagram showing the correspondence relationship between the mixing ratio of the four-pixel weighted average circuit 13 and the load capacitance of the vertical signal line. Fig. 15 shows α in equation (4), the load capacitance of the vertical signal line VSL of pixel value x1, and the combined load capacitance of the three vertical signal lines VSL of pixel values ​​x2, x3, and x4. As shown in Fig. 15, the load capacitance of the vertical signal line VSL for which the weighted average is calculated changes in 15 different ways depending on the value of the mixing ratio α.

[0134] FIG. 16 is a diagram showing the relationship between the load capacitance of the vertical signal line VSL, for which a weighted average is calculated, and the mixing ratio α. As shown in FIG. 16, as the load capacitance on the pixel value x1 side decreases, the mixing ratio α increases nonlinearly. That is, in the four-pixel weighted average circuit 13 of FIG. 14, the load capacitance of the vertical signal line VSL, for which a weighted average is calculated, cannot be linearly changed depending on the value of the mixing ratio α. Because the mixing ratio α is nonlinear, the degree of freedom in setting the weight is reduced. Furthermore, since the total capacitance seen from the comparator 101 changes, characteristic fluctuations may occur. Furthermore, the four-pixel weighted average circuit 13 of FIG. 14 has a risk of characteristic fluctuations because the load capacitance seen from the comparator 101 varies depending on the connection state of the switches.

[0135] Fig. 17 is a circuit diagram of the main parts of an analog-to-digital converter 83 of a photodetector 2004 according to a modified example of the third embodiment. The analog-to-digital converter 83 in Fig. 17 has a four-pixel weighted average circuit 13 having a different configuration from that in Fig. 13. Fig. 17 shows one four-pixel weighted average circuit 13 for calculating a weighted average. In reality, there are as many four-pixel weighted average circuits 13 similar to Fig. 17 as there are vertical signal lines VSL.

[0136] 17 includes first to third four-pixel weighted average circuits 13a, 13b, and 13c. The first to third four-pixel weighted average circuits 13a, 13b, and 13c have the same circuit configuration, but the types of vertical signal lines VSL for which the weighted average is to be calculated differ from those of the vertical signal lines VSL for which the weighted average is to be calculated. The circuit configuration of the first four-pixel weighted average circuit 13a will be described below.

[0137] The first 4-pixel weighted average circuit 13a includes four capacitors C1 to C4, switches SW1 and SW2 connected to one end of capacitor C1, switches SW3 and SW4 connected to one end of capacitor C2, switches SW5 and SW6 connected to one end of capacitor C3, and switches SW7 and SW8 connected to one end of capacitor C4. The other ends of the switches SW1, SW3, SW5, and SW7 are connected to the vertical signal line VSL for which the weighted average is to be calculated. The other ends of the switches SW2, SW4, and SW6 are connected to the adjacent vertical signal line VSLx2. The capacitance ratio of the capacitors C1 to C4 is 8:4:2:1.

[0138] The second 4-pixel weighted average circuit 13 is connected to the adjacent vertical signal line VSL transmitting the pixel value x3, and the third 4-pixel weighted average circuit 13 is connected to the adjacent vertical signal line VSL transmitting the pixel value x4.

[0139] The three weighted averages calculated by the first to third 4-pixel weighted average circuits 13a, 13b, and 13c are all input to a first input terminal of a comparator 101. The weighted average x'1 at the first input terminal is expressed by the following equation (5).

[0140] FIG. 18 is a diagram showing the correspondence relationship between the mixing ratio α in equation (5), the load capacitance of the vertical signal line VSL of pixel value x1, and the combined load capacitance of the three vertical signal lines VSL of adjacent pixel values ​​x2, x3, and x4.

[0141] Fig. 19 is a plot of the relationship between the load capacitance of the vertical signal line VSL, which is the target for calculating the weighted average in Fig. 18, and the mixing ratio α. In the four-pixel weighted average circuit 13 in Fig. 17, the mixing ratio α changes linearly with changes in the load capacitance. Furthermore, in the four-pixel weighted average circuit 13 in Fig. 17, the load capacitance seen from the comparator 101 is constant, so there is little fluctuation in the characteristics.

[0142] 13 to 18 show an example in which a weighted average is calculated by digital signal processing using the pixel values ​​of three pixels surrounding the pixel for which the weighted average is to be calculated, but the weighted average may also be calculated using the pixel values ​​of two or more pixels adjacent in the row direction and two or more pixels adjacent in the column direction to the pixel for which the weighted average is to be calculated.

[0143] As described above, in the third embodiment, the calculation of the weighted average with other pixels within the pixel region for which the weighted average is to be calculated in the boundary region BR is performed entirely by hardware circuits, which enables the weighted average to be calculated at high speed and reduces the processing load on the signal processing circuit 88. The four-pixel weighted average circuit 13 can have a variety of circuit configurations, and although the four-pixel weighted average circuit 13 of Fig. 17 has a more complex circuit configuration than the four-pixel weighted average circuit 13 of Fig. 13, it can adjust the weighted average linearly in accordance with the value of the mixture ratio α and is less likely to cause characteristic fluctuations.

[0144] Fourth Embodiment In the first to third embodiments described above, examples were shown in which two zones Z1 and Z2 were provided in the pixel section 81, but the number of zones may be arbitrary. Fig. 20 shows an example in which three zones Z1 to Z3 are provided. Zone Z3 is arranged to surround at least a portion of zone Z2. Zone Z1 reads out pixel signals in units of, for example, 1 x 1 pixels, zone Z2 reads out pixel signals by binning processing in units of, for example, 2 x 2 pixels, and zone Z3 reads out pixel signals by binning processing in units of, for example, 4 x 4 pixels.

[0145] A boundary region BR1 is provided between zones Z1 and Z2, and a boundary region BR2 is also provided between zones Z2 and Z3. In these two boundary regions BR1 and BR2, weighted average processing according to any of the first to third embodiments described above is performed. The weighted average processing for the two boundary regions BR1 and BR2 does not necessarily have to be the same. However, because the boundary region BR2 between zones Z2 and Z3 is larger in area than the boundary region BR1 between zones Z1 and Z2, providing a two-pixel weighted average circuit 12 or a four-pixel weighted average circuit 13 made up of hardware circuits could result in excessively large circuit size. Therefore, it is desirable to perform weighted average processing for the boundary region BR between zones Z2 and Z3 using digital signal processing in the downstream signal processing circuit 88. Even when four or more zones are provided, a boundary region BR occurs between two adjacent zones, so a two-pixel weighted average circuit 12 or a four-pixel weighted average circuit 13 is provided, or weighted average processing is performed by digital signal processing using the signal processing circuit 88.

[0146] Fifth Embodiment The mixing ratio α of the boundary region BR between two zones differs depending on whether or not different zones exist on both ends of the row. Specifically, when different zones exist on both ends of the row, the mixing ratio α must be different for adjacent vertical signal lines VSL in the row. On the other hand, when different zones do not exist on both ends of the row, the weighted average can be calculated using the same mixing ratio α for adjacent vertical signals in the row. To provide different mixing ratios α to adjacent vertical signal lines VSL in the row, a switch control circuit must be provided for each two-pixel weighted average circuit 12 or each four-pixel weighted average circuit 13, and each switch control circuit must switch and control each switch in the two-pixel weighted average circuit 12 or each four-pixel weighted average circuit 13 based on the different mixing ratio α.

[0147] 21 is a diagram illustrating the switch control circuit 14. As shown in Fig. 21, in a pixel region in the boundary region BR where different zones exist on both ends in the row direction, a different mixing ratio α needs to be applied to each vertical signal line VSL, and therefore, separate switch control circuits 14 are required.

[0148] 21 shows an example in which nine vertical signal lines VSL are provided within the pixel region described above, nine switch control circuits 14 corresponding to these vertical signal lines VSL, and two-pixel weighted average circuits 12 or four-pixel weighted average circuits 13. Mixing ratios α of 0.1 to 0.9, which differ in increments of 0.1, are input to the nine switch control circuits 14.

[0149] 21, the boundary region BR is usually disposed so as to surround the four sides of the zone Z1. The boundary region BR may be divided into first to fourth boundary regions BR1 to BR4 in accordance with the four sides of the zone Z1, and the weighted average process of any one of the first to fourth embodiments may be performed on each of the first to fourth boundary regions BR1 to BR4.

[0150] 22 is a diagram showing an example of weighted averaging processing for the first to fourth boundary regions BR1 to BR4. In Fig. 22, of the first to fourth boundary regions BR1 to BR4, weighted averaging processing is performed by digital signal processing by the signal processing circuit 88 for the first boundary region BR1 and the second boundary region BR2, which are located at both ends of the row direction of zone Z1, and weighted averages are calculated by the two-pixel weighted averaging circuit 12 or the four-pixel weighted averaging circuit 13 for the third boundary region BR3 and the fourth boundary region BR4, which are located at both ends of the column direction of zone Z1.

[0151] As described above, in FIG. 22, the first boundary region BR1 and the second boundary region BR2 have their additive averages calculated by analog arithmetic processing using the 2-pixel weighted average circuit 12 or the 4-pixel weighted average circuit 13, and the third boundary region BR3 and the fourth boundary region BR4 have their weighted averages calculated by digital signal processing using the signal processing circuit 88. Therefore, the third boundary region BR3 and the fourth boundary region BR4 do not require complex weight control using analog arithmetic processing, and there is no need to provide a switch control circuit 14, enabling the circuit size to be reduced.

[0152] 23 is a diagram showing the value of the mixing ratio α in the boundary region BR. As described above, when different zones exist at both ends of the boundary region BR in the row direction (first boundary region BR1, second boundary region BR2), the mixing ratio α needs to be changed for each vertical signal line VSL, whereas the mixing ratio α can be fixed in the third boundary region BR3 and fourth boundary region BR4, which are at both ends of zone Z1 in the column direction. In boundary regions BR1 and BR2, where the mixing ratio α varies for each vertical signal line, a switch control circuit 14 needs to be provided for each vertical signal line, whereas in boundary regions BR3 and BR4, where the mixing ratio α is fixed, a switch control circuit 14 only needs to be provided for each boundary region.

[0153] Fig. 24 is a diagram showing the connection relationship between the switch control circuit 14 and the first to third four-pixel weighted average circuits 13a, 13b, and 13c in the four-pixel weighted average circuit 13. As shown in Fig. 24, each switch in the first to third four-pixel weighted average circuits 13a, 13b, and 13c can be controlled by a single switch control circuit 14. In other words, there is no need to provide a switch control circuit 14 for each of the first to third four-pixel weighted average circuits 13a, 13b, and 13c.

[0154] 23, however, in the first boundary region BR1 and the second boundary region BR2, a plurality of capacitors and a plurality of switches are required for each of the plurality of vertical signal lines VSL, and a switch control circuit 14 is also required. This increases the circuit size of the analog-to-digital converter 83. The circuit size increases due to (1) switches SW1 to SW8, (2) the switch control circuit 14, and (3) capacitors C1 to C4 shown in FIG.

[0155] 25 is a planar layout diagram of the pixel section 81. The pixel section 81 is provided with HOPB (Horizontal Optical Black) regions 81b arranged on both sides of the aperture pixel region 81a in the row direction, VOPB (Vertical Optical Black) regions 81c arranged on both sides of the aperture pixel region in the column direction, and process dummy regions 81d arranged so as to surround each of the HOPB regions 81b and VOPB regions 81c.

[0156] Although pixels having the same structure as the aperture pixel region 81a are arranged in the HOPB region 81b and the process dummy region 81d, these pixels are not used for imaging or event detection. Therefore, in order to reduce the circuit size, it is desirable to arrange the two-pixel weighted average circuit 12 or the four-pixel weighted average circuit 13 only in the aperture pixel region.

[0157] Figure 26 is a planar layout diagram according to a modification of Figure 25. The human gaze is often detected near the center of the pixel section 81, and zone Z1 is often arranged near the center of the aperture pixel region. Therefore, the two-pixel weighted average circuit 12 or the four-pixel weighted average circuit 13 may be arranged only in the central region 81e of the aperture pixel region. If zone Z1 moves to an area where the two-pixel weighted average circuit 12 or the four-pixel weighted average circuit 13 is not arranged, weighted average processing can be performed by digital signal processing using the signal processing circuit 88.

[0158] Fig. 27 is a diagram showing an example in which the number of 2-pixel weighted average circuits 12 or 4-pixel weighted average circuits 13 is further reduced compared to Fig. 26. In Fig. 27, 2-pixel weighted average circuits 12 or 4-pixel weighted average circuits 13 are provided only on both ends of zone Z1 in the row direction, and weighted average processing is performed on the central side by digital signal processing using signal processing circuit 88. In this case, as explained in Fig. 21, not only 2-pixel weighted average circuits 12 or 4-pixel weighted average circuits 13 but also switch control circuit 14 is required.

[0159] (General Configuration of Information Processing System) The light detection device 2004 according to the present disclosure can be incorporated into a light detection system having a line of sight detection function.

[0160] (Schematic Configuration of Light Detection System) The light detection device 2004 according to the present disclosure can be incorporated into an information processing system 1001 having a gaze detection function.

[0161] Fig. 28 is a simplified block diagram of an information processing system 1001 including the light detection device 2004 according to the present disclosure shown in Fig. 2. The information processing system 1001 in Fig. 28 includes the light detection device 2004, a signal processing device 111, and a gaze detection device 112. The signal processing device 111 and the gaze detection device 112 in Fig. 28 are provided in at least one of the processing circuit 2001, the camera 2003, or the sensor 2008 in Fig. 2.

[0162] Specifically, the signal processing device 111 is configured by an ISP (Image Signal Processor) or an AP (Application Processor).

[0163] The photodetector 2004 has a configuration similar to that shown in Fig. 5. Fig. 28 shows a simplified internal configuration of the photodetector 2004 of Fig. 5, and includes at least a pixel unit 81, a signal processing circuit 88, and an interface circuit (IF) 113. The interface circuit (IF) 113 of Fig. 28 corresponds to the output circuit 89 of Fig. 5.

[0164] The signal processing device 111 includes a frame buffer 114 , a first processing unit (IFE) 115 , and a second processing unit (IPE) 116 .

[0165] The frame buffer 114 stores the image data for each frame output from the photodetector 2004. As will be described later, the photodetector 2004 generates image data for each of a plurality of zones. Thus, the frame buffer 114 stores the image data for each of the plurality of zones.

[0166] The first processing unit 115 performs various image processing on each image data to generate a luminance signal and a color difference signal (YUV signal).

[0167] The second processing unit 116 performs various image processing and image synthesis processing based on a plurality of luminance signals and color difference signals corresponding to a plurality of zones, and generates a synthesized luminance signal and color difference signal (YUV signal).

[0168] The gaze detection device 112 detects the gaze of the user and transmits information about the gaze position (hereinafter also referred to as the gaze position) to the signal processing device 111. More specifically, the signal processing device 111 calculates zone information based on the gaze detection information detected by the gaze detection device 112 and transmits the calculated zone information to the light detection device 2004.

[0169] The photodetector 2004 generates image data by changing the resolution for each zone based on the zone information transmitted from the signal processor 111 .

[0170] (Zone Setting Restrictions) The light detection device 2004 according to the present disclosure sets a plurality of zones in the pixel unit 81 based on the gaze position detected by the gaze detection device 112. In doing so, the light detection device 2004 needs to comply with the zone setting restrictions.

[0171] More specifically, in the photodetector 2004 according to the present disclosure, a region including the line of sight detected by the line of sight detection device 112 at its center position is defined as zone Z1, and a region including zone Z1 is defined as zone Z2. Zone Z1 is a region where photoelectrically converted pixel signals are output pixel by pixel. Zone Z2 is a region where binning processing is performed, in which pixel signals from multiple pixels are combined and output as a pixel signal for a single pixel. For example, if zone Z2 is subjected to 2×2 pixel binning processing, zone Z2 has a size that is a multiple of four of zone Z1.

[0172] The photodetector 2004 according to the present disclosure may have three or more zones. For example, if a zone Z3 is provided that encompasses zone Z2, binning processing is performed in zone Z3 using pixel regions larger in size than zone Z2. For example, if zone Z3 is subjected to 4x4 pixel binning processing, zone Z3 has a size that is a multiple of 16 of zone Z1.

[0173] There is a constraint that zone Z2 must contain the entire area of ​​zone Z1, and zone Z3 must contain the entire area of ​​zone Z2. For example, if all sides of zone Z1 are inside all sides of zone Z2 as in Fig. 29A, or if at least some sides of zone Z1 are in contact with any side of zone Z2 as in Fig. 29B, the zone setting is determined to be valid (OK), but if at least some sides of zone Z1 are located outside any side of zone Z2 as in Fig. 29C, the zone setting is determined to be invalid (NG: error).

[0174] The light detection device 2004 according to the present disclosure needs to set multiple zones based on the gaze position so as to satisfy the above-mentioned zone setting constraints. Since the gaze position detected by the gaze detection device 112 may change for each frame, the light detection device 2004 may change the positions and sizes of the multiple zones for each frame based on the zone information from the signal processing device 111.

[0175] (Zone Setting ID) The photodetector 2004 transmits image data for each zone to the signal processing device 111 using, for example, a virtual channel compliant with the MIPI (Mobile Industry Processor Interface) standard. At this time, the signal processing device 111 needs to know which zone of which frame the image data transmitted from the photodetector 2004 includes. Therefore, the photodetector 2004 according to the present disclosure is provided with a zone setting ID. The zone setting ID is information that identifies the frame to which the read image data belongs. The photodetector 2004 according to the present disclosure adds the zone setting ID to the image data and transmits it to the signal processing device 111.

[0176] The zone setting ID may be assigned by the photodetector 2004 or by the signal processing device 111 .

[0177] 30 is a sequence diagram showing the procedure for transmitting and receiving image data between the photodetector 2004 and the signal processing device 111 when the photodetector 2004 assigns a zone setting ID. When the signal processing device 111 receives gaze detection information from the gaze detection device 112 (step S1), it calculates zone information relating to the position and size of a zone Z1 that includes the gaze position at its center (step S2). The zone information is stored, for example, in a zone setting register, which will be described later.

[0178] The signal processing device 111 transmits the calculated zone information to the photodetector 2004 (step S3). The photodetector 2004 assigns a zone setting ID corresponding to the zone information (step S4). The zone setting ID is associated with a frame from which pixel signals are read out.

[0179] The photodetector 2004 transmits the zone setting ID to the signal processing device 111 (step S5). The photodetector 2004 sets zones Z1 to Z3 in the pixel unit 81, performs an exposure operation, and then reads out pixel signals. For example, in zone Z1, pixel signals are read out for each pixel, whereas in zones Z2 and Z3, pixel signals are read out by performing binning processing for each set of pixels (step S6). After the photodetector 2004 sets the zones, a time delay of N frames (N is an integer equal to or greater than 1) may occur before the photodetector 2004 reads out an image signal reflecting the zone setting.

[0180] The photodetector 2004 transmits a packet in which the corresponding zone setting ID is added to the image data for each frame to the signal processor 111 via a virtual channel conforming to the MIPI standard, for example (step S7).

[0181] The signal processing device 111 compares the zone setting ID attached to the received image data with the zone setting ID received in step S5 to identify which frame the received image data belongs to (step S8).

[0182] Fig. 31 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device 111 of Fig. 30. Fig. 31 shows an example of storing setting information for zones Z1 to Z3 in the zone setting register. The zone setting register stores the two-dimensional coordinates of the first pixel and the last pixel in each of zones Z1 to Z3.

[0183] Fig. 32 is a diagram showing an example of the data structure of image data that the photodetector 2004 transmits to the signal processor 111 in step S7 of Fig. 30. The image data in Fig. 32 is data in which the pixel values ​​(16-bit data) of each pixel are arranged in the order of zones Z3, Z2, and Z1, and this data is followed by 16-bit data representing the zone setting ID.

[0184] 33 is a sequence diagram showing the procedure for transmitting and receiving image data between the light detection device 2004 and the signal processing device 111 when the signal processing device 111 assigns a zone setting ID. When the signal processing device 111 receives gaze detection information from the gaze detection device 112 (step S11), it calculates zone information relating to the position and size of a zone Z1 that includes the gaze position at its center (step S12) and assigns a zone setting ID (step S13). This sequence differs from the sequence diagram of FIG. 30 in that the signal processing device 111 assigns a zone setting ID.

[0185] The signal processing device 111 adds the zone setting ID to the calculated zone information and transmits it to the photodetector 2004 (step S14).

[0186] The photodetector 2004 sets, for example, zones Z1 to Z3 in the pixel section 81, performs an exposure operation, and then reads out pixel signals from each zone. For example, in zone Z1, pixel signals are read out for each pixel, whereas in zones Z2 and Z3, pixel signals are read out by performing binning processing for each set of pixels (step S15).

[0187] The photodetector 2004 transmits a packet in which the corresponding zone setting ID is added to the image data for each frame to the signal processor 111 via a virtual channel conforming to the MIPI standard, for example (step S16).

[0188] The signal processing device 111 compares the zone setting ID attached to the received image data with the zone setting ID assigned by itself in step S13 to identify which frame the received image data is from (step S17).

[0189] Fig. 34 is a diagram showing an example of setting information in a zone setting register provided inside the signal processing device 111 of Fig. 33. The zone setting register of Fig. 34 contains information on the zone setting ID numbered in step S13. The other information is the same as that of Fig. 32.

[0190] The data structure of the image data that the photodetector 2004 transmits to the signal processing device 111 in step S16 is the same as that shown in FIG. 32, and therefore will not be described here.

[0191] (Zone Setting Information) In the above-described FIGS. 30 to 34 , an example was shown in which a zone setting ID was added when image data was transmitted from the photodetector 2004 to the signal processing device 111. However, it is preferable to add information regarding the position and size of each zone (hereinafter, "zone setting information") to the image data in addition to the zone setting ID and transmit the image data from the photodetector 2004 to the signal processing device 111. The reason for this is that the photodetector 2004 can change the position and size of each zone for each frame. Therefore, unless the photodetector 2004 notifies the signal processing device 111 of the position and size of each zone, the signal processing device 111 cannot grasp the position and size of each zone in each frame, resulting in poor utilization of the storage capacity of the frame buffer 114. Therefore, the photodetector 2004 according to the present disclosure adds zone setting information for all zones to the header of image data for each zone transmitted to the signal processing device 111. This allows the signal processing device 111 to grasp the position and size of all zones regardless of which zone's image data it receives, thereby ensuring the storage capacity of the frame buffer 114 for storing the received image data for each zone.

[0192] Fig. 35 is a diagram showing the data structure of image data for each zone transmitted from the photodetector 2004 to the signal processor 111. Fig. 35 shows an example of setting zones Z1 to Z3 in the pixel unit 81. Fig. 35 shows the data structures of zones Z3, Z2, and Z1, from left to right. The image data for each of zones Z1 to Z3 includes a frame start signal (FS), virtual channel information (VC), embedded data (EBD), a packet footer (PF), a data type (DT), and valid pixel data.

[0193] The zone setting information is included in, for example, the embedded data (EBD). As described above, the zone setting information for each of the zones Z1 to Z3 includes information about the positions and sizes of all of the zones Z1 to Z3.

[0194] The zone setting information needs to be updated whenever the position and / or size of a zone changes, and since the position and / or size of a zone may change from frame to frame, the zone setting information may also need to be updated from frame to frame.

[0195] However, the photodetector 2004 continues to use the zone setting information of the immediately preceding frame if there is no notification of zone information update from the signal processing device 111. Since the gaze detection device 112 and the photodetector 2004 usually operate asynchronously, the timing at which the gaze detection device 112 detects gaze movement does not necessarily coincide with the frame switching timing of the photodetector 2004. In general, the gaze detection cycle by the gaze detection device 112 is often shorter than the frame cycle of the photodetector 2004.

[0196] The above-described zone setting information may include only gaze coordinates instead of all coordinates of each zone. The left side of Figure 36 shows an example in which the coordinates of all pixels in each zone are included in the zone setting information. The right side of Figure 36 shows an example in which the sizes of zones Z1 and Z2 are registered in advance, and the center positions of each zone Z1 and Z2 are set as gaze positions, and only the coordinates of the gaze positions are included in the zone setting information. In the example on the right side of Figure 36, the sizes of zones Z1 and Z2 cannot be changed, but the coordinate information for zones Z1 and Z2 included in the zone setting information can be significantly reduced.

[0197] Furthermore, the sizes of zones Z1 and Z2 may be fixed to a predetermined size. The coordinates of zones Z1 and Z2 may be set so that the center positions of zones Z1 and Z2 are the gaze coordinates. In Fig. 28, the gaze information detected by the gaze detection device 112 is input to the signal processing device 111, but the gaze information detected by the gaze detection device 112 may also be input directly to the light detection device 2004.

[0198] The gaze detection device 112 may output gaze coordinates obtained by normalizing the coordinates of the pixel unit 81. Alternatively, the gaze detection device 112 may convert the physical gaze coordinates detected by the gaze detection device 112 into the coordinates of the pixel unit 81 and output the converted coordinates.

[0199] (Prediction of Gaze Coordinates) Gaze information detected by the gaze detection device 112 is input to the photodetector 2004 via the signal processing device 111. The photodetector 2004 performs exposure operations and pixel signal readout operations based on the gaze position detected by the gaze detection device 112, so there is a time lag between when the gaze detection device 112 detects the gaze position and when the photodetector 2004 reads out pixel signals from the zone corresponding to the gaze position. Furthermore, the period in which the gaze detection device 112 detects the gaze is different from the frame period for exposure and pixel signal readout by the photodetector 2004, and the two operate asynchronously.

[0200] FIG. 37 is a diagram showing the time lag between when the gaze detection device 112 detects the gaze position and when the photodetector 2004 reads out pixel signals from a zone corresponding to the gaze position. The vertical lines in FIG. 37 represent the gaze detection cycle and the frame switching timing of the VST sensor. That is, by the time the photodetector 2004 reads out pixel signals, the gaze may have moved to another position. Therefore, the gaze movement during the time lag shown in FIG. 37 may be predicted, and pixel signals may be read out from a zone corresponding to the predicted gaze position. Any algorithm may be used to predict the gaze position; for example, the gaze position may be predicted by linearly interpolating the gaze positions in two or more past frames.

[0201] (Gaze Position in Three-Dimensional Coordinates) Some gaze detection devices 112 output the gaze (gaze) position of a person as coordinates in three-dimensional space (three-dimensional coordinates). In addition, the light detection device 2004 according to the present disclosure may include a VST (Video See-Through) sensor that generates a captured image for the left eye and a VST sensor that generates a captured image for the right eye.

[0202] FIG. 38 is a diagram illustrating the gaze position and coordinate designation of each zone in two VST sensors for the left and right eyes. The photodetector 2004 projects three-dimensional coordinates 118 on a display plane 117 virtually arranged in three-dimensional space corresponding to the gaze (gaze) position output from the gaze detection device 112 onto an image plane 119 of the VST sensor for the left eye and an image plane 120 of the VST sensor for the right eye, respectively, to calculate coordinates corresponding to the gaze position on each image plane. Note that the virtual display plane 117 in FIG. 38 does not necessarily have to coincide with the physical display plane. The coordinates of zones Z1 to Z3 are designated on the image plane for the left eye and the image plane for the right eye, respectively.

[0203] Even when a VST sensor is used, as shown in Fig. 37, a time lag occurs between when the gaze position is detected by gaze detection device 112 and when image data for each zone is read out on each image plane 119, 120. Therefore, as shown in Fig. 37, the movement of the gaze during the time lag may be predicted, and pixel signals may be read out from the zone corresponding to the predicted gaze position.

[0204] Fig. 39 is a diagram showing data transmitted and received between two VST sensors (hereinafter referred to as CIS1(L) and CIS2(R)) that constitute the gaze detection device 112. Fig. 39 shows an example in which the signal processing device 111 is an AP (Application Processor). Fig. 39 shows an example in which CIS1(L) is the primary (main sensor) and CIS2(R) is the secondary (sub-sensor), but CIS2(R) may be the primary and CIS1(L) may be the secondary.

[0205] The AP 111 transmits zone information about the zone Z1, including the gaze position, to the CIS1(L) for the left eye via, for example, an internal integrated circuit (I2C) or an improved inter-integrated circuit (I3C). The zone information includes the position and size (x, y, w, h) of the zone Z1 and offset information (xoffset, yoffset) of the zone Z1 in the CIS2(R). x and y are the two-dimensional coordinates of the image plane of the CIS1(L), w is the width of the zone Z1, and h is the height of the zone Z1.

[0206] CIS1(L) generates zone setting information including the position and size (x, y, w, h) of zone Z1. The zone setting information includes offset information (xoffset, yoffset) of zone Z2 transmitted from AP 111.

[0207] CIS1(L) transmits the generated zone setting information to CIS2(R) for the right eye via, for example, I2C / I3C or SPI (Serial Peripheral Interface). CIS2(R) generates zone setting information including the position and size of zone Z2 (x+xoffset, y+yoffset, w, h). Note that since there is almost no deviation between the left and right lines of sight in the vertical direction of the image plane, yoffset may be omitted.

[0208] CIS1(L) and CIS2(R) each transmit image data to AP111 independently via, for example, MIPI.

[0209] There is a risk of a difference in the timing at which the zone setting information is reflected in CIS1(L) and CIS2(R). Therefore, reflection timing information may be added to the zone setting information transmitted from CIS1(L) to CIS2(R) to synchronize the setting reflection timing between CIS1(L) and CIS2(R). The reflection timing information is information indicating that the zone setting information will be reflected N frames later, assuming that CIS1(L) and CIS2(R) are synchronized. This prevents a problem of the settings being reflected in different frames between the left-eye image and the right-eye image due to a difference in the setting timing of the zone setting information between CIS1(L) and CIS2(R).

[0210] The image data generated by CIS1(L) and the image data generated by CIS2(R) may be concatenated and transmitted to AP 111 as a single image. Fig. 40 is a block diagram showing a case where two images generated by CIS1(L) and CIS2(R) are concatenated and transmitted to AP 111 as a single image. In the example of Fig. 40, the image data output from CIS2(R) is transmitted to CIS1(L) by, for example, MIPI. CIS1(L) concatenates the two image data and transmits the resulting image data to AP 111 via, for example, MIPI.

[0211] 41 is a diagram showing the data structure of image data transmitted by CIS1(L) to AP 111. The image data includes a frame start (FS), a frame header (FH), image data generated by CIS1(L) and image data generated by CIS(R), a frame footer (FF), and a frame end (FE).

[0212] In Figure 41, the image data generated by CIS1 (L) is placed on the higher bit side of the bit string, and the image data generated by CIS2 (R) is placed on the lower bit side of the bit string, but they may also be placed as separate bit strings.

[0213] In the case of Figure 41, AP 111 can acquire two pieces of image data almost simultaneously, so there is no need to synchronize the reception timing of the two pieces of image data or adjust the difference in reception timing, as in the case of Figure 40, and the processing of AP 111 can be simplified.

[0214] (Additional Information on Zone Setting ID) A frame number may be assigned to the frame currently being captured. The frame number is managed using a string of multiple bits. The photodetector 2004 according to the present disclosure may use the lowest N bits of the bit string representing the frame number as a zone setting ID and transmit it to the signal processing device 111.

[0215] Fig. 42 is a diagram showing the exposure and readout timing of the photodetector 2004. The photodetector 2004 transmits the above-mentioned zone setting ID to the signal processing device 111 (e.g., AP 111) using an existing communication interface. Fig. 42 shows an example in which the lowest two bits of the frame number are used as the zone setting ID using, for example, a GPIO (General Purpose Input / Output), which is an existing communication interface. "LL", "LH", "HL", and "HH" in Fig. 42 show examples in which the lowest two bits are "00", "01", "10", and "11", respectively.

[0216] 42, image data exposed within a frame period from time t1 to t2 has the value "0" of the lowest two bits of the frame number added to the image data via the MIPI interface during the period from time t3 to t4, and is transmitted to the AP 111. In this case, the lowest two bits "00" have been transmitted to the AP 111 in advance via GPIO, and therefore the AP 111 understands that the value of the lowest two bits added to the image data is "0," and therefore that the data was exposed within the frame period from time t1 to t2.

[0217] The GPIO is a communication interface that is prepared in advance to enable the photodetector 2004 and the AP 111 to send and receive various information. In this way, the photodetector 2004 can send, for example, the lowest two bits of an existing frame number to the AP 111 using an existing communication interface, so that the zone setting ID can be set and notified without adding new hardware. Note that, although the lowest two bits are shown here as an example, the number of bits is not important.

[0218] (Connecting Multiple CISs to a Single Lane) A plurality of photodetector devices 2004 according to the present disclosure can be connected to a single lane to communicate with the signal processing device 111.

[0219] 43 is a block diagram showing a schematic configuration of an information processing system 1001 in which a plurality of photodetectors 2004 are connected to a single lane 122 and communicate with a signal processing device 111. In FIG. 43, an example in which the signal processing device 111 is an AP 111 is shown.

[0220] In the information processing system 1001 in Fig. 43 , each photodetector 2004 connected to the single lane 122 can transmit and receive various information to and from the AP 111, separately from the other photodetectors 2004. In the single lane 122, packets containing image data from the multiple photodetectors 2004 are combined, for example, in a time-division manner and transmitted. In addition, in the single lane 122, packets containing various information to be sent from the AP 111 to the multiple photodetectors 2004 are transmitted. The header of each packet contains information identifying the destination photodetector 2004 or AP 111.

[0221] <Application Example to Intra-Vivo Information Acquisition System> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

[0222] FIG. 44 is a block diagram showing an example of a schematic configuration of a system for acquiring information from within a patient's body using a capsule endoscope, to which the technology according to the present disclosure (the present technology) can be applied.

[0223] The in-vivo information acquisition system 10001 includes a capsule endoscope 10100 and an external control device 10200 .

[0224] The capsule endoscope 10100 is swallowed by a patient during an examination. The capsule endoscope 10100 has an imaging function and a wireless communication function, and moves through the inside of organs such as the stomach and intestines by peristaltic movement or the like until it is naturally expelled from the patient, sequentially capturing images of the inside of the organs (hereinafter also referred to as in-vivo images) at predetermined intervals, and sequentially wirelessly transmitting information about the in-vivo images to an external control device 10200 outside the body.

[0225] The external control device 10200 comprehensively controls the operation of the in-vivo information acquisition system 10001. The external control device 10200 also receives information about the in-vivo images transmitted from the capsule endoscope 10100, and generates image data for displaying the in-vivo images on a display device (not shown) based on the received information about the in-vivo images.

[0226] In this way, the in-vivo information acquisition system 10001 can obtain in-vivo images of the state inside the patient's body at any time from the time the capsule endoscope 10100 is swallowed until it is expelled.

[0227] The configurations and functions of the capsule endoscope 10100 and the external control device 10200 will be described in more detail.

[0228] The capsule endoscope 10100 has a capsule-shaped housing 10101, which houses a light source unit 10111, an imaging unit 10112, an image processing unit 10113, a wireless communication unit 10114, a power supply unit 10115, a power supply unit 10116, and a control unit 10117.

[0229] The light source unit 10111 is composed of a light source such as an LED (light emitting diode), and irradiates the imaging field of the imaging unit 10112 with light.

[0230] The imaging unit 10112 is composed of an imaging element and an optical system consisting of multiple lenses provided in front of the imaging element. Reflected light (hereinafter referred to as observation light) of light irradiated onto the body tissue to be observed is collected by the optical system and incident on the imaging element. In the imaging unit 10112, the imaging element photoelectrically converts the incident observation light, generating an image signal corresponding to the observation light. The image signal generated by the imaging unit 10112 is provided to the image processing unit 10113.

[0231] The image processing unit 10113 is configured with processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and performs various signal processing on the image signal generated by the imaging unit 10112. The image processing unit 10113 provides the image signal that has been subjected to the signal processing to the wireless communication unit 10114 as RAW data.

[0232] The wireless communication unit 10114 performs predetermined processing such as modulation on the image signal that has been subjected to signal processing by the image processing unit 10113, and transmits the image signal to the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 also receives a control signal related to drive control of the capsule endoscope 10100 from the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 provides the control signal received from the external control device 10200 to the control unit 10117.

[0233] The power supply unit 10115 is composed of an antenna coil for receiving power, a power regeneration circuit that regenerates power from the current generated in the antenna coil, a boost circuit, etc. The power supply unit 10115 generates power using the principle of so-called contactless charging.

[0234] The power supply unit 10116 is configured by a secondary battery and stores the power generated by the power supply unit 10115. In Fig. 44, to avoid cluttering the drawing, arrows and the like indicating the destinations of the power supply unit 10116 are omitted, but the power stored in the power supply unit 10116 is supplied to the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the control unit 10117 and can be used to drive these units.

[0235] The control unit 10117 is composed of a processor such as a CPU, and appropriately controls the operation of the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the power supply unit 10115 in accordance with control signals transmitted from the external control device 10200.

[0236] The external control device 10200 is configured with a processor such as a CPU or a GPU, or a microcomputer or control board equipped with a processor and a storage element such as a memory. The external control device 10200 controls the operation of the capsule endoscope 10100 by transmitting a control signal to the control unit 10117 of the capsule endoscope 10100 via the antenna 10200A. In the capsule endoscope 10100, for example, the light irradiation conditions of the light source unit 10111 for the observation object can be changed by the control signal from the external control device 10200. Furthermore, the imaging conditions (e.g., the frame rate and exposure value of the imaging unit 10112) can be changed by the control signal from the external control device 10200. Furthermore, the control signal from the external control device 10200 can change the content of processing in the image processing unit 10113 and the conditions for transmitting image signals from the wireless communication unit 10114 (e.g., the transmission interval, the number of transmitted images, etc.).

[0237] The external control device 10200 also performs various image processing on the image signal transmitted from the capsule endoscope 10100 to generate image data for displaying the captured in-vivo image on a display device. The image processing can include various signal processing such as development processing (demosaic processing), image quality improvement processing (band enhancement processing, super-resolution processing, NR (Noise Reduction) processing, and / or image stabilization processing), and / or enlargement processing (electronic zoom processing). The external control device 10200 controls the driving of the display device to display the captured in-vivo image based on the generated image data. Alternatively, the external control device 10200 may record the generated image data in a recording device (not shown) or print it out on a printing device (not shown).

[0238] The above describes an example of an in-vivo information acquisition system to which the technology disclosed herein can be applied. The technology disclosed herein can be applied to the imaging unit 10112 of the above-described configuration. By applying the technology disclosed herein to the imaging unit 10112, clearer images of the surgical site can be obtained, thereby improving the accuracy of the examination.

[0239] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor is considered to be circuitry or processing circuitry, including transistors and other circuits. A processor may also be a programmed processor that executes a program stored in a memory.

[0240] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0241] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0242] The present technology may be configured as follows. (1) A photodetection device is provided, including: a pixel unit including a plurality of pixels each including a photoelectric conversion element; and a control unit configured to control a pixel signal of each pixel in a boundary region between a first region including two or more pixels among the plurality of pixels that include a gaze position and a second region that is disposed to surround at least a portion of the first region and has a different resolution from the first region, based on pixel signals of surrounding pixels. (2) The photodetection device described in (1), in which the control unit generates pixel signals that are closer to the pixel signals of the pixels in the first region for pixels in the boundary region that are closer to the first region, and generates pixel signals that are closer to the pixel signals of the pixels in the second region for pixels closer to the second region. (3) The photodetection device described in (1) or (2), in which the control unit controls the pixel signal of each pixel in the boundary region by a weighted average of pixel signals of one or more pixels adjacent in a first direction and pixel signals of one or more pixels adjacent in a second direction intersecting the first direction. (4) The photodetector according to (3), wherein the control unit calculates the weighted average after weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more pixels adjacent in a first direction, and the pixel signals of one or more pixels adjacent in a second direction intersecting the first direction with weights according to the pixel position of each pixel in the boundary region. (5) The photodetector according to (4), wherein the boundary region has a first boundary region and a second boundary region disposed on both ends of the first region in the first direction, and a third boundary region and a fourth boundary region disposed on both ends of the first region in the second direction, and the control unit controls the pixel signal for each pixel in the first to fourth boundary regions using the corresponding weight and surrounding pixels. (6) The photodetector according to (5), wherein the weighted average is calculated for the first boundary region and the second boundary region by analog arithmetic processing, and the weighted average is calculated for the third boundary region and the fourth boundary region by digital signal processing. (7) The photodetector device described in (5), wherein a weighted average is calculated for the first boundary area and the second boundary area by digital signal processing, and a weighted average is calculated for the third boundary area and the fourth boundary area by analog arithmetic processing.(8) The photodetector according to any one of (3) to (7), wherein the control unit has a signal processing circuit that performs digital signal processing to calculate a weighted average of pixel signals of each pixel in the boundary region between pixel signals of one or more pixels adjacent in the first direction and pixel signals of one or more pixels adjacent in the second direction. (9) The photodetector according to any one of (3) to (7), wherein the control unit has: an analog arithmetic processing circuit that calculates a weighted average of pixel signals of each pixel in the boundary region between pixel signals of one or more pixels adjacent in the first direction, and a digital signal processing unit that calculates a weighted average of pixel signals of each pixel weighted averaged by the analog arithmetic processing circuit between pixel signals of one or more pixels adjacent in the second direction weighted averaged by the analog arithmetic processing circuit. (10) The photodetector according to (9), wherein the analog arithmetic processing circuit and the digital signal processing unit calculate the weighted average after weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more pixels adjacent in a first direction, and the pixel signals of one or more pixels adjacent in a second direction intersecting the first direction with weights according to the pixel position of each pixel in the boundary region. (11) The photodetector according to (9) or (10), wherein the weighted average is calculated by the analog arithmetic processing circuit for some pixel regions in the boundary region, and the weighted average is calculated by the digital signal processing unit for pixel regions other than the some pixel regions. (12) The photodetector according to any one of (3) to (7), wherein the control unit has an analog arithmetic processing circuit that calculates a weighted average between the pixel signals of each pixel in the boundary region and the pixel signals of one or more pixels adjacent in the first direction and the pixel signals of one or more pixels adjacent in the second direction. (13) The photodetection device described in (12), wherein the analog arithmetic processing circuit calculates the weighted average after weighting the pixel signals of each pixel in the boundary region, the pixel signals of one or more pixels adjacent in the first direction, and the pixel signals of one or more pixels adjacent in the second direction with weights corresponding to the pixel positions of each pixel in the boundary region.(14) A digital video signal processing circuit includes a plurality of signal lines transmitting pixel signals of a plurality of pixels arranged in the second direction, wherein the analog arithmetic processing circuit includes: a plurality of weighted average circuits provided for each of the plurality of signal lines and calculating the weighted average; and a plurality of switch control circuits provided corresponding to the plurality of weighted average circuits, wherein each of the plurality of weighted average circuits includes: a plurality of first capacitors provided for each corresponding signal line in the boundary region, each having a different capacitance; a plurality of second capacitors provided corresponding to a plurality of pixels adjacent in the first direction and the second direction to a pixel connected to the corresponding signal line, each having the same capacitance; a plurality of first switches that switch whether to charge the plurality of first capacitors with charges corresponding to the pixel signals of the pixel; and a second switch that switches whether to combine the accumulated charges of the plurality of second capacitors and the accumulated charges of the plurality of first capacitors, wherein each of the plurality of switch control circuits controls switching of the plurality of first switches and the second switches in the corresponding weighted average circuit, wherein the weighted average calculated by each of the plurality of weighted average circuits has a nonlinear relationship with the corresponding weight. (13) A photodetector according to (13).(15) The photodetector according to (13), further comprising a plurality of signal lines transmitting pixel signals of a plurality of pixels arranged in the second direction, wherein the analog arithmetic processing circuit comprises: a plurality of weighted average circuits provided for each of the plurality of signal lines and calculating the weighted average; and a plurality of switch control circuits provided corresponding to the plurality of weighted average circuits, wherein each of the plurality of weighted average circuits comprises a plurality of sub-weighted average circuits calculating a weighted average between one pixel connected to a corresponding signal line and a plurality of different pixels arranged around the pixel, wherein each of the plurality of sub-weighted average circuits comprises: a plurality of capacitors; and a plurality of switches for switching whether or not to store charge in the plurality of capacitors according to the pixel value of the one pixel or the pixel value of another pixel arranged around the one pixel, wherein each of the plurality of switch control circuits controls switching of the plurality of switches in the corresponding one of the plurality of sub-weighted average circuits, wherein there is a linear relationship between a weighted average obtained by combining the weighted averages calculated by each of the plurality of sub-weighted average circuits and the corresponding weight. (16) The photodetector according to any one of (12) to (15), wherein the boundary region has a first boundary region and a second boundary region arranged at both ends in the first direction, and a third boundary region and a fourth boundary region arranged at both ends in the second direction, and an arithmetic average is calculated for the first boundary region, the second boundary region, the third boundary region, and the fourth boundary region by the analog arithmetic processing circuit. (17) The photodetector according to (15), comprising: an aperture pixel region having the plurality of pixels; an optical black region arranged to surround the aperture pixel region; and a process dummy region arranged to surround the optical black region, wherein the plurality of weighted average circuits and the switch control circuit are arranged to correspond to only at least a portion of the aperture pixel region. (18) The photodetector according to (17), wherein the plurality of weighted average circuits and the switch control circuit are arranged to correspond to only a partial region including a center position of the aperture pixel region.(19) The light detection device according to any one of (1) to (18), wherein the boundary region includes a first boundary region and a second boundary region, and the control unit controls a pixel signal of each pixel in each of the first boundary region between the first region and the second region and the second boundary region between the second region and a third region that is arranged to surround at least a portion of the second region and has a resolution different from that of the first region and the second region, based on pixel signals of surrounding pixels. (20) An information processing system comprising: a gaze point detection unit that detects a gaze position of a person; a pixel unit that includes a plurality of pixels each including a photoelectric conversion element; a control unit that controls a pixel signal of each pixel in a boundary region between a first region consisting of two or more pixels that includes the gaze position among the plurality of pixels and a second region that is arranged to surround at least a portion of the first region and has a resolution different from that of the first region, based on pixel signals of surrounding pixels; and a display unit that displays an image generated based on the pixel signals of each pixel in the first region, the second region, and the boundary region. (21) An information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit that has a plurality of pixels each including a photoelectric conversion element; a control unit that controls pixel signals of each pixel in a boundary region between a first region consisting of two or more pixels among the plurality of pixels that include the gaze position and a second region that is arranged to surround at least a portion of the first region and has a different resolution from the first region, based on pixel signals of surrounding pixels; and a processing circuit that generates pixel signals of each pixel in the boundary region based on pixel signals of surrounding pixels in the boundary region.(22) An information processing system comprising: a gaze point detection unit that detects a person's gaze position; a pixel unit having a plurality of pixels each including a photoelectric conversion element; a control unit that controls pixel signals of each pixel in a boundary region between a first region consisting of two or more pixels among the plurality of pixels that include the gaze position and a second region that is arranged to surround at least a portion of the first region and has a different resolution from the first region, based on pixel signals of surrounding pixels; and a display unit that combines and displays a first image based on the pixel signals of each pixel in the first region, a second image based on the pixel signals of each pixel in the second region, and a third image based on the pixel signals of each pixel in the boundary region.

[0243] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0244] 1 Light detection device, 4 Pixel weighted average circuit, 11 Digital signal generation circuit, 12 Pixel weighted average circuit, 13 Pixel weighted average circuit, 14 Switch control circuit, 31 Output mechanism unit, 32 Wearing mechanism unit, 33 Wearing band, 34 Housing, 35 Spatial imaging unit, 51 Control unit, 52 Input unit, 53 Output unit, 54 Memory unit, 55 Communication unit, 56 Drive, 57 Removable storage medium, 58 Bus, 59 Gazing point detection unit, 59A Image analysis device, 59AP Application processor, 59S Eyeball imaging sensor, 59SRL Eyeball imaging sensor, 59SRR Eyeball imaging sensor, 59STL Eyeball imaging sensor, 59STR Infrared, 59STR Eyeball imaging sensor, 60 Imaging unit, 61 Display unit, 61P Display image, 61PL Image, 61PR Image, 71 Image analysis unit, 72 display image generation unit, 81 pixel unit, 81a aperture pixel region, 81b HOPB region, 81c VOPB region, 81d process dummy region, 81e central region, 82 vertical scanning circuit, 83 analog-to-digital converter, 84 horizontal transfer scanning circuit, 85 timing control circuit, 86 horizontal transfer line, 87 amplifier circuit, 88 signal processing circuit, 89 output circuit, 90 region control unit, 100 reference voltage output unit, 101 comparator, 102 counter, 103 latch, 110 light detection system, 111 signal processing device, 112 line of sight detection device, 113 interface circuit (IF), 114 frame buffer, 115 first processing unit, 116 second processing unit, 117 three-dimensional space, 118 three-dimensional coordinates, 119 image plane, 120 image plane, 122 single lane, 10001 In-vivo information acquisition system, 10100 capsule endoscope, 10101 housing, 10111 light source unit, 10112 imaging unit, 10113 image processing unit, 10114 wireless communication unit, 10114A antenna, 10115 power supply unit, 10116 power supply unit, 10117 control unit, 10200 external control device, 10200A antenna

Claims

1. A light detection device comprising: a pixel portion including a plurality of pixels each including a photoelectric conversion element; a first region composed of two or more pixels including a fixation position among the plurality of pixels; and a control unit configured to control pixel signals of each pixel in a boundary region between the first region and a second region disposed so as to surround at least a part of the first region and having a resolution different from that of the first region, based on pixel signals of surrounding pixels.

2. The light detection device according to claim 1, wherein the control unit generates a pixel signal approximated by a pixel signal of a pixel in the first region for a pixel closer to the first region among the boundary region, and generates a pixel signal approximated by a pixel signal of a pixel in the second region for a pixel closer to the second region.

3. The light detection device according to claim 1, wherein the control unit controls a pixel signal of each pixel in the boundary region by weighted average of pixel signals of one or more pixels adjacent in a first direction and pixel signals of one or more pixels adjacent in a second direction intersecting the first direction.

4. The light detection device according to claim 3, wherein the control unit weights pixel signals of each pixel in the boundary region, pixel signals of one or more pixels adjacent in the first direction, and pixel signals of one or more pixels adjacent in the second direction intersecting the first direction with weights corresponding to pixel positions of each pixel in the boundary region, and then calculates the weighted average.

5. The boundary region includes a first boundary region and a second boundary region disposed on both end sides in the first direction in the first region, and a third boundary region and a fourth boundary region disposed on both end sides in the second direction in the first region, and the control unit controls pixel signals of each pixel in the first to fourth boundary regions using the corresponding weights and surrounding pixels. The light detection device according to claim 4.

6. In the light detection device according to claim 5, the weighted average is calculated by analog arithmetic processing for the first boundary region and the second boundary region, and the weighted average is calculated by digital signal processing for the third boundary region and the fourth boundary region.

7. In the light detection device according to claim 5, the weighted average is calculated by digital signal processing for the first boundary region and the second boundary region, and the weighted average is calculated by analog arithmetic processing for the third boundary region and the fourth boundary region.

8. The control unit has a signal processing circuit that performs digital signal processing for calculating a weighted average between the pixel signals of each pixel in the boundary region and the pixel signals of one or more adjacent pixels in the first direction and the pixel signals of one or more adjacent pixels in the second direction. The photodetection device according to claim 3.

9. The control unit includes an analog arithmetic processing circuit that calculates a weighted average between the pixel signals of each pixel in the boundary region and the pixel signals of one or more adjacent pixels in the first direction, and a digital signal processing unit that calculates a weighted average between the pixel signals of each pixel weighted-averaged by the analog arithmetic processing circuit and the pixel signals weighted-averaged by the analog arithmetic processing circuit for one or more adjacent pixels in the second direction. The photodetection device according to claim 3.

10. The analog arithmetic processing circuit and the digital signal processing unit calculate the weighted average after weighting the pixel signals of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction that intersects the first direction, with weights corresponding to the pixel positions of each pixel in the boundary region. The photodetection device according to claim 9.

11. For a part of the pixel regions in the boundary region, the weighted average is calculated by the analog arithmetic processing circuit, and for the pixel regions other than the part of the pixel regions, the weighted average is calculated by the digital signal processing unit. The photodetection device according to claim 9.

12. The control unit has an analog arithmetic processing circuit that calculates a weighted average between the pixel signals of each pixel in the boundary region and the pixel signals of one or more adjacent pixels in the first direction and the pixel signals of one or more adjacent pixels in the second direction. The photodetection device according to claim 3.

13. The analog arithmetic processing circuit calculates the weighted average after weighting the pixel signals of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction, with weights corresponding to the pixel positions of each pixel in the boundary region. The photodetection device according to claim 12.

14. The apparatus includes a plurality of signal lines for transmitting pixel signals of a plurality of pixels arranged in the second direction. The analog arithmetic processing circuit includes: a plurality of weighted average circuits provided for each of the plurality of signal lines to calculate the weighted average; and a plurality of switch control circuits provided corresponding to the plurality of weighted average circuits. Each of the plurality of weighted average circuits includes: a plurality of first capacitors provided for each corresponding signal line in the boundary region, each having a different capacitance; a plurality of second capacitors provided corresponding to a plurality of pixels adjacent to the one pixel in the first direction and the second direction and connected to the corresponding signal line, each having the same capacitance; a plurality of first switches for switching whether to charge the plurality of first capacitors with the charge corresponding to the pixel signal of the one pixel; and a second switch for switching whether to combine the stored charge of the plurality of second capacitors and the stored charge of the plurality of first capacitors. Each of the plurality of switch control circuits controls the switching of the plurality of first switches and the second switch in the corresponding weighted average circuit. The weighted average calculated by each of the plurality of weighted average circuits has a non-linear relationship with the corresponding weight. The light detection device according to claim 13.

15. A plurality of signal lines for transmitting pixel signals of a plurality of pixels arranged in the second direction, the analog arithmetic processing circuit includes a plurality of weighted average circuits provided for each of the plurality of signal lines to calculate the weighted average, and a plurality of switch control circuits provided in association with the plurality of weighted average circuits. Each of the plurality of weighted average circuits has a plurality of sub-weighted average circuits for calculating a weighted average between one pixel connected to a corresponding signal line and a plurality of different pixels arranged around this pixel. Each of the plurality of sub-weighted average circuits includes a plurality of capacitors and a plurality of switches for switching whether to accumulate charges corresponding to the pixel value of the one pixel or the pixel value of another pixel arranged around the one pixel in the plurality of capacitors. Each of the plurality of switch control circuits controls the switching of the plurality of switches in the corresponding plurality of sub-weighted average circuits, and the weighted average obtained by synthesizing the weighted averages calculated by each of the plurality of sub-weighted average circuits has a linear relationship with the corresponding weight. The optical detection device according to claim 13.

16. The boundary region has a first boundary region and a second boundary region arranged on both end sides in the first direction, and a third boundary region and a fourth boundary region arranged on both end sides in the second direction. The addition average is calculated by the analog arithmetic processing circuit for the first boundary region, the second boundary region, the third boundary region, and the fourth boundary region. The optical detection device according to claim 12.

17. An optical detection device comprising an aperture pixel region having the plurality of pixels, an optical black region arranged to surround the aperture pixel region, and a process dummy region arranged to surround the optical black region. The plurality of weighted average circuits and the switch control circuits are arranged corresponding to at least a part of the aperture pixel region. The optical detection device according to claim 15.

18. The plurality of weighted average circuits and the switch control circuits are arranged corresponding to only a part of the aperture pixel region including the central position of the aperture pixel region. The optical detection device according to claim 17.

19. The boundary region has a first boundary region and a second boundary region, and the control unit controls the pixel signals of the respective pixels in each of the first boundary region between the first region and the second region and the second boundary region between the third region having a resolution different from that of the first region and the second region and arranged so as to surround at least a part of the second region, based on the pixel signals of the surrounding pixels. The light detection device according to claim 1.

20. A gaze point detection unit that detects a person's gaze position, a pixel unit including a plurality of pixels each including a photoelectric conversion element, among the plurality of pixels, a first region composed of two or more pixels including the gaze position, and a second region having a resolution different from that of the first region and arranged so as to surround at least a part of the first region. A control unit that controls the pixel signals of the respective pixels in the boundary region between the two regions based on the pixel signals of the surrounding pixels, and a display unit that displays an image generated based on the pixel signals of the respective pixels in the first region, the second region, and the boundary region. An information processing system.

21. A gaze point detection unit that detects a person's gaze position, a pixel unit including a plurality of pixels each including a photoelectric conversion element, among the plurality of pixels, a first region composed of two or more pixels including the gaze position, and a second region having a resolution different from that of the first region and arranged so as to surround at least a part of the first region. A control unit that controls the pixel signals of the respective pixels in the boundary region between the two regions based on the pixel signals of the surrounding pixels, and a processing circuit that generates the pixel signals of the respective pixels in the boundary region based on the pixel signals of the pixels surrounding the boundary region. An information processing system.

22. A gaze point detection unit that detects a person's gaze position, a pixel unit including a plurality of pixels each including a photoelectric conversion element, among the plurality of pixels, a first region composed of two or more pixels including the gaze position, and a second region having a resolution different from that of the first region and arranged so as to surround at least a part of the first region. A control unit that controls the pixel signals of the respective pixels in the boundary region between the two regions based on the pixel signals of the surrounding pixels, and a display unit that synthesizes and displays a first image based on the pixel signals of the respective pixels in the first region, a second image based on the pixel signals of the respective pixels in the second region, and a third image based on the pixel signals of the respective pixels in the boundary region. An information processing system.

Images