Light detection device and information processing system

Abstract

[Problem] A light detection device is provided to prevent a sense of disharmony at boundary regions. [Solution] The light detection device according to the invention comprises: a pixel unit including a plurality of pixels, each pixel including a photoelectric conversion element; and a control unit that controls the pixel signals of respective pixels in a boundary region between a first region and a second region based on pixel signals of surrounding pixels, the first region being composed of two or more pixels among the plurality of pixels including a gaze position, the second region being arranged to surround at least a portion of the first region and having a resolution different from that of the first region.

Description

Technical Field

[0001] This disclosure relates to optical detection equipment and information processing systems. Background Technology

[0002] In wearable displays such as head-mounted displays, techniques are known to achieve augmented reality or virtual reality by capturing images of real space in the field of view according to the user's gaze and processing the captured images according to the real space.

[0003] For example, techniques are known that include a higher resolution in the area of ​​focus where the eyes of a user wearing a head-mounted display are focused than in other areas (see Patent Document 1).

[0004] Reference List

[0005] Patent documents

[0006] Patent document 1: WO 2019 / 171522 A Summary of the Invention

[0007] The problem to be solved by the present invention

[0008] Because the region of interest (ROI) differs in resolution from other regions, the resolution varies discontinuously in the boundary regions between them, resulting in images with a sense of incongruity. Specifically, the greater the resolution difference between the ROI and other regions, the greater the degree of incongruity is likely to be.

[0009] Therefore, this disclosure provides a light detection device and an information processing system to prevent a sense of disharmony in boundary areas.

[0010] Solution to the problem

[0011] To address the aforementioned issues, the optical detection device of the present invention includes: A pixel unit includes multiple pixels, each pixel including a photoelectric conversion element; and The control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels among the plurality of pixels, including the gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region.

[0012] The control unit can generate a pixel signal that is closer to a pixel in the first region relative to a pixel on the side closer to the first region in the boundary region, and can generate a pixel signal that is closer to a pixel in the second region relative to a pixel on the side closer to the second region.

[0013] The control unit can control the pixel signal of each pixel in the boundary region by weighted average of the pixel signals of one or more pixels adjacent in the first direction and the pixel signals of one or more pixels adjacent in the second direction intersecting the first direction.

[0014] The control unit can calculate a weighted average value by weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction intersecting the first direction, based on the pixel position of each pixel in the boundary region.

[0015] The boundary region may include: A first boundary region and a second boundary region, wherein the first boundary region is arranged on two end sides in the first direction; and The third and fourth boundary regions are arranged on the two end sides in the second direction within the first region, and The control unit can use the corresponding weights and peripheral pixels to control the pixel signal of each pixel in the first to fourth boundary regions.

[0016] In the first and second boundary regions, the weighted average can be calculated through analog processing, and in the third and fourth boundary regions, the weighted average can be calculated through digital signal processing.

[0017] In the first and second boundary regions, the weighted average can be calculated by digital signal processing, while in the third and fourth boundary regions, the weighted average can be calculated by analog processing.

[0018] The control unit may include signal processing circuitry that performs digital signal processing relative to the pixel signal of each pixel in the boundary region, calculating a weighted average between the pixel signals of one or more adjacent pixels in a first direction and the pixel signals of one or more adjacent pixels in a second direction.

[0019] The control unit may include: The analog processing circuit calculates a weighted average of the pixel signal of each pixel in the boundary region and the pixel signals of one or more adjacent pixels in a first direction; and The digital signal processing unit calculates a weighted average between the pixel signal of each pixel, which has been weighted by analog processing circuitry, and the pixel signal weighted average of one or more adjacent pixels in the second direction, which has also been weighted by analog processing circuitry.

[0020] The analog processing circuit and the digital signal processing unit can calculate a weighted average value by weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction intersecting the first direction, according to the pixel position of each pixel in the boundary region.

[0021] The analog processing circuit can calculate the weighted average value in some pixel regions of the boundary region, and the digital signal processing unit can calculate the weighted average value in some pixel regions outside the pixel region.

[0022] The control unit may include analog processing circuitry that, relative to the pixel signal of each pixel in the boundary region, calculates a weighted average between the pixel signals of one or more adjacent pixels in a first direction and the pixel signals of one or more adjacent pixels in a second direction.

[0023] The analog processing circuit can calculate the weighted average value by weighting the pixel signal of each pixel in the boundary region, the pixel signal of one or more neighboring pixels in the first direction, and the pixel signal of one or more neighboring pixels in the second direction according to the weight of the pixel position of each pixel in the boundary region.

[0024] It may further include multiple signal lines for transmitting pixel signals of multiple pixels arranged in the second direction. Analog processing circuits may include: Multiple weighted averaging circuits are configured in each of the multiple signal lines and calculate the weighted average; and Multiple switching control circuits are configured in conjunction with multiple weighted averaging circuits. Each of the multiple weighted average circuits may include: Multiple first capacitors are disposed in each corresponding signal line in the boundary region and have different capacitances; Multiple second capacitors are provided, corresponding to multiple pixel settings, wherein the multiple pixels are adjacent to a pixel connected to a corresponding signal line in a first direction and a second direction, and have the same capacitance; Multiple first switches toggle between charging multiple first capacitors using the charge from a pixel signal of a pixel; and The second switch toggles between combining the stored charges of the multiple second capacitors with the stored charges of the multiple first capacitors. Each switch in the multiple switch control circuit can control the switching of multiple first and second switches in the corresponding weighted average circuit, and The weighted average and the corresponding weights calculated in each of the multiple weighted average circuits can have a non-linear relationship.

[0025] It may further include multiple signal lines for transmitting pixel signals of multiple pixels arranged in the second direction. Analog processing circuits may include: Multiple weighted averaging circuits are configured in each of the multiple signal lines and calculate the weighted average; and Multiple switching control circuits are configured in conjunction with multiple weighted averaging circuits. Each of the multiple weighted averaging circuits may include multiple sub-weighted averaging circuits, which calculate a weighted average between a pixel connected to a corresponding signal line and multiple different pixels arranged around that pixel. Each of the multiple sub-weighted averaging circuits may include: Multiple capacitors; and Multiple switches toggle between storing charge in the multiple capacitors based on the pixel value of a pixel or the pixel value of another pixel surrounding a pixel. Each switch in a multi-switch control circuit can control the switching of multiple switches in corresponding sub-weighted averaging circuits, and The weighted average obtained by combining the weighted averages calculated by multiple sub-weighted average circuits can have a linear relationship with the corresponding weights.

[0026] The boundary region may include a first boundary region and a second boundary region disposed on two end sides in the first direction, and a third boundary region and a fourth boundary region disposed on two end sides in the second direction.

[0027] In the first boundary region, the second boundary region, the third boundary region, and the fourth boundary region, the additive average can be calculated by analog processing circuit.

[0028] It may further include: an aperture pixel area, comprising multiple pixels; The optical black area is arranged to surround the aperture pixel area; and It may further include: processing a virtual region, configured to surround an optically black region, and Multiple weighted averaging circuits and switching control circuits can be arranged to correspond to at least a portion of the aperture pixel region only.

[0029] Multiple weighted averaging circuits and switching control circuits can be configured to correspond only to a portion of the area including the center position of the aperture pixel region.

[0030] The boundary region may include a first boundary region and a second boundary region, and

[0031] The control unit can control the pixel signal of each pixel in each boundary region of the first boundary region between the first region and the second region and the second boundary region between the third region and the second region based on the pixel signal of the surrounding pixels. The third region is arranged to surround at least a portion of the second region and has a resolution different from that of the first region and the second region.

[0032] According to this disclosure, an information processing system is provided, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; A control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels, including the gaze position, among a plurality of pixels. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. The display unit displays an image generated based on the pixel signals of each pixel in the first region, the second region, and the boundary region.

[0033] According to this disclosure, an information processing system is provided, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; A control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels, including the gaze position, among a plurality of pixels. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. The processing circuit generates a pixel signal for each pixel in the boundary region based on the pixel signals of the surrounding pixels of the boundary region.

[0034] According to this disclosure, an information processing system is provided, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; A control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels, including the gaze position, among a plurality of pixels. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. The display unit combines and displays a first image based on pixel signals of each pixel in a first region, a second image based on pixel signals of each pixel in a second region, and a third image based on pixel signals of each pixel in a boundary region. Attached Figure Description

[0035] Figure 1A This is an illustration of an example used to describe the appearance of the information processing system of this disclosure.

[0036] Figure 1B This is a diagram illustrating a variation of the appearance of the information processing system described in this disclosure.

[0037] Figure 2 This is a block diagram illustrating a schematic configuration of the information processing system of this disclosure.

[0038] Figure 3 This is a diagram showing an example of the configuration of the gaze detection unit of an HMD.

[0039] Figure 4 This is a diagram illustrating a configuration example of the hardware as a gaze detection unit and a spatial image acquisition unit.

[0040] Figure 5 This is a block diagram showing the detailed configuration of the image picking unit.

[0041] Figure 6 This is a diagram showing an example of multiple zones set up by a zone control unit.

[0042] Figure 7 It is a block diagram used to describe the functionality of HMD.

[0043] Figure 8 It is a diagram used to describe the boundary region.

[0044] Figure 9 It shows Figure 5 A block diagram of a portion of the internal configuration of an analog-to-digital converter.

[0045] Figure 10 This is a circuit diagram of the analog-to-digital converter of the optical detection device according to the second embodiment.

[0046] Figure 11 It's enlarged. Figure 10 A portion of the circuit diagram.

[0047] Figure 12 This is a diagram showing the relationship between the mixing rate of a two-pixel weighted averaging circuit and the load capacitance of the vertical signal line.

[0048] Figure 13 This is a circuit diagram of the analog-to-digital converter of the optical detection device according to the third embodiment.

[0049] Figure 14 It's enlarged. Figure 13 A portion of the circuit diagram.

[0050] Figure 15 This is a diagram showing the relationship between the mixing rate of a four-pixel weighted averaging circuit and the load capacitance of the vertical signal line.

[0051] Figure 16 This is a graph showing the relationship between the load capacitance of the vertical signal line and the mixing rate for calculating its weighted average.

[0052] Figure 17 This is a circuit diagram of the main part of the analog-to-digital converter of the optical detection device according to a variation of the third embodiment.

[0053] Figure 18 This is a diagram showing the relationship between the mixing rate of the vertical signal line and the load capacitance.

[0054] Figure 19 The drawing requires calculation. Figure 18 A graph showing the relationship between the load capacitance of the vertical signal line with the weighted average value and the mixing rate.

[0055] Figure 20 This is a diagram showing an example of setting up three zones.

[0056] Figure 21 It is a diagram used to describe a switch control circuit.

[0057] Figure 22 This is a diagram illustrating an example of weighted average processing for the first to fourth boundary regions.

[0058] Figure 23 This is a graph showing the values ​​of the mixing rate in the boundary region.

[0059] Figure 24 This is a diagram showing the connection relationship between the switch control circuit in the four-pixel weighted averaging circuit and the first to third four-pixel weighted averaging circuits.

[0060] Figure 25 It is a planar layout diagram of pixel units.

[0061] Figure 26 It is based on Figure 25 A plan view of a modified example.

[0062] Figure 27 It is shown that... Figure 26 Illustrations comparing examples of reducing dual-pixel weighted averaging circuitry or quad-pixel weighted averaging circuitry.

[0063] Figure 28 It is a block diagram showing a general configuration of a light detection system including the light detection device according to this disclosure.

[0064] Figure 29 A, Figure 29 B and Figure 29 C is a diagram showing instances where the region constraints are OK and NG.

[0065] Figure 30 This is a sequence diagram illustrating the steps of sending and receiving image data between the optical detection device and the signal processing device when an ID is set in the allocation area of ​​the optical detection device.

[0066] Figure 31 It shows the setting Figure 30 A diagram illustrating an example of the settings information of the region setting register inside a signal processing device.

[0067] Figure 32 This is a diagram illustrating an example of a data structure for image data sent from a light detection device to a signal processing device.

[0068] Figure 33 This is a sequence diagram illustrating the steps of sending and receiving image data between a light detection device and a signal processing device when an ID is set in the allocation area of ​​the signal processing device.

[0069] Figure 34 It shows the setting Figure 33 A diagram illustrating an example of the settings information of the region setting register inside a signal processing device.

[0070] Figure 35 It is a diagram showing the data structure of image data sent from the light detection device to the signal processing device for each region.

[0071] Figure 36 This is a diagram showing the regional settings information.

[0072] Figure 37 This is a diagram showing the time delay from when the gaze detection device detects the gaze position to when the light detection device reads out the pixel signal from the area corresponding to the gaze position.

[0073] Figure 38It is a diagram used to describe the gaze position and coordinates of each region in the two VST sensors for the left and right eyes.

[0074] Figure 39 This is a diagram illustrating the data transmitted and received between two VST sensors that constitute a line-of-sight detection device.

[0075] Figure 40 This is a block diagram illustrating the scenario where two images generated by a VST sensor are connected as a single image and transmitted to an AP.

[0076] Figure 41 This is a diagram showing the data structure of image data sent from CIS1(L) to AP.

[0077] Figure 42 It is a diagram showing the exposure and readout timing of a light detection device.

[0078] Figure 43 This is a block diagram illustrating a general configuration of an optical detection system that connects multiple optical detection devices to a single channel and communicates with a signal processing device.

[0079] Figure 44 This is a block diagram illustrating an example of a schematic configuration of an in vivo information acquisition system. Detailed Implementation

[0080] In the following description, embodiments of the optical detection apparatus and information processing system according to the present disclosure will be described with reference to the accompanying drawings. The main components of the optical detection apparatus and information processing system according to the present disclosure will be described primarily below; however, the optical detection apparatus may have components and functions not shown or described. The following description is not intended to exclude components or functions not shown or described.

[0081] Figure 1A This is a diagram illustrating an example of the appearance of the information processing system 1001 of this disclosure. Figure 1B This is a modified diagram used to describe the appearance of the information processing system 1001 of this disclosure, and Figure 2 This is a block diagram illustrating a schematic configuration of the information processing system 1001 of this disclosure. For example... Figure 1A As shown, the information processing system 1001 according to this embodiment is configured as a head-mounted display (HMD). Referring to... Figure 1A An example of the appearance of the head-mounted display (HMD) of this embodiment is described.

[0082] In this example, HMD 1001 includes an output mechanism unit 1011 and a mounting mechanism unit 1012. The mounting mechanism unit 1012 includes a mounting strap 1013 worn by the user to wrap around the head and secure the device. Note that it does not need to wrap around the head as long as it is secured to the head.

[0083] The output mechanism unit 1011 includes a housing 1014, which has a shape that covers the left and right eyes when the user is wearing the HMD 1001, and includes a display panel inside to face the eyes when worn. Within the housing 1014, a further configuration is provided for the display panel (display unit 2005) located when the HMD 1001 is worn. Figure 2 A lens is placed between the user's eyes and the viewer's eyes, increasing the user's field of vision. A stereoscopic image corresponding to the parallax between the two eyes can be displayed in each area obtained by dividing the display panel into right and left, and stereoscopic vision can be achieved through this display.

[0084] In HMD 1001, the speaker or headphones may be further positioned at a location corresponding to the user's ear when worn. In this example, in HMD 1001, camera 1015 is disposed on the front surface of housing 1014, and the surrounding real space is imaged as a moving image within the user's field of vision corresponding to their line of sight. In this specification, camera 1015 may be referred to as light detection unit 1015.

[0085] Camera 1015 is, for example, a light detection unit and includes light detection devices (such as an image sensor, such as a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor or a range sensor) and an optical system, such as an imaging lens. For example, in Figure 1A In this system, a stereo camera is configured to capture images of the space in front of the user from left and right viewpoints corresponding to the user's left and right eyes. Note that the camera 1015 is not limited to this and can be a monocular camera or three or more multi-view cameras. Furthermore, combinations of various types of sensors can be used. In hand-tracking applications, the camera 1015 can be provided to capture images of the space below the information processing system. In eye-tracking or face-tracking applications, the camera 1015 can be provided to image the user's eyes or face.

[0086] The HMD 1001 also includes sensor 2008 ( Figure 2 The sensor group may include at least one of various sensors used to derive the motion, attitude, position, etc. of the HMD 1001, such as an accelerometer, a gyroscope, an angular velocity sensor, and a geomagnetic sensor.

[0087] The HMD 1001 can be connected to another processing device wirelessly or via a wired connection such as Universal Serial Bus (USB).

[0088] In this scenario, the HMD 1001 can drive online applications such as games in which multiple users can participate via a network. In this case, the HMD 1001 performs predetermined processing on images captured by the camera 1015 and generates and displays images within the field of view of the camera 1015.

[0089] Here, the content of the displayed image is not particularly restricted and can vary according to the functions required by the user system, the content of the activated application, etc.

[0090] For example, HMD 1001 can perform some processing on images captured by camera 1015 or can overlay and draw virtual objects that interact with real objects. Alternatively, HMD 1001 can draw a virtual world in a field of view corresponding to the user's field of view based on captured images, measurements from motion sensors included in the sensor group of HMD 1001, etc.

[0091] Representative examples of these include virtual reality (VR), augmented reality (AR), and mixed reality (MR). Furthermore, by directly using the image captured by the camera (light detection unit) 1015 as the display image in itself, it is possible to achieve a perspective form of the real world (video perspective (VST)) through the screen of the HMD 1001.

[0092] Figure 1B The information processing system 1101 shown in this embodiment is configured as a glass-type HMD.

[0093] The HMD body 1111 is used by wearing it on the user's head. The HMD body 1111 includes a front part 1112, a right temple 1113 located on the right side of the front part 1112, a left temple 1114 located on the left side of the front part 1112, and a glass part 1115 attached to the lower side of the front part 1112. Note that in Figure 1B In this case, the glasses are a single unit, but can have two separate lenses for each eye, or they can cover only one eye.

[0094] Display unit 1103 is a perspective-type display unit and is disposed on the surface of glass portion 1115. Display unit 1103 performs AR display of virtual objects under the control of processing circuit (control unit) 2001. Note that display unit 1103 can also be a non-perspective-type display unit. In this case, an image in which the virtual object is superimposed on the image currently captured by camera 1104 is displayed on display unit 2005, thereby performing AR display.

[0095] The camera 1104 is, for example, a light detection unit, and includes a light detection device such as an image sensor or a distance measurement sensor such as a CCD sensor or a CMOS sensor, and an optical system such as a camera lens. The camera 1104 is provided on the outer side of the outer surface of the front portion 1112, captures an object in the real space, and outputs image information obtained by imaging to the processing circuit (control unit) 2001. In Figure 1B for example, two cameras 1104 are provided in the front portion 1112 at a predetermined interval in the lateral direction. Note that the camera (light detection unit) 1015 is not limited to this, and may be a monocular camera or three or more multi-view cameras. In addition, a combination of multiple types of sensors may be used. In the application of hand tracking, the camera 1104 may be provided to capture an image of the space below the information processing system. In the application of eye tracking or face tracking, the camera 1104 may be provided to image the user's eyes or face.

[0096] The glass-type HMD 1101 also includes a sensor 2008 ( Figure 2 ). The sensor unit may include at least one of various sensors for deriving the motion, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0097] In addition, the glass-type HMD 1101 may include a communication IF 2009 ( Figure 2 ). Communicate with a smart phone or an external device other than a smart phone (for example, a personal computer (PC), a server device on a network, etc.) in a wired or wireless manner.

[0098] <Hardware Configuration Example of HMD>

[0099] Next, a hardware configuration example of the information processing system (HMD 1001 or glass-type HMD 1101) will be described with reference to Figure 2 As Figure 2 shown, the hardware of the information processing system includes a CPU 2001, a memory 2002, a camera 2003, a display unit 2005, an input unit 2006, an output unit 2007, a sensor 2008, a communication interface (IF) 2009, an external network 2010, and a secondary storage device 2011, which are connected to each other via a bus 2012 and are capable of sending and receiving data and programs.

[0100] The processing circuit 2001 operates based on a program stored in the memory 2002 or secondary storage device 2011, and controls all operations of the information processing systems 1001 and 1101. The processing circuit, for example, is a processor that reads from the memory 2002 and executes each program to implement the function corresponding to each read program. The processor may include any one or more of, for example, a multi-core processor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or an equivalent discrete logic circuit, or an integrated logic circuit. The processing circuit can be implemented by dividing it into multiple chips.

[0101] The memory 2020 can be implemented, for example, by semiconductor memory elements such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory, hard disk, optical disk, etc., and can include any form of memory for storing data and executing software instructions.

[0102] Camera 2003 corresponds to Figure 1A Camera 1015 and Figure 1B The camera 1104 includes a light detection device (such as an image sensor or distance measurement sensor, such as a CCD sensor or a CMOS sensor) and an optical system (such as an imaging lens). The camera 2003 includes the light detection device 2004.

[0103] The display unit 2005 is a display panel disposed inside the housing, and includes a display device including a liquid crystal display (LCD), an organic electroluminescent display (EL), etc.

[0104] Although not shown in the figure. Figure 1A and Figure 1B The input unit 2006 includes input devices such as a keyboard, mouse, touch panel, microphone, or controller for user input operation commands, and provides various input signals to the processing circuit (control unit) 2001.

[0105] The output unit 2007 includes audio output devices, such as speakers, tactile presentation devices, and odor presentation devices, which are controlled by the processing circuit 2001 and output the processing results as speech, touch, or odor.

[0106] Sensor 2008 may include at least one of various sensors for deriving motion, attitude, position, etc., of HMDs 1001 and 1101, such as an accelerometer, gyroscope, angular velocity sensor, and geomagnetic sensor. Additionally, it may include a biosensor for sensing human biometrics or a pressure sensor for sensing input.

[0107] Communication interface 2009 is the interface through which information processing systems 1001 and 1101 connect to external network 2010. For example, processing circuit 2001 receives data from another device or sends data generated by processing circuit 1100 to another device via communication interface 2009.

[0108] Examples of information processing systems to which the technology according to this disclosure can be applied have been described above. The technology according to this disclosure can be applied to the optical detection device 2004 in the above configuration.

[0109] The light detection device 2004 according to this disclosure implements the camera 1015 in FIG1, including a CCD sensor and a CMOS sensor, and outputs the captured image to the image analysis unit and the display image generation unit described later in the processing circuit 2001.

[0110] Based on the gaze point information provided by the gaze point detection unit described later, the light detection device 2004 considers the gaze point region, which is a range near the gaze point location in the image to be captured, as the region of interest that the user is concerned with, and changes the resolution in the region of interest and the region outside the region of interest.

[0111] It should be noted that in this specification, as shown in FIG1, the light detection device 2004 will be described as including a stereo camera that captures images of the space in front of the user from left and right viewpoints corresponding to the user's left and right eyes of the HMD 1001. Furthermore, unless otherwise specified, the camera 1015 and the light detection device 2004 will be described as having essentially the same structure.

[0112] The light detection device 2004 of this embodiment controls the pixel signals of each pixel in the boundary region of a first region (e.g., region Z1) containing two or more pixels at the gaze position and a second region (e.g., region Z2) configured to surround at least a portion of the first region and having a different resolution than the first region, based on the pixel signals of the surrounding pixels.

[0113] Figure 2 The display unit 2005 can display an image generated based on the pixel signals of each pixel in the first region, the second region, and the boundary region.

[0114] also, Figure 2 The processing circuit 2001 can generate a pixel signal for each pixel in the boundary region based on the pixel signals of the pixels surrounding the boundary region.

[0115] and, Figure 2The display unit 2005 can be combined to display a first image based on the pixel signal of each pixel in the first region, a second image based on the pixel signal of each pixel in the second region, and a third image based on the pixel signal of each pixel in the boundary region.

[0116] As described above, by reducing the resolution of the image output from the light detection device 2004, it is possible to display an image that will not cause discomfort to the user, and power consumption can be reduced except for the area of ​​interest containing the gaze position.

[0117] <Configuration Example of a Fove Point Detection Unit>

[0118] The following text will primarily describe the use of Figure 1A This invention relates to an example of the HMD 1001, but also to other HMD 1001 devices. Figure 1B HMD1011. Figure 3 and Figure 4 This is a diagram used to describe the configuration of HMD 1001 according to this disclosure. Figure 3 and Figure 4 The gaze detection unit 59 shown in the figure is set in Figure 2 In at least one of the processing circuit 2001 and the sensor 2008. Figure 3 This is a diagram illustrating an example of the configuration of the gaze detection unit 59 of the HMD 1001. Figure 4 This is a block diagram illustrating the connection relationship between the gaze detection unit 59, the application processor (hereinafter, AP) 59AP, and the light detection device 2004 according to an embodiment. In this specification, the light detection device 2004 of the embodiment is sometimes referred to as the image pickup unit 60.

[0119] The lower part of the accompanying figure schematically shows the HMD 1001 viewed from above and the head F of the user wearing the HMD 1001. As shown in the upper part, display image 61P is displayed on the HMD 1001, and the user views the display image using their left eye EL and right eye ER. Display image 61P includes images 61PL and 61PR viewed with the left eye EL and right eye ER, respectively. In this example, as the gaze detection unit 59, infrared LEDs 59STL and 59STR, eye imaging sensors 59SRL and 59SRR including an infrared camera or position-sensitive detector (PSD) sensor, and image analysis device 59A are provided. In this specification, the gaze detection unit 59 may be referred to as a gaze detection device.

[0120] Infrared LEDs 59STL and 59STR illuminate the user's left eye (EL) and right eye (ER) with infrared light, respectively. Eye imaging sensors 59SRL and 59SRR capture images of the user's left eye (EL) and right eye (ER), respectively, and provide the data to image analysis device 59A.

[0121] Image analysis device 59A specifies the position of infrared reflection in the cornea and the position of the pupil based on images captured by the left and right eyes, and specifies the user's gaze based on the positional relationship. This technology, as a corneal reflection method, has already been put into practical use in the field of gaze detection technology.

[0122] It should be noted that the methods for detecting gaze are not limited to this. For example, any general method, such as using a visible light camera to image the left and right eyes and specifying the gaze based on the positional relationship between the inner corner of the eye and the iris, can be used.

[0123] The image analysis device 59A sets the intersection points RPL and RPR between the gaze detected by the eye imaging sensors 59SRL and 59SRR and the display unit 61 (display panel) that displays the image 61P as the user's gaze point, detects its position coordinates, and provides the position coordinates to the image acquisition unit 60.

[0124] like Figure 4 As shown, the gaze detection unit 59 includes eye imaging sensors 59S (59STL and 59STR) for imaging the eyeball and an application processor 59AP that essentially functions as an image analysis device 59A, which acquires and detects the gaze point from the captured image of the eyeball.

[0125] Based on the gaze point information provided by the application processor 59AP, the image acquisition unit 60 sets the resolution of the region other than the region near the gaze point (the gaze point region) (i.e., the region other than the region of interest) to low and captures the image.

[0126] In this configuration, the architecture of the Mobile Industry Processor Interface (MIPI) can be used to transfer captured images from the eye imaging sensors 59S (59STL and 59STR) to the application processor 59AP.

[0127] Furthermore, the architecture of internal integrated circuits (i2C) or i3C can be used to transmit gaze-related information from the application processor 59AP to the image pickup unit 60.

[0128] It should be noted that the gaze detection unit 59 may have a configuration in which an eye imaging sensor 59S (59STL and 59STR) is stacked and integrated with an application processor 59AP.

[0129] <Configuration Example of Image Picking Unit>

[0130] Next, a configuration example of the image picking unit 60 will be described. Figure 5 This is a block diagram showing the detailed configuration of the image pickup unit 60. The image pickup unit 60 includes a pixel unit 81, a vertical scanning circuit 82, an analog-to-digital converter (ADC) group 83, a horizontal transfer scanning circuit 84, a timing control circuit 85, a horizontal transfer line 86, an amplifier circuit 87, a signal processing circuit 88, an output circuit 89, and a region control unit 90.

[0131] Pixel unit 81 is configured such that pixels, including photodiodes that perform photoelectric conversion, are arranged in a matrix. Each pixel retains a signal charge according to the amount of incident light. Vertical scanning circuit 82 drives pixels row by row by providing drive pulses to each pixel via pixel drive wiring (not shown). Therefore, analog signals of the pixels in that row are provided to ADC group 83 via vertical signal lines provided for each column.

[0132] The ADC group 83 is configured such that, as an analog-to-digital conversion circuit, the ADC includes: a comparator 101 that compares a reference voltage generated by a reference voltage output unit 100 with an analog signal obtained from a pixel via a vertical signal line; a counter 102 that counts the comparison time; and a latch 103 that holds the count result for each pixel column.

[0133] The analog signal read from the vertical signal line is compared with a reference voltage having a slope waveform by comparator 101, and a count value is obtained when the two match to be converted into a digital signal. The output of each latch 103 is connected to the horizontal transfer line 86.

[0134] The horizontal transfer scan circuit 84 includes, for example, a shift register, and sequentially outputs horizontal scan pulses to input a row of digital signals stored in the latch 103 to the amplifier circuit 87 and the signal processing circuit 88 via the horizontal transfer line 86.

[0135] The timing control circuit 85 generates and outputs clock and control signals based on the vertical synchronization signal, the horizontal synchronization signal, and the master clock, which serve as references for the operation of the vertical scanning circuit 82, the ADC group 83, and the horizontal transfer scanning circuit 84.

[0136] The signal processing circuit 88 performs predetermined processing on the input digital signal of each pixel, which serves as the readout processing circuit, such as defect correction, demosaic processing, and gamma correction, to generate data for the captured image.

[0137] The output circuit 89 appropriately buffers the data of the captured image generated by the signal processing circuit 88 and outputs the data for each line.

[0138] The region control unit 90 sets multiple regions in the pixel unit 81 based on the gaze point information provided by the gaze point detection unit 59. Figure 6 This is a diagram showing an example of multiple zones set up by the zone control unit 90. Figure 6 An example of the region control unit 90 setting regions Z1 and Z2 is shown. While the number of regions set by the region control unit 90 is not limited, the following description primarily focuses on an example of the region control unit 90 setting regions Z1 and Z2. Region Z1 is a pixel region including the gaze point location and is a high-resolution region that generates pixel signals in 1×1 pixel units. Region Z2 is a pixel region surrounding at least a portion of region Z1 and is a region used to perform merging processing, for example, generating pixel signals in 2×2 pixel units. Region Z2 is a pixel region with a lower resolution than region Z1.

[0139] More specifically, the region control unit 90, based on the gaze point information provided by the gaze point detection unit 59, specifies the pixel position of the coordinates corresponding to the gaze point in the image captured by the image picking unit 60, and, with the specified pixel position corresponding to the gaze point as a reference, specifies a rectangular gaze point region with a predetermined range in the horizontal and vertical directions as the region of interest.

[0140] It should be noted that an example of using an integrated configuration as the ADC constituting ADC group 83 has been described above, but other configurations may be used, and for example, a successive approximation ADC may be used.

[0141] <Functions implemented by the HMD in Figure 1>

[0142] Next, we will describe the... Figure 2 The functions implemented by HMD 1001. Figure 7 This is a block diagram used to describe the functionality of the HMD 1001. For example... Figure 7 As shown, the HMD 1001 enables the gaze detection unit 59, image pickup unit 60, image analysis unit 71, display image generation unit 72, and display unit 61 to function as follows: Figure 3 The diagram illustrates its functional operation, enabling imaging of real space, analysis of captured images, and processing based on the analysis results to generate and display the displayed images. Figure 7 The gaze detection unit 59, image analysis unit 71, and display image generation unit 72 are configured in Figure 2 In at least one of the processing circuit 2001 or the sensor 2008.

[0143] That is, the gaze detection unit 59 detects the gaze of the user wearing the HMD 1001 and outputs the detection result to the image acquisition unit 60 as gaze information.

[0144] Based on the gaze point information, the image acquisition unit 60 regards the gaze point region as the region of interest that the user is concerned about. The resolution of the pixels in the region of interest is set to high and the resolution of the pixels in the region outside the region of interest (the gaze point region) is set to low. The captured image is then output to the image analysis unit 71 and the display image generation unit 72.

[0145] For example, based on the image captured by the image picking unit 60, the image analysis unit 71 performs analysis processing such as face detection, face recognition, gesture recognition, visual tracking, position detection, or obtaining the pose of the HMD 1001 through visual simultaneous localization and mapping (SLAM), and outputs the analysis results to the display image generation unit 72.

[0146] The display image generation unit 72 performs predetermined processing on the image captured by the image picking unit 60 based on the analysis results provided by the image analysis unit 71 to generate a display image, and outputs the display image to the display unit 61 for display.

[0147] By operating as described above, based on the gaze point information detected by the gaze point detection unit 59, the image acquisition unit 60 treats the gaze point region as a region of interest that the user is focused on, sets the resolution to high, and captures the image while setting the resolution of regions other than the region of interest to low. Therefore, the amount of image data captured by the image acquisition unit 60 is effectively reduced, and the processing load in the analysis processing in the image analysis unit 71 and the display image generation processing in the display image generation unit 72 is reduced. As a result, delays associated with a series of processes can be suppressed.

[0148] The optical detection device 2004 disclosed herein can be applied to Figure 5 The image acquisition unit 60 is included. Furthermore, the light detection device 2004 according to this disclosure can also be applied to event-based vision sensors (EVS) for detecting events. Representative embodiments of the light detection device 2004 according to this disclosure will be described below.

[0149] (First Implementation)

[0150] The light detection device 2004 of the first embodiment has a viewing area and other areas in the pixel unit 81, and generates pixel signals through photoelectric conversion with different resolutions. In the following text, the viewing area is referred to as area Z1 or the first area, and the area other than the viewing area is referred to as area Z2 or the second area.

[0151] The light detection device 2004 of the first embodiment has a control unit. The control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels, each including a photoelectric conversion element, that include a gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region.

[0152] For example, the control unit is Figure 5 The control unit generates a pixel signal closer to a pixel in the first region relative to a pixel on the side of the boundary region closer to the first region, and a pixel signal closer to a pixel in the second region relative to a pixel on the side of the second region closer to the second region. The control unit controls the pixel signal of each pixel in the boundary region by a weighted average of the pixel signals of one or more pixels adjacent to each other in a first direction (e.g., row direction) X and the pixel signals of one or more pixels adjacent to each other in a second direction (e.g., column direction) Y intersecting the first direction X. The control unit calculates the weighted average by weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more pixels adjacent to each other in the first direction X, and the pixel signals of one or more pixels adjacent to each other in the second direction Y intersecting the first direction X, using weights corresponding to the pixel position of each pixel in the boundary region.

[0153] In the following text, the first region will be referred to as region Z1, and the second region will be referred to as region Z2.

[0154] Region Z1 is a region with a higher resolution than region Z2. In region Z1, for example, exposure and readout of the pixel signal are performed for each pixel. In region Z2, merging is performed on a multi-pixel basis, and the pixel signal is read out for each of the multiple pixels.

[0155] The boundary region is set between region Z1 and region Z2. Figure 8 This is a diagram used to describe the boundary region BR. Figure 8 A portion of the pixel region in pixel unit 81 is shown. Figure 8 An example is shown where a pixel signal is output for each pixel in region Z1, and a merging process is performed in region Z2 in units of 2×2 pixels represented by a single-dotted line box to output a single pixel signal. Figure 8 An example is shown where a boundary region BR, corresponding to four pixels, is set between regions Z1 and Z2, but the number of pixels in the boundary region BR is arbitrary.

[0156] Figure 8The dashed box in the diagram represents a 2×2 pixel region within the boundary region BR, and the four pixel values ​​within the dashed box are (x1, x2, x3, x4). In this embodiment, multiple pixels in the boundary region BR are divided into 2×2 pixel regions of the same size as region Z2, and the pixel values ​​(x1, x2, x3, x4) of each pixel in each pixel region are weighted and averaged using a blending rate α to generate pixel values ​​(x'1, x'2, x'3, x'4). The blending rate α is a coefficient that changes according to the distance between regions Z1 and Z2. In this specification, the blending rate α may be referred to as a weight. For example, the blending rate α decreases as it gets closer to region Z1 and increases as it gets closer to region Z2.

[0157] Figure 9 It is shown Figure 5 A block diagram of a portion of the internal configuration of the analog-to-digital converter 83. Figure 9 In this context, the analog-to-digital converter 83 is designated as a column analog-to-digital converter (ADC). For example... Figure 9 As shown, multiple pixels arranged in the second direction Y are connected to each of multiple vertical signal lines VSLs that extend in the second direction (column direction) Y and are arranged at predetermined intervals in the first direction (row direction) X. Pixel signals output from two adjacent pixels in the second direction Y are input to an analog-to-digital converter 83 via a shared vertical signal line VSL. The analog-to-digital converter 83 has a comparator 101 for each vertical signal line. Each comparator 101 compares the multiple pixel signals output from the multiple pixels via each of the multiple vertical signal lines VSLs with a reference voltage and outputs the comparison result.

[0158] The output voltage of comparator 101 is input to digital signal generation circuit 11. For example, as... Figure 5 As shown, the digital signal generation circuit 11 includes a counter 102, a latch 103, an amplifier circuit 87, etc. The digital pixel signal output from the digital signal generation circuit 11 is input to the signal processing circuit 88.

[0159] The signal processing circuit 88 performs a weighted average calculation based on multiple digital pixel signals (which correspond to multiple pixel signals output from multiple pixels included in the boundary region BR) through digital signal processing.

[0160] More specifically, the signal processing circuit 88 divides the multiple pixels included in the boundary region BR into 2×2 pixel units and calculates the weighted average of each pixel in 2×2 pixel units using the mixing rate α. In the pixel values ​​(x1, x2, x3, x4) of the 2×2 pixels, the weighted average x'1 of pixel value x1 is represented by the following (1).

[0161] [Mathematical Formula 1]

[0162] The other pixel values ​​(x2, x3, x4) of the 2×2 pixel are calculated in a similar manner to formula (1).

[0163] The signal processing circuit 88 divides each pixel in the boundary region BR into pixel regions of the same size as the merged processing region Z2, and converts the pixel value of each pixel into a weighted average of the pixel values ​​of other pixels in that pixel region. At this time, since the weighted average is calculated considering the mixing rate α, the weighted average is calculated after the influence of the pixel value in region Z1 further increases for pixels close to region Z1, and the influence of the pixel value in region Z2 further increases for pixels close to region Z2. Therefore, the pixel values ​​of each pixel in the boundary region BR can be changed without noticeable inconsistencies between regions Z1 and Z2.

[0164] When the signal processing circuit 88 calculates the weighted average of each pixel in the boundary region BR in units of 2×2 pixels, the hardware configuration can be compared with that in Figure 5 The hardware configuration is the same, and the weighted average value for each pixel can be calculated only by changing the processing content in the signal processing circuit 88. Therefore, the existing analog-to-digital converter 83 can be used as is.

[0165] Figure 9 An example is shown where a weighted average is calculated using the pixel values ​​of the three pixels surrounding the pixel to be weighted. However, a weighted average can be calculated using the pixel values ​​of two or more pixels adjacent in the row direction and two or more pixels adjacent in the column direction of the pixel to be weighted.

[0166] As described above, in Embodiment 1, the boundary region BR between regions Z1 and Z2 is divided into pixel regions of the same size as the merged processing area in region Z2. A weighted average of the pixel value of each pixel in each pixel region and the pixel values ​​of other pixels in that pixel region is calculated, and this pixel value is converted into the weighted average. Therefore, the pixel values ​​of each pixel in the boundary region BR can be made closer to the pixel values ​​of the region Z1 adjacent to region Z1, and the pixel values ​​of each pixel in the boundary region BR can be made closer to the pixel values ​​of the region Z2 adjacent to region Z2, thereby eliminating the sense of inconsistency in the boundary region BR between regions Z1 and Z2. Furthermore, in the first embodiment, since the weighted average is calculated through digital signal processing by the signal processing circuit 88, there is no need to change the hardware configuration of the analog-to-digital converter 83, and the application of the first embodiment becomes easier.

[0167] (Second Implementation)

[0168] In the second embodiment, a weighted average value among pixels that are adjacent to each other in the first direction (e.g., the row direction) X is calculated by hardware circuitry.

[0169] Figure 10 This is a circuit diagram showing the analog-to-digital converter 83 of the light detection device 2004 according to the second embodiment. Figure 11 It is magnification Figure 10 A portion of the circuit diagram.

[0170] The analog-to-digital converter 83 according to the second embodiment includes a two-pixel weighted averaging circuit 12 for each of a plurality of vertical signal lines VSL. The two-pixel weighted averaging circuit 12 calculates the weighted average of two pixels that are adjacent to each other in a first direction (e.g., row direction) X. Figure 11 As shown in detail, the dual-pixel weighted averaging circuit 12 includes four capacitors C1, C2, C3, and C4 with different capacitances, and multiple switches SW1 to SW6. The capacitance ratio of the four capacitors C1 to C4 is, for example, 8:4:2:1, and the difference is a multiple of 2. One end of the capacitor C1, which has the largest capacitance, is connected to the vertical signal line VSL for which the weighted average is to be calculated. Switches SW1, SW3, and SW5 are respectively connected between one end of the other three capacitors C2 to C4 and the vertical signal line VSL for which the weighted average is to be calculated, and switches SW2, SW4, and SW6 are respectively connected between one end of capacitors C2 to C4 and the adjacent vertical signal line VSL. The other ends of the four capacitors C1 to C4 are all connected to the first input terminal of comparator 101. The voltage at the first input terminal is the voltage level of the four capacitors C1 to C4 and the four switches weighted average. Multiple switches are switched based on the mixing rate α.

[0171] The reference voltage output from the reference voltage output unit 100, which includes the DAC, is supplied to the second input terminal of the comparator 101.

[0172] The output voltage of comparator 101 is input to digital signal generation circuit 11. For example, as... Figure 5 As shown, the digital signal generation circuit 11 includes a counter 102, a latch 103, an amplifier circuit 87, etc. The digital pixel signal output from the digital signal generation circuit 11 is input to the signal processing circuit 88.

[0173] The signal processing circuit 88 calculates the weighted average between two adjacent pixels in the second direction (column direction) Y, thereby calculating the weighted average in units of 2×2 pixels.

[0174] As described above, in the second embodiment, the two-pixel weighted averaging circuit 12 and the signal processing circuit 88 in the analog-to-digital converter 83 are used to calculate the weighted average value of each pixel in units of 2×2 pixels.

[0175] Figure 12 This is a diagram showing the relationship between the mixing rate of the two-pixel weighted averaging circuit 12 and the load capacitance of the vertical signal line.

[0176] The dual-pixel weighted averaging circuit 12 calculates the weighted average value x'k as expressed by the following formula (2) and inputs the weighted average value x'k to the first input terminal of the comparator 101. xk is the pixel value of the vertical signal line VSL for which the weighted average value is to be calculated, and xm is the pixel value of the adjacent vertical signal line VSL.

[0177] [Mathematical Formula 2]

[0178] Figure 12 The diagram illustrates the correspondence between α / 2, the load capacitance of the vertical signal line VSL with pixel value xk, and the load capacitance of the vertical signal line VSL with pixel value xm in formula (2). Figure 12 As shown, the load capacitance of the two vertical signal lines VSL can be switched in multiple ways by the value of the mixing rate α, and the pixel values ​​of the vertical signal lines VSL to be weighted can be averaged by dividing the capacitance of these load capacitances.

[0179] The signal processing circuit 88 calculates the weighted average value x"k represented by the following formula (3) through digital signal processing. In formula (3), the weighted average values ​​of two adjacent weighted average values ​​in the second direction (column direction) Y are calculated.

[0180] [Mathematical Formula 3]

[0181] By increasing the number of capacitors in the two-pixel weighted averaging circuit 12, the weighted average can be adjusted more finely according to the mixing rate α. For example, in the case of adjusting the weighted average in 2n ways, (n+1) capacitors and 2n switches are needed. Note that the capacitance ratio of the multiple capacitors does not have to be a multiple of two.

[0182] although Figure 10An example of a two-pixel weighted averaging circuit 12 calculating a weighted average between two adjacent pixels in the row direction is shown. However, the two-pixel weighted averaging circuit 12 can calculate a weighted average between two adjacent pixels in the column direction, and the subsequent signal processing circuit 88 can calculate a weighted average between two adjacent pixels in the row direction.

[0183] Figure 10 An example is shown in which the weighted average is calculated by digital signal processing using the pixel values ​​of the three pixels surrounding the pixel to which the weighted average is to be calculated. However, the weighted average can be calculated using the pixel values ​​of two or more pixels adjacent in the row direction and two or more pixels adjacent in the column direction of the pixel to which the weighted average is to be calculated.

[0184] As described above, in the second embodiment, for pixels adjacent to each other in the first direction (e.g., row direction) X within the pixel region used as a unit for calculating the weighted average in the boundary region BR, the two-pixel weighted averaging circuit 12 in the analog-to-digital converter 83 calculates the weighted average, and the subsequent signal processing circuit 88 performs digital signal processing to calculate the weighted average of pixels adjacent to each other in the second direction Y (e.g., column direction). Therefore, the processing load on the signal processing circuit 88 can be reduced, and since a portion of the weighted average calculation is performed by hardware circuitry, the processing can be accelerated. Note that combining analog and digital processing can improve processing speed.

[0185] (Third implementation method)

[0186] In the third embodiment, the hardware circuit calculates a weighted average value in a first direction (e.g., row direction) X and a second direction (e.g., column direction).

[0187] Figure 13 This is a circuit diagram of the analog-to-digital converter 83 of the optical detection device 2004 according to the third embodiment. Figure 14 It's enlarged. Figure 13 A portion of the circuit diagram. Yes. The analog-to-digital converter 83 according to the third embodiment includes a four-pixel weighted averaging circuit 13 for each of multiple vertical signal lines VSL. As Figure 14 As shown in detail, the four-pixel weighted averaging circuit 13 includes seven capacitors C1 to C7 and five switches SW1 to SW5. Capacitor C1 and switch SW1 are connected in series, capacitor C2 and switch SW2 are connected in series, capacitor C3 and switch SW3 are connected in series, and capacitor C4 and switch SW4 are connected in series. One end of each of switches SW1 to SW4 is connected to the vertical signal line VSL for which the weighted average is to be calculated. The other end of capacitors C1 to C4 is connected to the first input terminal of the corresponding comparator 101.

[0188] One end of capacitors C5 to C7 is connected to different vertical signal lines VSL in the pixel area. The other end of capacitors C5 to C7 is connected to one end of switch SW5. The other end of switch SW5 is connected to the first input terminal of the corresponding comparator 101.

[0189] The capacitance ratio of capacitors C1 to C4 is, for example, 8:4:2:1. The capacitances of capacitors C4 to C7 are the same.

[0190] Switches SW1 to SW5 are controlled to be on or off according to the mixing rate α. More specifically, the switch control circuit ( Figure 13 (Not shown) Switching control of switches SW1 to SW5 is executed according to the mixing rate α. The mixing rate α varies depending on the location of the pixel for which the weighted average is to be calculated. Therefore, if the pixel for which the weighted average is to be calculated is near region 1, the four-pixel weighted averaging circuit 13 calculates the weighted average when the pixel value of the pixel in region 1 has a greater influence; if the pixel for which the weighted average is to be calculated is near region 2, the four-pixel weighted averaging circuit calculates the weighted average when the pixel value of the pixel in region 2 has a greater influence.

[0191] exist Figure 14 In the calculation, the pixel value of the pixel connected to the vertical signal line VSL is set to x1, the pixel values ​​of the three pixels on the three surrounding vertical signal lines VSL are set to x2, x3 and x4, and the weighted average input to the first input of comparator 101 is set to x'1.

[0192] The four-pixel weighted average circuit 13 calculates the weighted average value x'k as expressed by the following formula (4).

[0193] [Mathematical Formula 4]

[0194] Figure 15 This is a diagram showing the relationship between the mixing rate of the four-pixel weighted averaging circuit 13 and the load capacitance of the vertical signal line. Figure 15 The diagram shows α in formula (4), the load capacitance of the vertical signal line VSL with pixel value x1, and the load capacitance obtained by combining three vertical signal lines VSL with pixel values ​​x2, x3, and x4. Figure 15 As shown, the load capacitance of the vertical signal line VSL for which the weighted average is to be calculated varies in 15 ways depending on the value of the mixing rate α.

[0195] Figure 16 This is a graph showing the relationship between the load capacitance of the vertical signal line VSL for which the weighted average is to be calculated and the mixing rate α. (See diagram below.) Figure 16As shown, as the load capacitance on the pixel value x1 side decreases, the mixing rate α increases non-linearly. That is, in Figure 14 In the four-pixel weighted averaging circuit 13, it is impossible to linearly change the load capacitance of the vertical signal line VSL, which is used to calculate the weighted average value through the mixing rate α. Since the mixing rate α is non-linear, the degrees of freedom for weight setting are reduced. Furthermore, characteristic fluctuations may occur due to the change in total capacitance observed from comparator 101. Additionally, in Figure 14 In the four-pixel weighted averaging circuit 13, there is a risk of characteristic fluctuations because the load capacitance viewed from the comparator 101 fluctuates according to the connection state of the switch.

[0196] Figure 17 This is a circuit diagram showing the main part of the analog-to-digital converter 83 of the optical detection device 2004, a modified example of the third embodiment. Figure 17 The analog-to-digital converter 83 in the middle includes having the same characteristics as in the middle. Figure 13 The four-pixel weighted averaging circuit 13 has different configurations. Figure 17 A four-pixel weighted averaging circuit 13 is shown to calculate its weighted average. In fact, there are as many as the vertical signal line VSL with... Figure 17 The four-pixel weighted average circuit 13 is similar to the four-pixel weighted average circuit 13 in the text.

[0197] Figure 17 The four-pixel weighted averaging circuit 13 includes first to third four-pixel weighted averaging circuits 13a, 13b, and 13c. The first to third four-pixel weighted averaging circuits 13a, 13b, and 13c have identical circuit configurations, but the types of the vertical signal lines VSL used to calculate the weighted average differ between them. The circuit configuration of the first four-pixel weighted averaging circuit 13a will be described below.

[0198] The first four-pixel weighted averaging circuit 13a includes four capacitors C1 to C4, switches SW1 and SW2 connected to one end of capacitor C1, switches SW3 and SW4 connected to one end of capacitor C2, switches SW5 and SW6 connected to one end of capacitor C3, and switches SW7 and SW8 connected to one end of capacitor C4. The other ends of switches SW1, SW3, SW5, and SW7 are connected to the vertical signal line VSL to which the weighted average is to be calculated. The other ends of switches SW2, SW4, and SW6 are connected to the adjacent vertical signal line VSLx2. The capacitance ratio of capacitors C1 to C4 is 8:4:2:1.

[0199] The second four-pixel weighted averaging circuit 13 is connected to the adjacent vertical signal line VSL that transmits pixel value x3, and the third four-pixel weighted averaging circuit 13 is connected to the adjacent vertical signal line VSL that transmits pixel value x4.

[0200] All three weighted averages calculated by the first to third four-pixel weighted averaging circuits 13a, 13b and 13c are input to the first input terminal of comparator 101. The weighted average value x'1 at the first input terminal is represented by the following formula (5).

[0201] [Mathematical Formula 5]

[0202] Figure 18 This is a diagram showing the correspondence between the mixing rate α of formula (5), the load capacitance of the vertical signal line VSL with pixel value x1, and the load capacitance obtained by combining three vertical signal lines VSL with adjacent pixel values ​​x2, x3, and x4.

[0203] Figure 19 The drawing requires calculation. Figure 18 A graph showing the relationship between the load capacitance of the vertical signal line VSL (weighted average) and the mixing rate α. Figure 17 In the four-pixel weighted averaging circuit 13, the mixing rate α varies linearly with the load capacitance. Furthermore, in... Figure 17 In the four-pixel weighted averaging circuit 13, the characteristic fluctuation is small because the load capacitance viewed from the comparator 101 is constant.

[0204] Figures 13 to 18 An example is shown where a weighted average is calculated using the pixel values ​​of the three pixels surrounding the pixel to be weighted. However, a weighted average can be calculated using the pixel values ​​of two or more pixels adjacent in the row direction and two or more pixels adjacent in the column direction of the pixel to be weighted.

[0205] As described above, in the third embodiment, since the calculation of the weighted average of all pixels in the pixel region where the weighted average is calculated in the boundary region BR is performed entirely by the hardware circuit, the weighted average can be calculated at high speed, and the processing load of the signal processing circuit 88 can be reduced. Although the four-pixel weighted averaging circuit 13 can have multiple circuit configurations, it is similar to... Figure 13 Compared to the four-pixel weighted averaging circuit 13 in the middle, Figure 17 The four-pixel weighted averaging circuit 13 has a complex circuit configuration, but the weighted average can be linearly adjusted according to the value of the mixing rate α, and almost no characteristic fluctuations occur.

[0206] (Fourth Implementation)

[0207] In the first to third embodiments described above, an example of setting two regions Z1 and Z2 in pixel unit 81 has been described, but the number of regions is arbitrary. Figure 20 An example of setting up three regions Z1 to Z3 is shown. Region Z3 is set to surround at least a portion of region Z2. In region Z1, pixel signals are read out in units of 1×1 pixels, for example; in region Z2, pixel signals are read out through merging processing, for example; in region Z3, pixel signals are read out through merging processing, for example; in region Z3, pixel signals are read out through merging processing, for example; in region Z3, pixel signals are read out through merging processing, for example; in region Z3, pixel signals are read out through merging processing.

[0208] Boundary region BR1 is located between region Z1 and region Z2, and boundary region BR2 is located between region Z2 and region Z3. In these two boundary regions BR1 and BR2, weighted averaging processing according to any one of the first to third embodiments described above is performed. The weighted averaging processing of the two boundary regions BR1 and BR2 is not necessarily the same. However, because the area of ​​boundary region BR2 between region Z2 and region Z3 is larger than the area of ​​boundary region BR1 between region Z1 and region Z2, the circuit size may be too large when a two-pixel weighted averaging circuit 12 or a four-pixel weighted averaging circuit 13 including hardware circuitry is provided. Therefore, it is desirable that the boundary region BR between region Z2 and Z3 undergoes weighted averaging processing via digital signal processing in a subsequent stage in the signal processing circuit 88. Even when four or more regions are provided, boundary regions BR are generated between two adjacent regions. Therefore, a two-pixel weighted averaging circuit 12 or a four-pixel weighted averaging circuit 13 is provided, or weighted averaging processing is performed via digital signal processing in the signal processing circuit 88.

[0209] (Fifth Implementation)

[0210] In the boundary region BR between two regions, the mixing rate α differs depending on whether there are different regions on the two ends in the row direction. Specifically, when there are different regions on the two ends in the row direction, the mixing rate α between adjacent vertical signal lines VSL in the row direction must be different. On the other hand, when there are no different regions on the two ends in the row direction, the same mixing rate α can be used to calculate the weighted average for adjacent vertical signals in the row direction. In order to give different mixing rates α to multiple adjacent vertical signal lines VSL in the row direction, it is necessary to provide a switch control circuit for each of the two-pixel weighted averaging circuit 12 or the four-pixel weighted averaging circuit 13, and each switch control circuit needs to perform switching control of each switch in the two-pixel weighted averaging circuit 12 or the four-pixel weighted averaging circuit 13 based on different mixing rates α.

[0211] Figure 21This is a diagram used to describe the switch control circuit 14. For example... Figure 21 As shown, in the pixel regions with different regions on the two ends in the row direction in the boundary region BR, different mixing rates α need to be provided to the vertical signal line VSL. Therefore, a separate switch control circuit 14 is required.

[0212] Figure 21 An example is shown where nine vertical signal lines VSL, nine switch control circuits 14 corresponding to these vertical signal lines VSL, and a two-pixel weighted averaging circuit 12 or a four-pixel weighted averaging circuit 13 are arranged in the aforementioned pixel region. Mixing rates α = 0.1 to 0.9 with different mixing rates α of 0.1 are input to the nine switch control circuits 14.

[0213] like Figure 21 As shown, the boundary region BR is typically arranged around the four sides of region Z1. The boundary region BR can be divided into a first boundary region BR1 to a fourth boundary region BR4 according to the four sides of region Z1, and a weighted average processing of any one of the first to fourth embodiments can be performed for each of the first boundary region BR1 to the fourth boundary region BR4.

[0214] Figure 22 This is a diagram illustrating an example of weighted average processing of the first boundary region BR1 to the fourth boundary region BR4. Figure 22 In the first boundary region BR1 to the fourth boundary region BR4, in the first boundary region BR1 and the second boundary region BR2 on the two ends of the row direction of region Z1, weighted average processing is performed by digital signal processing of signal processing circuit 88, and in the third boundary region BR3 and the fourth boundary region BR4 on the two ends of the column direction of region Z1, the weighted average value is calculated by two-pixel weighted average circuit 12 or four-pixel weighted average circuit 13.

[0215] As mentioned above, in Figure 22 In the first boundary region BR1 and the second boundary region BR2, the additive average is calculated by analog processing through the two-pixel weighted average circuit 12 or the four-pixel weighted average circuit 13, and the weighted average is calculated by digital signal processing through the signal processing circuit 88 in the third boundary region BR3 and the fourth boundary region BR4. In the third boundary region BR3 and the fourth boundary region BR4, the complex weight control by analog processing is unnecessary, and the switch control circuit 14 is not set, which can reduce the circuit size.

[0216] Figure 23This is a diagram showing the value of the mixing rate α in the boundary region BR. As described above, when different regions exist on the two ends in the row direction within the boundary regions BR (first boundary region BR1, second boundary region BR2), it is necessary to change the mixing rate α of each vertical signal line VSL. The mixing rate α can be fixed in the third boundary region BR3 and the fourth boundary region BR4 on the two ends in the column direction of region Z1. In the boundary regions BR1 and BR2 where the mixing rate α varies for each vertical signal line, a switch control circuit 14 needs to be provided for each vertical signal line. However, in the boundary regions BR3 and BR4 where the mixing rate α is fixed, a switch control circuit 14 can be provided for each boundary region.

[0217] Figure 24 This diagram illustrates the connection relationship between the switch control circuit 14 in the four-pixel weighted averaging circuit 13 and the first to third four-pixel weighted averaging circuits 13a, 13b, and 13c. Figure 24 As shown, each switch in the first to third four-pixel weighted averaging circuits 13a, 13b, and 13c can be controlled by a switch control circuit 14. That is, it is not necessary to provide a switch control circuit 14 for each of the first to third four-pixel weighted averaging circuits 13a, 13b, and 13c.

[0218] However, as Figure 23 As shown, in the first boundary region BR1 and the second boundary region BR2, for each of the multiple vertical signal lines VSL, multiple capacitors and multiple switches are required, as well as switch control circuitry 14. Therefore, the circuit size of the analog-to-digital converter 83 increases. The circuit size is increased by… Figure 24 The following are shown: (1) switches SW1 to SW8, (2) switch control circuit 14 and (3) capacitors C1 to C4.

[0219] Figure 25 This is a planar layout diagram of pixel unit 81. Pixel unit 81 is provided with horizontal optical black (HOPB) regions 81b arranged on both sides of the row direction of aperture pixel region 81a, vertical optical black (VOPB) regions 81c arranged on both sides of the column direction of aperture pixel region, and a processing virtual region 81d arranged to surround each of HOPB region 81b and VOPB region 81c.

[0220] Although pixels with a structure similar to that of the aperture pixel region 81a are arranged in the HOPB region 81b and the processing virtual region 81d, these pixels are not used in imaging or event detection applications. Therefore, in order to reduce the circuit size, it is preferable to arrange the two-pixel weighted averaging circuit 12 or the four-pixel weighted averaging circuit 13 only in the aperture pixel region.

[0221] Figure 26 It is based on Figure 25 A planar layout diagram of a modified example. Human gaze is typically detected near the center of pixel unit 81, and region Z1 is typically located near the center of the aperture pixel region. Therefore, the two-pixel weighted averaging circuit 12 or the four-pixel weighted averaging circuit 13 can be arranged only in the central region 81e of the aperture pixel region. If region Z1 moves to a region where the two-pixel weighted averaging circuit 12 or the four-pixel weighted averaging circuit 13 is not arranged, the weighted averaging process can be performed by digital signal processing of the signal processing circuit 88.

[0222] Figure 27 It is shown that... Figure 26 The illustration compares examples of further reducing the size of the two-pixel weighted averaging circuit 12 or the four-pixel weighted averaging circuit 13. Figure 27 In this configuration, a two-pixel weighted averaging circuit 12 or a four-pixel weighted averaging circuit 13 is provided only at both ends of the row direction in region Z1, and weighted averaging processing is performed at the center side through digital signal processing by the signal processing circuit 88. In this case, as shown in reference... Figure 21 The described circuit requires not only a two-pixel weighted averaging circuit 12 or a four-pixel weighted averaging circuit 13, but also a switch control circuit 14.

[0223] (Illustrative configuration of an information processing system)

[0224] Furthermore, the light detection device 2004 according to this disclosure can be mounted on a light detection system with line-of-sight detection function.

[0225] (Illustrative configuration of an optical detection system)

[0226] The light detection device 2004 according to this disclosure can be included in the information processing system 1001 which has a line-of-sight detection function.

[0227] Figure 28 It is based on Figure 2 The diagram shown is a simplified block diagram of the information processing system 1001 of this disclosure, which includes a photodetector 2004. Figure 28 The information processing system 1001 includes a light detection device 2004, a signal processing device 111, and a line-of-sight detection device 112. Figure 28 The signal processing device 111 and the line-of-sight detection device 112 are set in Figure 2 In at least one of the processing circuit 2001, camera 2003, or sensor 2008.

[0228] Specifically, the signal processing device 111 includes an image signal processor (ISP), an application processor (AP), etc.

[0229] Optical inspection equipment 2004, for example, with Figure 5 Configure it similarly. Figure 28 express Figure 5 The simplified internal structure of the optical detection device 2004 includes at least a pixel unit 81, a signal processing circuit 88, and an interface circuit (IF) 113. Figure 28 The interface circuit (IF) 113 in the middle corresponds to Figure 5 The output circuit 89 in the middle.

[0230] The signal processing device 111 includes a frame buffer 114, a first processing unit (IFE) 115, and a second processing unit (IPE) 116.

[0231] Frame buffer 114 stores image data for each frame output from light detection device 2004. As described later, light detection device 2004 generates image data for each region. Therefore, frame buffer 114 stores image data for each of multiple regions.

[0232] The first processing unit 115 performs various image processing operations on each image data to generate luminance signals and chromatic aberration signals (YUV signals).

[0233] The second processing unit 116 performs various types of image processing and image merging processing based on multiple luminance signals and chrominance signals corresponding to multiple regions, and generates combined luminance signals and chrominance signals (YUV signals).

[0234] The gaze detection device 112 detects the user's gaze and transmits information about the gaze position (hereinafter referred to as the gaze position) to the signal processing device 111. Specifically, the signal processing device 111 calculates area information based on the gaze detection information detected by the gaze detection device 112 and sends the calculated area information to the light detection device 2004.

[0235] The light detection device 2004 generates image data by changing the resolution of each region based on the region information sent from the signal processing device 111.

[0236] (Constraints on region settings)

[0237] The light detection device 2004 of this disclosure sets multiple regions in the pixel section 81 based on the line-of-sight position detected by the line-of-sight detection device 112. At this time, the light detection device 2004 needs to comply with the constraints of the region setting.

[0238] More specifically, in the light detection apparatus 2004 according to this disclosure, the region including the line of sight detected by the line-of-sight detection device 112 at the center position is region Z1, and the region including region Z1 is region Z2. Region Z1 is the region in which the photoelectric converted pixel signal is output in units of pixels. Region Z2 is the region of combined processing where the pixel signals of multiple pixels are combined and output as a single pixel signal. For example, in the case where region Z2 undergoes 2×2 pixel merging processing, region Z2 has a size four times that of region Z1.

[0239] The light detection device 2004 according to this disclosure can be configured with three or more regions. For example, in the case where region Z3 includes region Z2, merging processing is performed in region Z3 on a unit basis of pixel regions larger than the size of region Z2. For example, in the case where merging processing of 4×4 pixels is performed in region Z3, region Z3 has a size 16 times that of region Z1.

[0240] The following constraints apply: region Z2 must include the entire area of ​​region Z1, and region Z3 must include the entire area of ​​region Z2. For example, ... Figure 29 As shown in Figure A, in the case where all sides of region Z1 exist within all sides of region Z2, or as... Figure 29 As shown in B, the area setting is determined to be valid (OK) when at least some sides of region Z1 are in contact with any side of region Z2; however, if at least some sides of region Z1 are outside any side of region Z2, as shown in Figure B, the area setting is not valid (OK). Figure 29 As shown in C, the locale setting is invalid (NG: Error).

[0241] The optical detection device 2004 according to this disclosure needs to set multiple regions based on the gaze position to meet the constraints of the aforementioned region setting. Furthermore, since the gaze position detected by the gaze detection device 112 can change frame by frame, the optical detection device 2004 can change the position and size of the multiple regions on a per-frame basis according to the region information from the signal processing device 111.

[0242] (Regional Setting ID)

[0243] The optical detection device 2004 uses, for example, a virtual channel conforming to the Mobile Industry Processor Interface (MIPI) standard to transmit image data for each zone to the signal processing device 111. At this time, the signal processing device 111 needs to determine which frame and which zone the image data sent from the optical detection device 2004 belongs to. Therefore, the optical detection device 2004 according to this disclosure provides a region setting ID. The region setting ID is information used to identify the frame to which the read image data belongs. The optical detection device 2004 according to this disclosure adds the region setting ID to the image data and sends the image data to the signal processing device 111.

[0244] The zone setting ID can be assigned by the optical detection device 2004 or by the signal processing device 111.

[0245] Figure 30 This is a sequence diagram illustrating the steps of sending and receiving image data between the light detection device 2004 and the signal processing device 111 when a region setting ID is assigned to the light detection device 2004. Upon receiving gaze detection information from the gaze detection device 112 (step S1), the signal processing device 111 calculates region information regarding the position and size of the region Z1, which includes the gaze location as the center (step S2). The region information is stored, for example, in a region setting register, as described later.

[0246] The signal processing device 111 sends the calculated region information to the light detection device 2004 (step S3). The light detection device 2004 assigns a region setting ID corresponding to the region information (step S4). The region setting ID is associated with the frame from which the pixel signal is read.

[0247] The light detection device 2004 sends the region setting ID to the signal processing device 111 (step S5). For example, the light detection device 2004 sets regions Z1 to Z3 in the pixel unit 81, performs an exposure operation, and then reads the pixel signal. For example, in region Z1, the pixel signal is read for each pixel, while in regions Z2 and Z3, a merging process is performed for each pixel among multiple pixels to read the pixel signal (step S6). In the light detection device 2004, after the region setting is performed, a time delay of N (N is an integer greater than 1) frames sometimes occurs before the image signal reflecting the region setting is read out.

[0248] The optical detection device 2004, for example, sends packets obtained by adding a corresponding region setting ID to the image data of each frame to the signal processing device 111 in a virtual channel conforming to the MIPI standard (step S7).

[0249] The signal processing device 111 matches the region setting ID added to the received image data with the region setting ID received in step S5 to specify which frame of the image data the received image data belongs to (step S8).

[0250] Figure 31 It shows the setting Figure 30 A diagram illustrating an example of the setting information of the region setting register inside the signal processing device 111. Figure 31 This illustrates an example of how the setting information for regions Z1 to Z3 is stored in a region setting register. The region setting register stores the two-dimensional coordinates of the header pixel and the last pixel in each region Z1 to Z3.

[0251] Figure 32 It means in Figure 30 A diagram illustrating an example of the data structure of the image data sent from the light detection device 2004 to the signal processing device 111 in step S7. Figure 32 The image data consists of pixel values ​​(16-bit data) arranged sequentially in the order of zones Z3, Z2, and Z1, and includes 16-bit data representing the zone setting ID following that data.

[0252] Figure 33 This is a sequence diagram illustrating the steps of sending and receiving image data between the light detection device 2004 and the signal processing device 111 when the signal processing device 111 assigns a region setting ID. When receiving gaze detection information from the gaze detection device 112 (step S11), the signal processing device 111 calculates region information related to the position and size of the region Z1 centered on the gaze position (step S12) and assigns a region setting ID (step S13). The sequence diagram and... Figure 30 The difference in the sequence diagram is that the signal processing device 111 assigns an area setting ID.

[0253] The signal processing device 111 adds a region setting ID to the calculated region information and sends the region information to the optical detection device 2004 (step S14).

[0254] For example, the light detection device 2004 sets regions Z1 to Z3 in the pixel unit 81, performs an exposure operation, and then reads the pixel signal of each region. For example, in region Z1, the pixel signal is read for each pixel, while in regions Z2 and Z3, a merging process is performed for each pixel among multiple pixels to read the pixel signal (step S15).

[0255] The optical detection device 2004, for example, sends packets obtained by adding a corresponding region setting ID to the image data of each frame to the signal processing device 111 in a virtual channel conforming to the MIPI standard (step S16).

[0256] The signal processing device 111 matches the region setting ID added to the received image data with the region setting ID assigned by itself in step S13 to specify which frame of the image data the received image data belongs to (step S17).

[0257] Figure 34 It shows the setting Figure 33 A diagram illustrating an example of the setting information of the region setting register inside the signal processing device 111. Figure 34 The region setting register includes information about the region setting ID allocated in step S13. Other information and... Figure 32 The same as in.

[0258] In step S16, the data structure of the image data sent from the light detection device 2004 to the signal processing device 111 is... Figure 32 Since they are the same, their description is omitted.

[0259] (Regional settings information)

[0260] In the above Figures 30 to 34 The example described above illustrates an instance where a region setting ID is appended when image data is sent from the light detection device 2004 to the signal processing device 111. However, in addition to the region setting ID, it is desirable to append information related to the position and size of each region (region setting information) to the image data when sending image data from the light detection device 2004 to the signal processing device 111. This is because the light detection device 2004 can change the position and size of each region according to each frame; therefore, if the position and size of each region are not communicated to the signal processing device 111, the signal processing device 111 cannot grasp the position and size of each region in each frame, and the utilization efficiency of the storage capacity of the frame buffer 114 is reduced. Therefore, according to the present invention, the light detection device 2004 adds the region setting information of all regions to the header of the image data of each region sent to the signal processing device 111. Thus, the signal processing device 111 can grasp the position and size of all regions regardless of which region's image data is received, and can ensure the storage capacity of the frame buffer 114 storing the received image data of each region.

[0261] Figure 35 This is a diagram showing the data structure of image data for each region sent from the optical detection device 2004 to the signal processing device 111. Figure 35 An example is shown where regions Z1 to Z3 are set in pixel unit 81. Figure 35The data structures for regions Z3, Z2, and Z1 are shown from left to right. The image data in each of regions Z1 to Z3 includes the start of frame (FS), virtual channel information (VC), embedded data (EBD), packet footer (PF), data type (DT), and valid pixel data.

[0262] The zone setting information is included, for example, in the embedded data (EBD). As mentioned above, the zone setting information in each zone Z1 to Z3 includes information about the location and size of all zones Z1 to Z3.

[0263] When at least one of the location and size of a region changes, the region setting information needs to be updated accordingly. Because at least one of the location and size of a region can change from frame to frame, the region setting information can also be updated from frame to frame.

[0264] However, in the absence of an update notification of region information from signal processing device 111, optical detection device 2004 continues to use the region setting information from the previous frame. Typically, because gaze detection device 112 and optical detection device 2004 operate asynchronously, the timing of gaze movement detection by gaze detection device 112 may not coincide with the frame switching timing in optical detection device 2004. Typically, the gaze detection period performed by gaze detection device 112 is shorter than the frame period of optical detection device 2004.

[0265] The above area setting information may include only the line-of-sight coordinates, and not all the coordinates of each area. Figure 36 The left side shows an instance where the coordinates of all pixels in each zone are included in the zone setting information. Furthermore, Figure 36 The right side shows an instance where the dimensions of regions Z1 and Z2 are pre-registered, and only the coordinates of the line-of-sight positions are included in the zoning information with the center positions of regions Z1 and Z2 as the line-of-sight positions. Figure 36 In the example on the right, although the dimensions of regions Z1 and Z2 cannot be changed, the coordinate information of regions Z1 and Z2 included in the region setting information can be greatly reduced.

[0266] Furthermore, the dimensions of regions Z1 and Z2 can be fixed to predetermined dimensions. The coordinates of regions Z1 and Z2 can be set such that the center positions of regions Z1 and Z2 are line-of-sight coordinates. Figure 28 In this process, the gaze information detected by the gaze detection device 112 is input to the signal processing device 111. However, the gaze information detected by the gaze detection device 112 can be directly input to the light detection device 2004.

[0267] The gaze detection device 112 can output gaze coordinates obtained by normalizing the coordinates of the pixel unit 81. Alternatively, the physical gaze coordinates detected by the gaze detection device 112 can be converted into the coordinates of the pixel unit 81 and output.

[0268] (Prediction of line-of-sight coordinates)

[0269] The gaze information detected by the gaze detection device 112 is input to the light detection device 2004 via the signal processing device 111. The light detection device 2004 performs exposure and pixel signal readout operations based on the gaze position detected by the gaze detection device 112. Therefore, a time delay occurs from the time the gaze detection device 112 detects the gaze position until the light detection device 2004 reads out the pixel signal from the area corresponding to that gaze position. Furthermore, the gaze detection period of the gaze detection device 112 is different from the frame period used for exposure and pixel signal readout by the light detection device 2004; the two operate asynchronously.

[0270] Figure 37 This is a graph representing the time delay from when the gaze detection device 112 detects the gaze position to when the light detection device 2004 reads the pixel signal from the area corresponding to the gaze position. Figure 37 The vertical lines in the image represent the gaze detection period and frame switching timing of the VST sensor. That is, the gaze can be moved to other positions based on the timing of the pixel signal readout by the light detection device 2004. Therefore, it is possible to predict... Figure 37 The movement of the gaze during the time delay shown is illustrated, and pixel signals from the region based on the predicted gaze position can be read. While the algorithm used to predict the gaze position is not limited, the gaze position can be predicted by, for example, linear interpolation of gaze positions in two or more past frames.

[0271] (View position in 3D coordinates)

[0272] Some gaze detection devices 112 output the human gaze (gaze) position as coordinates in three-dimensional space (three-dimensional coordinates). Furthermore, the light detection device 2004 according to this disclosure may include a video perspective (VST) sensor for generating a captured image for the left eye and a VST sensor for generating a captured image for the right eye.

[0273] Figure 38This is a diagram used to describe the gaze position and coordinates of each region in the two VST sensors for the left and right eyes. The light detection device 2004 projects the three-dimensional coordinates 118 onto a display plane 117 virtually arranged in three-dimensional space corresponding to the gaze (gaze) position output from the gaze detection device 112, and projects them onto the image plane 119 of the left-eye VST sensor and the image plane 120 of the right-eye VST sensor respectively, and calculates the coordinates corresponding to the gaze position on each image plane. Note that... Figure 38 The virtual display plane 117 in the image does not necessarily coincide with the physical display plane. The coordinates of regions Z1 to Z3 are specified in each of the left-eye and right-eye image planes.

[0274] Even when using VST sensors, such as Figure 37 As shown, there is a time delay between the detection of the gaze position by the gaze detection device 112 and the reading of image data for each region on each image plane 119 and 120. Therefore, as Figure 37 As shown, it is possible to predict the movement of the gaze during the time lag and to read pixel signals from the region corresponding to the predicted gaze position.

[0275] Figure 39 This is a diagram showing the data transmitted and received between the two VST sensors (hereinafter referred to as CIS1(L) and CIS2(R)) constituting the line-of-sight detection device 112. Figure 39 The signal processing device 111 is shown as an example of an application processor (AP). Figure 39 An example is shown where CIS1(L) is the main sensor and CIS2(R) is the secondary sensor, but CIS2(R) can be the main sensor and CIS1(L) can be the secondary sensor.

[0276] AP 111 transmits region information about region Z1, including the gaze position, to the left eye's CIS1(L) via, for example, an internal integrated circuit (I2C) or an improved inter-integrated circuit (I3C). The region information includes the position and size (x, y, w, h) of region Z1 in CIS2(R) and the offset information (xoffset, yoffset) of region Z1. x and y are two-dimensional coordinates on the image plane of CIS1(L), w is the width of region Z1, and h is the height of region Z1.

[0277] CIS1(L) generates area setting information including the location and size (x, y, w, h) of area Z1. The area setting information includes the offset information (xoffset, yoffset) of area Z2 sent from AP111.

[0278] CIS1(L) transmits the generated region setting information to CIS2(R) of the right eye via, for example, I2C / I3C or a Serial Peripheral Interface (SPI). CIS2(R) generates region setting information including the position and size (x+xoffset, y+yoffset, w, h) of region Z2. Note that yoffset can be omitted because the left and right gazes move almost no vertically to the image plane.

[0279] CIS1(L) and CIS2(R) respectively send image data to AP111 via MIPI, for example.

[0280] There is a possibility of timing offsets reflecting locale setting information in each of CIS1(L) and CIS2(R). Therefore, reflection timing information can be added to the locale setting information sent from CIS1(L) to CIS2(R) to match the setting reflection timing of CIS1(L) and CIS2(R). The reflection timing information indicates the locale setting information to be reflected after N frames, assuming CIS1(L) and CIS2(R) are synchronized. Therefore, due to the setting timing offsets in the locale setting information of CIS1(L) and CIS2(R), there is no issue of different frame reflection settings between the left-eye and right-eye images.

[0281] Image data generated by CIS1(L) and image data generated by CIS2(R) can be connected and transmitted to AP111 as an image. Figure 40 This is a block diagram illustrating the scenario where two images generated by CIS1 (L) and CIS2 (R) are concatenated as a single image and transmitted to AP111. Figure 40 In this example, image data output from CIS2(R) is transmitted to CIS1(L) via, for example, MIPI. CIS1(L) connects the two image data sets and sends the image data as a single image data set to AP111 via, for example, MIPI.

[0282] Figure 41 This is a diagram illustrating the data structure of image data sent from CIS1(L) to AP 111. The image data includes Start of Frame (FS), Frame Header (FH), image data generated by CIS1(L) and image data generated by CIS(R), Frame Tail (FF), and Frame End (FE).

[0283] exist Figure 41 In this configuration, image data generated by CIS1(L) is set on the high-order bits of the bit string, while image data generated by CIS2(R) is set on the low-order bits of the bit string. However, image data can be set as a separate bit string.

[0284] exist Figure 41In this case, because AP 111 can acquire two image data sets almost simultaneously, there is no need to synchronize or adjust the timing of the reception of the two image data sets. Figure 40 This reduces the difference between the receiving timing and the actual timing, and simplifies the processing of AP 111.

[0285] (Supplement to the regional setting ID)

[0286] A frame number can be added to the frame currently being imaged. The frame number is managed by multiple bit strings. According to the optical detection device 2004 of this disclosure, the lower N bits can be used as a region setting ID to send the lower N bits of the bit string representing the frame number to the signal processing device 111.

[0287] Figure 42 This is a diagram illustrating the exposure and readout timing of the light detection device 2004. The light detection device 2004 uses an existing communication interface to send the aforementioned area setting ID to the signal processing device 111 (e.g., AP111). Figure 42 An example is shown in which the lower 2 bits of the frame number are used as a locale ID by using, for example, a general purpose input / output (GPIO) as an existing communication interface. Figure 42 The “LL”, “LH”, “HL” and “HH” in the text show instances where the lower two bits are “00”, “01”, “10” and “11”, respectively.

[0288] exist Figure 42 In this process, image data exposed during the frame period from time t1 to time t2 is transmitted to AP 111 by adding the lower two bits of the frame number, "0", to the image data via the MIPI interface during the period from time t3 to time t4. In this case, the lower two bits, "00", are transmitted to AP 111 in advance via GPIO, and therefore the lower two bits added to the image data are "0", enabling AP 111 to recognize that data was exposed during the frame period from time t1 to t2.

[0289] GPIO is a pre-prepared communication interface for the optical detection device 2004 and AP111 to send and receive various information. For example, the lower two bits of an existing frame number can be sent from the optical detection device 2004 to the AP111 using the existing communication interface, thus enabling the setting and notification of the zone ID without adding new hardware. Note that here, the lower two bits are shown as an example, but the bit depth is not limited.

[0290] (Connecting multiple CISs to a single channel)

[0291] Furthermore, the optical detection device 2004 of the present invention can connect multiple devices on a single channel that communicates with the signal processing device 111.

[0292] Figure 43 This is a block diagram showing the general structure of an information processing system 1001 that connects multiple optical detection devices 2004 to a single channel 122 and communicates with a signal processing device 111. Figure 43 The signal processing device 111 is shown as an example of AP 111.

[0293] exist Figure 43 In the information processing system 1001, each optical detection device 2004 connected to a single channel 122 can send various information to or receive various information from the AP 111 independently of other optical detection devices 2004. Within the single channel 122, data packets containing image data from multiple optical detection devices 2004 are synthesized and transmitted in a time-division manner. Additionally, within the single channel 122, data packets containing various information sent from the AP 111 to multiple optical detection devices 2004 are transmitted. The header of each data packet includes information identifying the optical detection device 2004 or the AP 111 as the destination.

[0294] <Example of its application in an in vivo information acquisition system>

[0295] The technology disclosed herein (the Technology) can be applied to a variety of products. For example, the Technology disclosed herein can be applied to endoscopic surgical systems.

[0296] Figure 44 This is a block diagram illustrating an example of a schematic configuration of a patient in vivo information acquisition system using a capsule endoscope, to which the technology (the technology) according to one embodiment of this disclosure can be applied.

[0297] The in vivo information acquisition system 10001 includes a capsule endoscope 10100 and an external control device 10200.

[0298] During the examination, the patient swallows a capsule endoscope 10100. The capsule endoscope 10100 has camera and wireless communication functions, and while moving through the stomach, intestines and other internal organs by peristalsis for a period of time before being naturally expelled from the patient, it continuously takes images of the internal organs (hereinafter referred to as in vivo images) at predetermined intervals. The information of the in vivo images is then transmitted wirelessly to an external control device 10200 outside the body.

[0299] The external control device 10200 controls the operation of the internal information acquisition system 10001. Further, the external control device 10200 receives information about internal images transmitted from the capsule endoscope 10100 and generates image data based on the received internal image information for displaying the internal images on a display device (not shown).

[0300] In the in vivo information acquisition system 10001, in vivo images of the patient's internal state can be acquired in this manner at any time during the period from swallowing until the capsule endoscope 10100 is expelled.

[0301] The construction and function of the capsule endoscope 10100 and the external control device 10200 are described in more detail below.

[0302] The capsule endoscope 10100 includes a capsule-shaped housing 10101, which houses a light source unit 10111, an image acquisition unit 10112, an image processing unit 10113, a wireless communication unit 10114, a power feed unit 10115, a power supply unit 10116, and a control unit 10117.

[0303] For example, the light source unit 10111 includes a light source such as a light-emitting diode (LED) and illuminates the field of view of the image pickup unit 10112.

[0304] Image pickup unit 10112 includes an image pickup element and an optical system. The optical system includes multiple lenses disposed at the front end of the image pickup element. Reflected light (hereinafter referred to as observation light) illuminating body tissue, which is the target of observation, is focused by the optical system and introduced into the image pickup element. In image pickup unit 10112, the image pickup element photoelectrically converts the incident observation light, thereby generating an image signal corresponding to the observation light. The image signal generated by image pickup unit 10112 is provided to image processing unit 10113.

[0305] The image processing unit 10113 includes a processor such as a central processing unit (CPU) or a graphics processing unit (GPU), and performs various signal processing on the image signal generated by the image acquisition unit 10112. The image processing unit 10113 provides the signal-processed image signal as RAW data to the wireless communication unit 10114.

[0306] The wireless communication unit 10114 performs predetermined processing, such as modulating the image signal that has already undergone signal processing by the image processing unit 10113, and transmits the resulting image signal to the external control device 10200 via the antenna 10114A. Further, the wireless communication unit 10114 receives control signals related to the drive control of the capsule endoscope 10100 from the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 provides the control signals received from the external control device 10200 to the control unit 10117.

[0307] The power feed unit 10115 includes an antenna coil for power reception, a power regeneration circuit for regenerating electricity from the current generated in the antenna coil, and a boost circuit. The power feed unit 10115 generates electricity using the principle of contactless charging.

[0308] The power supply unit 10116 includes a secondary battery and stores the power generated by the power feed unit 10115. Figure 44 In order to avoid a complicated schematic diagram, the arrow markings indicating the endpoints of the power supply from the power supply unit 10116 have been omitted. However, the power stored in the power supply unit 10116 is supplied to the light source unit 10111, the image acquisition unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the control unit 10117, thereby driving these units.

[0309] The control unit 10117 includes a processor such as a CPU, and appropriately controls the driving of the light source unit 10111, the image pickup unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the power feed unit 10115 according to the control signals transmitted to it from the external control device 10200.

[0310] The external control device 10200 includes a processor such as a CPU or GPU, a microcomputer, a control board, etc., which integrates the processor and storage elements such as a memory. The external control device 10200 transmits control signals to the control unit 10117 of the capsule endoscope 10100 via antenna 10200A to control the operation of the capsule endoscope 10100. In the capsule endoscope 10100, for example, the illumination conditions of the light source unit 10111 on the observed target can be changed according to the control signals from the external control device 10200. Furthermore, the imaging conditions (e.g., frame rate, exposure value, etc. of the image acquisition unit 10112) can be changed according to the control signals from the external control device 10200. Furthermore, the conditions of the content processed by the image processing unit 10113 or the image signal transmitted by the wireless communication unit 10114 (e.g., transmission interval, number of transmitted images, etc.) can be changed according to the control signals from the external control device 10200.

[0311] Furthermore, the external control device 10200 performs various image processing operations on the image signals transmitted from the capsule endoscope 10100 to generate image data for displaying the captured intraoperative image on a display device. Regarding image processing, various signal processing operations can be performed, such as image enhancement processing (de-mosaic processing), image quality improvement processing (bandwidth enhancement processing, super-resolution processing, noise reduction (NR) processing, and / or jitter correction processing), and / or magnification processing (electronic focusing processing). The external control device 10200 controls the drive of the display device, causing the display device to display the captured intraoperative image based on the generated image data. Alternatively, the external control device 10200 can also control a recording device (not shown) to record the generated image data or control a printing device (not shown) to print out the generated image data.

[0312] Examples of in vivo information acquisition systems to which the technology according to this disclosure can be applied have been described above. The technology according to this disclosure can be applied to the image acquisition unit 10112 in the above configuration. By applying the technology according to this disclosure to the image acquisition unit 10112, clearer images of the surgical site can be obtained, thereby improving the accuracy of the examination.

[0313] The functions implemented by the configuration elements described herein can be implemented in circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), central processing units (CPUs), conventional circuits, and / or combinations thereof, programmed to implement the described functions. A processor includes transistors and other circuitry and is considered as a circuit or processing circuit. A processor can be a programmable processor that executes programs stored in memory.

[0314] In this specification, circuits, units, and devices are hardware programmed to implement or perform the functions described herein. The hardware may be any hardware disclosed herein or any hardware programmed or known to perform the functions described herein.

[0315] In the case where hardware is considered as a processor, a circuit, device, or unit is a combination of hardware and software used to configure the hardware and / or processor.

[0316] It should be noted that this technology can have the following configurations.

[0317] (1) A light detection device, comprising: A pixel unit includes multiple pixels, each pixel including a photoelectric conversion element; and The control unit controls the pixel signal of each pixel in a boundary region based on the pixel signal of the surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels, including the gaze position, among a plurality of pixels. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region.

[0318] (2) The optical detection device according to (1), wherein, The control unit generates a pixel signal that is closer to the pixel signal in the first region than the pixel on the side closer to the first region in the boundary region, and generates a pixel signal that is closer to the pixel signal in the second region than the pixel on the side closer to the second region.

[0319] (3) The optical detection device according to (1) or (2), wherein, The control unit controls the pixel signal of each pixel in the boundary region by a weighted average of the pixel signals of one or more pixels adjacent in a first direction and the pixel signals of one or more pixels adjacent in a second direction intersecting the first direction.

[0320] (4) The optical detection device according to (3), wherein, The control unit calculates a weighted average value after weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction that intersects with the first direction, based on the pixel position of each pixel in the boundary region.

[0321] (5) The optical detection device according to (4), wherein, The boundary area includes: A first boundary region and a second boundary region are arranged on two end sides in a first direction within the first region; and The third and fourth boundary regions are arranged on the two end sides of the first region in the second direction, and The control unit uses corresponding weights and peripheral pixels to control the pixel signals of each pixel in the first to fourth boundary regions.

[0322] (6) The optical detection device according to (5), wherein, In the first and second boundary regions, a weighted average is calculated through analog processing, and in the third and fourth boundary regions, a weighted average is calculated through digital signal processing.

[0323] (7) The optical detection device according to (5), wherein, In the first and second boundary regions, a weighted average is calculated using digital signal processing, and in the third and fourth boundary regions, a weighted average is calculated using analog processing.

[0324] (8) The optical detection apparatus according to any one of (3) to (7), wherein, The control unit includes a signal processing circuit that performs digital signal processing for each pixel signal in the boundary region, calculating a weighted average between the pixel signals of one or more adjacent pixels in a first direction and the pixel signals of one or more adjacent pixels in a second direction.

[0325] (9) The optical detection apparatus according to any one of (3) to (7), wherein, The control unit includes: The analog processing circuit calculates a weighted average of the pixel signal of each pixel in the boundary region and the pixel signals of one or more adjacent pixels in a first direction; and The digital signal processing unit calculates a weighted average between the pixel signal of each pixel, which has been weighted by analog processing circuitry, and the pixel signal weighted average of one or more adjacent pixels in the second direction, which has also been weighted by analog processing circuitry.

[0326] (10) The optical detection device according to (9), wherein, The analog processing circuit and the digital signal processing unit calculate the weighted average value by weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction intersecting the first direction, according to the pixel position of each pixel in the boundary region.

[0327] (11) The optical detection device according to (9) or (10), wherein, The analog processing circuit calculates the weighted average value in some pixel regions of the boundary region, and the digital signal processing unit calculates the weighted average value in some pixel regions outside the boundary region.

[0328] (12) The optical detection apparatus according to any one of (3) to (7), wherein, The control unit includes an analog processing circuit that calculates a weighted average of the pixel signals of one or more adjacent pixels in a first direction and the pixel signals of one or more adjacent pixels in a second direction, relative to the pixel signal of each pixel in the boundary region.

[0329] (13) The optical detection device according to (12), wherein, The analog processing circuit calculates the weighted average value after weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction according to the pixel position of each pixel in the boundary region.

[0330] (14) The optical detection device according to (13) further includes: Multiple signal lines transmit pixel signals of multiple pixels arranged in the second direction. The analog processing circuit includes: Multiple weighted averaging circuits are configured in each of the multiple signal lines and calculate the weighted average; and Multiple switching control circuits are configured in conjunction with multiple weighted averaging circuits. Each of the multiple weighted average circuits includes: Multiple first capacitors are disposed in the boundary region for each corresponding signal line and have different capacitances; Multiple second capacitors are provided, corresponding to multiple pixel settings, wherein the multiple pixels are adjacent to a pixel connected to a corresponding signal line in a first direction and a second direction, and have the same capacitance; Multiple first switches toggle between charging multiple first capacitors using the charge from a pixel signal of a pixel; and The second switch toggles between combining the stored charges of the multiple second capacitors with the stored charges of the multiple first capacitors. Each switch in a multi-switch control circuit controls the switching of multiple first and second switches in a corresponding weighted average circuit, and The weighted average value and the corresponding weights calculated in each of the multiple weighted average circuits have a non-linear relationship.

[0331] (15) The optical detection device according to (13) further includes: Multiple signal lines transmit pixel signals of multiple pixels arranged in the second direction. The analog processing circuit includes: Multiple weighted averaging circuits are configured in each of the multiple signal lines and calculate the weighted average; and Multiple switching control circuits are configured in conjunction with multiple weighted averaging circuits. Each of the multiple weighted averaging circuits includes multiple sub-weighted averaging circuits, which calculate a weighted average between a pixel connected to a corresponding signal line and multiple different pixels arranged around that pixel. Each of the multiple sub-weighted averaging circuits includes: Multiple capacitors; and Multiple switches toggle between storing charge in the multiple capacitors based on the pixel value of a pixel or the pixel value of another pixel surrounding a pixel. Each of the multiple switch control circuits controls the switching of multiple switches in its corresponding multiple sub-weighted averaging circuits, and The weighted average obtained by combining the weighted averages calculated by multiple sub-weighted average circuits has a linear relationship with the corresponding weights.

[0332] (16) The optical detection apparatus according to any one of (12) to (15), wherein, The boundary region includes a first boundary region and a second boundary region disposed on two end sides in the first direction, and a third boundary region and a fourth boundary region disposed on two end sides in the second direction. In the first boundary region, the second boundary region, the third boundary region, and the fourth boundary region, the additive average is calculated by analog processing circuit.

[0333] (17) The optical detection device according to (15) further includes: The aperture pixel area includes multiple pixels; An optical black area is arranged to surround the aperture pixel area; and The virtual region is configured to surround the optical black area. Among them, multiple weighted averaging circuits and switching control circuits are arranged to correspond only to at least a portion of the aperture pixel region.

[0334] (18) The optical detection device according to (17), wherein, Multiple weighted averaging circuits and switching control circuits are arranged to correspond only to a portion of the area including the center position of the aperture pixel region.

[0335] (19) The optical detection apparatus according to any one of (1) to (18), wherein, The boundary region includes the first boundary region and the second boundary region, and The control unit controls the pixel signal of each pixel in each boundary region of the first boundary region between the first region and the second region and the second boundary region between the third region and the second region based on the pixel signal of the surrounding pixels. The third region is arranged to surround at least a portion of the second region and has a resolution different from that of the first region and the second region.

[0336] (20) An information processing system, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; A control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels among a plurality of pixels, including the gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. The display unit displays an image generated based on the pixel signals of each pixel in the first region, the second region, and the boundary region.

[0337] (21) An information processing system, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; A control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels among the plurality of pixels, including the gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. The processing circuit generates a pixel signal for each pixel in the boundary region based on the pixel signals of the surrounding pixels of the boundary region.

[0338] (22) An information processing system, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; A control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels, including the gaze position, among a plurality of pixels. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. The display unit combines and displays a first image based on pixel signals of each pixel in a first region, a second image based on pixel signals of each pixel in a second region, and a third image based on pixel signals of each pixel in a boundary region.

[0339] This disclosure is not limited to the individual embodiments described above, but includes various modifications that may be conceived by those skilled in the art, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions may be made without departing from the conceptual idea and spirit of this disclosure as defined in the claims and their equivalents.

[0340] Reference Symbol List

[0341] 1. Optical Detection Equipment

[0342] 4-pixel weighted averaging circuit

[0343] 11 Digital Signal Generation Circuit

[0344] 12-pixel weighted averaging circuit

[0345] 13-pixel weighted averaging circuit

[0346] 14 Switch control circuit

[0347] 31 Output Mechanism Unit

[0348] 32 Installation mechanism unit

[0349] 33 Installation belt

[0350] 34. Shell

[0351] 35 Spatial Image Acquisition Units

[0352] 51 Control Unit

[0353] 52 Input Units

[0354] 53 Output Unit

[0355] 54 storage units

[0356] 55 Communication Units

[0357] 56 drives

[0358] 57 Removable storage media

[0359] 58 bus

[0360] 59 Gaze Point Detection Units

[0361] 59A Image Analysis Equipment

[0362] 59AP Application Processor

[0363] 59S Eye Imaging Sensor

[0364] 59SRL Eye Imaging Sensor

[0365] 59SRR Eye Imaging Sensor

[0366] 59STL Eye Imaging Sensor

[0367] 59STR Infrared

[0368] 59STR Eye Imaging Sensor

[0369] 60 image acquisition units

[0370] 61 display units

[0371] 61P Display Image

[0372] 61PL image

[0373] 61PR image

[0374] 71 Image Analysis Unit

[0375] 72 Display Image Generation Unit

[0376] 81 pixel unit

[0377] 81a aperture pixel area

[0378] 81b HOPB area

[0379] 81c VOPB area

[0380] 81d handles virtual regions

[0381] 81e Central Area

[0382] 82 Vertical Scanning Circuit

[0383] 83 Analog-to-digital converter

[0384] 84 Horizontal Transfer Scanning Circuit

[0385] 85 Timing Control Circuit

[0386] 86 Horizontal Transfer Line

[0387] 87 Amplifier Circuit

[0388] 88 Signal Processing Circuit

[0389] 89 Output Circuit

[0390] 90 Area Control Unit

[0391] 100 Reference Voltage Output Unit

[0392] 101 comparator

[0393] 102 counter

[0394] 103 Latch

[0395] 110 Optical Detection System

[0396] 111 Signal processing equipment

[0397] 112 Line of sight detection equipment

[0398] 113 Interface Circuit (IF)

[0399] 114-frame buffer

[0400] 115 First Processing Unit

[0401] 116 Second Processing Unit

[0402] 117 Three-dimensional space

[0403] 118 Three-dimensional coordinates

[0404] 119 Image Plane

[0405] 120 Image Plane

[0406] 122 Single Channel

[0407] 10001 In-Vivo Information Acquisition System

[0408] 10100 Capsule Endoscope

[0409] 10101 Housing

[0410] 10111 Light Source Unit

[0411] 10112 Image Picking Unit

[0412] 10113 Image Processing Unit

[0413] 10114 Wireless Communication Unit

[0414] 10114A Antenna

[0415] 10115 Power Feed Unit

[0416] 10116 Power Supply Unit

[0417] 10117 Control Unit

[0418] 10200 External control equipment

[0419] 10200A Antenna

Claims

1. A light detection device, comprising: A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; as well as The control unit controls the pixel signal of each pixel in a boundary region based on the pixel signals of surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels among the plurality of pixels, including the gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region.

2. The optical detection device according to claim 1, wherein, The control unit generates a pixel signal that is closer to the pixel signal in the first region than the pixel on the side of the boundary region closer to the first region, and generates a pixel signal that is closer to the pixel signal in the second region than the pixel on the side of the boundary region closer to the second region.

3. The optical detection device according to claim 1, wherein, The control unit controls the pixel signal of each pixel in the boundary region by a weighted average of the pixel signals of one or more pixels adjacent in the first direction and the pixel signals of one or more pixels adjacent in the second direction intersecting the first direction.

4. The optical detection device according to claim 3, wherein, The control unit calculates a weighted average value after weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction intersecting the first direction, based on the pixel position of each pixel in the boundary region.

5. The optical detection device according to claim 4, wherein, The boundary region includes: A first boundary region and a second boundary region, wherein the first boundary region is arranged on two end sides in the first direction; and The third and fourth boundary regions are arranged on the two end sides in the second direction within the first region, and The control unit uses corresponding weights and peripheral pixels to control the pixel signal of each pixel in the first boundary region to the fourth boundary region.

6. The optical detection device according to claim 5, wherein, In the first and second boundary regions, a weighted average is calculated through analog computation, and in the third and fourth boundary regions, a weighted average is calculated through digital signal processing.

7. The optical detection device according to claim 5, wherein, In the first and second boundary regions, a weighted average value is calculated by digital signal processing, and in the third and fourth boundary regions, a weighted average value is calculated by analog computation.

8. The optical detection device according to claim 3, wherein, The control unit includes a signal processing circuit that performs digital signal processing for each pixel signal in the boundary region, calculating a weighted average between the pixel signals of one or more adjacent pixels in the first direction and the pixel signals of one or more adjacent pixels in the second direction.

9. The optical detection device according to claim 3, wherein, The control unit includes: The analog processing circuit calculates a weighted average between the pixel signal of each pixel in the boundary region and the pixel signals of one or more adjacent pixels in the first direction; and The digital signal processing unit calculates a weighted average between the pixel signal of each pixel after weighted averaging by the analog processing circuit and the pixel signal weighted average of one or more adjacent pixels in the second direction after weighted averaging by the analog processing circuit.

10. The optical detection device according to claim 9, wherein, The analog processing circuit and the digital signal processing unit calculate the weighted average value by weighting the pixel signal of each pixel in the boundary region, the pixel signal of one or more pixels adjacent in the first direction, and the pixel signal of one or more pixels adjacent in the second direction intersecting the first direction, according to the pixel position of each pixel in the boundary region.

11. The optical detection device according to claim 9, wherein, The analog processing circuit calculates the weighted average value in some pixel regions of the boundary region, and the digital signal processing unit calculates the weighted average value in pixel regions outside the some pixel regions.

12. The optical detection device according to claim 3, wherein, The control unit includes an analog processing circuit that calculates a weighted average value between the pixel signals of one or more pixels adjacent in the first direction and the pixel signals of one or more pixels adjacent in the second direction, relative to the pixel signal of each pixel in the boundary region.

13. The optical detection device according to claim 12, wherein, The analog processing circuit calculates the weighted average value after weighting the pixel signal of each pixel in the boundary region, the pixel signals of one or more adjacent pixels in the first direction, and the pixel signals of one or more adjacent pixels in the second direction according to the pixel position of each pixel in the boundary region.

14. The optical detection device according to claim 13, further comprising: Multiple signal lines transmit pixel signals of multiple pixels arranged in the second direction. The analog processing circuit includes: Multiple weighted averaging circuits are configured in each of the multiple signal lines and calculate the weighted average value; and Multiple switch control circuits are configured in association with the multiple weighted averaging circuits. Each of the plurality of weighted average circuits includes: Multiple first capacitors are disposed in each corresponding signal line in the boundary region and have different capacitances; Multiple second capacitors are provided, corresponding to multiple pixel arrangements, wherein the multiple pixels are adjacent to a pixel connected to a corresponding signal line in the first direction and the second direction, and have the same capacitance; A plurality of first switches toggle between charging the plurality of first capacitors using the charge of a pixel signal based on the pixel of the stated pixel; and The second switch toggles between combining the stored charge of the plurality of second capacitors with the stored charge of the plurality of first capacitors. Each switch in the plurality of switch control circuits controls the switching of the plurality of first switches and second switches in the corresponding weighted average circuit, and The weighted average value and the corresponding weights calculated in each of the multiple weighted average circuits have a non-linear relationship.

15. The optical detection device according to claim 13, further comprising: Multiple signal lines transmit pixel signals of multiple pixels arranged in the second direction. The analog processing circuit includes: Multiple weighted averaging circuits are configured in each of the multiple signal lines and calculate the weighted average value; and Multiple switch control circuits are configured in association with the multiple weighted averaging circuits. Each of the plurality of weighted averaging circuits includes a plurality of sub-weighted averaging circuits, which calculate a weighted average between a pixel connected to a corresponding signal line and a plurality of different pixels arranged around the pixel. Each of the plurality of sub-weighted average circuits includes: Multiple capacitors; and Multiple switches toggle between storing charge in the multiple capacitors based on the pixel value of a pixel or the pixel value of another pixel located around the pixel. Each of the plurality of switch control circuits controls the switching of the plurality of switches in the corresponding plurality of sub-weighted average circuits, and The weighted average value obtained by combining the weighted average values ​​calculated by the multiple sub-weighted average circuits has a linear relationship with the corresponding weights.

16. The optical detection device according to claim 12, wherein, The boundary region includes a first boundary region and a second boundary region disposed on two end sides in the first direction, and a third boundary region and a fourth boundary region disposed on two end sides in the second direction. In the first boundary region, the second boundary region, the third boundary region, and the fourth boundary region, the additive average is calculated by the analog processing circuit.

17. The optical detection device according to claim 15, further comprising: The aperture pixel area includes multiple pixels; An optical black area is arranged to surround the aperture pixel area; as well as The virtual region is processed and configured to surround the optical black region. The plurality of weighted averaging circuits and the switch control circuit are arranged to correspond only to at least a portion of the aperture pixel region.

18. The optical detection device according to claim 17, wherein, The multiple weighted averaging circuits and the switch control circuits are arranged to correspond only to a portion of the region including the center position of the aperture pixel region.

19. The optical detection device according to claim 1, wherein, The boundary region includes a first boundary region and a second boundary region, and The control unit controls the pixel signal of each pixel in each boundary region of the first boundary region between the first region and the second region and the second boundary region between the third region and the second region based on the pixel signals of the surrounding pixels, wherein the third region is arranged to surround at least a portion of the second region and has a resolution different from that of the first region and the second region.

20. An information processing system, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; The control unit controls the pixel signal of each pixel in a boundary region based on the pixel signal of the surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels among the plurality of pixels that include the gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. as well as The display unit displays an image generated based on the pixel signals of each pixel in the first region, the second region, and the boundary region.

21. An information processing system, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; The control unit controls the pixel signal of each pixel in a boundary region based on the pixel signal of the surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels among the plurality of pixels that include the gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. as well as The processing circuit generates a pixel signal for each pixel in the boundary region based on the pixel signals of the surrounding pixels of the boundary region.

22. An information processing system, comprising: A gaze point detection unit detects the person's gaze position. A pixel unit includes multiple pixels, and each pixel includes a photoelectric conversion element; The control unit controls the pixel signal of each pixel in a boundary region based on the pixel signal of the surrounding pixels. The boundary region is located between a first region and a second region. The first region includes two or more pixels among the plurality of pixels that include the gaze position. The second region is arranged to surround at least a portion of the first region and has a resolution different from that of the first region. as well as The display unit combines and displays a first image based on pixel signals of each pixel in the first region, a second image based on pixel signals of each pixel in the second region, and a third image based on pixel signals of each pixel in the boundary region.

Images