Photoelectric conversion device, apparatus, and signal processing method

Abstract

To provide a technique to improve the accuracy of detection of a detection object performed by pixels for difference detection.SOLUTION: A photoelectric conversion device comprises: a detection unit that has a plurality of pixels for obtaining pixel values based on the quantity of incident light into the respective pixels, and detects the difference between the plurality of pixel values; and a control unit that controls the detection unit. The control unit causes the detection unit to perform a first difference detection operation of detecting the difference between the plurality of pixel values obtained from the same pixel, and a second difference detection operation of detecting the difference between the plurality of pixel values obtained from the plurality of pixels.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a photoelectric conversion device, an apparatus, and a signal processing method. [Background technology]

[0002] An imaging device has been devised that includes frame pixels for detecting image data and difference detection pixels for obtaining data based on the difference between multiple pixel values. Patent Document 1 proposes an imaging device that performs so-called time difference detection, which outputs information indicating that there is a difference between multiple pixel values ​​at different times for the same pixel. Since the time difference detection pixels can output without synchronizing with the frame, they have the characteristic of being able to process at high speed, and may be used, for example, as an in-vehicle camera to detect pedestrians and oncoming vehicles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-96347 A Summary of the Invention [Problem to be solved by the invention]

[0004] As in Patent Document 1, in the case of an imaging device that performs time difference detection, there was a problem with the accuracy of difference detection depending on the input image. For example, when used as an in-vehicle camera, if both the car and the detection target are stopped, there is a problem that it is difficult to detect the detection target because there is no or only a small difference between multiple pixel values ​​at different times for the same pixel.

[0005] An object of the present disclosure is to provide a technique for improving the detection accuracy of a detection target using difference detection pixels. [Means for solving the problem]

[0006] One aspect of the present disclosure is a photoelectric conversion device having a plurality of pixels for obtaining pixel values ​​based on the amount of light incident on each pixel, a detection unit for detecting the difference between the plurality of pixel values, and a control unit for controlling the detection unit, wherein the control unit causes the detection unit to perform a first difference detection operation for detecting the difference between the plurality of pixel values ​​from the same pixel, and a second difference detection operation for detecting the difference between the plurality of pixel values ​​from the plurality of pixels. Effect of the Invention

[0007] According to at least one embodiment of the present disclosure, it is possible to improve the detection accuracy of a detection target using pixels for difference detection. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to a first embodiment; [Diagram 2] Example of pixel arrangement in pixel block according to the first embodiment [Diagram 3] FIG. 1 is a block diagram showing a configuration of a detection unit according to a first embodiment; [Figure 4] Operation flowchart according to the first embodiment [Diagram 5] Operational flow chart according to the second embodiment [Figure 6] FIG. 13 is a block diagram showing the configuration of an imaging apparatus according to a third embodiment. [Figure 7] Operational flow chart according to the third embodiment [Figure 8] FIG. 13 is a block diagram showing the configuration of an imaging apparatus according to a third embodiment. [Figure 9] FIG. 13 is a schematic diagram showing a program according to a third embodiment; [Figure 10] Operational flow chart according to the third embodiment [Figure 11] FIG. 13 is a schematic diagram showing a captured image according to a fourth embodiment; [Figure 12] FIG. 13 is a schematic diagram showing a setting example according to the fourth embodiment; [Figure 13]Operational flow chart according to the fourth embodiment [Figure 14] FIG. 13 is a block diagram showing the configuration of a detection unit according to a fifth embodiment. [Figure 15] FIG. 13 is a block diagram showing the configuration of a detection unit according to a fifth embodiment. [Figure 16] FIG. 13 is a schematic diagram showing an example of the arrangement of pixel blocks according to the fifth embodiment; [Figure 17] FIG. 13 is a schematic diagram showing a configuration of an apparatus according to a sixth embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following embodiments, an image pickup device will be mainly described as an example of a photoelectric conversion device. However, each embodiment is not limited to an image pickup device, and can be applied to other examples of photoelectric conversion devices. For example, a distance measuring device (a device for measuring distance using focus detection or TOF (Time Of Flight)) or a photometric device (a device for measuring the amount of incident light) may be used.

[0010] Furthermore, the disclosure of this specification includes the complement of the concepts described in this specification. In other words, if this specification contains a statement that "A is greater than B," even if the statement that "A is not greater than B" is omitted, this specification can be said to disclose that "A is not greater than B." This is because when a statement that "A is greater than B" is made, it is assumed that the case in which "A is not greater than B" is taken into consideration.

[0011] (First embodiment) 1 is a block diagram showing the hardware configuration of an image pickup apparatus according to this embodiment. The image pickup apparatus 10 includes a control unit 110, a detection unit 120, a post-processing unit 180, and an output IF (interface) unit 190.

[0012] The detection unit 120 has a plurality of pixel blocks 130. In the following description of the present embodiment, an example is shown in which one pixel block 130 is composed of 16 pixels P, but the number of pixels P included in the pixel block 130 is not limited to this.

[0013] 2 is an example of an arrangement of pixels P in a pixel block 130. W1 to W4, R1 to R4, and B1 to B4 are frame pixels used to obtain image data, and E1 to E4 are difference detection pixels used to output data based on the difference between a plurality of pixel values. The arrangement in Fig. 2 is one example, and for example, a configuration in which pixels for distance measurement that measure the distance to the subject are mixed in may also be used.

[0014] The frame pixels output pixel values ​​based on a fixed cycle. Here, W1 to W4, R1 to R4, and B1 to B4 are pixels using white, red, and blue color filters, respectively. However, this is just one example, and W1 to W4, R1 to R4, and B1 to B4 may be configured to use different color filters or may be configured to be used as black and white pixels without using color filters.

[0015] The difference detection pixels output pixel values ​​asynchronously with the period used by the frame pixels. The pixel values ​​output from the difference detection pixels are used, for example, to detect moving objects within the imaging area of ​​the imaging device by performing various difference detection modes described later. Preferably, the difference detection pixels are used as an in-vehicle camera to detect pedestrians and oncoming vehicles.

[0016] The post-processing unit 180 performs processing on the output from the detection unit 120. For example, it performs image processing based on the output of each frame pixel, difference detection processing based on the output of the difference detection pixel, and moving object recognition processing based on the detected difference. The results of these processes are output from the post-processing unit 180 to the control unit 110 and the output IF 190. The control unit 110 receives the processing results from the post-processing unit 180 and controls the detection unit 120 based on the processing results. The output IF unit 190 also outputs the output from the post-processing unit 180 to an external device.

[0017] 3 is a block diagram showing the configuration of the detection unit 120. Although only one pixel block 130 is shown in the figure, the detection unit 120 may have multiple pixel blocks 130. Each pixel P of the pixel block 130 outputs a pixel value according to the amount of incident light to a subsequent stage in response to an event synchronization signal and a frame synchronization signal from the control unit 110. The storage unit 320 stores the pixel value of each pixel P. The selection unit 330 selects an input to the subtraction unit 340 in response to a detection control signal from the control unit 110.

[0018] The subtraction unit 340 calculates the difference between two specific pixel values ​​among the multiple pixel values ​​input. The subtraction unit 340 is provided with one or multiple subtraction circuits. In this embodiment, since four difference detection pixels are arranged in one pixel block 130, the multiple subtraction circuits are configured to be able to simultaneously perform four sets of subtraction.

[0019] For example, in a time difference mode described later, subtraction is performed between a pixel value detected at a certain time in each of E1 to E4 and four sets of pixel values ​​detected immediately before that time. Also, for example, in a spatial difference mode described later, subtraction is performed between four sets of pixel values ​​detected from adjacent difference detection pixels, i.e., E1 and E2, E1 and E3, E2 and E4, and E3 and E4. The number of sets to be subtracted is not limited to this, and may be other numbers depending on the number of difference detection pixels and the number of subtraction circuits arranged.

[0020] Threshold control section 350 sets a threshold used in the calculation of comparison section 360. Comparison section 360 compares the value calculated by subtraction section 340 with the threshold set by threshold control section 350, and outputs if the value exceeds the threshold. Subtraction section 340, threshold control section 350, and comparison section 360 configure a comparison means. IF section 370 outputs the processing result of comparison section 360 to the subsequent stage.

[0021] The difference detection operation in this embodiment will be described below with reference to the flowchart in Fig. 4. The operation described below is based on the assumption that the imaging device is mounted on a vehicle as an on-board camera.

[0022] When the start of difference detection is instructed in step S401, the process proceeds to step S402. In step S402, the moving status of the vehicle is judged. That is, it is judged whether the vehicle is moving (moving) or stopped. For the judgment, a known technique such as a technique using an acceleration sensor may be used. If it is judged that the vehicle is moving, the process proceeds to step S403, and if it is judged that the vehicle is stopped, the process proceeds to step S404.

[0023] In step S403, the selection unit 330 receives a detection control signal from the control unit 110 and selects an input to the subtraction unit 340. In this step, the pixel value from pixel P and the pixel value output immediately before from the same pixel P read out from the storage unit 320 are selected. That is, each combination of difference detection pixels E1 and the pixel value of the previous E1, E2 and the pixel value of the previous E2, E3 and the pixel value of the previous E3, and E4 and the pixel value of the previous E4 are input to the subtraction unit 340.

[0024] A time difference value of the same pixel is derived by subtracting these inputs in subtraction unit 340. The time difference value is compared in comparison unit 360 with a threshold set in threshold control unit 350, and the result is output to post-processing unit 180 via IF unit 370. Such an operation mode is hereinafter referred to as a time difference mode.

[0025] In step S404, the selection unit 330 receives a detection control signal from the control unit 110 and selects an input to the subtraction unit 340. In this step, pixel values ​​from two adjacent difference detection pixels are selected as inputs. Here, adjacent pixels include not only directly adjacent pixels, such as E1 and E2, E3 and E4, E1 and E3, and E2 and E4, but also adjacent pixels adjacent via a frame pixel.

[0026] These inputs (pairs of pixel values) are subtracted from each other in subtraction unit 340, whereby the spatial difference value of two adjacent difference detection pixels is calculated in subtraction unit 340. The spatial difference value is compared in comparison unit 360 with a threshold set in threshold control unit 350, and the result is output to post-processing unit 180 via IF unit 370. Such an operation mode is hereinafter referred to as the spatial difference mode.

[0027] In step S405, it is determined whether or not an instruction to end the difference detection has been given, and if not, the process returns to step S402 to continue, whereas if an instruction to end the difference detection has been given, the difference detection process ends.

[0028] Through the above operations, it is possible to select and output difference information in either the time difference mode or the spatial difference mode depending on the state of the imaging device 10. In this way, in cases where it is difficult to recognize the detection target in one difference detection mode, it is possible to switch to the other difference detection mode, which has the effect of improving the detection accuracy of the detection target.

[0029] In this embodiment, the time difference mode and the space difference mode are switched depending on the state of the car, but other configurations may be used. For example, consider a case where an autofocus operation (focus detection operation) is performed using difference detection pixels in an imaging device. Since the autofocus operation searches for an appropriate focus state based on pixel values, if there is no output of pixel values, appropriate focus operation cannot be performed. For example, when processing is performed in the time difference mode, if there is no moving object within the shooting range of the imaging device, pixel values ​​are not output, and therefore focus operation cannot be performed.

[0030] Therefore, the determination step of step S402 in the operational flowchart of FIG. 4 may be a step for determining whether or not an autofocus operation is being performed, and if not, the time difference mode may be selected, and if performed, the spatial difference mode may be selected.

[0031] As another configuration, the time difference mode and the spatial difference mode may be switched depending on the presence or absence of illuminance fluctuation. If the time difference mode is used when there is a fluctuation in illuminance over the entire captured image, a difference will be detected over the entire image even if there is no moving object in the image. Therefore, the presence or absence of a fluctuation in illuminance may be detected, and the difference detection mode may be switched to the spatial difference mode. In this case, the illuminance detection unit that detects the illuminance may be realized by a known technology. For example, a part of the pixels of the imaging device 10 may be used as the illuminance detection unit, or a light meter or the like may be mounted on the imaging device 10 or a part of the car other than the imaging device to detect the illuminance.

[0032] In the above, the detection unit 120 has been described as having both frame pixels and difference detection pixels, but the detection unit 120 may be configured with only difference detection pixels. The pixels in the pixel block 130 may have an arrangement other than that shown in Fig. 2. The number of threshold values ​​set in the threshold control unit 350 is not limited to one, and multiple threshold values ​​may be set. For example, it may be possible to determine which of the multiple threshold values ​​has been exceeded and display the result in an identifiable manner on the image.

[0033] In addition, the difference detection pixels may be configured to use color filters, or may be configured to be used as black-and-white pixels without using color filters. When multiple color filters are used for the difference detection pixels, in the spatial difference mode, it is preferable to detect differences from multiple difference detection pixels provided with the same color filters.

[0034] Second embodiment In this embodiment, the difference detection mode is selected according to the comparison result in the comparison section 360. Note that only the points different from the above embodiment will be described.

[0035] 5 is a flowchart showing the operation of this embodiment. Steps that perform the same processes as in the previous embodiment are given the same numbers.

[0036] In step S401, when an instruction to start difference detection is given, the process proceeds to step S403, where processing starts in the time difference mode.

[0037] In step S501, it is determined whether there is an output from the comparison unit 360 within a preset fixed time. For example, when used as an in-vehicle camera, if there is no output in the time difference mode for 10 minutes or more, the car is likely to be parked in a location with few moving objects around, and in such a case, the mode is switched to the spatial difference mode. If there is no output within the fixed time, the process proceeds to step S404, where the process switches to the spatial difference mode. If there is an output within the fixed time, the process proceeds to step S403, where the process continues in the time difference mode.

[0038] In step S502, it is determined whether a preset fixed time has elapsed since switching to the spatial difference mode. If so, the process proceeds to step S403, where the process switches to the temporal difference mode. If not, the process proceeds to step S404, where the process continues in the spatial difference mode.

[0039] By executing each of the above steps, if a state of no output continues, it becomes possible to switch the difference detection mode and output information, which has the effect of improving the accuracy of various processes such as recognition processing in the post-processing unit 180.

[0040] In addition, for example, when used as a surveillance camera in a place with few moving objects, if there is no output in the time difference mode for a long period of time, it may be impossible to determine whether there is no moving object in the imaging area or whether the device is broken. According to this embodiment, by providing a detection method using the spatial difference mode, it is possible to output difference information even if there is no movement in the imaging area, which has the effect of making it easy to check whether there is a malfunction.

[0041] In the above, in step S501, the time difference mode is switched to the spatial difference mode when there is no output for a certain period of time, but this is not limited to the above. For example, the time difference mode may remain when there is no output, and the time difference mode may be switched to the spatial difference mode when there is output. In this case, since no output operation is performed when there is no change in the imaging area, the power consumption of the device can be reduced. Furthermore, the interval of time difference detection may be lengthened when the time when there is no output continues. In this case, the power consumption can be further reduced.

[0042] (Third embodiment) In this embodiment, the difference detection mode is switched according to a preset program. Note that only the differences from the previous embodiment will be described.

[0043] Fig. 6 shows an example of a hardware configuration block diagram in this embodiment. 601 is a storage unit that stores a program based on a switching pattern of the difference detection mode. Fig. 7 shows an operation flowchart in this embodiment. The same numbers are used for steps that perform the same processes as in the previous embodiment.

[0044] In step S701, a program in which a switching pattern for the difference detection mode is recorded is set in storage unit 601, and the process proceeds to step S702. In the following explanation, an example will be described in which 9 out of 10 difference detections are performed in the time difference mode and 1 in the spatial difference mode.

[0045] In step S702, a counter value N=0 is set in a detection counter (not shown) provided in the control unit 110, and the process proceeds to step S703.

[0046] In step S703, it is determined whether the counter value is N=9, and if N=9, the process proceeds to step S403 where processing is performed in the time difference mode, or if N=9, the process proceeds to step S404 where processing is performed in the spatial difference mode.

[0047] In step S704, the counter value N is set to N+1, and the process proceeds to S405. On the other hand, in step S705, the counter value N is set to 0, and the process proceeds to S405.

[0048] By the above operation, it becomes possible to switch between the time difference mode and the space difference mode in accordance with a preset operation program.

[0049] The program to be set may be switched based on time information. A block diagram of the hardware configuration in this embodiment is shown in Fig. 8. Time information can be read from an RTC (real-time clock) 801.

[0050] A schematic diagram of a program based on time information is shown in Figure 9. The time difference mode is set from 8:00 to 16:00 from Monday to Friday, and the space difference mode is set for the rest of the time period. This type of setting is an example that is suitable for surveillance cameras in an office.

[0051] An operational flowchart of this embodiment is shown in Figure 10. The same numbers are given to steps that perform the same processes as in the previous embodiment.

[0052] In step S1001, a program in which time and difference detection modes are associated with each other as shown in FIG. 9 is stored in the storage unit 601, and the process proceeds to step S401.

[0053] In step S1002, time information is read from the RTC 801, and a decision is made based on the program as to which difference detection mode to use. If the time difference mode is selected, the process proceeds to step S403 where the time difference mode is selected, and if the spatial difference mode is selected, the process proceeds to step S404 where the spatial difference mode is selected. The flowchart in Fig. 10 shows an example in which the time difference mode is selected during the day and the spatial difference mode is selected for other time periods, but the correspondence between the time and the difference detection mode is not limited to this.

[0054] According to the above, it is possible to switch to a suitable detection difference method in accordance with a preset program, and there is an effect of improving the accuracy of various processes such as the recognition process in the post-processing unit 180.

[0055] (Fourth embodiment) In the above-described embodiment, the difference detection mode switching is applied uniformly to all difference detection pixels, but in this embodiment, an example will be described in which this is applied, for example, to each pixel block 130. In this configuration, a detection control signal is sent from the control unit 110 to each pixel block 130 individually.

[0056] Fig. 11 shows a schematic diagram of an image captured by a surveillance camera installed outdoors. Vehicles pass along the road that crosses the center of the image, so moving objects are frequently detected, but there are almost no moving objects in other areas. Fig. 12 shows an example of setting the difference detection mode for each pixel block 130 in such a case. That is, the area including the road is set to the time difference mode, and the other areas are set to the spatial difference mode.

[0057] An example of an operation flowchart in this embodiment is shown in Fig. 13. The same numbers are given to steps that perform the same processes as in the previous embodiment.

[0058] In step S1301, the difference detection mode settings for each of the multiple pixel blocks 130 are stored in the storage unit 601, and the process proceeds to step S401.

[0059] In step S1302, the setting for the first pixel block 130 is read from the storage unit 601, and the process proceeds to step S1303.

[0060] In step S1303, it is determined which difference detection mode is set for processing the first pixel block 130. If the setting is the time difference mode, the process proceeds to step S403, and if the setting is the spatial difference mode, the process proceeds to step S404. Fig. 13 shows an example in which it is determined whether the difference detection mode is the time difference mode, and if the determination is Yes, processing is performed in the time difference mode, and if the determination is No, processing is performed in the spatial difference mode.

[0061] In step S1304, it is determined whether the difference detection process has been completed for all pixel blocks 130. If completed, the process proceeds to step S405, and if not completed, the process proceeds to step S1305.

[0062] In step S1305, the settings for the next pixel block 130 are read from the storage unit 601, and the process returns to step S1303 to repeat the process.

[0063] In step S405, it is determined whether detection has ended, and if not, the process returns to step S1302 and the process is repeated from the first pixel block 130.

[0064] According to the above, by setting the difference detection mode in accordance with the characteristics of each region of the captured image, there is an effect of improving the accuracy of various processes such as the recognition process in the post-processing unit 180. Also, in the above explanation, the difference detection mode is switched for each pixel block 130, but this is not limited thereto, and the difference detection mode may be switched for each set of multiple pixel blocks 130, for example.

[0065] Fifth embodiment In the first to fourth embodiments, the pixels in the pixel block 130 are configured so that both a time difference mode and a spatial difference mode can be selected as the difference detection mode, but a pixel block 130 equipped with only one of the modes may be selectively arranged on the imaging device.

[0066] Fig. 14 shows the configuration of detection section 120 when only the time difference mode is provided. Fig. 14 shows a configuration in which selection section 330 is removed from the configuration in Fig. 3. Fig. 15 shows a configuration in which only the spatial difference mode is provided. This configuration further shows that storage section 320 is removed from the configuration in Fig. 14.

[0067] 16 shows an example in which pixel block 1401 equipped only with a time difference mode and pixel block 1501 equipped only with a spatial difference mode are arranged with respect to detection unit 120. Pixel block 1401 is arranged at the position indicated by "T", and pixel block 1501 is arranged at the position indicated by "A". Pixel block 1401 and pixel block 1501 output information according to their respective difference detection modes.

[0068] According to this embodiment, it is possible to output using two difference detection modes, while having the effect of reducing the circuit scale and power consumption as compared to the above-described embodiments.

[0069] Sixth embodiment The sixth embodiment can be applied to any of the first to fifth embodiments. Fig. 17(a) is a schematic diagram for explaining an apparatus 9191 including a semiconductor device 930 according to this embodiment. The semiconductor device 930 can be a photoelectric conversion device (imaging device) according to each of the above-mentioned embodiments.

[0070] An apparatus 9191 including a semiconductor device 930 will be described in detail. As described above, the semiconductor device 930 can include a package 920 that houses the semiconductor device 910, in addition to the semiconductor device 910 having the semiconductor layer 902. The package 920 can include a base to which the semiconductor device 910 is fixed, and a lid such as glass that faces the semiconductor device 910. The package 920 can further include a bonding member such as a bonding wire or a bump that connects a terminal provided on the base and a terminal provided on the semiconductor device 910.

[0071] The device 9191 can include at least one of an optical device 940, a control device 950, a processing device 960, a display device 970, a storage device 980, and a mechanical device 990. The optical device 940 corresponds to the semiconductor device 930. The optical device 940 is, for example, a lens, a shutter, or a mirror. The control device 950 controls the semiconductor device 930. The control device 950 is, for example, a semiconductor device such as an ASIC.

[0072] The processing device 960 processes the signal output from the semiconductor device 930. The processing device 960 is a semiconductor device such as a CPU or ASIC for configuring an AFE (analog front end) or a DFE (digital front end). The display device 970 is an EL display device or a liquid crystal display device that displays information (images) obtained by the semiconductor device 930. The storage device 980 is a magnetic device or a semiconductor device that stores information (images) obtained by the semiconductor device 930. The storage device 980 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0073] The mechanical device 990 has a moving part or a propulsion part such as a motor or an engine. In the device 9191, the signal output from the semiconductor device 930 is displayed on the display device 970, or transmitted to the outside by a communication device (not shown) included in the device 9191. For this purpose, the device 9191 preferably further includes a memory device 980 and a processing device 960 in addition to the memory circuit and arithmetic circuit included in the semiconductor device 930. The mechanical device 990 may be controlled based on the signal output from the semiconductor device 930.

[0074] The device 9191 is also suitable for electronic devices such as information terminals (e.g., smartphones and wearable devices) with a photographing function and cameras (e.g., interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras). The mechanical device 990 in the camera can drive components of the optical device 940 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 990 in the camera can move the semiconductor device 930 for vibration isolation operation.

[0075] The device 9191 may be transportation equipment such as a vehicle, a ship, or an aircraft. The mechanical device 990 in the transportation equipment may be used as a moving device. The device 9191 as a transportation equipment is suitable for transporting the semiconductor device 930 or for assisting and / or automating driving (piloting) by using a photographing function. The processing device 960 for assisting and / or automating driving (piloting) can perform processing for operating the mechanical device 990 as a moving device based on information obtained by the semiconductor device 930. Alternatively, the device 9191 may be a medical device such as an endoscope, a measuring device such as a distance measuring sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0076] According to the above-described embodiment, it is possible to obtain good pixel characteristics. Therefore, the value of the semiconductor device can be increased. In this case, increasing the value corresponds to at least one of adding functions, improving performance, improving characteristics, improving reliability, improving manufacturing yield, reducing environmental load, reducing costs, reducing size, and reducing weight.

[0077] Therefore, if the semiconductor device 930 according to this embodiment is used in the equipment 9191, the value of the equipment can be improved. For example, by mounting the semiconductor device 930 on a transport equipment, excellent performance can be obtained when photographing the outside of the transport equipment or measuring the external environment. Therefore, in manufacturing and selling the transport equipment, it is advantageous to decide to mount the semiconductor device according to this embodiment on the transport equipment in order to improve the performance of the transport equipment itself. In particular, the semiconductor device 930 is suitable for transport equipment that performs driving assistance and / or automatic driving of the transport equipment using information obtained by the semiconductor device.

[0078] The photoelectric conversion system and the moving object of this embodiment will be described with reference to FIGS. 17(b) and 17(c).

[0079] Fig. 17(b) shows an example of a photoelectric conversion system related to an in-vehicle camera. The photoelectric conversion system 8000 has a photoelectric conversion device 80. The photoelectric conversion device 80 is the photoelectric conversion device (imaging device) described in any of the above embodiments. The photoelectric conversion system 8000 has an image processing unit 801 that performs image processing on a plurality of image data acquired by the photoelectric conversion device 80, and a parallax acquisition unit 802 that calculates parallax (phase difference of parallax images) from the plurality of image data acquired by the photoelectric conversion system 8000.

[0080] The photoelectric conversion system 8000 also includes a distance acquisition unit 803 that calculates the distance to the object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 802 and the distance acquisition unit 803 are examples of distance information acquisition means that acquire information on the distance to the object. That is, the distance information is information on the parallax, the defocus amount, the distance to the object, and the like. The collision determination unit 804 may determine the possibility of collision using any of these pieces of distance information.

[0081] The distance information acquisition means may be realized by dedicated hardware, a software module, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination of these.

[0082] The photoelectric conversion system 8000 is connected to a vehicle information acquisition device 810 and can acquire vehicle information such as vehicle speed, yaw rate, steering angle, etc. The photoelectric conversion system 8000 is also connected to a control ECU 820, which is a control device that outputs a control signal for generating a braking force for the vehicle based on the determination result of the collision determination unit 804. The photoelectric conversion system 8000 is also connected to an alarm device 830 that issues an alarm to the driver based on the determination result of the collision determination unit 804.

[0083] For example, when the collision probability is high as a result of the judgment by the collision judgment unit 804, the control ECU 820 performs vehicle control to avoid the collision and reduce damage by applying the brakes, releasing the accelerator, suppressing the engine output, etc. The warning device 830 warns the user by sounding a warning, displaying warning information on the screen of a car navigation system, etc., applying vibrations to the seat belt or steering wheel, etc.

[0084] In this embodiment, the surroundings of the vehicle, for example, the front or rear, are imaged by the photoelectric conversion system 8000. Fig. 17(c) shows the photoelectric conversion system when imaging the area in front of the vehicle (imaging range 850). The vehicle information acquisition device 810 sends an instruction to the photoelectric conversion system 8000 or the photoelectric conversion device 80. With this configuration, the accuracy of distance measurement can be further improved.

[0085] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from lanes, etc. Furthermore, the photoelectric conversion system is not limited to vehicles such as the vehicle itself, but can be applied to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the present invention can be applied not only to moving bodies, but also to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0086] (Other embodiments) It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features. For example, combinations of the elements of the above-mentioned embodiments are also within the scope of the present disclosure.

[0087] In addition, each of the embodiments described above can be appropriately modified without departing from the scope of the technical concept. The disclosure of this specification includes not only what is described in this specification, but also all matters that can be understood from this specification and the drawings attached to this specification.

[0088] The disclosure of this embodiment includes the following configurations and methods.

[0089] (Configuration 1) a detection unit having a plurality of pixels for obtaining pixel values ​​based on the amount of incident light to each pixel, the detection unit detecting a difference between the plurality of pixel values; A control unit that controls the detection unit, The control unit causes the detection unit to perform a first difference detection operation for detecting a difference between a plurality of the pixel values ​​of the same pixel, and a second difference detection operation for detecting a difference between a plurality of the pixel values ​​of the plurality of the pixels. A photoelectric conversion device comprising:

[0090] (Configuration 2) the plurality of pixels includes a plurality of difference detection pixels, The photoelectric conversion device according to configuration 1, wherein the control unit causes the detection unit to detect differences between the pixel values ​​of the plurality of closest difference detection pixels in the second difference detection operation.

[0091] (Configuration 3) The difference detection pixel further includes a color filter, The photoelectric conversion device described in configuration 2, wherein in the second difference detection operation, the control unit causes the detection unit to detect differences between a plurality of the pixel values ​​using the difference detection pixels having the color filters of the same color.

[0092] (Configuration 4) The photoelectric conversion device described in configuration 3, wherein in the second difference detection operation, the control unit causes the detection unit to detect differences in the pixel values ​​using the closest difference detection pixels among the difference detection pixels having the color filters of the same color.

[0093] (Configuration 5) 5. The photoelectric conversion device according to any one of configurations 2 to 4, wherein the plurality of pixels include pixels for obtaining image data.

[0094] (Configuration 6) The detection unit has a subtraction means for deriving a difference between a plurality of the pixel values, a selection means for selecting the pixel value to be input to the subtraction means in response to a signal from the control unit, and a means for comparing the difference with a predetermined value and outputting a result of the comparison to an outside of the detection unit. 6. The photoelectric conversion device according to any one of configurations 1 to 5, characterized in that:

[0095] (Configuration 7) A movement detection unit that detects whether the photoelectric conversion device is moving or stopped is further provided, The control unit causes the detection unit to perform either the first difference detection operation or the second difference detection operation depending on a result of detection by the movement detection unit. 7. The photoelectric conversion device according to any one of configurations 1 to 6, characterized in that:

[0096] (Configuration 8) The imaging device described in configuration 7, characterized in that the control unit, when the movement detection unit detects that the photoelectric conversion device is moving, causes the detection unit to perform the first difference detection operation, and when the movement detection unit detects that the photoelectric conversion device is stopped, causes the detection unit to perform the second difference detection operation.

[0097] (Configuration 9) The control unit causes the detection unit to perform the second difference detection operation when the photoelectric conversion device performs a focus detection operation based on the difference. 7. The photoelectric conversion device according to any one of configurations 1 to 6, characterized in that:

[0098] (Configuration 10) The illumination detecting unit further includes: The control unit causes the detection unit to perform the second difference detection operation when a fluctuation in illuminance is detected by the illuminance detection unit. 7. The photoelectric conversion device according to any one of configurations 1 to 6, characterized in that:

[0099] (Configuration 11) The control unit causes the detection unit to perform the second difference detection operation in response to a result of the first difference detection operation. 7. The photoelectric conversion device according to any one of configurations 1 to 6, characterized in that:

[0100] (Configuration 12) The control unit causes the detection unit to perform the second difference detection operation when a difference is not detected within a predetermined period in the first difference detection operation. 12. The photoelectric conversion device according to claim 11,

[0101] (Configuration 13) When a difference is detected within a predetermined period in the first difference detection operation, the control unit causes the detection unit to perform the second difference detection operation. 12. The photoelectric conversion device according to claim 11,

[0102] (Configuration 14) Further comprising a storage unit, The photoelectric conversion device described in any one of configurations 1 to 6, characterized in that the control unit causes the detection unit to perform either the first difference detection operation or the second difference detection operation in accordance with a program stored in the memory unit.

[0103] (Configuration 15) 15. The photoelectric conversion device according to configuration 14, wherein the program is based on time information.

[0104] (Configuration 16) the detection unit has a plurality of regions, each region including at least one pixel of the plurality of pixels; The control unit causes the detection unit to perform either the first difference detection operation or the second difference detection operation for each of the plurality of regions. 16. The photoelectric conversion device according to any one of configurations 1 to 15, characterized in that:

[0105] (Configuration 17) An apparatus including the photoelectric conversion device according to any one of configurations 1 to 16, an optical device corresponding to the photoelectric conversion device; A control device for controlling the photoelectric conversion device; a processing device that processes a signal output from the photoelectric conversion device; a display device that displays information obtained by the photoelectric conversion device; a storage device that stores information obtained by the photoelectric conversion device; and and a mechanical device that operates based on information obtained by the photoelectric conversion device.

[0106] (Configuration 18) A device including a photoelectric conversion device having a plurality of pixels for obtaining a pixel value based on an amount of incident light to each pixel, a detection unit for detecting a difference between the plurality of pixel values, and a control unit for controlling the detection unit, The control unit causes the detection unit to perform a first difference detection operation for detecting a difference between a plurality of the pixel values ​​of the same pixel, and a second difference detection operation for detecting a difference between a plurality of the pixel values ​​of the plurality of the pixels. The device characterized by:

[0107] (Method 1) A signal processing method for an apparatus including a photoelectric conversion device including a plurality of pixels for obtaining a pixel value based on an amount of incident light to each pixel, a detection unit for detecting a difference between the plurality of pixel values, and a control unit for controlling the detection unit, comprising: The control unit has a selection step of causing the detection unit to perform a first difference detection operation for detecting a difference between a plurality of the pixel values ​​of the same pixel, and a second difference detection operation for detecting a difference between a plurality of the pixel values ​​of the plurality of the pixels. A signal processing method comprising: [Explanation of symbols]

[0108] 10. Imaging device 110 Control section 120 Detection unit P pixels

Claims

1. A detection unit has multiple pixels for obtaining pixel values ​​based on the amount of light incident on each of them, and detects the difference between the multiple pixel values, A photoelectric conversion device comprising a control unit for controlling the detection unit, In order to detect the movement of the subject, the control unit causes the detection unit to perform a first difference detection operation to detect the difference between multiple pixel values ​​of the same pixel, and a second difference detection operation to detect the difference between multiple pixel values ​​of multiple pixels. A photoelectric conversion device characterized by the following.

2. The plurality of pixels include a plurality of difference detection pixels, The photoelectric conversion apparatus according to claim 1, characterized in that the control unit causes the detection unit to detect the difference between a plurality of pixel values ​​by the plurality of closest difference detection pixels in the second difference detection operation.

3. The aforementioned difference detection pixel further comprises a color filter, The photoelectric converter according to claim 2, characterized in that the control unit causes the detection unit to detect the difference between a plurality of pixel values ​​using the difference detection pixels provided with the color filter of the same color in the second difference detection operation.

4. The photoelectric converter according to claim 3, characterized in that, in the second difference detection operation, the control unit causes the detection unit to detect the difference between a plurality of pixel values ​​using the closest difference detection pixel among the difference detection pixels provided with the same color filter.

5. The photoelectric conversion device according to claim 2, characterized in that the plurality of pixels include pixels for obtaining image data.

6. The detection unit includes subtraction means for deriving the difference between a plurality of pixel values, selection means for selecting the pixel value to be input to the subtraction means in accordance with a signal from the control unit, and means for comparing the difference with a predetermined value and outputting the result of the comparison to the outside of the detection unit. The photoelectric conversion device according to claim 1, characterized by the following:

7. The photoelectric converter further comprises a movement detection unit that detects whether the device is moving or stopped. The control unit causes the detection unit to perform either the first difference detection operation or the second difference detection operation, depending on the result of the detection by the movement detection unit. The photoelectric conversion device according to claim 1, characterized by the following:

8. The imaging apparatus according to claim 7, characterized in that the control unit causes the detection unit to perform the first difference detection operation when the movement detection unit detects that the photoelectric converter is moving, and causes the detection unit to perform the second difference detection operation when the movement detection unit detects that the photoelectric converter is stopped.

9. The control unit, when the photoelectric converter performs a focus detection operation based on the difference, causes the detection unit to perform the second difference detection operation. The photoelectric conversion device according to claim 1, characterized by the following:

10. It further includes an illuminance detection unit, The control unit, when the illuminance detection unit detects a change in illuminance, causes the detection unit to perform the second difference detection operation. The photoelectric conversion device according to claim 1, characterized by the following:

11. The control unit causes the detection unit to perform the second difference detection operation according to the result of the first difference detection operation. The photoelectric conversion device according to claim 1, characterized by the following:

12. If no difference is detected within a predetermined period in the first difference detection operation, the control unit causes the detection unit to perform the second difference detection operation. The photoelectric conversion device according to claim 9, characterized by the following:

13. If a difference is detected within a predetermined period during the first difference detection operation, the control unit causes the detection unit to perform the second difference detection operation. The photoelectric conversion device according to claim 11, characterized by the following:

14. It also has a memory unit, The photoelectric converter according to claim 1, characterized in that the control unit causes the detection unit to perform either the first difference detection operation or the second difference detection operation according to a program stored in the storage unit.

15. The photoelectric converter according to claim 14, characterized in that the program is based on time information.

16. The detection unit has a plurality of regions, each region containing at least one pixel of the plurality of pixels, The control unit causes the detection unit to perform either the first difference detection operation or the second difference detection operation for each of the plurality of regions. The photoelectric conversion device according to claim 1, characterized by the following:

17. A device comprising a photoelectric converter according to any one of claims 1 to 16, Optical device corresponding to the aforementioned photoelectric converter, A control device for controlling the aforementioned photoelectric converter, A processing device that processes the signal output from the aforementioned photoelectric converter, A display device that displays information obtained by the aforementioned photoelectric converter. A storage device for storing information obtained by the photoelectric converter, and The apparatus is characterized by further comprising at least one of the following: a mechanical device that operates based on information obtained from the photoelectric converter.

18. A device comprising: a photoelectric converter having a plurality of pixels for obtaining pixel values ​​based on the amount of incident light on each; a detection unit for detecting the difference between the plurality of pixel values; a post-processing unit for detecting the movement of an object using the output from the detection unit; and a control unit for controlling the detection unit, The control unit causes the detection unit to perform a first difference detection operation to detect the difference between multiple pixel values ​​of the same pixel, and a second difference detection operation to detect the difference between multiple pixel values ​​of multiple pixels. A device characterized by the following.

19. A signal processing method for a device comprising: a photoelectric converter having a plurality of pixels for obtaining pixel values ​​based on the amount of incident light to each; a detection unit for detecting the difference between the plurality of pixel values; and a control unit for controlling the detection unit, To detect the movement of a subject, the control unit has a selection step of causing the detection unit to perform a first difference detection operation that detects the difference between multiple pixel values ​​of the same pixel, and a second difference detection operation that detects the difference between multiple pixel values ​​of multiple pixels. A signal processing method characterized by the following:

Images