Imaging device, imaging element, imaging method, and system

Abstract

The present disclosure relates to an imaging device, an imaging element, an imaging method, and a system that make it possible to acquire an edge image having an appropriate characteristic. Provided is an imaging device comprising: an acquisition unit that acquires an edge image, which is an image obtained by extracting an edge; a control unit that performs control to evaluate an edge characteristic, which is a characteristic related to an edge obtained from the acquired edge image, and perform feedback so that an appropriate edge characteristic can be obtained from the edge image acquired by the acquisition unit; and a signal processing unit that performs signal processing using the acquired edge image. The present disclosure can be applied to, for example, an imaging device that acquires an edge image.

Description

Imaging device, imaging element, imaging method, and system

[0001] The present disclosure relates to an imaging device, an imaging element, an imaging method, and a system, and more particularly to an imaging device, an imaging element, an imaging method, and a system that are capable of acquiring an edge image having appropriate characteristics.

[0002] In order to perform recognition with low power consumption, a sensor has been proposed that performs edge extraction using an analog circuit and performs signal processing using the resulting edge image. Patent Document 1 discloses a technology for extracting edge information from image information output from a pixel array unit having a plurality of pixels arranged two-dimensionally. Patent Document 2 discloses a technology for performing edge extraction and edge evaluation on an RGB image in a system that performs recognition processing using an RGB image, and controls an imaging device based on the evaluation value.

[0003] JP 2021-150814 A JP 2011-188094 A

[0004] Due to their characteristics, sensors that capture edge images are easily affected by the surrounding environment and the movement of the subject, making it difficult to stabilize the characteristics of edge images. Therefore, there was a demand for a technology to capture edge images with appropriate characteristics.

[0005] The present disclosure has been made in view of such circumstances, and makes it possible to acquire an edge image having appropriate characteristics.

[0006] An imaging device according to one aspect of the present disclosure is an imaging device that includes an acquisition unit that acquires an edge image, which is an image in which edges have been extracted; a control unit that evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit; and a signal processing unit that performs signal processing using the acquired edge image.

[0007] An imaging element according to one aspect of the present disclosure is an imaging element that includes an acquisition unit that acquires an edge image, which is an image in which an edge has been extracted, and a control unit that evaluates edge characteristics, which are characteristics related to the edge obtained from the acquired edge image, and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit.

[0008] An imaging method according to one aspect of the present disclosure is an imaging method in which an imaging device acquires an edge image, which is an image in which edges have been extracted, evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image, performs feedback control so that appropriate edge characteristics are obtained from the acquired edge image, and performs signal processing using the acquired edge image.

[0009] In the imaging device, imaging element, and imaging method according to one aspect of the present disclosure, an edge image, which is an image in which an edge has been extracted, is acquired, edge characteristics, which are characteristics related to the edge obtained from the acquired edge image, are evaluated, feedback control is performed so that appropriate edge characteristics are obtained from the acquired edge image, and signal processing is performed using the acquired edge image.

[0010] A system according to one aspect of the present disclosure is a system comprising an imaging device and a signal processing device, wherein the imaging device comprises an acquisition unit that acquires an edge image, which is an image in which edges have been extracted, and a control unit that evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit, and the signal processing device comprises a signal processing unit that performs signal processing using the edge image output from the imaging device.

[0011] In a system according to one aspect of the present disclosure, an imaging device acquires an edge image, which is an image in which edges have been extracted, and edge characteristics, which are characteristics related to the edges obtained from the acquired edge image, are evaluated, and feedback control is performed so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit, and a signal processing device performs signal processing using the edge image output from the imaging device.

[0012] The imaging device and the imaging element according to one aspect of the present disclosure may be independent devices or may be internal blocks constituting a single device.

[0013] 1 is a block diagram illustrating an example configuration of an embodiment of an imaging device to which the present disclosure is applied. FIG. 1 is a block diagram illustrating an example configuration of the image sensor of FIG. 1. FIG. 2 is a diagram illustrating an example method of selecting pixels to be compared. FIG. 3 is a circuit diagram illustrating a first example configuration of the edge determination unit of FIG. 2. FIG. 4 is a circuit diagram illustrating a second example configuration of the edge determination unit of FIG. 2. FIG. 5 is a diagram illustrating setting of a threshold value using edge gain and offset. FIG. 6 is a circuit diagram illustrating an example configuration of an edge determination unit in which the edge gain and offset are changeable. FIG. 7 is a diagram illustrating an example of an edge amount. FIG. 8 is a diagram illustrating a first example relationship between edge direction information and the number of output bits. FIG. 9 is a diagram illustrating a second example relationship between edge direction information and the number of output bits. FIG. 10 is a diagram illustrating an example of pixels to be connected. FIG. 11 is a diagram illustrating a first example method of calculating the degree of connection between edge pixels. FIG. 12 is a diagram illustrating a second example method of calculating the degree of connection between edge pixels. FIG. 13 is a diagram illustrating an example method of controlling edge gain and offset by feedback. FIG. 14 is a flowchart illustrating a processing flow in an imaging device. FIG. 15 is a block diagram illustrating a first example of another configuration of the image sensor. FIG. 16 is a block diagram illustrating a second example of another configuration of the image sensor. FIG. 17 is a block diagram illustrating a third example of another configuration of the image sensor. FIG. 18 is a flowchart illustrating a processing flow when a moving object detection region is specified. FIG. 19 is a flowchart illustrating a processing flow when a user designation region is specified. 10 is a flowchart illustrating a processing flow when specifying a mask region. FIG. 10 is a flowchart illustrating a processing flow when adjusting appropriate edge characteristics for each channel number of an edge image. FIG. 10 is a flowchart illustrating a processing flow when adjusting appropriate edge characteristics according to a signal processing result. FIG. 10 is a diagram illustrating an example of an edge amount for each image frame. FIG. 10 is a block diagram illustrating another example configuration of an embodiment of an imaging device to which the present disclosure is applied. FIG. 10 is a flowchart illustrating a processing flow when adjusting appropriate edge characteristics using sensing information. FIG. 10 is a diagram illustrating an overview of optical flow estimation. FIG. 10 is a flowchart illustrating a processing flow when adjusting edge characteristics to be suitable for optical flow estimation. FIG. 10 is a block diagram illustrating another example configuration of an embodiment of an imaging device to which the present disclosure is applied. FIG. 10 is a flowchart illustrating a processing flow when feedback is given to shutter control and lens control.1 is a block diagram illustrating a configuration example of an embodiment of a system to which the present disclosure is applied.

[0014] <Device Configuration> Fig. 1 is a block diagram showing an example configuration of an embodiment of an imaging device to which the present disclosure is applied. In Fig. 1, the imaging device 1 captures an image of a subject, acquires an edge image, and performs signal processing on the acquired edge image. The imaging device 1 has an imaging element 11 such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a signal processing unit 12 such as a DSP (Digital Signal Processor) or ISP (Image Signal Processor).

[0015] The image sensor 11 extracts edges by determining whether the difference in luminance values ​​between pixels arranged in the pixel array exceeds a threshold, thereby obtaining an edge image and outputting the edge image to the signal processing unit 12. The edge image is an image in which the boundaries between bright and dark areas of an image are extracted as edges.

[0016] The signal processing unit 12 performs digital signal processing (hereinafter simply referred to as signal processing) using the edge image output from the imaging element 11. For example, the signal processing unit 12 performs inference processing using a trained model using the edge image. The trained model is generated in advance by machine learning using training data and is provided to the signal processing unit 12. Specifically, signal processing can be performed using a deep neural network (DNN), a convolutional neural network (CNN), or the like. Signal processing may also be performed using a support vector machine (SVM), a histogram of oriented gradients (HOG), or the like.

[0017] The signal processing unit 12 outputs the signal processing result obtained by the signal processing to the outside. The signal processing unit 12 may output the signal processing result to the image sensor 11. The image sensor 11 can perform processing using the signal processing result output from the signal processing unit 12. The signal processing unit 12 may also calculate a control value for controlling the image sensor 11 and output it to the image sensor 11. The image sensor 11 can operate based on the control value output from the signal processing unit 12. Alternatively, the signal processing unit 12 may output the control value to a control unit (not shown) that controls the image sensor 11. In this case, the image sensor 11 will operate according to the control from the control unit.

[0018] Fig. 2 is a block diagram showing an example configuration of the image sensor 11 of Fig. 1. In Fig. 2, the image sensor 11 has a pixel array section 21, a row scanning section 22, an edge determination section 23, a column scanning section 24, a logic section 25, and an input / output section 26. In the image sensor 11, the pixel array section 21 and the edge determination section 23 are configured as analog circuit sections that process analog signals, and the logic section 25 is configured as a digital circuit section that processes digital signals.

[0019] The pixel array unit 21 has a plurality of pixels arranged two-dimensionally (in a matrix). Each pixel has a photodiode as a photoelectric conversion element and pixel transistors including a transfer transistor, an amplification transistor, a reset transistor, and a selection transistor. The row scanning unit 22 selects a specific drive line from the connected drive lines and outputs a drive signal to the selected drive line to drive the pixels, thereby driving the pixels row by row.

[0020] The edge determination unit 23 is composed of comparators and the like provided corresponding to the pixels in the column direction. In the edge determination unit 23, the comparators compare the difference between pixel signals (analog signals) input from each of the pixels to be compared with a threshold value, and output a signal (digital signal) corresponding to the comparison result. In other words, the edge determination unit 23 performs analog-to-digital conversion of the difference between the two analog signals based on edge determination as to whether or not there is an edge, and outputs the result as a 1-bit signal. The column scanning unit 24 sequentially outputs column scanning pulses to select each of the edge determination unit 23 (comparators) in turn, and cause each of the edge determination unit 23 (comparators) to output a signal.

[0021] The logic unit 25 has a timing control unit 31, an operation control unit 32, a counter 33, an edge characteristic calculation unit 34, and an edge characteristic control unit 35. The timing control unit 31 generates a clock signal that serves as a reference for the operation of each unit, such as the row scanning unit 22 and the column scanning unit 24, and outputs the clock signal to each unit. The operation control unit 32 generates a control signal that controls the operation of each unit, such as the row scanning unit 22 and the column scanning unit 24, and outputs the control signal to each unit.

[0022] The counter 33 counts the number of edges based on the signal input from the edge determination unit 23, and outputs the count result to the edge characteristic calculation unit 34. The signal output from the edge determination unit 23 passes through the counter 33 and is also input to the edge characteristic calculation unit 34. The edge characteristic calculation unit 34 calculates edge characteristics such as the edge amount based on the count result input from the counter 33 and the signal input from the edge determination unit 23, and outputs the calculated edge characteristics to the edge characteristic control unit 35.

[0023] The edge characteristic control unit 35 performs feedback control on the edge determination unit 23 based on the edge characteristics input from the edge characteristic calculation unit 34. For example, the edge characteristic control unit 35 evaluates the edge characteristics input from the edge characteristic calculation unit 34 and performs feedback control so that appropriate edge characteristics can be obtained from the edge image acquired by the edge determination unit 23. Here, the threshold used in the edge determination can be changed by feeding back a parameter that sets the threshold used in the edge determination by the edge determination unit 23.

[0024] The input / output unit 26 is composed of terminals and the like, and is electrically connected to the signal processing unit 12. The input / output unit 26 outputs an edge image obtained by edge determination to the signal processing unit 12. The input / output unit 26 also supplies signals such as signal processing results and control values ​​input from the signal processing unit 12 to the logic unit 25 and the like.

[0025] In the image sensor 11 configured as described above, the edge determination unit 23 performs edge determination to acquire an edge image, and the counter 33, edge characteristic calculation unit 34, and edge characteristic control unit 35 perform feedback control to control the edges included in the edge image (edges to be extracted as the edge image). Therefore, it can be said that the edge determination unit 23 configures an edge image acquisition unit 41 that acquires the edge image. It can also be said that the counter 33, edge characteristic calculation unit 34, and edge characteristic control unit 35 configure an edge control unit 42 that controls the edges to be extracted as the edge image.

[0026] <Edge Determination> The edge determination unit 23 in Fig. 2 compares the luminance values ​​of two neighboring pixels from among the multiple pixels arranged in the pixel array unit 21 as comparison pixels, and can determine that an edge exists if the difference between the luminance values ​​exceeds a threshold. Fig. 3 is a diagram showing examples of methods for selecting pixels to be compared. In Fig. 3, five selection methods A to E are shown in planar layout.

[0027] In the planar layout of FIG. 3 , 8×8 squares represent pixels arranged in the pixel array unit 21, and pixels that obtain charges corresponding to each color component are arranged as indicated by G (Green), B (Blue), R (Red), and G in the 2×2 pixels in the upper left. For simplicity's sake, FIG. 3 does not include a letter indicating each color for every pixel. However, every 2×2 pixels, like the 2×2 pixels in the upper left, are arranged with pixels that obtain charges corresponding to each color component of G, B, R, and G. That is, in the pixel array unit 21, the pixels are arranged in a predetermined arrangement pattern (e.g., a Bayer arrangement). The Bayer arrangement is an arrangement pattern in which G pixels are arranged in a checkerboard pattern, and R pixels and B pixels are arranged alternately in rows in the remaining portion. The Bayer arrangement is not limited to this, and other arrangement patterns, such as a quad-Bayer arrangement, may also be used. Alternatively, a configuration may be adopted in which color filters that transmit light of specific wavelengths corresponding to each color are not provided. The relationship between these squares and pixels is the same in other drawings described later.

[0028] In A of Fig. 3, pixels marked with dots indicate pixels to be compared, and arrows C11 and C12 indicate pairs of pixels to be compared. Specifically, the upper left 2x2 pixels indicated by one direction of arrow C11 and the lower right 2x2 pixels indicated by the other direction are selected as pixels to be compared, and edge determination is performed. Also, the upper right 2x2 pixels indicated by one direction of arrow C12 and the lower left 2x2 pixels indicated by the other direction are selected as pixels to be compared, and edge determination is performed.

[0029] Similarly, in B of Fig. 3, 2 x 2 pixels indicated by one direction of arrow C21 or C22 and 2 x 2 pixels indicated by the other direction of arrow C21 or C22 are selected as pixels to be compared, and edge determination is performed. In C of Fig. 3, 2 x 2 pixels indicated by one direction of arrow C31 or C32 and 2 x 2 pixels indicated by the other direction of arrow C31 or C32 are selected as pixels to be compared, and edge determination is performed.

[0030] In Fig. 3D, 2 x 2 pixels indicated by one of the directions of arrow C41 or C42 and 2 x 2 pixels indicated by the other direction of arrow C41 or C42 are selected as pixels to be compared, and edge determination is performed. In Fig. 3E, 2 x 2 pixels indicated by one of the directions of arrow C51 or C52 and 2 x 2 pixels indicated by the other direction of arrow C51 or C52 are selected as pixels to be compared, and edge determination is performed.

[0031] Fig. 4 is a circuit diagram showing an example of the configuration of the edge determination unit 23 of Fig. 2. As shown in Fig. 4, the edge determination unit 23 is provided with a plurality of comparators corresponding to the pixels arranged in the pixel array unit 21. In Fig. 4, comparators 52-1 and 52-2 are provided for comparing pixels 51-1 and 51-2, which are neighboring pixels arranged in the pixel array unit 21. For example, the comparator 52-1 is a comparator provided in an even-numbered column (EVEN CLM), and the comparator 52-2 is a comparator provided in an odd-numbered column (ODD CLM).

[0032] The pixel 51-1 has a photodiode 61-1, a transfer transistor 62-1, an amplification transistor 63-1, a reset transistor 64-1, and a selection transistor 65-1. In the pixel 51-1, the pixel transistors operate in response to a drive signal, and a pixel signal corresponding to the charge obtained by photoelectric conversion in the photodiode 61-1 is output to the comparators 52-1 and 52-2 via a vertical signal line (VSL1). The pixel 51-2 is configured similarly to the pixel 51-1, and outputs a pixel signal corresponding to the charge obtained by photoelectric conversion in the photodiode 61-2 to the comparators 52-1 and 52-2 via a vertical signal line (VSL2).

[0033] The pixel signal output from pixel 51-1 and the pixel signal output from pixel 51-2 are input to comparator 52-1. Capacitors 53A-1 and 53B-1 are provided on the input side of comparator 52-1. Comparator 52-1 compares the input pixel signals, and if the difference exceeds a threshold, outputs a 1-bit signal indicating "1". The pixel signal output from pixel 51-1 and the pixel signal output from pixel 51-2 are input to comparator 52-2. Capacitors 53A-2 and 53B-2 are provided on the input side of comparator 52-2. Comparator 52-2 compares the input pixel signals, and if the difference exceeds a threshold, outputs a 1-bit signal indicating "0".

[0034] In this way, in the edge determination unit 23, the comparators 52-1 and 52-2 return a pair of "1" and "0" when the difference between the pixel signals (luminance values) of two neighboring pixels exceeds a threshold value, thereby performing edge determination. In the subsequent logic unit 25, the counter 33 can count edges based on the signals output from the comparators 52-1 and 52-2 of the edge determination unit 23. Alternatively, instead of the subsequent counter 33, the edge determination unit 23 may count a pair of "1" and "0" or a pair of "0" and "1" that are outputs from the comparators 52-1 and 52-2 as edges.

[0035] 5, edge determination may be performed by providing one comparator 52 for pixels to be compared, that is, pixel 51-1 and pixel 51-2, comparing pixel signals, and outputting a 1-bit signal indicating "1" when the difference exceeds a threshold. In this case, the counter 33 or edge determination unit 23 at the subsequent stage can count the output of a 1-bit signal indicating "1" as an edge.

[0036] The edge determination unit 23 can change the threshold value used in edge determination by changing the edge gain and offset. The threshold value for edge determination can be calculated by the following equation (1).

[0037] th = |px1 - px2| - g * max (px1, px2) - c = px1 - px2 - g * px1 - c = (1 - g) * px1 - px2 - c = G * px1 - px2 - c ... (1)

[0038] However, when expanding equation (1), the condition px1 > px2 must be satisfied. In equation (1), |px1 - px2| indicates the difference in luminance value (pixel difference) between pixel 51-1 (px1) and pixel 51-2 (px2), g*max(px1, px2) indicates a threshold, and c indicates an offset. Note that equation (1) is an example of a method for calculating the threshold for edge determination, and other calculation methods may also be used.

[0039] FIG. 6 is a diagram illustrating the setting of thresholds based on edge gain and offset. In FIG. 6, the horizontal axis represents the luminance value of pixel 51-1 (px1) and the vertical axis represents the luminance value of pixel 51-2 (px2), and the diagram schematically illustrates how the edge detection region changes when the edge gain (Gain) and offset (Offset) are changed. FIG. 6A shows the edge detection region when Gain = 0.5 and Offset = 5 are set. FIG. 6B shows the edge detection region when Gain = 0.8 and Offset = 10 are set.

[0040] In Figures 6A and 6B, when the difference in brightness values ​​between px1 and px2 is large, an edge detection region is formed and an edge is detected. However, in Figure 6B, the set edge gain (Gain = 0.8) is greater than the edge gain set in Figure 6A (Gain = 0.5), further increasing the difference in brightness values ​​between px1 and px2, thereby expanding the edge detection region. Also, in Figure 6B, the set offset (Offset = 10) is greater than the offset set in Figure 6A (Offset = 5), increasing the offset in the direction of the brightness values ​​of px1 and px2. As a result, the number of black circles (corresponding to the brightness values ​​of px1 and px2) included in the edge detection region in Figure 6B increases compared to Figure 6A.

[0041] FIG. 7 is a circuit diagram showing an example configuration of the edge determination unit 23 that allows for changeable edge gain and offset. In FIG. 7, capacitors 53A-1 and 53B-1, which are provided on the input side of comparator 52-1, are each composed of multiple capacitors. Capacitors 53A-1 and 53B-1 operate according to the capacitance Cm and voltage Vm input to each capacitor, thereby allowing for changeable edge gain and offset, which set the threshold for edge determination by comparator 52-1. Similarly, in comparator 52-2, capacitors 53A-2 and 53B-2 allow for changeable edge gain and offset, which set the threshold for edge determination. Note that the circuit configuration shown in FIG. 7 is an example of a circuit that allows for changeable edge gain and offset in the edge determination unit 23; other circuit configurations may also be employed. In the edge control unit 42, the edge characteristic control unit 35 includes a feedback circuit and has a mechanism for controlling the reference voltage, bias voltage, and the like supplied to the internal circuit of the edge determination unit 23 based on the settings of the edge characteristic calculation unit 34.

[0042] In this way, the edge image acquisition unit 41 acquires an edge image by performing edge determination by the edge determination unit 23. Furthermore, the edge image acquisition unit 41 can adjust the edge characteristics, which are characteristics of the edge image, to appropriate edge characteristics (desired edge characteristics) by making it possible to set variable thresholds according to the edge gain and offset as thresholds used in the edge determination by the edge determination unit 23. The edge gain and offset are examples of parameters for which the thresholds (variable thresholds) used in the edge determination can be set, and other parameters may also be used.

[0043] <Example of Edge Characteristics> When the edge image acquisition unit 41 extracts edges, the edge control unit 42 in Fig. 2 performs control to dynamically feedback parameters that set thresholds used in edge determination by the edge determination unit 23, thereby enabling an edge image with appropriate edge characteristics to be acquired. For example, the edge characteristics include at least one of edge amount, edge direction information, and edge connectivity information. The edge amount indicates the proportion of edge pixels to all pixels in the valid area. The edge direction information is information about the edge direction. The edge connectivity information is information indicating the degree of connectivity of edge pixels.

[0044] FIG. 8 is a diagram illustrating an example of edge amounts. FIG. 8A shows an imaged scene for comparison with FIG. 8B and FIG. 8C. FIG. 8B and FIG. 8C show edge images corresponding to the imaged scene of FIG. 8A, but with different edge amounts. FIG. 8B shows an edge image with an edge amount of approximately 10%. FIG. 8C shows an edge image with an edge amount of a few percent, i.e., an edge image with a smaller edge amount than the edge image shown in FIG. 8B. For example, in a dark scene, the amount of light incident on the image sensor 11 decreases, the difference in luminance values ​​between pixels becomes smaller, and fewer pixels are determined to exceed the threshold. This results in a smaller edge amount, as shown in the edge image shown in FIG. 8C. Thus, the edge amount is the proportion of edge pixels (pixels from which edges have been extracted) to all pixels in the effective area of ​​the pixel array unit 21, and is the proportion of edges to a predetermined area in the edge image (an area corresponding to all pixels in the effective area).

[0045] The edge direction information will be described with reference to Figures 9 and 10. Figure 9 is a diagram showing an example of the relationship between the edge direction information and the number of output bits when pixels are binned. Figure 9 shows the coordinates (height, width) of pixels to be compared when 2x2 pixels, two pixels each in the row and column directions in a planar layout, are binned and the upper left pixel (R pixel) is set as the origin. For the sake of explanation, the planar layout shows column numbers corresponding to the height in the column direction and row numbers corresponding to the width in the row direction.

[0046] The 4ch edge has one bit of information indicating which is larger in each of the horizontal and vertical pixel comparisons (comparison of the luminance values ​​of each pixel). For example, in the horizontal pixel comparison, as indicated by arrow C1, when binning pixel (2,0) is compared with binning pixel (1,3), and when binning pixel (1,3) is compared with binning pixel (2,0), one bit of information indicating which binning pixel has a larger luminance value is obtained. In the vertical pixel comparison, as indicated by arrow C2, when binning pixel (0,1) is compared with binning pixel (3,2), and when binning pixel (3,2) is compared with binning pixel (0,1), one bit of information indicating which binning pixel has a larger luminance value is obtained. In other words, the 4ch edge has four bits of information, consisting of two bits of information obtained in the horizontal pixel comparison and two bits of information obtained in the vertical pixel comparison.

[0047] A 2ch edge has information from a 4ch edge that is selected from either the horizontal or vertical direction. For example, one bit of information or the other bit of information from two bits of information obtained by pixel comparison in the horizontal direction is used as one bit of horizontal information. Also, one bit of information or the other bit of information from two bits of information obtained by pixel comparison in the vertical direction is used as one bit of vertical information. A 1ch edge has information from a 2ch edge that is selected from either the horizontal or vertical direction. For example, one bit of information selected from two bits of information obtained by pixel comparison in the horizontal direction or one bit of information selected from two bits of information obtained by pixel comparison in the vertical direction is used as one bit of 1ch edge information.

[0048] In this way, for example, a 4ch edge has four-directional information using 4 bits of information, and a 2ch edge has two-directional information using 2 bits of information, so these pieces of information can be used as edge direction information, which is information about the edge direction. Here, luminance gradient direction information can be obtained depending on the method of selecting pixels to be compared. Note that while Figure 9 illustrates 4ch edges, 2ch edges, and 1ch edges, this can be extended to N-ch edges depending on the method of selecting pixels to be compared. An N-ch edge has N bits of information.

[0049] While Fig. 9 shows a case where edges are detected by binning pixels, edges may also be detected without binning pixels, as shown in Fig. 10. Fig. 10 shows the coordinates (height, width, channel) of the comparison pixel when the upper left pixel (R pixel) is set as the origin without binning pixels. Fig. 10 also shows the column number corresponding to the height and the row number corresponding to the width for the planar layout. R, G, and B are specified as the channel according to the pixel of each color.

[0050] The 3ch edge has one bit of information indicating which pixel has a larger luminance value in each of the horizontal and vertical pixel comparisons (comparison of the luminance values ​​of each pixel). For example, the 3ch edge includes one bit of information indicating which pixel has a larger luminance value in a horizontal pixel comparison between pixel (5, 1, B) and pixel (2, 6, R) as shown by arrow C11, or in a vertical pixel comparison between pixel (1, 3, B) and pixel (6, 4, R) as shown by arrow C21. The 3ch edge also includes one bit of information indicating which pixel has a larger luminance value in a horizontal pixel comparison between pixel (5, 0, G) and pixel (2, 7, G) as shown by arrow C12, or in a vertical pixel comparison between pixel (1, 2, G) and pixel (6, 5, G) as shown by arrow C22. Furthermore, the 3ch edge contains 1 bit of information indicating which pixel has a larger luminance value when comparing pixels (4, 0, R) and (3, 7, B) in a horizontal pixel comparison as shown by arrow C13, or when comparing the difference between pixels (0, 2, R) and (7, 5, B) in a vertical pixel comparison as shown by arrow C23. In other words, the 3ch edge has 3 bits of information obtained by comparing pixels for each color in the horizontal and vertical directions.

[0051] The 1ch edge has 1 bit of information selected from the 3 bits of information obtained by comparing the pixels of each color in the 3ch edge. For example, the 1 bit of information of the 1ch edge is selected from 1 bit of information obtained by comparing B pixels and R pixels in the horizontal or vertical direction, 1 bit of information obtained by comparing G pixels and G pixels in the horizontal or vertical direction, or 1 bit of information obtained by comparing R pixels and B pixels in the horizontal or vertical direction. Here, edges between different colors can be detected by comparing pixels of different colors, such as B pixels and R pixels, or R pixels and B pixels. Edges between the same colors can also be calculated by comparing pixels of the same color, such as G pixels and G pixels.

[0052] The edge connectivity information will be described with reference to Figures 11 to 13. When an image of a certain subject is captured, edges should ideally be extracted for all pixels on the boundary between the background and the subject. In a recognition task using signal processing performed by the signal processing unit 12, the recognition rate may decrease if the edge of the subject is broken. Therefore, the degree of connectivity between edge pixels is important, and control is performed using edge connectivity information indicating the degree of connectivity between edge pixels. For example, in order to improve the recognition rate in a recognition task, a higher degree of connectivity between edge pixels is considered to be better, so the edge connectivity information can be used to control the degree of connectivity between edge pixels to be higher.

[0053] FIG. 11 is a diagram showing pixels to be connected. In the case of 4-connected pixels, as shown in FIG. 11A, the four neighboring pixels above, below, left, and right of the central pixel are connected. In the case of 8-connected pixels, as shown in FIG. 11B, the eight neighboring pixels to be connected are the four neighboring pixels above, below, left, and right of the central pixel, plus the pixels at the top left, bottom left, top right, and bottom right. A set of connected pixels can be considered a connected component. The degree of connectivity can be calculated using the average number of pixels constituting the extracted connected component, as shown in the following equation (2).

[0054] Connectivity = number of edge pixels / number of connected components (2)

[0055] In the case of a 1ch edge, connectivity can be determined based on whether the edge pixel is an edge pixel. FIG. 12 illustrates an example of a method for calculating the connectivity between edge pixels in the case of a 1ch edge. In FIG. 12, black squares represent edge pixels. In the case of 4-connected edges, the pixels above, below, left, and right are connected, so the four edge pixels surrounded by frame L11 are considered connected edge pixels, and the set of these pixels can be considered a connected component. Therefore, in the case of 4-connected edges, the connectivity is 8 / 5 = 1.6. In the case of 8-connected edges, the pixels above, below, left, and right, as well as the upper left, lower left, upper right, and lower right pixels are connected, so the six edge pixels surrounded by frame L12 are considered connected edge pixels, and the set of these pixels can be considered a connected component. Therefore, in the case of 8-connected edges, the connectivity is 8 / 3 = 2.7.

[0056] In the case of a 4ch edge, connectivity can be determined based on whether the directional vectors of the edge pixels match. There are two cases for the directional vectors: a perfect match and a match in directional components. FIG. 13 is a diagram illustrating an example of a method for calculating the degree of connectivity between edge pixels in the case of a 4ch edge. In FIG. 13, patterned squares represent edge pixels, with different patterns indicating pixels with different directions, and the arrows on the squares represent the directional vectors of the edge pixels.

[0057] FIG. 13A illustrates an example of edge pixel connectivity when connectivity determination is performed based on a perfect match of directional vectors. In the case of 4-connectivity, the pixels above, below, left, and right are connected, and the directional vectors must be perfectly consistent. Therefore, the three edge pixels surrounded by frame L21 (edge ​​pixels with an upward directional vector) and the four edge pixels surrounded by frame L22 (edge ​​pixels with a rightward directional vector) are considered connected edge pixels, and the set of these pixels can be considered a connected component. Therefore, for a 4ch edge, the connectivity degree for 4-connectivity is 9 / 4 = 2.25. In the case of 8-connectivity, the upper left, lower left, upper right, and lower right pixels are also connected, but since there are no pixels with perfectly consistent directional vectors among these pixels, the edge pixels surrounded by frame L21 and frame L22 are considered connected, just as in the case of 4-connectivity, and the connectivity degree is 9 / 4 = 2.25. In other words, the results for 4-connectivity and 8-connectivity are the same in FIG. 13A.

[0058] FIG. 13B shows an example of edge pixel connectivity when connectivity determination is performed based on the matching of directional components of directional vectors. When connectivity determination is performed based on the matching of directional components, all edges are detected as the same connected component if the directional vectors change continuously. In the case of 4-connected edges, the top, bottom, left, and right pixels are considered as connection targets, and connectivity determination is performed to determine whether the directional components match. Therefore, the four edge pixels surrounded by frame L31 (edge ​​pixels having upward and upper-right directional vectors) and the four edge pixels surrounded by frame L32 (edge ​​pixels having rightward directional vectors) are considered as connected edge pixels, and a set of these pixels can be considered as a connected component. Therefore, for a 4-channel edge, the connectivity degree in the case of 4-connected edges is 9 / 3 = 3. In the case of 8-connected edges, the upper left, lower left, upper right, and lower right pixels are also considered as connection targets, so the eight edge pixels surrounded by frame L33 (edge ​​pixels having upward, upper-right, and rightward directional vectors) are considered as connected edge pixels, and a set of these pixels can be considered as a connected component. Therefore, in the case of 8-connection at the 4ch edge, the degree of connection is 9 / 2 = 4.5.

[0059] In this way, the edge connectivity information is information that calculates the degree of connectivity between edge pixels by assuming that the edge pixels are connected when they are adjacent in either the up-down, left-right, or up-down, left-right, or diagonal directions, or when the vector directions of adjacent edge pixels are the same. Using the edge connectivity information to select and output edge pixels to be connected whose connectivity between edge pixels is equal to or greater than a threshold is also effective in removing noise. For example, by taking the dot product of the vectors of adjacent edge pixels, it is easy to determine whether the direction information matches.

[0060] The degree of connectivity of edge pixels is not limited to the calculation method described above, and may be calculated by other calculation methods. For example, the signal processing unit 12 may detect connected components using a labeling processing algorithm and feed the detected connected components back to the image sensor 11. In the labeling processing, the same ID is assigned to pixels with consecutive edges in the edge image, and the number of label IDs is the number of connected components. Therefore, the edge control unit 42 can obtain edge connectivity information according to the number of label IDs.

[0061] Furthermore, connectivity can be calculated using a line buffer that stores pixel information obtained by scanning the pixel array unit 21 in a predetermined direction. In a labeling algorithm used in signal processing, edge pixels are searched for and labeled by raster scanning starting from the upper left pixel in the edge image. Here, labeling of a given edge pixel is determined using only information on pixels scanned up to that point. In other words, whether the pixel to be linked is 4-connected or 8-connected, labeling is possible as long as information on the pixels of the currently scanned line and the line immediately preceding it is stored in memory. For example, the line buffer is included in the row scanning unit 22 (or the column scanning unit 24) in FIG. 2 , and stores information output from the row scanning unit 22 (information on the line immediately preceding it) in a memory (not shown) so that it can be read by the logic unit 25. This allows edge connectivity information to be calculated from information on the pixels of the currently scanned line (first line) and the line scanned immediately preceding the first line (second line). When calculating connectivity, if you simply want to calculate connectivity only in the horizontal (or vertical) direction, information about the pixels on the previous line is not necessary; it is sufficient to determine whether edge pixels are adjacent on the line currently being scanned.

[0062] <Parameter Control Method> A control method will be described in which the edge control unit 42 dynamically feeds back parameters that set thresholds used in edge determination. The edge control unit 42 can perform control so as to reduce the overall error in the evaluation values ​​of each characteristic to be controlled. Specifically, if the difference between the evaluation value (Y) of the edge characteristic to be controlled of the edge image to be output and the target value (P) is defined as the error amount (E), the relationship of the following equation (3) holds.

[0063] E = P - Y (3)

[0064] With respect to the target value (P), ±E is set as an appropriate value, and when the error amount (E) exceeds the appropriate value, the edge gain and offset are changed in a stepwise manner (for example, by one step) in the direction that reduces the error amount (E). For example, the evaluation value (Y) of the edge characteristic to be controlled is the edge amount ratio, the edge amount ratio for each channel, the average edge connectivity, etc.

[0065] Here, it is assumed that feedback control is performed so that the edge amount and edge connectivity approach the target values. In this case, the error in the edge amount is expressed as E q The error in the average edge connectivity is defined as E n Then, the relationship of the following equation (4) holds: In equation (4), α and β are real numbers equal to or greater than 0.

[0066] E = αE q + βE n ... (4)

[0067] Also, the target edge amount is set as Y q Let the current edge amount be P q Then, the relationship of the following equation (5) holds. Let Y n Let the current average edge connectivity be P n Then, the relationship of the following equation (6) holds.

[0068] E q = P q - Y q ... (5) E n = P n - Y n ... (6)

[0069] For example, regarding the edge amount, P q = 4 (%), Y q = 10 (%), and the average edge connectivity is n = 3, Y n = 5, then, using equations (5) and (6), E q = -6, E n = -2 is obtained. Then, when the weighting coefficients are α = 1 and β = 1, E = -6α - 2β = -8 is obtained using equation (4). Based on the error amount (E) obtained in this way, the edge control unit 42 performs control to change the edge gain and offset in stages.

[0070] FIG. 14 is a diagram illustrating an example of a control method for edge gain and offset. In FIG. 14, the horizontal axis represents the error amount (E), and the vertical axis represents the manipulated variables (ΔG, ΔO) of the edge gain and offset. In FIG. 14, the manipulated variables (ΔG, ΔO) are determined according to the error amount (E), and feedback control is performed so that the edge gain and offset are changed according to the manipulated variables, which are changed in stages. Specifically, when ±E1, which is an appropriate value for the target value (P), is set to E1 = 2, and the manipulation stages are indicated by manipulation level A, which is represented by a thick line in the diagram, the following occurs:

[0071] That is, operation level A ０ As shown in Figure 1, when E = -2 to 2, the edge gain and offset are not manipulated. １ As shown in Fig. 1, when E = -6 to -2, the edge gain and offset are increased by one step, and the operation level is A. ２ As shown in Fig. 1, when E = -10 to -6, the edge gain and offset are increased by two steps. －１ As shown in Figure 1, when E = 2 to 6, the edge gain and offset are lowered by one step, and the operation level is A. －２ As shown in Fig. 1, when E = 6 to 10, the edge gain and offset are decreased by two steps.

[0072] In the above example, since the error amount (E) is found to be E = -8, the edge control unit 42 performs feedback control to increase the edge gain by two stages. The edge gain and offset can be manipulated as parameters for feedback control, but normally only the edge gain is manipulated, and when the edge gain exceeds a certain amount, the offset can also be manipulated. In other words, the edge determination threshold can be changed by feeding back at least one of the edge gain and offset.

[0073] In the imaging device 1, the signal processing unit 12 may calculate control values ​​for controlling the imaging element 11 and output them to the imaging element 11. In this case, control values ​​that facilitate signal processing by the signal processing unit 12 can be calculated. For example, a technology related to CIS (CMOS Image Sensor) is known as Deep ISP (Image Signal Processor). This technology realizes optimal sensor control for signal processing by providing feedback to the ISP control values ​​in a neural network (NN) that performs noise reduction tasks. In the imaging device 1, the signal processing unit 12 can use Deep ISP technology to calculate control values ​​for black level adjustment, HDR (High Dynamic Range), gain, exposure, and the like, and provide feedback to the imaging element 11.

[0074] <Processing Flow> The processing flow in the imaging device 1 will be described with reference to the flowchart of FIG.

[0075] In step S11, the edge image acquisition unit 41 acquires an edge image. In the edge image acquisition unit 41, the edge determination unit 23 performs edge determination by determining whether or not the difference between pixel signals (luminance values ​​of two neighboring pixels) output from pixels to be compared and arranged in the pixel array unit 21 exceeds a threshold, thereby acquiring the edge image.

[0076] In step S12, the counter 33 counts the edges included in the edge image. The edges can be counted for each image frame. In step S13, the edge characteristic calculation unit 34 calculates edge characteristics. As the edge characteristics, edge amount based on the counted edges, edge direction information, and edge connectivity information can be calculated.

[0077] In step S14, the edge characteristic control unit 35 determines whether the edge characteristics of the edge image to be processed are appropriate. Here, the difference between the evaluation value (e.g., edge amount ratio) of the edge characteristics to be controlled and the target value is defined as the error amount, an appropriate value is set from the error amount relative to the target value, and whether the error amount exceeds the appropriate value is determined, thereby making it possible to determine whether the edge characteristics are appropriate.

[0078] If it is determined in step S14 that the edge characteristics are not appropriate, the process proceeds to step S15. In step S15, the edge characteristic control unit 35 performs feedback control so that appropriate edge characteristics can be obtained from the edge image acquired by the edge image acquisition unit 41. Here, as shown in Fig. 14, if the error amount exceeds an appropriate value, feedback is performed so that the edge gain and offset are changed in the direction in which the error amount decreases.

[0079] When the process of step S15 is completed, the process returns to step S11, and the processes of steps S11 to S14 are repeated. That is, the edge determination unit 23 performs edge determination again using the threshold value changed according to the edge gain and offset changed by feedback. Then, the edge characteristics are calculated from the edge image obtained in accordance with the edge determination using the threshold value changed, and it is determined whether the edge characteristics are appropriate.

[0080] On the other hand, if it is determined in step S14 that the edge characteristics are appropriate, the process proceeds to step S16. In step S16, the signal processing unit 12 performs signal processing such as human detection processing using CNN, using the edge image input from the image sensor 11. When the process of step S16 ends, the series of processes ends.

[0081] As described above, in the imaging device 1, the imaging element 11 acquires an edge image from pixel signals output from pixels arranged in the pixel array unit 21, evaluates edge characteristics obtained from the acquired edge image, and performs feedback control so that appropriate edge characteristics are obtained from the acquired edge image. As a result, the edge characteristics obtained from the acquired edge image are fed back so that they become edge characteristics suitable for the operating environment, and the characteristics of the edge image output from the imaging element 11 to the signal processing unit 12 are stable regardless of the operating environment, so that the quality of signal processing (e.g., inference processing) by the signal processing unit 12 can be maintained constant.

[0082] Here, due to their characteristics, image sensors (and imaging devices having them) that capture edge images are easily affected by factors such as the surrounding environment and subject movement during image capture, resulting in unstable edge image characteristics. For example, when capturing an image in a dark location, the luminance difference between pixels is reduced, making edges less visible. Furthermore, edges are also less visible when blur occurs due to the movement of the image sensor or subject, or when there are blurred areas. On the other hand, when there are high-frequency components in the background or subject, a large number of edges appear (excessive edges), resulting in the loss of edge information about the subject. In an imaging device, if the characteristics of the edge image output from the image sensor change, the accuracy of signal processing (tasks processed by the DSP, etc.) in the downstream signal processing unit (e.g., DSP) may be significantly reduced.

[0083] In contrast, in the imaging device 1 to which the present disclosure is applied, the edge gain and offset, which are parameters used when performing edge determination in the imaging element 11, are dynamically controlled according to the scene, thereby making it possible to stabilize the characteristics of the edge image output from the imaging element 11 regardless of the operating environment, thereby suppressing a decrease in the accuracy of signal processing by the downstream signal processing unit 12. In other words, in the imaging device 1, the imaging element 11 can always output an edge image that is easy for the downstream signal processing unit 12 to process.

[0084] The inventors of the present disclosure have confirmed through detailed simulations that adjusting the edge amount as an edge characteristic in feedback control to which the present disclosure is applied improves the object detection accuracy (person detection accuracy) in object detection processing (person detection processing) using edge images. In the simulations, the signal processing unit 12 performs inference processing using a CNN that performs a person detection task using edges. The inference processing was performed on edge images obtained by changing only the edge gain and offset, which are parameters used in edge determination in the image sensor 11, and the person recognition probability (person confidence) obtained for each parameter (edge ​​amount) was compared. The simulations used 4ch edge images.

[0085] Specifically, when capturing an image in a dark location with the edge gain and offset set to the default values ​​(Gain = 0.8, Offset = 5), the brightness change is small and the subject's edges are not clearly visible (the edge amount is small). This results in insufficient capture of the subject's edge information, resulting in reduced human detection accuracy. In this simulation, when the default values ​​of Gain = 0.8 and Offset = 5 were used, the edge amount was 5.878%, and the human recognition probability was 41%, but this was not sufficient for accurate human detection. In this simulation, the edge gain and offset values ​​were changed to Gain = 0.9 and Offset = 50 to ensure the desired edge amount. This resulted in the edge amount being 10.36%, sufficient subject edge information being captured, and the human recognition probability was 99%, confirming sufficient accuracy for human detection results.

[0086] In this way, the simulation confirmed that adjusting the edge gain and offset values ​​to ensure the desired edge amount improves human detection accuracy. When a sudden change in brightness occurs, the edge amount suddenly decreases or increases, but even in such cases, by applying feedback control to the present disclosure to ensure the desired edge amount or maintain the original edge amount, it is possible to improve detection accuracy in the same way as described above.

[0087] <Other Configurations> The image sensor 11 may be configured to allow a count area in which edges are counted to be designated, and counting may be performed within the designated count area. Configurations of the image sensor 11 that allow the count area to be designated are shown in Figures 16 to 18. In Figures 16 to 18, parts corresponding to those in Figure 2 are given the same reference numerals, and descriptions thereof will be omitted where appropriate.

[0088] 16 , compared to FIG. 2 , the logic unit 25 is provided with an edge control unit 42A instead of the edge control unit 42. The edge control unit 42A has a counter 33, an edge characteristic calculation unit 34, an edge characteristic control unit 35, and an area designation unit 36. The area designation unit 36 ​​controls the row scanning unit 22 to count edges based on pixel signals output from pixels arranged in a specific row among the pixels arranged two-dimensionally in the pixel array unit 21. That is, the row scanning unit 22 designates an effective area A1 in the row direction in the analog circuit unit including the pixel array unit 21 and the edge determination unit 23, and the counter 33 can count edges included in a count area (a specific area on the edge image) corresponding to the effective area A1.

[0089] 17 , the logic unit 25 is provided with an edge control unit 42A instead of the edge control unit 42, as in FIG. 16 . In the edge control unit 42A, the area designation unit 36 ​​controls the column scanning unit 24 to count edges based on pixel signals output from pixels arranged in a specific column among the pixels arranged two-dimensionally in the pixel array unit 21. That is, the column scanning unit 24 designates an effective area A2 in the column direction in the analog circuit unit including the pixel array unit 21 and the edge determination unit 23, and the counter 33 can count edges included in a count area (a specific area on the edge image) corresponding to the effective area A2.

[0090] 18, compared to FIG. 2, the logic unit 25 is provided with an edge control unit 42B instead of the edge control unit 42. The edge control unit 42B has a counter 33, an edge characteristics calculation unit 34, an edge characteristics control unit 35, and an area designation unit 37. The area designation unit 37 designates a count area in which the counter 33 will perform counting for the signal (digital signal) output from the edge determination unit 23, and outputs a signal corresponding to the designated count area to the counter 33. In other words, the area designation unit 37 designates the count area in the digital circuit unit including the counter 33, allowing the counter 33 to count edges included in the count area (a specific area on the edge image).

[0091] In this way, edges can be counted not only within one image frame, but also within a specific region on the edge image by specifying the count region. The count region can also be considered to be the region for which the edge characteristics calculation unit 34 calculates edge characteristics. Note that the region specification unit 36 ​​may control both the row scanning unit 22 and the column scanning unit 24 so that edges are counted based on pixel signals output from pixels arranged in a specific row and column.

[0092] <Specifying a Moving Object Detection Region> The imaging device 1 may detect a moving object contained in an edge image, calculate edge characteristics in the region where the moving object is detected (hereinafter also referred to as a moving object detection region), and control the moving object detection region so that the edge characteristics are appropriate. In this case, edge control is not performed on regions of the edge image excluding the moving object detection region. Adjusting the edge characteristics in the moving object detection region is particularly effective for edge images with a large amount of high-frequency components in the background. In addition, a Region of Interest (ROI) may be specified for the moving object detection region, which can reduce the power consumption of signal processing. Specifying an ROI is effective when you want to detect only moving objects.

[0093] The flow of processing when specifying a moving object detection region in the imaging device 1 will be described with reference to the flowchart of FIG.

[0094] In step S31, similar to step S11 of FIG. 15 , an edge image is acquired by the edge image acquisition unit 41. In step S32, the logic unit 25 performs signal processing on the edge image to detect a moving object (e.g., a moving person) included in the edge image. It should be noted that the detection of a moving object included in the edge image may be performed not only by the logic unit 25 but also by another signal processing unit (including the signal processing unit 12), and the detection result of the moving object may be input to the edge control unit 42. In step S33, the edge control unit 42 designates a moving object detection region in the edge image based on the detection result of the moving object. In step S34, the edge characteristic calculation unit 34 calculates edge characteristics in the designated moving object detection region. For example, when an edge amount is used as the edge characteristic, the counter 33 counts edges within the motion detection region, and the edge characteristic calculation unit 34 can calculate the edge amount based on the count result.

[0095] In steps S35 and S36, similar to steps S14 and S15 in Fig. 15, the edge characteristic control unit 35 determines whether the calculated edge characteristics are appropriate, and if it is determined that the edge characteristics are inappropriate (S35: No), feedback control is performed so that appropriate edge characteristics are obtained from the specified moving object detection region. In step S37, similar to step S16 in Fig. 15, if it is determined that the calculated edge characteristics are appropriate (S35: Yes), the signal processing unit 12 performs signal processing using the edge image. When the processing of step S37 is completed, the series of processes ends.

[0096] <Specifying a User-Specified Region> The imaging device 1 may calculate edge characteristics in a region specified by a user (hereinafter also referred to as a user-specified region) and control the user-specified region so that the edge characteristics are appropriate. In this case, edge control is not performed on a region of the edge image other than the user-specified region. Here, an ROI may be specified so that an edge image corresponding to the user-specified region is output as the edge image output from the imaging element 11.

[0097] The flow of processing when a user designates a user-designated area in the imaging device 1 will be described with reference to the flowchart of FIG.

[0098] In step S51, similar to step S11 in Fig. 15, the edge image acquisition unit 41 acquires an edge image. In step S52, the edge control unit 42 designates a user-designated region in the edge image based on an instruction from the user. For example, the instruction from the user can be given by operating an input unit (not shown), such as a button or a touch panel, provided in the imaging device 1. In step S53, the edge characteristic calculation unit 34 calculates the edge characteristics in the designated user-designated region.

[0099] In steps S54 and S55, similar to steps S14 and S15 in Fig. 15, the edge characteristic control unit 35 determines whether the calculated edge characteristics are appropriate, and if it is determined that the edge characteristics are inappropriate (S54: No), feedback control is performed so that appropriate edge characteristics can be obtained from the specified user-designated area. In step S56, similar to step S16 in Fig. 15, if it is determined that the calculated edge characteristics are appropriate (S54: Yes), the signal processing unit 12 performs signal processing using the edge image. When the processing of step S56 is completed, the series of processes ends.

[0100] <Specifying Mask Area> The imaging device 1 may mask a specific area of ​​the edge image, calculate edge characteristics in the area excluding the masked area (hereinafter also referred to as the mask area), and control the area excluding the mask area so that the edge characteristics are appropriate. In this case, edge control is not performed on the mask area in the edge image. This is effective, for example, when there is a lot of high-frequency components (noise) in the specific area of ​​the edge image.

[0101] The flow of processing when specifying a mask area in the imaging device 1 will be described with reference to the flowchart of FIG.

[0102] In step S71, similar to step S11 in Fig. 15, the edge image acquisition unit 41 acquires an edge image. In step S72, the edge control unit 42 specifies a mask region for the edge image. In step S73, the edge characteristic calculation unit 34 calculates edge characteristics in the region excluding the specified mask region.

[0103] In steps S74 and S75, similar to steps S14 and S15 in Fig. 15, the edge characteristic control unit 35 determines whether the calculated edge characteristics are appropriate, and if it is determined that the edge characteristics are inappropriate (S74: No), feedback control is performed so that appropriate edge characteristics are obtained from the area excluding the specified mask area. In step S76, similar to step S16 in Fig. 15, if it is determined that the calculated edge characteristics are appropriate (S74: Yes), the signal processing unit 12 performs signal processing using the edge image. When the processing of step S76 is completed, the series of processes ends.

[0104] As described above, in the imaging device 1, a specific region in the edge image can be designated as a region for calculating edge characteristics and the edge characteristics of the specific region can be adjusted by designating a moving object detection region, a user-designated region, or a mask region. Here, the specific region in the edge image can be designated by, for example, employing any of the configurations shown in Figures 16 to 18 and designating an effective region in the row or column direction in the analog circuit section or a count region in the digital circuit section using the region designation unit 36 ​​and the region designation unit 37.

[0105] <Adjusting Appropriate Edge Characteristics for Each Channel Number> Edge images with different channel numbers, such as 1ch edge, 2ch edge, and 4ch edge, are expected to have different edge characteristics suitable for signal processing by the signal processing unit 12. For example, 1ch edge images do not contain directional information, while 2ch edge images do, and 4ch edge images have even more detailed directional information than 2ch edge images. Therefore, it is expected that edge characteristics suitable for each channel number exist. For example, in signal processing using edge images with directional information, the more sets of edge directional vectors that match, the more likely it is that the subject's edges can be properly extracted. Therefore, when edge images with different channel numbers are acquired in the imaging device 1, appropriate edge characteristics can be individually adjusted for each channel number of the edge image. In other words, the edge characteristics can be adjusted for each edge direction information.

[0106] The flow of processing when adjusting appropriate edge characteristics for each channel number of edge images in the imaging device 1 will be described with reference to the flowchart of FIG.

[0107] In step S91, the edge image acquisition unit 41 acquires an N-ch edge image. For example, N is an integer equal to or greater than 1, such as 1, 2, or 4, and an edge image such as a 4-ch edge image is acquired. In step S92, the counter 33 counts the edges included in the acquired N-ch edge image. In step S93, the edge characteristic calculation unit 34 calculates edge characteristics such as edge amount and edge direction information based on the counted edges.

[0108] In step S94, the edge characteristic control unit 35 determines whether the edge characteristics in the N-ch edge image to be processed are appropriate. If it is determined in step S94 that the edge characteristics are not appropriate, the process proceeds to step S95. In step S95, the edge characteristic control unit 35 performs feedback control so that appropriate edge characteristics are obtained from the N-ch edge image acquired by the edge image acquisition unit 41.

[0109] Upon completion of step S95, the process returns to step S91, and steps S91 to S94 are repeated. That is, edge determination is performed again using thresholds according to the parameters changed by feedback, and edge characteristics are calculated from the resulting N-channel edge image to determine whether the edge characteristics are appropriate. If it is determined in step S94 that the edge characteristics in the N-channel edge image are appropriate, the process proceeds to step S96. In step S96, the signal processing unit 12 performs signal processing using the N-channel edge image from the imaging element 11. Upon completion of step S96, the series of processes ends. For example, the series of processes may be repeated for each edge image with a different number of channels, such as performing the series of processes on a 2-channel edge image after completing the series of processes on a 4-channel edge image.

[0110] <Adjusting Appropriate Edge Characteristics According to Signal Processing Results> When detecting an object from an edge image in the signal processing unit 12, if a certain subject is captured (even though the subject appears when the edge image is displayed on the screen) and the signal processing fails to recognize the subject, one possible cause is that the edge amount is inappropriate. Thus, when a specific subject cannot be recognized, the target edge amount can be increased or decreased to improve the recognition probability (confidence) of the specific subject, thereby adjusting the edge amount to a level appropriate for recognition. Furthermore, even when a specific subject is recognized, adjusting the edge amount to further improve the recognition probability allows for more stable recognition of the subject.

[0111] The flow of processing when adjusting edge characteristics appropriately in accordance with the signal processing results in the imaging device 1 will be described with reference to the flowchart of FIG.

[0112] In steps S111 to S116, similar to steps S11 to S16 in Fig. 15, the edge image acquisition unit 41 acquires an edge image (S111), the counter 33 counts edges (S112), and the edge characteristic calculation unit 34 calculates edge characteristics (S113). If the edge characteristics are not appropriate (S114: No), feedback control (S115) is performed by the edge characteristic control unit 35. If the edge characteristics are appropriate (S114: Yes), signal processing is performed by the signal processing unit 12 (S116). Here, object detection processing using the edge image is performed as the signal processing.

[0113] In step S117, the signal processing unit 12 determines whether a specific subject has been detected by the object detection processing. If it is determined in step S117 that a specific subject has not been detected, the processing proceeds to step S118. In step S118, the edge characteristic control unit 35 increases or decreases (increases or decreases) the target edge amount based on the signal processing result from the signal processing unit 12. When the processing of step S118 ends, the processing proceeds to step S115. In step S115, the edge characteristic control unit 35 performs feedback control so that appropriate edge characteristics (edge ​​amount) corresponding to the increased or decreased target edge amount are obtained from the edge image acquired by the edge image acquisition unit 41.

[0114] If it is determined in step S117 that a specific subject has been detected, the process proceeds to step S119. In step S119, the signal processing unit 12 determines whether or not to further adjust the edge characteristics. If it is determined in step S119 that the edge characteristics are to be further adjusted, the process proceeds to step S118. In steps S118 and S115, the edge characteristics control unit 35 increases or decreases the target edge amount based on the signal processing result from the signal processing unit 12, and performs feedback control to achieve appropriate edge characteristics corresponding to the increased or decreased target edge amount. On the other hand, if it is determined in step S119 that the edge characteristics are not to be further adjusted, the process ends.

[0115] An example of a processing flow will now be described with reference to FIG. 24. FIG. 24 is a diagram showing an example of the edge amount for each image frame. FIG. 24A shows an image capture scene for comparison with FIG. 24B to FIG. 24D. FIG. 24B shows the edge image of the first frame, FIG. 24C shows the edge image of the second frame, and FIG. 24C shows the edge image of the third frame, showing that adjusting the edge amount for each image frame improves the recognition probability of a specific subject. It is assumed that the initial target edge amount is set to 7 to 10%.

[0116] 23 are performed in order, the edge image of the first frame (B in FIG. 24) acquired in step S111 is used as the edge image to be processed, and if the calculated edge amount is 3%, feedback control is performed so that the edge amount becomes the target amount (7 to 10%). When step S115 is completed, the process returns to step S111.

[0117] Next, by performing the processes of steps S111 to S114 in order for the second time, the edge image of the second frame (C in FIG. 24) acquired in the process of step S111 is processed as the edge image to be processed. If the calculated edge amount is 8%, this is the target edge amount (7 to 10%), and the process proceeds to step S116. In step S116, the signal processing unit 12 performs object detection processing using the edge image of the second frame, but because the recognition process failed (S117: No), the target edge amount is increased and feedback control is performed (S118, S115). When the process of step S115 is completed, the process returns to step S111.

[0118] Next, by performing the processes of steps S111 to S114 in order for the third time, the edge image of the third frame (D in FIG. 24) acquired in the process of step S111 is processed as the edge image to be processed, and if the calculated edge amount is 11%, it becomes the target edge amount (the target edge amount increased from the initial target edge amount of 7 to 10%), so the process proceeds to step S116. In step S116 for the second time, the signal processing unit 12 performs object detection processing using the edge image of the third frame, and since the recognition process was also successful (S117: Yes), if further adjustment of the edge amount is not to be performed (S119: No), the series of processes ends.

[0119] <Adjusting Appropriate Edge Characteristics Using Sensing Information> If the imaging device 1 has a sensor that senses the surrounding environment, physical information, etc., the edge characteristics can be adjusted using sensing information (external world information) obtained by sensing with the sensor. Fig. 25 shows a configuration in which a sensor is provided in the imaging device 1. In Fig. 25, parts corresponding to those in Fig. 1 are assigned the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0120] Compared to the imaging device 1 of FIG. 1 , the imaging device 1 of FIG. 25 additionally includes a sensor 13 that senses the surrounding environment, physical information, etc., and a control unit 14 that controls each unit including the sensor 13. The sensor 13 is composed of various sensors such as an acceleration sensor, a rotational angular acceleration sensor, a gyro sensor, a magnetic field sensor, a barometric pressure sensor, and a temperature sensor. The sensor 13 performs sensing under control of the control unit 14 and outputs sensing information obtained by the sensing to the imaging element 11. The imaging element 11 adjusts edge characteristics using the sensing information input from the sensor 13. Note that the sensor 13 may be provided external to the imaging device 1, and sensing information obtained by the external sensor 13 may be input to the imaging element 11 via a predetermined interface.

[0121] For example, if an acceleration sensor is provided as the sensor 13, it is expected that when the acceleration detected by the acceleration sensor is large, blurring occurs, making it difficult to detect edges in the edge image. Therefore, the image sensor 11 adjusts the target edge characteristics using acceleration information from the acceleration sensor, thereby adjusting the edge gain and offset that set the thresholds used in edge determination. As a result, the image sensor 11 can acquire an edge image suitable for signal processing while suppressing blurring through sensor fusion with the sensor 13.

[0122] The flow of processing when adjusting appropriate edge characteristics using sensing information in the imaging device 1 will be described with reference to the flowchart of FIG.

[0123] In step S131, the edge characteristic control unit 35 acquires sensing information input from the sensor 13. In step S132, the edge characteristic control unit 35 adjusts the target edge characteristic (for example, the target edge amount) based on the acquired sensing information.

[0124] In steps S133 to S135, similar to steps S11 to S13 in FIG. 15, the edge image is acquired by the edge image acquisition unit 41 (S133), the edges are counted by the counter 33 (S134), and the edge characteristics are calculated by the edge characteristic calculation unit 34 (S135).

[0125] In step S136, the edge characteristic control unit 35 determines whether the edge characteristics of the edge image to be processed are appropriate. Here, the difference between the evaluation value of the edge characteristic to be controlled and the target value is defined as the error amount, an appropriate value is set from the error amount relative to the target value, and whether the error amount exceeds the appropriate value is determined to determine whether the edge characteristics are appropriate. However, the target value corresponds to the target edge characteristic adjusted based on the sensing information in the processing of step S132.

[0126] In step S137, similar to step S15 in Fig. 15, if it is determined that the edge characteristics are not appropriate (S136: No), feedback control is performed by the edge characteristics control unit 35. Also, in step S138, similar to step S16 in Fig. 16, if it is determined that the edge characteristics are appropriate (S136: Yes), signal processing is performed by the signal processing unit 12. When the processing of step S138 is completed, the series of processing steps ends.

[0127] <Adjusting Edge Characteristics Suitable for Optical Flow Estimation> Signal processing by the signal processing unit 12 may involve signal processing according to optical flow estimation using an edge image. Optical flow estimation calculates the direction and amount of movement of a subject between image frames. FIG. 27 is a diagram illustrating an overview of optical flow estimation. FIG. 27A shows a subject S rotating in the direction of the arrow R. FIG. 27B shows an example of optical flow inferred from an image including the subject S. The optical flow is a vector representing the movement of the subject S. As shown in FIG. 27C, in optical flow, directional information can be represented by hue (circle H in the figure), and velocity information can be represented by brightness. Note that there has been much research into edge detection and optical flow estimation using signal processing on RGB images.

[0128] The edge image can retain information on the spatial direction of pixels where discontinuous luminance changes occur, as well as information on luminance gradients. This makes it easy to search for corresponding points between image frames, making it well suited to optical flow estimation. Optical flow estimation involves matching feature points between image frames, so a lack of edges can lead to a decrease in the accuracy of the inference process by the signal processing unit 12. However, simply increasing the gain increases noise in the edge image, increasing the calculation cost for searching for corresponding points between image frames. Therefore, the edge control unit 42 controls the edge gain and offset to increase the degree to which the luminance gradient direction vectors of adjacent edges match.

[0129] This makes it easier to obtain feature points when searching for corresponding points between image frames using edge images, and makes it possible to obtain an appropriate edge image as the edge image used for optical flow estimation by the signal processing unit 12. In other words, it is expected that appropriate edge characteristics will differ depending on the task of signal processing performed using the edge image, but the present disclosure can also handle such cases by changing the feedback according to differences in the signal processing task.

[0130] The flow of processing when adjusting edge characteristics to be suitable for optical flow estimation in the imaging device 1 will be described with reference to the flowchart in FIG.

[0131] In steps S151 to S154, similar to steps S11 to S14 in FIG. 15, the edge image is acquired by the edge image acquisition unit 41 (S151), the edges are counted by the counter 33 (S152), and the edge characteristics are calculated by the edge characteristic calculation unit 34 (S153), and it is determined whether the edge characteristics of the edge image to be processed are appropriate (S154).

[0132] If it is determined in step S154 that the edge characteristics are not appropriate, the process proceeds to step S155. In step S155, the edge characteristics control unit 35 performs feedback control to obtain edge characteristics suitable for optical flow estimation by the signal processing unit 12. For example, the edge characteristics control unit 35 adjusts the edge gain and offset that set the threshold used in edge determination so that the degree of coincidence between the directional vectors of the brightness gradients of adjacent edges increases.

[0133] If it is determined in step S154 that the edge characteristics are appropriate, the process proceeds to step S156. In step S156, the signal processing unit 12 performs signal processing according to the optical flow estimation using the edge image input from the image sensor 11. When the process of step S156 ends, the series of processes ends.

[0134] <Feedback to Shutter Control and Lens Control> Whether or not an edge can be extracted depends greatly on the amount of light incident on the light receiving surface of the image sensor 11 and the blur of the subject. Therefore, in the image capture device 1, shutter control and lens control are included as targets of feedback by the edge control unit 42, so that exposure time and focusing can be adjusted. Figure 29 shows a configuration for feedback control of shutter control and lens control in the image capture device 1. In Figure 29, parts corresponding to those in Figure 1 are assigned the same reference numerals, and their description will be omitted as appropriate.

[0135] 29 is additionally provided with a control unit 14 that controls the operation of each unit, and an optical system 15 that includes a lens that focuses light from the subject on the light receiving surface of the image sensor 11, as compared to the image sensor 1 of FIG. 1. The control unit 14 controls the lenses included in the optical system 15 in accordance with feedback from (the edge control unit 42 of) the image sensor 11. There are various shutter methods, but for example, when controlling the image sensor 11 to achieve a shutter function, the control unit 14 controls the shutter function of the image sensor 11 in accordance with feedback from (the edge control unit 42 of) the image sensor 11.

[0136] For example, when the amount of light incident on the light receiving surface of the image sensor 11 is small, edges are less likely to appear, so the control unit 14 performs shutter control to lengthen the exposure time in accordance with feedback from the edge control unit 42. By lengthening the exposure time through shutter control, a sufficient amount of light can be secured, making edges more likely to appear. On the other hand, by shortening the exposure time through shutter control, it becomes more difficult to secure the amount of light, making edges less likely to appear.

[0137] Furthermore, when blurring occurs in the edge image due to the movement of the subject, the edge is less likely to appear, so the control unit 14 performs shutter control to shorten the exposure time in accordance with feedback from the edge control unit 42. By shortening the exposure time through shutter control, it is possible to reduce blurring caused by the movement (movement) of the subject. Furthermore, when blurring occurs in the edge image due to the subject being out of focus, it is less likely to appear, so the control unit 14 performs lens control to bring the subject into focus in accordance with feedback from the edge control unit 42. By focusing on the subject through lens control, it is possible to reduce blurring of the subject. In this way, a more appropriate edge image can be obtained by combining shutter control and lens control with the adjustment of edge gain and offset as feedback control.

[0138] The flow of processing when feedback is given to shutter control and lens control in the imaging device 1 will be described with reference to the flowchart of FIG.

[0139] In steps S171 to S174, similar to steps S11 to S14 in FIG. 15, the edge image is acquired by the edge image acquisition unit 41 (S171), the edges are counted by the counter 33 (S172), and the edge characteristics are calculated by the edge characteristic calculation unit 34 (S173), and it is determined whether the edge characteristics of the edge image to be processed are appropriate (S174).

[0140] If it is determined in step S174 that the edge characteristics are not appropriate, the process proceeds to step S175. In step S175, the edge characteristics control unit 35 performs feedback control not only on the edge image acquisition unit 41 but also on the control unit 14, which performs shutter control and lens control, so that appropriate edge characteristics can be obtained from the edge image acquired by the edge image acquisition unit 41. This allows the control unit 14 to perform at least one of shutter control and lens control in accordance with feedback from the edge control unit 42 (the edge characteristics control unit 35 thereof).

[0141] 15, in step S176, if it is determined that the edge characteristics are appropriate (S174: Yes), signal processing is performed by the signal processing unit 12. When the processing of step S176 ends, the series of processes ends.

[0142] <Modification> In the above description, the imaging device 1 is provided with the signal processing unit 12. However, the signal processing unit may be provided external to the imaging device 1. For example, the signal processing unit may be provided in a signal processing device such as a PC (Personal Computer) to process edge images. FIG. 31 is a block diagram showing an example configuration of an embodiment of a system to which the present disclosure is applied. In FIG. 31, the system 101 includes an imaging device 1 equipped with an imaging element 11 and a signal processing device 102 such as a PC. The imaging device 1 and the signal processing device 102 are connected via a wireless or wired communication line. The signal processing device 102 has a signal processing unit having the same function as the signal processing unit 12 and performs signal processing using edge images output from the imaging device 1. The signal processing device 102 can also output signal processing results obtained by the signal processing and control values ​​for controlling the imaging element 11 to the imaging device 1. Note that in FIG. 31, the imaging device 1 includes the imaging element 11, but whether or not the signal processing unit 12 is provided is optional. The edge images output from the imaging element 11 or the signal processing unit 12 are input to the signal processing device 102. The imaging device 1 and the imaging element 11 can also be considered as a sensor (edge ​​sensor) that acquires an edge image. The imaging device 1 may be mounted on an electronic device such as a smartphone.

[0143] In the above description, the edge control unit 42 of the logic unit 25 of the image sensor 11 performs feedback control on the edge image acquisition unit 41 based on the edge characteristics obtained from the edge image output from the edge image acquisition unit 41 as well as the signal processing result output from the signal processing unit 12. However, the control may also be performed based only on the signal processing result output from the signal processing unit 12. As shown in FIG. 2 and other figures, the counter 33 is necessary in the logic unit 25 of the image sensor 11 when the edge amount is used as the edge characteristic, but is not necessarily required when the edge amount is not used as the edge characteristic. In other words, in the logic unit 25 of the image sensor 11, the edge control unit 42 may be composed of the edge characteristic calculation unit 34 and the edge characteristic control unit 35.

[0144] As described above, the present disclosure provides an imaging device 1 including an edge image acquisition unit 41 that acquires an edge image, which is an image in which edges have been extracted; an edge control unit 42 that evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the edge image acquisition unit 41; and a signal processing unit 12 that performs signal processing using the acquired edge image. The imaging device 1 configured as described above can acquire an edge image having appropriate characteristics, such as characteristics suited to the operating environment, through feedback control. This makes it possible to acquire an edge image suitable for signal processing, and to perform signal processing using an edge image that maintains consistent quality regardless of the surrounding environment or the subject. For example, signal processing that is robust to environmental fluctuations can be realized.

[0145] Due to their characteristics, sensors that capture edge images (edge ​​sensors) are more susceptible to the influence of the surrounding environment and subject movement than sensors that capture RGB images (RGB sensors), making it difficult to stabilize the characteristics of the edge image. Edge sensors use analog circuits to determine whether the difference in luminance values ​​between pixels exceeds a threshold and output the result as an edge image. When performing signal processing using this edge image, discontinuous or excessively large edges in the subject can make subsequent signal processing difficult. Therefore, the threshold used for edge determination must be appropriately adjusted depending on the surrounding brightness and the shape and material of the subject. This threshold is determined by parameters such as edge gain and offset. In the present disclosure, the sensor is controlled to output edges suitable for signal processing by feeding back appropriate edge characteristics (e.g., edge amount) to these parameters. By controlling the edge characteristics of the entire edge image or a specific region to be appropriate (desired edge characteristics), the edge image can be maintained at a consistent quality regardless of the sensor's surrounding environment or the subject, enabling signal processing that is robust to environmental fluctuations.

[0146] The aforementioned Patent Document 2 discloses a technology for performing edge extraction and edge evaluation on an RGB image in a system that performs recognition processing using an RGB image, and controls an imaging device based on the evaluation value. However, the technology disclosed in the aforementioned Patent Document 2 does not proceed to the evaluation step if the initial recognition fails, and therefore cannot respond to environmental changes (such as sudden changes in brightness or blurring) that occur during recognition. It is a technology that is only intended to output an image that is more suitable for recognition when recognition is successful. Furthermore, although edge direction information is obtained in the edge extraction stage, it is converted into a histogram, which presents a problem in that spatial direction information, such as the relationship between adjacent edges, cannot be used.

[0147] It should be noted that the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0148] The present disclosure can also be configured as follows.

[0149] (1) An imaging device comprising: an acquisition unit that acquires an edge image that is an image in which edges are extracted; a control unit that evaluates edge characteristics that are characteristics related to the edges obtained from the acquired edge image and controls feedback so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit; and a signal processing unit that performs signal processing using the acquired edge image. (2) The imaging device described in (1), wherein the control unit controls feedback so that the edge characteristics obtained from the edge image become edge characteristics appropriate for the operating environment. (3) The imaging device described in (1), wherein the control unit controls feedback so that appropriate edge characteristics are obtained from the edge image based on a signal processing result by the signal processing unit. (4) The imaging device described in any of (1) to (3), wherein the acquisition unit has an edge determination unit that compares a difference in luminance values ​​between a first pixel and a second pixel that are arranged nearby among a plurality of pixels arranged two-dimensionally in a pixel array unit with a threshold value to determine an edge, and acquires the edge image based on a determination result by the edge determination unit. (5) The imaging device according to (4), wherein the threshold is a variable threshold set by an edge gain and an offset, and the control unit controls the threshold by feeding back at least one of the edge gain and the offset so as to obtain appropriate edge characteristics from the edge image. (6) The imaging device according to any of (1) to (3), wherein the control unit has: an edge characteristic calculation unit that calculates edge characteristics obtained from the acquired edge image, and an edge characteristic control unit that adjusts an amount of a parameter to be fed back based on the calculated edge characteristics so as to obtain appropriate edge characteristics from the edge image acquired by the acquisition unit. (7) The imaging device according to (6), wherein the edge characteristics include at least one of an edge amount indicating the proportion of edge pixels to all pixels in an effective area, edge direction information related to the direction of the edge, and edge connectivity information indicating the degree of connectivity of the edge pixels.(8) The imaging device according to (7), wherein the edge connectivity information is information obtained by calculating the degree of connectivity between edge pixels, assuming that edge pixels are connected when they are adjacent in any of the up-down, left-right, or up-down, left-right, or diagonal directions, or when adjacent edge pixels have the same vector direction. (9) The imaging device according to (6), wherein the acquisition unit has an edge determination unit that determines an edge by comparing a difference in luminance values ​​between a first pixel and a second pixel that are arranged nearby among a plurality of pixels arranged two-dimensionally in a pixel array unit with a threshold, and the edge characteristic calculation unit calculates the edge characteristic based on the determination result output from the edge determination unit. (10) The imaging device according to (9), wherein the control unit further has a counter that counts the number of edges based on the determination result output from the edge determination unit, and the edge characteristic calculation unit calculates the edge characteristic based on the count result by the counter. (11) The imaging device described in (7), wherein the edge connectivity information is information obtained by calculating the degree of connectivity between edge pixels using a line buffer that stores pixel information obtained by scanning a pixel array unit in which a plurality of pixels are arranged two-dimensionally in a predetermined direction. (12) The imaging device described in any of (1) to (3), wherein the control unit performs feedback control to bring current edge characteristics, which are edge characteristics obtained from the edge image, closer to target edge characteristics. (13) The imaging device described in (12), wherein the control unit adjusts the target edge characteristics based on signal processing results from the signal processing unit. (14) The imaging device described in (12), wherein the control unit adjusts the target edge characteristics based on sensing information obtained by sensing by a sensor. (15) The imaging device described in any of (1) to (3), wherein the signal processing unit uses the edge image to perform inference processing using a trained model. (16) The imaging device according to (1), wherein the signal processing unit performs signal processing according to optical flow estimation using the edge image, and the control unit performs feedback control so that edge characteristics obtained from the edge image become edge characteristics suitable for the optical flow estimation.(17) The imaging device according to any of (1) to (3), wherein the control unit feeds back information regarding control according to an evaluation result of edge characteristics obtained from the edge image to another control unit that performs at least one of control related to a shutter during imaging and control related to a lens. (18) An imaging element comprising: an acquisition unit that acquires an edge image that is an image from which edges have been extracted; and a control unit that evaluates edge characteristics that are characteristics related to the edges obtained from the acquired edge image and performs feedback control so that appropriate edge characteristics can be obtained from the edge image acquired by the acquisition unit. (19) An imaging method, wherein an imaging device acquires an edge image that is an image from which edges have been extracted, evaluates edge characteristics that are characteristics related to the edges obtained from the acquired edge image, and performs feedback control so that appropriate edge characteristics can be obtained from the acquired edge image, and performs signal processing using the acquired edge image. (20) A system comprising an imaging device and a signal processing device, wherein the imaging device comprises: an acquisition unit that acquires an edge image, which is an image in which edges are extracted; and a control unit that evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit; and the signal processing device comprises a signal processing unit that performs signal processing using the edge image output from the imaging device.

[0150] DESCRIPTION OF SYMBOLS 1 Imaging device, 11 Imaging element, 12 Signal processing unit, 13 Sensor, 14 Control unit, 15 Optical system, 21 Pixel array unit, 22 Row scanning unit, 23 Edge determination unit, 24 Column scanning unit, 25 Logic unit, 26 Input / output unit, 31 Timing control unit, 32 Operation control unit, 33 Counter, 34 Edge characteristic calculation unit, 35 Edge characteristic control unit, 36 Area designation unit, 37 Area designation unit, 41 Edge image acquisition unit, 42, 42A, 42B Edge control unit, 51, 51-1, 51-2 Pixel, 52, 52-1, 52-2 Comparator, 53A-1, 53A-2, 53B-1, 53B-2 Capacitor, 101 System, 102 Signal processing device

Claims

1. An imaging device comprising: an acquisition unit that acquires an edge image, which is an image in which edges are extracted; a control unit that evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit; and a signal processing unit that performs signal processing using the acquired edge image.

2. The imaging device according to claim 1, wherein the control unit performs feedback control so that edge characteristics obtained from the edge image become edge characteristics suited to the operating environment.

3. The imaging device according to claim 1, wherein the control unit performs feedback control based on the result of signal processing by the signal processing unit so that appropriate edge characteristics are obtained from the edge image.

4. The imaging device described in claim 1, wherein the acquisition unit has an edge determination unit that determines an edge by comparing a difference in luminance value between a first pixel and a second pixel that are arranged in close proximity to one another among a plurality of pixels that are arranged two-dimensionally in a pixel array unit with a threshold value, and acquires the edge image based on a determination result by the edge determination unit.

5. The imaging device according to claim 4, wherein the threshold is a variable threshold set based on an edge gain and an offset, and the control unit feeds back at least one of the edge gain and the offset to control the threshold so that appropriate edge characteristics are obtained from the edge image.

6. The imaging device according to claim 1, wherein the control unit has: an edge characteristic calculation unit that calculates edge characteristics obtained from the acquired edge image; and an edge characteristic control unit that adjusts the amount of a parameter to be fed back based on the calculated edge characteristics so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit.

7. The imaging device according to claim 6, wherein the edge characteristics include at least one of the following information: edge amount indicating the proportion of edge pixels to all pixels in the effective area, edge direction information regarding the edge direction, and edge connectivity information indicating the degree of connectivity of edge pixels.

8. The imaging device described in claim 7, wherein the edge connectivity information is information obtained by calculating the degree of connectivity between edge pixels when the edge pixels are adjacent in either the up / down, left / right, or up / down, left / right and diagonal directions, or when the vector directions of adjacent edge pixels are the same, the edge pixels are considered to be connected.

9. The imaging device described in claim 6, wherein the acquisition unit has an edge determination unit that determines an edge by comparing a difference in luminance value between a first pixel and a second pixel that are arranged in close proximity to one another among a plurality of pixels that are arranged two-dimensionally in a pixel array unit with a threshold value, and the edge characteristic calculation unit calculates the edge characteristics based on the determination result output from the edge determination unit.

10. The imaging device of claim 9, wherein the control unit further has a counter that counts the number of edges based on the judgment result output from the edge judgment unit, and the edge characteristic calculation unit calculates the edge characteristics based on the counting result by the counter.

11. The imaging device according to claim 7, wherein the edge connectivity information is information obtained by calculating the degree of connectivity between edge pixels using a line buffer that stores pixel information obtained by scanning a pixel array section in which a plurality of pixels are arranged two-dimensionally in a predetermined direction.

12. The imaging device according to claim 1, wherein the control unit performs feedback control so as to bring current edge characteristics, which are edge characteristics obtained from the edge image, closer to target edge characteristics.

13. The imaging device according to claim 12, wherein the control unit adjusts the target edge characteristics based on a signal processing result from the signal processing unit.

14. The imaging device according to claim 12, wherein the control unit adjusts edge characteristics of the target based on sensing information obtained by sensing using a sensor.

15. The imaging device according to claim 1, wherein the signal processing unit uses the edge image to perform inference processing based on a trained model.

16. The imaging device according to claim 1, wherein the signal processing unit uses the edge image to perform signal processing according to optical flow estimation, and the control unit performs feedback control so that edge characteristics obtained from the edge image become edge characteristics suitable for the optical flow estimation.

17. The imaging device according to claim 1, wherein the control unit feeds back information regarding control in response to the evaluation result of edge characteristics obtained from the edge image to another control unit that performs at least one of control related to the shutter during imaging and control related to the lens.

18. An imaging element comprising: an acquisition unit that acquires an edge image, which is an image in which edges are extracted; and a control unit that evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image, and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit.

19. An imaging method in which an imaging device acquires an edge image which is an image in which edges are extracted, evaluates edge characteristics which are characteristics related to the edges obtained from the acquired edge image, performs feedback control so that appropriate edge characteristics are obtained from the acquired edge image, and performs signal processing using the acquired edge image.

20. A system consisting of an imaging device and a signal processing device, wherein the imaging device comprises an acquisition unit that acquires an edge image, which is an image with extracted edges, and a control unit that evaluates edge characteristics, which are characteristics related to the edges, obtained from the acquired edge image and performs feedback control so that appropriate edge characteristics are obtained from the edge image acquired by the acquisition unit, and the signal processing device comprises a signal processing unit that performs signal processing using the edge image output from the imaging device.

Images