Image recognition device and image recognition method
By using multi-exposure technology to generate high-sensitivity, medium-sensitivity and low-sensitivity images within the same frame period, and using DNN to identify subjects from image data that have not been synthesized by HDR, the reduction in subject recognition accuracy in the prior art is solved. The problem has achieved higher recognition accuracy and accuracy.
Patent Information
- Application Number
- CN202080038430.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-30
- Filing Date
- 2020-05-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the recognition accuracy of the subject may sometimes be reduced, especially in high dynamic range images due to the influence of artifacts.
Multiple exposures are performed in the same frame period using imaging pixels that receive visible and infrared light to generate high-sensitivity, medium-sensitivity and low-sensitivity images, and are identified by deep neural networks (DNNs) from image data that is not synthesized by HDR. The subject eliminates the influence of artifacts.
The recognition accuracy of the subject is improved, the impact of artifacts on the recognition results is reduced, and the accuracy of the image recognition system is enhanced.
Smart Images

Figure CN113875217B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image recognition device and an image recognition method. Background Art
[0002] There is an imaging device that is provided with a high dynamic range (HDR) imaging mode that generates an HDR image by imaging and synthesizing a plurality of images having different sensitivities in one frame period, and recognizes a subject from the HDR image (for example, see Patent Document 1).
[0003] Furthermore, there is an imaging device that generates an HDR image by synthesizing images imaged by using imaging pixels that receive visible light and imaging pixels that receive infrared light, and recognizes a subject from the HDR image.
[0004] Prior art literature
[0005] Patent Literature
[0006] Patent Document 1: JP 2014-103643 A Summary of the invention
[0007] Technical Problems to be Solved by the Invention
[0008] However, in the above conventional technology, the recognition accuracy of the subject may be reduced. Therefore, the present disclosure proposes an image recognition device and an image recognition method capable of improving the recognition accuracy of the subject.
[0009] Solution to the problem
[0010] The image recognition device according to the present disclosure includes an imaging unit and a recognition unit. The imaging unit uses imaging pixels that receive visible light and imaging pixels that receive infrared light, and images multiple images at the same exposure timing in one frame period to generate image data. The recognition unit recognizes a subject from each piece of image data. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 is an explanatory diagram showing a pixel array according to the present disclosure;
[0012] Figure 2A is an explanatory diagram showing an example of using a general-purpose DNN;
[0013] Figure 2B is an explanatory diagram showing an example of using a DNN according to the present disclosure;
[0014] Figure 3 is a diagram showing a configuration example of an image recognition system according to the present disclosure;
[0015] Figure 4is an explanatory diagram of processing performed by a signal processing unit according to the present disclosure;
[0016] Figure 5A is an explanatory diagram of a process performed by a recognition unit according to the present disclosure;
[0017] Figure 5B is an explanatory diagram of a process performed by a recognition unit according to the present disclosure;
[0018] Figure 6 is an explanatory diagram showing a modified example of processing performed by a signal processing unit according to the present disclosure;
[0019] Figure 7 is an explanatory diagram showing a modified example of a DNN used by a recognition unit according to the present disclosure;
[0020] Figure 8 is an explanatory diagram showing a first modified example of a pixel array according to the present disclosure;
[0021] Fig. 9 is an explanatory diagram showing a second modification example of the pixel array according to the present disclosure;
[0022] Fig.10 is an explanatory diagram showing an example of using a DNN when a second modification example of a pixel array according to the present disclosure is adopted;
[0023] Fig.11 is an explanatory diagram showing a third modification example of the pixel array according to the present disclosure;
[0024] Fig.12 is a block diagram showing an example of a schematic configuration of a vehicle control system;
[0025] Fig.13 2 is an explanatory diagram showing an example of installation positions of the vehicle exterior information detection unit and the imaging unit. DETAILED DESCRIPTION
[0026] A detailed description of embodiments of the present disclosure will be given below with reference to the accompanying drawings. Note that in the following respective embodiments, the same reference numerals are given to the same parts, and repeated descriptions will be omitted.
[0027] [1. Overview of image recognition methods]
[0028] First, a description of an overview of an image recognition method according to the present disclosure will be given. In the image recognition method according to the present disclosure, multiple images are imaged at the same exposure timing in one frame period using imaging pixels that receive visible light and imaging pixels that receive infrared light to generate image data, and a subject is recognized from each piece of image data. Here, such imaging at the same exposure timing means, for example, imaging an image by starting exposure of multiple pixels at multiple separate timings and ending exposure at the same timing.
[0029] In order to image the multiple images, a pixel array for imaging a high dynamic range (HDR) image is used. Figure 1 is an explanatory diagram showing a pixel array according to the present disclosure. Figure 1 As shown, in a pixel array A1 for imaging an HDR image, imaging pixels R receiving red light, imaging pixels Gr and Gb receiving green light, and imaging pixels B receiving blue light are arranged in a Bayer array.
[0030] Note that the imaging pixel Gr is an imaging pixel disposed in the row where the imaging pixel R is located, and receives green light. The imaging pixel Gb is an imaging pixel disposed in the row where the imaging pixel B is located, and receives green light.
[0031] Each of the imaging pixels R, Gr, Gb, and B includes three imaging elements, which are arranged in an L shape and have the same light receiving area. The imaging element L is a long exposure element having an exposure time longer than that of the imaging elements M and S. The imaging element S is a short exposure element having an exposure time shorter than that of the imaging elements L and M.
[0032] The imaging element M is an intermediate-time exposure element having an exposure time longer than that of the imaging element S and shorter than that of the imaging element L. In addition, at each position surrounded on three sides by the imaging elements L, M, and S that receive light of the same color and are arranged in an L shape, the pixel array A1 includes an imaging element IR serving as an imaging pixel that receives infrared light.
[0033] The imaging element L has a long exposure time, so a sufficient amount of received light can be obtained even when its periphery is dark. Therefore, even if the subject is a subject with too low brightness and underexposed in a normal exposure time, the imaging element L can image a high-sensitivity image on which the image of the subject is clearly reflected (hereinafter, the high-sensitivity image will also be referred to as a "long-exposure image" in some cases).
[0034] The imaging element S has a short exposure time, and therefore, is not saturated even when its periphery is bright. Therefore, even if the subject is a subject with excessively high brightness and overexposed within a normal exposure time, the imaging element S can image a low-sensitivity image on which the image of the subject is clearly reflected (hereinafter, the low-sensitivity image will also be referred to as a "short-time exposure image" in some cases). Note that the imaging element M can image a medium-sensitivity image (hereinafter, referred to as an "intermediate-time exposure image" in some cases) within an exposure time similar to that of an ordinary digital camera.
[0035] The imaging element IR receives infrared light, and therefore can image an infrared (IR) image in which the following images are clearly reflected, for example, the image includes: pedestrians or vehicles in the dark, which are difficult to see with the naked eye; pedestrians that are difficult to see due to vehicle headlights or spotlights; and subjects that are difficult to see due to smoke. In addition, the imaging element infrared can also sense the thermal information of the subject.
[0036] An HDR image is generated by performing HDR synthesis on the high-sensitivity image, medium-sensitivity image, low-sensitivity image, and infrared image as described above. Therefore, the HDR image becomes an image in which all subjects (e.g., dark subjects, bright subjects, and subjects that are difficult to see with the naked eye) are clearly reflected.
[0037] Note that Figure 1 In the pixel array A1 shown, the exposure time is differentiated, whereby a high-sensitivity image, a medium-sensitivity image, or a low-sensitivity image is imaged by the imaging element L, M, or S having the same light receiving area; however, this is an example. For example, the pixel array A1 differentiates the transmittance of the color filter to be stacked on each imaging pixel, so that a high-sensitivity image, a medium-sensitivity image, or a low-sensitivity image can be imaged even if the exposure time setting is the same.
[0038] In this case, in the pixel array A1, the color filter whose transmittance is higher than that of a normal imaging pixel becomes a high-sensitivity imaging pixel, the color filter whose transmittance is the same as that of a normal imaging pixel becomes a medium-sensitivity imaging pixel, and the color filter whose transmittance is lower than that of a normal imaging pixel becomes a low-sensitivity imaging pixel.
[0039] Note that even if a method of equalizing the exposure start time and distinguishing the exposure end time, a method of distinguishing the exposure start time and equalizing the exposure end time, or a method of distinguishing the exposure start time and the exposure end time is used as a method of distinguishing the exposure start time, a high-sensitivity image, a medium-sensitivity image, or a low-sensitivity image can be imaged.
[0040] In addition, by using such a pixel array A1, a high-sensitivity image and a low-sensitivity image can be imaged at the same time, so an HDR image can be imaged by performing HDR synthesis on the two images. In addition, by distinguishing the light receiving areas of the imaging pixels, it is also possible to use the imaging pixels with the same color filter transmittance and the same exposure time as high-sensitivity imaging pixels or low-sensitivity imaging pixels.
[0041] Herein, as an example of a method for recognizing a subject from image data, there is an image recognition method using a deep neural network (DNN). DNN is an algorithm having a multi-layer structure that uses a human brain neural circuit (neural network) designed by machine learning as a model to recognize features (patterns) of a subject from image data.
[0042] Figure 2A is an explanatory diagram showing an example of using a general-purpose DNN. For example, in the case of recognizing a subject from an HDR image, as Figure 2A As shown, when the signals R, G and B of the corresponding pixels of red R, green G and blue B in the HDR image that has been HDR synthesized (image data of the HDR image) are input to the DNN, the recognition result of the subject is output from the DNN.
[0043] However, since the HDR image is HDR-synthesized, artifacts that do not actually exist are sometimes reflected in it. Therefore, when the signals R, G, and B of each pixel in the HDR image are input to the DNN, the DNN's recognition accuracy of the subject is sometimes reduced due to the harmful effects of the artifacts.
[0044] Therefore, in the present disclosure, the subject is identified from each of the high-sensitivity image, the medium-sensitivity image, the low-sensitivity image and the infrared image that have not yet been subjected to HDR synthesis, thereby eliminating the influence of such artifacts and improving the recognition accuracy of the subject. Figure 2B is an explanatory diagram showing an example of using a DNN according to the present disclosure.
[0045] like Figure 2B As shown, in the present disclosure, for example, long time exposure signals R, Gr, Gb, and B (image data of a high-sensitivity image) to be output from the imaging element L and intermediate time exposure signals R, Gr, Gb, and B (image data of a medium-sensitivity image) to be output from the imaging element M are input to the DNN. In addition, in the present disclosure, short time exposure signals R, Gr, Gb, and B (image data of a low-sensitivity image) to be output from the imaging element S and infrared light signals IRr, IRgr, IRgb, and IRb (image data of an infrared image) to be output from the imaging element IR are input to the DNN.
[0046] Therefore, the DNN outputs recognition results of recognizing the subject from the high-sensitivity image, the medium-sensitivity image, the low-sensitivity image, and the infrared image.
[0047] At this time, the DNN recognizes the subject from the high-sensitivity image, the medium-sensitivity image, the low-sensitivity image, and the infrared image that do not include artifacts, and therefore, can output a high-precision recognition result of the subject without being affected by artifacts.
[0048] As described above, in the image recognition method according to the present disclosure, imaging pixels that receive visible light and imaging pixels that receive infrared light are used to image the subject at multiple exposure times in one frame cycle, and the subject is identified by using all of these, thereby improving the recognition accuracy of the subject.
[0049] [2. Configuration of image recognition system]
[0050] Next, refer to Figure 3 , a description will be given of the configuration of an image recognition system according to the present disclosure. Figure 3 : is a diagram showing a configuration example of an image recognition system according to the present disclosure. Figure 3 As shown, the image recognition system 100 according to the present disclosure includes: an image sensor 1, which is an example of an image recognition device; and an application processor (hereinafter referred to as AP 2).
[0051] The image sensor 1 includes an imaging unit 10, a signal processing unit 13, a recognition unit 14, a data transmission determination unit 15, a selector (hereinafter referred to as SEL 16), and a transmission unit 17. The imaging unit 10 includes a pixel array A1 and an analog / digital (A / D) conversion unit 12.
[0052] The pixel array A1 images a high-sensitivity image, a medium-sensitivity image, a low-sensitivity image, and an infrared image through the imaging elements L, M, S, and IR. Then, the pixel array A1 outputs analog pixel signals corresponding to the amount of received light from the imaging elements L, M, S, and IR to the A / D conversion unit 12.
[0053] The A / D conversion unit 12 performs A / D on the analog pixel signal input from the pixel array A1 to convert it into a digital pixel signal, thereby generating image data of a high-sensitivity image, medium-sensitivity image data, low-sensitivity image data, and infrared light image data, and outputs the generated image data to the signal processing unit 13.
[0054] The signal processing unit 13 includes: a microcomputer having a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like; and various circuits.
[0055] The signal processing unit 13 performs predetermined signal processing on each of the image data of the high-sensitivity image, the medium-sensitivity image data, the image data of the low-sensitivity image, and the image data of the infrared light input from the A / D conversion unit 12. Then, the signal processing unit 13 outputs the image data of the high-sensitivity image, the medium-sensitivity image data, the image data of the low-sensitivity image, and the image data of the infrared light that have been subjected to the signal processing to the recognition unit 14 and the SEL 16.
[0056] In this article, reference Figure 4 , a description will be given of the flow of processing performed by the signal processing unit 13. Figure 4 FIG. 1 is an explanatory diagram of processing performed by a signal processing unit according to the present disclosure. Figure 4 As shown, for image data to be input, the signal processing unit 13 first performs shading correction, and then performs mixed color correction.
[0057] Thereafter, the signal processing unit 13 performs digital gain adjustment on the image data, and then performs white balance gain adjustment Thereafter, the signal processing unit 13 performs demosaicing on the image data, and then finally performs gamma correction thereon, and outputs the image data that has been gamma corrected.
[0058] Note that in demosaicing, processing is performed to supplement the colors of each pixel of the high-sensitivity image, the medium-sensitivity image, the low-sensitivity image, and the infrared image with the colors of the surrounding pixels. Therefore, each of the image data of the high-sensitivity image, the medium-sensitivity image, and the low-sensitivity image that have not yet been demosaiced is four types of pixel signals R, Gr, Gb, and B; however, each image data that has been demosaiced becomes three types of pixel signals R, G, and B. In addition, the image data of the IR image that has not yet been demosaiced is four types of pixel signals IRr, IRgr, IRgb, and IRb; however, the image data that has been demosaiced becomes one type of image signal IR.
[0059] return Figure 3 , the recognition unit 14 includes: a microcomputer having a CPU, a ROM, a RAM, etc.; and various circuits. The recognition unit 14 includes: an object recognition unit 31, which operates in such a manner that the CPU executes an object recognition program stored in the ROM by using the RAM as a work area; and an object recognition data storage unit 32, which is set in the RAM or the ROM. In the object recognition data storage unit 32, a DNN is stored for each type of object used as a recognition target.
[0060] The object recognition unit 31 reads the DNN corresponding to the type of recognition target to be set from the object recognition data storage unit 32, outputs the recognition result of the subject obtained by inputting the image data into the DNN and outputting it from the DNN to the data sending determination unit 15, and outputs the metadata of the recognition result to the SEL 16.
[0061] In this article, reference Figure 5A and Figure 5B , a description will be given of the flow of processing performed by the recognition unit 14. Figure 5A and Figure 5B FIG. 2 is an explanatory diagram of a process performed by a recognition unit according to the present disclosure. Figure 5A As shown, according to the size and input value of the DNN, the recognition unit 14 first normalizes the size and input value of the image data to be input, inputs the normalized image data to the DNN, and performs object recognition. Then, the recognition unit 14 outputs the recognition result of the subject output from the DNN to the data transmission determination unit 15, and outputs the metadata of the recognition result to the SEL 16.
[0062] At this time, if Figure 5B As shown, the recognition unit 14 inputs the long exposure signals R, G and B, the intermediate exposure signals R, G and B, the short exposure signals R, G and B, and the infrared light signal IR to the DNN. Therefore, the recognition unit 14 recognizes the subject from each of the high sensitivity image, the medium sensitivity image, the low sensitivity image, and the infrared image without artifacts, so that the recognition accuracy of the subject can be improved.
[0063] return Figure 3 According to the recognition result input from the recognition unit 14, the data transmission determination unit 15 outputs a control signal for switching data to the SEL 16 so that the SEL 16 outputs data therefrom. When the recognition unit 14 recognizes the subject, the data transmission determination unit 15 outputs a control signal to the SEL 16 so that the SEL 16 outputs the image data and metadata indicating the recognition result to the transmission unit 17.
[0064] Meanwhile, when the recognition unit 14 does not recognize the subject, the data transmission determination unit 15 outputs a control signal to the SEL 16 for causing the SEL 16 to output information to that effect (no data) to the transmission unit 17. According to the control signal input from the data transmission determination unit 15, the SEL 16 outputs a set of image data and metadata or no data to the transmission unit 17.
[0065] The transmission unit 17 is a communication interface (I / F) that performs data communication with the AP 2 , and transmits a set of image data and metadata input from the SEL 16 or no data to the AP 2 .
[0066] As described above, the image sensor 1 transmits image data to the AP 2 only when the subject is recognized, and does not transmit image data to the AP 2 when the subject is not recognized, so that power consumption required for transmitting image data can be reduced.
[0067] The AP 2 includes a microcomputer having a CPU, a ROM, a RAM, etc., which executes various application programs corresponding to the use of the image recognition system 100 , and various circuits. The AP 2 includes a receiving unit 21 , an authentication unit 22 , and an authentication data storage unit 23 .
[0068] In the authentication data storage unit 23, an authentication program, authentication image data, etc. for authenticating a subject recognized by the image sensor 1 are stored. The receiving unit 21 is a communication I / F that performs data communication with the image sensor 1. The receiving unit 21 receives a set of image data and metadata or no data from the image sensor 1, and outputs the received one to the authentication unit 22.
[0069] The authentication unit 22 is inactive when no data is received from the receiving unit 21, and is activated when a set of image data and metadata is received. Once activated, the authentication unit 22 reads an authentication program from the authentication data storage unit 23 and executes the program, and authenticates the subject recognized by the image sensor 1.
[0070] For example, upon receiving a set of metadata indicating that a subject is a person and image data, the authentication unit 22 collates the image data and human authentication image data with each other and performs processing for identifying who is the person thus identified or the like.
[0071] At this time, the authentication unit 22 recognizes a person based on the image data of the high-sensitivity image, the medium-sensitivity image, the low-sensitivity image, and the infrared image without artifacts, and wherein the subject is recognized as a person with high accuracy by the image sensor 1. Therefore, the authentication unit 22 can accurately recognize who the recognized person is. Note that the above-described embodiment is merely an example and can be modified in various ways. Next, a modified example of an embodiment according to the present disclosure will be described.
[0072] [3. Modification of Processing Performed by Signal Processing Unit]
[0073] Figure 6 is an explanatory diagram illustrating a modified example of processing performed by the signal processing unit according to the present disclosure. Figure 7 is an explanatory diagram showing a modification of the example using DNN according to the present disclosure.
[0074] like Figure 6As shown, the signal processing unit according to the modification performs shading correction, mixed color correction, digital gain adjustment, white balance gain adjustment, and gamma correction on input image data, and outputs the image data that has been signal processed to the recognition unit 14 and the SEL 16.
[0075] As mentioned above, the signal processing unit can Figure 4 De-mosaicing is omitted in the signal processing shown. In this case, the recognition unit 14 recognizes the subject from the image data that has not been de-mosaiced. As described above, each of the image data of the high-sensitivity image, the medium-sensitivity image, and the low-sensitivity image that have not been de-mosaiced becomes four types of pixel signals R, Gr, Gb, and B. In addition, the image data of the infrared image that has not been de-mosaiced becomes pixel signals IRr, IRgr, IRgb, and IRb.
[0076] Therefore, the recognition unit 14 inputs the long exposure signals R, Gr, Gb and B, the intermediate exposure signals R, Gr, Gb and B, the short exposure signals R, Gr, Gb and B, and the infrared light signals IRr, IRgr, IRgb and IR (see FIG. Figure 2B ). In this case, in the recognition unit 14, since the number of input channels to the DNN increases, its throughput increases. However, the image data of the high-sensitivity image, the medium-sensitivity image, the low-sensitivity image, and the infrared image are input to the DNN separately, and therefore, the recognition unit 14 can recognize the subject with high accuracy without being affected by artifacts.
[0077] Note that in other modified examples, the signal processing unit itself may be omitted. Similarly, in this case, the recognition unit 14 inputs the long exposure signals R, Gr, Gb, and B, the intermediate exposure signals R, Gr, Gb, and B, the short exposure signals R, Gr, Gb, and B, and the infrared light signals IRr, IRgr, IRgb, and IR to the DNN. Therefore, although the throughput of the recognition unit 14 is increased, the throughput of the entire image sensor 1 is greatly reduced by the amount of signal processing that is not performed.
[0078] Note that in the above embodiment, when a subject is recognized, the data transmission determination unit 15 causes the SEL 16 to output metadata and image data of the recognition result every time; however, the data caused to be output by the SEL 16 may be selected according to the remaining capacity of the battery.
[0079] For example, when the subject is recognized by the recognition unit 14, in the normal mode in which the remaining capacity of the battery is equal to or greater than the predetermined remaining capacity, the data transmission determination unit 15 causes the SEL 16 to output metadata and image data. In addition, in the low power consumption mode in which the remaining capacity of the battery is less than the predetermined remaining capacity, the data transmission determination unit 15 causes the SEL 16 to output only metadata. Therefore, when the remaining capacity of the battery is insufficient, the data transmission determination unit 15 can suppress the power consumption of the image sensor 1 to a low level.
[0080] [4. Modification Example of Recognition of Subject by Recognition Unit]
[0081] Figure 7 : is an explanatory diagram showing a modified example of a DNN to be used by the recognition unit according to the present disclosure. Figure 2B and Figure 5B In the example shown, the recognition unit 14 recognizes the subject by using one DNN; however, two DNNs may also be used, such as Figure 7 shown.
[0082] Specifically, the long exposure image, the intermediate exposure image, and the short exposure image have a hue; however, the infrared image has no hue. Therefore, the features of the subject therein are different between the long exposure image, the intermediate exposure image, the short exposure image, and the infrared image.
[0083] Therefore, the recognition unit according to the modified example recognizes the subject by using the first DNN and the second DNN. The first DNN is a learning model that is machine-learned by specifically recognizing the subject from the long-exposure image, the medium-exposure image, and the short-exposure image. The second DNN is a learning model that is machine-learned by specifically recognizing the subject from the infrared image.
[0084] The recognition unit according to the modified example inputs the long exposure signals R, G and B, the intermediate exposure signals R, G and B, and the short exposure signals R, G and B to the first DNN, and recognizes the subject from the long exposure image, the intermediate exposure image, and the short exposure image. In addition, the recognition unit inputs the infrared light signal to the second DNN, and recognizes the subject from the infrared image. Therefore, the recognition unit can further improve the recognition accuracy of the subject.
[0085] [5. Modification Example of Pixel Array]
[0086] Figure 8 is an explanatory diagram showing a first modified example of a pixel array according to the present disclosure. Fig. 9 is an explanatory diagram showing a second modification example of the pixel array according to the present disclosure. Fig.10is an explanatory diagram showing an example of using a DNN when the second modification example of the pixel array according to the present disclosure is adopted. Fig.11 is an explanatory diagram showing a third modification example of the pixel array according to the present disclosure.
[0087] like Figure 8 As shown, the pixel array A2 according to the first modification includes a time-of-flight (ToF) sensor instead of Figure 1 The imaging element IR shown. In the case of adopting the pixel array A2, the image sensor 1 includes a light emitting unit that applies infrared light to a subject.
[0088] Each ToF sensor is a sensor that measures the distance to a subject based on the time from applying infrared light until receiving infrared light reflected on the subject. By adopting such a pixel array A2, the image sensor 1 can measure the distance to a subject and recognize the subject.
[0089] In addition, if Fig. 9 As shown, the pixel array A3 according to the second modified example includes: imaging pixels R that receive red light; imaging pixels G that receive green light; imaging pixels B that receive blue light; and imaging pixels IR that receive infrared light.
[0090] The imaging pixel R includes a long-time exposure imaging element RL and a short-time exposure imaging element RS. The imaging pixel G includes a long-time exposure imaging element GL and a short-time exposure imaging element GS. The imaging pixel B includes a long-time exposure imaging element BL and a short-time exposure imaging element BS.
[0091] As described above, in the pixel array A3, the imaging pixels R, G, and B include imaging elements RL and RS, GL and GS, and BL and BS in a one-to-two relationship, respectively. Generally, such a pixel array A3 is used for autofocusing, which evenly sets the exposure times of all imaging elements RL and RS, GL and GS, and BL and BS in the imaging pixels R, G, and B, and automatically adjusts the focus based on the phase difference between each light received by the right imaging element and the left imaging element.
[0092] In the present disclosure, the exposure times of the imaging elements RL and RS, GL and GS, and BL and BS in the imaging pixels R, G and B of such a pixel array A3 are individually controlled, thereby making the imaging elements RL, GL and BL serve as long-exposure pixels and making the imaging elements RS, GS and BS serve as short-exposure pixels.
[0093] Therefore, the pixel array A3 can image the long exposure image and the short exposure image. In addition, the pixel array A3 includes an imaging element IR, and therefore, can also image the IR image simultaneously with the long exposure image and the short exposure image.
[0094] When the pixel array A3 is used, Fig.10 As shown, the recognition unit 14 inputs the long exposure signals output from the imaging pixels RL, GL and BL, the short exposure signals output from the imaging pixels RS, GS and BS, and the infrared light signal output from the imaging element IR to the DNN. Therefore, the recognition unit 14 can separately recognize the subject from the long exposure image, the short exposure image and the infrared image with high accuracy.
[0095] In addition, similar to Fig. 9 The pixel array A3 shown in FIG. Fig.11 As shown, the pixel array A4 according to the fourth modification includes a pair of imaging elements R and R receiving red light, a pair of imaging elements G and G receiving green light, and imaging elements B and B receiving blue light. In addition, the pixel array A4 includes a pair of imaging elements IR and IR receiving infrared light. Each of the imaging elements R, G, B and IR includes two photoelectric conversion elements PD.
[0096] In such a pixel array A4, for example, among the four photoelectric conversion elements PD provided in the imaging element pair R and R, one is exposed for a long time, two are exposed for an intermediate time, and the remaining one is exposed for a short time. Furthermore, in the pixel array A4, among the four photoelectric conversion elements PD provided in a pair of imaging elements G and G, one is exposed for a long time, two are exposed for an intermediate time, and the remaining one is exposed for a short time.
[0097] In addition, in the pixel array A4, for example, among the four photoelectric conversion elements PD provided in the imaging element pair B and B, one is exposed for a long time, two are exposed for an intermediate time, and the remaining one is exposed for a short time. Therefore, the pixel array A4 can image a long-time exposure image, an intermediate-time exposure image, and a short-time exposure image.
[0098] In addition, the pixel array A4 exposes the imaging elements IR and IR, thereby being able to image an IR image. When such a pixel array A4 is adopted, the recognition unit 14 inputs the image data of the long exposure image, the intermediate exposure image, the short exposure image, and the infrared image into the DNN, thereby being able to recognize the subject with high accuracy without being affected by artifacts.
[0099] [6. Application examples of mobile subjects]
[0100] The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device installed on any type of mobile body, such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, and a robot.
[0101] Fig.12 is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to the embodiment of the present disclosure can be applied.
[0102] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig.12 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as the functional configuration of the integrated control unit 12050.
[0103] The drive system control unit 12010 controls the operation of the devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 is used as a control device to control: a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine, a drive motor, etc., a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating the braking force of the vehicle, etc.
[0104] The body system control unit 12020 controls the operation of various types of equipment configured for the body according to various programs. For example, the body system control unit 12020 is used as a control device to control the following items: a keyless entry system, a smart key system, a power window device, or various lights such as headlights, reverse lights, brake lights, turn signals, fog lights, etc. In this case, the body system control unit 12020 can receive as input radio waves sent from a mobile device that replaces the key or signals from various switches. The body system control unit 12020 receives these input radio waves or signals to control the door lock device, power window device, lights, etc. of the vehicle.
[0105] The vehicle exterior information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to an imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to image an image of the vehicle exterior, and receives the imaged image. Based on the received image, the vehicle exterior information detection unit 12030 may perform a process of detecting an object (such as a person, vehicle, obstacle, sign, symbol, etc. on the road), or perform a process of detecting the distance to the object.
[0106] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.
[0107] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. The in-vehicle information detection unit 12040 may be connected to a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that photographs the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 may calculate the driver's fatigue level or the driver's concentration level, or may distinguish whether the driver is dozing off.
[0108] The microcomputer 12051 can calculate a control target value for a driving force generation device, a steering mechanism, or a braking device based on information about the interior or exterior of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS), including collision avoidance or impact buffering for the vehicle, following driving based on vehicle-to-vehicle distance, vehicle speed maintenance driving, an alarm for vehicle collision, an alarm for vehicle lane deviation, and the like.
[0109] In addition, the microcomputer 12051 can control the driving force generating device, the steering mechanism, and the braking device based on the information about the outside or inside of the vehicle obtained by the outside information detection unit 12030 or the inside information detection unit 12040, thereby performing collaborative control intended for automatic driving, etc. that is independent of the driver's operation.
[0110] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information about the exterior of the vehicle obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 can control the headlights to change from high beam to low beam based on the position of the preceding vehicle or oncoming vehicle detected by the exterior information detection unit 12030, thereby performing cooperative control aimed at preventing glare by controlling the headlights.
[0111] The sound / image output unit 12052 sends an output signal of at least one of sound and image to an output device, which can notify information to the passengers of the vehicle or the outside of the vehicle in a visual or auditory manner. Fig.12In the example of FIG. 1 , an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are shown as output devices. The display portion 12062 may include, for example, at least one of an in-vehicle display and a head-up display.
[0112] Fig.13 12031 is a diagram showing an example of the installation position of the imaging unit 12031.
[0113] exist Fig.13 In the figure, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104 and 12105.
[0114] The imaging units 12101, 12102, 12103, 12104 and 12105 may be disposed at the positions of the front nose, side mirrors, rear bumper, rear door and the upper portion of the windshield inside the vehicle 12100. The imaging unit 12101 disposed at the front nose and the imaging unit 12105 disposed at the upper portion of the windshield inside the vehicle mainly obtain images in front of the vehicle 12100. The imaging units 12102 and 12103 disposed at the side mirrors mainly obtain images of the sides of the vehicle 12100. The imaging unit 12104 disposed at the rear bumper or rear door mainly obtains images of the rear of the vehicle 12100. The imaging unit 12105 disposed at the upper portion of the windshield inside the vehicle is mainly used to detect vehicles ahead, pedestrians, obstacles, signals, traffic signs, lanes, etc.
[0115] By the way, Fig.13 An example of the shooting range of the imaging units 12101 to 12104 is shown. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided at the front nose. Imaging ranges 12112 and 12113 respectively indicate the imaging ranges of the imaging units 12102 and 12103 provided at the side mirrors. Imaging range 12114 indicates the imaging range of the imaging unit 12104 provided at the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 12100 observed from above can be obtained by superimposing image data imaged by the imaging units 12101 to 12104.
[0116] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.
[0117] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and thereby extract the nearest three-dimensional object as the leading vehicle, which specifically exists on the driving path of the vehicle 12100 and travels at a predetermined speed (e.g., equal to or greater than 0 km / h) in the substantially same direction as the vehicle 12100. In addition, the microcomputer 12051 can pre-set the following distance to be maintained from the leading vehicle, and perform automatic braking control (including following parking control), automatic acceleration control (including following starting control), etc. Therefore, it is possible to perform cooperative control intended for automatic driving, etc. that does not depend on the operation of the driver.
[0118] For example, the microcomputer 12051 can classify the three-dimensional object data about the three-dimensional object into three-dimensional object data of two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104, and extract the classified three-dimensional object data for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes whether the obstacles around the vehicle 12100 are obstacles that can be visually identified by the driver of the vehicle 12100, or obstacles that are difficult for the driver of the vehicle 12100 to visually identify. Then, the microcomputer 12051 determines the collision risk, which indicates the risk of collision with each obstacle. In the case where the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 outputs an alarm to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or avoidance steering via the drive system control unit 12010. Thus, the microcomputer 12051 can assist driving to avoid collisions.
[0119] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 is capable of identifying pedestrians by determining whether there are pedestrians in the imaging images of the imaging units 12101 to 12104. This pedestrian recognition is performed, for example, by the following programs: a program for extracting characteristic points in the imaging images of the imaging units 12101 to 12104 as infrared cameras, and a program for determining whether it is a pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there are pedestrians in the imaging images of the imaging units 12101 to 12104 and thus recognizes the pedestrian, the sound / image output unit 12052 controls the display unit 12062 to display a square outline superimposed on the recognized pedestrian for emphasizing the recognized pedestrian. The sound / image output unit 12052 can also control the display unit 12062 to display an icon representing a pedestrian at a desired position, etc.
[0120] An example of a vehicle control system to which the technology of the present disclosure is applicable has been described above. The technology of the present disclosure is applicable to, for example, the vehicle exterior information detection unit 12030, the imaging unit 12031, the vehicle interior information detection unit 12040, the driver state detection unit 12041, etc., among the above components. For example, Figure 1 The image recognition device 1 can be applied to the in-vehicle information detection unit 12040 and the driver state detection unit 12041. The technology disclosed in the present invention is applied to the in-vehicle information detection unit 12040 and the driver state detection unit 12041, thereby improving the recognition accuracy of the driver as the subject. Therefore, the driver's fatigue and concentration, whether the driver is dozing off, etc. can be detected more accurately.
[0121] [7. Effect]
[0122] The image sensor 1 as an example of an image recognition device includes an imaging unit 10 and a recognition unit 14. The imaging unit 10 uses imaging pixels R, Gr, Gb, and B that receive visible light and imaging pixels IR that receive infrared light, and images a plurality of images at the same exposure timing in one frame period to generate image data. The recognition unit recognizes a subject from each piece of image data. Therefore, the image sensor 1 eliminates the influence of artifacts, thereby improving the recognition accuracy of the subject.
[0123] In addition, the imaging unit 10 has a pixel array A1 in which a plurality of imaging pixels R, Gr, Gb, and B having the same light receiving area and different exposure times and an imaging pixel IR receiving infrared light are two-dimensionally arranged. Therefore, the image sensor 1 recognizes a subject from each of the image data having different exposure times and infrared images, thereby being able to recognize the subject with high accuracy without being affected by artifacts.
[0124] In addition, the pixel array A1 includes an imaging element L which is an example of a long-exposure imaging pixel that receives visible light, an imaging element M which is an example of an intermediate-time exposure imaging pixel that receives the imaging element, and an imaging element S which is an example of a short-time exposure imaging pixel. Therefore, the image sensor 1 recognizes a subject from each of the long-exposure image, the intermediate-time exposure image, and the short-time exposure image, and thereby can recognize the subject with high accuracy without being affected by artifacts.
[0125] In addition, in the pixel array A3, each pixel is composed of imaging elements RL and RS, GL and GS, or BL and BS, and each pair of imaging elements has two imaging elements with different exposure times. Therefore, the image sensor 1 can image a long exposure image and a short exposure image by using phase difference pixels for autofocus.
[0126] In addition, in the pixel array A4, each pixel is composed of imaging elements R and R, G and G, or B and B, and each pair of imaging elements has two imaging elements. Each of the imaging elements R, G, and B includes two photoelectric conversion elements PD with different exposure times. Therefore, the image sensor 1 controls the exposure time of each photoelectric conversion element PD, so that a long exposure image, an intermediate exposure image, and a short exposure image can be imaged.
[0127] Furthermore, in the case where the subject is recognized, the recognition unit 14 outputs the recognition result and image data of the subject to the AP 2 as an example of a subsequent device, and in the case where the subject is not recognized, outputs information to that effect to the AP 2. Therefore, the image sensor 1 does not output image data in the case where the subject is not recognized, and thus power consumption can be reduced.
[0128] Furthermore, when the remaining capacity of the battery is equal to or greater than a predetermined remaining capacity, the recognition unit 14 outputs the recognition result of the subject and the image data to the AP 2, and when the remaining capacity of the battery is less than the predetermined remaining capacity, the recognition unit 14 outputs the recognition result of the subject to the AP 2 without outputting the image data to the AP 2. Therefore, when the remaining capacity of the battery is less than the predetermined remaining capacity, the image sensor 1 does not output the image data even in the case of recognizing the subject, and thus, power consumption can be reduced.
[0129] Furthermore, the recognition unit 14 recognizes the subject from the demosaiced image data. Therefore, when the subject is recognized by using the DNN, the image sensor 1 reduces the number of input channels of the DNN, thereby reducing the processing load.
[0130] In addition, the recognition unit 14 recognizes the subject from the image data that has not been de-mosaiced. Therefore, although the throughput is increased when the subject is recognized by using the DNN, the image sensor 1 can recognize the subject with high accuracy from the image data of the high-sensitivity image, the medium-sensitivity image, the low-sensitivity image, and the infrared image.
[0131] Furthermore, the recognition unit 14 recognizes the subject from the image data input from the imaging unit 10. Therefore, although the throughput increases in the case of recognizing the subject by using the DNN, the image sensor 1 can greatly reduce the throughput of the entire image sensor 1 by the amount of signal processing not performed.
[0132] In addition, the imaging pixel that receives infrared light is a time-of-flight (ToF) sensor. Therefore, the image sensor 1 can measure the distance to the subject and recognize the subject.
[0133] In addition, the image recognition method uses imaging pixels that receive visible light and imaging pixels that receive infrared light to image multiple images at the same exposure timing in one frame period to generate image data, and recognizes the subject from each piece of image data. According to this image recognition method, the influence of artifacts is eliminated, thereby improving the recognition accuracy of the subject.
[0134] Note that the effects described in this specification are merely examples and not limitations, and there may be other effects.
[0135] Note that the present technology can also adopt the following configurations.
[0136] (1) An image recognition device comprising:
[0137] an imaging unit that uses imaging pixels that receive visible light and imaging pixels that receive infrared light and that images a plurality of images at the same exposure timing in one frame period to generate image data; and
[0138] The recognition unit recognizes the subject from each piece of image data.
[0139] (2) The image recognition device according to (1), wherein:
[0140] The imaging unit includes:
[0141] A pixel array in which a plurality of imaging pixels having the same light receiving area and different exposure times and imaging pixels receiving infrared light are two-dimensionally arranged.
[0142] (3) The image recognition device according to (2), wherein:
[0143] The pixel array includes:
[0144] Long-time exposure imaging pixels receiving visible light, intermediate-time exposure imaging pixels receiving visible light, and short-time exposure imaging pixels receiving visible light.
[0145] (4) The image recognition device according to (2), wherein:
[0146] In the pixel array,
[0147] Each pixel consists of two imaging elements with different exposure times.
[0148] (5) The image recognition device according to (2), wherein:
[0149] In the pixel array,
[0150] Each pixel consists of two imaging elements, and
[0151] Each imaging element includes
[0152] Two photoelectric conversion elements with different exposure times.
[0153] (6) The image recognition device according to any one of (1) to (5), wherein:
[0154] The recognition unit outputs the recognition result and image data of the subject to a subsequent device if the subject is recognized, and outputs information to that effect to the subsequent device if the subject is not recognized.
[0155] (7) The image recognition device according to (6), wherein:
[0156] When the remaining capacity of the battery is equal to or greater than a predetermined remaining capacity, the recognition unit outputs the recognition result of the subject and image data to a subsequent device, and when the remaining capacity of the battery is less than a predetermined remaining capacity, the recognition unit outputs the recognition result of the subject to the subsequent device without outputting the image data to the subsequent device.
[0157] (8) The image recognition device according to any one of (1) to (7), wherein:
[0158] Identification unit:
[0159] The subject is recognized from the demosaiced image data.
[0160] (9) The image recognition device according to any one of (1) to (8), wherein:
[0161] Identification unit:
[0162] The subject is recognized from the image data which has not been demosaiced.
[0163] (10) The image recognition device according to any one of (1) to (8), wherein:
[0164] Identification unit:
[0165] A subject is recognized from image data to be input from the imaging unit.
[0166] (11) The image recognition device according to any one of (1) to (10), wherein:
[0167] The imaging pixels that receive infrared light are:
[0168] Time of Flight (ToF) sensor.
[0169] (12) An image recognition method comprising:
[0170] Using imaging pixels that receive visible light and imaging pixels that receive infrared light, and imaging a plurality of images at the same exposure timing in one frame period to generate image data; and
[0171] A subject is identified from each piece of image data.
[0172] Reference numerals list
[0173] 100 Image Recognition System
[0174] 1 Image sensor
[0175] 10 Imaging Units
[0176] A1, A2, A3, A4 pixel array
[0177] 12A / D conversion unit
[0178] 13Signal Processing Unit
[0179] 14 Identification Unit
[0180] 15 Data transmission confirmation unit
[0181] 16SEL
[0182] 17 Sending Unit
[0183] 2AP
[0184] 21 receiving unit
[0185] 22 certification units
[0186] 23. Authentication data storage unit
[0187] 31 Object Recognition Unit
[0188] 32 object identification data storage units.
Claims
1. An image recognition device, comprising: an imaging unit that uses imaging pixels that receive visible light and imaging pixels that receive infrared light, and images a plurality of images at the same exposure timing in one frame period to generate image data, wherein the imaging pixels that receive visible light include a plurality of imaging elements having the same light receiving area and different exposure times, and the imaging pixels that receive infrared light include imaging elements that receive the infrared light, and wherein the imaging unit includes a pixel array in which the imaging pixels that receive visible light and the imaging pixels that receive infrared light are two-dimensionally arranged; and A recognition unit is used to recognize a subject from each piece of image data, wherein: The imaging pixel receiving visible light comprises: a long-exposure imaging element for receiving the visible light, a medium-exposure imaging element for receiving the visible light, and a short-exposure imaging element for receiving the visible light, In which, an imaging element for receiving the infrared light is arranged at each corresponding position of the pixel array, and the three sides of the corresponding position are surrounded by the long-exposure imaging element, the intermediate-time exposure imaging element and the short-time exposure imaging element which receive light of the same color and are arranged in an L shape.
2. The image recognition device according to claim 1, wherein: The recognition unit outputs a recognition result of the object and the image data to a subsequent device when the object is recognized, and outputs information indicating that the object is not recognized to a subsequent device when the object is not recognized.
3. The image recognition device according to claim 2, wherein: When the remaining capacity of the battery is equal to or greater than a predetermined remaining capacity, the recognition unit outputs the recognition result of the subject and the image data to the subsequent device, and when the remaining capacity of the battery is less than the predetermined remaining capacity, the recognition unit outputs the recognition result of the subject to the subsequent device without outputting the image data to the subsequent device.
4. The image recognition device according to claim 1, wherein: The identification unit: The object is identified from the demosaiced image data.
5. The image recognition device according to claim 1, wherein: The identification unit: The object is identified from the image data that has not been demosaiced.
6. The image recognition device according to claim 1, wherein: The imaging pixels that receive infrared light are: Time-of-Flight ToF sensor.
7. An image recognition method, comprising: Using imaging pixels that receive visible light and imaging pixels that receive infrared light, and imaging a plurality of images at the same exposure timing in one frame period to generate image data, wherein the imaging pixels that receive visible light include a plurality of imaging elements having the same light receiving area and different exposure times, and the imaging pixels that receive infrared light include imaging elements that receive the infrared light, and wherein the imaging pixels that receive visible light and the imaging pixels that receive infrared light are arranged two-dimensionally; and identifying a subject from each piece of image data, Wherein, the imaging pixel receiving visible light includes: a long-exposure imaging element for receiving the visible light, a medium-exposure imaging element for receiving the visible light, and a short-exposure imaging element for receiving the visible light, Among them, an imaging element that receives the infrared light is set at a position surrounded on three sides by long-exposure imaging elements, intermediate-time exposure imaging elements, and short-time exposure imaging elements that receive light of the same color and are arranged in an L shape.
