Image recognition system

Abstract

To improve accuracy of image recognition by optimizing a condition of an imaging apparatus when acquiring an image.SOLUTION: An image recognition system 10 comprises: an image acquisition unit 101 which acquires an image captured by an imaging apparatus 11; a recognition model generation unit 102 which has at least one environment recognition part 111, recognizes the image with the at least one environment recognition part 111 and generates a recognition model; and a state estimation unit 103 which estimates the state of the imaging apparatus 11 and the characteristics of the imaging apparatus 11 corresponding to the state on the basis of the image and the recognition model. The state estimation unit 103 further generates an imaging correction parameter for correcting the imaging condition of the imaging apparatus 11 on the basis of at least one of the estimated state and characteristics and the recognition model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to image recognition technology, and more particularly to an image recognition system mounted on a moving body that moves while recognizing its surroundings, represented by a driving assistance system, an autonomous driving system, an autonomous mobile robot, and the like.

Background Art

[0002] In recent years, image recognition technology mounted on moving bodies has been required to be robust against environmental variations and external disturbances. For example, depending on the location where the moving body travels, the recognition target may not be appropriately illuminated, resulting in a decrease in contrast, or the imaging may be performed with a changed hue depending on the type of light source such as sunlight, halogen lamp, or neon sign, etc. Environmental variations may occur.

[0003] In addition, in an indoor environment, there may be various external disturbances such as backlight from the setting sun or light sources such as a PC monitor that enters through a window, raindrop adhesion to the camera lens in an outdoor environment, and generation of dark noise during nighttime operation. On the other hand, there are limitations in the dynamic range of the camera imaging device itself and in the inability to select the installation location conveniently for image recognition. Thus, image recognition using an imaging device has a reduced recognition performance due to environmental variations and external disturbances. In other words, in image recognition technology, the recognition performance for an object decreases in a state different from the imaging state assumed at the time of construction.

[0004] Therefore, for example, in Patent Document 1, "a first acquisition means for acquiring the imaging conditions under which the image used for generating the learning model was taken, an input means for inputting a captured image to be processed from an imaging device, a second acquisition means for acquiring the imaging conditions under which the captured image to be processed was taken, and a conversion means for converting the captured image to be processed based on the imaging conditions acquired by the first and second acquisition means", so that the input image is converted to approach the imaging state assumed when the image recognition was constructed, and even when the imaging conditions do not match between the environment for pre-learning and the environment for actual recognition, a technique for improving the accuracy of image recognition is provided.

Prior Art Documents

Patent Document

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the above prior art, it is assumed that the imaging conditions of the images used for learning are constant, and the imaging conditions at the time of actual imaging can be made to match the imaging conditions at the time of learning by image conversion of the captured images. However, mobile bodies typified by driving support systems, autonomous driving systems, autonomous mobile robots, etc. are assumed to move to various places, and the learning images are often those collected under various imaging conditions, and there are cases where learning is not performed using images acquired in the same environment for a plurality of types of recognition targets. In addition, since it is premised on improving the accuracy of image recognition by converting the acquired images, when information is already missing or degenerated when the images are acquired by the imaging device, the accuracy of image recognition cannot be sufficiently improved.

[0007] Therefore, the present invention is made from the viewpoint that it is important to match the environment in which the performance of devices etc. used for image recognition appears, rather than matching the environment at the time of learning of image recognition as in the above prior art, and the object thereof is to provide an image recognition system capable of improving the accuracy of image recognition by optimizing the conditions of the imaging device when acquiring images.

Means for Solving the Problems

[0008] To solve the above problems, the image recognition system of the present invention includes a processor and a memory, and has an image acquisition unit that acquires an image captured by an imaging device, at least one environment recognition unit that recognizes the image with at least one environment recognition unit and generates a recognition model, and a state estimation unit that estimates the state of the imaging device and the characteristics of the imaging device corresponding to the state based on the image and the recognition model. The state estimation unit further generates an imaging correction parameter for correcting the imaging conditions of the imaging device based on at least one of the estimated state and characteristics and the recognition model.

Advantages of the Invention

[0009] According to the present invention, it is possible to optimize the conditions of the imaging device when acquiring an image and improve the image recognition accuracy. Further features related to the present invention will become apparent from the description of this specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0011] Hereinafter, an example of the image recognition system according to the present invention will be described with reference to the drawings, taking the case where it is applied to an in-vehicle camera as an example.

[0012] <Example 1> FIG. 1 is a block configuration diagram for explaining each functional block of the image recognition system 10 according to Example 1 of the present invention. As shown in FIG. 1, the image recognition system 10 is connected to the imaging device 11 via a wired or wireless network, and receives imaging data and imaging parameters obtained from the imaging device 11. Then, based on these data, an imaging correction parameter for correcting the imaging conditions of the imaging device is generated and transmitted to the imaging device 11 as feedback.

[0013] In the imaging device 11, the exposure control unit 116 controls the shutter opening time and the aperture opening degree, and allows the video to pass through the inside of the imaging device 11 only for a certain period of time via a light collecting element such as a lens. The imaging device 11 incorporates an imaging element 117 (also called a photoelectric element or an imager), and an external image signal passes through the lens and is received by the imaging element 117 and converted into electric charges. The electric charges generated at each part on the imaging element 117 are subjected to image signal processing such as gain adjustment, smoothing, edge enhancement, demosaicking, noise removal, white balance adjustment, HDR processing (High Dynamic Range), and low-bit conversion (conversion to 24 bit / pixel, etc. if necessary) based on image signal processing parameters in the image signal processing unit 118 (also called ISP: Image Signal Processor). The data subjected to image signal processing is transmitted to the image recognition system 10 as image data. At this time, the imaging device 11 may transmit observation parameters such as imaging parameters and exposure values at the time of image acquisition to the image recognition system together with the image data.

[0014] The image recognition system 10 includes an image acquisition unit 101, a recognition model generation unit 102, and a state estimation unit 103. The recognition model generation unit 102 includes a plurality of environment recognition units 111. The state estimation unit 103 includes an imaging device state estimation unit 112, an imaging device characteristic estimation unit 113, a parameter generation unit 114, and a parameter stabilization unit 115.

[0015] In addition, in FIG. 1, the imaging device 11 and the image recognition system 10 are separately described. However, they may be mounted on the same hardware. Conversely, any one or more of the image acquisition unit 101, the recognition model generation unit 102, and the state estimation unit 103 included in the image recognition system 10 may be divided and mounted on different hardware and operate in cooperation through communication or the like.

[0016] Specifically, the image recognition system 10 is an electronic control unit (ECU, Electronic Control Unit) equipped with hardware such as a computing device (processor) such as a CPU, a main memory device such as a semiconductor memory, an auxiliary storage device, and a communication device. The computing device executes a program loaded into the main memory device to realize each function such as the recognition model generation unit 102 and the state estimation unit 103. However, hereinafter, while appropriately omitting well-known techniques in the computer field, the details of each unit will be described.

[0017] The image recognition system 10 acquires image data and the like received from the imaging device 11 by the image acquisition unit 101, accumulates it in a memory (not shown), and outputs it to the recognition model generation unit 102 and the state estimation unit 103.

[0018] The recognition model generation unit 102 has one or more environment recognition units 111 inside. Taking the image data obtained from the image acquisition unit as input, it recognizes the image data based on a recognition model obtained in advance through learning or manually designed, and outputs the result. Here, the recognition model refers to, for example, when this system is applied to an in-vehicle camera mounted on an automatic driving device, the environment recognition unit 111 recognizes one or more recognition objects such as moving objects like other vehicles, pedestrians, motorcycles, bicycles, animals, installations installed in the environment such as traffic lights, signs, billboards, bus stops, road markings such as lane dividing lines, stop lines, road shoulders, safety zones, guiding signs, regulatory signs, structures such as walls, fences, utility poles, curbs, buildings, obstacles such as falling rocks, tires, and road surface conditions such as road surface unevenness, road gradient, dryness and humidity. It refers to a model that includes this information.

[0019] The recognition model generation unit 102 performs external world recognition for each environment recognition unit 111, and outputs recognition results such as the rectangular area, center point, size, recognition confidence level, type of recognition target, and attributes of the recognized target of the area obtained as a result of the recognition to the state estimation unit 103.

[0020] The state estimation unit 103 receives image data and the like from the image acquisition unit 101 and recognition results and the like from the recognition model generation unit 102, generates imaging correction parameters for correcting the imaging conditions of the imaging device 11 based on these data, and transmits them to the imaging device 11. Note that the correction of the imaging conditions in the present invention refers to correcting one or more imaging correction parameters among, for example, the shutter release time, aperture opening degree, and image signal processing parameters of the imaging device.

[0021] The state estimation unit 103 further includes an imaging device state estimation unit 112, an imaging device characteristic estimation unit 113, a parameter generation unit 114, and a parameter stabilization unit 115.

[0022] Based on the received image data and the like, the imaging device state estimation unit 112 estimates the state of the imaging device, including one or more of raindrop adhesion on the lens, shielding due to fog, wiper, etc., and exposure state. That is, the state of the imaging device can also be said to be a general term for events that occur in the imaging device due to the environment in which the imaging device is placed (external factors such as weather, temperature, humidity, and interference by surrounding objects).

[0023] The imaging device characteristic estimation unit 113 estimates the characteristics of the imaging device, including one or more of the sensitivity of the imaging element 117, the signal-to-noise ratio (SNR), the sensitivity and SNR of each pixel of the imaging element 117, lens characteristics such as the amount of blur and distortion of the imaging device 11, and color correction characteristics such as white balance characteristics and gain characteristics. That is, the characteristics of the imaging device can also be said to be a general term for events that occur in the imaging device due to the functions of internal elements that make up the imaging device, such as the light-receiving element and the processing circuit, as opposed to the state of the imaging device described above.

[0024] Based on the results output from the imaging device state estimation unit 112, the imaging device characteristic estimation unit 113, and the recognition model generation unit 102, the parameter generation unit 114 performs one or more corrections of the shutter release time, aperture opening degree, and image signal processing parameters, and calculates the influence on the recognition result of the recognition model generation unit 102.

[0025] Based on both or one of the time-series change of the imaging correction parameters calculated by the parameter generation unit 114 and the influence on the recognition result of the recognition model generation unit 102, the parameter stabilization unit 115 monitors whether the imaging correction parameters are in an unstable state such as hunting, and finally determines the imaging correction parameters to be output to the imaging device 11.

[0026] The general outline of the series of processes performed by the above-described recognition model generation unit 102, imaging device state estimation unit 112, imaging device characteristic estimation unit 113, and parameter generation unit 114 will be described with reference to FIG. 2.

[0027] Here, as shown in FIG. 2, an example will be described in which a vehicle is recognized in a certain area of the image acquired by the image acquisition unit by the vehicle recognition unit among the plurality of environment recognition units included in the recognition model generation unit 102.

[0028] From the vehicle recognition unit, together with the image obtained by the image acquisition unit 101, the information on vehicle recognition is transmitted to the imaging device state estimation unit 112. Assume that "blur" is detected in the area where the vehicle is recognized there.

[0029] Then, the information is transmitted to the imaging device characteristic estimation unit 113. The imaging device characteristic estimation unit 113 estimates the amount of blur from the information on the aperture opening degree of the imaging device that caused the blur.

[0030] When the information regarding the amount of blur is transmitted to the parameter generation unit 114, the parameter generation unit 114 generates an edge enhancement degree correction parameter from among the image signal processing correction parameters so as to correct the amount of blur to a value at which the vehicle recognition performance of the imaging device is normally exhibited, and feeds it back to the imaging device 11 via a parameter stabilization unit 115 (not shown).

[0031] Through the series of processes as described above, the imaging device 11 can monitor and correct these in real time even when the imaging environment changes due to the running of the vehicle, changes in the weather, etc., and the state and characteristics of the imaging device change, and can always perform imaging under optimal conditions.

[0032] Note that the example shown in FIG. 2 is an example of the process. The same process is performed even when the environment recognition unit recognizes other objects, and there may be a plurality of characteristics corresponding to each state and parameters for correcting each characteristic. In addition, each state, characteristic, and parameter will be described in detail later.

[0033] Hereinafter, each part of the image recognition system according to the present invention will be described in more detail.

[0034] <Image acquisition unit 101> The image acquisition unit 101 receives the image data and imaging parameters sent from the imaging device 11 via a communication method such as USB (Universal Serial Bus), LVDS (Low Voltage Differential Signaling), I2C (Inter-Integrated Circuit), or Ethernet. Assume that the image data is obtained as a color image such as RGB888 or a grayscale luminance image. Assume that the imaging parameters include the shutter release time and aperture opening degree at the time of imaging.

[0035] The image acquisition unit 101 receives the image data and imaging parameters and records them in a memory (not shown) or the like. At this time, it may be stored in association with the time data unique to the image recognition system 10 or the time data of the entire system. By recording the image data and imaging parameters in association with the time data, it is possible to detect whether data is being periodically sent from the imaging device or whether a delay has occurred in the communication path or transmission path. Also, since it is possible to know which imaging data the calculation result of the imaging correction parameters described later is associated with, a more stable imaging correction parameter output can be achieved in the parameter stabilization unit 115 described later.

[0036] <Recognition model generation unit 102> The recognition model generation unit 102 recognizes a predetermined recognition target based on the image data acquired from the image acquisition unit 101 and outputs the recognition result to the state estimation unit 103. Since there are many known techniques for this image recognition technology, the details will be omitted, but some examples in the case of an automatic driving device equipped with an in-vehicle camera will be described.

[0037] When the recognition target is a moving object or a fixed object, a technique that can obtain recognition results such as the rectangular region, center point, size, recognition confidence, type of recognition target, and attributes of the recognized target of the recognition target included in the image using a convolutional neural network (CNN), which is a type of deep learning, has been publicly disclosed, and such a technique may be used.

[0038] There are also several known techniques for the case where the object to be recognized is a road surface marking. For example, by using the technique described in Patent Document 2, recognition results such as the area, center point, size, and type of the object to be recognized can be obtained. Furthermore, since the confidence level of recognition can also be obtained based on the contrast and noise amount of the image of the road surface marking portion, such techniques may be utilized.

[0039] Similarly, for the case where the object to be recognized is a structure or an obstacle, there are several known techniques. For example, when using a stereo camera, based on the parallax between the left and right cameras, the distance from the camera to the point corresponding to each point in the image can be calculated. Furthermore, it is known that the road plane can be calculated by performing a plane approximation of the point group on the lower side of the screen of the obtained distance information, and structures and obstacles can be calculated by extracting points that are a certain distance or more away from the road plane as three-dimensional objects.

[0040] For the case where the object to be recognized is the road surface condition, a technique for estimating based on the texture in the image and the continuity with the surroundings using a convolutional neural network (CNN), which is a type of deep learning, has been published, and such techniques may be utilized.

[0041] The above-mentioned individual recognition processes may be implemented by being shared among the plurality of environment recognition units 111 for each object to be recognized. In that case, the image data and imaging parameters obtained from the image acquisition unit 101 are transmitted in parallel to the environment recognition unit 111 and are assumed to be input.

[0042] Next, the detailed functions of the imaging device state estimator 112, imaging device characteristic estimator 113, parameter generator 114, and parameter stabilizer 115 included in the state estimator 103 described above will be described.

[0043] <Imaging Device State Estimator 112> Based on the image data and the like received from the image acquisition unit 101, the imaging device state estimation unit 112 detects factors that change the imaging conditions of the image sensor, such as raindrops or mud adhering to the windshield or the lens itself, fog or blur occurring between the imaging element and the subject, detects the intermittent shielding by the wiper blade during wiper operation, detects fluctuations in the exposure state that change due to the external weather, time zone, or artificial lighting, and obtains the adjustment value by the automatic exposure adjustment mechanism (AE: automatic exposure) or the shutter speed value by the automatic shutter speed adjustment mechanism separately provided in the imaging device.

[0044] Hereinafter, several examples of the detection methods will be given. The detection of raindrop or mud adhesion is performed as follows. In the moving image continuously acquired while driving, when observing the variation of the coordinates of the feature points that can be extracted by feature point extraction methods such as Harris, SIFT, and SURF over time, in the area where no raindrops are attached, the coordinates of the feature points change smoothly, while in the area where raindrops are approaching, the amount of change in the coordinates of the feature points changes. That is, in the area where raindrops are attached, the coordinates of the feature points change suddenly or discontinuously. Also, when mud is attached, the extraction of feature points fails in the area where the mud is approaching and tracking becomes impossible. By observing such changes in the amount of change in the coordinates of the feature points and the success or failure of feature point extraction, the adhesion of raindrops or mud can be detected. Furthermore, by accumulating the change points of the amount of change in the coordinates and the change points of the success or failure of feature point extraction, the adhesion area of raindrops or mud can be estimated.

[0045] Detection of fogging and blurring is performed as follows. When imaging while moving, a subject imaged in an image captured at a certain moment is imaged blurred by the apparent movement amount during the exposure time. The above phenomenon, also called motion blur, has a short exposure time, little influence on a sufficiently distant subject, and a small amount of blur. Also, the amount of blur corresponding to the spatial frequency characteristics of the lens expressed by MTF (Modulation Transfer Function) or the like is superimposed. Therefore, when analyzing the frequency components of the captured image and no frequency components higher than the threshold value considering the above blur amount are detected, it is determined that lens fogging due to factors other than motion blur and lens characteristics, that is, water vapor or dirt adhesion, has occurred. Since the lens fogging state does not change rapidly, instead of instantaneously judging from a single image, the frequency components in time series are accumulated, and it may be determined that lens fogging has occurred when no high frequency components are detected until a certain time or a certain driving distance has elapsed.

[0046] Detection of shielding by the wiper blade is performed as follows. Since the on / off state of the wiper switch and the wiper operation speed flow through an in-vehicle network such as CAN (Controller Area Network), the content is acquired to determine the presence or absence of shielding by the wiper. Alternatively, since it is possible to know at what angle and how it appears in the camera field of view when the wiper blade operates, for example, by collating with a shape registered in advance by a template matching method or the like, it is possible to detect where the wiper blade exists in the image.

[0047] The detection of the exposure state is performed as follows. For the pixels in the image, statistical quantities such as the average, variance, multimodality, and frequency components are calculated and compared with a pre-set threshold value to determine day and night, inside and outside the tunnel, or to determine whether the shutter speed of the camera is within an appropriate range. Also, the image may be divided into partial regions and the same determination as described above may be made. That is, when divided into three parts horizontally and two parts vertically, if the average luminance value of the upper left region is high, bimodal, and has few high-frequency components, it can be seen that there is a light source such as the sun in the upper left sky within the field of view and there is overexposure. Also, when divided in the same way, if the luminance variance of the lower region is large and the frequency component is high, it can be seen that the ground plane portion including the driving region is properly imaged. By observing these state changes, the detection of fluctuations in the exposure state can be achieved.

[0048] The acquisition of adjustment values and the like in the imaging device is realized by receiving values corresponding to the necessary identification numbers (IDs) among the values flowing through the in-vehicle network such as CAN and storing them in a buffer or the like.

[0049] <Imaging device characteristic estimation unit 113> There are variations in sensitivity in the imaging element due to non-uniformity of sensitivity (PRNU) to photons and non-uniformity of dark current (DCNU) under the set operating conditions, etc., and furthermore, the value of the dark current changes with temperature. Therefore, the imaging device characteristic estimation unit 113 measures the dark current or measures the current under a uniform light source in advance to measure the sensitivity based on the image data and the like received from the image acquisition unit 101 and stores the result. Also, since the dark current has temperature dependence as described above, the sensitivity measurement results for each temperature may be recorded during measurement.

[0050] Similarly, the imaging device characteristic estimation unit 113 measures and records in advance the signal-to-noise ratio (SNR) of the signal flowing through the imaging element. Further, geometric characteristics of the lens such as the amount of blur and distortion of the imaging device 11, and color correction characteristics such as output characteristics with respect to the white balance adjustment value and output characteristics with respect to the gain adjustment value may be recorded. These can be calculated and stored at the timing of camera calibration.

[0051] For example, the amount of blur and distortion are obtained by calculating how much the object or boundary spreads to surrounding pixels and how much it is distorted when an object or linear object boundary of a known shape and size is observed within the camera's field of view, and are expressed as an MTF curve or internal parameters, etc. Since the degree of the amount of blur and distortion changes depending on the aperture opening, the amount of blur and distortion for each aperture opening may be recorded.

[0052] Characteristics regarding white balance and gain can be measured and recorded in advance using a color chart or the like to determine how they are observed at each illuminance level. Note that these may be recorded for the entire imaging area, or may be recorded for appropriately divided areas such as four divisions or nine divisions.

[0053] Based on the sensitivity measurement results and the like recorded in advance obtained above, the dark noise level in the actual driving environment can be estimated. For example, since the dark current depends on the temperature, the ratio of the dark current to the output current when imaging in the driving environment can be calculated. Specifically, the correspondence relationship between the temperature and the dark current is recorded as a table in the imaging device characteristic estimation unit 113, and by referring to this table based on the measured temperature, it becomes possible to calculate the signal-to-noise ratio of the signal flowing through the imaging element. Also, it is possible to estimate the illuminance of the imaging environment from the average luminance value of the acquired image and estimate the parameters for color tone correction taking into account the characteristics of white balance. Similarly, it is also possible to estimate the amount of blur and distortion based on the aperture opening at the time of imaging received from the imaging device state estimation unit 112 in the actual driving environment.

[0054] <Parameter generation unit 114> Based on the output results of the imaging device state estimation unit 112, the imaging device characteristic estimation unit 113, and the recognition model generation unit 102, the parameter generation unit 114 calculates imaging correction parameters for performing one or more corrections among the shutter release time, aperture opening degree, and image signal processing, and calculates the influence on the recognition result of the recognition model generation unit 102.

[0055] Hereinafter, several examples of a method for calculating imaging correction parameters for correcting imaging conditions such as the shutter release time, aperture opening degree, and image signal processing will be given. Regarding the imaging correction parameters for correcting the shutter release time, parameters are calculated so as to correct the current values obtained from the photometric points set for the entire screen or a plurality of locations in the screen so that they are distributed within a certain range. That is, when the median value Vmid of the current values obtained with the setting of a certain shutter release time T1 falls between a previously set threshold value THRmin and threshold value THRmax, it is determined that correction is unnecessary and T1 is output as it is. When the obtained current value Vmid is smaller than THRmin, T1 + ΔT is output to increase the current value by increasing the shutter release time, and conversely, when the current value is larger than THRmax, T1 - ΔT is output to decrease the current value by shortening the shutter release time. Here, ΔT is a constant determined in advance, and THRmax and THRmin may be provided in multiple steps.

[0056] Note that the above describes the method using photometric points set for the entire screen or a plurality of locations in the screen. Instead, the shutter release time may be output so that the current values obtained from the image areas or photometric points included in the area where the recognition target object output by the recognition model generation unit 102 exists are distributed within a certain range.

[0057] Regarding the aperture opening degree, the aperture opening A is set as follows. When the output value of the shutter opening time T1 described above is equal to or less than a preset threshold value Tmax0 and the current value C is equal to or greater than a preset threshold value Cthr0, parameters are output such that the aperture opening A becomes a value smaller than the current opening degree. Also, when the output value of the shutter opening time T1 described above is equal to or greater than a preset threshold value Tmax1, or when the current value C is equal to or greater than a preset threshold value Cthr1, parameters are output such that the aperture opening A becomes a value larger than the current opening degree. Here, Tmax0 < Tmax1 and Cthr0 < Cthr1. Also, at this time, Tmax1 and Cthr1 are set to sufficiently large values. For example, in the case of daytime, cloudy weather, and outdoors, the threshold values are set such that the aperture opening is fully opened. As a result, normally the aperture opening is in the maximum state, and when a light source such as the sun is within the field of view, the aperture becomes smaller, making it possible to reduce subject blur due to movement during imaging.

[0058] The imaging correction parameters for correcting the image signal processing include, for example, one or more of a smoothing parameter, an edge enhancement parameter, a white balance correction parameter, an offset correction and gain correction parameter, and a gamma correction parameter. Below, the calculation methods of these parameters will be described.

[0059] The smoothing parameter sets the process and parameters for smoothing the image according to the dark noise level estimated by the imaging device characteristic estimation unit 113. For example, when the signal-to-noise ratio due to dark noise is sufficiently large (when it is equal to or greater than the threshold value Dthr1), it is determined that noise removal is not necessary and the smoothing process is not performed. When the signal-to-noise ratio is less than Dthr1 and equal to or greater than Dthr2, an averaging process in a 3×3 pixel range is performed. When the signal-to-noise ratio is less than Dthr2 and equal to or greater than Dthr3, a median value process in a 3×3 pixel range is performed. When the signal-to-noise ratio is less than Dthr3, a median value process in a 5×5 pixel range is performed, etc. The process and parameters for smoothing the image are switched so that the noise removal process is stronger as the signal-to-noise ratio becomes smaller. Regarding the above threshold values, Dthr1 > Dthr2 > Dthr3, and each value is separated by about several times.

[0060] The edge enhancement parameter sets the process and parameter for sharpening the image according to the amount of blur estimated by the imaging device characteristic estimation unit 113. For example, as a technique for sharpening an image (enhancing edges), unsharp masking processing is generally known. Unsharp masking processing is a method of sharpening the original image by blurring the original image according to a certain Gaussian distribution and subtracting the obtained image from the original image. At this time, the kernel size and Gaussian filter coefficient are switched as parameters, and the parameters are adjusted so that the greater the amount of blur, the stronger the sharpening.

[0061] The white balance correction parameter is calculated using the gray world algorithm. The gray world algorithm generally used for white balance correction is an algorithm based on the statistical fact that if all colors in the screen are averaged, it will be close to achromatic color. It sets the gain of white balance so that the average signal levels of each color of R (red), G (green), and B (blue) in the screen are equal. By photographing a pre-known color chart or the like in advance and recording the gain of white balance, the white balance correction parameter can be obtained from the comparison with the gain of white balance obtained in the actual driving environment. Also, since it is conceivable that the white balance correction parameter changes depending on the environmental illumination, the white balance correction parameters for a plurality of environmental illuminations, for example, when the intensity and type of the light source are changed, are recorded, and the white balance correction parameter may be switched depending on which environmental illumination the video obtained during driving is close to.

[0062] The offset correction and gain correction parameters are parameters for correcting the correspondence relationship between the amount of current generated according to the amount of photons received by the imaging device and the luminance value of each pixel of the image when converting the amount of current into the luminance value. Let the amount of current input to the pixel be V0, the output luminance value be I0, the offset value (offset correction parameter) be C0, and the gain value (gain correction parameter) be G0. Then, it can be expressed as I0 = G0 × V0 + C0. At this time, due to the manufacturing error of the imaging device, the amount of received photons and the generated amount of current do not have an exact proportional relationship. Therefore, a lookup table for correcting the error corresponding to the amount of current V is denoted as L(V), and it may be set as I0 = G0 × V0 + C0 + L(V0). These C0, G0, and L(V) can be obtained by photographing a pre-known chart or the like in advance and recording the output values.

[0063] The gamma correction parameter is a parameter for adjusting the assignment of the luminance value of the output pixel so that the object to be observed in each luminance region such as the low luminance region, the middle luminance region, and the high luminance region can be expressed with sufficient gradation. Generally, when outputting the imaging result by a camera as an image, it is necessary to express the gradation with a finite number of bits such as 8 bits, 10 bits, 12 bits, 24 bits, 32 bits, etc. per pixel. For example, when part or all of the recognition object output by the recognition model generation unit 102 exists in the low luminance region of the screen and the number of gradations of the luminance values included in that region is equal to or less than a certain threshold Gthr, the gamma correction parameter value is reduced to increase the number of gradations assigned to the low luminance region. Here, let the luminance value of the input image be I1, the luminance value of the output image be I2, and the value of the gamma correction parameter be γ. Then, it can be expressed as I2 / C = (I1 / C)^γ. Here, C is a constant for normalizing the range of the luminance value to 0 to 1, and ^ is an arithmetic symbol representing exponentiation.

[0064] As described above, the imaging device state estimation unit 112 estimates events occurring in the imaging device 11 due to external factors, and the imaging device characteristic estimation unit estimates, according to those events, the estimation of dark current by temperature measurement, the estimation of the signal-to-noise ratio of the estimated dark current, the estimation of the illuminance of the imaging environment using the average luminance value of the acquired image, the estimation of the color tone correction amount considering white balance, the estimation of the amount of blur and distortion using the aperture opening degree at the time of imaging, and so on. Then, based on the estimated values, the parameter generation unit 114 generates imaging correction parameters for correcting the shutter release time, aperture opening degree, and image signal processing, which are imaging conditions.

[0065] That is, even in a situation where the environment around the imaging device 11 continues to change, such as when the vehicle on which the imaging device 11 is mounted moves, the imaging conditions are corrected in real time by the above-described processing, and it is possible to always acquire an image under optimal imaging conditions.

[0066] Although imaging correction parameters for correcting the shutter release time, aperture opening degree, and image signal processing can be calculated by the above-described calculation method, since these parameters affect each other, it may be necessary to determine a single parameter to be applied in consideration of the range within which the recognition performance of each environment recognition unit 111 described below is stable.

[0067] Specifically, the state estimation unit 103 stores in advance, as the range within which the recognition performance of each environment recognition unit 111 is stable, the parameter range within which the recognition performance is stable as shown in FIG. 3 (each hatched portion in FIG. 2). Then, each parameter is determined so that the imaging conditions of the imaging device 11 fall within the region where the recognition performance is stable.

[0068] For example, assuming a vehicle recognition function as one of the functions of the environment recognition unit 111, as shown in FIG. 3, the contrast range in which the recognition performance of vehicle recognition operates well is a relatively wide range, but the allowable blurring range is relatively narrow. These ranges can be obtained depending on what contrast range and image blurring range are learned when constructing the vehicle recognition function, that is, when inputting learning images and teacher data into the vehicle recognition algorithm for machine learning.

[0069] From the perspective of vehicle recognition, the following processing is performed to optimize the imaging conditions of the imaging device 11. First, the environment recognition unit 111 performs vehicle recognition on the video acquired during driving, calculates the contrast and blurring within the image area recognized as the area where the vehicle exists, and calculates the positioning with respect to the contrast range and the blurring range. Then, imaging correction parameters for correcting the shutter opening time, aperture opening degree, and image signal processing are calculated so as to head toward the position where the recognition performance is considered to be most stable, that is, the center of the contrast range and the blurring range (the point indicated by A in the hatched area). Further, it is recorded whether the recognition result of the recognition model generation unit 102 is improved, that is, whether the position has actually approached point A as a result of the correction.

[0070] That is, for example, when the center of the contrast range is larger than the contrast within the recognized image area, the imaging correction parameters are calculated so as to increase the contrast more. Specifically, one or more of the following processes are performed: reducing THRmin and increasing ΔT to extend the shutter opening time, increasing the threshold value Tmax to widen the aperture opening degree, increasing the gain value G0 of the gain correction, and changing the gamma value of the gamma correction.

[0071] In addition, as an influence on the recognition result of the recognition model generation unit 102, it is determined whether the performance stable range for each environment recognition unit 111 changes. Specifically, here, it is determined whether the contrast range and the blurring range in which the recognition performance is stable approach or move away, and based on this, either the prospect of performance improvement or the prospect of performance degradation is output.

[0072] When performing adjustment on a plurality of environment recognition units 111 simultaneously, the logical product of each parameter range, that is, the overlapping area, may be processed as described above. In this case, the influence on the recognition result of the recognition model generation unit 102 calculated varies depending on the type of the environment recognition unit 111.

[0073] Regarding the performance stable range of the environment recognition unit 111, instead of obtaining the contrast range and the blur range from the input learning image and the teacher data, an evaluation image may be input to the vehicle recognition algorithm after learning is completed, and the contrast range and the blur range may be obtained from the recognition success or failure thereof.

[0074] In FIG. 3, as an example, a case where the regions where the recognition performances of vehicle recognition, white line recognition, and signal recognition are stable respectively have a logical product (there is an overlapping region) has been described. However, there may be a case where there is no overlapping region in the parameter range where the recognition performance is stable as shown in FIG. 4. In that case, imaging may be performed with different parameters depending on the image acquisition timing. Specifically, the imaging timing may be divided into odd-numbered times and even-numbered times, and parameter determination may be performed from the region of parameter A in the odd-numbered imaging and from the region of parameter B in the even-numbered imaging. Alternatively, the region of the parameter used may be switched in accordance with the execution cycle of the environment recognition unit 111.

[0075] <Parameter stabilization unit 115> When calculating each imaging correction parameter by the above-described calculation method and performing feedback on the imaging device 11 to execute imaging, hunting may occur. That is, for example, if an image captured at a certain point in time is darker than an image suitable for the environment recognition unit 111, the shutter opening time is extended to capture an image in order to obtain a brighter image. However, if the resulting image is too bright compared to an image suitable for the environment recognition unit 111 this time, the shutter opening time is shortened to capture an image in order to obtain a darker image again, and this may be repeatedly performed periodically in a short time. That is, under certain specific conditions, the calculated parameters may vibrate, and there is a risk that the recognition performance may become unstable. Therefore, the parameter stabilization unit 115 accumulates each imaging correction parameter calculated by the parameter generation unit 114 along the time series, monitors its variation period, and when detecting vibrations over a certain period, reduces the amount of change in the output parameter.

[0076] Specifically, for example, assuming that the shutter opening time at time t1 is Tt1, the shutter opening time at time t2 is Tt2,..., and the shutter opening time at time tn is Ttn, the variation period of this shutter opening time is obtained by Fourier transform or the like within the most recent fixed time. When a high frequency equal to or higher than the threshold value is detected, the output shutter opening time can be made the moving average of the parameters calculated by the parameter generation unit 114 in the time series, for example, {Tt(n - 1)+Ttn} / 2.

[0077] <Processing Contents of Image Recognition System 10> Next, the processing contents of each part according to this embodiment will be described using a flowchart. FIG. 5 is a flowchart showing the processing contents of the image recognition system 10 of the first embodiment.

[0078] This image recognition system 10 acquires an image and imaging parameters from the imaging device 11 (S10). Generally, after imaging is completed by the imaging device 11, a signal serving as a basis for transfer such as a synchronization signal, an interrupt signal, or a data transfer start signal is sent, and based on that timing, the data acquisition process on the image recognition system 10 side is activated, and the data to be subsequently transmitted is sequentially stored in the memory.

[0079] After data acquisition is completed, further image recognition processing is performed by the recognition model generation unit 102. When something is recognized as a recognition result (Yes in S11), subsequent imaging correction parameter calculation processing and the like are performed on the recognized area. On the other hand, when nothing is recognized (No in S11), subsequent imaging correction parameter calculation processing and the like are performed on the entire screen.

[0080] In the process of estimating the imaging device state and imaging device characteristics (S12), the estimation processes by the imaging device state estimation unit 112 and the imaging device characteristics estimation unit 113 described above are performed.

[0081] In the process of calculating imaging correction parameters for the recognition result area of environmental recognition (S13), according to the recognition result area in the recognition model generation unit 102, the parameter generation unit 114 performs the calculation process of imaging correction parameters.

[0082] After this imaging correction parameter calculation process, when the frequency component of the accumulation result within a certain period of the imaging correction parameter is obtained and it fluctuates above a predetermined frequency (Yes in S14), it is considered that hunting has occurred. In order to stabilize the parameter, the hunting is suppressed by the parameter stabilization unit 115 by taking the average with the previous output value or other methods (S15).

[0083] At this time, as described above, when the parameter generation unit 114 has no overlapping area in the parameter range where recognition is stable and performs multiple shootings with different parameters, the frequency calculation may be divided for each parameter.

[0084] Also, when image recognition processing is performed by the recognition model generation unit 102 and nothing is recognized as a recognition result (No in S11), the calculation process of the imaging correction parameters is performed on the area of the entire acquired image.

[0085] As described above, according to the image recognition system of the present invention, it becomes possible to improve the accuracy of image recognition by optimizing the conditions of the imaging device when acquiring an image according to the environment where the performance of image recognition is exhibited.

[0086] <Example 2> FIG. 6 is a block diagram for explaining the image recognition system 10 according to Example 2 of the present invention. The image recognition system 10 in this example is connected via a wired or wireless network to the imaging device 11, receives the imaging data and imaging parameters obtained from the imaging device 11, and transmits the imaging correction parameters to the imaging device 11, which is the same as in Example 1. However, in this example, it is assumed that the imaging device 11 is mounted on a moving vehicle such as an autonomous vehicle, and the difference is that it further has a vehicle information acquisition unit 119. Hereinafter, the details of each part will be described while appropriately omitting the parts overlapping with Example 1.

[0087] The image acquisition unit 101 and the recognition model generation unit 102 in this example are the same as those described in Example 1, so the description thereof is omitted.

[0088] <Vehicle information acquisition unit 119> The vehicle information acquisition unit 119 acquires information regarding the vehicle speed and turning angle of the vehicle on which the imaging device 11 is mounted via a network such as CAN. Considering these information together with the camera's field of view angle, the mounting position and attitude information of the camera with respect to the vehicle, and a vehicle model such as the Ackermann model, it is possible to estimate the change in the camera's field of view accompanying the behavior of the vehicle. Various existing techniques for this estimation method are known, and they can be preferably used.

[0089] <State estimation unit 103> Similar to Example 1, the state estimation unit 103 receives image data and the like from the image acquisition unit 101 and recognition results and the like from the recognition model generation unit 102, and transmits imaging correction parameters to the imaging device 11. The state estimation unit 103 has one or more of an imaging device state estimation unit 112, an imaging device characteristic estimation unit 113, a parameter generation unit 114, and a parameter stabilization unit 115 inside. Among these, since the parameter generation unit 114 is different from that in Example 1, it will be mainly described around this unit.

[0090] <Parameter generation unit 114> The parameter generation unit 114 in Example 2 calculates the imaging correction parameters and the influence on the recognition results of the recognition model generation unit 102 based on the results of the imaging device state estimation unit 112, the imaging device characteristic estimation unit 113, the recognition model generation unit 102, and the output of the vehicle information acquisition unit 119.

[0091] As described above, the vehicle information acquisition unit 119 can estimate the change in the field of view of the camera. Therefore, in this embodiment, considering whether the recognition object included in the image output by the recognition model generation unit 102 is near the center of the screen, near the outer periphery, or outside the turning, among the regions where the recognition object exists, it is characterized in that the regions not used during parameter calculation are determined.

[0092] Specifically, an outer peripheral region determined depending on the speed V is set, and a threshold value for determination is set as Ethr(V). Recognition objects whose region center of the recognition object is included within a range of a distance Ethr(V) from the outer periphery of the screen can be made not to be treated as recognition objects within the parameter generation unit 114. Also, when the turning angle is θ and the speed is V, a threshold value for determination of the outside of the turn is set as Othr(θ, V). Similarly, recognition objects whose region center of the recognition object is included within a range of a distance Ethr(V) from the outer periphery of the screen can be made not to be treated as recognition objects within the parameter generation unit 114.

[0093] This can prevent the parameters from becoming suitable for objects that will soon go out of the field of view and become invisible among the recognition objects included in the captured image during driving and turning, and enables the calculation of parameters suitable for recognition objects present at the center of the screen or in the traveling direction.

[0094] As described above, according to the image recognition system of the present invention, it is possible to improve the accuracy of image recognition by optimizing the conditions of the imaging device when acquiring an image according to the environment in which the performance of image recognition is exhibited.

[0095] According to the embodiments of the present invention described above, the following operational effects can be obtained. (1) The image recognition system according to the present invention includes a processor and a memory, and has an image acquisition unit that acquires an image captured by an imaging device, and at least one environment recognition unit that recognizes the image with at least one environment recognition unit and generates a recognition model. And a state estimation unit that estimates the state of the imaging device and the characteristics of the imaging device corresponding to the state based on the image and the recognition model. The state estimation unit further generates an imaging correction parameter for correcting the imaging conditions of the imaging device based on at least one of the estimated state and characteristics and the recognition model.

[0096] With the above configuration, even if the imaging environment changes due to the running of the vehicle, the change in the weather, etc., and the state and characteristics of the imaging device change, it is possible to monitor and correct them in real time, and optimize the conditions of the imaging device when acquiring an image, thereby improving the image recognition accuracy.

[0097] (2) The state estimation unit includes an imaging device state estimation unit that estimates the state of the imaging device including at least one of the raindrop adhesion state and the cloud adhesion state on the light collecting element of the imaging device, the shielding object between the imaging device and the imaging object, and the exposure state; an imaging device characteristic estimation unit that estimates the characteristics of the imaging device including at least one of the imaging element characteristics, lens characteristics, and color correction characteristics of the imaging device; and a parameter generation unit that generates imaging correction parameters based on at least one of the estimated state and characteristics and the recognition model. Thereby, it becomes possible to accurately grasp each state and characteristic inside and outside the imaging device and generate parameters for correcting them.

[0098] (3) The recognition model generation unit generates a recognition model by recognizing the region where the imaging object exists, and the state estimation unit generates imaging correction parameters for each region where the imaging object exists on the recognition model. Thereby, even when there are a plurality of recognition objects in the acquired image, the present invention can be preferably applied to each of those objects.

[0099] (4) It further has a parameter stabilization unit that converges the imaging correction parameters generated by the parameter generation unit when a hunting phenomenon occurs. Thereby, even when the generated imaging correction parameters cause a hunting phenomenon, it becomes possible to stabilize the parameters, so that it is possible to prevent the behavior of the imaging device from becoming unstable.

[0100] (5) When there are a plurality of imaging conditions to be corrected, the imaging conditions to be corrected are switched every predetermined number of imaging times. Thereby, even when there is no logical product region of the region where the recognition performance of the imaging device is stable on the parameter region or when it is divided into a plurality of regions, it becomes possible to perform optimal parameter adjustment for each region.

[0101] (6) The imaging correction parameters are parameters for correcting at least one of the shutter release time, aperture opening degree, and image signal processing of the imaging device. As a result, almost all functions of the imaging device become correction targets, and thus the present invention can be preferably applied to almost all imaging device states / characteristics.

[0102] (7) The image recognition system is mounted on a vehicle and further includes a vehicle information acquisition unit that acquires vehicle information including at least one of the vehicle speed and turning speed of the vehicle. Based on the vehicle information, the state estimation unit corrects the target area for generating the imaging correction parameters. As a result, even when the recognition target object frequently appears / disappears on the image due to the movement of the vehicle, it becomes possible to select the target object to be actually recognized, and it becomes possible to reduce the arithmetic processing load of the system.

[0103] Note that the present invention is not limited to the above-described embodiments, and various design changes can be made without departing from the spirit of the present invention described in the claims. For example, the above embodiments have been described in detail to assist in understanding the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Also, for a part of the configuration of each embodiment, addition, deletion, or replacement with other configurations is possible.

Description of Reference Numerals

[0104] 10 Image recognition system, 102 Recognition model generation unit, 103 State estimation unit, 112 Imaging device state estimation unit, 113 Imaging device characteristic estimation unit, 114 Parameter generation unit, 115 Parameter stabilization unit, 119 Vehicle information acquisition unit

Claims

1. comprising a processor and a memory, an image acquisition unit that acquires an image captured by an imaging device, having at least one environmental recognition unit, recognizing the image by the at least one environmental recognition unit, recognizing the image based on a recognition model, and outputting a recognition result thereof; a state estimation unit that estimates a state of the imaging device representing an event occurring in the imaging device due to an environment in which the imaging device is placed and a characteristic of the imaging device representing an event occurring in the imaging device due to a function of an internal element constituting the imaging device, based on the image and the recognition result output from the recognition model; the state estimation unit an imaging device state estimation unit that estimates the state of the imaging device including at least one of a raindrop adhesion state and a cloud adhesion state on a light collecting element of the imaging device and a shielding object between the imaging device and an imaging target; an imaging device characteristic estimation unit that estimates the characteristic of the imaging device including at least one of an imaging element characteristic, a lens characteristic, and a color correction characteristic of the imaging device; a parameter generation unit that generates an imaging correction parameter for correcting an imaging condition of the imaging device based on the estimated state and characteristic and the recognition result output from the recognition model; An image recognition system characterized by the above.

2. The image recognition system according to claim 1, wherein the recognition result output unit generates the recognition model by recognizing a region where an imaging target exists, and the state estimation unit generates the imaging correction parameter for each region where the imaging target exists on the recognition model. An image recognition system characterized by the above.

3. The image recognition system according to claim 1, further comprising a parameter stabilization unit that converges the imaging correction parameter generated by the parameter generation unit when a hunting phenomenon occurs. An image recognition system characterized by the above.

4. The image recognition system according to claim 1, wherein when there are a plurality of imaging conditions to be corrected, the imaging conditions to be corrected are switched every predetermined number of imaging times. An image recognition system characterized by the above.

5. The image recognition system according to claim 1, wherein when there are a plurality of imaging conditions to be corrected, the imaging conditions to be corrected are switched in accordance with an execution period of the environmental recognition unit. An image recognition system characterized by the above.

6. The image recognition system according to claim 1, The imaging correction parameter is a parameter for correcting at least one of a shutter release time, an aperture opening degree, and image signal processing of the imaging device. An image recognition system characterized by this.

7. The image recognition system according to claim 1, wherein the image recognition system is mounted on a vehicle and further includes a vehicle information acquisition unit that acquires vehicle information including at least one of a vehicle speed and a turning speed of the vehicle, and based on the vehicle information, the state estimation unit corrects a target area for generating the imaging correction parameter. An image recognition system characterized by this.

Images