Image recognition support device, image recognition support method, and program

Abstract

To provide an image recognition support device, an image recognition support method, and a program for displaying a more appropriate display image.SOLUTION: An image recognition support device includes: a first processing unit for performing first image processing relative to a photographed image to generate a first image; a second processing unit for performing second image processing for image recognition relative to the first image to generate a second image; a recognition processing unit for performing object recognition processing based on the second image; a determination unit for determining whether or not the first image and the second image are appropriate as a display image; and a display image output unit for outputting the display image using at least one of the first image and the second image according to determination of the determination unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image recognition support device, an image recognition support method, and a program. [Background technology]

[0002] In an image recognition system, the image quality of an image input to a recognition engine that performs recognition processing has a large effect on the recognition accuracy. For example, if a dark image, an image that is too bright, or an image with a lot of noise is input, the recognition accuracy of the recognition engine decreases. Patent Document 1 discloses an image processing device that performs image processing for image recognition separately from image processing for image display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-158682 A Summary of the Invention [Problem to be solved by the invention]

[0004] In some cases, an image for image recognition is more appropriate as a display image than an image for image display. In such cases, conventional image recognition systems have a problem in that they are unable to display a more appropriate display image.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an image recognition support device, an image recognition support method, and a program that display a more appropriate display image. [Means for solving the problem]

[0006] The image recognition support device according to the present disclosure comprises: a first processing unit that performs a first image processing on the captured image to generate a first image; a second processing unit that performs a second image processing for image recognition on the first image to generate a second image; A recognition processing unit that performs object recognition processing based on the second image; a determination unit that determines whether the first image and the second image are appropriate as display images; a display image output unit that outputs the display image using at least one of the first image and the second image based on the determination by the determination unit; Equipped with.

[0007] The image recognition assistance method according to the present disclosure includes: performing a first image processing on the captured image to generate a first image; performing a second image processing for image recognition on the first image to generate a second image; performing an object recognition process based on the second image; determining whether the first image and the second image are appropriate as display images; outputting the display image using at least one of the first image and the second image based on the determination in the determining step; Includes.

[0008] The program according to the present disclosure is performing a first image processing on the captured image to generate a first image; performing a second image processing for image recognition on the first image to generate a second image; performing an object recognition process based on the second image; determining whether the first image and the second image are appropriate as display images; outputting the display image using at least one of the first image and the second image based on the determination in the determining step; to be executed by the computer. Effect of the Invention

[0009] The present disclosure makes it possible to provide an image recognition assistance device, an image recognition assistance method, and a program that display a more appropriate display image. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an overall configuration of an image recognition system including an image recognition support device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a detailed configuration of an image synthesis unit according to the first embodiment. [Diagram 3] 4 is a flowchart showing an example of the operation of the image recognition support device according to the first embodiment. [Figure 4] 4 is a diagram for explaining effects achieved by the image recognition support device according to the first embodiment. FIG. [Diagram 5] 5A to 5C are diagrams for explaining the operation of the boundary processing unit according to the first embodiment. [Figure 6] 6 is a flowchart showing an example of the operation of the boundary processing unit according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing the overall configuration of an image recognition system including an image recognition support device according to a second embodiment. [Figure 8] FIG. 11 is a block diagram showing detailed configurations of a determination unit and an image switching unit according to the second embodiment. [Figure 9] 11 is a flowchart showing a first example of the operation of the image recognition support device according to the second embodiment. [Figure 10] 13 is a flowchart showing a second example of the operation of the image recognition support device according to the second embodiment. [Figure 11] 13 is a flowchart showing a third example of the operation of the image recognition support device according to the second embodiment. [Figure 12] 13 is a flowchart showing a fourth example of the operation of the image recognition support device according to the second embodiment. [Figure 13] FIG. 11 is a block diagram showing the overall configuration of an image recognition system according to a third embodiment. [Figure 14] FIG. 11 is a block diagram showing the configuration of a processing recombination unit according to a third embodiment. [Figure 15] FIG. 11 is a block diagram showing the configuration of a determination unit according to the third embodiment. [Figure 16] 13 is a flowchart showing a first example of the operation of the image recognition support device according to the third embodiment. [Figure 17] 13 is a flowchart showing a second example of the operation of the image recognition support device according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, specific embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0012] EMBODIMENT 1 1 is a block diagram showing an overall configuration of an image recognition system 1000 including an image recognition support device 200 according to the first embodiment. The image recognition system 1000 includes a camera 100, an image recognition support device 200, an image recognition engine 300, and a display device 400. The camera 100 is an imaging device that captures a landscape including people, cars, and the like, and captures image data as a captured image 41. The camera 100 may input the captured video data to the image recognition support device 200 in succession in frame image units. The camera 100 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0013] The image recognition support device 200 performs standard image quality adjustment and image quality adjustment for recognition on the captured image 41. The image recognition support device 200 adjusts the adjustment values ​​2110 to 211nmax of the adjustment value group 211 to 21nmax in order based on the recognition result 43 of the image recognition on the adjusted image 42nmax. Each adjustment value group is associated with at least one image quality adjustment parameter. The adjustment value groups 211 to 21nmax may be associated with different image quality adjustment parameters. By repeating the feedback of the recognition result 43, the image recognition support device 200 continues image quality adjustment until the recognition result 43 is stabilized at a high recognition rate. The same image may be repeatedly used as the captured image 41, or a new image captured by the camera 100 may be used as the captured image 41 each time.

[0014] The image recognition engine 300 performs image recognition processing on the adjusted image 42nmax output from the image recognition support device 200, and outputs a recognition result 43. The recognition result 43 includes the presence or absence of a recognition target, the type of the recognition target, the area or position of the recognition target, and a recognition rate. The presence or absence of a recognition target is information indicating whether or not a recognition target is recognized, that is, identified, by the image recognition processing on the adjusted image 42nmax. The recognition target is, for example, a person, a car, etc. The recognition target type is information indicating the type of the recognition target. The recognition target area is a coordinate group that defines the range of an area including the recognition target recognized in the adjusted image 42nmax. The recognition target area is, for example, a range specified by pixel values ​​in an XY coordinate system. The recognition target position is the position of the recognition target recognized in the adjusted image 42nmax, for example, a representative point such as a central coordinate. The recognition rate is an example of the degree of certainty of a recognition result by image recognition. That is, the recognition rate is numerical information indicating the presence or absence of a recognition target, the recognition target type, and the recognition accuracy of the recognition target area of ​​the recognition target object recognized by the image recognition process for the adjusted image 42nmax. The recognition rate may be indicated, for example, from 0 to 100%. In addition, for example, a threshold indicating the similarity with the recognition target object, the number of stages passed by the classifier, etc. may be used to calculate the recognition rate. Note that, when multiple recognition targets are recognized, a set of the recognition target type, the recognition target area, and the recognition rate may be generated as the recognition result 43 for each recognition target object. Also, an area including multiple recognition targets may be set as the recognition target area. In this case, the recognition rate may be generated for each recognition target object. Note that the term "recognition" described above may be replaced with "identification". The image recognition engine 300 is also called a recognition processing unit.

[0015] The image recognition engine 300 is hardware or software capable of executing a known image recognition process, or a combination of these. For example, the image recognition engine 300 may be a computer on which a known image recognition processing program is executed. The image recognition engine 300 may be redundantly implemented in a plurality of computers, and each functional block may be implemented in a plurality of computers. The image recognition engine 300 may be implemented in a form in which each is connected via a communication network, such as a client-server system or a cloud computing system. The function of the image recognition engine 300 may be provided in a SaaS (Software as a Service) format. Alternatively, the image recognition engine 300 may be implemented in the same computer as the image recognition support device 200.

[0016] The display device 400 is a display device or the like that displays the display image 44 generated by the image recognition support device 200. The display image 44 may be processed using the recognition result 43. The display device 400 may display, for example, a bounding box surrounding the recognition target area in the display image 44, character information corresponding to the recognition target type, recognition determination results such as a recognition rate, etc., on an OSD (On-Screen Display). The recognition target area in the display image 44 may be an image area corresponding to the recognition target area recognized in the adjusted image 42nmax.

[0017] Image recognition support device 200 is an information processing device including image quality adjustment unit 21, acquisition unit 22, setting unit 23, determination unit 24, and image synthesis unit 25. In image quality adjustment unit 21, standard adjustment value group 210 is set in advance, and adjustment value group 211 to 21nmax (nmax is a natural number equal to or greater than 2) is set according to recognition result 43. The hardware configuration of image recognition support device 200 will be described later.

[0018] The standard adjustment value group 210 is a collection of adjustment values ​​used when performing standard image quality adjustment on the signal of the captured image 41. The standard adjustment value group 210 may be a collection of initial values ​​set in advance for the adjustment value of each image quality adjustment parameter. An "adjustment value" is a value of each image quality adjustment parameter. The image quality adjustment parameter is also referred to as an image quality type. The adjustment value 2110 is an adjustment value included in the standard adjustment value group 210.

[0019] The adjustment value group 211 is used when performing image quality adjustment for image recognition on the signal of the adjusted image 420 obtained by adjusting the standard adjustment value group 210. The adjustment value groups 212 to 21nmax are used when performing image quality adjustment for image recognition on the signals of the adjusted images 421 to 42(nmax-1) obtained by adjusting the adjustment value groups 211 to 21(nmax-1), respectively. The adjustment values ​​2111 to 211nmax are adjustment values ​​included in the adjustment value groups 211 to 21nmax, respectively. The adjustment value 2111 and the like are determined by the setting unit 23 in response to feedback of the recognition result 43. Each of the adjustment value groups 211 to 21nmax is associated with at least one image quality adjustment parameter. Each adjustment value group may include one adjustment value or may include multiple adjustment values.

[0020] The image quality adjustment unit 21 adjusts the image quality of the photographed image 41 using the standard adjustment value group 210, the adjustment value group 211, etc., and outputs the adjusted image 42nmax to the image recognition engine 300. That is, the image quality adjustment unit 21 performs pre-processing of the image recognition engine 300 on the photographed image 41. For example, the image quality adjustment unit 21 performs standard image quality adjustment (also called first image processing) on ​​the photographed image 41 using the standard adjustment value group 210 as the first image quality adjustment. Here, standard image quality adjustment refers to adjustment to a level of image quality that is statistically evaluated as clean and of high image quality when viewed by various people. For example, the image quality adjustment unit 21 adjusts the signal of an image photographed under low illuminance to image data that appears as bright as possible by balancing the S / N ratio, resolution, color reproducibility, etc. Then, the image quality adjustment unit 21 performs image quality adjustment for recognition (also referred to as second image processing) on ​​the adjusted image 420 (also referred to as the first image) after the standard image quality adjustment, using an adjustment value (any of the adjustment values ​​2111 to 211nmax) set in response to feedback of the recognition result 43.

[0021] The second image processing includes a plurality of image processing steps corresponding to the adjustment value group 211 to 21nmax, and the adjustment value group 211 to 21nmax is adjusted in order. The image quality adjustment unit 21 performs the second image processing for image recognition to generate the adjusted image 421 to 42nmax. The adjusted image 421 to 42nmax is also called the second image. As an example of the order of adjustment of the adjustment value group 211 to 21nmax, the adjustment may be performed in order from the setting value that has the greatest effect on the recognition rate. For example, the order is brightness adjustment, noise reduction processing, and edge emphasis. The brightness adjustment may be performed in the whole frame, or in the method of dividing the frame into several parts and adjusting each area. The noise reduction processing mainly removes components of high frequency components of the image that have an amplitude less than a threshold value as noise. The edge emphasis is a processing to mainly increase the high frequency components of the image. The order of adjustment of the adjustment value group 211 to 21nmax may be arbitrarily changed, or an arbitrary adjustment value group may be skipped. The order of adjustment and the type of processing are merely examples and may be other methods. The image quality adjustment unit 21 outputs the adjusted image 420 in which the standard adjustment value group 210 has been adjusted, and the adjusted images 421-42nmax in which the adjustment value groups 211-21nmax have been adjusted, respectively, to the image synthesis unit 25. As already described, the adjusted image 42nmax is also output to the image recognition engine 300. The object recognition process by the image recognition engine 300 described above is performed based on the second image.

[0022] The adjustment target when the image quality adjustment unit 21 adjusts the image quality is also called an image quality adjustment parameter. The image quality adjustment parameter is, for example, but not limited to, luminance, S / N (Signal / Noise) ratio, resolution, etc. Note that luminance may also be called brightness, brightness, etc. Resolution may be called contour emphasis, enhancement, aperture, etc.

[0023] The acquisition unit 22 acquires a recognition result 43 obtained by performing image recognition on the adjusted image 42nmax by the image recognition engine 300, and outputs the recognition result 43 to the setting unit 23. The acquisition unit 22 acquires at least a recognition rate of the recognition object included in the recognition result 43, and a recognition target area including the recognition object.

[0024] The setting unit 23 sets the adjustment value of each image quality adjustment parameter for each recognition target region in the image quality adjustment unit 21 based on the recognition result 43. The adjustment values ​​2111 to 211nmax are adjusted to improve the next recognition accuracy. In particular, when the recognition rate included in the recognition result 43 is less than a predetermined value, the setting unit 23 sets the adjustment values ​​2111 to 211nmax in the image quality adjustment unit 21 so that the recognition rate becomes equal to or greater than the predetermined value. This adjusts the image quality of the recognition target region in the captured image 41, improving the next recognition accuracy. The setting unit 23 may further include a function of setting an initial value to the adjustment value 2110. The setting unit 23 may adjust the standard adjustment value group 210 based on the recognition result 43.

[0025] The determination unit 24 calculates information regarding the suitability of each of the adjusted images 420 to 42nmax as a display image. For example, the determination unit 24 calculates α1=K1-H1 for the adjusted image 421. The deviation coefficient K1 is an arbitrary constant. H1 is the absolute value of the deviation between the setting value of the adjustment value 2111 and the adjustment reference value. The adjustment reference value represents a predetermined optimum value of the adjustment value 2111. The larger α1 is, the more appropriate the adjusted image 421 is as a display image. It can also be said that the adjustment reference value represents a visual optimum value.

[0026] Similarly, the determination unit 24 calculates α2=K2-H2 for the adjusted image 422 in which the adjustment value group 212 (not shown) has been adjusted. The deviation coefficient K2 is an arbitrary constant. H2 is the absolute value of the deviation between the set value of the adjustment value 2112 and the adjustment reference value. The adjustment reference value represents a predetermined optimum value of the adjustment value 2112. The larger α2 is, the more appropriate the adjusted image 422 is as a display image.

[0027] Similarly, the determination unit 24 calculates αn=Kn-Hn for an adjusted image 42n in which an unillustrated adjustment value group 21n has been adjusted. n represents any integer between 1 and nmax. The deviation coefficient Kn is any constant. Hn is the absolute value of the deviation between the adjustment value 211n and the adjustment reference value. The adjustment reference value represents a predetermined optimum value of the adjustment value 211n. The larger αn is, the more appropriate the adjusted image 42n is as a display image.

[0028] Similarly, the determination unit 24 calculates αnmax=Knmax-Hnmax for the adjusted image 42nmax. The deviation coefficient Knmax is an arbitrary constant. Hnmax is the absolute value of the deviation between the adjustment value 211nmax and the adjustment reference value. The adjustment reference value represents a predetermined optimal value of the adjustment value 211nmax. The larger αnmax is, the more appropriate the adjusted image 42nmax is as a display image.

[0029] The determination unit 24 calculates α0=1-{(normalization coefficient 1)*α1+(normalization coefficient 2)*α2+···+(normalization coefficient n)*αn+···+(normalization coefficient nmax)*αnmax} for the adjusted image 420, where 0≦α0≦1. The larger α0 is, the more appropriate the adjusted image 420 is as a display image compared to the adjusted images 421 to 42nmax.

[0030] α0 to αnmax are also referred to as information regarding the appropriateness of the adjusted images 420 to 42nmax. The information regarding appropriateness is an index indicating the degree of appropriateness as a display image. If the information regarding appropriateness is zero, the corresponding adjusted image is not appropriate, and the adjusted image is not superimposed on a superimposed image, which will be described later. It can be said that the determination unit 24 determines whether the adjusted images 420 to 42nmax are appropriate or not.

[0031] The determination unit 24 outputs the calculated values ​​α0 to αnmax to the image synthesis unit 25. Note that the method of calculating α0 to αnmax is not limited to the above-mentioned method using the deviation coefficients K1 to Knmax and the absolute values ​​of deviation H1 to Hnmax.

[0032] The image synthesis unit 25 outputs a superimposed image obtained by superimposing the adjusted images 420-42nmax based on α0-αnmax to the display device 400. The image synthesis unit 25 is an example of the display image output unit described above. The adjusted image 420 corresponds to the first image, and the adjusted images 421-42nmax correspond to the second image. When α0 is sufficiently larger than α1-αnmax, the image synthesis unit 25 outputs the first image as the display image. When α1-αnmax are sufficiently larger than α0, the image synthesis unit 25 outputs a display image using the second image.

[0033] The image synthesis unit 25 sets the weight of the adjusted image 420 to α0, for example. The image synthesis unit 25 sets the weight assigned to the adjusted image 421 to (normalization coefficient 1)*α1, for example. The image synthesis unit 25 similarly sets the weight assigned to the adjusted image 422 to (normalization coefficient 2)*α2. The image synthesis unit 25 similarly sets the weight assigned to the adjusted image 42n to (normalization coefficient n)*αn. The image synthesis unit 25 sets the weight assigned to the adjusted image 42nmax to (normalization coefficient nmax)*αnmax, where α0+(normalization coefficient 1)*α1+(normalization coefficient 2)*α2+···+(normalization coefficient n)*αn+···+(normalization coefficient nmax)*αnmax=1.

[0034] Furthermore, the image synthesis unit 25 may input α0 to αnmax to a trained model, and superimpose the adjusted images 420 to 42nmax in accordance with the optimal weights output from the trained model.

[0035] The function of the image synthesis unit 25 will be described in more detail with reference to Fig. 2. The image synthesis unit 25 includes a calculation unit 251, a boundary processing unit 252, and a recognition result superimposition processing unit 253.

[0036] The calculation unit 251 receives adjusted image 420 in which standard adjustment value group 210 has been adjusted, and receives adjusted images 421-42nmax in which adjustment value groups 211-21nmax have been adjusted, respectively. The calculation unit 251 also receives α0-αnmax, normalization coefficient 1, ... normalization coefficient n, ... normalization coefficient nmax from the determination unit 24. The calculation unit 251 then generates a superimposed image in which adjusted images 420-42nmax are superimposed based on α0-αnmax and the respective normalization coefficients.

[0037] Image quality adjustment using the adjustment value group 211-21nmax may be performed for each recognition target region. In this case, the calculation unit 251 receives α0-αnamx for each recognition region, the respective normalization coefficients, and adjusted images 421-42nmax, and generates a superimposed image for each recognition region. Therefore, when generating a display image, multiple superimposed images need to be stitched together.

[0038] The boundary processing unit 252 performs boundary processing for smoothly joining a plurality of superimposed images output from the calculation unit 251. Details of the boundary processing will be described later.

[0039] The recognition result overlay processing unit 253 overlays the recognition result 43 on the image on which boundary processing has been performed by the boundary processing unit 252. The recognition result overlay processing unit 253 may add a bounding box surrounding the recognition target to the image on which boundary processing has been performed.

[0040] Returning to FIG. 1, the image recognition support device 200 may be realized, for example, by one computer. When the image recognition support device 200 is mounted on an automobile or the like, the computer may be, but is not limited to, an ECU (Electronic Control Unit). The image recognition support device 200 may be redundantly configured with a plurality of computers, and each functional block may be realized by a plurality of computers. Alternatively, all or a part of the functions of the image recognition support device 200 may be realized by a general-purpose or dedicated circuit such as a semiconductor device. In these cases, the image recognition support device 200 may be communicably connected to the camera 100 and the image recognition engine 300 via a communication network.

[0041] The image recognition support device 200 includes, as hardware configuration, a storage unit (not shown), an IF (Interface) unit (not shown) for communicating with the outside, and a control unit (not shown). The storage unit includes a non-volatile storage device such as a hard disk or flash memory, and a memory such as a RAM (Random Access Memory), that is, a volatile storage device. The storage unit stores an image recognition support program (not shown), a standard adjustment value group 210, and adjustment value groups 211 to 21nmax.

[0042] The control unit is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), or a quantum processor (quantum computer control chip). The control unit loads the image recognition support program from a non-volatile storage device in the storage unit into the memory, and executes the image recognition support program. In this way, the control unit realizes the functions of the image quality adjustment unit 21, the acquisition unit 22, the setting unit 23, the determination unit 24, and the image synthesis unit 25. Note that some or all of the image quality adjustment unit 21, the acquisition unit 22, the setting unit 23, the determination unit 24, and the image synthesis unit 25 may be realized by hardware other than the control unit, for example, a general-purpose or dedicated circuit realized by a semiconductor device.

[0043] 3 is a flowchart showing an example of the operation of the image recognition support device 200. First, the image quality adjustment unit 21 performs standard image quality adjustment on the captured image 41 captured by the camera 100 using the standard adjustment value group 210 (step S101). Then, the image quality adjustment unit 21 performs recognition image quality adjustment on the adjusted image 420 that has been subjected to the standard image quality adjustment using the adjustment value group 211, etc. (step S102). Note that if the adjustment value group 211, etc. have not been set the first time, step S102 may be omitted.

[0044] Then, the image quality adjustment unit 21 outputs the adjusted image 42nmax to the image recognition engine 300, and the image recognition engine 300 performs image recognition on the adjusted image 42nmax (step S103). The image recognition engine 300 outputs the recognition result 43, and the acquisition unit 22 acquires the recognition result 43 from the image recognition engine 300.

[0045] Then, the setting unit 23 acquires a determination result of whether the recognition rate included in the acquired recognition result 43 is equal to or greater than a predetermined value or less. Then, the setting unit 23 calculates the recognition frequency according to the presence or absence of a recognition target included in the recognition result 43 and the determination result. Here, the "recognition frequency" is the number of successful image recognitions in the total number of image recognitions in step S103 within a certain period of time. Specifically, the setting unit 23 accumulates the number of times that a predetermined recognition target is successfully recognized in the adjusted image 42nmax as the recognition frequency, and calculates the recognition frequency per total number of steps S103 as the recognition frequency. For example, when the presence or absence of a recognition target indicates "present", the setting unit 23 adds 1 to the recognition frequency. Furthermore, the setting unit 23 may add 1 to the recognition frequency when the recognition rate is equal to or greater than a predetermined value.

[0046] Then, the setting unit 23 judges whether the recognition frequency is 0 or not (step S104). Here, when the recognition frequency is 0, it means that the number of recognitions within a certain period of time is 0. For example, this corresponds to a case where the brightness is significantly insufficient in the first few image quality adjustments, particularly in standard image quality adjustments, and the object to be recognized cannot be identified at all by image recognition. Therefore, when the recognition frequency is 0, it also means that the recognition rate immediately before is 0, that is, the recognition rate is less than a predetermined value.

[0047] If the recognition frequency is 0 (NO in step S104), the setting unit 23 sets the recognition target area to the entire captured image 41, and sets the image quality adjustment unit 21 to significantly increase or decrease the adjustment value of the image quality adjustment parameter (e.g., brightness) (step S105). The setting unit 23 sets the brightness setting value so as to significantly increase the brightness compared to standard image quality adjustment, for example. The setting unit 23 may also set the brightness for the entire captured image 41 so as to significantly increase it compared to standard image quality adjustment. In other words, when the recognition rate is less than a predetermined value, the setting unit 23 sets the adjustment value so that the recognition rate in the next image recognition will be equal to or greater than the predetermined value.

[0048] Then, the image quality adjustment unit 21 performs recognition image quality adjustment on the image that has been subjected to the standard image quality adjustment in step S101, using the adjustment value group 211 set in step S105, etc. (step S102). Then, steps S103 to S104 are performed as described above. If it is determined in step S104 that the recognition frequency is greater than 0, the setting unit 23 may set the recognition target area included in the recognition result 43 to the image quality adjustment unit 21.

[0049] Then, the setting unit 23 judges whether the recognition is stable, that is, whether the recognition frequency is equal to or greater than the stable number (step S106). For example, if the image recognition process is performed 30 times per second, the stable number may be 20 times. In this case, if the recognition frequency is 2 / 3 or more, the recognition is stable, and if it is less than 2 / 3, the recognition is unstable. Note that the stable number and the predetermined time (1 second) are merely examples and are not limited to these.

[0050] If the recognition is not stable in step S106, the setting unit 23 sets the image quality adjustment unit 21 to gradually increase or decrease the adjustment value of the image quality adjustment parameter (e.g., luminance) (step S107). That is, the amount of increase or decrease in step S107 is smaller than the amount of increase or decrease in step S105. For example, if the recognition frequency has increased to a certain extent but is unstable, the setting unit 23 may set the luminance to be increased by a fixed width Y smaller than the increase amount X in step S105. In particular, by narrowing the recognition target area from the entire captured image 41 to the image-recognized area and adjusting the luminance, the image quality is adjusted more finely than in the standard image quality adjustment, and the recognition accuracy is improved. In addition, repeated adjustments may cause the luminance to increase too much, making the recognition target area too bright, which may increase noise, reduce the recognition rate, and make the recognition frequency unstable. In this case, the setting unit 23 may set the luminance of the recognition target area to be decreased by a fixed width Y to reduce noise. Then, the image quality adjustment unit 21 performs recognition image quality adjustment on the image that has been subjected to the standard image quality adjustment using the adjustment value group 211 adjusted in step S107 (step S102). Then, steps S103 to S106 are performed as described above.

[0051] If noise increases due to excessive brightness, the image quality adjustment unit 21 may perform noise reduction processing or noise removal processing. Here, the noise removal processing is processing to remove noise in pixel signals caused by defects in the imaging element of the camera by median processing or the like. And the median processing is a filtering processing to convert the pixel values ​​of the target pixel into a pixel with a median value by comparing the magnitudes of the surrounding pixels.

[0052] If the recognition is stable in step S106, the setting unit 23 calculates whether the recognition rate is saturated, that is, whether the difference in the change in the recognition rate is less than a threshold value (step S108). First, the setting unit 23 calculates the average recognition rate for each recognition rate in multiple image recognitions for a predetermined number of times. For example, the setting unit 23 calculates the average recognition rate A1 in the first to tenth image recognitions and the average recognition rate A2 in the eleventh to twentieth image recognitions. Then, the setting unit 23 calculates the difference between A1 and A2. Then, the setting unit 23 determines whether the difference is less than a threshold value.

[0053] A difference less than the threshold value means that the recognition frequency is sufficient and stable even when the adjustment value of a specific image quality adjustment parameter is changed, the recognition rate is stable within a certain period of time, and recognition is saturated. For example, if the average recognition rate before changing the adjustment value of the image quality adjustment parameter "brightness" is 70%, and the average recognition rate after changing the adjustment value by a certain range changes to between 68 and 72%, recognition is saturated. Note that the recognition rate may vary in image recognition processing using different adjustment values ​​or different image quality adjustment parameters, so a threshold value or number of stable times may be set that takes the variation into consideration.

[0054] If the recognition rate is not saturated, that is, if the difference is equal to or greater than the threshold (NO in step S108), the setting unit 23 gradually increases or decreases the adjustment value of the image quality adjustment parameter (e.g., luminance) for the recognition target region (step S109). If the difference is equal to or greater than the threshold, the recognition rate may be increasing, so the setting unit 23 may increase the luminance by a fixed width Z smaller than X, for example. The adjustment value Z may be different from the adjustment value Y used in step S107. If the recognition rate is decreasing even if the difference is equal to or greater than the threshold, the setting unit 23 may set the luminance of the recognition target region to be decreased by a fixed width Z. The relationship between the increase and decrease in the fixed width Z corresponding to the increase and decrease in the recognition rate may be reversed. Then, the image quality adjustment unit 21 performs image quality adjustment for recognition on the adjusted image 420 that has been subjected to the standard image quality adjustment, using the adjustment value group 211 adjusted in step S109, etc. (step S102). Then, steps S103 to S108 are performed as described above.

[0055] If the recognition rate is saturated in step S108, the determination unit 24 completes the adjustment of one image quality adjustment parameter (e.g., luminance), and calculates information regarding the appropriateness of the adjusted image in which the image quality adjustment parameter has been adjusted (step S110). The information regarding the appropriateness may be calculated based on the deviation between a setting value set by the setting unit 23 and an adjustment reference value. The determination unit 24 outputs the information regarding the appropriateness to the image synthesis unit 25. In addition, the image quality adjustment unit 21 may output the corresponding adjusted image to the image synthesis unit 25.

[0056] Then, setting unit 23 determines whether any unadjusted image quality adjustment parameters remain (step S111). For example, when recognition is saturated for the image quality adjustment parameter "brightness," other image quality adjustment parameters such as "S / N ratio" and "resolution" may remain unadjusted. If there is an unadjusted image quality adjustment parameter, setting unit 23 changes the image quality adjustment parameter (step S112) and returns to step S102. Steps S102 to S111 may be repeated for each recognition target region, that is, for each image quality adjustment section.

[0057] If there are no unadjusted image quality adjustment parameters remaining in step S111, the image quality adjustment for image recognition ends. Image synthesis unit 25 calculates α0 based on α1 to αnmax, and outputs a superimposed image in which adjusted images 420 to 42nmax are superimposed based on α0 to αnmax.

[0058] If the recognition frequency remains at 0 for a certain period of time without improvement even when the adjustment value is increased or decreased significantly within a certain range, it is possible to move to the next parameter adjustment with α=0 for the target α. If it never stabilizes for all parameters, it is possible that the object to be recognized does not exist, in which case adjustment can be started again from the initial adjustment value after a certain period of time. If the frequency remains at 0 for several adjustment value groups from the start and α stabilizes for the first time during that period, the stable parameters are adopted to determine the subsequent adjustment value groups, but the adjustment value before stability can be readjusted.

[0059] 4 is a diagram for explaining the effect of embodiment 1. Note that adjusted images other than the adjusted images 420, 421, and 42nmax are omitted in the illustration.

[0060] The adjusted image 420 represents an image that has been subjected to only standard image quality adjustment using the standard adjustment value group 210. The adjusted image 420 represents, for example, an image that has been subjected to standard image quality adjustment on an image taken at night, and includes a low-illuminance area. The adjusted image 420 that includes a low-illuminance area is not necessarily appropriate as a display image. The standard image quality adjustment is performed to achieve a balance between S / N, resolution, color reproduction, and the like, and to adjust the image so that it is visually optimal as a frame for various scenes. Therefore, although there is no visual problem with the image of the entire frame, in scenes with a relatively wide dynamic range, such as nighttime shooting or near the exit of a tunnel on a sunny day, brightness is insufficient in some areas. Depending on the purpose of the system and the user's image quality preferences, in scenes with a wide dynamic range, it may be determined that compositing an image with an adjustment that has a high recognition rate of objects to be recognized in low-illuminance areas with the standard adjusted image provides better visibility in low-illuminance areas and is visually better.

[0061] The adjusted image 421 represents an image whose luminance has been adjusted using the adjustment value group 211. The adjusted image 421 is easy for humans to recognize and is an appropriate image to be displayed. Moreover, compared to the adjusted image 420, the adjusted image 421 is an appropriate image for the image recognition engine 300.

[0062] The adjusted image 42nmax represents an image that has undergone edge enhancement processing using the adjustment value group 21nmax. The adjusted image 42nmax is an image that has undergone all image quality adjustments, and is therefore an optimal image for the image recognition engine 300. However, the adjusted image 42nmax may be an unnatural image with too much edge enhancement, or an image with excessive noise reduction that has lost fine texture. In this case, the adjusted image 42nmax is not appropriate as a display image. Note that, although an adjusted image nmax with too much edge enhancement is shown, the type of image that is obtained as the adjusted image nmax is determined by the image recognition engine 300.

[0063] In this case, the image synthesis unit 25 outputs the display image 44 that is close to the adjusted image 421. The display image 44 is a visually appropriate image.

[0064] Next, a detailed description will be given of the boundary processing performed by boundary processing unit 252 of image synthesis unit 25. Fig. 5 is a diagram for explaining an example of boundary processing when synthesizing four superimposed images A to D. Boundary processing unit 252 performs boundary processing on image regions A1 to A3 in superimposed image A, image regions B1 to B3 in superimposed image B, image regions C1 to C3 in superimposed image C, and image regions D1 to D3 in superimposed image D.

[0065] The luminance of each pixel included in the image regions A2, B2, C2, and D2 is corrected based on both the distance between the pixel and a boundary line (called a first boundary line) extending vertically in FIG. 5 and the distance between the pixel and a boundary line (called a second boundary line) extending horizontally in FIG. 5. Meanwhile, the luminance of each pixel included in the image regions A1, B1, C1, and D1 is corrected based on the distance between the pixel and the first boundary line. The luminance of each pixel included in the image regions A3, B3, C3, and D3 is corrected based on the distance between the pixel and the second boundary line.

[0066] For example, when correcting the luminance PLa2n of the pixel Pa2 in the image area A2, the boundary processing unit 252 calculates a first offset OffV2ab with respect to the first boundary line V2ab and a second offset OffV2ac with respect to the second boundary line V2ac. The boundary processing unit 252 calculates the corrected luminance PLa2n (1+OffV2ab / 2+OffV2ac / 2).

[0067] The first offset OffV2ab is calculated by the formula OffV2ab=((TwV2ab-TpV2ab) / TwV2ab)*(V2abave-A2ave). TwV2ab represents the dimension of the image area A2 in the horizontal direction in FIG. 5. TpV2ab represents the distance between the boundary line V2ab and the pixel Pa2. V2abave is calculated by the formula V2abave=(A2ave+B2ave) / 2. A2ave is the average luminance value in the image area A2, and B2ave is the average luminance value in the image area B2.

[0068] The second offset OffV2ac is calculated by the formula OffV2ac=((TwV2ac-TpV2ac) / TwV2ac)*(V2acave-A2ave), where TwV2ac represents the dimension of the image area A2 in the vertical direction in FIG. 5. TpV2ac represents the distance between the boundary line V2ac and pixel Pa2. V2acave is calculated by the formula V2acave=(A2ave+C2ave) / 2, where C2ave is the average luminance value in the image area C2.

[0069] Furthermore, for example, when correcting the luminance PLa1n of the pixel Pa1 in the image area A1, the boundary processing unit 252 calculates a first offset OffV1ab related to the boundary line V1ab. The first offset OffV1ab is calculated in the same manner as the first offset OffV2ab. The boundary processing unit 252 calculates PLa1n (1+OffV1ab) as the corrected value of the luminance Pla1n.

[0070] Furthermore, for example, when correcting the luminance PLa3n of the pixel Pa3 in the image area A3, the boundary processing unit 252 calculates a second offset OffV3ac related to the boundary line V3ac. The second offset OffV3ac is calculated in the same manner as the second offset OffV2ac. The boundary processing unit 252 calculates PLa3n(1+OffV3ac) as the corrected value of the luminance PLa3n.

[0071] The boundary processing unit 252 reduces the brightness difference at the boundary through the above processing, and can output a more natural display image. Note that the method for reducing the brightness difference at the boundary is not limited to the above method, and other known methods may be used.

[0072] Furthermore, when joining image region A, image region B, image region C, and image region D, the boundary processing unit 252 may correct the gain of pixels in each image region so that the average luminance of all regions becomes equal. FIG. 6 is a flowchart showing an example of the operation of the boundary processing unit 252. First, the boundary processing unit 252 obtains the average luminance value of each of the image regions A, B, C, and D (step S201). The average luminance values ​​of the image regions A, B, C, and D are set to ARAAve, ARBAve, ARCAve, and ARDAve, respectively.

[0073] Next, the boundary processing unit 252 calculates the average luminance values ​​ARAAve, ARBAve, ARCAve, and ARDAve, and determines the image region having the average luminance value closest to the average values ​​as the reference region (step S202).

[0074] Next, the boundary processing unit 252 calculates the correction gain of the image area based on the average brightness value of the reference area and the average brightness value of each image area (step S203). For example, when the image area B is the reference area, the correction gain of the image area A is calculated as ARBAve / ARAAve. The correction gain for the reference area is 1.

[0075] Next, the boundary processing unit 252 corrects the luminance of the pixels included in each image region based on the correction gain (step S204). When the correction of all image regions is completed, the boundary processing unit 252 ends the process.

[0076] In the above description, the average luminance value of each image region is used, but the ratio of B and R components of RGB (Red Green Blue) may be used instead of the average luminance value. In this case, the boundary processing unit 252 calculates, for example, (total value of R / G) / (total value of B / G) for each image region. Then, similar to the case where luminance is used, the boundary processing unit 252 determines a reference region, calculates a correction gain for each image region, and corrects the color of each pixel based on the correction gain. This makes it possible to reduce color errors in multiple image regions with different white balances.

[0077] The method of matching the average luminance values ​​and the RGB ratios of a plurality of image regions is not limited to the above-mentioned method, and any other method may be used. The luminance and color may be corrected by the method described with reference to FIG.

[0078] Next, the effects of the image recognition support device according to the first embodiment will be described. In the conventional image recognition support device, the process of generating an image for display and the process of generating an image for image recognition are separated, and there are cases where an appropriate image cannot be displayed. In addition, in the conventional image recognition support device, the two processes are clearly separated, which causes a problem of an increase in the number of circuits and the area. On the other hand, an image recognition support device configured to display only an image for image recognition may display a visually inappropriate image. On the other hand, the image recognition support device 200 according to the first embodiment superimposes the adjusted images 420 to 42nmax based on α0 to αnmax and their respective normalization coefficients, and therefore can output an appropriate display image. In addition, since the two processes are not completely separated, the image recognition support device 200 can reduce the number of circuits and the area.

[0079] EMBODIMENT 2 The second embodiment is a modification of the first embodiment. The image recognition support device according to the first embodiment outputs a superimposed image in which the adjusted images 420 to 42nmax are superimposed as a display image. The image recognition support device according to the second embodiment outputs one of the adjusted images 420 to 42nmax as a display image.

[0080] Fig. 7 is an explanatory diagram showing a configuration of an image recognition system 1000a according to the second embodiment. Comparing Fig. 1 with Fig. 7, the image recognition support device 200 is replaced with an image recognition support device 200a. Also, the determination unit 24 and the image synthesis unit 25 of the image recognition support device 200 are replaced with a determination unit 24a and an image switching unit 25a of the image recognition support device 200a, respectively.

[0081] The determination unit 24a calculates information indicating the appropriateness of each adjusted image, and determines whether or not the adjusted image is appropriate as a display image. The determination unit 24a outputs the determination result to the image switching unit 25a. The determination unit 24a may also analyze the image quality of the adjusted image received from the image switching unit 25a, and output the analysis result to the image switching unit 25a.

[0082] The image switching unit 25a outputs one of the adjusted images 420 to 42nmax to the display device 400 based on the information received from the determination unit 24a.

[0083] An example of the configuration of the determination unit 24a and the image switching unit 25a will be described in more detail with reference to Fig. 8. The determination unit 24a includes a setting value analysis unit 241 and an image quality analysis unit 242. The image switching unit 25a includes a recognition result superimposition processing unit 253, a first selection unit 254, and a second selection unit 255. The function of the recognition result superimposition processing unit 253 has already been described.

[0084] The setting value analysis section 241 of the determination section 24a receives the setting values ​​from the setting section 23, and determines whether or not each of the adjusted images 421 to 42nmax is appropriate.

[0085] The setting value analysis unit 241 determines, for example, whether H1 is less than K1 for the adjusted image 421. As already explained, K1 is an arbitrary constant, and H1 is the absolute value of the deviation between the setting value and the adjustment reference value. As already explained, the adjustment reference value represents a predetermined optimum value of the adjustment value included in the adjustment value group. It can also be said that the determination unit 24a determines whether the difference value K1-H1 is greater than 0. The difference value corresponds to the above-mentioned α1.

[0086] Similarly, the setting value analysis unit 241 determines whether H2 is less than K2 for the adjusted image 422. The setting value analysis unit 241 determines whether Hn is less than Kn for the adjusted image 42n. The setting value analysis unit 241 determines whether Hnmax is less than Knmax for the adjusted image 42nmax. That is, the setting value analysis unit 241 determines whether each of the above-mentioned α2 to αnmax is equal to or greater than a threshold value. The setting value analysis unit 241 outputs information indicating the determination result as first selection information to the first selection unit 254 of the image switching unit 25a.

[0087] The image quality analysis section 242 of the determination section 24a analyzes the image quality of the candidate image 45 selected by the first selection section 254. The image quality analysis section 242 may further analyze the image quality of the currently displayed image 46 currently displayed on the display device 400. The image quality analysis section 242 outputs information indicating whether the candidate image 45 has high image quality or not to the second selection section 255 as second selection information. The image quality analysis section 242 may output information indicating which of the candidate image 45 and the currently displayed image 46 has higher image quality.

[0088] The image quality analysis unit 242 may, for example, measure the noise of each image and determine that an image with a large S / N value has high image quality. Specifically, the image quality analysis unit 242 measures the S / N of an image region with a small brightness level difference, that is, a relatively flat brightness level, compares the S / N with a standard value, and outputs the determination result to the second selection unit 255. In this case, the second selection unit 255 selects an image with an S / N greater than the standard value.

[0089] Note that an image region with a flat brightness level may be detected as an image region with small brightness variance when the adjusted image is divided into multiple image regions. Image regions with a large amount of movement, image regions with extremely low illuminance, image regions with high illuminance and the possibility of brightness saturation, etc. may not be able to accurately measure the S / N ratio, so these image regions may be excluded from detection targets. The S / N ratio may be measured using a method standardized by ISO-15739, for example.

[0090] Moreover, since images with low contrast tend to have low resolution, the image quality analysis unit 242 may measure the luminance histogram of each image and determine that an image with a variance (spread coefficient) within a desired range is of high image quality. The image quality analysis unit 242 may determine that an image with a spread coefficient equal to or greater than a threshold is of high image quality. However, the image quality analysis unit 242 may not determine that an image with an excessively large spread coefficient is of high image quality. The image quality analysis unit 242 may determine that an image with a spread coefficient close to an ideal spread coefficient is of high image quality. Moreover, an image with an excessively high average luminance value is extremely bright, and an image with an excessively low average luminance value is extremely dark. Therefore, the image quality analysis unit 242 may determine that an image with an average luminance value closer to a reference value is of high image quality. Moreover, the image quality analysis unit 242 may analyze the image quality of an image by taking into account both the spread coefficient and the average luminance value.

[0091] The first selection unit 254 of the image switching unit 25a outputs the adjusted image according to the first selection information to the second selection unit 255 as a candidate image 45. In addition, the first selection unit 254 outputs a currently displayed image 46 currently being output to the display device 400 to the second selection unit 255, separately from the candidate image 45.

[0092] The second selection unit 255 of the image switching unit 25a selects either the candidate image 45 or the currently displayed image 46 depending on the second selection information received from the image quality analysis unit 242, and outputs the selected image to the recognition result superposition processing unit 253.

[0093] The determination unit 24a may not include either the set value analysis unit 241 or the image quality analysis unit 242. It is also possible to select the display image 44 according to the analysis result by the set value analysis unit 241, or to select the display image 44 according to the analysis result by the image quality analysis unit 242. The image quality analysis unit 242 may analyze the image quality of the adjusted image 420. In this case, the image switching unit 25a may be configured to select the adjusted image 420 when the image quality of the adjusted image 420 is high, and to select one of the adjusted images 421 to 42nmax when the image quality of the adjusted image 420 is low.

[0094] Fig. 9 is a flowchart showing a first example of the operation of the image recognition support device 200a. Comparing Fig. 3 with Fig. 9, step S301 is added before step S101, and step S110 is replaced with steps S302 to S304.

[0095] In step S301, settings are made for the image switching unit 25a and the like so as to output the adjusted image 420, which has been subjected to standard adjustment, to the display device 400. As a result, after step S101, the adjusted image 420 is output to the display device 400.

[0096] In step S302, judgment unit 24a judges whether the absolute value H of the deviation for all image quality adjustment parameters adjusted before the previous time is less than a predetermined constant K. If the absolute value H for any image quality adjustment parameter is equal to or greater than K, step S111 is executed.

[0097] If H is less than K for all image quality adjustment parameters (YES in step S302), judgment unit 24a judges whether the absolute value H of the deviation for the currently adjusted image quality adjustment parameter is less than a predetermined constant K (step S303). If H is K or greater for the currently adjusted image quality adjustment parameter (NO in step S303), step S111 is executed.

[0098] If the absolute value H of the deviation for the currently adjusted image quality adjustment parameter is less than the predetermined constant K (YES in step S303), the image switching unit 25a outputs the currently adjusted image (step S304).

[0099] The image recognition support device 200a is configured not to output an adjusted image that has been adjusted to the extent that it becomes visually inappropriate. The image recognition support device 200a can output an adjusted image that has been adjusted within an appropriate range.

[0100] Fig. 10 is a flowchart showing a second example of the operation of the image recognition support device 200a. Comparing Fig. 3 with Fig. 10, step S301 is replaced with step S301a. In step S301a, the adjusted image 42nmax, which has been subjected to image quality adjustment for image recognition, is set to be output to the display device 400. As a result, after step S101, the adjusted image 42nmax is output to the display device 400. When the initial values ​​of the adjustment values ​​2111 to 211nmax are set so that the absolute values ​​H1 to Hnmax of the deviations become 0, the adjusted image 42nmax is an image similar to the adjusted image 420. Therefore, an image appropriate as a display image is displayed.

[0101] Fig. 11 is a flowchart showing a third example of the operation of the image recognition support device 200a. Comparing Fig. 3 with Fig. 11, step S305 is added after steps S105, S107, and S109.

[0102] In step S305, the determination unit 24a determines whether the adjusted image obtained by adjusting the image quality adjustment parameters is appropriate, and if the adjusted image is determined to be appropriate, the image switching unit 25a switches to displaying the adjusted image. This allows the displayed image to be updated every time the image quality adjustment parameters are adjusted. In the third example, when the image quality of the second image becomes higher than the image quality of the first image while adjusting any of the adjustment value groups, the second image at that time is output.

[0103] In addition, in the initial stage of image quality adjustment, the adjustment value may be significantly changed in step S105. In this case, since the adjusted image may not be suitable as a display image, the image recognition support device 200a may be configured to output the adjusted image 420.

[0104] Fig. 12 is a flowchart showing a fourth example of the operation of the image recognition support device 200a. Fig. 12 shows the flow of the operation for performing image quality analysis of the currently displayed image and the candidate image.

[0105] First, the image quality analysis unit 242 of the determination unit 24 analyzes the image quality of the currently displayed image 46 and calculates an evaluation value of the image quality of the currently displayed image 46 (step S401). Then, the image quality analysis unit 242 analyzes the image quality of the candidate image 45, which is an adjusted image in which the image quality adjustment parameters have been adjusted this time, and calculates an evaluation value of the image quality of the candidate image 45 (step S402). Then, the image quality analysis unit 242 determines whether the evaluation value of the image quality of the candidate image 45 is higher (step S403). If the evaluation value of the image quality of the candidate image 45 is equal to or lower than the evaluation value of the image quality of the currently displayed image 46 (NO in step S403), the image switching unit 25a continues to display the currently displayed image 46 (step S404). If the evaluation value of the candidate image 45 is higher (YES in step S403), the image switching unit 25a switches to displaying the candidate image 45.

[0106] The second embodiment can also achieve the same effects as the first embodiment.

[0107] EMBODIMENT 3 The third embodiment is a specific example of the first and second embodiments. The image recognition support device according to the third embodiment changes the adjustment order of the group of adjustment values ​​211 to 21nmax.

[0108] FIG. 13 is a block diagram showing an example of the configuration of an image recognition system 1000b according to the third embodiment. Comparing FIG. 7 with FIG. 13, the image recognition support device 200a is replaced with the image recognition support device 200b. The image quality adjustment unit 21 and the determination unit 24a of the image recognition support device 200 are replaced with the processing recombination unit 21b and the determination unit 24b, respectively. The processing recombination unit 21b includes a rearranged adjustment value group 311-31nmax obtained by rearranging the above-mentioned adjustment value group 211-21nmax. For example, the rearranged adjustment value group 31n represents an adjustment value group that is adjusted n-th among the adjustment value group 211-21nmax. The adjustment values ​​3111-311nmax are set to the rearranged adjustment value group 311-31nmax. The rearranged adjustment value group 311-31nmax outputs the processing 1 output image 61-processing nmax output image 6nmax, respectively. In addition, the image recognition support device 200b may output a superimposed image to the display device 400 as in embodiment 1, or may output any of the processing 1 output image 61 to the processing nmax output image 6nmax to the display device 400 as in embodiment 2.

[0109] The determination unit 24b determines the adjustment order of the above-mentioned adjustment value group 211-21nmax based on at least one of the first recombination information, the second recombination information, and the third recombination information. The adjustment order can also be said to be the processing order of a plurality of image processes. The first recombination information is based on pixel information (e.g., illuminance) of the captured image 41. The second recombination information is based on the set values ​​of the adjustment value group 211-21nmax. The third recombination information is based on the adjusted images 421-42nmax of the adjustment value group 211-21nmax. The determination unit 24b outputs recombination information 51 indicating the adjustment order to the processing recombination unit 21b.

[0110] The process recombination unit 21b is configured to be able to change the adjustment order of the group of adjustment values ​​211 to 21nmax. The process recombination unit 21b sequentially adjusts the group of adjustment values ​​211 to 21nmax based on the adjustment order determined by the determination unit 24b.

[0111] 14 is a block diagram showing the configuration of the process recombination unit 21b. The process recombination unit 21b includes adjustment value groups 211-21nmax, switches SWI1-SWInmax, and a switch SWO. When the switches SWI1-SWInmax are not distinguished from one another, they may be referred to as switches SWI. Note that switches SWI other than switches SWI1, SWI2, SWIn, and SWInmax are not shown in the figure.

[0112] An adjusted image 420 of the standard adjustment value group 210 is input to SWI1 to SWInmax.

[0113] To the switch SWI1, an adjusted image of an adjustment value group other than adjustment value group 211 is input. Similarly, to the switch SWI2, an adjusted image of an adjustment value group other than adjustment value group 212 is input. Similarly, to the switch SWIn, an adjusted image of an adjustment value group other than adjustment value group 21n is input. Similarly, to the switch SWInmax, an adjusted image of an adjustment value group other than adjustment value group 21nmax is input.

[0114] Moreover, recombination information 51 indicating the processing order determined by the determination unit 24b is input to the switches SWI1 to SWInmax.

[0115] The switch SWI1 outputs one of the input adjusted images to the adjustment value group 211 in accordance with the recombination information 51. Similarly, the switch SWI2 outputs one of the input adjusted images to the adjustment value group 212 in accordance with the recombination information 51. Similarly, the switch SWIn outputs one of the input adjusted images to the adjustment value group 21n in accordance with the recombination information 51. Similarly, the switch SWInmax outputs one of the input adjusted images to the adjustment value group 21nmax in accordance with the recombination information 51.

[0116] The adjustment value group 211 outputs the adjusted image to the switches SWI other than the switch SWI1 and the switch SWO. Similarly, the adjustment value group 212 outputs the adjusted image to the switches SWI other than the switch SWI2 and the switch SWO. Similarly, the adjustment value group 21n outputs the adjusted image to the switches SWI other than the switch SWIn and the switch SWO. Similarly, the adjustment value group SWnmax outputs the adjusted image to the switches SWI other than the switch SWInmax and the switch SWO.

[0117] Switch SWO outputs adjusted image 52 of the last adjusted adjustment value group to image recognition engine 300. After recombination processing based on recombination information 51 is performed, switch SWO outputs processing 1 output image 61 to processing nmax output image 6nmax to image switching unit 25a. Processing 1 output image 61 is the adjusted image of the first adjusted adjustment value group. Processing 2 output image 62 is the adjusted image of the second adjusted adjustment value group. Processing n output image 6n is the adjusted image of the nth adjusted adjustment value group. Processing nmax output image 6nmax is the adjusted image of the nmaxth adjusted adjustment value group.

[0118] An example of the configuration of the determination unit 24b will be described with reference to Fig. 15. The determination unit 24b includes a setting value analysis unit 241, an image quality analysis unit 242, an image information analysis unit 243, and a rearrangement order setting unit 244. Note that the determination unit 24b does not need to include all of the setting value analysis unit 241, the image quality analysis unit 242, and the image information analysis unit 243.

[0119] The setting value analysis unit 241 acquires the setting value of each adjustment value group from the setting unit 23. The setting value analysis unit 241 calculates the absolute value of the deviation α between the setting value and a predetermined optimal value, and compares the absolute value of the deviation with a threshold value. The threshold value indicates the allowable range of the absolute value of the deviation. The setting value analysis unit 241 determines, as a selection candidate image, an adjusted image of one or more image quality adjustment value groups whose absolute value of the deviation is within the allowable range. Then, the setting value analysis unit 241 acquires an adjustment order (second recombination information) by arranging the one or more image quality adjustment groups in descending order of the absolute value of the deviation. The setting value analysis unit 241 outputs the second recombination information to the recombination order setting unit 244. Note that the setting value analysis unit 241 may further output the analysis result of the setting value to the image switching unit 25a, as in the second embodiment.

[0120] The image quality analysis unit 242 analyzes the image quality of the adjusted image for each adjustment value group, and outputs third rearrangement information based on the analysis result to the rearrangement order setting unit 244. The image quality analysis unit 242 analyzes, for example, the amount of noise increased by adjusting each adjustment value group. Then, the image quality analysis unit 242 determines the processing order of the adjustment value groups (third rearrangement information) such that the adjustment value group with the smaller increase in noise is placed earlier in the order. Note that the image quality analysis unit 242 may further output the image quality analysis result to the selection unit 256 of the image switching unit 25a.

[0121] The image information analysis section 243 determines the priority of the adjustment order of the adjustment value group from the image information of the photographed image 41. The image information analysis section 243 outputs the adjustment order based on the priority to the rearrangement order setting section 244 as first rearrangement information.

[0122] The recombination order setting unit 244 outputs recombination information 51 (adjustment order) based on at least one of the first recombination information, the second recombination information, and the third recombination information to the processing recombination unit 21b. The recombination order setting unit 244 can also determine the recombination information 51 by considering all of the first recombination information, the second recombination information, and the third recombination information. The processing recombination unit 21b switches the switch SWI and the switch SWO according to the recombination information 51. The recombination order setting unit 244 may further output the recombination information 51 to the image switching unit 25a. The image switching unit 25a may determine an image to be output as the currently displayed image 46 based on the analysis result of the input setting value, the recombination information 51, and the information on the analysis result of the image quality. The selection unit 256 of the image switching unit 25a determines which adjustment value group each of the processing 1 output image 61 to the processing nmax output image 6nmax is an adjusted image for, based on the recombination information 51. The selection unit 256 can select an image to be output to the current display image 46 from among the processing 1 output image 61 to the processing nmax output image 6nmax, based on the analysis results of the setting values ​​and the image quality.

[0123] An example of the adjustment order set by the rearrangement order setting unit 244 will be described with reference to FIG. 14. For example, it is assumed that the deviations between the set values ​​and the predetermined optimal values ​​are α1=0.7, α2=0.6, αn=-1.5, αnmax=0.2. When the processing order is determined based on the set values, the processing order is, for example, the adjustment value group 211, the adjustment value group 212, the adjustment value group 21nmax, and the adjustment value group 21n. In this case, for example, the switch SWInmax outputs the adjusted image of the adjustment value group 212 to the adjustment value group 21nmax. Then, the switch SWIn outputs the adjusted image of the adjustment value group 21nmax to the adjustment value group 21n. The switch SWO outputs the adjusted image of the adjustment value group 21n to the image recognition engine 300. The switch SWO outputs the adjusted image of the adjustment value group 211 as the processing 1 output image 61. The switch SWO outputs the adjusted image of the adjustment value group 212 as a processing 2 output image 62. The switch SWO outputs the adjusted image of the adjustment value group 21nmax as a processing n output image 6n. The switch SWO outputs the adjusted image of the adjustment value group 21n as a processing nmax output image 6nmax.

[0124] The image quality analysis unit 242 will be described in detail with reference to FIG. 15. The image quality analysis unit 242 measures the S / N (signal / noise) value of the input image and the output image of each adjustment process. The image quality analysis unit 242 can determine that an input image or an output image with a large S / N is an image with little noise, that is, an image that is visually appropriate, and generates the third recombination information according to the determination result. The image quality analysis unit 242 may select an area with a relatively flat luminance level and measure the S / N of the selected area. The signal of the S / N usually means the average value of all pixel values ​​in the area. When an area with little change in pixel value over time is selected, the image quality analysis unit 242 can average the pixel values ​​in the area in the time axis direction by integrating the pixel values ​​in the area in the time axis direction, and calculate a signal that is a more accurate reference value. Note that the method of measuring noise is not limited to the above method.

[0125] Next, the image information analysis unit 243 will be described in detail. The image information analysis unit 243 can determine whether the time period during which the captured image was taken is nighttime or daytime from the image captured by the camera 100, that is, pixel information (pixel level) of the RawData. The image information analysis unit 243 can also determine whether the capture environment in which the captured image was taken is indoors (e.g., inside a tunnel) or outdoors. The image information analysis unit 243 may use position information such as a Global Positioning System (GPS) to accurately determine that the capture environment is inside a tunnel. The image information analysis unit 243 may use the time of day as auxiliary information when determining the time period.

[0126] When the illuminance of the shooting environment is low (for example, the illuminance is below the low illuminance threshold), the image information analysis unit 243 predicts that the captured image will have a large amount of noise components, and places the processing by the noise reduction module in the front stage. This makes it possible to prevent noise from increasing in the processing in the rear stage. The image information analysis unit 243 determines that contrast enhancement processing in low illuminance induces an increase in noise, which is visually disadvantageous, and places the contrast enhancement processing in the rear stage.

[0127] When the illuminance of the captured image is high (for example, the illuminance is equal to or higher than a high illuminance threshold), the image information analysis unit 243 predicts that the captured image has few noise components. Then, the image information analysis unit 243 arranges the contrast enhancement process in an earlier stage than other processes.

[0128] When there are many saturated areas in the captured image and there is a concern that the recognition rate of the classification target may decrease, the image information analysis unit 243 places the processing by the brightness adjustment block at an earlier stage than other processing.

[0129] Fig. 16 is a flowchart showing a first example of the operation of the image recognition support device 200b. Comparing Fig. 11 with Fig. 16, step S401 is added before step S101. If YES in step S108, step S402 is performed. If NO in step S111, step S403 is performed. Note that step S305 may be performed before step S111, as in the second embodiment.

[0130] In step S401, the image recognition support device 200b determines the adjustment order of the adjustment value group. Then, the image recognition support device 200b outputs the image after adjustment of the standard adjustment value group to the display device. The process recombination unit 21b of the image recognition support device 200b may determine that the adjustment of the adjustment value group 211 to 21nmax is performed in the 1st to nmaxth order, respectively.

[0131] In step S402, the setting value analysis unit 241 of the image recognition support device 200b stores in the storage device the absolute value α of the deviation of the most recently adjusted image quality adjustment parameter.

[0132] In step S403, the determination unit 24b of the image recognition support device 200b determines the adjustment order of the adjustment value group according to the α stored in the storage device. Then, the process recombination unit 21b of the image recognition support device 200 changes the adjustment order of the adjustment value group.

[0133] 17 is a flowchart showing a second example of the operation of image recognition support device 200b. In the second example, step S403 is performed following step S112. In this case, each time adjustment of image quality adjustment parameters is completed, determination unit 24b determines the adjustment order of the image quality adjustment parameters, and process recombination unit 21b changes the adjustment order of the image quality parameters.

[0134] The image recognition support device according to the third embodiment adjusts a plurality of image quality adjustment parameters in an appropriate order, and therefore may be able to display a more appropriate display image.

[0135] While the present invention has been described above in accordance with the above-described embodiment, the present invention is not limited to the configuration of the above-described embodiment, and naturally includes various modifications, alterations, and combinations that may be made by a person skilled in the art within the scope of the invention as defined in the claims of the present application.

[0136] In the above embodiment, the present disclosure has been described as a hardware configuration, but is not limited to this. Any process in the present disclosure can be realized by causing a CPU to execute a computer program.

[0137] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals. [Explanation of symbols]

[0138] 1000, 1000a, 1000b Image Recognition System 100 Cameras 200, 200a, 200b Image recognition support device 300 Image Recognition Engine 400 display device 21 Image quality adjustment section 210 Standard adjustment value group 211~21nmax adjustment value group 311~31nmax Adjusted value group after recombination 2110~211nmax, 3111~311nmax Adjustment value 22 Acquisition Department 23 Setting section 24, 24a, 24b Judgment section 241 Setting value analysis unit 242 Image quality analysis section 25 Image synthesis unit 251 Arithmetic section 252 Boundary Processing Section 253 Recognition result overlay processing unit 254 First Selection Section 255 Second Selection Section 256 Selection 41 Images 420~42nmax, 52 adjusted images 43 Recognition results 44 Display images 45 candidate images 46 Current image 21b Processing and Recombination Department SWI, SWI1~SWInmax, SWO Switch 51 Recombination Information 61 Processing 1 output image 62 Processing 2 output image 6n Processing n Output Images 6nmax processing nmax output image 243 Image Information Analysis Unit 244 Recombination Order Setting Section

Claims

1. a first processing unit that performs a first image processing on the captured image to generate a first image; a second processing unit that performs a second image processing for image recognition on the first image to generate a second image; A recognition processing unit that performs object recognition processing based on the second image; a determination unit that determines whether the first image and the second image are appropriate as display images; a display image output unit that outputs the display image using at least one of the first image and the second image based on the determination by the determination unit; An image recognition assistance device comprising:

2. In the second image processing, a plurality of different image quality adjustment parameters are adjusted in sequence; a setting value for each image quality adjustment parameter is determined based on a recognition result by the recognition processing unit; The determination unit calculates a difference value by subtracting an absolute value of a deviation between the set value and a predetermined optimum value from an arbitrary constant. The image recognition support device according to claim 1 .

3. the determination unit calculates the difference value for an adjusted image in which each image quality adjustment parameter has been adjusted; The display image output unit outputs, as the display image, a superimposed image obtained by superimposing the first image and the adjusted image on each other based on the difference value. The image recognition support device according to claim 2 .

4. The determination unit is a rearrangement order determination unit that determines a processing order of each adjustment process of the second image processing based on at least one of first rearrangement information based on pixel information of the captured image, second rearrangement information based on a setting value determined from a recognition result in the recognition processing unit, and third rearrangement information based on an adjusted image of each adjustment process of the second image processing, In the second image processing, a plurality of mutually different image quality adjustment parameters are adjusted in the processing order determined by the rearrangement order determination unit, and the second image is output to the recognition processing unit and also to the display image output unit. The image recognition support device according to claim 1 .

5. The second recombination information is obtained by comparing an absolute value of a deviation between the set value and a predetermined optimum value with a threshold value indicating an allowable range of the absolute value of the deviation, and sorting one or more adjustment processes whose absolute value of the deviation is within the allowable range in descending order of the absolute value of the deviation. The image recognition support device according to claim 4.

6. performing a first image processing on the captured image to generate a first image; performing a second image processing for image recognition on the first image to generate a second image; performing an object recognition process based on the second image; determining whether the first image and the second image are appropriate as a display image; outputting the display image using at least one of the first image and the second image based on the determination in the determining step; An image recognition assistance method comprising:

7. performing a first image processing on the captured image to generate a first image; performing a second image processing for image recognition on the first image to generate a second image; performing an object recognition process based on the second image; determining whether the first image and the second image are appropriate as a display image; outputting the display image using at least one of the first image and the second image based on the determination in the determining step; A program that causes a computer to execute the following.

Images