IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND IMAGE PROCESSING SYSTEM

Abstract

This image processing device comprises: an acquisition unit that acquires a first image captured by an imaging device mounted on a mobile body; and a generation unit that generates a second image in which the brightness of a second region lower than a first region of the first image in a direction perpendicular to a moving surface of the mobile body is corrected on the basis of the brightness of the first region.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, a program, and an image processing system. [Background technology]

[0002] Conventionally, there is known a technology for displaying images of the rear and surroundings of a vehicle on a display unit of a navigation device, etc., based on images captured by an on-board camera. These are also called, for example, a back monitor and an around view monitor, etc.

[0003] Conventionally, there are also known techniques for performing driving assistance and automatic driving based on images captured by an in-vehicle camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-055437 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, when a light source is present in a part of the camera's angle of view, for example when a vehicle travels under a street light at night, halation (flare, ghost) may occur.

[0006] According to one aspect, an object of the present invention is to provide a technique capable of appropriately correcting the brightness of each frame. [Means for solving the problem]

[0007] In one proposal, an image processing device is mounted on a moving object. a first area set based on a mounting position of the imaging device on the moving body; an acquisition unit for acquiring a first image, and a forward image in a direction perpendicular to the moving plane of the moving object; Record number The brightness of a second region below the first region is corrected based on the brightness of the first region. , the brightness of the first region is not corrected. A generation unit that generates a second image; an output unit that outputs the second image; has. Effect of the Invention

[0008] According to one aspect, the brightness of each frame can be appropriately corrected. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating a configuration of a control system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of a moving body according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of the hardware configuration of an image processing device and a control device according to an embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a configuration of an image processing device and a control device according to an embodiment. [Diagram 5] 5 is a flowchart illustrating an example of an image correction process performed by the image processing device according to the embodiment. [Figure 6] 13A to 13C are diagrams illustrating an example of a process for calculating a target brightness value of a currently processed frame according to the embodiment. [Figure 7] 11A to 11C are diagrams illustrating an example of a process for adjusting a correction amount according to pixel brightness according to an embodiment. [Figure 8] 5A and 5B are diagrams illustrating an example of a halation detection region according to the embodiment. [Figure 9] FIG. 4 is a diagram illustrating an example of an output image according to the embodiment. [Figure 10A] FIG. 1 is a diagram illustrating an example of a configuration of an imaging device according to an embodiment. [Figure 10B] FIG. 1 is a diagram illustrating an example of a configuration of an imaging device according to an embodiment. [Figure 10C] FIG. 1 is a diagram illustrating an example of a configuration of an imaging device according to an embodiment. [Figure 11]FIG. 1 is a diagram illustrating an example in which an image processing device according to an embodiment is realized by an analog circuit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] <Overall composition> Fig. 1 is a diagram illustrating the configuration of a control system 500 according to an embodiment. In the example of Fig. 1, the control system 500 includes a moving object 1 and a server 50. The number of moving objects 1 and servers 50 is not limited to the example of Fig. 1.

[0012] The mobile object 1 and the server 50 communicate via a network such as a mobile phone network such as 5G (5th Generation, 5th generation mobile communication system), 4G, LTE (Long Term Evolution), or 3G, a wireless LAN (Local Area Network), or the Internet.

[0013] The moving body 1 is, for example, a moving machine such as a vehicle that runs on land using wheels, a robot that moves using legs or the like, an aircraft, an unmanned aerial vehicle (drone), etc. Note that the vehicle includes, for example, an automobile, a motorcycle (motorbike), a robot that moves using wheels, a railroad car that runs on a railroad, etc. Note that the automobile also includes automobiles that run on roads, streetcars, construction vehicles used for construction purposes, military vehicles, industrial vehicles for loading and transporting goods, agricultural vehicles, etc.

[0014] The server 50 provides the moving body 1 with various services necessary for the automatic driving of the moving body 1, for example.

[0015] <Example of imaging device placement> Fig. 1 shows the appearance of a moving object 1, which is an automobile, as viewed from directly above. In the example of Fig. 1, the moving object 1 has an imaging device 12A, an imaging device 12B, an imaging device 12C, an imaging device 12D, and an imaging device 12E (hereinafter, when there is no need to distinguish between them, they will be simply referred to as "imaging devices 12").

[0016] The imaging device 12 is a device that captures an image. The imaging device 12 may be, for example, a camera.

[0017] Imaging device 12A is an imaging device (rear camera, rear camera, back view camera) that captures the rear as seen from moving body 1 (the opposite direction to the normal traveling direction). Imaging device 12B is an imaging device (left camera) that captures the left as seen from moving body 1. Imaging device 12C is an imaging device (right camera) that captures the right as seen from moving body 1. Imaging devices 12D and 12E are imaging devices (front cameras) that capture the front as seen from moving body 1 (the normal traveling direction).

[0018] The imaging device 12 may be, for example, an imaging device for capturing images for an advanced driver-assistance system (ADAS) that assists the driver in driving operations or an image capturing device for automatic driving. The imaging device 12 may also be, for example, a camera that captures images for an omnidirectional monitor (around view, panoramic view, multi-view, top view) that generates an image of the moving body 1 as if seen from directly above. The imaging device 12 may also be, for example, an imaging device for a drive recorder that records images of the outside of the moving body 1.

[0019] The imaging device 12A may be, for example, a camera that captures an image to be displayed on a room mirror (back mirror) monitor. Also, the imaging device 12A may be, for example, a camera that captures an image to be displayed on the screen of the navigation device 18 when the moving object 1 moves backward (backing up).

[0020] The image capturing device 12B may be, for example, a camera that captures an image to be displayed on the left side mirror monitor, and the image capturing device 12C may be, for example, a camera that captures an image to be displayed on the right side mirror monitor.

[0021] 1, the imaging device 12D is provided on the front surface (the leading edge surface when the moving body 1 moves forward) of the moving body 1. Also, the imaging device 12E is provided inside the vehicle interior, that is, inside the windshield of the moving body 1. In this case, the imaging device 12E may be provided, for example, near a rearview mirror (room mirror, back mirror) inside the vehicle interior, on the dashboard, or the like.

[0022] The imaging device 12D and the imaging device 12E that capture images of the front (normal traveling direction) as seen from the moving body 1 may be stereo cameras having multiple cameras. Also, the moving body 1 may have only one of the imaging device 12D and the imaging device 12E.

[0023] <Configuration of moving object 1> Fig. 2 is a diagram illustrating an example of the configuration of a moving object 1 according to an embodiment. In the example of Fig. 2, the moving object 1 includes an image processing device 10, a control device 11, an imaging device 12, an ECU 13, a wireless communication device 14, a sensor 15, a drive device 16, a lamp device 17, a navigation device 18, and a display device 19.

[0024] These components are connected to each other via an internal network (eg, an in-vehicle network) such as a Controller Area Network (CAN) or Ethernet (registered trademark).

[0025] The image processing device 10 generates an image whose brightness has been corrected so as to reduce hunting, based on an image (frame) captured at each time point by the imaging device 12. The image processing device 10 may also record the generated image in an external or internal recording device.

[0026] The control device 11 is a computer (information processing device) that controls each part of the moving object 1. The control device 11 recognizes objects outside the moving object 1 based on the image generated by the image processing device 10. The objects may include, for example, other vehicles, pedestrians, bicycles, white lines, side walls of the road, and obstacles.

[0027] Furthermore, the control device 11 tracks the recognized object based on the images at each point in time generated by the image processing device 10. The control device 11 controls an ECU (Electronic Control Unit) 13 and the like of the moving object 1 based on the detected objects (the recognized object and the tracked object), thereby controlling the movement and the like of the moving object 1.

[0028] The control device 11 may realize any level of autonomous driving, for example, from level 0, in which a driver (user, driver, passenger) operates the main control system (acceleration, steering, braking, etc.), to level 5, in which the vehicle is unmanned, by controlling the movement of the mobile body 1.

[0029] The ECU 13 is a device that controls each device of the mobile object 1. The ECU 13 may have multiple ECUs. The wireless communication device 14 communicates with devices outside the mobile object 1, such as a server 50 and a server on the Internet, by wireless communication such as a mobile phone network.

[0030] The sensor 15 is a sensor that detects various types of information. The sensor 15 may include, for example, a position sensor that acquires current position information of the mobile object 1. Note that the position sensor may be, for example, a sensor that uses a satellite positioning system such as a Global Positioning System (GPS).

[0031] The sensor 15 may also include a speed sensor that detects the speed of the moving body 1. The speed sensor may be, for example, a sensor that measures the number of rotations of an axle of a wheel. The sensor 15 may also include an acceleration sensor that detects the acceleration of the moving body 1. The sensor 15 may also include a yaw axis angular velocity sensor that detects the yaw axis angular velocity (yaw rate) of the moving body 1.

[0032] The sensor 15 may also include an operation sensor that detects the amount of operation of the mobile object 1 by the driver and the control device 11. The operation sensor may include, for example, an accelerator sensor that detects the amount of depression of an accelerator pedal, a steering sensor that detects the rotation angle of a steering wheel, a brake sensor that detects the amount of depression of a brake pedal, a shift position sensor that detects the gear position, and the like.

[0033] The driving device 16 is various devices for moving the moving body 1. The driving device 16 may include, for example, an engine, a steering device, and a braking device (brake).

[0034] The lamp device 17 is various lighting devices mounted on the mobile body 1. The lamp device 17 may include, for example, a headlight (headlamp, headlight), a turn signal (blinker) lamp for indicating the direction to the surroundings when turning right or left or changing course (lane change), a backlight provided at the rear of the mobile body 1 and turned on when the gear is in the reverse range, a brake lamp, and the like.

[0035] The navigation device 18 is a device (car navigation) that guides the route to a destination by voice and display. Map information may be recorded in the navigation device 18. The navigation device 18 may also transmit information on the current position of the mobile object 1 to an external server that provides a car navigation service, and obtain map information on the surroundings of the mobile object 1 from the external server. Note that the map information may include, for example, information on nodes indicating nodes such as intersections, and links that are road sections between the nodes.

[0036] The display device 19 displays images and the like generated by the image processing device 10. For example, the display device 19 may display an image of the rear of the mobile object 1 captured when the mobile object 1 moves backward (backing up). Note that the display device 19 may display an image output from the navigation device 18, for example.

[0037] <Computer hardware configuration> 3 is a diagram illustrating an example of the hardware configuration of the image processing device 10 and the control device 11 according to the embodiment. In the following, the image processing device 10 will be described as an example. The hardware configuration of the control device 11 may be the same as that of the image processing device 10.

[0038] In the example of FIG. 3, the image processing apparatus 10 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, and an interface device 1005, all of which are connected to one another via a bus B.

[0039] An information processing program for realizing processing in the image processing device 10 is provided by a recording medium 1001. When the recording medium 1001 on which the information processing program is recorded is set in the drive device 1000, the information processing program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the information processing program does not necessarily have to be installed from the recording medium 1001, and may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed information processing program as well as necessary files, data, and the like.

[0040] When an instruction to start a program is received, the memory device 1003 reads out the program from the auxiliary storage device 1002 and stores it. The CPU 1004 executes processing in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network.

[0041] An example of the recording medium 1001 is a portable recording medium such as a CD-ROM, a DVD disk, or a USB memory. An example of the auxiliary storage device 1002 is a hard disk drive (HDD) or a flash memory. Both the recording medium 1001 and the auxiliary storage device 1002 correspond to computer-readable recording media.

[0042] The image processing device 10 may be realized by, for example, an analog circuit, or an integrated circuit such as an Application Specific Integrated Circuit (ASIC) or a Field-Programmable Gate Array (FPGA).

[0043] <Configuration of image processing device 10 and control device 11> Next, the configurations of the image processing device 10 and the control device 11 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configurations of the image processing device 10 and the control device 11 according to the embodiment.

[0044] <Image processing device 10> The image processing device 10 includes an acquisition unit 101, a determination unit 102, a generation unit 103, and an output unit 104. Each of these units may be realized by cooperation between one or more programs installed in the image processing device 10 and hardware such as a CPU 1004 of the image processing device 10.

[0045] The acquisition unit 101 acquires data from another device. For example, the acquisition unit 101 acquires an image captured by the imaging device 12 from the imaging device 12.

[0046] The determination unit 102 determines the amount of correction for the brightness of the image based on the brightness of a predetermined region of the image acquired by the acquisition unit 101 (a region for detecting halation; an example of a "first region").

[0047] The generation unit 103 generates an image (corrected image, output image) in which the brightness of the area below the halation detection area in the direction perpendicular to the moving plane of the moving body 1 is corrected according to the correction amount determined by the determination unit 102.

[0048] The output unit 104 outputs the image generated by the generation unit 103 to the display device 19, the control device 11, and the like.

[0049] ≪Control device 11≫ The control device 11 includes a storage unit 111, a recognition unit 112, a tracking unit 113, and a control unit 114. Each of these units may be realized by cooperation between one or more programs installed in the control device 11 and hardware such as a CPU of the control device 11.

[0050] The storage unit 111 stores various types of information. For example, the storage unit 111 stores a trained model distributed by the server 50.

[0051] The recognition unit 112 recognizes an object captured in an image based on the trained model stored in the storage unit 111 and the image output by the image processing device 10. The recognition unit 112 may recognize, for example, the type of the object and its relative position (distance) from the moving object 1. Note that the recognition unit 112 may classify the object into types such as vehicles, motorcycles, bicycles, humans, and others.

[0052] The tracking unit 113 tracks the object recognized by the recognition unit 112 based on the image output by the image processing device 10 at each time point, across the time points.

[0053] The control unit 114 controls the moving object 1 based on the distance between the moving object 1 and each object being tracked by the tracking unit 113 .

[0054] <Processing> Next, the image correction process by the image processing device 10 will be described with reference to Fig. 5 to Fig. 9. Fig. 5 is a flowchart showing an example of the image correction process by the image processing device 10 according to the embodiment. Fig. 6 is a diagram explaining an example of the process of calculating a target value of brightness of a frame to be processed currently according to the embodiment. Fig. 7 is a diagram explaining an example of the process of adjusting the correction amount according to the brightness of a pixel according to the embodiment. Fig. 8 is a diagram explaining an example of a halation detection region according to the embodiment. Fig. 9 is a diagram explaining an example of an output image according to the embodiment.

[0055] The following process may be executed, for example, every time an image (frame) is captured by the imaging device 12. In this case, for example, when the imaging device 12 captures images (frames) 60 times per second (60 fps), the following process is executed 60 times per second.

[0056] In step S1, the acquisition unit 101 of the image processing device 10 acquires an image (hereinafter also referred to as "the current processing target frame") captured by the imaging device 12. Here, the image processing device 10 acquires an image with a screen resolution of N1×M, where N1 is the number of horizontal pixels and M is the number of vertical pixels. In this case, the image processing device 10 may acquire an image with a screen resolution of 1920×1200, for example, WUXGA (Wide Ultra-XGA).

[0057] Next, the determination unit 102 of the image processing device 10 calculates the brightness (luminance) of a predetermined area (halation detection area) in the image (step S2). The halation detection area is an area where halation occurs when a light source such as a street light is reflected in the halation detection area. The halation detection area may be set in advance when the image processing device 10 is mounted on the moving body 1 and shipped. The halation detection area may be set according to the type (for example, vehicle type) of the moving body 1. In this way, for example, when halation (flare, ghost) occurs due to re-reflection on the inner surface of the windshield of the moving body 1, the halation detection area can be set according to the shape and material of the windshield for each vehicle type, the mounting position of the imaging device 12E, etc. In the example of FIG. 8 and FIG. 9, the halation detection area 802 is set in the upper area in the vertical direction with respect to the moving surface of the moving body 1. The vertical pixel count of the halation detection area 802 may be, for example, a predetermined ratio (for example, 5%) of the vertical pixel count of the input image. Furthermore, the number of horizontal pixels of the halation detection region 802 may be the same as the number of horizontal pixels of the input image, for example.

[0058] Here, the image processing device 10 may calculate a representative value of the values ​​of each pixel included in the halation detection region as the brightness of the halation detection region. In this case, the image processing device 10 may set the average value, the median value, or the most frequent value (the peak of the frequency of occurrence in a histogram of pixel values) as the representative value. Note that the value of each pixel may be, for example, the sum of the values ​​of each bit of RGB.

[0059] The image processing device 10 may also calculate the maximum value of the values ​​of the pixels included in the halation detection region as the brightness of the halation detection region, or may calculate the sum of the values ​​of the pixels included in the halation detection region as the brightness of the halation detection region.

[0060] Next, the determination unit 102 of the image processing device 10 determines the amount of brightness correction for the current processing target frame based on the brightness of the halation detection region of the current processing target frame (step S3). Here, the image processing device 10 may determine the amount of brightness correction for the current processing target frame so as to reduce the amount of brightness variation between each previously acquired frame and the current processing target frame.

[0061] In this case, the image processing device 10 may calculate a target brightness value for the current processing target frame using a predetermined filter function based on the brightness of the halation detection region in each frame acquired within a predetermined period up to the present (for example, the most recent one second).Then, the image processing device 10 may set the ratio between the target value and the brightness of the halation detection region of the current processing target frame as a correction coefficient.

[0062] In this case, the image processing device 10 may use a finite impulse response (FIR) filter as the predetermined filter function to calculate a moving average of brightness for the currently processed frame. The image processing device 10 may set the moving average value for the currently processed frame as a target value and calculate the ratio between the target value and the brightness of the halation detection region in the currently processed frame as a correction coefficient. The image processing device 10 may also use an infinite impulse response (IIR) filter as the predetermined filter function.

[0063] 6 shows a transition 601 of brightness of the halation detection region in a frame acquired (photographed) at each time point, and a target value (e.g., moving average) 602 calculated using a predetermined filter function. Also shown is a time point (current time) 611 when the current processing target frame was acquired (photographed), and a correction amount 612 of brightness for the current processing target frame. The correction amount 612 is the difference between the original brightness of the current processing target frame and the target value for the current processing target frame.

[0064] Furthermore, the image processing device 10 may adjust the value of a correction coefficient for a target value for each pixel in accordance with the brightness of the pixel in the currently processed frame, as shown in Fig. 7. In the example of Fig. 7, the image processing device 10 does not correct pixel values ​​whose brightness is less than a threshold value 701, and determines an adjusted correction coefficient 703 for pixel values ​​whose brightness is equal to or greater than the threshold value 701 such that the correction coefficient increases as the brightness increases. This makes it possible to reduce the amount of correction of brightness in dark areas in the currently processed frame, for example.

[0065] 7, the correction coefficient for pixel values ​​whose brightness is less than the threshold value 701 is 1. The correction coefficient for a brightness equal to that of the halation detection region is set to the ratio between the target value 711 and the brightness of the halation detection region of the current processing target frame.

[0066] Furthermore, the image processing device 10 may determine the value of the correction coefficient for each region (an example of the "third region") included in the corrected image of the currently processed frame according to the distance from the halation detection region. In this case, the image processing device 10 may determine the correction coefficient so that the correction coefficient becomes larger as the pixel distance from the halation detection region becomes closer (the number of pixels between the halation detection region becomes smaller). Thereby, for example, the amount of brightness correction can be reduced for regions of the currently processed frame that are far from the halation detection region and are considered to be less susceptible to the effects of halation.

[0067] In this case, the image processing device 10 may divide an area included in the corrected image of the current frame to be processed into a predetermined number of areas (for example, three) in the horizontal direction, and set the correction coefficient value for the bottom area to 1 (no correction).Then, for the intermediate area, the correction coefficient value may be closer to 1 than that for the top area.

[0068] Next, the generating unit 103 of the image processing device 10 generates an image (corrected image, output image) whose brightness has been corrected according to the correction amount determined by the determining unit 102, based on the currently processed frame (step S4). Here, the image processing device 10 may generate the corrected image by updating the pixel value of each pixel in an area included in the corrected image, of the currently processed frame, with a value obtained by multiplying the pixel value by a correction coefficient.

[0069] The image processing device 10 may perform correction only on a rectangular (rectangular) region not including the halation detection region in the current processing target frame, and generate a corrected image including only the region. This can improve the processing speed because correction processing is not performed on each pixel of the halation detection region. In this case, as shown in FIG. 8, the image processing device 10 corrects an N2×M region 803 with N2 pixels in the horizontal direction and M pixels in the vertical direction, excluding the halation detection region 802, out of the entire region 801 of the current processing target frame (input image) having a screen resolution of N1×M, and generates the corrected region 803 as a corrected image (output image). In this case, the image processing device 10 may generate an image with a screen resolution of, for example, FHD (Full-HD) of 1920×1080.

[0070] As shown in FIG. 8, the area 803 included in the corrected image of the frame currently being processed is located below the halation detection area in the vertical direction (vertical direction if the moving body 1 is a vehicle, etc.) relative to the moving surface (e.g., the road surface) of the moving body 1.

[0071] Next, the output unit 104 of the image processing device 10 outputs the image generated by the generation unit 103 (step S5). In the example of FIG. 9, since a street lamp 902 exists diagonally above the moving object 1 when the moving object 1 runs on the road 901, the street lamp 902 is reflected in the halation detection area 802 of the image captured by the imaging device 12. In this case, the area 803 is corrected by the above-mentioned processing from step S1 to step S5, and is output as a corrected image. Note that, in the entire area 801 of FIG. 8, there are areas where the halation detection area 802 and the area 803 are not arranged, but such areas may not be provided. Also, the ends of the halation detection area 802 and the area 803 may be arranged to the ends of the entire area 801, or the halation detection area 802 and the area 803 may be arranged so that there is no space between them.

[0072] (Regarding the imaging device 12) Next, an example of the configuration of an imaging device according to an embodiment will be described with reference to Figures 10A to 10C. Figures 10A to 10C are diagrams illustrating an example of the configuration of an imaging device according to an embodiment.

[0073] The imaging device 12 may have a mechanism for diffusing light from a light source such as a street lamp reflected in the halation detection area to capture the image. As a result, even if the pixel value of a pixel capturing light from a light source such as a street lamp exceeds the dynamic range (the range of light that can be exposed) in the absence of the mechanism, the light is diffused by the mechanism, so that each pixel value can be made below the dynamic range. Therefore, more appropriate correction can be achieved by the processes from step S1 to step S5 in FIG. 5 described above.

[0074] 10A, the imaging device 12 may have a diffusion plate 124 in front of a portion of the sensor surface of the image sensor 122 that converts the light incident from the lens 121 into an electric signal, the portion capturing the image of the halation detection region 123. The diffusion plate 124 is a member that diffuses light, and may be, for example, a substrate such as glass, acrylic, or film having an uneven surface.

[0075] In the example of FIG. 10A, the light incident from the lens 121 is upside down by the lens 121, so that the portion capturing the image of the halation detection region 123 is located below the sensor surface of the image sensor (imaging element) 122.

[0076] 10B, the imaging device 12 may have a diffusion plate 125 in front of the lens 121 at a position through which light incident from the lens 121 to the halation detection region 123 passes. The diffusion plate 125 is a member that diffuses light, and may be, for example, a substrate such as glass, acrylic, or film having an uneven surface.

[0077] Furthermore, as shown in FIG. 10C, the imaging device 12 may have a wide-angle lens 126 in front of the lens 121 for collecting light incident on the halation detection region 123 from the lens 121 at a wide angle.

[0078] The imaging device 12 may be a general imaging device that does not have the above-mentioned light diffusing glass, etc. In this case, the moving body 1 may use, as the imaging device 12, an imaging device having an image sensor with a high dynamic range (for example, 24 bits).

[0079] In addition, the image processing device 10 may capture multiple frames while changing the exposure using the imaging device 12 having an image sensor with a low dynamic range (e.g., 12 bit), and generate a frame with a high dynamic range as the input image by combining the frames.

[0080] (Example of implementation using analog circuits) At least a part of the processing of each functional unit of the image processing device 10 may be executed by, for example, an analog circuit. Fig. 11 is a diagram for explaining an example of the case where the image processing device 10 according to the embodiment is realized by an analog circuit. In the example of Fig. 11, the image processing device 10 has a wiring 1101, an amplifier circuit 1101A, a decision circuit 1111, a wiring 1102, an amplifier circuit 1102A, a wiring 1103, and a wiring 1104. In addition, the decision circuit 1111 has a differentiation circuit 1112 and an integration circuit 1113.

[0081] 11, a signal from an image sensor that captures an area 123 for halation detection on the sensor surface of an image sensor 122 is input to a decision circuit 1111 via a wiring 1101 and an amplifier circuit 1101A (e.g., a line amplifier). In the decision circuit 1111, a differentiation circuit 1112 detects a change in brightness over time, and an integration circuit 1113 (low-pass filter) blocks low-frequency components of the change in brightness.

[0082] In addition, a signal from an imaging element that captures an area (e.g., area 803 in FIG. 8) other than the halation detection area 123 on the sensor surface of the image sensor 122 is output via one or more wirings 1102 and an amplifier circuit 1102A (e.g., a line amplifier).

[0083] Further, a control signal (feedback current) based on a signal from the imaging element in the halation detection area 123 is input to the amplifier circuit 1102A from the decision circuit 1111 via the wiring 1103. The amplifier circuit 1102A adjusts the amplification of the output of the imaging element in the area other than the halation detection area 123 based on the signal (e.g., the change over time of the current value) from the imaging element in the halation detection area 123. More specifically, when halation occurs in the halation detection area 123, the feedback current input to the amplifier circuit 1102A fluctuates, and the amount of amplification in the amplifier circuit 1102A fluctuates based on the feedback current, thereby reducing the brightness of the output image. This makes it possible to suppress the occurrence of halation.

[0084] <Modification> Each functional unit of the image processing device 10 and the control device 11 may be realized by, for example, cloud computing provided by one or more computers. The image processing device 10 and the control device 11 may be configured as an integrated device. The image processing device 10 and the imaging device 12 may be configured as an integrated device. Various processes of the image processing device 10 may be performed by a server 50. The moving object 1 may have a semiconductor device, and one semiconductor device may include the image processing device 10 and the control device 11. The moving object 1 may have a plurality of semiconductor devices, one of which may include the image processing device 10 and another of which may include the control device 11.

[0085] <Advantages of this disclosure> Conventionally, when a moving object 1 moves under street lights at night, halation (flare, ghost) may occur due to re-reflection on the glass of the housing of the moving object 1, such as the windshield, and on the lens of the imaging device 12.

[0086] According to the present disclosure, the brightness of each frame can be appropriately corrected. Therefore, for example, when a moving image of each frame is displayed on an omnidirectional monitor, a rear monitor, or the like, it is possible to reduce fluctuations in brightness (hunting) caused by halation. In addition, it is considered that the accuracy of detecting an object based on an image can be improved by, for example, reducing ghosting.

[0087] Furthermore, when the correction amount is determined based on the entire frame, it is necessary to calculate the brightness of the entire frame. Furthermore, the image processing device 10 cannot perform processing on the current target frame until the pixel values ​​of all pixels of the frame are input from the imaging device 12 and stored in the memory.

[0088] On the other hand, in the technology disclosed herein, the correction amount is determined based only on each pixel in the halation detection area among each pixel of the frame, so that the correction process can be performed faster (with lower latency) compared to when the correction amount is determined based on the entire frame.

[0089] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0090] 500 Control System 1. Mobile 10 Image processing device 101 Acquisition Department 102 Decision Section 103 Generation part 104 Output section 11 Control device 111 Storage section 112 Recognition part 113 Tracking Department 114 Control section 12A Imaging device 12B Imaging device 12C Imaging device 12D imaging device 12E Imaging device 14 Wireless communication devices 15 Sensors 16 Drive unit 17 Lamp unit 18 Navigation devices 19 Display device 50 Servers

Claims

1. an acquisition unit that acquires a first image captured by an imaging device mounted on a moving body and including a first area set based on an installation position of the imaging device on the moving body; a generation unit that corrects brightness of a second region that is below the first region in a direction perpendicular to a moving plane of the moving object based on the brightness of the first region, and generates a second image in which the brightness of the first region is not corrected; an output unit that outputs the second image; having The generation unit is generating the second image including the second region and not including the first region; Image processing device.

2. an acquisition unit that acquires a first image captured by an imaging device mounted on the moving object; a generating unit that generates a second image by correcting brightness of a second region that is below the first region of the first image in a direction perpendicular to a moving plane of the moving object based on the brightness of the first region; an output unit that outputs the second image; having The generation unit is generating the second image including the second region and not including the first region; Image processing device.

3. The generation unit is correcting the brightness of each pixel included in the second region based on the brightness of the pixel; 3. The image processing device according to claim 1 or 2.

4. The generation unit is correcting the brightness of the third region based on a distance from the first region to the third region included in the second region; The image processing device according to claim 1 .

5. The first area is imaged by the imaging device through a diffusion plate. The image processing device according to claim 1 .

6. The image processing device A process of acquiring a first image captured by an imaging device mounted on a moving body, the first image including a first area set based on an installation position of the imaging device on the moving body; a process of correcting the brightness of a second region below the first region in a direction perpendicular to the moving plane of the moving object based on the brightness of the first region, and generating a second image in which the brightness of the first region is not corrected; outputting the second image; Run the generating process generates the second image including the second region and not including the first region. Image processing methods.

7. The image processing device includes: A process of acquiring a first image captured by an imaging device mounted on a moving body, the first image including a first area set based on an installation position of the imaging device on the moving body; a process of correcting the brightness of a second region below the first region in a direction perpendicular to the moving plane of the moving object based on the brightness of the first region, and generating a second image in which the brightness of the first region is not corrected; outputting the second image; Run the command, the generating process generates the second image including the second region and not including the first region. program.

8. An image processing system including an image processing device and an imaging device mounted on a moving object, The image processing device includes: an acquisition unit that acquires a first image captured by the imaging device and including a first area that is set based on an installation position of the imaging device on the moving object; a generation unit that corrects brightness of a second region that is below the first region in a direction perpendicular to a moving plane of the moving object based on the brightness of the first region, and generates a second image in which the brightness of the first region is not corrected; an output unit that outputs the second image; having The generating unit generates the second image including the second region and not including the first region. Image processing system.

Images