Image processing device

Abstract

An image processing device comprising the following image processing devices, the image processing device comprising: an acquisition unit, which acquires information of each image captured as a separate segment by a plurality of image capture devices, each of which captures a specific perspective that can supplement the field of view of a driver of a vehicle, and acquires attribute information of each image; a calculation unit, which calculates each correction value for aligning the dynamic range of each image capture device based on the attribute information acquired by the acquisition unit; an image merging unit, which merges the respective image information into an image with a single dynamic range based on the respective correction values ​​calculated by the calculation unit and the attribute information of the respective image information; and an output unit, which outputs the image information merged by the image merging unit to a display device.

Description

Technical Field

[0001] The present disclosure relates to an image processing device and an image processing program for merging segmentally captured vehicle surrounding images. Background Art

[0002] Japanese Patent Application Laid-Open (JP-A) No. 2017-175480 describes merging overlapping images, and in particular, paragraph 0066 of JP-A No. 2017-175480 describes displaying the result on an interior rearview mirror. However, no mention is made of dynamic range, and there are still issues with highlight and shadow clipping.

[0003] JP-A No. 2012-6551 describes a display unit and an image generation unit. The display unit is capable of selectively displaying images captured by a front camera, a left camera, a right camera, and a rear camera, and the image generation unit generates a surround view image (as if looking down at the vehicle from above) by performing a viewpoint conversion. The surround view image is displayed on the display unit. Summary of the invention

[0004] Technical issues

[0005] However, for example, the prior art is only adjusted to a specific brightness (luminance), and is not able to reflect all information when each image is captured separately with appropriate exposure for a separate imaging area. Therefore, when the images are configured as a single image, the difference in dynamic range between the images may generate an image that looks unnatural, and may cause objects (e.g., obstacles that appear during travel) to be misjudged (including being ignored).

[0006] In particular, JP-A No. 2012-6551 describes a technique related to synthesizing a surround view. However, there is no mention of merging in a manner that takes into account the contrast of images captured by the respective cameras when generating a surround view, for example.

[0007] An object of the present disclosure is to obtain an image processing device and an image processing program capable of generating an image that looks natural based on image capturing parameters and the like of a plurality of images when combining information of the respective images and capable of expressing a wide dynamic range.

[0008] Solution to the problem

[0009] An image processing device according to one aspect of the present disclosure includes: an acquisition unit that acquires information of individual images captured as separate segments by a plurality of image capture devices, each of which captures a specific perspective that can supplement the field of view of a driver of a vehicle, and acquires attribute information of the individual images; a calculation unit that calculates individual correction values ​​for aligning a dynamic range of the individual image capture devices based on the attribute information acquired by the acquisition unit; an image merging unit that merges the individual image information into an image with a single dynamic range based on the individual correction values ​​calculated by the calculation unit and the attribute information of the individual image information; and an output unit that outputs the image information merged by the image merging unit to a display device.

[0010] In an aspect of the present disclosure, a dynamic range of an image merged by the image merging section has a high dynamic range wider than a dynamic range of any one of the image capturing devices.

[0011] In one aspect of the present disclosure, an adjustment unit is further included that adjusts the image information merged by the image merging unit to a dynamic range suitable for the specification of the display device.

[0012] In one aspect of the present disclosure, the attribute information includes image capture parameters when capturing each image, including sensitivity, exposure time, and aperture.

[0013] In one aspect of the present disclosure, the attribute information is inter-individual difference information about respective captured images obtained from overlapping image capturing areas of the respective image information.

[0014] In the above aspects of the present disclosure, the attribute information of the information of the multiple images can be used to obtain a correction value for aligning the dynamic range of the information of the multiple images. Examples of the attribute information include image capture parameters at the time of capture, the image capture parameters including sensitivity, exposure time, and aperture, which can be obtained together with the captured image information.

[0015] The information of multiple images is merged using correction values ​​and image capture parameters (such as coordinates) to generate information of a single image. This enables a natural-looking image to be obtained, thereby preventing misjudgment of an object, etc.

[0016] Note that the attribute information may be inter-individual difference information about the respective captured images obtained from overlapping image-capture areas of the respective image information.

[0017] Note that since the synthesized image information has a high dynamic range (HDR), adjustment by the adjustment section, for example, adjustment by tone mapping, may be performed to compress the high dynamic range into a specification (dynamic range) suitable for the device.

[0018] Note that although a perspective that supplements the rear view of the vehicle is given as an example of a specific perspective, the specific perspective is not limited to the rear view and may be, for example, a perspective that supplements the vehicle surroundings when looking down from above (so-called surround view).

[0019] In one aspect of the present disclosure, each of a plurality of image capture devices is mounted to a vehicle and includes a RH camera that captures in a direction right and rearward from the vehicle, a LH camera that captures in a direction left and rearward from the vehicle, and a RR camera that captures in a direction rearward from the vehicle. Each of the image capture devices has a defined viewing angle for image capture such that image information captured by the respective image capture devices at least partially overlaps.

[0020] In one aspect of the present disclosure, the display device is a monitor installed instead of a mirror for checking the rear of a vehicle.

[0021] There are more blind spots in the field of view behind the vehicle than in the field of view ahead. Therefore, for example, obstacles may be overlooked in the image on a mirror surface such as an interior rearview mirror or a side mirror, particularly due to differences in the brightness of the surroundings.

[0022] The above aspects of the present disclosure make it possible to eliminate blind spots in the rear field of view when driving a vehicle, and also prevent misjudgment (including neglect) of obstacles when the vehicle is reversing, thereby achieving safe driving.

[0023] In one aspect of the present disclosure, the display device is a storage device that performs cyclic storage of captured image information corresponding to predetermined time intervals such that captured image information from old time intervals is deleted and captured image information from a latest time interval is stored.

[0024] In the above aspects of the present disclosure, the image information output from the output unit can be applied as an image from a driving recorder that records the collision before and after the vehicle. That is, the wide-angle image of the present disclosure with natural-looking boundaries, etc. can be used as analysis material to accurately find the cause of the collision involving the vehicle.

[0025] An image processing program of another aspect of the present disclosure causes a computer to operate as any of the above image processing devices.

[0026] Advantageous Effects of the Invention

[0027] As explained above, the present disclosure exhibits advantageous effects of enabling generation of natural-looking images based on image capture parameters and the like of a plurality of images while combining information of the respective images, and enabling expression of a wide dynamic range. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] [ Figure 1A ] Figure 1A is a plan view showing a vehicle according to an exemplary embodiment.

[0029] [ Figure 1B ] Figure 1B yes Figure 1A A side view of the vehicle is shown in FIG.

[0030] [ Figure 2 ] Figure 2 is a schematic diagram showing the interior of a cabin of a vehicle according to an exemplary embodiment viewed from the center of a rear seat toward a front seat.

[0031] [ Figure 3 ] Figure 3 is a schematic diagram illustrating an image processing apparatus for merging images according to an exemplary embodiment.

[0032] [ Figure 4 ] Figure 4 is a control flow chart illustrating a flow of processing performed by an image processing apparatus according to an exemplary embodiment.

[0033] [ Figure 5 ] Figure 5 An image obtained by simple merging of the images captured by the individual cameras is shown.

[0034] [ Figure 6 ] Figure 6 Shown by making Figure 5 The image shown in is an image obtained by subjecting the image to high dynamic range processing.

[0035] [ Figure 7 ] Figure 7 Shows when Figure 6 The image shown in FIG. 1 is an image when it is displayed on a display device as it is (with highlight clipping).

[0036] [ Figure 8 ] Figure 8 Shows when Figure 6 The image shown in FIG. 1 is an image when it is displayed as it is on a display device (with shadow clipping).

[0037] [ Fig. 9 ] Fig. 9 Shows when Figure 6 The image shown in FIG. 1 is displayed on a display device after undergoing a single dynamic range compression.

[0038] [ Fig.10 ] Fig.10 Shows Figure 6 The image shown in FIG. 1 is displayed on a display device after being subjected to local tone compression. DETAILED DESCRIPTION

[0039] Figure 1A is a plan view showing a vehicle 10 according to an exemplary embodiment, and Figure 1B 1 is a side view showing the vehicle 10 .

[0040] like Figure 1A As shown, the right camera adapter 12 is attached to the right side of the vehicle 10 relative to the forward direction. The RH camera 14 whose optical axis extends toward the right rear of the vehicle 10 is attached to the right camera adapter 12. The RH camera 14 captures Figure 1A The area AR surrounded by the single-dot chain line R in FIG.

[0041] In addition, if Figure 1A As shown, the left camera adapter 16 protrudes from the left side of the vehicle 10 relative to the forward direction. An LH camera 18 having an optical axis extending toward the left rear of the vehicle 10 is attached to the left camera adapter 16. The LH camera 18 captures Figure 1A The area AL surrounded by the single-dot chain line L in FIG.

[0042] Likewise Figure 1A As shown, the RR camera 20 having an optical axis extending rearward from the vehicle 10 (the optical axis is substantially parallel to the road surface) is attached to the vehicle width direction center portion of the rear portion of the vehicle 10. The RR camera 20 captures Figure 1A The area AC surrounded by the single-dot chain line C in FIG.

[0043] The area AR and the area AC partially overlap each other, so that the images captured by them partially overlap. The area AL and the area AC also partially overlap each other, so that the images captured by them partially overlap.

[0044] like Figure 2 As shown, a dedicated backup camera 22 is also attached to the rear of the vehicle 10 separately from the RR camera 20. The dedicated backup camera 22 is configured to capture an area AB surrounded by a dotted line B, and when the vehicle 10 is reversed, the captured image is displayed on an information display monitor 24 (see FIG. Figure 2 ) on. The optical axis is set so that the area AB includes the rear end portion of the vehicle 10 (e.g., the rear bumper) and faces downward compared to the optical axis of the RR camera 20.

[0045] That is, the above-mentioned RH camera 14, LH camera 18 and RR camera 20 are used to assist the driver during normal driving of the vehicle 10 (whether moving forward, reversing or stationary), and therefore have a different purpose from the dedicated reversing camera device 22, which is specifically used to assist the driver when the vehicle 10 is reversing.

[0046] Figure 2 It is a schematic diagram of the interior of the cabin of the vehicle 10 viewed from the center of the rear seats toward the front seats.

[0047] The above-mentioned right camera adapter 12 is positioned in front of the right side window 26 of the vehicle 10 , and the RH camera 14 is attached to the right camera adapter 12 .

[0048] The RH monitor 32 is attached to a lower portion of the right A-pillar 30 between the right window 26 and the front window 28. The RH monitor 32 is configured to monitor the right rear area AR of the vehicle captured by the RH camera 14 (see FIG. Figure 1A ) is displayed in a mirror-inverted state. That is, the RH camera 14 and the RH monitor 32 have a function equivalent to that of an optical right door mirror.

[0049] The above-mentioned left camera adapter 16 is positioned in front of the left side window 34 of the vehicle 10 , and the LH camera 18 is attached to the left camera adapter 16 .

[0050] The LH monitor 38 is attached to the lower portion of the left A-pillar 36 between the left window 34 and the front window 28. The LH monitor 38 is configured to monitor the left rear area AL of the vehicle captured by the LH camera 18 (see FIG. Figure 1A ) is displayed in a mirror-inverted state. That is, the LH camera 18 and the LH monitor 38 have a function equivalent to that of the optical left door mirror.

[0051] The information display monitor 24 is attached to the upper portion of the center console 40 in the vehicle cabin. The information display monitor 24 displays an image captured by the dedicated backup camera 22 when a shift lever 42 of a transmission (not shown) of the vehicle 10 has been placed in the reverse range (R range).

[0052] The information display monitor 24 is used as a multi-function monitor, and for example, in addition to displaying the vehicle rear image as described above, the information display monitor 24 cooperates with a navigation system and an audio system not shown to display a navigation screen (a screen with navigation information, etc.) or an audio screen (a screen with information related to music, etc.) when a switching operation is performed. For example, based on the position information acquired using the GPS function installed in the vehicle 10, a map image and an image of the vehicle's running position on the map image are displayed on the navigation screen. Note that in the case where the information display monitor 24 is configured by a touch panel, the information display monitor 24 can also be used as an input device.

[0053] Note that the inside rearview mirror 44 attached to the upper center portion of the front window 28 (or ceiling) of the vehicle 10 of the present exemplary embodiment serves as a display device of the present disclosure.

[0054] That is, the inside rearview mirror 44 has both a mirror function of serving as an optical mirror and a monitor function of displaying an image.

[0055] The rearview mirror function enables the driver or the like to check the situation behind the vehicle within their field of view (by moving their line of sight) while driving.

[0056] The monitor function enables display of a composite image constructed from the images captured by the RH camera 14 , the LH camera 18 and the RR camera 20 .

[0057] like Figure 3 As shown, the interior rearview mirror 44 includes a display driver 45 and a display unit 48, which includes a built-in display device 46 such as an LCD and has a mirror-finished surface. Due to reasons related to light intensity, when the display device 46 does not display an image, the display unit 48 acts as a mirror, and when the display device 46 displays an image, the display unit 48 acts as a monitor.

[0058] Note that the reason related to light intensity is that since the brightness of the image displayed by the display device 46 is greater than the brightness of the image optically reflected by the mirror surface, the image display by the display device 46 is prioritized.

[0059] Figure 3 An image processing device 50 for merging images is schematically shown.

[0060] The following description assumes that the display device 46 of the inside rearview mirror 44 operates in the monitor function to display the synthetic image. However, the monitor function for the synthetic image is not limited to the inside rearview mirror 44, and may be a function of the information display monitor 24. Alternatively, a dedicated monitor for displaying the synthetic image according to the present disclosure may also be additionally provided in the vehicle cabin separately from the inside rearview mirror 44 and the information display monitor 24.

[0061] like Figure 3 As shown, the image information of the area AR imaged by the RH camera 14 is temporarily stored in the image information memory 52. ​​In synchronization with the storage of the image information in the image information memory 52, the image capture parameters (including sensitivity, exposure time, and aperture) of the RH camera 14 when captured by the RH camera 14 are stored in the image capture parameter information memory 54.

[0062] The image information of the area AC imaged by the RR camera 20 is temporarily stored in the image information memory 56. In synchronization with the storage of the image information in the image information memory 56, the image capture parameters (including sensitivity, exposure time, and aperture) when captured by the RR camera 20 are stored in the image capture parameter information memory 58.

[0063] The image information of the area AL imaged by the LH camera 18 is temporarily stored in the image information memory 60. In synchronization with the storage of the image information in the image information memory 60, the image capturing parameters (including sensitivity, exposure time, and aperture) at the time of capturing by the LH camera 18 are stored in the image capturing parameter information memory 62.

[0064] The image information memory 52, the image information memory 56, and the image information memory 60 are each connected to the brightness merging section 63. The brightness merging section 63 is connected to the image merging section 64.

[0065] The image capturing parameter information memory 54 , the image capturing parameter information memory 58 , and the image capturing parameter information memory 62 are each connected to the inter-camera correction value calculation section 66 .

[0066] The inter-camera correction value calculation section 66 calculates a correction value for information of each image to adjust the brightness between the cameras and, in particular, the brightness of the overlapping portion so that the respective brightnesses are closer. When performing this operation, the correction value is not calculated using the information of any one image as a reference. Instead, the correction value for information of each image is calculated based on the respective image capture parameters to generate information of a single high dynamic range (HDR) image capable of combining images of three dynamic range types.

[0067] The inter-camera correction value calculation section 66 is connected to the brightness combining section 63 .

[0068] The brightness combining section 63 corrects the image brightness of each recorded image based on the correction value calculated by the inter-camera correction value calculating section 66. In order to avoid overflow and digital loss during this correction, the image information is converted into a storage format having a high dynamic range.

[0069] Using the image information that has been brightness corrected by the brightness merging section 63, the image merging section 64 combines the image information captured by the RH camera 14, the image information captured by the RR camera 20, and the image information captured by the LH camera 18, using the area where the image capture overlaps as a reference. Note that since rear image assistance is provided in the present exemplary embodiment, the image merging section 64 not only merges the images but also performs mirror inversion processing. This results in an image equivalent to a left-right inverted image on a mirror surface.

[0070] That is, since the overlapping portion is a region captured at the same timing, processing such as aligning the position of a contour portion where there is a large difference between light and dark enables correction of misalignment at the time of merging.

[0071] Note that although overlapping capture areas are created in the present exemplary embodiment, overlapping capture is not essential as long as image capture is at least adjacent (images are continuous with each other).

[0072] Figure 5 An image obtained by simply merging images captured by the RH camera 14 , the RR camera 20 , and the left LH camera 18 (collectively referred to as cameras) is shown, and does not represent an image displayed on the display device 46 or the like in the present exemplary embodiment.

[0073] If the light intensity between the various fields of view of the cameras is different, the automatic exposure function of each camera will result in a different exposure for each camera. In such a case, the problem that arises is that the image brightness between the various cameras in the overlapping area is different.

[0074] The image combining section 64 corrects the brightness of each piece of image information based on the correction value acquired from the inter-camera correction value calculation section 66. The respective brightness-corrected images are combined with the positional misalignment corrected.

[0075] Figure 6 Shown with Figure 5 The image corresponds to the synthesized image after being subjected to the high dynamic range processing by the image merging section 64. Although there is no clipping of shadows and highlights in the image data, the image data exceeds the dynamic range that the display device 46 can actually display, and therefore it is difficult to truly display the image data.

[0076] Right now, Figure 7 and Figure 8 It is shown separately Figure 6 The image shown in FIG. 1 is an image when displayed as it is on a display device. Since this is a high dynamic range image, it cannot be displayed as it is using the display device 46, which is a standard configuration that does not provide a sufficient dynamic range. Therefore, in the bright area (see Figure 7 ) or on the darker side (see Figure 8 ) Shadow clipping occurs.

[0077] The image merging section 64 is connected to the dynamic range compressing section 68. The synthesized image information is output to the dynamic range compressing section 68.

[0078] The dynamic range compression section 68 includes a background brightness estimation section 70 and a brightness difference / hue compression section 72 .

[0079] Basically, the dynamic range compression section 68 has the following functions: suppressing shadow clipping or highlight clipping depending on the position in the synthetic image information subjected to HDR processing, and converting the synthetic image information into a dynamic range that the display device 46 can display.

[0080] The background brightness estimation section 70 estimates the background brightness as a low-frequency component in the synthesized image information subjected to the HDR process.

[0081] The brightness difference / tone compression section 72 extracts the background brightness difference from the synthesized image subjected to HDR processing. Then, the brightness difference / tone compression section 72 compresses the contrast of the background brightness estimated by the background brightness estimation section 70, and re-merges using the extracted background brightness difference. The re-merged image retains the local contrast while having a compressed overall dynamic range. Note that a configuration may be made in which the dynamic range compression section does not isolate the background / brightness difference, but simply reduces the contrast of the synthesized image information subjected to HDR processing ( Fig. 9 ), or locally reduce contrast (see Fig.10 ). However, in the case of simply reducing the contrast, the image is relatively less easy for the occupants to see.

[0082] The dynamic range compression section 68 is connected to the display driver 45 of the interior rearview mirror 44 via an output section 74 .

[0083] The output section 74 outputs the image information subjected to the compression process by the dynamic range compression section 68 to the display driver 45 based on an output instruction not shown in the figure. The display driver 45 displays the image information on the display device 46 (see Fig. 9 or Fig.10 ).

[0084] Note that during output to the output section 74, the output instruction may always be present, or a sensor or the like may detect when the rear view on the mirror surface becomes poor (e.g., during rain, when the sun is low in the sky at dawn or dusk, at night, etc.) and switch to display on the display device 46. Alternatively, display using the display device 46 may be performed when instructed by the driver.

[0085] Refer to the following Figure 4 The operation of this exemplary embodiment is described with reference to the flowchart of FIG.

[0086] Figure 4 1 is a control flow chart showing a flow of image processing by the image processing device 50 on image information captured by the RH camera 14 , the RR camera 20 , and the LH camera 18 until the image is displayed on the display device 46 .

[0087] At step 100, it is determined whether there is a rear image output instruction. In the case of a negative determination, the routine ends. In the case of an affirmative determination at step 100, the process proceeds to step 102. Note that the affirmative determination at step 100 includes the case where there is a constant output instruction.

[0088] At step 102, image information captured by each camera (RH camera 14, RR camera 20, and LH camera 18) is acquired. Then, the process proceeds to step 104, where the image information is temporarily stored in image information memory 52, image information memory 56, and image information memory 60, respectively. Then, the process proceeds to step 106.

[0089] At step 106, image capture parameters at the time of capture are acquired for each camera (RH camera 14, RR camera 20, and LH camera 18). Then, the process proceeds to step 108, where the image capture parameters are temporarily stored in image capture parameter information memory 54, image capture parameter information memory 58, and image capture parameter information memory 62, respectively. Then, the process proceeds to step 110.

[0090] At step 110 , based on the respective image capturing parameters temporarily stored in the image capturing parameter information memory 54 , the image capturing parameter information memory 58 , and the image capturing parameter information memory 62 , the inter-camera correction values ​​are calculated using the image capturing parameters.

[0091] At the next step 112, the overlapping area is extracted based on the image information stored in the image information memory 52, the image information memory 56, and the image information memory 60. Then, the process proceeds to step 114.

[0092] At step 114, the image brightness of each recorded image is corrected based on the correction value calculated by the inter-camera correction value calculation section 66. In order to avoid overflow and digital loss during correction, the image information is converted into a high dynamic range storage format.

[0093] At the next step 116, an image merging process is performed on the brightness corrected image information.

[0094] The image generated by the merging process at step 116 corresponds to high dynamic range image information, such as Figure 6 Although the image information includes information ranging from the darkest areas to the brightest areas, the specifications of the display device 46 may mean that shadow clipping occurs in the dark areas, while highlight clipping occurs in the bright areas.

[0095] At the next step 118, the background brightness is estimated from the synthesized HDR image generated by the merging process at step 116. Then, the process goes to step 120, where the brightness difference of the background brightness is obtained, and the brightness difference / tone compression section 72 of the dynamic range compression section 68 performs tone compression.

[0096] At the next step 122, the image generated by the tone compression at step 120 is output to the display device 46. Fig. 9 or Fig.10 As shown, an image with a single dynamic range is displayed on the display device 46. The image displayed on the display device 46 has been adapted to the dynamic range of the display device 46, thereby eliminating shadow clipping and highlight clipping.

[0097] In the present exemplary embodiment described above, image information captured by the plurality of camera devices (RH camera 14, RR camera 20, and LH camera 18) is merged into an HDR image with adjusted brightness, and then subjected to tone compression for display suitable for the dynamic range that can be displayed by the display device 46. Therefore, for example, even an image in which both direct sunlight and shadows are present, resulting in a very large contrast within the image, can be displayed by the display device 46 in a state in which shadow clipping in dark areas and highlight clipping in bright areas are suppressed.

[0098] Relationship between HDR processing and display device 46

[0099] As an example, in a case where the range behind the vehicle 10 captured by the RH camera 14, the RR camera 20, and the LH camera 18 corresponds to a scene with shadow on one side and direct sunlight on the other side, the dynamic range of the scene (the ratio of the brightest area to the darkest area) becomes wider and the contrast ratio may exceed 10,000:1.

[0100] However, the contrast ratio of the dynamic range of the light receivers (eg, CMOS sensors or CCD sensors) of the RH camera 14, the RR camera 20, and the LH camera 18 is in the range of 1000:1.

[0101] Therefore, the RH camera 14, the RR camera 20, and the LH camera 18 each independently convert their respective dynamic ranges (change the exposure level) to capture using a dynamic range suitable for display on the display device 46. Therefore, when displayed on the display device 46, a simple merging process may result in an extreme jump in image brightness at the junction between the images from the various cameras (see Figure 5 ).

[0102] Therefore, in this exemplary embodiment, brightness correction is performed to adjust the images from each camera to images within the same brightness range. Applying a small gain to an image with a high exposure level and applying a large gain to an image with a low exposure level enables the images to be adjusted to images with the same exposure level. When performing such gain correction, the corrected images are retained in a storage format with a wide dynamic range to avoid overflow and digital loss. Then, an image merging process is performed on these brightness-corrected images. The resulting composite image has a wide dynamic range (see Figure 6 ).

[0103] Therefore, if the composite image is displayed as it is on the display device 46, with respect to image brightness clipping, shadow clipping may occur in some relatively dark areas and highlight clipping may occur in some relatively bright areas at the connection between the images from the respective cameras (see Figure 7 and Figure 8 ).

[0104] In the present exemplary embodiment, the high dynamic range of the synthesized image is compressed by the dynamic range compression section 68 into a (single) dynamic range suitable for the display device 46 (see Fig. 9 or Fig.10 ), so that the HDR composite image information is converted into a dynamic range that the display device 46 can display with shadow clipping and highlight clipping locally suppressed.

[0105] Therefore, if Fig. 9 and Fig.10 As shown, even in Figure 7 and Figure 8 In the bright areas with highlight clipping and the dark areas with shadow clipping in the image shown in , the image actually displayed on the display section 48 of the inside rearview mirror 44 (display device 46) can also be clearly perceived as rearview field information.

[0106] In the present exemplary embodiment, retaining the composite image as HDR and compressing the dynamic range when displayed is itself a unique technology in the field of technology that captures the surroundings of a vehicle as separate segments and then merges them. In other words, heretofore, information of separate image segments has been merged by taking the information of one image as a reference image and adjusting the brightness of the other image information accordingly, without performing HDR processing on the information of a plurality of separately captured image segments.

[0107] For example, Figure 5As shown, shadow clipping and the like may sometimes occur in a simple form of HDR image. However, with the HDR image, it is possible to detect the gaze position of the driver within the display area of ​​the display device 46 and perform various processes using the dynamic range compression section 68, such as adjusting the contrast of a dark image to make it easier to see when the gaze position corresponds to a dark area of ​​the image, or adjusting the contrast of a bright image to make it easier to see when the gaze corresponds to a bright area of ​​the image.

[0108] That is, the present exemplary embodiment relates to a technology that captures the surroundings of a vehicle in separate segments and then merges them into an image that is processed as an HDR image, thereby making it possible to suppress the possibility of overlooking obstacles, etc. while presenting an image that is more suitable for assisting the driver than when using an image in which the image brightness differs at the boundary between camera devices.

[0109] Furthermore, in the present exemplary embodiment, the synthesized image information having a high dynamic range is subjected to dynamic range compression suitable for the display device 46 employed, thereby making it possible to express a range from a dark area to a bright area in a normal display state of the display device 46 (particularly, a state in which the contrast is not adjusted according to the gaze as described above) (see Fig. 9 and Fig.10 ).

[0110] Note that in the present exemplary embodiment, when image information is acquired from each camera (RH camera 14, RR camera 20, and LH camera 18), image capture parameters at the time of capture, including sensitivity, exposure time, and aperture, are also acquired as attribute information required when merging images. However, such attribute information may be obtained from the image information. That is, the correction value may be calculated as attribute information of the brightness difference between the overlapping portions of each information of each image.

[0111] Furthermore, although in the present exemplary embodiment, an image behind the vehicle 10 is captured to assist the field of view behind the vehicle 10, the image processing device 50 of the present exemplary embodiment may also be applied to a technique of combining other images such as an image showing a surrounding view, a front view, etc. In such a case, in addition to the four camera devices described in the present exemplary embodiment, one or more additional camera devices are provided to capture toward the front side.

[0112] In the present exemplary embodiment, the images captured by the RH camera 14, the LH camera 18, and the RR camera 20 are subjected to HDR processing and the like before being displayed on the display device 46. However, it is possible to make the application record images using the driving recorder without displaying the images, so that the recording is stopped in the case of a vehicle collision and the like and the images before and after the vehicle collision are saved. A recording device that records images can be applied so that images with start and end prompts (in a loop) are recorded at uniform time intervals. When applied to the driving recorder, clear image information can be obtained regardless of dark areas or bright areas, and this can be used for accurate analysis of vehicle collisions.

[0113] Furthermore, in the present exemplary embodiment, the acquired image information to be merged is limited to the camera devices (RH camera 14, LH camera 18, and RR camera 20) mounted to the vehicle 10. However, the communication unit may be employed to acquire information captured by infrastructure-based camera devices mounted along the road being traveled or camera devices mounted to other vehicles in the surrounding environment, which is used as information to supplement the areas corresponding to the blind spots of the RH camera 14, LH camera 18, and RR camera 20, for example.

[0114] The following supplements related to the above-described exemplary embodiments are disclosed.

[0115] Supplement 1

[0116] An image processing device, comprising:

[0117] an acquisition section that acquires individual information of images captured as separate segments by a plurality of image capture devices, each of which captures a specific viewing angle capable of supplementing a field of view of a driver of a vehicle, and acquires attribute information of the individual images;

[0118] a calculation section that calculates respective correction values ​​for aligning the dynamic ranges of the respective image capturing devices based on the attribute information acquired by the acquisition section;

[0119] an image combining section that combines the respective image information into an image having a single dynamic range based on the respective correction values ​​calculated by the calculating section and the attribute information of the respective image information; and

[0120] An output unit outputs the image information merged by the image merging unit to a display device.

[0121] Supplement 2

[0122] The image processing device according to Supplement 1, wherein the image information merged by the image merging section has a dynamic range wider than a dynamic range of any one of the image capturing devices.

[0123] Supplement 3

[0124] The image processing device according to Supplement 1 or Supplement 2 further includes an adjustment unit that adjusts the image information merged by the image merging unit to a dynamic range that can be displayed by the display device.

[0125] Supplement 4

[0126] The image processing device according to any one of Supplement 1 to Supplement 3, wherein the attribute information includes image capturing parameters when each image is captured, the image capturing parameters including sensitivity, exposure time, and aperture.

[0127] Supplement 5

[0128] The image processing device according to any one of Supplement 1 to Supplement 3, wherein the attribute information is inter-individual difference information about the respective image capturing devices obtained from overlapping image capturing areas of the respective image information.

[0129] Supplement 6

[0130] An image processing device according to any one of Supplements 1 to 5, wherein each of a plurality of image capture devices is mounted to a vehicle and includes a RH camera, an LH camera, and a RR camera, the RH camera captures in directions to the right and rearward from the vehicle, the LH camera captures in directions to the left and rearward from the vehicle, the RR camera captures in a direction to the rear from the vehicle, and each of the image capture devices has a limited viewing angle for image capture, so that image information captured by each image capture device at least partially overlaps.

[0131] Supplement 7

[0132] The image processing device according to any one of Supplement 1 to Supplement 6, wherein the display device is a monitor installed instead of a mirror for checking the rear of the vehicle.

[0133] Supplement 8

[0134] An image processing device according to any one of Supplements 1 to 6, wherein the display device is a storage device, performing cyclic storage of captured image information corresponding to a predetermined time interval, so that captured image information from an old time interval is deleted and captured image information from a recent time interval is stored.

[0135] Supplement 9

[0136] An image processing program that causes a computer to operate as the image processing device according to any one of Supplement 1 to Supplement 8.

[0137] The disclosure of Japanese Patent Application No. 2019-130370 filed on Jul. 12, 2019 is incorporated herein by reference in its entirety.

[0138] All references, patent applications, and technical standards mentioned in this specification are incorporated by reference into the specification to the same extent as if each individual reference, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0139] Reference Mark List

[0140] 10 Vehicles

[0141] 12 Right camera adapter

[0142] 14 RH camera (image capture device)

[0143] 16 Left camera adapter

[0144] 18 LH camera (image capture device)

[0145] 20 RR camera (image capture device)

[0146] 22 Dedicated reversing camera

[0147] 24 Information Display Monitor

[0148] 26 Right window

[0149] 28 front window

[0150] 30 Right A-pillar

[0151] 32 RH Monitor

[0152] 34 Left window

[0153] 36 Left A-pillar

[0154] 38 LH Monitor

[0155] 40 Center console

[0156] 42 Gear lever

[0157] 44 Interior rearview mirror

[0158] 45 Display Driver

[0159] 46 Display device

[0160] 48 Display

[0161] 50 Image processing device

[0162] 52 Image information storage (acquisition unit)

[0163] 54 Image capture parameter information storage (acquisition unit)

[0164] 56 Image information storage (acquisition unit)

[0165] 58 Image capture parameter information storage (acquisition unit)

[0166] 60 Image information storage (acquisition unit)

[0167] 62 Image capture parameter information memory (acquisition unit)

[0168] 63 Brightness Merging Unit

[0169] 64 Image merging unit (image merging unit)

[0170] 66: Inter-camera correction value calculation unit (calculation unit)

[0171] 68 Dynamic range compression unit (adjustment unit)

[0172] 70 Background brightness estimation unit

[0173] 72 Brightness difference / hue compression unit

[0174] 74 Output

[0175] R single dot dash line

[0176] AR field of view

[0177] L single dot dash line

[0178] AL Region

[0179] C single dot dash line

[0180] AC Area

[0181] B Dashed Line

[0182] AB area.

Claims

1. An image processing device, comprising: an acquisition section that acquires information of respective images captured as separate segments by a plurality of image capture devices, each of which captures a specific viewing angle capable of supplementing a field of view of a driver of a vehicle, and acquires attribute information of the respective images; a calculation unit that calculates respective correction values ​​for aligning the dynamic range of respective image capture devices based on the attribute information acquired by the acquisition unit, wherein the correction values ​​are calculated based on respective image capture parameters; an image merging section that merges the respective image information into an image that is not displayed on a display device and has a high dynamic range width capable of simply merging each image information, based on the respective correction values ​​calculated by the calculating section and the attribute information of the respective image information; an adjustment unit capable of locally adjusting the contrast of a portion of the region of the image having the dynamic range while maintaining image information including information from the darkest portion to the brightest portion captured by the plurality of capture devices, and configured to adjust the image information merged by the image merging unit to a dynamic range displayable on the display device; and An output unit outputs the image information adjusted by the adjustment unit to the display device.

2. The image processing device according to claim 1, wherein: The attribute information includes image capture parameters when capturing the respective images, and the image capture parameters include sensitivity, exposure time, and aperture.

3. The image processing device according to claim 1, wherein: The attribute information is inter-individual difference information about the respective image capturing devices obtained from overlapping image capturing areas of the respective image information.

4. The image processing device according to claim 1, wherein: Each of the multiple image capture devices is mounted to the vehicle and includes a RH camera, an LH camera, and a RR camera, wherein the RH camera captures in a direction to the right and rearward from the vehicle, the LH camera captures in a direction to the left and rearward from the vehicle, and the RR camera captures in a direction to the rear from the vehicle, and each of the image capture devices has a limited viewing angle for image capture so that image information captured by the respective image capture devices at least partially overlaps.

5. The image processing device according to claim 1, wherein: The display device is a monitor installed in place of a mirror for checking the rear of the vehicle.

6. The image processing device according to claim 1, wherein: The display device is a storage device that performs cyclic storage of captured image information corresponding to predetermined time intervals so that captured image information from an old time interval is deleted and captured image information from a latest time interval is stored.

Images