Image Processing Device

Abstract

To provide an image processing apparatus which can reduce an influence of a heating wire on a processing result of image processing.SOLUTION: The image processing apparatus relating to one embodiment of the present disclosure comprises: a sub-image generation unit which generates a plurality of sub-images by laterally dividing a first image being one of a left image and a right image captured by a stereo camera, through a front glass having a heating wire formed therein; a brightness ratio map generation unit which selects one of the plurality of sub-images in sequence and generates map data showing a ratio of brightness values on the basis of the selected sub-image and a second image being the other of the left image and the right image; and a heating wire pattern estimation unit which estimates a wiring pattern of the heating wire on the basis of map data for the respective sub-images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device that performs image processing based on stereo images. [Background technology]

[0002] Some vehicles, such as automobiles, detect surrounding objects based on captured images obtained by an imaging device. Some vehicles are equipped with heating wires on the windshield or rear window to prevent fogging. For example, Patent Document 1 discloses a technology for recognizing, as heating wires, portions of an image captured by an on-board camera that do not change over time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-4255 Summary of the Invention [Problem to be solved by the invention]

[0004] In image processing devices, it is desirable to reduce the influence of the heating wire on the results of image processing based on captured images, and further reduction of the influence on the image processing results is expected.

[0005] It is desirable to provide an image processing device that can reduce the influence of the heating wire on the results of image processing based on a captured image. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present disclosure includes a sub-image generation unit, a brightness ratio map generation unit, and a heating wire pattern estimation unit. The sub-image generation unit is configured to generate multiple sub-images by horizontally dividing a first image, which is one of a left image and a right image captured by a stereo camera through a windshield on which a heating wire is formed. The brightness ratio map generation unit is configured to sequentially select one of the multiple sub-images and generate map data indicating a ratio of brightness values ​​based on the selected sub-image and a second image, which is the other of the left image and the right image. The heating wire pattern estimation unit is configured to estimate a wiring pattern of the heating wire based on the map data for each of the multiple sub-images. [Effects of the Invention]

[0007] According to an image processing device according to an embodiment of the present disclosure, it is possible to reduce the influence of the heating wire on the processing results of image processing. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of the arrangement of the stereo cameras shown in FIG. [Figure 3] FIG. 3 is an explanatory diagram illustrating an example of a wiring pattern of the heating wire shown in FIG. 2. [Figure 4] FIG. 2 is an explanatory diagram illustrating an example of a left image and a right image. [Figure 5] FIG. 3 is an explanatory diagram illustrating another example of the wiring pattern of the heating wire shown in FIG. [Figure 6] FIG. 2 is an explanatory diagram illustrating an example of a left image and a right image. [Figure 7] 2 is an explanatory diagram illustrating an example of an operation of the sub-image generating unit illustrated in FIG. 1. [Figure 8] 2 is a flowchart illustrating an example of the operation of the heating wire detection unit shown in FIG. 1. [Figure 9] 9 is a flowchart illustrating an example of the detection process shown in FIG. 8. [Figure 10] 2 is a characteristic diagram illustrating an example of a detection result of the heating wire detection unit illustrated in FIG. 1. [Figure 11] 2 is an explanatory diagram illustrating an example of the operation of the heating wire detection unit shown in FIG. 1. FIG. [Figure 12] 1. FIG. 4 is another explanatory diagram illustrating an example of the operation of the heating wire detection unit shown in FIG. [Figure 13] 1. FIG. 4 is another explanatory diagram illustrating an example of the operation of the heating wire detection unit shown in FIG. [Figure 14] FIG. 10 is an explanatory diagram illustrating an example of a wiring pattern of a heating wire according to a modified example. [Figure 15] FIG. 10 is an explanatory diagram illustrating an example of a wiring pattern of a heating wire according to another modified example. [Figure 16] FIG. 10 is an explanatory diagram illustrating an example of a wiring pattern of a heating wire according to another modified example. [Figure 17] FIG. 10 is a block diagram illustrating an example of the configuration of an image processing device according to another modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] <Embodiment> [Configuration example] 1 shows an example of the configuration of an image processing device (image processing device 1) according to an embodiment. The image processing device 1 includes a stereo camera 11 and a processing unit 20. In this example, the image processing device 1 is mounted on a vehicle 10 such as an automobile.

[0011] The stereo camera 11 is configured to capture an image of the area ahead of the vehicle 10, thereby generating a pair of images (a left image PL and a right image PR) having a parallax therebetween. The stereo camera 11 includes a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor.

[0012] FIG. 2 shows an example of the arrangement of the stereo camera 11 in a vehicle 10. In this example, the left camera 11L and the right camera 11R are arranged inside the vehicle 10 near the top of the windshield 9 of the vehicle 10, spaced a predetermined distance apart in the width direction of the vehicle 10. The left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL and the right image PR form a stereo image PIC. The stereo camera 11 performs an imaging operation at a predetermined frame rate (e.g., 60 fps) to generate a series of stereo images PIC and supply the generated stereo images PIC to the processing unit 20.

[0013] An electric heating wire 8 is provided on the windshield 9 in front of the left camera 11L and the right camera 11R. The electric heating wire 8 is configured to generate heat when a current flows from a drive circuit (not shown). In the vehicle 10, the electric heating wire 8 generates heat, thereby preventing the windshield 9 from fogging up.

[0014] Fig. 3 shows an example of the wiring pattern of the heating wire 8 on the windshield 9. Fig. 3 shows the wiring pattern of the heating wire when viewed from inside the vehicle 10 looking forward. Fig. 3 also shows the imaging range 9L of the left camera 11L and the imaging range 9R of the right camera 11R.

[0015] The heating wire 8 has a linear shape extending laterally near the imaging range 9L, and similarly has a linear shape extending laterally near the imaging range 9R. The position of the heating wire 8 in the imaging range 9L is different from the position of the heating wire 8 in the imaging range 9R. Specifically, in this example, the heating wire 8 is provided above the vertical center of the imaging range 9L in the imaging range 9L, and is provided below the vertical center of the imaging range 9R in the imaging range 9R. The heating wire 8 near the imaging range 9L and the heating wire 8 near the imaging range 9R are connected to each other via a stepped wiring pattern in the region between the imaging range 9L and the imaging range 9R. In this way, the wiring patterns of the heating wire 8 in the imaging range 9L and the imaging range 9R are different and asymmetric.

[0016] 4 shows an example of a left image PL and a right image PR. In this case, the left image PL generated by the left camera 11L includes a line portion 8L that extends horizontally on the upper side in the vertical direction and has a lower brightness than the surrounding area, and the right image PR generated by the right camera 11R includes a line portion 8R that extends horizontally on the lower side in the vertical direction and has a lower brightness than the surrounding area. These low-brightness line portions 8L and 8R are generated in response to the heating wire 8. That is, the heating wire 8 blocks some of the light from the subject at the line portions 8L and 8R, so the brightness is lower than the surrounding area.

[0017] When the stereo camera 11 is attached to the vehicle 10, the stereo camera 11 may be attached with its orientation slightly misaligned.

[0018] Fig. 5 shows an example of the positional relationship between the imaging ranges 9L and 9R and the heating wire 8 when the stereo camera 11 is installed with a slightly misaligned orientation. Note that Fig. 5 is exaggerated for ease of explanation. In this example, the stereo camera 11 is installed so that the right side is slightly elevated and the left side is slightly lowered. Even in this case, the wiring patterns of the heating wire 8 in the imaging range 9L and the imaging range 9R are different from each other and are asymmetrical.

[0019] Fig. 6 shows an example of the left image PL and the right image PR in the case shown in Fig. 5. In this case, the left image PL generated by the left camera 11L includes a line portion 8L on the upper side in the vertical direction that slopes downward to the right and has a lower brightness than the surrounding area, and the right image PR generated by the right camera 11R includes a line portion 8R on the lower side in the vertical direction that slopes downward to the right and has a lower brightness than the surrounding area.

[0020] Because the windshield 9 is provided with the heating wire 8, the left image PL includes a line portion 8L that is less bright than the surrounding area, and the right image PR includes a line portion 8R that is less bright than the surrounding area. The positions of the line portion 8L in the left image PL and the line portion 8R in the right image PR are different from each other and do not overlap. The stereo camera 11 generates the left image PL and the right image PR and supplies the generated left image PL and right image PR to the processing unit 20.

[0021] The processing unit 20 (FIG. 1) is configured to recognize an object ahead of the vehicle 10 based on the stereo image PIC supplied from the stereo camera 11. The vehicle 10, for example, can perform driving control of the vehicle 10 based on information about the object recognized by the processing unit 20, or display information about the recognized object on a console monitor. The processing unit 20 is configured, for example, with a CPU (Central Processing Unit) that executes a program, a RAM (Random Access Memory) that temporarily stores processed data, and a ROM (Read Only Memory) that stores the program. The processing unit 20 has a heating wire detection unit 30, an image correction unit 21, a parallax image generation unit 22, and an object recognition unit 24.

[0022] The heating wire detection unit 30 is configured to detect the heating wire 8 based on the left image PL and the right image PR. The heating wire detection unit 30 has a sub-image generation unit 31, a brightness ratio map generation unit 32, and a heating wire pattern estimation unit 33.

[0023] The sub-image generating unit 31 is configured to generate a plurality of sub-images PS (three sub-images PSA, PSB, and PSC in this example) by dividing the right image PR in the horizontal direction.

[0024] 7 shows an example of the operation of the sub-image generation unit 31. In this example, the image area of ​​the right image PR is divided into three image areas R (image areas RA, RB, and RC) arranged horizontally. The sub-image generation unit 31 generates a sub-image PSA based on the image in image area RA included in the right image PR, generates a sub-image PSB based on the image in image area RB included in the right image PR, and generates a sub-image PSC based on the image in image area RC included in the right image PR. Note that in this example, three sub-images PS are generated, but this is not limited to this; for example, two sub-images PS or four or more sub-images PS may be generated.

[0025] The brightness ratio map generation unit 32 is configured to generate three pieces of brightness ratio map data MAPA, MAPB, and MAPC regarding the brightness ratios between the left image PL and each of the sub-images PSA, PSB, and PSC. Specifically, the brightness ratio map generation unit 32 detects corresponding points, including corresponding image points in the left image PL and image points in the sub-image PSA, for example, by performing template matching based on the left image PL and the sub-image PSA. The brightness ratio map generation unit 32 repeats this process to detect multiple corresponding points. The brightness ratio map generation unit 32 generates the brightness ratio map data MAPA by using these multiple corresponding points to calculate the ratio of brightness values ​​of corresponding pixels on a pixel-by-pixel basis based on the left image PL and the sub-image PSA. Similarly, the brightness ratio map generation unit 32 generates the brightness ratio map data MAPB based on the left image PL and the sub-image PSB, and generates the brightness ratio map data MAPC based on the left image PL and the sub-image PSC.

[0026] The heating wire pattern estimation unit 33 is configured to estimate the wiring pattern of the heating wires 8 in the imaging range 9L of the left camera 11L and the wiring pattern of the heating wires 8 in the imaging range 9R of the right camera 11R, among the heating wires 8 on the windshield 9, based on the brightness ratio map data MAPA, MAPB, MAPC.

[0027] The image correction unit 21 is configured to generate a left image PL1 and a right image PR1 by correcting the left image PL and the right image PR based on the wiring pattern of the heating wire 8 estimated by the heating wire pattern estimation unit 33. Specifically, the image correction unit 21 increases the luminance values ​​of multiple pixels at the positions of the wiring pattern of the heating wire 8 in the left image PL based on, for example, the wiring pattern of the heating wire 8 in the imaging range 9L estimated by the heating wire pattern estimation unit 33. That is, the image correction unit 21 increases the luminance values ​​of multiple pixels at positions corresponding to the line portions 8L shown in FIGS. 4 and 6. The image correction unit 21 generates a left image PL1 by performing such correction on the left image PL. Similarly, the image correction unit 21 increases the luminance values ​​of multiple pixels at the positions of the wiring pattern of the heating wire 8 in the right image PR based on, for example, the wiring pattern of the heating wire 8 in the imaging range 9R estimated by the heating wire pattern estimation unit 33. That is, the image correction unit 21 increases the luminance values ​​of a plurality of pixels at positions corresponding to the line portion 8R shown in Figures 4 and 6. The image correction unit 21 performs such correction on the right image PR to generate the right image PR1.

[0028] The parallax image generator 22 is configured to generate a parallax image PD by performing predetermined image processing, including stereo matching, on the left image PL1 and the right image PR1. The parallax image PD has a plurality of pixel values. Each of the pixel values ​​indicates a value for the parallax at the corresponding pixel. In other words, each of the pixel values ​​corresponds to a distance to a point corresponding to the corresponding pixel in three-dimensional real space.

[0029] The parallax image generation unit 22 includes a corresponding point detection unit 23. The corresponding point detection unit 23 is configured to detect corresponding points including image points in the left image PL1 and image points in the right image PR1 that correspond to each other by performing stereo matching processing based on the left image PL1 and the right image PR1. The corresponding point detection unit 23 may detect corresponding points by, for example, template matching, or feature amount matching based on local features. The parallax image generation unit 22 is configured to generate a parallax image PD based on the detection result of the corresponding point detection unit 23.

[0030] The object recognition unit 24 is configured to recognize an object ahead of the vehicle 10 based on the left image PL1, the right image PR1, and the parallax image PD generated by the parallax image generation unit 22. The object recognition unit 24 then outputs data on the recognition result.

[0031] Here, the stereo camera 11 corresponds to a specific example of a "stereo camera" in the present disclosure. The sub-image generation unit 31 corresponds to a specific example of a "sub-image generation unit" in the present disclosure. For example, the right image PR corresponds to a specific example of a "first image" in the present disclosure. The brightness ratio map generation unit 32 corresponds to a specific example of a "brightness ratio map generation unit" in the present disclosure. For example, the left image PL corresponds to a specific example of a "second image" in the present disclosure. The heating wire pattern estimation unit 33 corresponds to a specific example of a "heating wire pattern estimation unit" in the present disclosure. The image correction unit 21 corresponds to a specific example of an "image correction unit" in the present disclosure. The parallax image generation unit 22 and the object recognition unit 24 correspond to specific examples of a "processing unit" in the present disclosure.

[0032] [Actions and Actions] Next, the operation and function of the image processing device 1 of this embodiment will be described.

[0033] (Overview of overall operation) First, an overview of the overall operation of the image processing device 1 will be described with reference to FIG. 1. The stereo camera 11 captures images of the area ahead of the vehicle 10 to generate a stereo image PIC including a left image PL and a right image PR. In the processing unit 20, the sub-image generation unit 31 of the heating wire detection unit 30 horizontally divides the right image PR to generate three sub-images PSA, PSB, and PSC. The brightness ratio map generation unit 32 generates three pieces of brightness ratio map data MAPA, MAPB, and MAPC regarding the brightness ratios between the left image PL and each of the sub-images PSA, PSB, and PSC. The heating wire pattern estimation unit 33 estimates, based on the brightness ratio map data MAPA, MAPB, and MAPC, the wiring patterns of the heating wires 8 in the windshield 9 within the imaging range 9L of the left camera 11L and within the imaging range 9R of the right camera 11R. The parallax image generator 22 generates parallax images PD by performing predetermined image processing, including stereo matching, on the left image PL1 and the right image PR1. The object recognizer 24 recognizes objects ahead of the vehicle 10 based on the left image PL1, the right image PR1, and the parallax images PD generated by the parallax image generator 22.

[0034] (Detailed operation) 8 shows an example of the operation of the heating wire detection unit 30. Every time a stereo image PIC is supplied from the stereo camera 11 to the processing unit 20, the heating wire detection unit 30 performs the following process.

[0035] First, the sub-image generating unit 31 of the heating wire detecting unit 30 generates three sub-images PSA, PSB, and PSC by dividing the right image PR in the horizontal direction, as shown in FIG. 7 (step S101).

[0036] Next, the heating wire detection unit 30 performs detection process A (step S102). The heating wire detection unit 30 generates heating wire position data including data about the position of the heating wire 8 in the left image PL and the right image PR based on the left image PL and the sub-images PSA, PSB, and PSC.

[0037] FIG. 9 shows an example of a subroutine of the detection process A.

[0038] First, the brightness ratio map generating unit 32 selects a sub-image PS that has not yet been selected from the three sub-images PSA, PSB, and PSC generated by the sub-image generating unit 31 (step S111).

[0039] Next, the brightness ratio map generation unit 32 detects multiple corresponding points based on the left image PL and the selected sub-image PS (step S112). Specifically, the brightness ratio map generation unit 32 performs template matching processing or the like based on the left image PL and the selected sub-image PS to detect corresponding points including image points in the left image PL and image points in the selected sub-image PS that correspond to each other. The brightness ratio map generation unit 32 repeats this processing to detect multiple corresponding points.

[0040] Next, the brightness ratio map generation unit 32 generates brightness ratio map data MAP by calculating the ratio of brightness values ​​at corresponding pixels on a pixel-by-pixel basis using the corresponding points obtained in step S112 based on the left image PL and the selected sub-image PS (step S113). In this example, the brightness ratio map generation unit 32 calculates the brightness ratio by dividing the brightness value of the left image PL by the brightness value of the sub-image PS. If the selected sub-image PS is sub-image PSA, the brightness ratio map generation unit 32 generates brightness ratio map data MAPA. If the selected sub-image PS is sub-image PSB, the brightness ratio map generation unit 32 generates brightness ratio map data MAPB. If the selected sub-image PS is sub-image PSC, the brightness ratio map generation unit 32 generates brightness ratio map data MAPC.

[0041] Next, the heating wire pattern estimation unit 33 calculates the average value of the luminance ratios for one pixel line for each of the multiple pixel lines extending in the horizontal direction in the luminance ratio map data MAP generated in step S113 (step S114).

[0042] FIG. 10 shows an example of the distribution of average values ​​of luminance ratios in the luminance ratio map data MAP. The average value of the luminance ratio is about "1" for many pixel lines. This indicates that the luminance of corresponding pixels in the left image PL and the right image PR is approximately equal. In this example, the average value of the luminance ratio is projected in the direction smaller than "1" near pixel line L1, and is projected in the direction smaller than "1" near pixel line L2. Nearby In this case, it protrudes in a direction greater than "1".

[0043] Next, the heating wire pattern estimation unit 33 identifies a pixel line corresponding to the position of the heating wire 8 based on the result obtained in step S114 (step S115). Specifically, the heating wire pattern estimation unit 33 identifies a pixel line corresponding to the position of the heating wire 8 based on the result obtained in step S114, for example, by using a technique such as pattern patching or machine learning.

[0044] FIG. 11 shows an image in image region RA of the left image PL shown in FIG. 6. FIG. 12 shows a sub-image PSA in image region RA of the right image PR shown in FIG. 6. Near pixel line L1, there is a line portion 8L in the left image PL, but there is no line portion 8R in the right image PR (sub-image PS). This corresponds to the fact that in FIG. 10, the average luminance ratio near pixel line L1 is projected in a direction smaller than "1." Near pixel line L2, there is a line portion 8R in the right image PR (sub-image PS), but there is no line portion 8L in the left image PL. This corresponds to the fact that in FIG. 10, the average luminance ratio near pixel line L2 is projected in a direction larger than "1."

[0045] In this way, the heating wire pattern estimation unit 33 identifies pixel lines corresponding to the position of the heating wire 8 in the left image PL, and also identifies pixel lines corresponding to the position of the heating wire 8 in the sub-image PS, which is part of the right image PR.

[0046] If the left image PL and the right image PR are color images, the heating wire detection unit 30 can perform the processes of steps S112 to S115 based on the luminance values ​​of red (R), green (G), and blue (B). Specifically, first, in step S112, the luminance ratio map generation unit 32 detects multiple corresponding points based on, for example, the red image of the left image PL and the red image of the selected sub-image PS. In step S113, the luminance ratio map generation unit 32 generates luminance ratio map data for the ratio of red luminance values ​​based on the red image of the left image PL and the red image of the sub-image PS. In step S114, the heating wire pattern estimation unit 33 calculates the average value of the luminance ratio for one pixel line for each of the multiple pixel lines in the luminance ratio map data for red. In this way, the heating wire pattern estimation unit 33 obtains the distribution of average values ​​of the luminance ratio for red, as shown in FIG. 10. Similarly, the heating wire detection unit 30 obtains a distribution of average values ​​of luminance ratios related to green and a distribution of average values ​​of luminance ratios related to blue. Then, in step S115, the heating wire pattern estimation unit 33 can identify pixel lines corresponding to the position of the heating wire 8 based on the distribution of average values ​​of luminance ratios related to red, green, and blue. Note that while this example has been described using the RGB color space as an example, this is not limiting and other color spaces such as YUV may be used instead. The left image PL and the right image PR may also be monochrome images.

[0047] Next, the heating wire detection unit 30 checks whether all sub-images PS have been selected in step S111 (step S116). If all sub-images PS have not yet been selected ("N" in step S116), the process returns to step S111, and steps S111 to S116 are repeated until all sub-images PS have been selected. If all sub-images PS have been selected ("Y" in step S116), this subroutine of detection process A ends.

[0048] In this way, based on the left image PL and right image PR included in one stereo image PIC, the heating wire detection unit 30 identifies pixel lines corresponding to the position of the heating wire 8 in each of the three image regions RA to RC of the left image PL, and identifies pixel lines corresponding to the position of the heating wire 8 in each of the three image regions RA to RC of the right image PR. Then, the heating wire detection unit 30 stores data on these pixel lines as heating wire position data.

[0049] Next, as shown in Fig. 8, it is confirmed whether or not a predetermined number of pieces of heating wire position data have been obtained (step S103). If the predetermined number of pieces of heating wire position data have not yet been obtained ("N" in step S103), this flow ends. The process of Fig. 8 is performed each time a stereo image PIC is supplied from the stereo camera 11 to the processing unit 20. Therefore, based on each of the multiple stereo images PIC supplied sequentially, a pixel line corresponding to the position of the heating wire 8 is identified, and the heating wire position data is stored sequentially. In this way, the heating wire position data is accumulated.

[0050] In step S103, if a predetermined number or more of heating wire position data are obtained ("Y" in step S103), the heating wire pattern estimation unit 33 calculates the variance value of the positions of pixel lines corresponding to the positions of the heating wire 8 in each of the image areas RA, RB, and RC of the left image PL based on the accumulated heating wire position data, and calculates the variance value of the positions of pixel lines corresponding to the positions of the heating wire 8 in each of the image areas RA, RB, and RC of the right image PR (step S104).

[0051] Next, the heating wire pattern estimation unit 33 checks whether all of the six variance values ​​obtained in step S104 are within a predetermined threshold value (step S105). If this condition is not met ("N" in step S105), this flow ends.

[0052] In step S105, if all six variance values ​​obtained in step S104 are within a predetermined threshold value ("Y" in step S105), the heating wire pattern estimation unit 33 determines a representative position of pixel lines corresponding to the position of the heating wire 8 in each of the image regions RA, RB, and RC of the left image PL based on the accumulated heating wire position data, and also determines a representative position of pixel lines corresponding to the position of the heating wire 8 in each of the image regions RA, RB, and RC of the right image PR (step S106). The representative position may be, for example, an average position, or a position corresponding to the most frequent value when a histogram of the positions of the heating wire 8 is generated.

[0053] FIG. 13 shows the representative positions of pixel lines obtained in step S106. In this example, the left image PL and the right image PR are the images shown in FIG. 6. The thick dotted lines indicate the pixel lines indicated by the representative positions. In the left image PL, in each of the image regions RA, RB, and RC, the pixel lines indicated by the representative positions are located on the upper side of the left image PL in the vertical direction, with the pixel line in image region RA at the top and the pixel line in image region RC at the bottom. These pixel lines are located along a line portion 8L that is lower in brightness than its surroundings. In the right image PR, in each of the image regions RA, RB, and RC, the pixel lines indicated by the representative positions are located on the lower side of the right image PR in the vertical direction, with the pixel line in image region RA at the top and the pixel line in image region RC at the bottom. These pixel lines are located along a line portion 8R that is lower in brightness than its surroundings.

[0054] Next, the heating wire pattern estimation unit 33 estimates the wiring pattern of the heating wire 8 based on the representative position of the pixel line obtained in step S106 (step S107). As shown in FIG. 13, in the left image PL, the three pixel lines indicated by the representative position are located along a line portion 8L that is less bright than its surroundings. Therefore, the heating wire pattern estimation unit 33 can obtain the position and orientation of the line portion 8L based on this representative position, and can estimate the wiring pattern of the heating wire 8 in the imaging range 9L of the left camera 11L. Similarly, in the right image PR, the three pixel lines indicated by the representative position are located along a line portion 8R that is less bright than its surroundings. Therefore, the heating wire pattern estimation unit 33 can obtain the position and orientation of the line portion 8R based on this representative position, and can estimate the wiring pattern of the heating wire 8 in the imaging range 9R of the right camera 11R. The wiring pattern of the heating wire 8 can be expressed, for example, using a function.

[0055] This is the end of this flow.

[0056] In this way, the heating wire detection unit 30 estimates the wiring pattern of the heating wire 8. The image correction unit 21 generates a left image PL1 and a right image PR1 by correcting the left image PL and the right image PR based on the wiring pattern of the heating wire 8 estimated by the heating wire pattern estimation unit 33. Specifically, the image correction unit 21 increases the luminance values ​​of multiple pixels at the positions of the wiring pattern of the heating wire 8 in the left image PL based on, for example, the wiring pattern of the heating wire 8 in the imaging range 9L estimated by the heating wire pattern estimation unit 33. That is, the image correction unit 21 increases the luminance values ​​of multiple pixels at positions corresponding to the line portion 8L shown in FIGS. 4 and 6. The image correction unit 21 generates a left image PL1 by performing this correction on the left image PL. Similarly, the image correction unit 21 increases the luminance values ​​of multiple pixels at the positions of the wiring pattern of the heating wire 8 in the right image PR based on, for example, the wiring pattern of the heating wire 8 in the imaging range 9R estimated by the heating wire pattern estimation unit 33. That is, the image correcting unit 21 increases the luminance values ​​of a plurality of pixels at positions corresponding to the line portion 8R shown in Figures 4 and 6. The image correcting unit 21 performs such correction on the right image PR to generate a right image PR1.

[0057] As described above, the image processing device 1 includes a sub-image generation unit 31 that generates three sub-images PS by horizontally dividing the right image PR, which is one of the left image PL and right image PR captured by the stereo camera 11 through the windshield 9 on which the heating wire 8 is formed; a brightness ratio map generation unit 32 that sequentially selects one of the three sub-images PS and generates brightness ratio map data MAP indicating the ratio of brightness values ​​based on the selected sub-image PS and the left image PL, which is the other of the left image PL and right image PR; and a heating wire pattern estimation unit 33 that estimates the wiring pattern of the heating wire 8 based on the brightness ratio map data MAP for each of the three sub-images PS. As a result, the image processing device 1, for example, can have the image correction unit 21 correct the left image PL and right image PR based on the estimated wiring pattern of the heating wire 8, thereby reducing the influence of the heating wire 8 on the corrected left image PL and right image PR. This also reduces the influence of the heating wire 8 on various image processing operations based on the corrected left image PL and right image PR. In this way, the image processing device 1 can reduce the influence of the heating wire 8 on the results of image processing.

[0058] In the image processing device 1, the multiple sub-images PS include a first sub-image (e.g., sub-image PSA) and a second sub-image (e.g., sub-image PSC), and the heating wire pattern estimation unit 33 calculates a first representative value that is a representative value of the luminance ratios for one pixel line for each of multiple horizontally extending pixel lines in the luminance ratio map data MAP (e.g., luminance ratio map data MAPA) for the first sub-image (e.g., sub-image PSA), and calculates a second representative value that is a representative value of the luminance ratios for one pixel line for each of multiple horizontally extending pixel lines in the luminance value map data MAP (luminance ratio map data MAPC) for the second sub-image (e.g., sub-image PSC), and estimates the wiring pattern of the heating wires based on the multiple first representative values ​​and the multiple second representative values. As a result, the image processing device 1 can obtain a distribution of representative luminance values ​​as shown in FIG. 10 in, for example, the image region RA of the sub-image PSA and the image region RC of the sub-image PSC. Therefore, for example, by identifying pixel lines that protrude in a direction where the representative brightness value is greater than "1," it is possible to estimate the wiring pattern of the heating wire 8 in the imaging range 9R of the right camera 11R. Similarly, for example, by identifying pixel lines that protrude in a direction where the representative brightness value is less than "1," it is possible to estimate the wiring pattern of the heating wire 8 in the imaging range 9L of the left camera 11L. As a result, as shown in FIG. 13, it is possible to estimate an inclined wiring pattern. As a result, in the image processing device 1, for example, the image correction unit 21 can correct the left image PL and the right image PR based on the estimated wiring pattern of the heating wire 8, thereby reducing the influence of the heating wire 8 in the corrected left image PL and right image PR.

[0059] [effect] As described above, this embodiment includes a sub-image generation unit that generates three sub-images by horizontally dividing the right image, which is one of the left and right images captured by the stereo camera through the windshield on which the heating wire is formed, a brightness ratio map generation unit that sequentially selects one of the three sub-images and generates brightness ratio map data for the ratio of brightness values ​​based on the selected sub-image and the left image, which is the other of the left and right images, and a heating wire pattern estimation unit that estimates the wiring pattern of the heating wire based on the brightness ratio map data for each of the three sub-images. This makes it possible to correct the left and right images based on the estimated wiring pattern of the heating wire, thereby reducing the influence of the heating wire on the processing results of the image processing.

[0060] In this embodiment is complex The number of sub-images includes a first sub-image and a second sub-image, and the heating wire pattern estimation unit calculates a first representative value that is a representative value of the luminance ratios for one pixel line for each of a plurality of horizontally extending pixel lines in the luminance ratio map data for the first sub-image, calculates a second representative value that is a representative value of the luminance ratios for one pixel line for each of a plurality of horizontally extending pixel lines in the luminance value map data for the second sub-image, and estimates the wiring pattern of the heating wires based on the plurality of first representative values ​​and the plurality of second representative values, thereby reducing the influence of the heating wires on the processing results of the image processing.

[0061] [Variation 1] In the above embodiment, the wiring pattern of the heating wire 8 is the wiring pattern shown in Fig. 3, but this is not limited to this and various asymmetric wiring patterns can be used. For example, as shown in Fig. 14, the heating wire 8 may have a linear shape so as to penetrate the imaging range 9L and the imaging range 9R and extend in an oblique direction. Even in this case, the wiring patterns of the heating wire 8 in the imaging range 9L and the imaging range 9R are different from each other and are asymmetric.

[0062] 15, the heating wires 8 may be provided separately in the imaging range 9L and the imaging range 9R. The heating wires 8 include a heating wire 8A provided in the imaging range 9L and a heating wire 8B provided in the imaging range 9R. The heating wire 8A crosses the imaging range 9L and has a shape that moves back and forth in the left-right direction within this imaging range 9L. The heating wire 8B crosses the imaging range 9R and has a shape that moves back and forth in the left-right direction within this imaging range 9R. Even in this case, the wiring patterns of the heating wires 8 in the imaging range 9L and the imaging range 9R are different from each other and are asymmetrical.

[0063] 16, the heating wire 8 may be provided only in one of the imaging range 9L and the imaging range 9R. In this example, the heating wire 8 is provided only in the imaging range 9L. The heating wire 8 has a shape that crosses the imaging range 9L and moves back and forth in the left-right direction within this imaging range 9L.

[0064] [Variation 2] In the above embodiment, the sub-image generation unit 31 generates multiple sub-images PS by dividing the right image PR horizontally, and the brightness ratio map generation unit 32 generates three pieces of brightness ratio map data MAPA, MAPB, and MAPC for the brightness ratios between the left image PL and each of the sub-images PSA, PSB, and PSC, but this is not limited to this. Alternatively, the sub-image generation unit 31 may generate multiple sub-images PS by dividing the left image PL horizontally, and the brightness ratio map generation unit 32 may generate three pieces of brightness ratio map data MAPA, MAPB, and MAPC for the brightness ratios between the right image PR and each of the sub-images PSA, PSB, and PSC.

[0065] [Variation 3] In the above embodiment, the image correction unit 21 is provided, and this image correction unit 21 corrects the left image PL and the right image PR based on the wiring pattern of the heating wire 8 estimated by the heating wire pattern estimation unit 33, but the present invention is not limited to this. An image processing device 1C according to this modified example will be described in detail below.

[0066] 17 shows an example of the configuration of an image processing device 1C. The image processing device 1C includes a processing unit 20C. The processing unit 20C includes a heating wire detection unit 30, a parallax image generation unit 22C, and an object recognition unit 24C. That is, the processing unit 20C is obtained by omitting the image correction unit 21 from the processing unit 20 (FIG. 1) according to the above embodiment, and by replacing the parallax image generation unit 22 and the object recognition unit 24 with the parallax image generation unit 22C and the object recognition unit 24C, respectively.

[0067] Similar to the parallax image generation unit 22 according to the above embodiment, the parallax image generation unit 22C is configured to generate a parallax image PD by performing predetermined image processing including stereo matching processing based on the left image PL and the right image PR.

[0068] The parallax image generation unit 22C includes a corresponding point detection unit 23C. The corresponding point detection unit 23C is configured to perform stereo matching based on the left image PL and the right image PR to detect corresponding points including corresponding image points in the left image PL and the right image PR. When matching conditions for the stereo matching process are met, the corresponding point detection unit 23C detects corresponding points including corresponding image points in the left image PL and the right image PR. The corresponding point detection unit 23C relaxes the matching conditions around the wiring pattern of the heating wire 8 estimated by the heating wire pattern estimation unit 33. That is, as shown in FIGS. 4 and 6, for example, the position of the line portion 8L in the left image PL and the position of the line portion 8R in the right image PR are different from each other, which may make it difficult to detect corresponding points using the stereo matching process. Therefore, the corresponding point detection unit 23C relaxes the matching conditions around the wiring pattern of the heating wire 8 to reduce the possibility of difficulty in detecting corresponding points. This allows the corresponding point detection unit 23C to effectively perform stereo matching processing even when the line portions 8L and 8R are present.

[0069] The object recognition unit 24C is configured to recognize an object ahead of the vehicle 10 based on the left image PL, the right image PR, and the parallax image PD generated by the parallax image generation unit 22C. The object recognition unit 24C recognizes an object when a predetermined processing condition is satisfied. The object recognition unit 24C relaxes the processing condition around the wiring pattern of the heating wire 8 based on the wiring pattern estimated by the heating wire pattern estimation unit 33. That is, it may be difficult to recognize an object near the line portion 8L in the left image PL or the line portion 8R in the right image PR. Therefore, the object recognition unit 24C reduces the possibility of difficulty in recognizing an object by relaxing the processing condition around the wiring pattern of the heating wire 8. This allows the object recognition unit 24C to effectively recognize an object even when the line portions 8L and 8R are present.

[0070] Here, the parallax image generating unit 22C and the object recognizing unit 24C correspond to a specific but not limitative example of a "processing unit" in the disclosure.

[0071] [Other variations] Two or more of these variations may also be combined.

[0072] The present technology has been described above by giving embodiments and some modified examples, but the present technology is not limited to these embodiments and can be modified in various ways.

[0073] For example, in the above embodiment, the stereo camera 11 is configured to capture an image in front of the vehicle 10, but this is not limited thereto. For example, the stereo camera 11 may capture an image of the side or rear of the vehicle 10.

[0074] The effects described in this specification are merely examples and are not limiting, and other effects may also be present. [Explanation of symbols]

[0075] 1,1C...image processing device, 8,8A,8B...heating wire, 8L,8R...line portion, 9...windshield, 9L,9R...imaging range, 10...vehicle, 11...stereo camera, 11L...left camera, 11R...right camera, 20,20C...processing unit, 21...image correction unit, 22,22C...parallax image generation unit, 23,23C...corresponding point detection unit, 24,24C...object recognition unit, 30...heating wire detection unit, 31...sub-image generation unit, 32...brightness ratio map generation unit, 33...heating wire pattern estimation unit, MAP...brightness ratio map data, PIC...stereo image, PL,PL1...left image, PR,PR1...right image, PS,PSA,PSB,PSC...sub-image, RA,RB,RC...image area.

Claims

1. a sub-image generating unit that generates a plurality of sub-images by horizontally dividing a first image, which is one of a left image and a right image captured by the stereo camera through a windshield on which a heating wire is formed; a brightness ratio map generator that sequentially selects one of the plurality of sub-images and generates map data indicating a ratio of brightness values ​​based on the selected sub-image and a second image that is the other of the left image and the right image; a heating wire pattern estimation unit that estimates a wiring pattern of the heating wire based on the map data for each of the plurality of sub-images; An image processing device comprising:

2. the plurality of sub-images includes a first sub-image and a second sub-image; The heating wire pattern estimation unit calculating a first representative value that is a representative value of luminance ratios for one pixel line in each of a plurality of pixel lines extending in a horizontal direction in the map data for the first sub-image; calculating a second representative value that is a representative value of the luminance ratios of one pixel line for each of a plurality of pixel lines extending in a horizontal direction in the map data for the second sub-image; A wiring pattern of the heating wire is estimated based on the plurality of first representative values ​​and the plurality of second representative values. The image processing device according to claim 1 .

3. an image correction unit that corrects image portions of the left image and the right image that relate to the wiring pattern of the heating wire based on the estimation result of the heating wire pattern estimation unit; a processing unit that performs predetermined image processing based on the left image and the right image corrected by the image correction unit; 3. The image processing device according to claim 1, further comprising:

4. a processing unit that performs predetermined image processing based on the left image and the right image when a predetermined processing condition is satisfied, The processing unit relaxes the processing conditions for image portions related to the wiring pattern of the heating wire in the left image and the right image based on the estimation result of the heating wire pattern estimation unit.

3. The image processing device according to claim 1.

5. The wiring patterns of the heating wires in the imaging range of the left camera that generates the left image and the imaging range of the right camera that generates the right image are asymmetrical with each other. The image processing device according to any one of claims 1 to 4.

Images