Image processing apparatus

Abstract

To provide an image processing apparatus which can reduce an influence of a heating wire on a processing result of image processing.SOLUTION: The image processing apparatus relating to one embodiment of the present disclosure comprises: a sub-image generation unit which generates a plurality of sub-images by laterally dividing a first image being one of a left image and a right image captured by a stereo camera, through a front glass having a heating wire formed therein; a brightness ratio map generation unit which selects one of the plurality of sub-images in sequence and generates map data showing a ratio of brightness values on the basis of the selected sub-image and a second image being the other of the left image and the right image; and a heating wire pattern estimation unit which estimates a wiring pattern of the heating wire on the basis of map data for the respective sub-images.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] ,

[0006] , , , , , ,

[0005] , , , , , ,

[0001] The present disclosure relates to an image processing apparatus that performs image processing based on stereoscopic images.

Background Art

[0002] Some vehicles such as automobiles detect surrounding objects based on captured images obtained by an imaging device. Also, some vehicles have electric heating wires provided on the glass so that, for example, the front glass and the rear glass do not become cloudy. For example, Patent Document 1 discloses a technique for recognizing a portion that does not change over time as an electric heating wire in a captured image of an in-vehicle camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an image processing apparatus, it is desirable to reduce the influence of the electric heating wire on the processing result of image processing based on a captured image, and further reduction of the influence on the image processing result is expected.

[0005] It is desirable to provide an image processing apparatus capable of reducing the influence of the electric heating wire on the processing result of image processing based on a captured image.

Means for Solving the Problems

[0006] An image processing apparatus according to one embodiment of the present disclosure comprises a sub-image generation unit, a luminance ratio map generation unit, and a heating wire pattern estimation unit. The sub-image generation unit is configured to generate a plurality of sub-images by horizontally dividing a first image, which is one of the left image and the right image captured by a stereo camera through a windshield on which heating wires are formed. The luminance ratio map generation unit is configured to sequentially select one of the plurality of sub-images and generate map data indicating the ratio of luminance values ​​based on the selected sub-image and a second image, which is the other of the left image and the right image. The heating wire pattern estimation unit is configured to estimate the wiring pattern of the heating wires based on the map data for each of the plurality of sub-images. [Effects of the Invention]

[0007] According to an image processing apparatus according to one embodiment of the present disclosure, the influence of the heating element on the processing result of the image processing can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of an image processing apparatus according to one embodiment of the present disclosure. [Figure 2] This is an explanatory diagram illustrating an example of the arrangement of the stereo cameras shown in Figure 1. [Figure 3] Figure 2 is an explanatory diagram illustrating an example of a wiring pattern for a heating element. [Figure 4] This is an explanatory diagram showing an example of the left and right images. [Figure 5] This is an explanatory diagram showing another example of the wiring pattern for the heating element shown in Figure 2. [Figure 6] This is an explanatory diagram showing an example of the left and right images. [Figure 7] Figure 1 is an explanatory diagram illustrating an example of the operation of the sub-image generation unit. [Figure 8] Figure 1 is a flowchart illustrating an example of the operation of the heating element detection unit. [Figure 9] Figure 8 is a flowchart illustrating an example of the detection process. [Figure 10] It is a characteristic diagram showing an example of the detection result of the heating wire detection unit shown in FIG. 1. [Figure 11] It is an explanatory diagram showing an example of an operation of the heating wire detection unit shown in FIG. 1. [Figure 12] It is another explanatory diagram showing an example of an operation of the heating wire detection unit shown in FIG. 1. [Figure 13] It is another explanatory diagram showing an example of an operation of the heating wire detection unit shown in FIG. 1. [Figure 14] It is an explanatory diagram showing an example of the wiring pattern of the heating wire according to the modification example. [Figure 15] It is an explanatory diagram showing an example of the wiring pattern of the heating wire according to another modification example. [Figure 16] It is an explanatory diagram showing an example of the wiring pattern of the heating wire according to another modification example. [Figure 17] It is a block diagram showing an example of a configuration of an image processing apparatus according to another modification example.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] <Embodiment> [Configuration Example] FIG. 1 shows a configuration example of an image processing apparatus (image processing apparatus 1) according to an embodiment. The image processing apparatus 1 includes a stereo camera 11 and a processing unit 20. In this example, the image processing apparatus 1 is mounted on a vehicle 10 such as an automobile.

[0011] The stereo camera 11 is configured to generate a set of images (left image PL and right image PR) having parallax with each other by imaging the front of the vehicle 10. The stereo camera 11 includes a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor.

[0012] Figure 2 shows an example of the arrangement of the stereo camera 11 in the vehicle 10. In this example, the left camera 11L and the right camera 11R are arranged inside the vehicle 10 near the upper part of the windshield 9 of the vehicle 10, separated from each other by a predetermined distance in the width direction of the vehicle 10. The left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL and the right image PR constitute a stereo image PIC. The stereo camera 11 performs an imaging operation at a predetermined frame rate (for example, 60 [fps]) to generate a series of stereo images PIC, and supplies the generated stereo images PIC to the processing unit 20.

[0013] On the windshield 9, a heating wire 8 is provided in the front portion of the left camera 11L and the right camera 11R. The heating wire 8 is configured to generate heat when an electric current flows from a drive circuit (not shown). In the vehicle 10, by the heating wire 8 generating heat, the windshield 9 can be prevented from becoming cloudy.

[0014] Figure 3 shows an example of the wiring pattern of the heating wire 8 on the windshield 9. Figure 3 shows the wiring pattern of the heating wire when viewed from the inside of the vehicle 10 towards the front of the vehicle 10. In this Figure 3, the imaging range 9L of the left camera 11L and the imaging range 9R of the right camera 11R are also drawn.

[0015] The heating wire 8 has a linear shape extending horizontally in the vicinity of the imaging range 9L, and similarly has a linear shape extending horizontally in the vicinity of the imaging range 9R. The position of the heating wire 8 in the imaging range 9L is different from the position of the heating wire 8 in the imaging range 9R. Specifically, in this example, the heating wire 8 is provided above the center in the vertical direction of the imaging range 9L in the imaging range 9L, and is provided below the center in the vertical direction of the imaging range 9R in the imaging range 9R. The heating wire 8 near the imaging range 9L and the heating wire 8 near the imaging range 9R are connected to each other through a stepped wiring pattern in the region between the imaging range 9L and the imaging range 9R. Thus, the wiring patterns of the heating wire 8 in the imaging range 9L and the imaging range 9R are different from each other and asymmetric.

[0016] Figure 4 shows an example of a left image PL and a right image PR. In this case, the left image PL, generated by the left camera 11L, includes a line portion 8L that extends horizontally on the upper side in the vertical direction and has lower brightness than the surrounding area, and the right image PR, generated by the right camera 11R, includes a line portion 8R that extends horizontally on the lower side in the vertical direction and has lower brightness than the surrounding area. These low-brightness line portions 8L and 8R are generated in response to the heating element 8. That is, in the line portions 8L and 8R, some of the light from the subject is blocked by the heating element 8, so the brightness is lower than the surrounding area.

[0017] When the stereo camera 11 is attached to the vehicle 10, it is possible that the stereo camera 11 may be attached with a slight misalignment.

[0018] Figure 5 shows an example of the positional relationship between the imaging ranges 9L and 9R and the heating element 8 when the stereo camera 11 is mounted with a slight misalignment. Note that Figure 5 is exaggerated for illustrative purposes. In this example, the stereo camera 11 is mounted so that the right side is slightly higher and the left side is slightly lower. Even in this case, the wiring patterns of the heating element 8 in imaging ranges 9L and 9R are different and asymmetrical.

[0019] Figure 6 shows an example of the left image PL and right image PR in the case shown in Figure 5. In this case, the left image PL generated by the left camera 11L includes a line portion 8L on the upper side in the vertical direction that slopes downward to the right and has lower brightness than the surrounding area, and the right image PR generated by the right camera 11R includes a line portion 8R on the lower side in the vertical direction that slopes downward to the right and has lower brightness than the surrounding area.

[0020] As described above, since heating elements 8 are provided on the windshield 9, the left image PL includes a line portion 8L with lower brightness than the surrounding area, and the right image PR includes a line portion 8R with lower brightness than the surrounding area. The positions of the line portion 8L in the left image PL and the line portion 8R in the right image PR are different from each other and do not overlap. The stereo camera 11 generates such a left image PL and a right image PR, and supplies the generated left image PL and right image PR to the processing unit 20.

[0021] The processing unit 20 (Figure 1) is configured to recognize objects in front of the vehicle 10 based on the stereo image PIC supplied from the stereo camera 11. The vehicle 10 can, for example, control its driving based on the information about the object recognized by the processing unit 20, or display information about the recognized object on a console monitor. The processing unit 20 is composed of, for example, a CPU (Central Processing Unit) for executing programs, a RAM (Random Access Memory) for temporarily storing processing data, and a ROM (Read Only Memory) for storing programs. The processing unit 20 includes a heating element detection unit 30, an image correction unit 21, a disparity image generation unit 22, and an object recognition unit 24.

[0022] The heating element detection unit 30 is configured to detect the heating element 8 based on the left image PL and the right image PR. The heating element detection unit 30 includes a sub-image generation unit 31, a brightness ratio map generation unit 32, and a heating element pattern estimation unit 33.

[0023] The sub-image generation unit 31 is configured to generate multiple sub-images PS (in this example, three sub-images PSA, PSB, and PSC) by dividing the right image PR in the horizontal direction.

[0024] Figure 7 shows an example of the operation of the sub-image generation unit 31. In this example, the image area of ​​the right image PR is divided into three horizontally aligned image areas R (image areas RA, RB, RC). The sub-image generation unit 31 generates a sub-image PSA based on the image in image area RA included in the right image PR, generates a sub-image PSB based on the image in image area RB included in the right image PR, and generates a sub-image PSC based on the image in image area RC included in the right image PR. In this example, three sub-images PS were generated, but this is not limited to this; for example, two sub-images PS may be generated, or four or more sub-images PS may be generated.

[0025] The luminance ratio map generation unit 32 is configured to generate three luminance ratio map data sets, MAPA, MAPB, and MAPC, for the luminance ratio between the left image PL and the sub-images PSA, PSB, and PSC, respectively. Specifically, the luminance ratio map generation unit 32 detects corresponding points, including image points in the left image PL and image points in the sub-image PSA, by performing, for example, template matching processing based on the left image PL and sub-image PSA. The luminance ratio map generation unit 32 detects multiple corresponding points by repeating this process. Based on the left image PL and sub-image PSA, the luminance ratio map generation unit 32 generates the luminance ratio map data MAPA by calculating the ratio of luminance values ​​in corresponding pixels on a pixel-by-pixel basis using these multiple corresponding points. Similarly, the luminance ratio map generation unit 32 generates the luminance ratio map data MAPB based on the left image PL and sub-image PSB, and generates the luminance ratio map data MAPC based on the left image PL and sub-image PSC.

[0026] The heating wire pattern estimation unit 33 is configured to estimate the wiring pattern of the heating wires 8 on the windshield 9 in the imaging range 9L of the left camera 11L and the wiring pattern of the heating wires 8 in the imaging range 9R of the right camera 11R, based on the brightness ratio map data MAPA, MAPB, and MAPC.

[0027] The image correction unit 21 is configured to generate left image PL1 and right image PR1 by correcting the left image PL and right image PR based on the wiring pattern of the heating wire 8 estimated by the heating wire pattern estimation unit 33. Specifically, the image correction unit 21 increases the brightness values ​​of multiple pixels at the locations of the wiring pattern of the heating wire 8 in the left image PL, based on the wiring pattern of the heating wire 8 in the imaging range 9L estimated by the heating wire pattern estimation unit 33. That is, the image correction unit 21 increases the brightness values ​​of multiple pixels at the locations corresponding to the line portion 8L shown in Figures 4 and 6. The image correction unit 21 generates left image PL1 by performing such correction on the left image PL. Similarly, the image correction unit 21 increases the brightness values ​​of multiple pixels at the locations of the wiring pattern of the heating wire 8 in the right image PR, based on the wiring pattern of the heating wire 8 in the imaging range 9R estimated by the heating wire pattern estimation unit 33. In other words, the image correction unit 21 increases the brightness values ​​of multiple pixels at positions corresponding to the line portion 8R shown in Figures 4 and 6. The image correction unit 21 generates the right image PR1 by performing such correction on the right image PR.

[0028] The disparity image generation unit 22 is configured to generate a disparity image PD by performing predetermined image processing, including stereo matching processing, based on the left image PL1 and the right image PR1. The disparity image PD has multiple pixel values. Each of the multiple pixel values ​​represents a value for the disparity at each pixel. In other words, each of the multiple pixel values ​​corresponds to the distance to the point corresponding to each pixel in three-dimensional real space.

[0029] The disparity image generation unit 22 includes a correspondence point detection unit 23. The correspondence point detection unit 23 is configured to detect corresponding points, including image points in the left image PL1 and the right image PR1, by performing stereo matching processing based on the left image PL1 and the right image PR1. The correspondence point detection unit 23 may detect corresponding points by, for example, template matching, or by feature matching based on local features. The disparity image generation unit 22 generates a disparity image PD based on the detection results of the correspondence point detection unit 23.

[0030] The object recognition unit 24 is configured to recognize an object in front of the vehicle 10 based on the left image PL1, the right image PR1, and the disparity image PD generated by the disparity image generation unit 22. The object recognition unit 24 then outputs data about the recognition result.

[0031] Here, the stereo camera 11 corresponds to one specific example of the "stereo camera" in this disclosure. The sub-image generation unit 31 corresponds to one specific example of the "sub-image generation unit" in this disclosure. For example, the right image PR corresponds to one specific example of the "first image" in this disclosure. The luminance ratio map generation unit 32 corresponds to one specific example of the "luminance ratio map generation unit" in this disclosure. For example, the left image PL corresponds to one specific example of the "second image" in this disclosure. The heating wire pattern estimation unit 33 corresponds to one specific example of the "heating wire pattern estimation unit" in this disclosure. The image correction unit 21 corresponds to one specific example of the "image correction unit" in this disclosure. The disparity image generation unit 22 and the object recognition unit 24 correspond to one specific example of the "processing unit" in this disclosure.

[0032] [Action and function] Next, the operation and function of the image processing apparatus 1 of this embodiment will be described.

[0033] (Overview of overall operation) First, with reference to Figure 1, the overall operation of the image processing device 1 will be explained. The stereo camera 11 generates a stereo image PIC, which includes the left image PL and the right image PR, by capturing images of the area in front of the vehicle 10. In the processing unit 20, the sub-image generation unit 31 of the heating element detection unit 30 generates three sub-images PSA, PSB, and PSC by dividing the right image PR horizontally. The luminance ratio map generation unit 32 generates three luminance ratio map data MAPA, MAPB, and MAPC for the luminance ratio between the left image PL and each of the sub-images PSA, PSB, and PSC. The heating element pattern estimation unit 33 estimates the wiring pattern of the heating element 8 in the windshield 9, specifically the wiring pattern of the heating element 8 in the imaging range 9L of the left camera 11L and the wiring pattern of the heating element 8 in the imaging range 9R of the right camera 11R, based on the luminance ratio map data MAPA, MAPB, and MAPC. The parallax image generation unit 22 generates a parallax image PD by performing predetermined image processing, including stereo matching, based on the left image PL1 and the right image PR1. The object recognition unit 24 recognizes an object in front of the vehicle 10 based on the left image PL1 and the right image PR1, and the parallax image PD generated by the parallax image generation unit 22.

[0034] (Detailed operation) Figure 8 shows an example of the operation of the heating element detection unit 30. The heating element detection unit 30 performs the following processing each time a stereo image PIC is supplied from the stereo camera 11 to the processing unit 20.

[0035] First, the sub-image generation unit 31 of the heating element detection unit 30 generates three sub-images PSA, PSB, and PSC by dividing the right image PR horizontally, as shown in Figure 7 (step S101).

[0036] Next, the heating element detection unit 30 performs detection process A (step S102). Based on the left image PL and sub-images PSA, PSB, PSC, the heating element detection unit 30 generates heating element position data that includes data about the position of the heating element 8 in the left image PL and the right image PR.

[0037] Figure 9 shows an example of a subroutine for detection process A.

[0038] First, the luminance ratio map generation unit 32 selects the sub-image PS, which has not yet been selected from the three sub-images PSA, PSB, and PSC generated by the sub-image generation unit 31 (step S111).

[0039] Next, the luminance ratio map generation unit 32 detects multiple corresponding points based on the left image PL and the selected sub-image PS (step S112). Specifically, the luminance ratio map generation unit 32 detects corresponding points, including image points in the left image PL and image points in the selected sub-image PS, by performing template matching processing and the like based on the left image PL and the selected sub-image PS. The luminance ratio map generation unit 32 detects multiple corresponding points by repeating this process.

[0040] Next, the luminance ratio map generation unit 32 generates luminance ratio map data MAP (step S113) by calculating the ratio of luminance values ​​at corresponding pixels on a pixel-by-pixel basis using a plurality of corresponding points obtained in step S112, based on the left image PL and the selected sub-image PS. In this example, the luminance ratio map generation unit 32 calculates the ratio of luminance values ​​by dividing the luminance value of the left image PL by the luminance value of the sub-image PS. The luminance ratio map generation unit 32 generates luminance ratio map data MAPA if the selected sub-image PS is sub-image PSA, generates luminance ratio map data MAPB if the selected sub-image PS is sub-image PSB, and generates luminance ratio map data MAPC if the selected sub-image PS is sub-image PSC.

[0041] Next, the heating wire pattern estimation unit 33 calculates the average value of the luminance ratio for one pixel line in each of the multiple pixel lines extending horizontally in the luminance ratio map data MAP generated in step S113 (step S114).

[0042] Figure 10 shows an example of the distribution of the average luminance ratio in the luminance ratio map data (MAP). The average luminance ratio is approximately "1" for many pixel lines. This indicates that the luminances of corresponding pixels in the left image PL and the right image PR are approximately equal. In this example, the average luminance ratio is skewed towards values ​​less than "1" near pixel line L1 and towards values ​​greater than "1" near pixel line L2.

[0043] Next, the heating wire pattern estimation unit 33 identifies pixel lines corresponding to the positions of the heating wires 8 based on the results obtained in step S114 (step S115). Specifically, the heating wire pattern estimation unit 33 identifies pixel lines corresponding to the positions of the heating wires 8 based on the results obtained in step S114, for example, using techniques such as pattern patching or machine learning.

[0044] Figure 11 shows the image in the image region RA of the left image PL shown in Figure 6. Figure 12 shows the sub-image PSA in the image region RA of the right image PR shown in Figure 6. Near pixel line L1, the left image PL has a line portion 8L, but the right image PR (sub-image PS) does not have a line portion 8R. This corresponds to the fact that in Figure 10, the average value of the luminance ratio protrudes in the direction of being less than "1" near pixel line L1. Near pixel line L2, the right image PR (sub-image PS) has a line portion 8R, but the left image PL does not have a line portion 8L. This corresponds to the fact that in Figure 10, the average value of the luminance ratio protrudes in the direction of being greater than "1" near pixel line L2.

[0045] In this way, the heating wire pattern estimation unit 33 identifies pixel lines corresponding to the positions of the heating wires 8 in the left image PL, and also identifies pixel lines corresponding to the positions of the heating wires 8 in the sub-image PS, which is part of the right image PR.

[0046] If the left image PL and the right image PR are color images, the heating element detection unit 30 can perform the processing in steps S112 to S115 based on the brightness values ​​of red (R), green (G), and blue (B). Specifically, in step S112, the brightness ratio map generation unit 32 detects multiple corresponding points based, for example, the red image of the left image PL and the red image of the selected sub-image PS. In step S113, the brightness ratio map generation unit 32 generates brightness ratio map data for the ratio of red brightness values ​​based on the red image PL and the red image of the sub-image PS. In step S114, the heating element pattern estimation unit 33 calculates the average value of the brightness ratio for one pixel line for each of the multiple pixel lines in this brightness ratio map data related to red. In this way, the heating element pattern estimation unit 33 obtains the distribution of the average value of the brightness ratio related to red, as shown in Figure 10. Similarly, the heating element detection unit 30 obtains the distribution of average values ​​of the brightness ratio related to green and the distribution of average values ​​of the brightness ratio related to blue. Then, in step S115, the heating element pattern estimation unit 33 can identify the pixel line corresponding to the position of the heating element 8 based on the distribution of average values ​​of the brightness ratio related to red, the distribution of average values ​​of the brightness ratio related to green, and the distribution of average values ​​of the brightness ratio related to blue. In this example, the RGB color space was used as an example, but it is not limited to this, and other color spaces such as YUV may be used instead. Also, the left image PL and the right image PR may be monochrome images.

[0047] Next, the heating element detection unit 30 checks whether all sub-images PS were selected in step S111 (step S116). If not all sub-images PS have been selected yet ("N" in step S116), the process returns to step S111 and repeats steps S111 to S116 until all sub-images PS are selected. If all sub-images PS have been selected ("Y" in step S116), this detection process A subroutine terminates.

[0048] In this way, the heating element detection unit 30 identifies pixel lines corresponding to the position of the heating element 8 in each of the three image regions RA to RC of the left image PL, and also identifies pixel lines corresponding to the position of the heating element 8 in each of the three image regions RA to RC of the right image PR, based on the left image PL and right image PR contained in one stereo image PIC. The heating element detection unit 30 then stores the data for these pixel lines as heating element position data.

[0049] Next, as shown in Figure 8, it is checked whether a predetermined number or more of heating wire position data have been obtained (step S103). If a predetermined number or more of heating wire position data have not yet been obtained ("N" in step S103), this flow ends. The process in Figure 8 is performed each time a stereo image PIC is supplied from the stereo camera 11 to the processing unit 20. Therefore, based on each of the multiple stereo image PICs supplied sequentially, the pixel line corresponding to the position of the heating wire 8 is identified, and the heating wire position data is stored sequentially. In this way, the heating wire position data is accumulated.

[0050] In step S103, if a predetermined number or more of heating wire position data are obtained (in step S103, "Y"), the heating wire pattern estimation unit 33 calculates the variance value of the pixel line positions corresponding to the positions of the heating wires 8 in each of the image regions RA, RB, and RC of the left image PL, based on the accumulated heating wire position data, and also calculates the variance value of the pixel line positions corresponding to the positions of the heating wires 8 in each of the image regions RA, RB, and RC of the right image PR (step S104).

[0051] Next, the heating wire pattern estimation unit 33 checks whether all six variance values ​​obtained in step S104 are within a predetermined threshold (step S105). If this condition is not met ("N" in step S105), this flow terminates.

[0052] In step S105, if all six variance values ​​obtained in step S104 are within a predetermined threshold ("Y" in step S105), the heating wire pattern estimation unit 33 determines a representative position of the pixel line corresponding to the position of the heating wire 8 in each of the image regions RA, RB, and RC of the left image PL, based on the accumulated heating wire position data, and also determines a representative position of the pixel line corresponding to the position of the heating wire 8 in each of the image regions RA, RB, and RC of the right image PR (step S106). The representative position may be, for example, the average position, or it may be the position corresponding to the mode when a histogram of the positions of the heating wire 8 is generated.

[0053] Figure 13 shows the representative positions of the pixel lines obtained in step S106. In this example, the left image PL and the right image PR are the images shown in Figure 6. The thick dotted lines indicate the pixel lines indicated by the representative positions. In the left image PL, in each of the image regions RA, RB, and RC, the pixel lines indicated by the representative positions are located on the upper side in the vertical direction of the left image PL, with the pixel line in image region RA at the top and the pixel line in image region RC at the bottom. These pixel lines are positioned along the line portion 8L, which has lower brightness than its surroundings. In the right image PR, in each of the image regions RA, RB, and RC, the pixel lines indicated by the representative positions are located on the lower side in the vertical direction of the right image PR, with the pixel line in image region RA at the top and the pixel line in image region RC at the bottom. These pixel lines are positioned along the line portion 8R, which has lower brightness than its surroundings.

[0054] Next, the heating wire pattern estimation unit 33 estimates the wiring pattern of the heating wire 8 based on the representative positions of the pixel lines obtained in step S106 (step S107). As shown in Figure 13, in the left image PL, the three pixel lines indicated by the representative positions are located along the line portion 8L, which has lower brightness than its surroundings. Therefore, the heating wire pattern estimation unit 33 can obtain the position and orientation of the line portion 8L based on these representative positions, and thus estimate the wiring pattern of the heating wire 8 in the imaging range 9L of the left camera 11L. Similarly, in the right image PR, the three pixel lines indicated by the representative positions are located along the line portion 8R, which has lower brightness than its surroundings. Therefore, the heating wire pattern estimation unit 33 can obtain the position and orientation of the line portion 8R based on these representative positions, and thus estimate the wiring pattern of the heating wire 8 in the imaging range 9R of the right camera 11R. The wiring pattern of the heating wire 8 can be expressed, for example, using a function.

[0055] This concludes this flow.

[0056] In this way, the heating element detection unit 30 estimates the wiring pattern of the heating element 8. The image correction unit 21 generates left image PL1 and right image PR1 by correcting the left image PL and right image PR based on the wiring pattern of the heating element 8 estimated by the heating element pattern estimation unit 33. Specifically, the image correction unit 21 increases the brightness values ​​of multiple pixels at the locations of the wiring pattern of the heating element 8 in the left image PL, based on the wiring pattern of the heating element 8 in the imaging range 9L estimated by the heating element pattern estimation unit 33. That is, the image correction unit 21 increases the brightness values ​​of multiple pixels at the locations corresponding to the line portion 8L shown in Figures 4 and 6. The image correction unit 21 generates left image PL1 by performing such correction on the left image PL. Similarly, the image correction unit 21 increases the brightness values ​​of multiple pixels at the locations of the wiring pattern of the heating element 8 in the right image PR, based on the wiring pattern of the heating element 8 in the imaging range 9R estimated by the heating element pattern estimation unit 33. In other words, the image correction unit 21 increases the brightness values ​​of multiple pixels at positions corresponding to the line portion 8R shown in Figures 4 and 6. The image correction unit 21 generates the right image PR1 by performing such correction on the right image PR.

[0057] Thus, the image processing device 1 includes a sub-image generation unit 31 that generates three sub-images PS by horizontally dividing the right image PR, which is one of the left image PL and right image PR captured by the stereo camera 11, through a windshield 9 on which heating wires 8 are formed; a luminance ratio map generation unit 32 that sequentially selects one of the three sub-images PS and generates luminance ratio map data MAP for the ratio of luminance values ​​based on the selected sub-image PS and the left image PL, which is the other of the left image PL and right image PR; and a heating wire pattern estimation unit 33 that estimates the wiring pattern of the heating wires 8 based on the luminance ratio map data MAP for each of the three sub-images PS. As a result, in the image processing device 1, for example, the image correction unit 21 can correct the left image PL and right image PR based on the estimation result of the wiring pattern of the heating wires 8, thereby suppressing the influence of the heating wires 8 on the corrected left image PL and right image PR. This reduces the influence of the heating wires 8 on various image processing based on the corrected left image PL and right image PR. In this way, the image processing device 1 can reduce the influence of the heating element 8 on the image processing results.

[0058] Furthermore, in the image processing device 1, the multiple sub-images PS include a first sub-image (e.g., sub-image PSA) and a second sub-image (e.g., sub-image PSC). The heating wire pattern estimation unit 33 calculates a first representative value, which is a representative value of the luminance ratio for one pixel line, for each of the multiple pixel lines extending horizontally in the luminance ratio map data MAP (e.g., luminance ratio map data MAPA) for the first sub-image (e.g., sub-image PSA). It also calculates a second representative value, which is a representative value of the luminance ratio for one pixel line, for each of the multiple pixel lines extending horizontally in the luminance value map data MAP (luminance ratio map data MAPC) for the second sub-image (e.g., sub-image PSC). Based on the multiple first representative values ​​and the multiple second representative values, the heating wire wiring pattern is estimated. As a result, the image processing device 1 can obtain, for example, the distribution of representative values ​​of luminance values ​​as shown in Figure 10, for example, in the image region RA of sub-image PSA and the image region RC of sub-image PSC. Therefore, for example, by identifying pixel lines that protrude in the direction where the representative value of the brightness is greater than "1", the wiring pattern of the heating wire 8 in the imaging range 9R of the right camera 11R can be estimated. Similarly, for example, by identifying pixel lines that protrude in the direction where the representative value of the brightness is less than "1", the wiring pattern of the heating wire 8 in the imaging range 9L of the left camera 11L can be estimated. Then, as shown in Figure 13, a tilted wiring pattern can be estimated. As a result, in the image processing device 1, for example, the image correction unit 21 can correct the left image PL and the right image PR based on the estimation result of the wiring pattern of the heating wire 8, thereby suppressing the influence of the heating wire 8 in the corrected left image PL and right image PR.

[0059] [effect] As described above, this embodiment includes a sub-image generation unit that generates three sub-images by horizontally dividing the right image, which is one of the left and right images captured by a stereo camera, through a windshield on which heating wires are formed; a luminance ratio map generation unit that sequentially selects one of the three sub-images and generates luminance ratio map data for the ratio of luminance values ​​based on the selected sub-image and the left image, which is the other of the left and right images; and a heating wire pattern estimation unit that estimates the wiring pattern of the heating wires based on the luminance ratio map data for each of the three sub-images. As a result, for example, the left and right images can be corrected based on the estimation result of the heating wire wiring pattern, thereby reducing the influence of the heating wires on the processing results of the image processing.

[0060] In this embodiment, the plurality of sub-images include a first sub-image and a second sub-image. The heating wire pattern estimation unit calculates a first representative value, which is a representative value of the luminance ratio for one pixel line, for each of the plurality of pixel lines extending horizontally in the luminance ratio map data for the first sub-image. It also calculates a second representative value, which is a representative value of the luminance ratio for one pixel line, for each of the plurality of pixel lines extending horizontally in the luminance value map data for the second sub-image. Based on the plurality of first representative values ​​and the plurality of second representative values, the heating wire wiring pattern is estimated, thereby enabling the estimation of the heating wire wiring pattern. This reduces the influence of the heating wire on the image processing results.

[0061] [Example 1] In the above embodiment, the wiring pattern of the heating element 8 is as shown in Figure 3, but it is not limited to this, and various asymmetrical wiring patterns can be used. For example, as shown in Figure 14, the heating element 8 may have a linear shape so as to penetrate the imaging area 9L and imaging area 9R and extend diagonally. Even in this case, the wiring patterns of the heating element 8 in the imaging area 9L and imaging area 9R are different from each other and asymmetrical.

[0062] Alternatively, as shown in Figure 15, for example, the heating wires 8 may be provided separately in the imaging range 9L and the imaging range 9R. The heating wires 8 include a heating wire 8A provided in the imaging range 9L and a heating wire 8B provided in the imaging range 9R. The heating wire 8A has a shape that crosses the imaging range 9L and moves back and forth in the left-right direction within this imaging range 9L. The heating wire 8B has a shape that crosses the imaging range 9R and moves back and forth in the left-right direction within this imaging range 9R. Even in this case, the wiring patterns of the heating wires 8 in the imaging range 9L and the imaging range 9R are different from each other and asymmetrical.

[0063] Alternatively, as shown in Figure 16, for example, the heating element 8 may be provided in only one of the imaging range 9L and imaging range 9R. In this example, the heating element 8 is provided only in the imaging range 9L. The heating element 8 has a shape that crosses the imaging range 9L and moves back and forth across the imaging range 9L in the left-right direction.

[0064] [Differentiation 2] In the above embodiment, the sub-image generation unit 31 generates multiple sub-images PS by dividing the right image PR horizontally, and the luminance ratio map generation unit 32 generates three luminance ratio map data MAPA, MAPB, and MAPC for the luminance ratio between the left image PL and each of the sub-images PSA, PSB, and PSC, respectively. However, the embodiment is not limited to this. Alternatively, the sub-image generation unit 31 may generate multiple sub-images PS by dividing the left image PL horizontally, and the luminance ratio map generation unit 32 may generate three luminance ratio map data MAPA, MAPB, and MAPC for the luminance ratio between the right image PR and each of the sub-images PSA, PSB, and PSC, respectively.

[0065] [Difference 3] In the above embodiment, an image correction unit 21 is provided, and this image correction unit 21 corrects the left image PL and the right image PR based on the wiring pattern of the heating wire 8 estimated by the heating wire pattern estimation unit 33, but it is not limited to this. The image processing apparatus 1C according to this modified example will be described in detail below.

[0066] Figure 17 shows an example configuration of the image processing device 1C. The image processing device 1C includes a processing unit 20C. The processing unit 20C includes a heating element detection unit 30, a disparity image generation unit 22C, and an object recognition unit 24C. In other words, the processing unit 20C is the same as the processing unit 20 (Figure 1) according to the above embodiment, but with the image correction unit 21 omitted and the disparity image generation unit 22 and object recognition unit 24 replaced by a disparity image generation unit 22C and an object recognition unit 24C, respectively.

[0067] The disparity image generation unit 22C is configured to generate a disparity image PD by performing predetermined image processing, including stereo matching processing, based on the left image PL and the right image PR, similar to the disparity image generation unit 22 in the above embodiment.

[0068] The disparity image generation unit 22C has a corresponding point detection unit 23C. The corresponding point detection unit 23C is configured to detect corresponding points, including image points in the left image PL and the right image PR, by performing stereo matching processing based on the left image PL and the right image PR. The corresponding point detection unit 23C detects corresponding points, including image points in the left image PL and the right image PR, when the matching conditions of the stereo matching processing are met. The corresponding point detection unit 23C relaxes the matching conditions around the wiring pattern of the heating wire 8 estimated by the heating wire pattern estimation unit 33. That is, as shown in Figures 4 and 6, for example, the position of line portion 8L in the left image PL and the position of line portion 8R in the right image PR are different, so it may be difficult to detect corresponding points by stereo matching processing. Therefore, the corresponding point detection unit 23C reduces the possibility of difficulty in detecting corresponding points by relaxing the matching conditions near the wiring pattern of the heating wire 8. As a result, the corresponding point detection unit 23C can effectively perform stereo matching processing even when there are linear portions 8L and 8R.

[0069] The object recognition unit 24C is configured to recognize an object in front of the vehicle 10 based on the left image PL and the right image PR, and the disparity image PD generated by the disparity image generation unit 22C. The object recognition unit 24C recognizes an object when predetermined processing conditions are met. Based on the wiring pattern of the heating element 8 estimated by the heating element pattern estimation unit 33, the object recognition unit 24C relaxes the processing conditions around the wiring pattern. That is, it may be difficult to recognize an object near the line portion 8L in the left image PL and near the line portion 8R in the right image PR. Therefore, the object recognition unit 24C reduces the possibility of difficulty in recognizing an object by relaxing the processing conditions near the wiring pattern of the heating element 8. As a result, the object recognition unit 24C can effectively recognize an object even when there are line portions 8L and 8R.

[0070] Here, the disparity image generation unit 22C and the object recognition unit 24C correspond to one specific example of the "processing unit" in this disclosure.

[0071] [Other variations] Furthermore, two or more of these variations may be combined.

[0072] Although the present technology has been described above with reference to embodiments and several modifications, the present technology is not limited to these embodiments, and various modifications are possible.

[0073] For example, in the above embodiment, the stereo camera 11 is configured to capture images of the area in front of the vehicle 10, but it is not limited to this, and for example, it may capture images of the side or rear of the vehicle 10.

[0074] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur. [Explanation of symbols]

[0075] 1,1C...Image processing device, 8,8A,8B...Heating wire, 8L,8R...Line portion, 9...Windshield, 9L,9R...Imaging range, 10...Vehicle, 11...Stereo camera, 11L...Left camera, 11R...Right camera, 20,20C...Processing unit, 21...Image correction unit, 22,22C...Disparity image generation unit, 23,23C...Corresponding point detection unit, 24,24C...Object recognition unit, 30...Heating wire detection unit, 31...Sub-image generation unit, 32...Brightness ratio map generation unit, 33...Heating wire pattern estimation unit, MAP...Brightness ratio map data, PIC...Stereo image, PL,PL1...Left image, PR,PR1...Right image, PS,PSA,PSB,PSC...Sub-image, RA,RB,RC...Image region.

Claims

1. a sub-image generating unit that generates a plurality of sub-images by horizontally dividing a first image, which is one of a left image and a right image captured by the stereo camera through a windshield on which a heating wire is formed, and a brightness ratio map generator that sequentially selects one of the plurality of sub-images and generates map data indicating a ratio of brightness values ​​based on the selected sub-image and a second image that is the other of the left image and the right image; a heating wire pattern estimation unit that estimates a wiring pattern of the heating wire based on the map data for each of the plurality of sub-images; An image processing device comprising:

2. the plurality of sub-images includes a first sub-image and a second sub-image; The heating wire pattern estimation unit is calculating a first representative value that is a representative value of luminance ratios for one pixel line in each of a plurality of pixel lines extending in a horizontal direction in the map data for the first sub-image; calculating a second representative value that is a representative value of luminance ratios for one pixel line in each of a plurality of pixel lines extending in a horizontal direction in the map data for the second sub-image; A wiring pattern of the heating wire is estimated based on the plurality of first representative values ​​and the plurality of second representative values. The image processing device according to claim 1 .

3. an image correction unit that corrects image portions of the left image and the right image that relate to the wiring pattern of the heating wire based on an estimation result of the heating wire pattern estimation unit; a processing unit that performs predetermined image processing based on the left image and the right image corrected by the image correction unit; The image processing device according to claim 1 or 2, further comprising:

4. a processing unit that performs a predetermined image processing based on the left image and the right image when a predetermined processing condition is satisfied, The processing unit relaxes the processing conditions for image portions in the left image and the right image that relate to a wiring pattern of the heating wire, based on an estimation result of the heating wire pattern estimation unit.

3. The image processing device according to claim 1 or 2.

5. The wiring patterns of the heating wires in the imaging range of the left camera that generates the left image and the imaging range of the right camera that generates the right image are asymmetrical with each other. The image processing device according to claim 1 .

Images