Image processing device, image processing system, and image processing program

Abstract

To provide an image processing device, an image processing system, and an image processing program capable of improving the visibility of images generated by taking photographs through the window glass of a vehicle equipped with heating elements. [Solution] The image processing apparatus 1 according to the present disclosure includes a heating element area identification unit 12 that identifies an image area whose brightness or luminance is less than or equal to a predetermined brightness or luminance among a plurality of image areas constituting a captured image generated by taking a photograph through the window glass of a vehicle on which heating elements are arranged, as a heating element area for displaying heating elements; a corrected image generation unit 13 that corrects the brightness or luminance of the heating element area included in the captured image to match the brightness or luminance of an image area other than the heating element area adjacent to the heating element area, and generates a corrected image of the captured image; and an image storage unit 15 that stores the corrected image.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing system, and an image processing program for processing a captured image generated by a camera installed in a vehicle.

Background Art

[0002] Conventionally, various techniques for processing a captured image generated by a camera installed in a vehicle have been proposed. As an example of such a technique, Patent Document 1 discloses an image processing apparatus that detects an edge of a reflected image that overlaps a captured image generated by an in-vehicle camera imaging an external target through a windshield.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the image processing apparatus disclosed in Patent Document 1, since it does not detect the heating wire shown in the captured image, there is a problem that the visibility of the captured image generated by imaging through the window glass of the vehicle where the heating wire is arranged cannot be improved.

[0005] The present disclosure solves such problems, and an object thereof is to provide an image processing apparatus, an image processing system, and an image processing program capable of improving the visibility of a captured image generated by imaging through a window glass of a vehicle where a heating wire is arranged.

Means for Solving the Problems

[0006] The image processing apparatus according to the present disclosure is A heating element region identification unit identifies, among multiple image regions constituting a captured image generated by taking a photograph through the window glass of a vehicle equipped with heating elements, an image region whose brightness or luminance is less than or equal to a predetermined brightness or luminance, as a heating element region for displaying heating elements. A corrected image generation unit corrects the brightness or luminance of the heating element region included in the captured image to match the brightness or luminance of the image region adjacent to the heating element region, thereby generating a corrected image of the captured image. It includes an image storage unit for saving corrected images.

[0007] The image processing system related to this disclosure is A heating element region identification unit identifies, among multiple image regions constituting a captured image generated by taking a photograph through the window glass of a vehicle equipped with heating elements, an image region whose brightness or luminance is less than or equal to a predetermined brightness or luminance, as a heating element region for displaying heating elements. A corrected image generation unit corrects the brightness or luminance of the heating element region included in the captured image to match the brightness or luminance of the image region adjacent to the heating element region, thereby generating a corrected image of the captured image. It includes an image storage unit for saving corrected images.

[0008] The image processing program relating to this disclosure is provided to a computer, Among the multiple image regions that make up the captured image generated by taking a photograph through the window glass of a vehicle equipped with heating wires, the image region whose brightness or luminance is less than or equal to a predetermined brightness or luminance is identified as the heating wire region where the heating wires are to be displayed. The brightness or luminance of the heating element region included in the captured image is corrected to match the brightness or luminance of the image region adjacent to the heating element region, thereby generating a corrected image of the captured image. Display the corrected image. [Effects of the Invention]

[0009] This disclosure provides an image processing device, an image processing system, and an image processing program that can improve the visibility of captured images generated by taking photographs through the window glass of a vehicle equipped with heating wires. [Brief explanation of the drawing]

[0010] [Figure 1] This block diagram shows an example of the configuration of an image processing device related to this disclosure. [Figure 2] This is an example of a photograph showing heating elements placed on the windows of a vehicle. [Figure 3] This figure shows an example of an image region that makes up a captured image. [Figure 4] This figure shows another example of a photograph that captures heating elements placed on the windows of a vehicle. [Figure 5] This flowchart shows an example of a process performed by the image processing device according to the first embodiment. [Figure 6] This figure shows an example of the process for identifying the heating element region. [Figure 7] This figure shows an example of the process for identifying the heating element region. [Figure 8] This figure shows an example of processing performed by the image processing apparatus according to the second embodiment. [Modes for carrying out the invention]

[0011] <First Embodiment> Figure 1 is a block diagram showing an example of the configuration of the image processing device 1 according to this disclosure. The image processing device 1 is a device that processes captured images generated by taking pictures through the window glass of a vehicle equipped with a heating element. A specific example of the image processing device 1 is a dashcam, etc. Vehicles include automobiles, trains, and other vehicles.

[0012] The image processing apparatus 1 includes an arithmetic unit 10, a communication interface 20, and a storage device 30. The communication interface 20 is an interface for transmitting and receiving signals between the image processing apparatus 1 and other devices. The other devices include various in-vehicle devices. Specific examples of the in-vehicle devices include an in-vehicle camera that captures the rear of the vehicle through the rear glass from inside the vehicle.

[0013] The storage device 30 is a storage device that stores programs executed by the arithmetic unit 10 and various information processed by the arithmetic unit 10.

[0014] The arithmetic unit 10 is a processor such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit). The arithmetic unit 10 executes the program stored in the storage device 30 to execute the method defined by the program. Note that an integrated circuit such as an FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) may execute the above-described program. The integrated circuits such as the processor, FPGA, and ASIC correspond to a computer.

[0015] The arithmetic unit 10 includes a reception determination unit 11, a heating wire region specifying unit 12, a corrected image generation unit 13, a correction condition determination unit 14, and an image storage unit 15. These functions can be realized by a program.

[0016] The reception determination unit 11 determines whether it has received the captured image transmitted by the in-vehicle camera via the communication interface 20. The captured image is generated by capturing through the window glass of the vehicle on which the heating wire is arranged. FIG. 2 is an example of a captured image in which the heating wire arranged on the window glass of the vehicle is shown. In the present embodiment, the heating wire arranged so as to extend in the horizontal direction is targeted. As shown in FIG. 2, the heating wire shown in the captured image is not necessarily straight, and may be curved depending on the distance between the captured image and the window glass and the shape of the window glass. Further, the brightness or luminance of the image area (hereinafter referred to as the "heating wire area") displaying the heating wire is lower than the brightness or luminance of the surrounding image area. Also, the scenery behind the heating wire often appears in the heating wire area. Hereinafter, the vertical direction of the captured image is referred to as the vertical direction, and the horizontal direction of the captured image is referred to as the horizontal direction.

[0017] The heating wire area specifying unit 12 specifies, as the heating wire area, an image area in which the brightness or luminance is equal to or lower than a predetermined brightness or luminance among a plurality of image areas constituting the captured image transmitted by the in-vehicle camera. The heating wire area specifying unit 12 includes an image area selection unit 120, a representative value calculation unit 121, a representative value determination unit 122, a candidate image area registration unit 123, an image area determination unit 124, a candidate image area determination unit 125, a distance calculation unit 126, a distance determination unit 127, and a heating wire area registration unit 128.

[0018] The image area selection unit 120 selects an image area constituting the captured image. Specifically, the image area selection unit 120 sequentially selects a plurality of image columns, which are a plurality of image areas continuous in the vertical direction. FIG. 3 is a diagram showing an example of the image area constituting the captured image. As shown in FIG. 3, the captured image is composed of a plurality of image areas. The image area selection unit 120 sequentially selects, for example, from the image column 40 at one end in the horizontal direction. Note that the captured image shown in FIG. 3 is composed of 1920 pixels × 1024 pixels, but the size of the captured image is not limited to this and may be any size.

[0019] Furthermore, the image region selection unit 120 sequentially selects multiple image regions that make up the selected image sequence. For example, if image sequence 40 is selected, the image region selection unit 120 sequentially selects from the multiple image regions that make up image sequence 40, starting with the image region 41 at one end in the vertical direction.

[0020] The representative value calculation unit 121 calculates a representative value of brightness or luminance for the image region selected by the image region selection unit 120. In this embodiment, as the representative value of brightness or luminance, for example, the median, average, or minimum value of the brightness or luminance of multiple pixels constituting the selected image region can be adopted.

[0021] The representative value determination unit 122 determines whether the representative value of brightness or luminance calculated by the representative value calculation unit 121 is less than or equal to a predetermined threshold corresponding to the brightness or luminance of the heating element area. Specifically, the representative value of the brightness or luminance of the pixel displaying the heating element, among the multiple pixels constituting the captured image generated by photographing the heating element placed on the window glass with an in-vehicle camera, can be adopted as the predetermined threshold. The representative value of the brightness or luminance of the pixel displaying the heating element can be, for example, the maximum value, median value, or average value of the brightness or luminance of that pixel.

[0022] The candidate image area registration unit 123 stores identification information of a candidate image area in the storage device 30. This candidate image area is an image area that has been determined to have a representative value of brightness or luminance that is less than or equal to a predetermined threshold corresponding to the brightness or luminance of the heating element area. The candidate image area is an image area that may be a heating element area. In this embodiment, the position coordinates (x,y) of the candidate image area can be used as identification information for the candidate image area. For example, the position coordinates of the four corners of the candidate image area can be used as identification information for the candidate image area.

[0023] The image region determination unit 124 determines whether the image region selected by the image region selection unit 120 is an image region at a predetermined location. Specifically, the image region determination unit 124 determines whether the selected image region is an image region at the other end in the vertical direction. For example, when the process of selecting images sequentially from the image region 41 (Figure 3) at one end in the vertical direction is executed, the image region determination unit 124 determines whether the currently selected image region is an image region 42 (Figure 3) at the other end in the vertical direction.

[0024] Furthermore, the image region determination unit 124 determines whether the image sequence region selected by the image region selection unit 120 is an image sequence at a predetermined position. Specifically, the image region determination unit 124 determines whether the currently selected image sequence is an image sequence at the other end in the horizontal direction. For example, when the process of selecting images sequentially starting from the image sequence 40 (Figure 3) at one end in the horizontal direction is executed, the image region determination unit 124 determines whether the currently selected image sequence is an image sequence 47 (Figure 3) at the other end in the horizontal direction.

[0025] The candidate image region determination unit 125 determines whether a single image sequence contains multiple candidate image regions. If multiple candidate image regions exist in a single image sequence, the x-coordinate values ​​of these candidate image regions are the same. Therefore, the candidate image region determination unit 125 can determine whether multiple candidate image regions exist in a single image sequence by determining whether the x-coordinate values ​​of the multiple candidate image regions stored in the storage device 30 are the same.

[0026] Furthermore, the candidate image region determination unit 125 determines whether or not candidate image regions exist in a predetermined number of horizontally adjacent image sequences. When candidate image regions exist in multiple horizontally adjacent image sequences, the x-coordinate values ​​of these candidate image regions are configured to be consecutive. Specifically, the x-coordinate of the rightmost edge of a certain candidate image region is the same as the x-coordinate of the rightmost edge of another candidate image region adjacent to the right of that candidate image region. Therefore, the candidate image region determination unit 125 can determine whether or not candidate image regions exist in multiple horizontally adjacent image sequences by determining whether or not the x-coordinate values ​​of the candidate image regions stored in the storage device 30 are consecutive.

[0027] The predetermined number of image sequences used for this determination shall be less than or equal to the number of image sequences that make up the entire captured image. Figure 4 shows another example of a captured image in which heating wires placed on the windows of a vehicle are visible. Two heating wires are visible in the captured image shown in Figure 4. The heating wires visible at the top of the captured image do not extend from one end to the other in the horizontal direction of the captured image, but are interrupted midway. In order to identify an image region displaying such heating wires as a heating wire region, it is preferable that the predetermined number of image sequences used for the above determination be less than the number of image sequences that make up the entire captured image.

[0028] The distance calculation unit 126 calculates the vertical distance between multiple candidate image regions when multiple candidate image regions are included in multiple horizontally adjacent image sequences. More specifically, the distance calculation unit 126 calculates the vertical distance between these candidate image regions based on the position coordinates of the multiple candidate image regions. The vertical distance between candidate image regions can be calculated using the vertical position coordinates of each candidate image region. For example, the distance calculation unit 126 can calculate the vertical distance between multiple candidate image regions by using the difference in vertical position coordinates of their edges, such as the upper right edge, lower right edge, upper left edge, and lower left edge. Alternatively, instead of using the position coordinates of the edges of each candidate image region, the distance calculation unit 126 may calculate the vertical distance between these candidate image regions by using the difference in vertical position coordinates of the center points of each candidate image region.

[0029] The distance determination unit 127 determines whether the maximum vertical distance among the candidate image regions calculated by the distance calculation unit 126 is less than or equal to a predetermined threshold. The predetermined threshold is a value less than or equal to the vertical size of the captured image. The predetermined threshold can be the vertical distance between multiple heating element image regions included in the captured image generated when the in-vehicle camera actually photographs heating elements placed on the window glass. For example, as shown in Figure 3, the vertical distance between heating element region 44 and heating element region 45 can be used as the predetermined threshold. The larger this predetermined threshold, the more likely it is to identify heating elements with a greater degree of curvature. On the other hand, the smaller this predetermined threshold, the more likely it is to identify straight heating elements.

[0030] The heating element area registration unit 128 stores the position coordinates of the heating element area in the captured image in the storage device 30. Specifically, the heating element area registration unit 128 stores the position coordinates of multiple candidate image areas whose vertical distance is determined to be less than or equal to a predetermined threshold in the storage device 30 as the position coordinates of the heating element area. In this embodiment, the heating element area registration unit 128 can, for example, store the position coordinates of the four corners of a candidate image area as the position coordinates of the heating element area.

[0031] The corrected image generation unit 13 corrects the brightness or luminance of the heating element region included in the captured image to match the brightness or luminance of the image region adjacent to the heating element region, and generates a corrected image of the captured image. More specifically, the corrected image generation unit 13 can change the brightness or luminance of the heating element region identified by the heating element region identification unit 12 to the brightness or luminance of the image region vertically adjacent to that image region. For example, the corrected image generation unit 13 can change the brightness or luminance of the heating element region 45 shown in Figure 3 to the brightness or luminance of the image region 46 vertically adjacent to the heating element region 45.

[0032] The correction condition determination unit 14 determines whether or not the captured image satisfies the correction conditions. Specifically, the correction condition determination unit 14 can determine that the captured image does not satisfy the correction conditions if the overall brightness or luminance of the captured image is below a predetermined threshold. The predetermined threshold can be the threshold corresponding to the brightness or luminance of the heating element region described above.

[0033] Furthermore, the correction condition determination unit 14 can determine that the correction conditions for the captured image are not met if the headlights of the vehicle on which the image processing device 1 is installed are turned on. The correction condition determination unit 14 can determine that the headlights are on if it receives a signal indicating that the headlights are on, such as a CAN (Controller Area Network) signal or a DIMMER line signal.

[0034] Furthermore, the correction condition determination unit 14 can determine that the correction conditions for the captured image are not met if an impact is detected on the vehicle on which the image processing device 1 is installed. The correction condition determination unit 14 can determine that an impact on the vehicle has been detected if the detected value of the acceleration sensor provided by the image processing device 1 is greater than or equal to a predetermined threshold. This predetermined threshold can be a value that is expected when an impact on the vehicle is detected.

[0035] Furthermore, the correction condition determination unit 14 can determine that the correction conditions for the captured image are not met if a moving object is detected around the vehicle on which the image processing device 1 is installed. The correction condition determination unit 14 can detect moving objects around the vehicle using a program that detects moving objects using the captured image around the vehicle, such as a multimodal LLM (Large Language Models).

[0036] Furthermore, the correction condition determination unit 14 can determine that the correction conditions for the captured image are not met if aggressive driving is detected against a vehicle equipped with the image processing device 1. The correction condition determination unit 14 can detect aggressive driving using a program that detects aggressive driving using a captured image of the rear of the vehicle, such as a multimodal LLM.

[0037] Furthermore, the correction condition determination unit 14 can determine that the correction conditions for the captured image are not met when the vehicle on which the image processing device 1 is installed is traveling at a predetermined location. The correction condition determination unit 14 can determine whether the vehicle is traveling at a predetermined location by calculating the position of the vehicle using the positioning signal transmitted by the positioning satellite and determining whether the calculated position is within a predetermined distance range from the predetermined location. Any distance can be adopted as the predetermined distance range.

[0038] Furthermore, the correction condition determination unit 14 can determine that the correction conditions for captured images are not met if the image processing device 1 is in a parking surveillance state where it saves captured images when the vehicle on which the image processing device is installed is parked. The correction condition determination unit 14 can determine whether or not it is in a parking surveillance state by, for example, determining whether or not it has received a parking brake signal from the vehicle.

[0039] The image storage unit 15 stores either the corrected or uncorrected captured image in the storage device 30. More specifically, if the correction condition determination unit 14 determines that the captured image meets the correction conditions, the image storage unit 15 stores the corrected captured image. On the other hand, if the correction condition determination unit 14 determines that the captured image does not meet the correction conditions, the image storage unit 15 stores the uncorrected captured image.

[0040] Figure 5 is a flowchart showing an example of processing performed by the image processing device 1 according to the first embodiment. In step S1, the reception determination unit 11 of the image processing device 1 determines whether or not it has received the captured image transmitted by the in-vehicle camera. If it is determined that the captured image has not been received (NO), the processing in step S1 is executed again. On the other hand, if it is determined that the captured image has been received (YES), the processing branches to step S2.

[0041] In step S2, the correction condition determination unit 14 determines whether the captured image satisfies the correction conditions. If it is determined that the captured image satisfies the correction conditions (YES), in step S3, the heating wire area identification unit 12 uses the received captured image to perform a heating wire area identification process to identify the heating wire area from among multiple image areas that constitute the captured image. Details of the heating wire area identification process will be described later with reference to Figures 6 and 7.

[0042] In step S4, the corrected image generation unit 13 corrects the heating element region identified in the heating element region identification process and generates a corrected image. In step S5, the image storage unit 15 saves the corrected captured image, and the process returns to step S1.

[0043] On the other hand, if it is determined in step S2 that the correction conditions for the captured image are not met (NO), in step S6 the image storage unit 15 saves the received uncorrected captured image, and the process returns to step S1.

[0044] Figures 6 and 7 show an example of the heating element area identification process. In step S10, the image area selection unit 120 selects an image sequence at one end in the horizontal direction from among a plurality of image sequences that make up the captured image. In step S11, the image area selection unit 120 selects an image sequence at one end in the vertical direction from among a plurality of image sequences that make up the selected image sequence.

[0045] In step S12, the representative value calculation unit 121 calculates a representative value of the brightness or luminance of the selected image area. In step S13, the representative value determination unit 122 determines whether the representative value of brightness or luminance calculated in step S12 is less than or equal to a predetermined threshold corresponding to the brightness or luminance of the heating element area. If it is determined that the calculated representative value of brightness or luminance is less than or equal to the predetermined threshold (YES), in step S14, the candidate image area registration unit 123 saves the identification information of the selected image area to the storage device 30 as the identification information of the candidate image area. On the other hand, if it is determined that the calculated brightness or luminance exceeds the predetermined threshold (NO), the process branches to step S15.

[0046] In step S15, the image region determination unit 124 determines whether the selected image region is the image region at the other end in the vertical direction. If it is determined that the selected image region is not the image region at the other end in the vertical direction (NO), the process branches to step S16.

[0047] In step S16, the image area selection unit 120 selects an image area adjacent to one side of the selected image area in the vertical direction, and the process returns to step S12. For example, in an embodiment in which image areas are selected sequentially from the top of the captured image, the image area selection unit 120 selects an image area adjacent to the bottom of the selected image area. In an embodiment in which image areas are selected sequentially from the bottom of the captured image, the image area selection unit 120 selects an image area adjacent to the top of the selected image area.

[0048] On the other hand, if it is determined that the selected image region is the image region at the other end in the vertical direction (YES), in step S17, the image region determination unit 124 determines whether or not the selected image sequence is the image sequence at the other end in the horizontal direction. If it is determined that the selected image sequence is not the image sequence at the other end in the horizontal direction (NO), the process branches to step S18.

[0049] In step S18, the image region selection unit 120 selects an image row adjacent to one side of the selected image row in the horizontal direction, and the process returns to step S12. For example, in an embodiment in which image rows are selected sequentially from the left side of the captured image, the image region selection unit 120 selects an image row adjacent to the right side of the selected image row. In an embodiment in which image rows are selected sequentially from the right side of the captured image, the image region selection unit 120 selects an image row adjacent to the left side of the selected image row.

[0050] On the other hand, if it is determined in step S17 that the selected image sequence is the image sequence at the other end in the horizontal direction (YES), the process branches to step S19. In step S19, the candidate image region determination unit 125 uses the candidate image region identification information saved in step S14 to determine whether or not a single image sequence contains multiple candidate image regions. If it is determined that a single image sequence does not contain multiple candidate image regions (NO), the process branches to step S20.

[0051] In step S20, the candidate image region determination unit 125 uses the candidate image region identification information saved in step S14 to determine whether or not there are candidate image regions in multiple horizontally adjacent image rows. If it is determined that there are no candidate image regions in multiple horizontally adjacent image rows (NO), the heating wire region identification process ends. Therefore, if there are no candidate image regions in multiple horizontally adjacent image rows, in other words, if there are no horizontally consecutive candidate image regions, the candidate image region whose identification information was saved in step S14 will not be identified as a heating wire region. This prevents candidate image regions that are not horizontally consecutive from being identified as heating wire regions.

[0052] On the other hand, if it is determined that there are candidate image regions in multiple horizontally adjacent image sequences (YES), the process branches to step S21. In step S21, the distance calculation unit 126 calculates the vertical distance of multiple candidate image regions based on the position coordinates of the multiple candidate image regions included in the multiple horizontally adjacent image sequences.

[0053] In step S22, the distance determination unit 127 determines whether the maximum vertical distance among the calculated candidate image regions is less than or equal to a predetermined threshold. If it is determined that the maximum vertical distance of the candidate image region exceeds the predetermined threshold (NO), the heating element region identification process ends. Therefore, even if an image region has a brightness or luminance below the predetermined threshold, if the maximum vertical distance exceeds the predetermined threshold, the image region will not be identified as a heating element region. This prevents image regions that do not display heating elements from being identified as heating element regions.

[0054] On the other hand, if it is determined that the maximum vertical distance of the candidate image regions is less than or equal to a predetermined threshold (YES), in step S23 the heating element region registration unit 128 saves the position coordinates of these candidate image regions to the storage device 30 as the position coordinates of the heating element regions, and the heating element region identification process ends.

[0055] In step S19, if it is determined that a single image sequence contains multiple candidate image regions (YES), the process branches to step S24. In step S24, the image region selection unit 120 selects one candidate image region from among the multiple candidate image regions contained in a single image sequence.

[0056] In step S25, the candidate image region determination unit 125 determines whether or not there are candidate image regions in multiple horizontally adjacent image rows for the selected candidate image region. If it is determined that there are no candidate image regions in multiple horizontally adjacent image rows (NO), the heating wire region identification process ends. Therefore, if there are no candidate image regions in multiple horizontally adjacent image rows, in other words, if there are no horizontally consecutive candidate image regions, the candidate image region whose identification information was saved in step S14 is not identified as a heating wire region. This prevents candidate image regions that are not horizontally consecutive from being identified as heating wire regions.

[0057] On the other hand, if it is determined that there are candidate image regions in multiple horizontally adjacent image sequences (YES), the process branches to step S26. In step S26, the distance calculation unit 126 calculates the vertical distance of multiple candidate image regions based on the position coordinates of the multiple candidate image regions included in the multiple horizontally adjacent image sequences.

[0058] In step S27, the distance determination unit 127 determines whether the maximum vertical distance among the calculated candidate image regions is less than or equal to a predetermined threshold. If it is determined that the maximum vertical distance of the candidate image region exceeds the predetermined threshold (NO), the heating element region identification process ends. Therefore, even if an image region has a brightness or luminance below the predetermined threshold, if the maximum vertical distance exceeds the predetermined threshold, the image region will not be identified as a heating element region. This prevents image regions that do not display heating elements from being identified as heating element regions.

[0059] On the other hand, if it is determined that the maximum vertical distance of the candidate image regions is less than or equal to a predetermined threshold (YES), in step S28 the heating element region registration unit 128 saves the position coordinates of these candidate image regions to the storage device 30 as the position coordinates of the heating element regions, and the heating element region identification process ends.

[0060] In step S29, the candidate image region determination unit 125 determines whether there are any other candidate image regions that were not selected in step S24. If it is determined that there are no other unselected candidate image regions (NO), the heating wire region identification process ends. On the other hand, if it is determined that there are other unselected candidate image regions (YES), the process branches to step S30. In step S30, the image region selection unit 120 selects the other candidate image region, and the process returns to step S25. If there are multiple other candidate image regions, the image region selection unit 120 selects one of them.

[0061] In the embodiment described above, the heating element area identification unit 12 identifies, among a plurality of image areas constituting the captured image generated by photographing through the window glass of a vehicle on which heating elements are arranged, an image area whose brightness or luminance is less than or equal to a predetermined brightness or luminance, as a heating element area to display the heating elements. Next, the corrected image generation unit 13 corrects the brightness or luminance of the heating element areas included in the captured image to match the brightness or luminance of the image areas other than the heating element areas adjacent to the heating element areas, and generates a corrected image of the captured image. The image storage unit 15 stores the corrected image generated by the corrected image generation unit 13.

[0062] The heating element region has lower brightness or luminance than the surrounding image region due to the presence of the heating element, and often displays the scenery behind the heating element. The corrected image generation unit 13 corrects the brightness or luminance of such heating element regions to match the brightness or luminance of the image regions adjacent to the heating element region. This makes it possible to match the brightness or luminance of the heating element region, which has been reduced due to the presence of the heating element, with the surrounding image region, thereby improving the visibility of the captured image generated by shooting through the window glass of a vehicle equipped with heating elements.

[0063] Furthermore, in the embodiments described above, the correction condition determination unit 14 determines that the correction conditions for the captured image are not met if the overall brightness or luminance of the captured image is below a predetermined threshold, if the headlights of the vehicle on which the image processing device 1 is installed are on, if an impact to the vehicle is detected, if a moving object is detected around the vehicle, if aggressive driving towards the vehicle is detected, if the vehicle is traveling in a predetermined position, or if the vehicle is in a parking surveillance state where captured images are saved when parked. If it is determined that the correction conditions for the captured image are not met, the corrected image generation unit 13 does not correct the captured image, and the image storage unit 15 saves the uncorrected captured image.

[0064] By adopting this configuration, in the specified cases described above, the captured image will not be corrected, and the uncorrected image will be saved. If the overall brightness or luminance of the captured image is below a predetermined threshold, the heating element area may not be correctly identified, and the image may not be corrected properly. Therefore, saving the uncorrected image prevents a decrease in the visibility of the captured image.

[0065] Furthermore, if the vehicle on which the image processing device 1 is installed has a function that turns on the headlights when the external illumination falls below a predetermined level, the overall brightness or luminance of the captured image may be low when the vehicle's headlights are on. In this case, the heating element area may not be correctly identified, and the image may not be corrected properly. Therefore, saving the uncorrected captured image can prevent a decrease in the visibility of the captured image.

[0066] Furthermore, if an impact to a vehicle is detected, or if aggressive driving against a vehicle is detected, the uncorrected captured image can be saved as evidence, or as a recorded image.

[0067] Furthermore, if a moving object is detected around a vehicle, the likelihood of an accident increases, and therefore, uncorrected images can be recorded as evidence. Similarly, if a vehicle is traveling through a designated location such as an accident-prone area, the likelihood of an accident increases, and therefore, uncorrected images can be recorded as evidence.

[0068] Furthermore, by not correcting the captured image in the above-mentioned predetermined cases, the power consumption and memory usage due to the image correction process can be reduced.

[0069] Furthermore, in the above-described embodiment, the heating element region identification unit 12 identifies the image region as a heating element region when an image region with a brightness or luminance below a predetermined level is continuous in the horizontal direction of the captured image. As a result, when an image region with a brightness or luminance below a predetermined level exists in the captured image, if that image region is not continuous in the horizontal direction of the captured image, that image region is not identified as a heating element region. For example, if a black object that does not extend horizontally is captured in the image, the image region displaying such an object is not identified as a heating element region. Therefore, by adopting this configuration, the accuracy of identifying the heating element region can be improved.

[0070] <Second Embodiment> Figure 8 shows an example of the processing performed by the image processing device 1 according to the second embodiment. The differences from the second embodiment will be explained below.

[0071] In step S31, the reception determination unit 11 of the image processing device 1 determines whether or not it has received the captured image transmitted by the in-vehicle camera. If it is determined that the captured image has not been received (NO), the process in step S31 is executed again. On the other hand, if it is determined that the captured image has been received (YES), the process branches to step S32.

[0072] In step S32, the correction condition determination unit 14 determines whether the captured image satisfies the correction conditions. If it is determined that the captured image satisfies the correction conditions (YES), in step S33, the heating element area identification unit 12 performs heating element area identification processing using the received captured image. In the second embodiment, it is preferable to use an image generated by the in-vehicle camera capturing a white background through the window glass. This improves the accuracy of identifying the heating element area.

[0073] In step S34, the corrected image generation unit 13 corrects the heating element region identified in the heating element region identification process and generates a corrected image. In step S35, the image storage unit 15 saves the corrected captured image.

[0074] On the other hand, if it is determined in step S32 that the correction conditions for the captured image are not met (NO), in step S36 the image storage unit 15 saves the uncorrected captured image received in step S31, and the process proceeds to step S37.

[0075] In step S37, the reception determination unit 11 determines whether or not it has received the captured image transmitted by the in-vehicle camera. If it is determined that the captured image has not been received (NO), the process in step S37 is executed again. On the other hand, if it is determined that the captured image has been received (YES), in step S38, the correction condition determination unit 14 determines whether or not the correction conditions for the captured image are met. If it is determined that the correction conditions for the captured image are met (YES), the process branches to step S34. On the other hand, if it is determined that the correction conditions for the captured image are not met (NO), the process branches to step S36.

[0076] As described above, in the second embodiment, the heating element area identification process is executed only once. Therefore, the power and memory usage consumed by the heating element area identification process can be reduced.

[0077] <Other Embodiments> In the embodiments described above, the case in which the image processing device 1 is installed in a vehicle was explained. However, in other embodiments, an image processing system including multiple information processing devices such as servers not installed in a vehicle may realize the functions of the image processing device 1. Therefore, in the image processing system, the heating element area identification unit 12, the corrected image generation unit 13, and the image storage unit 15 can be implemented in individual information processing devices. These information processing devices can communicate with each other with data such as captured images and corrected images. In this case, the information processing device can receive captured images generated by a camera installed in the vehicle from the vehicle via wireless and wired communication. The information processing device can also store the captured images and corrected images in other storage devices, which are external devices.

[0078] In other embodiments, the image processing device 1 may have a display unit that displays captured images and corrected images on a display device such as a display. The display unit can be implemented by a program executed by the arithmetic unit 10. In this case, specific examples of the image processing device 1 include information processing devices such as PCs (Personal Computers) and smartphones. The image processing device 1 can receive captured images generated by cameras installed on vehicles via wireless and wired communication. The image processing device 1 can also acquire captured images stored on portable recording media such as memory cards.

[0079] Furthermore, in other embodiments, instead of performing the heating element region identification process shown in Figures 6 and 7, the image processing device 1 may identify the heating element region in the captured image using a trained model, which is a program learned by machine learning such as deep learning. This model can be trained using training data that takes a captured image generated by taking a picture through the window glass of a vehicle where heating elements are placed as input information, and outputs the position coordinates of the heating element region contained in the captured image as output information. When a captured image containing heating elements is input, the trained model trained in this way can output the position coordinates of the heating element region.

[0080] In the above example, the program can be stored and provided to the computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Alternatively, the program may be provided to the computer using various types of transient computer-readable medium. Examples of transient computer-readable medium include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable medium can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0081] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from the spirit of this disclosure. [Explanation of Symbols]

[0082] 1: Image processing device 10: Arithmetic device 11: Reception determination unit 12: Heating wire area identification part 120: Image area selection section 121: Representative Value Calculation Unit 122: Representative Value Determination Unit 123: Candidate image region registration unit 124: Image area determination unit 125: Candidate image region determination unit 126: Distance calculation unit 127: Distance determination unit 128: Heating wire area registration section 13: Corrected Image Generation Unit 14: Correction condition judgment section 15: Image storage section 20: Communication Interface 30: Storage device 40,47: Image sequence 41-43: Image area 44,45: Heating wire area 46: Image area

Claims

1. A heating element region identification unit identifies, among multiple image regions constituting a captured image generated by photographing through the window glass of a vehicle on which heating elements are arranged, an image region whose brightness or luminance is less than or equal to a predetermined brightness or luminance, as a heating element region for displaying the heating elements. A corrected image generation unit that corrects the brightness or luminance of the heating element region included in the captured image to match the brightness or luminance of an image region other than the heating element region adjacent to the heating element region, thereby generating a corrected image of the captured image. The corrected image is stored in the image storage unit. Image processing device including

2. Includes a correction condition determination unit that determines whether or not the correction conditions of the captured image are met, The correction condition determination unit determines that the correction conditions for the captured image are not met if the overall brightness or luminance of the captured image is below a predetermined threshold, if the headlights of the vehicle (automobile) on which the image processing device is installed are on, if an impact to the vehicle is detected, if a moving object is detected around the vehicle, if aggressive driving towards the vehicle is detected, if the vehicle is traveling at a predetermined position, or if the vehicle is in a parking surveillance state where the captured image is saved when parked. If it is determined that the correction conditions for the captured image are not met, the corrected image generation unit does not correct the captured image, and the image storage unit stores the uncorrected captured image, as described in claim 1.

3. The image processing apparatus according to claim 1 or 2, wherein the heating element region identification unit identifies the image region that is continuous in the horizontal direction as the heating element region when the image region with a predetermined brightness or luminance is continuous in the horizontal direction of the captured image.

4. A heating element region identification unit identifies, among multiple image regions constituting a captured image generated by photographing through the window glass of a vehicle on which heating elements are arranged, an image region whose brightness or luminance is less than or equal to a predetermined brightness or luminance, as a heating element region for displaying the heating elements. A corrected image generation unit that corrects the brightness or luminance of the heating element region included in the captured image to match the brightness or luminance of an image region other than the heating element region adjacent to the heating element region, thereby generating a corrected image of the captured image. The corrected image is stored in the image storage unit. An image processing system including [specific components / features].

5. For computers, Among the multiple image regions that constitute the captured image generated by taking a photograph through the window glass of a vehicle equipped with heating wires, the image region whose brightness or luminance is less than or equal to a predetermined brightness or luminance is identified as the heating wire region where the heating wires are displayed. The brightness or luminance of the heating element region included in the captured image is corrected to match the brightness or luminance of the image region adjacent to the heating element region, thereby generating a corrected image of the captured image. Display the corrected image. Image processing program.

Images