Image processing device and on-board control device

Abstract

To solve a problem in which it is not possible to increase the accuracy of the likelihood indicating that a captured image is in an unsatisfactory state.SOLUTION: An image processing device includes an input unit that acquires each captured image from a plurality of cameras installed in a vehicle, a capturing state detection unit that detects whether a malfunction has occurred in the capturing state of each of the captured images, a likelihood calculation unit that calculates, for each of the cameras, a likelihood indicating the degree of malfunction of the capturing state on the basis of each of the captured images, and a likelihood updating unit that updates the likelihood to a new one on the basis of a determination obtained by integrating the likelihoods for the respective cameras that are calculated by the likelihood calculating unit. The likelihood updated by the likelihood updating unit is output as the likelihood for each of the cameras.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an on-vehicle control device. [Background technology]

[0002] In recent years, there has been progress in the development of technology that recognizes the environment around a vehicle based on images captured by a camera installed in the vehicle and provides driving assistance based on the recognition results.In image recognition processing, if there are problems with the captured images by the camera due to water droplets, mud, backlight, etc. that hinder recognition, there will be cases where the image is incorrectly recognized or cannot be recognized at all.

[0003] Therefore, methods have been devised to detect camera malfunctions from captured images. For example, in Patent Document 1, halation is detected based on the brightness value in the image captured by the camera, and water droplets and dirt are detected based on edge feature amounts. In Patent Document 2, images captured by the left and right cameras of a stereo camera are compared, and dirt is determined to be present in areas where differences other than parallax occur. In Patent Document 3, reliability is defined based on image feature amounts of white line areas, and the reliability history of each camera is compared to determine that an abnormality has occurred in the lens of a camera with a relatively low reliability. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-197863 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-293672 [Patent Document 3] Japanese Patent Application Laid-Open No. 2014-115814 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 cannot detect the presence of a water film when the entire camera lens is covered with the water film because the effect of the water film does not appear in the edge feature. Furthermore, the technology disclosed in Patent Document 2 cannot detect the presence of a water film when the water film is equally present on both the left and right cameras and there is no difference between the left and right captured images other than parallax. In these cases, the water film distorts light like a lens, making the area of ​​water film adhesion appear enlarged in the camera image, resulting in calculated values ​​that differ from the true value in image recognition processing, etc. The technology disclosed in Patent Document 3 has the problem of being unable to improve the accuracy of the likelihood that a captured image indicates a problem with the shooting condition. [Means for solving the problem]

[0006] The image processing device according to the present invention includes an input unit that acquires each captured image from a plurality of cameras installed in a vehicle, a capture state detection unit that detects whether a problem has occurred in the capture state of each of the captured images, a likelihood calculation unit that calculates a likelihood indicating the degree of the problem in the capture state for each of the cameras based on each of the captured images, and a likelihood update unit that updates the likelihood to a new likelihood based on a determination that integrates the likelihood for each of the cameras calculated by the likelihood calculation unit. a target recognition unit that recognizes targets present in each of the images captured by the plurality of cameras; Equipped with the likelihood update unit compares the lane width recognized by the target object recognition unit or the curvature of the road during travel calculated from the target object recognized by the target object recognition unit between the multiple cameras, and if there is a camera among the multiple cameras whose lane width or road curvature significantly deviates from the other cameras, updates the likelihood by that camera to increase it; if there is no camera among the multiple cameras whose lane width or road curvature significantly deviates from the other cameras, compares the lane width or road curvature with the lane width or curvature of the road recorded in map data, and if the comparison results in a significant difference, updates the likelihood by each of the multiple cameras to increase it; The likelihood updated by the likelihood update unit is output as the likelihood for each camera. The in-vehicle control device according to the present invention includes an image processing device and a control processing device that executes processing control based on the likelihood updated by the likelihood update unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to increase the accuracy of the likelihood that a problem has occurred in the shooting state of an image captured by a camera. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an on-board control device according to a first embodiment. [Figure 2]4 is a flowchart showing the processing of the image processing device in the first embodiment. [Figure 3] 4A to 4C are diagrams illustrating likelihood update processing by a likelihood update unit in the first embodiment. [Figure 4] FIG. 10 is a configuration diagram of an on-board control device according to a second embodiment. [Figure 5] FIG. 10 is a diagram illustrating a likelihood update process performed by a likelihood update unit in the second embodiment. [Figure 6] FIG. 10 is a configuration diagram of an on-board control device according to a third embodiment. [Figure 7] FIG. 13 is a diagram illustrating a likelihood update process performed by a likelihood update unit in the fourth embodiment. [Figure 8] FIG. 13 is a diagram illustrating a likelihood update process performed by a likelihood update unit in the fifth embodiment. [Figure 9] FIG. 20 is a diagram illustrating a likelihood update process performed by a likelihood update unit in the eighth embodiment. [Figure 10] FIG. 20 is a diagram illustrating a likelihood update process performed by a likelihood update unit in the ninth embodiment. [Figure 11] FIG. 22 is a configuration diagram of an on-board control device according to a tenth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0010] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0011] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. However, when there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0012] [First Example] 1 is a configuration diagram of an on-board control device 1000 according to the first embodiment. The on-board control device 1000 includes an image processing device 100 and a control processing device 200.

[0013] The image processing device 100 is connected to multiple cameras 2, and also acquires vehicle specification data 3, CAN data 4, and time data 5. As will be described in detail later, it calculates and updates a likelihood indicating the degree of malfunction in the shooting conditions, and outputs the updated likelihood to the control processing device 200.

[0014] The cameras 2 are multiple on-board cameras installed on the vehicle, and are installed at predetermined positions on the vehicle, for example, at the front, rear, left, and right of the vehicle, each at a predetermined angle, to capture images of the surroundings of the vehicle. The cameras 2 may be installed either outside or inside the vehicle. The captured images obtained by each camera 2 are output to the image processing device 100 as analog data or after A / D conversion via a transmission path such as a dedicated line.

[0015] The vehicle specification data 3 includes information on the dimensions of the vehicle, the mounting position and angle of the camera 2, the angle of view, etc., and these values ​​are mainly used for calculating likelihoods in the image processing device 100. The vehicle specification data 3 may be stored in a medium in the image processing device 100, on which the values ​​are recorded, or may be acquired via a transmission path.

[0016] The CAN data 4 is vehicle information relating to the behavior of the vehicle, such as the vehicle speed, steering angle, wiper operation information, etc., and is input to the image processing device 100 via a CAN (Controller Area Network). By using the input vehicle information, the image processing device 100 determines whether the vehicle is traveling, whether it is traveling straight or curved, etc.

[0017] The time data 5 is input to the image processing device 100 from a GPS, a clock, etc. The time data 5 is used when the image processing device 100 checks time-series changes in the images captured by the camera 2. The GPS, the clock, etc. may be built into the image processing device 100.

[0018] The image processing device 100 includes an input unit 101, a shooting state detection unit 102, and a likelihood calculation unit 103 corresponding to each camera 2. The image processing device 100 further includes a likelihood update unit 104 and an output unit 105.

[0019] The input unit 101 acquires a captured image from the camera 2 and passes it to the capture state detection unit 102 . The shooting condition detection unit 102 analyzes each image captured by each camera 2 and detects for each camera 2 whether there is a problem with the shooting condition. If the shooting condition is poor, it detects that the camera 2 that captured the image is malfunctioning. If a malfunction has occurred, it determines the type of malfunction that is causing the malfunction. Types of malfunction include, for example, adhesion of water droplets, water film, mud, snow, etc., and backlighting. To detect or distinguish these, the position or size of the area on the captured image may be obtained, and the occurrence or type of malfunction may be determined based on these. Note that known techniques can be used to detect adhesion of matter to the lens of the camera 2 or backlighting.

[0020] The likelihood calculation unit 103 calculates a likelihood indicating the degree of malfunction of the shooting condition for each type of malfunction for each camera 2 based on each captured image captured by each camera 2 detected by the shooting condition detection unit 102. The likelihood is a value indicating the degree of malfunction to which the shooting condition is considered to be. This likelihood can be calculated from the shooting condition of the captured image by a known method for determining the shooting condition using machine learning or image features. In this embodiment, the likelihood is expressed as %. For example, if the likelihood is 80%, it is considered that there is a high possibility of malfunction. If the likelihood is 20%, it is considered that there is a low possibility of malfunction. If the likelihood is 0%, it is considered that there is an extremely low possibility of malfunction (or that the condition is normal). Note that the likelihood may be expressed in units or values ​​other than %.

[0021] The likelihood update unit 104 updates the likelihood to a new likelihood based on a determination that integrates the likelihoods for each camera 2 calculated by the likelihood calculation unit 103. For example, if there are multiple cameras 2 that have detected water droplets, the likelihoods for each camera 2 are integrated to determine that the vehicle is traveling in the rain or has just been washed, and the likelihood of the image capture state regarding water droplets for the camera 2 that detected the water droplets is increased to become the new likelihood for this camera 2.

[0022] The output unit 105 outputs the new likelihood updated by the likelihood update unit 104 to the control processing device 200 as the likelihood for each camera 2. At this time, if a malfunction has occurred, the type of malfunction and the position and area on the image captured by the camera 2 where the malfunction has occurred may be output for each camera 2.

[0023] The control processing device 200 is a driving assistance device 201, a cleaning control device 202, etc., and executes process control based on the updated likelihood. For example, it determines whether or not various malfunctions have occurred in the shooting state of the camera 2 based on the updated likelihood. In addition, it changes the process control according to the updated likelihood.

[0024] The driving assistance device 201 recognizes various objects such as surrounding vehicles, motorcycles, pedestrians, bicycles, stationary objects, signs, white lines, curbs, and guardrails based on the images captured by the camera 2, and issues an alert to the user or controls the behavior of the vehicle based on the recognition results and the updated likelihood. Note that the area on the image captured by the camera 2 that performs recognition differs for each target object. In particular, the driving assistance device 201 refers to the likelihood output from the image processing device 100 to continue or interrupt processing, or to perform function degradation that gradually restricts the driving assistance function.

[0025] For example, the driving assistance device 201 executes driving assistance processing control when the updated likelihood is lower than a threshold. Since the degree of impact of a malfunction is thought to differ depending on the target of image recognition, a threshold is set for each recognition target and each type of malfunction. For example, in the case of a lane departure warning device based on the result of white line detection by image recognition processing, if the likelihood of a malfunction is higher than a predetermined threshold, it may be determined that the image recognition processing cannot operate normally and the warning processing may simply be stopped. Alternatively, even if the likelihood of a malfunction is higher than the predetermined threshold, it may be determined whether to stop or continue the warning processing based on whether the area on the captured image used for white line detection overlaps with the area where the malfunction has occurred.

[0026] The cleaning control device 202 includes a wiper for wiping off water droplets and the like from the glass surface of an optical input / output device such as the camera 2, a washer for washing off dirt, etc. The cleaning control device 202 controls the start, continuation, and end of wiping and cleaning based on the likelihood and the like output from the image processing device 100. The optical input / output devices include the camera 2, headlights, backlights, and drive recorders.

[0027] For example, if the updated likelihood is higher than a threshold, the cleaning control device 202 executes cleaning control of the optical input / output device provided in the vehicle. Specifically, if the likelihood of water droplets or water film is higher than a predetermined threshold, the cleaning control device 202 operates the wiper attached to the target camera 2, and if the likelihood of mud is higher than another predetermined threshold, the cleaning control device 202 operates the washer.

[0028] As described above, the likelihood of the photographing state needs to be calculated with high accuracy because it is an important parameter that determines the behavior of the processing control of the control processing device 200, such as the driving assistance device 201 and the cleaning control device 202. In this embodiment, as will be described later, it is possible to increase the accuracy of the likelihood that a problem has occurred in the photographing state of the image photographed by the camera.

[0029] FIG. 2 is a flowchart showing the processing of the image processing device 100. In step S201, images captured by each camera 2 are acquired from the input unit 101.

[0030] In process S202, the shooting condition detection unit 102 detects the shooting condition of each camera 2 (normal, water droplets, water film, mud, snow, backlight, etc.) from the acquired images shot by each camera 2. If it is determined that the condition is not normal, i.e. that a malfunction has occurred in a camera 2, the area on the image shot by the malfunctioning camera 2 is identified, and the position, size, etc. of the water droplets, water film, mud, snow, backlight, etc. are extracted to detect the shooting condition. Known techniques can be used to identify the shooting condition.

[0031] In step S203, the likelihood calculation unit 103 calculates the likelihood of each of the detected image capturing states of each camera 2.

[0032] In process S204, the likelihood update unit 104 updates the likelihood of each shooting state of each camera 2 based on the determination that integrates the likelihood of each shooting state of each camera 2. The update of the likelihood will be described in detail later.

[0033] In process S205, the output unit 105 outputs the updated likelihood of each shooting state of each camera 2 to the control processing device 200.

[0034] 3 is a diagram illustrating the likelihood update process performed by the likelihood update unit 104. When there are multiple cameras 2 in which the shooting condition detection unit 102 has detected the same type of malfunction, the likelihood update unit 104 updates the likelihood of the shooting condition of each camera 2.

[0035] In the example of FIG. 3, four cameras 2a, 2b, 2l, and 2r (front, rear, left, and right) are connected to the image processing device 100. The likelihoods determined in process S203 are as follows: the likelihood of water droplets R for the front camera 2a is 50%, the likelihood of water droplets R for the left camera 2l is 50%, the likelihood of water droplets R for the right camera 2r is 60%, and the likelihood of water droplets R for the rear camera 2b is 10%. In this case, for example, a likelihood of 50% or more is determined to indicate the possibility of water droplets. If there are multiple cameras with the possibility of water droplets, it is determined that the vehicle was traveling in the rain or had just been washed, and the likelihood of water droplets R for each camera 2a, 2b, 2l, and 2r is increased by 30%. The likelihoods for each camera 2a, 2b, 2l, and 2r updated in this way are output from the image processing device 100. Note that the 30% increase in the likelihood of water droplets R is merely an example, and the appropriate increase should be determined using actual data. In addition, in the example of Figure 3, the increment in likelihood of water droplets R for each camera 2 is set to be equal, but the increment in likelihood of the captured state by the front camera 2a and rear camera 2b, which are often used for downstream functions such as lane departure warning, may be set to be larger than that of the left camera 2l and right camera 2r.

[0036] The conditions for updating the likelihood are not limited to the number of cameras 2 on which water droplets are likely to adhere, but for example, wiper information from CAN data may be used to determine that when the wipers are operating, it is likely that it is raining or the car has just been washed, and the likelihood of water droplets or a water film may be increased.Furthermore, in addition to water droplets and water films, for example, mud is expected to adhere to multiple cameras 2 due to being kicked up while driving on an unpaved road, and snow is expected to adhere to multiple cameras 2 while driving during snowfall, so the likelihood of these malfunctions may also be increased when there are multiple cameras 2 on which adhesion is likely to occur.

[0037] As described above, by updating to a new likelihood based on a judgment that integrates the likelihood based on each captured image of each camera 2, the accuracy of the likelihood based on each captured image of each camera 2 can be improved, and erroneous processing control caused by the capturing state of camera 2 can be avoided in the processing control of the control processing device 200, which is the subsequent processing.

[0038] [Second Example] Fig. 4 is a configuration diagram of an on-board control device 1000 in the second embodiment. The second embodiment differs from the first embodiment in that it includes a target recognition unit 106. The same components as those in the first embodiment shown in Fig. 1 are denoted by the same reference numerals, and their description will be omitted.

[0039] As shown in FIG. 4, the image processing device 100 includes an input unit 101, a shooting state detection unit 102, a likelihood calculation unit 103, a likelihood update unit 104, an output unit 105, and a target recognition unit 106.

[0040] The target recognition unit 106 recognizes targets such as white lines, signs, curbs, and guardrails that exist in each image captured by the multiple cameras 2. Publicly known techniques can be used for the recognition process of each target. The target recognition results by the target recognition unit 106 are output to the likelihood update unit 104.

[0041] If there are multiple cameras 2 that have detected the same type of malfunction by the imaging condition detection unit 102, the likelihood update unit 104 compares the target recognition results of each camera 2 and updates the likelihood of the imaging condition from each camera 2.

[0042] FIG. 5 is a diagram illustrating the likelihood update process performed by the likelihood update unit 104. In FIG. In the example of Figure 5, four cameras 2a, 2b, 2l, and 2r on the front, rear, left, and right sides of the vehicle are connected to the image processing device 100. If a water film W adhering to the lens of the rear camera 2b or the vehicle window is captured in the captured image and the water film W distorts the light like a lens, the area of ​​the water film adhering to the image captured by the rear camera 2b will be enlarged. However, if the likelihood is not updated, incorrect operations will be performed in the processing control at the later stage, and it may be difficult to detect the water film using a single camera 2.

[0043] In this embodiment, the likelihood update unit 104 compares the width of the white line L recognized by the target object recognition unit 106 between the cameras 2a, 2b, 2l, and 2r, and updates the likelihood of the image capture state of the water film W captured by the rear camera 2b based on the comparison result. Specifically, as shown in Fig. 5, it is assumed that the likelihood calculation unit 103 has obtained the likelihood of the image capture state of the four cameras 2a, 2b, 2l, and 2r on the front, rear, left, and right sides, and that the target object recognition unit 106 has obtained the white line width on the image captured by each of the cameras 2a, 2b, 2l, and 2r.

[0044] Specifically, the likelihood calculated by the likelihood calculation unit 103 is 20% for the rear camera 2b only and 10% for the other cameras 2a, 2l, and 2r. The width of the white line L calculated by the target recognition unit 106 is 20 cm for the rear camera 2b only and 15 cm for the other cameras 2a, 2l, and 2r. In this case, since the width of the white line L is 20 cm for the rear camera 2b only and 15 cm for the other cameras 2a, 2l, and 2r, the likelihood update unit 104 determines that there is a possibility that a water film W is attached to the rear camera 2b and increases the likelihood of the water film W for the rear camera 2b from 20% to 80%. This makes it possible to suppress the influence of the attachment of the water film W by the rear camera 2b in subsequent processing control. However, the 60% increase in the likelihood of the water film W is just an example, and the appropriate increase should be determined using actual data.

[0045] In addition, if a water film W is attached to all of the cameras 2 connected to the image processing device 100 in the same manner, there will be no difference when comparing the recognition results of the width of the white line L from each camera 2, making it difficult to detect the water film W. However, since there is a specified value for the width of the white line L set by law, it is possible to update the likelihood that a water film W may be attached to a camera 2 that recognizes a width of the white line L that deviates from this specified value.

[0046] In this way, the likelihood update unit 104 updates the likelihood of the image capture state by comparing the target recognition results by the target recognition unit 106 between the cameras 2. Specifically, adhesion of a water film W, which is difficult to detect with a single camera 2, is determined by integrating the target recognition results from multiple cameras 2, for example, by treating the target recognition results from many cameras 2 as positive. Therefore, by updating the likelihood of the image capture state of the water film W of a camera 2 on which there is a possibility of adhesion of the water film W, erroneous process control caused by the image capture state of the camera 2 can be avoided in the process control of the control processing device 200, which is a subsequent process.

[0047] [Third Example] Fig. 6 is a configuration diagram of an on-board control device 1000 in a third embodiment. The third embodiment differs from the second embodiment in that it references map data 6. The same components as those in the second embodiment shown in Fig. 4 are denoted by the same reference numerals, and their description will be omitted.

[0048] 6, the image processing device 100 refers to map data 6 from a map database or the like. The map database or the like stores values ​​related to landmarks such as lane width (the width between white lines) and curvature of each road.

[0049] The target recognition unit 106 recognizes targets such as white lines on the image captured by the camera 2, and calculates the lane width of the road on which the vehicle is traveling.

[0050] The likelihood update unit 104 compares the target recognition results of each camera 2 with the target-related values ​​in the map data 6 to determine whether or not a malfunction has occurred in the camera 2, and updates the likelihood of the shooting state by the camera 2 that has been determined to be malfunctioning.

[0051] An example will be described in which the target object is the lane width of a road. The likelihood update unit 104 compares the lane widths recognized by the target object recognition unit 106 between the cameras 2 and updates the likelihood of the water film captured by each camera 2 based on the comparison results. That is, the likelihood update unit 104 compares the lane width calculation values ​​of each camera 2, determines that the camera 2 with a larger lane width than the others has a possibility of water film adhesion, and increases the likelihood of the water film for that camera 2. Furthermore, if the difference in the lane width values ​​calculated between the cameras 2 is small, it determines that each camera 2 has a possibility of water film adhesion, and compares the calculated lane width value with the lane width of the road currently being traveled recorded in the map data 6. If the values ​​differ significantly as a result, it determines that a water film has adhered to each camera 2, and increases the likelihood of the water film captured by each camera 2.

[0052] An example will be described in which the landmark is the curvature of a road. The landmark recognition unit 106 calculates the curvature of the road on which the vehicle is traveling from landmarks such as white lines, guardrails, and curbs. The likelihood update unit 104 compares the calculated curvature values ​​of each camera 2, determines that the camera 2 with a curvature value that deviates from the others may have a water film attached, and increases the likelihood of the water film-related image capture state of that camera 2. This utilizes the phenomenon in which the calculated curvature value differs from the true value as a result of the distortion of the contour of the landmark in the image captured by the camera 2 due to the attachment of a water film. Furthermore, if the difference in the curvature values ​​calculated between the cameras 2 is small, the calculated curvature is compared with the curvature of the road on which the vehicle is traveling recorded in the map data 6. If the resulting values ​​differ significantly, it is determined that a water film has attached to each camera 2, and increases the likelihood of the water film-related image capture state of each camera 2.

[0053] In this way, by referring to the map data 6 and updating the likelihood regarding the photographing state of the water film W of the camera 2 where there is a possibility of the water film W adhering, the likelihood can be more accurately reflected, thereby avoiding erroneous processing control caused by the photographing state of the camera 2 in the processing control of the control processing device 200, which is the subsequent processing.

[0054] [Fourth Example] In this embodiment, the time series change of the water droplets is monitored and the likelihood is updated. The configuration diagram of the on-board control device 1000 in this embodiment is the same as that shown in Fig. 1 in the first embodiment, Fig. 4 in the second embodiment, or Fig. 6 in the third embodiment, and therefore is not shown.

[0055] FIG. 7 is a diagram illustrating the likelihood update process performed by the likelihood update unit 104. In FIG. The likelihood update unit 104 monitors the time series changes in the water droplets R in the images captured by each camera 2 detected by the image capture state detection unit 102, compares the time series changes in the water droplets R with predefined patterns of time series changes in the water droplets R, and determines that an external factor common to each camera 2 has occurred if there are multiple images captured by the camera 2 that show time series changes in the water droplets R similar to the pattern, and updates the likelihood of the image capture state of each camera 2 based on the determined external factor. Here, the predefined pattern of time series changes in the water droplets R is, for example, a pattern of time series changes in the water droplets R obtained by test driving a vehicle. The external factor common to each camera 2 is rainfall, snowfall, etc.

[0056] 7, a front camera 2a and a right camera 2r of a vehicle are connected to an image processing device 100. Water droplets R adhering to the lenses of the front camera 2a and the right camera 2r and the vehicle windows are captured in the captured image, and the positions of the water droplets R move due to wind pressure while the vehicle is moving. Whether the vehicle is moving or not is determined by whether the vehicle speed value in the CAN data 4 is equal to or greater than a predetermined threshold.

[0057] While the vehicle is traveling, the position of the adhering water droplets R moves, and as time passes from t to t+1 to t+2, the water droplets R are detected by the image capturing state detection unit 102 based on the images captured by the cameras 2, and the positions of the water droplets R on the images captured by each camera 2 also gradually move.

[0058] 7, the likelihood update unit 104 calculates the movement direction of the water drop R from the movement of the position of the water drop R in the images captured by each camera 2 from time t to time t+2. Then, by comparing the movement direction of the water drop R with the movement direction of the water drop R in the images captured by each camera 2 previously obtained in a driving experiment or the like, if the movement direction is similar, it is determined that the water drop R is likely to be an external factor. The movement direction of the water drop R can be calculated, for example, by the least squares method or the like from the position of the water drop R region at each time, and whether the directions are similar or not is determined, for example, by whether the absolute value of the dot product of the calculated movement direction of the water drop R and the previously obtained movement direction of the water drop R is less than a predetermined threshold.

[0059] If there are multiple cameras 2 that have determined that the image is likely to be a water droplet R, i.e., if an external factor occurs that is common to all cameras 2, it is determined that the vehicle is traveling in the rain, and the likelihood of the water droplet R captured by each camera 2 is increased. In the example of FIG. 7, the likelihood of the water droplet R is increased from 60% to 90% for the front camera 2a and from 60% to 80% for the right camera 2r. However, the increase in the likelihood of the water droplet R is merely an example, and the appropriate increase is determined using actual data. Note that while FIG. 7 focuses on two of the cameras 2, the front camera 2a and the right camera 2r, the likelihood update process may also be similarly implemented in a configuration with three or more cameras 2. Furthermore, while this embodiment focuses on the case of a water droplet R, the likelihood update process can also be performed in the case of an object that moves across the image captured by the camera 2 while traveling, such as melting snow.

[0060] In this way, by detecting adhesions such as water droplets R or snow moving on the image captured by camera 2 and updating the likelihood, the adhesions can be accurately reflected in the likelihood, thereby avoiding erroneous processing control caused by the shooting state of camera 2 in the processing control of the control processing device 200, which is the subsequent processing.

[0061] [Fifth Example] In this embodiment, backlight is detected and the likelihood is updated. The configuration diagram of the on-board control device 1000 in this embodiment is the same as that shown in Fig. 1 in the first embodiment, Fig. 4 in the second embodiment, or Fig. 6 in the third embodiment, and therefore is not shown in the figures.

[0062] FIG. 8 is a diagram illustrating the likelihood update process performed by the likelihood update unit 104. In FIG. The upper diagram in Figure 8 shows an example in which sunlight 300 is shining from behind vehicle 100a during the day, and the likelihood of the rear camera 2b of cameras 2 capturing an image related to backlighting is high. Whether it is daytime or not is determined from time data 5. During the daytime, backlighting does not occur on camera 2a, which is installed in the opposite direction to camera 2b, which is experiencing backlighting. The lower diagram of FIG. 8 shows an example in which the image captured by the front camera 2a includes a white line L, and the image captured by the rear camera 2b includes the white line L and a backlit area RF. The likelihood update unit 104 determines whether the likelihood of backlighting is equal to or greater than a predetermined threshold based on the backlighting region RF captured by the rear camera 2b. For example, in this example, if the threshold is 70% or greater, it determines that the front camera 2a, which is installed in the opposite direction, is unlikely to experience backlighting, and reduces the likelihood of the backlighting-related image capture state of the front camera 2a. In the example of FIG. 8, the backlighting likelihood of the front camera 2a is reduced from 30% to 10%. The backlighting likelihood of the rear camera 2b remains unchanged at 70%. However, the amount by which the backlighting likelihood is reduced is merely an example, and the appropriate value is determined using actual data.

[0063] That is, when backlighting is detected as a type of malfunction by the shooting condition detection unit 102, the likelihood update unit 104 updates the likelihood of the shooting condition related to backlighting by the camera 2a, which is installed facing in the opposite direction to the camera 2b for which backlighting was detected, so as to decrease it. This allows the likelihood to be updated depending on whether the direction is backlighting or not, so that errors in the processing control of the control processing device 200, which is the subsequent processing, can be avoided even when the camera 2 is backlit.

[0064] The likelihood update unit 104 may switch and update the likelihood of the captured state depending on the installation position of the camera 2 on the vehicle. For example, the likelihood of the captured state by the rear camera 2b may be increased more than the likelihood of the captured state by the front camera 2a, or the likelihood of the captured state by a camera located in the traveling direction depending on whether the vehicle is moving forward or backward may be decreased, or the likelihood of the captured state by a camera 2 located in the opposite direction depending on the vehicle's orientation and time.

[0065] [Sixth Example] In this embodiment, the likelihood is updated by regarding the shooting state of the highly reliable camera 2 as positive. The configuration diagram of the on-board control device 1000 in this embodiment is the same as that shown in Fig. 1 in the first embodiment, Fig. 4 in the second embodiment, or Fig. 6 in the third embodiment, and therefore is not shown.

[0066] When the camera 2 is installed inside the vehicle, particularly when the camera 2 is installed inside a window with a wiper, it can take pictures in an environment where the wiper has removed water droplets, water film, mud, snow, and other adhering matter, and poor image quality caused by adhering matter is less likely to occur compared to a camera 2 installed outside the vehicle. In other words, the image quality taken by the camera 2 installed inside the vehicle is more reliable than the image quality taken by the camera 2 installed outside the vehicle.

[0067] The likelihood update unit 104 determines whether a malfunction has occurred in the shooting state of the camera 2 by comparing the shooting state and the target recognition result of the camera 2 installed in the vehicle cabin as correct, and updates the likelihood of the shooting state of the camera 2 determined to be malfunctioning. Specifically, it increases the likelihood of the shooting state of the camera 2 determined to be malfunctioning.

[0068] Furthermore, even if a camera 2 is installed outside the vehicle, if there is a camera 2 equipped with a wiping function for wiping off deposits such as with a wiper, the likelihood update unit 104 may compare the image captured by the camera 2 as an input with the image captured by the camera 2 as positive, and update the likelihood of the image captured by each camera 2. Specifically, the likelihood of the image captured by the camera 2 determined to be malfunctioning is increased.

[0069] In addition, a case will be described in which a camera 2 installed inside or outside the vehicle has a wiping function for wiping off deposits with a wiper or the like. In this case, the likelihood update unit 104 may receive a signal indicating the operating state of the wiping function, and may perform a comparison assuming that the image captured by the camera 2 within a predetermined time immediately after the wiping function is activated is the positive image capture state, and update the likelihood of the image capture state of each camera 2. Specifically, the likelihood of the image capture state by the camera 2 determined to be malfunctioning is increased.

[0070] Alternatively, the likelihood update unit 104 may determine whether a malfunction has occurred in the camera 2 by comparing the current captured image of each camera 2 with the captured image of the camera 2 that has been determined to be malfunctioning in the past based on a previous driving experiment or the like, and update the likelihood of the captured state by the camera 2 that has been determined to be malfunctioning. Specifically, the likelihood of the captured state by the camera 2 that has been determined to be malfunctioning is increased.

[0071] Furthermore, the likelihood update unit 104 may regard the target recognition result from the past when the shooting state of the camera 2 was normal as positive, compare it with the current target recognition result from each camera 2, determine whether or not a malfunction has occurred in the camera 2, and update the likelihood of the shooting state by the camera 2 determined to be malfunctioning. Specifically, the likelihood of the shooting state by the camera 2 determined to be malfunctioning is increased.

[0072] In this way, by comparing the captured images with the shooting state of the highly reliable camera 2 as the correct one, the likelihood regarding the shooting state of camera 2 is updated, thereby avoiding erroneous processing control caused by the shooting state of camera 2 in the processing control of the control processing device 200, which is the subsequent processing.

[0073] [Seventh Example] In this embodiment, the likelihood is updated using a pre-recorded target recognition result during normal operation. The configuration diagram of the on-board control device 1000 in this embodiment is the same as that shown in Fig. 1 in the first embodiment, Fig. 4 in the second embodiment, or Fig. 6 in the third embodiment, and therefore is not shown.

[0074] In the second, third, and sixth embodiments, the target recognition results are used to update the likelihood of the captured image state, but in many cases the values ​​of the white line width and lane width do not change over time. In this embodiment, the likelihood update unit 104 pre-records the target recognition results when the captured image state is normal in a storage unit, and compares the stored target recognition results with the current target recognition results, assuming them to be correct, to update the likelihood of the captured image state of each camera 2. Specifically, the likelihood of the captured image state by the camera 2 determined to be malfunctioning is increased.

[0075] In this way, by using the pre-recorded recognition results of targets under normal conditions, the likelihood regarding the shooting state of camera 2 is appropriately updated even when there is no specified value for the white line width or map data 6, so that erroneous processing control caused by the shooting state of camera 2 can be avoided in the processing control of the control processing device 200, which is the subsequent processing.

[0076] [Eighth Example] In this embodiment, the likelihood is updated using the shooting state of the common shooting area on the captured image. The configuration diagram of the on-board control device 1000 in this embodiment is the same as that of the first embodiment shown in Fig. 1, the second embodiment shown in Fig. 4, or the third embodiment shown in Fig. 6, and therefore is not shown in the figures.

[0077] FIG. 9 is a diagram illustrating the likelihood update process performed by the likelihood update unit 104. In FIG. Among the sets of cameras 2, there may be a set that captures a common area in three-dimensional space depending on the installation position, direction, and angle of view of the cameras 2. The area in which that area is captured on the images captured by each camera 2 is called the common capturing area C. Note that the common capturing area C can be calculated using the vehicle specification data 3, and although it will actually have a complex shape, it is shown as a rectangle in Figure 9 for ease of explanation.

[0078] When a pair of cameras 2 each having a common imaging area C on the captured image of the cameras 2 detects a malfunction in the common imaging area C of only one of the cameras 2, the likelihood update unit 104 updates the likelihood of the imaging state of this one camera 2 to increase it.

[0079] Specifically, as shown in FIG. 9, the case is illustrated where the front camera 2a and the right camera 2r of the cameras 2 have a common imaging area C. For the right camera 2r, there is no water droplet R in the common imaging area C, and the image captured by the right camera 2r has a likelihood of 10%, which is approximately normal. On the other hand, for the front camera 2a, the image captured is a water droplet, which has a high likelihood of 60%, and the water droplet R is present in the common imaging area C. When an object in three-dimensional space is captured in the common imaging area C, the object appears in both the common imaging area C of the front camera 2a and the common imaging area C of the right camera 2r in the image captured by the front camera 2a. Therefore, if an object appears in only one of the common imaging areas C, it is determined that there is a high possibility that the area is experiencing a problem with the imaging condition of the corresponding camera 2.

[0080] In the example shown in Fig. 9, the likelihood update unit 104 determines that there is a high possibility that a water droplet R is captured in the common imaging area C of the front camera 2a, and increases the likelihood of the image capture state of the front camera 2a from 60% to 80%. However, the increase in likelihood due to the water droplet R is just an example, and an appropriate value will be determined using actual data. Note that while Fig. 9 shows the example of a water droplet R, the likelihood of the malfunction may also be updated in the same way when a malfunction of another type of camera 2 occurs in the common imaging area C.

[0081] In this way, by using the shooting state of the common imaging area on the captured image, the likelihood regarding the shooting state of camera 2 is appropriately updated, thereby avoiding erroneous processing control caused by the shooting state of camera 2 in the processing control of the control processing device 200, which is the subsequent processing.

[0082] [Ninth Example] In this embodiment, the likelihood is updated using a target in a common imaging area on the captured image. The configuration diagram of the on-board control device 1000 in this embodiment is the same as that of the first embodiment shown in FIG. 1, the second embodiment shown in FIG. 4, or the third embodiment shown in FIG. 6, and therefore is not shown in the figures.

[0083] FIG. 10 is a diagram illustrating the likelihood update process performed by the likelihood update unit 104. In FIG. As in the eighth embodiment, the pair of cameras 2 has a common imaging area C, and FIG. 10 shows an example in which the common imaging area C exists between the front camera 2a and the right camera 2r.

[0084] The likelihood update unit 104 increases the likelihood of the imaging state of the camera 2 in which the target does not exist when the target recognized by the target recognition unit 106 exists only in the common imaging area C of one of the cameras 2 in a pair of cameras 2 each having a common imaging area C on the imaging image of the camera 2.

[0085] Specifically, as shown in FIG. 10, when the image capturing conditions of the front camera 2a and the right camera 2r are both normal, if the target recognition unit 106 recognizes a white line L from the common image capturing area C of the image captured by the right camera 2r, the white line L should also be recognized from the common image capturing area C of the image captured by the front camera 2a. If the white line L is recognized from the common image capturing area C of the image captured by the right camera 2r but the white line L is not recognized from the common image capturing area C of the image captured by the front camera 2a, the likelihood update unit 104 determines that there may be a malfunction in the common image capturing area C of the front camera 2a. However, whether or not a target is recognized cannot determine what type of malfunction is occurring in the camera 2. Therefore, the likelihood update unit 104 increases the likelihood of each type of the front camera 2a. Note that the appropriate value for the likelihood increase is determined using actual data.

[0086] In this way, by using targets within the common imaging area on the captured image, the likelihood regarding the imaging state of camera 2 is appropriately updated, thereby avoiding erroneous processing control caused by the imaging state of camera 2 in the processing control of the control processing device 200, which is the subsequent processing.

[0087] [Tenth Example] Fig. 11 is a configuration diagram of an on-board control device 1000 in a tenth embodiment. The tenth embodiment differs from the second embodiment in that it includes a radar 7. The same components as those in the second embodiment shown in Fig. 4 are denoted by the same reference numerals, and their description will be omitted.

[0088] 11, a radar 7 is connected to the image processing device 100. The radar 7 is a three-dimensional information detection unit that detects three-dimensional information of targets present around the vehicle. The likelihood update unit 104 compares the three-dimensional information of the target detected by the three-dimensional information detection unit with the image captured by the camera 2, and if there is no target corresponding to the three-dimensional information in the captured image, increases the likelihood of the captured state of the captured image.

[0089] Specifically, as shown in FIG. 11 , the likelihood update unit 104 compares the image of each target recognized by the target recognition unit 106 captured by the camera 2 with the 3D information of the target obtained from the radar 7. If there is a target that is detected by the radar 7 but not in the image captured by the camera 2, the likelihood update unit 104 determines that the image captured by the camera 2 is in a poor state and increases the likelihood of the image captured by the camera 2. Note that because the type of malfunction cannot be identified by target comparison alone, the likelihood update unit 104 increases the likelihood for each type. Note that the appropriate value for the likelihood increase is determined using actual data. Furthermore, although the 3D information detection unit has been described using the example of the radar 7, other means capable of acquiring 3D information of targets, such as millimeter waves or ultrasonic waves, may also be used.

[0090] In this way, by using the three-dimensional information detection unit, the likelihood regarding the shooting state of camera 2 is appropriately updated, thereby avoiding erroneous processing control caused by the shooting state of camera 2 in the processing control of the control processing device 200, which is the subsequent processing.

[0091] The image processing device 100 shown in each of the above embodiments has been described as including an input unit 101, an image capture state detection unit 102, a likelihood calculation unit 103, a likelihood update unit 104, an output unit 105, and a target recognition unit 106. However, some or all of these components may be realized by a processor (e.g., a CPU or a GPU) and a program executed by the processor. When executed by the processor, the program performs a predetermined process using storage resources (e.g., a memory) and / or an interface device (e.g., a communication port) as appropriate, so the processor may be the subject of the process. Similarly, the subject of the process performed by executing the program may be a controller, device, system, computer, or node having a processor. The subject of the process performed by executing the program may be a calculation unit, and may include a dedicated circuit (e.g., an FPGA or an ASIC) that performs a specific process.

[0092] A program may be installed on a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Furthermore, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0093] Information such as programs, tables, and files that realize some or all of the functions of the image processing device 100 can be stored in a storage device such as a memory, a hard disk, or a solid state drive (SSD), or in a recording medium such as an IC card, an SD card, or a DVD. Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines necessary for implementation. In reality, it can be assumed that almost all of the components are interconnected.

[0094] According to the embodiment described above, the following effects can be obtained. (1) The image processing device 100 includes an input unit 101 that acquires each captured image from a plurality of cameras 2 installed in a vehicle, a capture condition detection unit 102 that detects whether a problem has occurred in the capture condition of each captured image, a likelihood calculation unit 103 that calculates a likelihood indicating the degree of the problem in the capture condition for each camera 2 based on each captured image, and a likelihood update unit 104 that updates the likelihood to a new likelihood based on a determination that integrates the likelihood for each camera 2 calculated by the likelihood calculation unit 103, and outputs the likelihood updated by the likelihood update unit 104 as the likelihood for each camera 2. This makes it possible to improve the accuracy of the likelihood that indicates a problem has occurred in the capture condition of the images captured by the cameras.

[0095] The present invention is not limited to the above-described embodiments, and various embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention as long as they do not impair the characteristics of the present invention. Furthermore, configurations combining the above-described embodiments may be used. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations. [Explanation of symbols]

[0096] 2 Camera, 3 Vehicle specification data, 4 CAN data, 5 Time data, 100 Image processing device, 101 Input unit, 102 Shooting state detection unit, 103 Likelihood calculation unit, 104 Likelihood update unit, 105 Output unit, 200 Control processing device, 201 Driving assistance device, 202 Cleaning control device, 1000 On-board control device.

Claims

1. an input unit that acquires images captured by a plurality of cameras installed in the vehicle; a photographing condition detection unit that detects whether a malfunction has occurred in the photographing condition of each of the photographed images; a likelihood calculation unit that calculates a likelihood indicating a degree of poor photographing condition for each of the cameras based on each of the photographed images; a likelihood update unit that updates the likelihood to a new likelihood based on a determination that integrates the likelihoods calculated for the cameras by the likelihood calculation unit; a target recognition unit that recognizes targets present in each of the images captured by the plurality of cameras, The likelihood update unit comparing the lane width recognized by the target object recognition unit or the curvature of the road on which the vehicle is traveling, calculated from the target object recognized by the target object recognition unit, between the plurality of cameras; If there is a camera among the plurality of cameras whose lane width or road curvature is significantly different from that of the other cameras, updating is performed to increase the likelihood of the camera; If there is no camera among the plurality of cameras whose lane width or curvature of the road is significantly different from that of the other cameras, the lane width or curvature of the road is compared with the lane width or curvature of the road recorded in map data, and if the comparison result is significantly different, the likelihoods of the respective cameras are updated so as to be increased; The image processing device outputs the likelihood updated by the likelihood update unit as the likelihood for each camera.

2. 2. The image processing device according to claim 1, the photographing state detection unit detects a malfunction of the camera based on the photographing state and determines a type of the malfunction; the likelihood calculation unit calculates a likelihood related to the malfunction for each type of the malfunction; The likelihood update unit is an image processing device that updates the likelihood related to a water film among the likelihoods for each type of malfunction calculated by the likelihood calculation unit.

3. 3. The image processing device according to claim 2, When there are a plurality of cameras in which the shooting state detection unit has detected the same type of malfunction, the likelihood update unit updates the likelihood of the shooting state of each camera.

4. The image processing device according to any one of claims 1 to 3; a control processor that executes processing control based on the likelihood updated by the likelihood update unit.

5. The on-board control device according to claim 4, The control processing device is a driving assistance device, and the driving assistance device is an in-vehicle control device that executes driving assistance processing control when the updated likelihood is lower than a threshold value.

6. The on-board control device according to claim 4, The control processing device is a cleaning control device, and the cleaning control device is an on-vehicle control device that executes cleaning control of an optical input / output device provided in a vehicle when the updated likelihood is higher than a threshold value.

Images