Method for determining whether camera is contaminated in real time and computing device using the same

Abstract

To provide a method for determining whether a camera is contaminated in real time.SOLUTION: A method includes the steps of: determining whether or not reliability indicating whether an input video is suitable for contamination detection satisfies a predetermined condition if the input video is created by a camera; executing a process of applying calculation of convolution neutral network to the input video and classifying a contamination state according to each predetermined fine part area when it is determined that the input video satisfies the predetermined condition, and deriving a frame contamination degree for each of the fine part areas on the basis of the classified contamination states, and a process of acquiring an edge map of the input video for each of the fine part areas by utilizing a brightness difference between adjacent pixels, and deriving frame sharpness for each of the fine part areas on the basis of the acquired edge map; and determining whether the camera is contaminated by referring to the frame contamination degree and the frame sharpness.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method for determining in real time whether a camera is contaminated or not, and a computing device using the same. [Background technology]

[0002] Recently, autonomous driving systems or advanced driver assistance systems (ADAS) have been applied to vehicles to ensure driver convenience and safety. These autonomous driving systems or advanced driver assistance systems require sensors such as cameras, radars, and LIDAR to function properly.

[0003] However, if contamination occurs on the camera lens, for example, if water droplets or dust get on it, the quality of the image captured through the camera deteriorates rapidly. If an autonomous driving system or advanced driver assistance system uses an image of degraded quality, stable system operation cannot be guaranteed, which may pose a risk to the safety of the driver or pedestrians.

[0004] In relation to this, there was a prior art (Korean Patent Application No. 10-2016-0048179) that determined lens contamination by comparing the clarity and sensitivity of an image with the reference values ​​of a normal image.

[0005] However, according to Korean Patent Application No. 10-2016-0048179, there was a problem that accuracy was reduced because it was difficult to define a generalized standard value and areas such as the sky, ground, or the surface of a flat object showed values ​​similar to contaminated patterns.

[0006] According to another prior art (Korean Patent Application No. 10-2021-0034734), the camera compares images before and after being illuminated by a vehicle light source such as a headlight, and if there is no change in the image, it is determined that there is contamination.

[0007] However, according to Korean Patent Application No. 10-2021-0034734, when there is translucent contamination, the contaminated area is affected by the light source, making it difficult to distinguish and reducing accuracy.

[0008] In addition, according to another conventional technique (JP Patent Publication 2008-060874), pixel movement due to vehicle movement is measured and compared with pixel movement estimates based on optical flow to determine camera contamination.

[0009] However, according to JP 2008-060874 A, it is only possible to determine the presence or absence of contamination for specific parts where pixel movement can be measured, and there was a problem that it was difficult to estimate optical flow in environments where the difference between pixels is not large, such as the sky or road surfaces.

[0010] Also, according to another prior art (US Pat. No. 9,319,637), the pixels of an image were continuously monitored and if there was no significant change in brightness, the camera was determined to be contaminated.

[0011] However, according to US Pat. No. 9,319,637, there is a problem in that accuracy decreases in a vehicle driving environment where objects with little difference, such as the sky and road surfaces, are repeatedly observed.

[0012] In addition, according to another prior art (Korean Patent Application No. 10-2017-0140891), camera contamination was detected by extracting image features using a Convolution Neural Network and combining them with similarly extracted features from consecutive images.

[0013] However, according to Korean Patent Application No. 10-2017-0140891, the problem of diverse distribution of contamination shapes or locations was simplified to the problem of determining the contamination state, which required learning for various contamination patterns; otherwise, there was a problem of reduced accuracy in localized contamination or areas such as the ground or sky.

[0014] Therefore, in terms of a method for determining contamination on a camera lens, there is currently no camera contamination detection system that can distinguish between transparent and semi-transparent contamination, determine contamination even when the brightness difference between pixels is small, such as on roads, ground, and sky, and overcome inaccuracies that may be contained in the inference results of a convolutional neural network. [Prior art documents] [Patent documents]

[0015] [Patent Document 1] Korean Patent Application No. 10-2016-0048179 [Patent Document 2] Korean Patent Application No. 10-2021-0034734 [Patent Document 3] JP 2008-060874 A [Patent Document 4] U.S. Pat. No. 9,319,637 [Patent Document 5] Korean Patent Application No. 10-2017-0140891 Summary of the Invention [Problem to be solved by the invention]

[0016] An object of the present invention is to solve all of the above problems.

[0017] Another object of the present invention is to effectively detect contamination patterns that are difficult to classify using generalized criteria, such as translucent contamination, by using a convolutional neural network that has been trained on various image patterns.

[0018] Another object of the present invention is to accurately determine in real time whether a camera is contaminated or not based on an image captured by the camera.

[0019] Another object of the present invention is to accurately detect contaminated areas in real time even in areas with monotonous patterns such as the sky or ground by combining the inference results of a deep neural network with image clarity. [Means for solving the problem]

[0020] According to an embodiment of the present invention, there is provided a method for determining whether or not a camera is contaminated in real time, comprising the steps of: (a) when an input image is generated by a camera, a contamination detection system determines whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition using a contamination detection time determination unit; (b) when it is determined that the input image satisfies the preset condition, the contamination detection system performs the steps of: (i) providing the input image to a contamination degree calculation unit, and then using the contamination degree calculation unit to classify a contamination state for each detailed region by applying a convolutional neural network operation to the input image, and then deriving a frame contamination degree for each detailed region based on the classification; and (ii) providing the input image to a sharpness calculation unit, and then using the sharpness calculation unit to obtain an edge map for each detailed region using a brightness difference between adjacent pixels for the input image, and then deriving a frame sharpness for each detailed region based on the classification; and (c) causing a contamination determination unit in the contamination detection system to determine whether or not the camera is contaminated by referring to the frame contamination degree and the frame definition.

[0021] In one example, in step (a), the contamination detection system acquires behavior information including a moving distance and a rotation angle of the camera from an external sensor, and provides the behavior information to the contamination detection time determination unit. With respect to the preset condition for determining whether the input image is suitable for contamination detection, the contamination detection time determination unit determines at least (i) whether the moving distance acquired from the behavior information exceeds a preset critical moving distance, (ii) whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) whether feature points of the input image acquired from the camera are matched and a ratio of the matching feature points is less than a preset critical matching feature point ratio, and (iv) whether an intermediate value of feature point disparity of the input image is calculated and whether the intermediate value of feature point disparity exceeds a preset critical value of feature point disparity, and based on the determined result, determines whether the reliability satisfies at least one preset condition.

[0022] In one example, the contamination detection system matches previous feature points extracted from a previous input image previously acquired by the camera with current feature points extracted from a current input image currently acquired by the camera using the contamination detection time determination unit, and if it is determined that a ratio of matched matching feature points is equal to or greater than the critical matching feature point ratio, obtains a feature point disparity vector from positions and directions between the matched previous feature points and the current feature points, calculates a median value of a magnitude distribution of the feature point disparity vector, and determines whether the median value of the magnitude distribution of the feature point disparity vector exceeds a critical value of the feature point disparity.

[0023] In one example, the contamination detection system, using the contamination detection time determination unit, (i) refers to the behavior information acquired from the external sensor and compares the movement distance with the critical movement distance, and if it is determined that the movement distance exceeds the critical movement distance, performs the step (b), and if it is determined that the movement distance is equal to or less than the critical movement distance, acquires the rotation angle from the external sensor, and (ii) refers to the behavior information acquired from the external sensor and compares the rotation angle acquired from the external sensor with the critical rotation angle, and if it is determined that the rotation angle exceeds the critical rotation angle, performs the step (b), and if it is determined that the rotation angle is equal to or less than the critical rotation angle, performs the step (b). (iii) if it is determined that a ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio, the step (b) is performed, and if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio, the feature point disparity is obtained, and (iv) if it is determined that the intermediate value of the distribution of the magnitude of the feature point disparity exceeds the critical value of the feature point disparity as a result of calculating the intermediate value, the step (b) is performed, and if it is determined that the intermediate value is equal to or less than the critical value of the feature point disparity, the contamination detection and determination process is terminated.

[0024] In one example, in step (b), when n contamination states, which are any contamination states including at least a transparent state, a semi-transparent state, and an opaque state, are set, the contamination detection system uses the output values ​​obtained for each of the detailed regions through the convolutional neural network operation by the contamination degree calculation unit to obtain a contamination probability map indicating the contamination state for each of the detailed regions, and applies a weight vector individually preset for each of the contamination states to the contamination probability map to derive the frame contamination degree for each of the detailed regions.

[0025] In one example, the contamination detection system is configured to increase the weight vector as the degree of contamination approaches the opaque state among the n contamination states, which are in a range from the transparent state to the opaque state, and to have a negative weight vector for a contamination state that is close to the transparent state within a preset critical value among the n contamination states, and to apply the weight vector to the contamination probability map using the contamination degree calculation unit to derive the frame contamination degree for each detailed region.

[0026] In one example, in step (b), the contamination detection system acquires the frame clarity for each detailed region by applying the edge map and an attenuation constant (the attenuation constant is a constant for adjusting the amount of change in the frame clarity due to the previous frame clarity) to the previous frame clarity derived for each detailed region for the previous input image by the clarity calculation unit.

[0027] In one example, in step (c), the contamination detection system uses the contamination judgment unit to combine the frame contamination level and the frame clarity obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculates a corrected contamination level for each detailed region by referring to the previous contamination level and the current contamination level derived for each detailed region for the previous input image, and then determines whether the corrected contamination level is greater than or equal to a preset value. If a certain number of first specific detailed regions are greater than or equal to the preset value, the contamination detection system determines that the camera is contaminated.

[0028] In one example, when a region importance calculated by referring to at least a portion of the position importance of each detailed region, the class importance of the object, and the moving direction of the object is set for each detailed region of the camera, the contamination detection system determines that the camera is contaminated using the contamination determination unit when the sum of the region importances of a second specific detailed region whose corrected contamination level is equal to or greater than a preset value is equal to or greater than a preset critical contaminated region importance.

[0029] In one example, the contamination detection system is characterized in that the clarity calculation unit is configured to output the first frame clarity when a specific frame clarity exceeds a predetermined first frame clarity, to output the second frame clarity (the second frame clarity is a value smaller than the first frame clarity) when the specific frame clarity is less than a predetermined second frame clarity, to output the specific frame clarity when the specific frame clarity has a value between the second frame clarity and the first frame clarity, to output the first corrected contamination degree when a specific corrected contamination degree exceeds a predetermined first corrected contamination degree, to output the second corrected contamination degree (the second corrected contamination degree is a value smaller than the first corrected contamination degree) when the specific corrected contamination degree is less than a predetermined second corrected contamination degree, and to output the specific corrected contamination degree when the specific corrected contamination degree has a value between the second corrected contamination degree and the first corrected contamination degree.

[0030] According to another embodiment of the present invention, a computing device for determining whether a camera is contaminated in real time includes at least one memory for storing instructions; and at least one processor configured to execute the instructions. The processor includes: (I) a process for, when an input image is generated by a camera, determining whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition using a contamination detection time determination unit; (II) if it is determined that the input image satisfies the preset condition, (i) providing the input image to a contamination degree calculation unit, and then performing a convolutional neural network (CNNN) on the input image using the contamination degree calculation unit. and (iii) a process for determining whether or not the camera is contaminated by a contamination determination unit by referring to the frame contamination degree and the frame clarity.

[0031] In one example, in the process (I), the processor acquires behavior information including a moving distance and a rotation angle of the camera from an external sensor, and provides the behavior information to the contamination detection time determination unit. With respect to the preset condition for determining whether the input image is suitable for contamination detection, the processor at least (i) determines whether the moving distance acquired from the behavior information exceeds a preset critical moving distance, (ii) determines whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) matches feature points of the input image acquired from the camera, and determines whether a ratio of the matching feature points is less than a preset critical matching feature point ratio, and (iv) calculates an intermediate value of feature point disparity of the input image, and determines whether the intermediate value of the feature point disparity exceeds a preset critical value of feature point disparity, and determines whether the reliability satisfies at least one preset condition based on the above.

[0032] In one example, the processor matches previous feature points extracted from a previous input image previously acquired by the camera with current feature points extracted from a current input image currently acquired, using the contamination detection time determination unit, and when it is determined that a ratio of matched matching feature points is equal to or greater than the critical matching feature point ratio, obtains a feature point disparity vector from positions and directions between the matched previous feature points and the current feature points, calculates a median value of a magnitude distribution of the feature point disparity vector, and determines whether the median value of the magnitude distribution of the feature point disparity vector exceeds a critical value of the feature point disparity.

[0033] In one example, the processor, using the contamination detection time determination unit, (i) refers to the behavior information acquired from the external sensor and compares the movement distance with the critical movement distance, and if it is determined that the movement distance exceeds the critical movement distance, performs the (II) process, and if it is determined that the movement distance is equal to or less than the critical movement distance, acquires the rotation angle from the external sensor, and (ii) refers to the behavior information acquired from the external sensor and compares the rotation angle acquired from the external sensor with the critical rotation angle, and if it is determined that the rotation angle exceeds the critical rotation angle, performs the (II) process, and if it is determined that the rotation angle is equal to or less than the critical rotation angle, performs the (II) process. (iii) if it is determined that a ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio, the process (II) is performed; if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio, the feature point disparity is obtained; (iv) if it is determined that the intermediate value of the distribution of the magnitude of the feature point disparity exceeds the critical value of the feature point disparity as a result of calculating the intermediate value, the process (II) is performed; if it is determined that the intermediate value is equal to or less than the critical value of the feature point disparity, the contamination detection and determination process is terminated.

[0034] In one example, in the process (II), the processor sets n contamination states, which are any contamination states including at least a transparent state, a semi-transparent state, and an opaque state, and uses the output values ​​obtained for each of the detailed regions through the convolutional neural network operation by the contamination degree calculation unit to obtain a contamination probability map indicating the contamination state for each of the detailed regions, and applies a weight vector individually preset for each of the contamination states to the contamination probability map to derive the frame contamination degree for each of the detailed regions.

[0035] In one example, the processor is configured to increase the weighting value of the weight vector as the degree of contamination approaches the opaque state among the n contamination states, which are in a range from the transparent state to the opaque state, and to have a negative weighting value for a contamination state that is close to the transparent state within a preset critical value among the n contamination states, and to apply the weighting value vector to the contamination probability map using the contamination degree calculation unit to derive the frame contamination degree for each detailed region.

[0036] In one example, in the (II) process, the processor acquires the frame clarity for each detailed region by reflecting the edge map and an attenuation constant (the attenuation constant is a constant for adjusting the amount of change in the frame clarity due to the previous frame clarity) for each detailed region to the previous frame clarity derived for each detailed region for the previous input image using the clarity calculation unit.

[0037] In one example, in the (III) process, the processor uses the contamination judgment unit to combine the frame contamination level and the frame clarity obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculates a corrected contamination level for each detailed region by referring to the previous contamination level and the current contamination level derived for each detailed region for the previous input image, and then determines whether the corrected contamination level is greater than or equal to a preset value, and determines that the camera is contaminated if there are a certain number or more of first specific detailed regions that are greater than or equal to the preset value.

[0038] In one example, the processor is configured to set a region importance for each detailed region of the camera, the region importance being calculated by referring to at least a portion of the position importance for each detailed region, the class importance of the object, and the moving direction of the object, and to determine that the camera is contaminated using the contamination determination unit when the sum of the region importances of a second specific detailed region, the corrected contamination level of which is equal to or greater than a preset value, is equal to or greater than a preset critical contaminated region importance.

[0039] In one example, the processor controls the clarity calculation unit to output the first frame clarity when a specific frame clarity exceeds a predetermined first frame clarity, to output the second frame clarity (the second frame clarity is a value smaller than the first frame clarity) when the specific frame clarity is less than a predetermined second frame clarity, to output the specific frame clarity when the specific frame clarity has a value between the second frame clarity and the first frame clarity, to output the first corrected pollution degree when a specific corrected pollution degree exceeds a predetermined first corrected pollution degree, to output the second corrected pollution degree (the second corrected pollution degree is a value smaller than the first corrected pollution degree) when the specific corrected pollution degree is less than a predetermined second corrected pollution degree, and to output the specific corrected pollution degree when the specific corrected pollution degree has a value between the second corrected pollution degree and the first corrected pollution degree. Effect of the Invention

[0040] The present invention has the effect of effectively detecting contamination patterns that are difficult to classify using generalized criteria, such as translucent contamination, by using a convolutional neural network that has been trained on various image patterns.

[0041] In addition, the present invention has an effect of accurately determining in real time whether a camera is contaminated or not based on the image captured by the camera.

[0042] In addition, the present invention has the effect of accurately detecting contaminated areas in real time even in areas with monotonous patterns such as the sky or ground by combining the inference results of a deep neural network with the clarity of an image.

[0043] The following drawings, which are attached to be used for explaining the embodiments of the present invention, are merely a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter, "ordinary engineers") can derive each of the other drawings based on these drawings without performing any inventive work. [Brief description of the drawings]

[0044] [Figure 1] 1 is a simplified diagram of a contamination detection system for determining in real time whether a camera is contaminated or not according to an embodiment of the present invention. [Diagram 2] 1 is a simplified flowchart illustrating an example of a process for determining whether a camera is contaminated in real time according to an embodiment of the present invention. [Diagram 3] 1 is an exemplary flowchart illustrating a process of determining a contamination detection time by a contamination detection time determination unit according to an embodiment of the present invention. [Figure 4] 13 is an exemplary diagram illustrating obtaining feature point disparity vectors from an input image according to an embodiment of the present invention. [Diagram 5] 4 illustrates an example of a process for acquiring frame definition from an input image according to an embodiment of the present invention. [Figure 6] 4 illustrates an example process of deriving a frame clarity and a frame corruption level from an input image and acquiring a corruption level according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] The following detailed description of the present invention will be made with reference to the accompanying drawings, which show, by way of example, specific embodiments in which the present invention may be practiced in order to clarify the objects, technical solutions, and advantages of the present invention. These embodiments are described in sufficient detail to enable those of ordinary skill in the art to practice the present invention.

[0046] Furthermore, throughout the detailed description of the present invention and the claims, the word "comprises" and variations thereof are not intended to exclude other technical features, additions, components or steps. Other objects, advantages and characteristics of the present invention will become apparent to those of ordinary skill in the art, in part from the description and in part from the practice of the present invention. The following examples and drawings are provided as illustrations and are not intended to limit the present invention.

[0047] Moreover, the present invention covers all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, a particular shape, structure, and characteristic described herein in relation to one embodiment may be embodied in another embodiment without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention is limited only by the appended claims with the full scope of equivalents that such claims are entitled to if properly interpreted. In the drawings, like reference numerals refer to the same or similar functions across the various aspects.

[0048] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order that those skilled in the art to which the present invention pertains can easily carry out the present invention.

[0049] FIG. 1 is a simplified diagram of a contamination detection system for determining in real time whether a camera is contaminated or not according to an embodiment of the present invention.

[0050] 1, the contamination detection system 100 may include a memory 110 that stores instructions for detecting contamination of a camera, and a processor 120 that executes the instructions stored in the memory 110. In this case, the contamination detection system 100 may include various computing devices such as a server, a PC (Personal Computer), a tablet, a mobile computer, a PDA / EDA, a mobile phone, a smartphone, an IOT device, and the like.

[0051] In particular, the contamination detection system 100 may typically utilize a combination of computing devices (e.g., devices that may include computer processors, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers, switches, and the like; and electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular manner) to achieve the desired system performance.

[0052] The processor of the computing device may include hardware components such as a micro processing unit (MPU) or central processing unit (CPU), a cache memory, a data bus, etc. The computing device may further include software components such as an operating system and an application for performing a specific purpose.

[0053] However, this does not exclude the case where the computing device includes an integrated processor in which a medium, a processor and a memory for implementing the present invention are integrated.

[0054] Meanwhile, the contamination detection system 100 may be comprised of multiple devices performing various functions.

[0055] FIG. 2 is a simplified flowchart illustrating an example process for determining whether a camera is contaminated in real time according to an embodiment of the present invention.

[0056] 2, when an input image is generated by a camera, the contamination detection system 100 acquires the input image from the camera (S200), transmits the input image to a contamination detection time determination unit, and causes the contamination detection time determination unit to evaluate the reliability of the input image and determine whether to proceed with the contamination detection and determination process (S210). At this time, the reliability of the input image is a value indicating whether the input image is in a state suitable for contamination detection, and can be obtained by determining whether it satisfies at least one of preset conditions described below. Here, the contamination detection time determination unit may be included in the contamination detection system 100, but is not limited thereto.

[0057] If the contamination detection time determination unit determines that the reliability of the input image is sufficient, the contamination detection system 100 may provide the input image to the contamination degree calculation unit and the sharpness calculation unit. Here, the contamination degree calculation unit and the sharpness calculation unit may also be included in the contamination detection system 100, but are not limited thereto.

[0058] The contamination detection system 100 can classify the contamination state for each detailed region by applying a convolutional neural network operation to the input image using a contamination degree calculation unit, and based on this, derive a frame contamination degree for each detailed region (S221). Separately, the contamination detection system 100 can obtain an edge map for each detailed region by using a brightness difference between adjacent pixels for the input image using a sharpness calculation unit, and based on this, derive a frame sharpness for each detailed region (S222).

[0059] Thereafter, the contamination detection system 100 can determine whether or not the camera is contaminated by referring to the frame contamination degree and the frame definition using a contamination determination unit (S230). Here, the contamination determination unit may also be included in the contamination detection system 100, but is not limited thereto.

[0060] The operation of the contamination detection time determination unit will be described below.

[0061] First, the contamination detection system 100 may acquire an input image from a camera, and may further acquire behavior information including a movement distance and a rotation angle of the camera from an external sensor, and provide the behavior information to a contamination detection time determination unit. Thereafter, in order to determine whether a preset condition for a reliability for determining whether an input image is in a state suitable for contamination detection is satisfied, the contamination detection time determination unit may determine at least whether (i) a movement distance acquired from the behavior information exceeds a preset critical movement distance, (ii) a rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) a feature point of an input image acquired from a camera is matched, and a ratio of matching feature points is less than a preset critical ratio of matching feature points, and (iv) a feature point disparity median value of an input image is calculated, and a feature point disparity median value of the input image exceeds a preset feature point disparity threshold value. If an input image and behavior information corresponding to the input image satisfy at least one preset condition based on preset conditions including the above conditions, the input image may be determined to have a reliability suitable for contamination detection.

[0062] If the camera moves or rotates a lot, it is considered to have enough information to determine contamination since it has information about other physical positions, and even if the camera itself does not move or rotate enough, it is considered to have enough information to determine contamination if, for example, there is a sudden change in the surrounding environment or new information is added to the image due to the movement of surrounding objects, etc. Therefore, input images that satisfy these conditions are considered to have a reliability suitable for contamination detection (e.g., a reliability above a preset threshold value).

[0063] Hereinafter, a process of evaluating the reliability of an input image by the contamination detection time determination unit will be described with reference to FIG.

[0064] FIG. 3 is a flow chart illustrating an example of a process for determining a contamination detection time by a contamination detection time determination unit according to an embodiment of the present invention.

[0065] Referring to FIG. 3, the pollution detection system 100 may first acquire 300 an input image from a camera and 310 acquire behavior information from an external sensor, and provide the information to a pollution detection time determination unit. The pollution detection time determination unit may first acquire 320 a movement distance (of the camera) by referring to the behavior information, and compare 320 the movement distance with a critical movement distance. The movement distance of the camera may depend on, but is not limited to, the movement distance of the vehicle in which the camera is mounted. If it is determined that the movement distance exceeds the critical movement distance, a pollution detection and determination process is performed 370, and if it is determined that the movement distance is equal to or less than the critical movement distance, it may acquire a rotation angle (of the camera) by referring to the behavior information (determining that the camera does not have a reliability suitable for pollution detection). The rotation angle of the camera may depend on, but is not limited to, the movement angle of the vehicle in which the camera is mounted. Then, the rotation angle is compared with the critical rotation angle (330), and if it is determined that the rotation angle exceeds the critical rotation angle, a contamination detection and determination process is performed (370), and if it is determined that the rotation angle is equal to or less than the critical rotation angle, a previous feature point and a current feature point are obtained from the input image, and the previous feature point and the current feature point are matched (340). Then, a ratio of matching feature points obtained by matching the previous feature point and the current feature point is compared with a critical matching ratio (350). If it is determined that the ratio of matching feature points is less than the critical matching ratio, a contamination detection and determination process is performed (370), and if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature ratio, a feature point disparity is obtained based on the matched previous feature point and the current feature point. Then, the median value of the distribution of the magnitude of the feature point disparity is calculated, and the median value is compared 360 with the feature point disparity critical value. If it is determined that the median value exceeds the feature point disparity critical value, the contamination detection and judgment process is performed 370. If it is determined that the median value is equal to or less than the feature point disparity critical value, the contamination detection and judgment process is terminated 380.

[0066] The acquisition of the feature point disparity vector will be described with reference to FIG.

[0067] FIG. 4 is a diagram illustrating an example of obtaining feature point disparity vectors from an input image according to an embodiment of the present invention.

[0068] As described above, the contamination detection time determination unit generates a matching feature point ratio, which is a ratio of matched feature points between the previous feature point and the current feature point, and compares the matching feature point ratio with a critical matching feature point ratio. If it is determined that the matching feature point ratio is equal to or greater than the critical matching feature point ratio, a feature point disparity vector can be obtained from the positions and directions between the matched previous feature point and the current feature point.

[0069] Referring to FIG. 4, disparity vectors of feature points in an image at the current time t and feature points in a matched image at time t-dt are visualized. Since the magnitude of disparity is widely distributed due to the perspective effect, it is possible to effectively determine whether an image has changed by taking the median value as a representative value. That is, as shown in FIG. 4, the median value of the distribution for the magnitudes of multiple feature point disparity vectors is calculated, and it is possible to determine whether the median value exceeds the critical value of the feature point disparity.

[0070] Returning to FIG. 2, the process in which the sharpness calculation unit derives the frame sharpness (S222) after it is determined that the reliability of the input image satisfies the preset condition will be described in detail below. The frame sharpness can be calculated using the characteristic that an uncontaminated image has a large brightness difference between adjacent pixels. The sharpness calculation unit can generate an edge map by removing ambiguous change components from the input image and detecting only edges with strong components. Then, the frame sharpness can be calculated for each detailed region based on the edge map to derive the frame sharpness. At this time, the sharpness for each detailed region can be derived from an average value, a representative value, or a median value of the edge components of each detailed region. In the present invention, the "detailed region" may be each region generated when an image is divided into a predetermined grid shape, but is not limited thereto.

[0071] The process of deriving the frame sharpness will be described with reference to FIG.

[0072] FIG. 5 illustrates an example process of acquiring frame definition from an input image according to an embodiment of the present invention.

[0073] 5, edges can be detected 520 from an input image 510, and an edge map 530 can be obtained based on the detected edges 520. At this time, referring to the detected edges 520, it can be seen that when extracting edges from contaminated parts that can be seen in the input image 510, edges are hardly detected because the brightness difference between adjacent pixels is not large, unlike non-contaminated parts. The sharpness calculation unit generates an edge map 530 based on the edges 520 extracted from the input image 510, and can calculate the sharpness for each detailed area using the edge map 530.

[0074] 2, for convenience, the process of deriving frame sharpness (S222) will be described first. The contamination detection system 100 can reflect the sharpness of the previous frame when calculating the frame sharpness from the edge map using the sharpness calculation unit. Also, an attenuation constant can be further included to adjust the amount of change in the frame sharpness depending on the sharpness of the previous frame. This will be described in more detail later.

[0075] In addition, the contamination detection system 100 can output the first frame definition when the specific frame definition exceeds a predetermined first frame definition by using a definition calculation unit, output the second frame definition when the specific frame definition is less than a predetermined second frame definition, and output the specific frame definition as it is when the specific frame definition has a value between the second frame definition and the first frame definition, where the second frame definition corresponds to a value smaller than the first frame definition.

[0076] Therefore, the calculation formula for the frame sharpness for each detailed area performed by the sharpness calculation unit is as follows.

number

number

number

number

number

number

number

number

number

number

number

number

number

[0077] 2, looking in detail at the process of deriving the frame contamination level (S221), n ​​contamination states can be set for the contamination detection system 100, and the n contamination states can include at least a transparent state, a semi-transparent state, and an opaque state. The contamination detection system 100 can obtain a contamination probability map indicating the contamination state for each detailed region using output values ​​obtained for each detailed region through a convolutional neural network operation using a contamination level calculation unit.

[0078] At this time, an example of the contamination probability map can be seen in FIG.

[0079] FIG. 6 illustrates an example process of deriving a frame clarity and a frame corruption level from an input image and acquiring a corruption level according to an embodiment of the present invention.

[0080] 6, a contamination probability map 610 can be seen which visualizes the most probable contamination state inferred from a contamination probability map in which a contamination probability is derived for each specific detailed region by applying a convolutional neural network operation based on four contamination states. It can be seen that in the visualized contamination probability map 610, the opaque state which is the most probable contamination state based on the four contamination states is displayed in orange, a semi-transparent state close to the opaque state is displayed in yellow, a semi-transparent state close to the transparent state is displayed in gray, and a transparent state is displayed in white.

[0081] In addition, referring to Figure 2, the process of calculating the frame contamination level can be explained. After the contamination probability map is derived, a weight vector individually set for each contamination state can be applied to the contamination probability map to derive the frame contamination level for each detailed region.

[0082] In this case, the weight vector may be weighted higher as the contamination level approaches the opaque state among n contamination states, which are a range of contamination levels from transparent to opaque. Also, a contamination state that is closer to the transparent state within a preset critical value among the n contamination states may be set to have a negative weight. Based on the preset weight vector, the contamination detection system 100 may apply the weight vector to each corresponding contamination state to the contamination probability map using a contamination level calculation unit, thereby deriving a frame contamination level for each detailed region.

[0083] Once the frame contamination degree and frame clarity are derived from the contamination degree calculation unit and the clarity calculation unit, the contamination detection system 100 provides the frame contamination degree and frame clarity to the contamination judgment unit, and enables the contamination judgment unit to calculate the current contamination degree calculated for each detailed region by combining the frame contamination degree obtained for each detailed region and the frame clarity.

[0084] At this time, the contamination detection system 100 may calculate a corrected contamination level by reflecting the immediately preceding contamination level derived for each detailed region of the immediately preceding input image using the contamination determination unit on the current contamination level.

[0085] The pollution detection system 100 can also output a first corrected pollution level when the specific corrected pollution level exceeds a first predetermined corrected pollution level, output a second corrected pollution level when the specific corrected pollution level is less than a second predetermined corrected pollution level, and output a specific corrected pollution level when the specific corrected pollution level has a value between the second corrected pollution level and the first corrected pollution level. In this case, the second corrected pollution level can have a value smaller than the first corrected pollution level.

[0086] Therefore, the formula for calculating the corrected contamination level for each detailed area performed by the contamination determination unit is as follows.

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0087] The contamination detection system 100 determines whether the corrected contamination level calculated by the contamination determination unit is equal to or greater than a preset value, and if there are a certain number or more of first specific detailed areas, which are detailed areas having a value equal to or greater than the preset value, the contamination detection system 100 can determine that the camera is contaminated.

[0088] At this time, for each part of the area photographed by the camera, a region importance calculated by referring to at least a part of the position importance of each detailed area, the class importance of the object, and the moving direction of the object may be set for each detailed area. For example, the position importance of each detailed area may be higher toward the center, the class importance of the object may be set to reflect a person, a vehicle, or a guardrail, and the moving direction of the object may be set to reflect the moving direction of the person or the vehicle. In addition, the region importance may be set by taking into consideration driving habits, traffic regulations, or driving environment. When the contamination detection system 100 determines whether the camera is contaminated using the contamination determination unit, if the sum of the region importance of second specific detailed areas, which are detailed areas having a corrected contamination level equal to or higher than a preset value, is equal to or higher than a preset critical contamination region importance, the camera may be determined to be contaminated. Here, the second specific detailed area may be the same as the first specific detailed area, but is not limited thereto.

[0089] 6, the contamination detection system 100 uses a contamination degree calculation unit to derive a frame contamination degree for each detailed region by referring to a contamination probability map 610, uses a sharpness calculation unit to derive a frame sharpness for each detailed region by referring to an edge map 620, and then uses a contamination determination unit to derive a corrected contamination degree by referring to the frame contamination degree and the frame sharpness. Referring to FIG. 6, a corrected contamination degree 630 visualized for each detailed region can be seen.

[0090] Although the present invention has been described above using specific details such as specific components and limited examples and drawings, this is provided merely to aid in a more general understanding of the present invention, and the present invention is not limited to the above examples. Those skilled in the art will be able to make various modifications and variations from such descriptions.

[0091] Therefore, the concept of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the scope of the claims, as well as the scope of the claims described below, are within the scope of the concept of the present invention.

Claims

1. In a method for determining in real time whether a camera is contaminated or not, (a) when an input image is generated by a camera, the contamination detection system causes a contamination detection time determination unit to determine whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition; (b) if it is determined that the input image satisfies the preset condition, the contamination detection system performs: (i) a process of providing the input image to a contamination degree calculation unit, and then using the contamination degree calculation unit to classify contamination states into predetermined detailed regions by applying a convolutional neural network operation to the input image, and deriving a frame contamination degree for each detailed region based on the classification; and (ii) a process of providing the input image to a sharpness calculation unit, and then using the sharpness calculation unit to obtain an edge map for each detailed region using a brightness difference between adjacent pixels for the input image, and then deriving a frame sharpness for each detailed region based on the edge map; and (c) causing a contamination determination unit of the contamination detection system to determine whether or not the camera is contaminated by referring to the frame contamination degree and the frame definition; A method comprising:

2. In the step (a), 2. The method of claim 1, wherein the contamination detection system acquires behavior information including a moving distance and a rotation angle of the camera from an external sensor, and provides the behavior information to the contamination detection time determination unit, and, for the preset condition for determining whether the input image is suitable for contamination detection, determines, using the contamination detection time determination unit, at least: (i) whether the moving distance acquired from the behavior information exceeds a preset critical moving distance; (ii) whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle; (iii) match feature points of the input image acquired from the camera and determine whether a ratio of the matching feature points is less than a preset critical matching feature point ratio; and (iv) calculate an intermediate value of feature point disparity of the input image and determine whether the intermediate value of the feature point disparity exceeds a preset critical value of feature point disparity, and based on the determined condition, determines whether the reliability satisfies at least one preset condition.

3. 3. The method of claim 2, wherein the contamination detection system matches previous feature points extracted from a previous input image previously acquired by the camera with current feature points extracted from a current input image currently acquired, and if it is determined that a ratio of matched matching feature points is equal to or greater than a critical matching feature point ratio, obtains a feature point disparity vector from a position and a direction between the matched previous feature point and the current feature point, calculates a median value of a distribution of magnitudes of the feature point disparity vector, and determines whether the median value of the distribution of magnitudes of the feature point disparity vector exceeds a critical value of the feature point disparity.

4. The contamination detection system, using the contamination detection time determination unit, (i) compares the movement distance with the critical movement distance with reference to the behavior information acquired from the external sensor, and if it is determined that the movement distance exceeds the critical movement distance, performs step (b), and if it is determined that the movement distance is equal to or less than the critical movement distance, acquires the rotation angle from the external sensor, and (ii) compares the rotation angle acquired from the external sensor with the critical rotation angle with reference to the behavior information acquired from the external sensor, and if it is determined that the rotation angle exceeds the critical rotation angle, performs step (b), and if it is determined that the rotation angle is equal to or less than the critical rotation angle, performs step (b).

4. The method of claim 3, further comprising: obtaining a previous feature point and the current feature point; (iii) performing step (b) if it is determined that a ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio; obtaining the feature point disparity if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature ratio; and (iv) performing step (b) if it is determined that the intermediate value exceeds a critical value of the feature point disparity as a result of calculating the intermediate value of the feature point disparity magnitude distribution; and terminating a contamination detection and determination process if it is determined that the intermediate value is equal to or less than the critical value of the feature point disparity.

5. In the step (b), 2. The method of claim 1, wherein, in a state where n contamination states, which are any contamination states including at least a transparent state, a semi-transparent state, and an opaque state, are set, the contamination detection system obtains a contamination probability map indicating the contamination state for each detailed region using an output value obtained for each detailed region through the convolutional neural network operation by the contamination degree calculation unit, and applies a weight vector individually preset for each of the contamination states to the contamination probability map to derive the frame contamination degree for each detailed region.

6. The method of claim 5, characterized in that the contamination detection system applies the weight vector to the contamination probability map using the contamination degree calculation unit to derive the frame contamination degree for each detailed region, in a state where the weight vector is increased as the contamination degree approaches the opaque state among the n contamination states, which are in a range from the transparent state to the opaque state, and a contamination state among the n contamination states that is close to the transparent state within a preset critical value has a negative weight.

7. In the step (b), 2. The method of claim 1, wherein the contamination detection system obtains the frame sharpness for each detailed region by applying the edge map and an attenuation constant (the attenuation constant is a constant for adjusting the amount of change in the frame sharpness due to the previous frame sharpness) to the previous frame sharpness derived for each detailed region for the previous input image by the sharpness calculation unit.

8. In the step (c), The method of claim 1, wherein the contamination detection system uses the contamination judgment unit to combine the frame contamination level and the frame clarity obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculates a corrected contamination level for each detailed region by referring to the previous contamination level and the current contamination level derived for each detailed region for the previous input image, and then determines whether the corrected contamination level is greater than or equal to a preset value, and determines that the camera is contaminated if a certain number of first specific detailed regions are greater than or equal to the preset value.

9. The method of claim 8, characterized in that, when a region importance calculated by referring to at least a portion of the position importance for each detailed region, the object class importance, and the object's movement direction is set for each detailed region for the camera, the contamination detection system determines that the camera is contaminated using the contamination determination unit when the sum of the region importances of a second specific detailed region whose corrected contamination level is equal to or greater than a preset value is equal to or greater than a preset critical contaminated region importance.

10. the contamination detection system is configured to output the first frame definition when a specific frame definition exceeds a predetermined first frame definition using the definition calculation unit, to output the second frame definition when the specific frame definition is less than a predetermined second frame definition, the second frame definition being a value smaller than the first frame definition, and to output the specific frame definition when the specific frame definition has a value between the second frame definition and the first frame definition, 9. The method according to claim 8, characterized in that the contamination judgment unit is configured to output the first corrected contamination degree when a specific corrected contamination degree exceeds a predetermined first corrected contamination degree, to output the second corrected contamination degree (the second corrected contamination degree is a value smaller than the first corrected contamination degree) when the specific corrected contamination degree is less than a predetermined second corrected contamination degree, and to output the specific corrected contamination degree when the specific corrected contamination degree has a value between the second corrected contamination degree and the first corrected contamination degree.

11. A computing device that determines in real time whether a camera is contaminated or not. At least one memory storing instructions; and at least one processor configured to execute said instructions; 2. The computing device according to claim 1 , wherein the processor includes: (I) a process for, when an input image is generated by a camera, determining using a contamination detection time determination unit whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition; (II) a process for, when it is determined that the input image satisfies the preset condition, (i) a process for providing the input image to a contamination degree calculation unit, and then causing the contamination degree calculation unit to apply a convolutional neural network operation to the input image to classify a contamination state for each predetermined detailed region, and deriving a frame contamination degree for each detailed region based on the classification; and (ii) a process for providing the input image to a clarity calculation unit, and then causing the clarity calculation unit to obtain an edge map for each detailed region using a brightness difference between adjacent pixels for the input image, and then deriving a frame clarity for each detailed region based on the edge map; and (III) a process for determining whether the camera is contaminated or not by referring to the frame contamination degree and the frame clarity.

12. In the process (I), 12. The computing device of claim 11, wherein the processor acquires behavior information including a moving distance and a rotation angle of the camera from an external sensor, and provides the behavior information to the contamination detection time determination unit; and, for the preset condition for determining whether the input image is suitable for contamination detection, causes the contamination detection time determination unit to determine at least (i) whether the moving distance acquired from the behavior information exceeds a preset critical moving distance, (ii) whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) match feature points of the input image acquired from the camera and determine whether a ratio of the matching feature points is less than a preset critical matching feature point ratio, and (iv) calculate an intermediate value of feature point disparity of the input image and determine whether the intermediate value of feature point disparity exceeds a preset critical value of feature point disparity, and based on the determined result, determine whether the reliability satisfies at least one preset condition.

13. 13. The computing device of claim 12, wherein the processor matches previous feature points extracted from a previous input image previously acquired by the camera with current feature points extracted from a current input image currently acquired, and if it is determined that a ratio of matched matching feature points is equal to or greater than the critical matching feature point ratio, obtains a feature point disparity vector from a position and a direction between the matched previous feature point and the current feature point, calculates a median value of a distribution of magnitudes of the feature point disparity vector, and determines whether the median value of the distribution of magnitudes of the feature point disparity vector exceeds a critical value of the feature point disparity.

14. The processor, using the contamination detection time determination unit, (i) compares the movement distance with the critical movement distance with reference to the behavior information acquired from the external sensor, and if it is determined that the movement distance exceeds the critical movement distance, performs the process (II), and if it is determined that the movement distance is equal to or less than the critical movement distance, acquires the rotation angle from the external sensor, (ii) compares the rotation angle acquired from the external sensor with the critical rotation angle with reference to the behavior information acquired from the external sensor, and if it is determined that the rotation angle exceeds the critical rotation angle, performs the process (II), and if it is determined that the rotation angle is equal to or less than the critical rotation angle, performs the process (II).

14. The computing device of claim 13, further comprising: (iii) performing the process (II) if it is determined that a ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio; and (iv) performing the process (II) if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio; and (iv) performing the process (II) if it is determined that the intermediate value exceeds a critical value of the feature point disparity as a result of calculating the intermediate value for the feature point disparity magnitude distribution; and (v) terminating the contamination detection and determination process if it is determined that the intermediate value is equal to or less than the critical value of the feature point disparity.

15. In the process (II), 12. The computing device of claim 11, wherein the processor is configured to set n contamination states, which are any contamination states including at least a transparent state, a semi-transparent state, and an opaque state, and to obtain a contamination probability map indicating the contamination state for each detailed region using output values ​​obtained for each detailed region through the convolutional neural network operation by the contamination degree calculation unit, and to apply a weight vector individually preset for each of the contamination states to the contamination probability map to derive the frame contamination degree for each detailed region.

16. 16. The computing device of claim 15, wherein the processor is configured to increase a weight for the weight vector as the degree of contamination approaches the opaque state among the n contamination states that divide the range from the transparent state to the opaque state, and to have a negative weight for a contamination state that is close to the transparent state within a preset critical value among the n contamination states, and to apply the weight vector to the contamination probability map using the contamination degree calculation unit to derive the frame contamination degree for each detailed region.

17. In the process (II), 12. The computing device of claim 11, wherein the processor acquires the frame clarity for each detailed region by applying the edge map and a decay constant (the decay constant is a constant for adjusting the amount of change in the frame clarity due to the previous frame clarity) for each detailed region to the previous frame clarity derived for each detailed region for the previous input image by the clarity calculation unit.

18. In the process (III), 12. The computing device of claim 11, wherein the processor uses the contamination determination unit to combine the frame contamination level and the frame clarity obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculates a corrected contamination level for each detailed region by referring to the previous contamination level and the current contamination level derived for each detailed region for the previous input image, and then determines whether the corrected contamination level is greater than or equal to a preset value, and determines that the camera is contaminated if a certain number of first specific detailed regions are greater than or equal to the preset value.

19. 20. The computing device of claim 18, wherein the processor determines that the camera is contaminated when a sum of the region importances of a second specific detailed region, in which the corrected contamination level is equal to or greater than a preset value, is equal to or greater than a preset critical contaminated region importance, in a state in which a region importance calculated by referring to at least a portion of the position importance of each detailed region, the class importance of the object, and the moving direction of the object is set for each detailed region with respect to the camera, using the contamination determination unit.

20. the processor, using the definition calculation unit, outputs the first frame definition when a specific frame definition exceeds a predetermined first frame definition, outputs the second frame definition when the specific frame definition is less than a predetermined second frame definition (the second frame definition is a value smaller than the first frame definition), and outputs the specific frame definition when the specific frame definition has a value between the second frame definition and the first frame definition; 20. The computing device of claim 18, wherein the pollution judgment unit is configured to output the first corrected pollution degree when a specific corrected pollution degree exceeds a predetermined first corrected pollution degree, to output the second corrected pollution degree (the second corrected pollution degree is a value smaller than the first corrected pollution degree) when the specific corrected pollution degree is less than a predetermined second corrected pollution degree, and to output the specific corrected pollution degree when the specific corrected pollution degree has a value between the second corrected pollution degree and the first corrected pollution degree.

Images