METHOD FOR DETERMINING WHETHER CAMERA IS CONTAMINATED IN REAL TIME AND COMPUTING DEVICE USING THE SAME

Abstract

To provide a method for determining whether a camera is contaminated in real time.SOLUTION: A method includes the steps of: determining whether or not reliability indicating whether an input video is suitable for contamination detection satisfies a predetermined condition if the input video is created by a camera; executing a process of applying calculation of convolution neutral network to the input video and classifying a contamination state according to each predetermined fine part area when it is determined that the input video satisfies the predetermined condition, and deriving a frame contamination degree for each of the fine part areas on the basis of the classified contamination states, and a process of acquiring an edge map of the input video for each of the fine part areas by utilizing a brightness difference between adjacent pixels, and deriving frame sharpness for each of the fine part areas on the basis of the acquired edge map; and determining whether the camera is contaminated by referring to the frame contamination degree and the frame sharpness.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a method for determining whether a camera is contaminated in real time, and a computing device using the same. [Background technology]

[0002] Recently, autonomous driving systems or advanced driver assistance systems (ADAS) have been applied to vehicles to ensure driver convenience and safety. These autonomous driving systems or ADAS systems require sensors such as cameras, radar, and LIDAR to function properly.

[0003] However, if the camera lens becomes contaminated, for example, if water droplets or dust get on it, the quality of the image captured through the camera will deteriorate rapidly. If an autonomous driving system or an advanced driver assistance system uses the degraded image, stable system operation cannot be guaranteed, which may even pose a risk to the safety of the driver or pedestrians.

[0004] In relation to this, there was a prior art technology (Korean Patent Application No. 10-2016-0048179) that judged lens contamination by comparing the clarity and sensitivity of the image with the reference value of a normal image.

[0005] However, according to Korean Patent Application No. 10-2016-0048179, there was a problem that it was difficult to establish a generalized standard value, and areas such as the sky, ground, or the surface of a flat object showed values ​​similar to contaminated patterns, resulting in reduced accuracy.

[0006] According to another prior art (Korean Patent Application No. 10-2021-0034734), the camera compares images before and after being illuminated by a vehicle light source such as a headlight, and if there is no change in the image, it is determined that contamination has occurred.

[0007] However, according to Korean Patent Application No. 10-2021-0034734, when there is translucent contamination, the contaminated area is affected by the light source, making it difficult to distinguish and reducing accuracy.

[0008] In addition, according to another prior art technique (Japanese Patent Application Laid-Open No. 2008-060874), pixel movement due to vehicle movement is measured and compared with pixel movement estimates based on optical flow to determine camera contamination.

[0009] However, according to JP 2008-060874 A, it is only possible to determine whether or not there is contamination in specific areas where pixel movement can be measured, and there was a problem in that it was difficult to estimate optical flow in environments where there is not much difference between pixels, such as the sky or road surface.

[0010] Also, according to another prior art (US Pat. No. 9,319,637), the camera continuously monitors the pixels of the image and determines that there is contamination if there is no significant change in brightness.

[0011] However, according to US Pat. No. 9,319,637, there is a problem in that accuracy decreases in a vehicle driving environment where objects with little difference, such as the sky or road surface, are repeatedly observed.

[0012] In addition, according to another prior art (Korean Patent Application No. 10-2017-0140891), camera contamination was detected by extracting image features using a convolutional neural network and combining them with similarly extracted features from consecutive images.

[0013] However, according to Korean Patent Application No. 10-2017-0140891, the problem of diverse distribution of contamination shapes or locations was simplified to the problem of determining the contamination state, which required learning for various contamination patterns. Otherwise, there was a problem of reduced accuracy in areas with localized contamination or areas such as the ground or sky.

[0014] Therefore, in terms of a method for determining contamination on a camera lens, there is currently no camera contamination detection system that can distinguish between transparent and translucent contamination, determine contamination even when there is little brightness difference between pixels, such as on roads, ground, or sky, and overcome inaccuracies that may be present in the inference results of convolutional neural networks. [Prior art documents] [Patent documents]

[0015] [Patent Document 1] Korean Patent Application No. 10-2016-0048179 [Patent Document 2] Korean Patent Application No. 10-2021-0034734 [Patent Document 3] Japanese Patent Application Laid-Open No. 2008-060874 [Patent Document 4] U.S. Patent No. 9,319,637 [Patent Document 5] Korean Patent Application No. 10-2017-0140891 Summary of the Invention [Problem to be solved by the invention]

[0016] An object of the present invention is to solve all of the above-mentioned problems.

[0017] Another object of the present invention is to effectively detect contamination patterns that are difficult to classify using generalized criteria, such as translucent contamination, by using a convolutional neural network that has been trained on various image patterns.

[0018] Another object of the present invention is to accurately determine whether a camera is contaminated in real time through an image captured by the camera.

[0019] Another object of the present invention is to accurately detect contaminated areas in real time even in areas with monotonous patterns such as the sky or ground by combining the inference results of a deep neural network with image clarity. [Means for solving the problem]

[0020] According to one embodiment of the present invention, there is provided a method for determining whether a camera is contaminated in real time, comprising: (a) when an input image is generated by a camera, a contamination detection system causes a contamination detection time determination unit to determine whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition; (b) when it is determined that the input image satisfies the preset condition, the contamination detection system performs the following steps: (i) providing the input image to a contamination degree calculation unit, and then causing the contamination degree calculation unit to classify a contamination state into predetermined detailed regions by applying a convolutional neural network operation to the input image, and then deriving a frame contamination degree for each detailed region based on the classification; and (ii) providing the input image to a sharpness calculation unit, and then causing the sharpness calculation unit to obtain an edge map for each detailed region using a brightness difference between adjacent pixels in the input image, and then deriving a frame sharpness for each detailed region based on the edge map; and (c) the contamination detection system has a contamination determination unit determine whether the camera is contaminated by referring to the frame contamination level and the frame clarity.

[0021] In one example, in step (a), the contamination detection system acquires behavior information including a movement distance and a rotation angle of the camera from an external sensor and provides the behavior information to the contamination detection time determination unit. With respect to the preset conditions for determining whether the input image is suitable for contamination detection, the contamination detection time determination unit determines at least (i) whether the movement distance acquired from the behavior information exceeds a preset critical movement distance, (ii) whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) whether feature points of the input image acquired from the camera are matched and a ratio of the matching feature points is less than a preset critical ratio of the matching feature points, and (iv) whether the median value of the feature point disparity of the input image is calculated and a ratio of the feature point disparity exceeds a preset critical value of the feature point disparity, and based on the calculated results, determines whether the reliability satisfies at least one preset condition.

[0022] In one example, the contamination detection system uses the contamination detection time determination unit to match previous feature points extracted from a previous input image previously acquired by the camera with current feature points extracted from a current input image currently acquired, and if it is determined that a ratio of matched matching feature points is equal to or greater than the critical ratio of matching feature points, obtains a feature point disparity vector from positions and directions between the matched previous feature points and the current feature points, calculates a median value of a distribution of magnitudes of the feature point disparity vector, and determines whether the median value of the distribution of magnitudes of the feature point disparity vector exceeds a critical value of the feature point disparity.

[0023] In one example, the contamination detection system, using the contamination detection time determination unit, (i) refers to the behavior information acquired from the external sensor and compares the movement distance with the critical movement distance, and if it is determined that the movement distance exceeds the critical movement distance, performs step (b), and if it is determined that the movement distance is equal to or less than the critical movement distance, acquires the rotation angle from the external sensor; (ii) refers to the behavior information acquired from the external sensor and compares the rotation angle acquired from the external sensor with the critical rotation angle, and if it is determined that the rotation angle exceeds the critical rotation angle, performs step (b), and if it is determined that the rotation angle is equal to or less than the critical rotation angle, performs step (b). (iii) if it is determined that the ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio, it performs step (b); if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio, it performs step (b); and if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio, it performs step (b); and if it is determined that the ratio of matching feature points is equal to or less than the critical matching feature point ratio, it terminates the contamination detection and determination process.

[0024] In one example, in step (b), when n contamination states are set, which are arbitrary contamination states including at least a transparent state, a semi-transparent state, and an opaque state, the contamination detection system uses the output values ​​obtained for each of the detailed regions through the convolutional neural network operation by the contamination degree calculation unit to obtain a contamination probability map indicating the contamination state for each of the detailed regions, and applies a weight vector individually preset for each of the contamination states to the contamination probability map to derive the frame contamination degree for each of the detailed regions.

[0025] In one example, the contamination detection system is configured such that the weight vector is increased as the contamination level approaches the opaque state among the n contamination states, which are in a range from the transparent state to the opaque state, and a contamination state that is closer to the transparent state within a predetermined critical value among the n contamination states has a negative weight, and the contamination detection system applies the weight vector to the contamination probability map using the contamination level calculation unit to derive the frame contamination level for each detailed region.

[0026] In one example, in step (b), the contamination detection system acquires the frame sharpness for each detailed region by applying the edge map and an attenuation constant (the attenuation constant is a constant for adjusting the amount of change in the frame sharpness depending on the previous frame sharpness) to the previous frame sharpness derived for each detailed region for the previous input image using the sharpness calculation unit.

[0027] In one example, in step (c), the contamination detection system uses the contamination determination unit to combine the frame contamination level and the frame clarity obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculate an adjusted contamination level for each detailed region by referring to the previous contamination level and the current contamination level derived for each detailed region for the previous input image, and then determine whether the adjusted contamination level is greater than or equal to a preset value.If there are a certain number or more first specific detailed regions that are greater than or equal to the preset value, the contamination detection system determines that the camera is contaminated.

[0028] In one example, when a region importance calculated by referring to at least a portion of the position importance of each detailed region, the class importance of the object, and the movement direction of the object is set for each detailed region of the camera, the contamination detection system determines that the camera is contaminated using the contamination determination unit if the sum of the region importances of second specific detailed regions whose corrected contamination level is equal to or greater than a preset value is equal to or greater than a preset critical contamination region importance.

[0029] In one example, the contamination detection system uses the clarity calculation unit to output the first frame clarity when a specific frame clarity exceeds a predetermined first frame clarity, to output the second frame clarity (the second frame clarity is a value smaller than the first frame clarity) when the specific frame clarity is less than a predetermined second frame clarity, and to output the specific frame clarity when the specific frame clarity has a value between the second frame clarity and the first frame clarity, and to use the contamination judgment unit to output the first corrected contamination degree when a specific corrected contamination degree exceeds a predetermined first corrected contamination degree, to output the second corrected contamination degree (the second corrected contamination degree is a value smaller than the first corrected contamination degree) when the specific corrected contamination degree is less than a predetermined second corrected contamination degree, and to output the specific corrected contamination degree when the specific corrected contamination degree has a value between the second corrected contamination degree and the first corrected contamination degree.

[0030] According to another embodiment of the present invention, a computing device for determining whether a camera is contaminated in real time includes at least one memory for storing instructions; and at least one processor configured to execute the instructions. The processor includes: (I) a process for determining, when an input image is generated by a camera, whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition using a contamination detection time determination unit; (II) if it is determined that the input image satisfies the preset condition, (i) providing the input image to a contamination level calculation unit, and then running a convolutional neural network (CNNN) on the input image using the contamination level calculation unit. a process for applying a computation of a network (Network) to classify contamination states for each of the detailed regions and deriving a frame contamination level for each of the detailed regions based on the classification; (ii) a process for providing the input image to a sharpness calculation unit, and then using the sharpness calculation unit to obtain an edge map for each of the detailed regions using a brightness difference between adjacent pixels in the input image, and deriving a frame clarity for each of the detailed regions based on the edge map; and (III) a process for using a contamination determination unit to determine whether the camera is contaminated by referring to the frame contamination level and the frame clarity.

[0031] In one example, in the process (I), the processor acquires behavior information including a movement distance and a rotation angle of the camera from an external sensor and provides the behavior information to the contamination detection time determination unit. With respect to the preset conditions for determining whether the input image is suitable for contamination detection, the contamination detection time determination unit determines at least (i) whether the movement distance acquired from the behavior information exceeds a preset critical movement distance, (ii) whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) whether feature points of the input image acquired from the camera are matched and a ratio of the matching feature points is less than a preset critical ratio of the matching feature points, and (iv) whether the median value of the feature point disparity of the input image is calculated and a ratio of the feature point disparity exceeds a preset critical value of the feature point disparity, and based on the calculated results, determines whether the reliability satisfies at least one preset condition.

[0032] In one example, the processor, using the contamination detection time determination unit, matches previous feature points extracted from a previous input image previously acquired by the camera with current feature points extracted from a currently acquired current input image, and if it is determined that a ratio of matched matching feature points is equal to or greater than the critical ratio of matching feature points, obtains a feature point disparity vector from positions and directions between the matched previous feature points and the current feature points, calculates a median value of a distribution of magnitudes of the feature point disparity vector, and determines whether the median value of the distribution of magnitudes of the feature point disparity vector exceeds a critical value of the feature point disparity.

[0033] In one example, the processor, using the contamination detection time determination unit, (i) refers to the behavior information acquired from the external sensor and compares the movement distance with the critical movement distance, and if it is determined that the movement distance exceeds the critical movement distance, performs the process (II), and if it is determined that the movement distance is equal to or less than the critical movement distance, obtains the rotation angle from the external sensor; (ii) refers to the behavior information acquired from the external sensor and compares the rotation angle acquired from the external sensor with the critical rotation angle, and if it is determined that the rotation angle exceeds the critical rotation angle, performs the process (II), and if it is determined that the rotation angle is equal to or less than the critical rotation angle, performs the process (II). (iii) if it is determined that the ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio, it performs the process (II); if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio, it performs the process (II); and if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio, it performs the process (II); and if it is determined that the ratio of matching feature points is equal to or less than the critical matching feature point ratio, it terminates the contamination detection and determination process.

[0034] In one example, in the process (II), the processor sets n contamination states, which are arbitrary contamination states including at least a transparent state, a semi-transparent state, and an opaque state, and causes the contamination degree calculation unit to obtain a contamination probability map indicating the contamination state for each detailed region using the output value obtained for each detailed region through the convolutional neural network operation, and applies a weight vector individually preset for each of the contamination states to the contamination probability map to derive the frame contamination degree for each detailed region.

[0035] In one example, the processor is configured to increase the weight of the weight vector as the degree of contamination approaches the opaque state among the n contamination states, which are divided into a range from the transparent state to the opaque state, and to assign a negative weight to a contamination state that is close to the transparent state within a predetermined critical value among the n contamination states, and to apply the weight vector to the contamination probability map using the contamination degree calculation unit to derive the frame contamination degree for each detailed region.

[0036] In one example, in the (II) process, the processor acquires the frame sharpness for each detailed region by reflecting the edge map and an attenuation constant (the attenuation constant is a constant for adjusting the amount of change in the frame sharpness due to the previous frame sharpness) for each detailed region to the previous frame sharpness derived for each detailed region for the previous input image using the sharpness calculation unit.

[0037] In one example, in the (III) process, the processor uses the contamination determination unit to combine the frame contamination level and the frame clarity obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculate a corrected contamination level for each detailed region by referring to the previous contamination level and the current contamination level derived for each detailed region for the previous input image, and then determine whether the corrected contamination level is greater than or equal to a preset value.If there are a certain number or more first specific detailed regions that are greater than or equal to the preset value, the processor determines that the camera is contaminated.

[0038] In one example, the processor sets a region importance for each detailed region of the camera, calculated by referring to at least a portion of the position importance for each detailed region, the class importance of the object, and the movement direction of the object, and uses the contamination determination unit to determine that the camera is contaminated if the sum of the region importances of second specific detailed regions, whose corrected contamination level is equal to or greater than a preset value, is equal to or greater than a preset critical contamination region importance.

[0039] In one example, the processor uses the clarity calculation unit to output the first frame clarity when a specific frame clarity exceeds a predetermined first frame clarity, to output the second frame clarity (the second frame clarity is a value smaller than the first frame clarity) when the specific frame clarity is less than a predetermined second frame clarity, and to output the specific frame clarity when the specific frame clarity has a value between the second frame clarity and the first frame clarity, to output the specific frame clarity, to use the contamination determination unit to output the first corrected contamination degree when a specific corrected contamination degree exceeds a predetermined first corrected contamination degree, to output the second corrected contamination degree (the second corrected contamination degree is a value smaller than the first corrected contamination degree) when the specific corrected contamination degree is less than a predetermined second corrected contamination degree, and to output the specific corrected contamination degree when the specific corrected contamination degree has a value between the second corrected contamination degree and the first corrected contamination degree. [Effects of the Invention]

[0040] The present invention has the effect of effectively detecting contamination patterns that are difficult to classify using generalized criteria, such as translucent contamination, by using a convolutional neural network that has been trained on various image patterns.

[0041] Furthermore, the present invention has the effect of accurately determining whether a camera is contaminated in real time through the image captured by the camera.

[0042] Furthermore, the present invention has the effect of accurately detecting contaminated areas in real time even in areas with monotonous patterns such as the sky or ground by combining the inference results of a deep neural network with the clarity of the image.

[0043] The following drawings, which are attached to be used for explaining the embodiments of the present invention, are merely a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter referred to as "ordinary engineer") can derive each of the other drawings based on these drawings without performing any inventive work. [Brief explanation of the drawings]

[0044] [Figure 1] 1 is a simplified diagram of a contamination detection system for determining whether a camera is contaminated in real time according to an embodiment of the present invention. [Figure 2] 1 is a simplified flowchart illustrating an example process for determining whether a camera is contaminated in real time according to an embodiment of the present invention. [Figure 3] 10 is a flowchart illustrating a process of determining a contamination detection time by a contamination detection time determination unit according to an embodiment of the present invention; [Figure 4] 10 is a diagram illustrating an example of obtaining a feature point disparity vector from an input image according to an embodiment of the present invention. [Figure 5] 10 illustrates an example of a process for acquiring frame definition from an input image according to an embodiment of the present invention. [Figure 6] 10 illustrates an example process of deriving a frame clarity and a frame corruption level from an input image and acquiring a corruption level according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] The following detailed description of the present invention refers to the accompanying drawings which show, by way of example, specific embodiments in which the present invention may be practiced, in order to clarify the objects, technical solutions, and advantages of the present invention. These embodiments are described in sufficient detail to enable those of ordinary skill in the art to practice the present invention.

[0046] Furthermore, throughout the detailed description of the present invention and the claims, the word "comprises" and variations thereof are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and characteristics of the present invention will become apparent to those of ordinary skill in the art, in part from the description and in part from the practice of the present invention. The following examples and drawings are provided as illustrations and are not intended to limit the present invention.

[0047] Furthermore, the present invention covers all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention, although different from one another, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein in connection with one embodiment may be embodied in other embodiments without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the present invention. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention is limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly interpreted. Like reference numerals in the drawings refer to the same or similar functionality throughout the various aspects.

[0048] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention.

[0049] FIG. 1 is a simplified diagram of a contamination detection system for determining in real time whether a camera is contaminated or not, according to one embodiment of the present invention.

[0050] 1, the contamination detection system 100 may include a memory 110 that stores instructions for detecting contamination of a camera, and a processor 120 that executes the instructions stored in the memory 110. In this case, the contamination detection system 100 may include various computing devices such as a server, a personal computer (PC), a tablet, a mobile computer, a PDA / EDA, a mobile phone, a smartphone, an IoT device, etc.

[0051] Specifically, contamination detection system 100 may typically utilize a combination of computing devices (e.g., devices that may include computer processors, memory, storage, input and output devices, and other components of conventional computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular manner) to achieve desired system performance.

[0052] The processor of a computing device may include hardware components such as a microprocessing unit (MPU) or central processing unit (CPU), cache memory, and data bus. The computing device may also include software components such as an operating system and applications that perform specific purposes.

[0053] However, this does not exclude the case where the computing device includes an integrated processor in which the medium, processor, and memory for implementing the present invention are integrated.

[0054] Meanwhile, the contamination detection system 100 may be comprised of multiple devices that perform various functions.

[0055] FIG. 2 is a simplified flowchart illustrating an exemplary process for determining whether a camera is contaminated in real time according to an embodiment of the present invention.

[0056] 2, when an input image is generated by a camera, contamination detection system 100 acquires the input image from the camera (S200), transmits the input image to a contamination detection time determination unit, and causes the contamination detection time determination unit to evaluate the reliability of the input image and determine whether to proceed with the contamination detection and determination process (S210). At this time, the reliability of the input image, which indicates whether the input image is in a state suitable for contamination detection, can be obtained by determining whether at least one of preset conditions described below is satisfied. Here, the contamination detection time determination unit may be included in contamination detection system 100, but is not limited thereto.

[0057] If the contamination detection time determination unit determines that the reliability of the input image is sufficient, the contamination detection system 100 can provide the input image to the contamination degree calculation unit and the clarity calculation unit. Here, the contamination degree calculation unit and the clarity calculation unit may also be included in the contamination detection system 100, but are not limited thereto.

[0058] The contamination detection system 100 can classify the contamination state for each detailed region by applying a convolutional neural network operation to the input image using a contamination degree calculation unit, and derive a frame contamination degree for each detailed region based on the classification (S221). Separately, the contamination detection system 100 can obtain an edge map for each detailed region using a brightness difference between adjacent pixels in the input image using a sharpness calculation unit, and derive a frame sharpness for each detailed region based on the edge map (S222).

[0059] Thereafter, the contamination detection system 100 may determine whether or not the camera is contaminated by referring to the frame contamination level and the frame clarity using a contamination determination unit (S230). Here, the contamination determination unit may also be included in the contamination detection system 100, but is not limited thereto.

[0060] The operation of the contamination detection time determination unit will be described below.

[0061] First, the contamination detection system 100 acquires an input image from a camera and further acquires behavior information, including a camera movement distance and a camera rotation angle, from an external sensor and provides the behavior information to a contamination detection time determination unit. Thereafter, the contamination detection time determination unit determines whether a preset condition for reliability for determining whether the input image is in a state suitable for contamination detection is satisfied by determining at least (i) whether the movement distance acquired from the behavior information exceeds a preset critical movement distance, (ii) whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) whether the ratio of matching feature points obtained by matching feature points in the input image acquired from the camera is less than a preset critical ratio of matching feature points, and (iv) whether the median value of the feature point disparity in the input image exceeds a preset critical value of the feature point disparity. If the input image and the behavior information corresponding to the input image satisfy at least one preset condition based on the preset conditions, including the above conditions, the input image can be determined to have a reliability suitable for contamination detection.

[0062] If the camera moves or rotates a lot, it is considered to have sufficient information for contamination detection because it has information about other physical positions, and even if the camera itself does not move or rotate enough, it is considered to have sufficient information for contamination detection if new information is added to the image due to, for example, a sudden change in the surrounding environment or the movement of surrounding objects. Therefore, input images that meet these conditions are considered to have a reliability suitable for contamination detection (e.g., reliability above a preset threshold value).

[0063] Hereinafter, the process of evaluating the reliability of an input image by the contamination detection time determination unit will be described with reference to FIG.

[0064] FIG. 3 is a flowchart illustrating an example of a process for determining a contamination detection time by a contamination detection time determination unit according to an embodiment of the present invention.

[0065] Referring to FIG. 3, the pollution detection system 100 may first acquire (300) an input image from a camera and acquire (310) behavior information from an external sensor, and provide the acquired information to a pollution detection time determination unit. The pollution detection time determination unit may first acquire (the camera's) movement distance by referring to the behavior information and compare (320) the movement distance with a critical movement distance. The camera's movement distance may depend on, but is not limited to, the movement distance of the vehicle on which the camera is mounted. If it is determined that the movement distance exceeds the critical movement distance, it performs (370) a pollution detection and determination process. If it is determined that the movement distance is equal to or less than the critical movement distance, it may determine that the pollution detection does not have sufficient reliability and acquire (the camera's) rotation angle by referring to the behavior information. The camera's rotation angle may depend on, but is not limited to, the rotation angle of the vehicle on which the camera is mounted. Then, the rotation angle is compared with the critical rotation angle (330). If it is determined that the rotation angle exceeds the critical rotation angle, a contamination detection and determination process is performed (370). If it is determined that the rotation angle is equal to or less than the critical rotation angle, the previous feature point and the current feature point are acquired from the input image, and the previous feature point and the current feature point are matched (340). Then, the ratio of matching feature points obtained by matching the previous feature point and the current feature point is compared with the critical matching ratio (350). If it is determined that the ratio of matching feature points is less than the critical matching ratio, a contamination detection and determination process is performed (370). If it is determined that the ratio of matching feature points is equal to or greater than the critical matching ratio, a feature point disparity is obtained based on the matched previous feature point and the current feature point. Then, the median value of the distribution of the magnitude of the minutiae disparity is calculated, and the median value is compared with the critical value of the minutiae disparity (360). If it is determined that the median value exceeds the critical value of the minutiae disparity, the contamination detection and determination process is performed (370). If it is determined that the median value is equal to or less than the critical value of the minutiae disparity, the contamination detection and determination process is terminated (380).

[0066] The acquisition of the feature point disparity vector will be described below with reference to FIG.

[0067] FIG. 4 is a diagram illustrating an example of obtaining a feature point disparity vector from an input image according to an embodiment of the present invention.

[0068] As described above, the contamination detection time determination unit generates a matching feature point ratio, which is the ratio of matched feature points between the previous feature point and the current feature point, and compares the matching feature point ratio with the critical matching feature point ratio. If it is determined that the matching feature point ratio is equal to or greater than the critical matching feature point ratio, it can obtain a feature point disparity vector from the positions and directions between the matched previous feature point and the current feature point.

[0069] Referring to Figure 4, the disparity vectors of feature points in an image at the current time t and the matched feature points in an image at time t-dt are visualized. Because the magnitude of disparity is widely distributed due to the perspective effect, the presence or absence of changes in the image can be effectively determined by taking the median value as a representative value. That is, as shown in Figure 4, the median value of the distribution for the magnitudes of multiple feature point disparity vectors can be calculated, and it can be determined whether the median value exceeds the critical value of the feature point disparity.

[0070] Returning to FIG. 2 , the process by which the sharpness calculation unit derives frame sharpness (S222) after determining that the reliability of the input image satisfies the preset conditions will be described in detail below. The frame sharpness can be calculated using the characteristic that an uncontaminated image has a large brightness difference between adjacent pixels. The sharpness calculation unit can generate an edge map by eliminating vague change components from the input image and detecting only edges with strong components. Then, the frame sharpness can be calculated for each detailed region based on the edge map. In this case, the sharpness for each detailed region can be derived from the average value, representative value, or median value of the edge components of each detailed region. In the present invention, the "detailed region" may be, but is not limited to, each region generated when an image is divided into a predetermined grid.

[0071] The process of deriving the frame sharpness will be described below with reference to FIG.

[0072] FIG. 5 illustrates an exemplary process of acquiring frame definition from an input image according to an embodiment of the present invention.

[0073] 5, edges can be detected 520 from an input image 510, and an edge map 530 can be obtained based on the detected edges 520. At this time, referring to the detected edges 520, it can be seen that when extracting edges from contaminated portions visible in the input image 510, edges are hardly detected because, unlike non-contaminated portions, the brightness difference between adjacent pixels is not large. The sharpness calculation unit generates an edge map 530 based on the edges 520 extracted from the input image 510, and can calculate the sharpness for each detailed region using the edge map 530.

[0074] 2, for convenience, the process of deriving frame sharpness (S222) will be described first. The contamination detection system 100 can reflect the sharpness of the previous frame when calculating the frame sharpness from the edge map using a sharpness calculation unit. Furthermore, an attenuation constant can be further included to adjust the amount of change in frame sharpness depending on the sharpness of the previous frame. This will be described in more detail later.

[0075] In addition, the contamination detection system 100 includes a definition calculation unit that outputs the first frame definition when the specific frame definition exceeds a predetermined first frame definition, outputs the second frame definition when the specific frame definition is less than a predetermined second frame definition, and outputs the specific frame definition as is when the specific frame definition has a value between the second frame definition and the first frame definition, where the second frame definition is a value smaller than the first frame definition.

[0076] Therefore, the calculation formula for the frame sharpness for each detailed area performed by the sharpness calculation unit is as follows.

number

number

number

number

number

number

number

number

number

number

number

number

number

[0077] 2, looking in detail at the process of deriving the frame contamination level (S221), n ​​contamination states can be set for the contamination detection system 100, and the n contamination states can include at least a transparent state, a semi-transparent state, and an opaque state. The contamination detection system 100 can then use the output values ​​obtained for each detailed region through a convolutional neural network operation using a contamination level calculation unit to obtain a contamination probability map indicating the contamination state for each detailed region.

[0078] At this time, an example of the contamination probability map can be seen in FIG.

[0079] FIG. 6 illustrates an exemplary process of deriving frame clarity and frame corruption from an input image and obtaining a corruption level according to an embodiment of the present invention.

[0080] 6, a contamination probability map 610 can be seen, which visualizes the contamination state with the highest probability inferred from a contamination probability map obtained by applying a convolutional neural network operation based on four contamination states to derive the contamination probability for each specific detailed region. In the visualized contamination probability map 610, it can be seen that the opaque state, which is the contamination state with the highest probability based on the four contamination states, is displayed in orange, a semi-transparent state close to the opaque state is displayed in yellow, a semi-transparent state close to the transparent state is displayed in gray, and a transparent state is displayed in white.

[0081] Referring to Figure 2, the process of calculating the frame contamination level will be explained. Once the contamination probability map has been derived, a weight vector individually set for each contamination state can be applied to the contamination probability map to derive the frame contamination level for each detailed region.

[0082] In this case, the weight vector may be weighted more as the contamination level approaches the opaque state among n contamination states, which are a range of contamination levels from transparent to opaque. Also, a contamination state that is closer to the transparent state within a preset threshold value among the n contamination states may be set to have a negative weight. Based on the preset weight vector, the contamination detection system 100 may use a contamination level calculation unit to apply the weight vector to each corresponding contamination state in the contamination probability map, thereby deriving a frame contamination level for each detailed region.

[0083] Once the frame contamination level and frame clarity are derived from the contamination level calculation unit and the clarity calculation unit, the contamination detection system 100 provides the frame contamination level and frame clarity to the contamination determination unit, and the contamination determination unit can combine the frame contamination level obtained for each detailed region and the frame clarity to calculate the current contamination level calculated for each detailed region.

[0084] At this time, the contamination detection system 100 can calculate a corrected contamination level by reflecting the immediately preceding contamination level derived for each detailed region of the immediately preceding input image using the contamination determination unit on the current contamination level.

[0085] The pollution detection system 100 may also have a pollution determination unit that outputs a first corrected pollution level when the specific corrected pollution level exceeds a predetermined first corrected pollution level, outputs a second corrected pollution level when the specific corrected pollution level is less than a predetermined second corrected pollution level, and outputs a specific corrected pollution level when the specific corrected pollution level has a value between the second corrected pollution level and the first corrected pollution level. In this case, the second corrected pollution level may have a value smaller than the first corrected pollution level.

[0086] Therefore, the formula for calculating the corrected contamination level for each detailed area performed by the contamination determination unit is as follows:

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0087] The contamination detection system 100 determines whether the corrected contamination level calculated by the contamination determination unit is equal to or greater than a preset value, and if there are a certain number or more first specific detailed areas, which are detailed areas having a value equal to or greater than the preset value, the contamination detection system 100 can determine that the camera is contaminated.

[0088] In this case, for each part of the area captured by the camera, a region importance calculated by referring to at least some of the position importance of each region, the object class importance, and the object movement direction can be set for each detailed region. For example, the position importance of each detailed region increases toward the center, the object class importance can be set to reflect people, vehicles, guardrails, etc., and the object movement direction can be set to reflect the movement direction of the people, vehicles, etc. In addition, the region importance can be set by taking driving habits, traffic regulations, or driving environment into more consideration. When determining whether the camera is contaminated using the contamination determination unit, the contamination detection system 100 can determine that the camera is contaminated if the sum of the region importances of second specific detailed regions, which are detailed regions having an adjusted contamination level equal to or greater than a predetermined value, is equal to or greater than a predetermined critical contamination region importance. Here, the second specific detailed region may be the same as the first specific detailed region, but is not limited thereto.

[0089] 6, the contamination detection system 100 uses a contamination degree calculation unit to derive a frame contamination degree for each detailed region by referring to a contamination probability map 610, a sharpness calculation unit to derive a frame sharpness for each detailed region by referring to an edge map 620, and a contamination determination unit to derive a corrected contamination degree by referring to the frame contamination degree and the frame sharpness. Referring to FIG. 6, a corrected contamination degree 630 visualized for each detailed region can be confirmed.

[0090] Although the present invention has been described above using specific details such as specific components and limited examples and drawings, this is provided merely to aid in a more general understanding of the present invention, and the present invention is not limited to the above examples. Those skilled in the art will be able to make various modifications and variations from such descriptions.

[0091] Therefore, the concept of the present invention should not be limited to the above-described embodiments, and all modifications equivalent to or equivalent to the scope of the claims, as well as the scope of the claims described below, are within the scope of the concept of the present invention.

Claims

1. In a method to determine whether a camera is contaminated or not in real time, (a) when an input image is generated by a camera, the contamination detection system causes a contamination detection time determination unit to determine whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition; (b) if it is determined that the input image satisfies the preset condition, the contamination detection system (i) provides the input image to a contamination degree calculation unit, and then uses the contamination degree calculation unit to classify contamination states into predetermined detailed regions, which are regions generated when the input image is divided into a predetermined grid shape, by applying a convolutional neural network operation to the input image, and derives a frame contamination degree for each detailed region based on the classification; and (ii) provides the input image to a sharpness calculation unit, and then uses the sharpness calculation unit to obtain an edge map for each detailed region using a brightness difference between adjacent pixels in the input image, and derives a frame sharpness for each detailed region based on the edge map; and (c) causing the contamination detection system to determine whether the camera is contaminated by a contamination determination unit with reference to the frame contamination level and the frame clarity; Including, In the step (a), the contamination detection system acquires behavior information including a movement distance and a rotation angle of the camera from an external sensor and provides the behavior information to the contamination detection time determination unit; and, with respect to the preset conditions for determining whether the input image is suitable for contamination detection, causes the contamination detection time determination unit to determine at least (i) whether the movement distance acquired from the behavior information exceeds a preset critical movement distance, (ii) whether the rotation angle acquired from the behavior information exceeds a preset critical rotation angle, (iii) match feature points of the input image acquired from the camera and determine whether a ratio of the matching feature points is less than a preset critical ratio of matching feature points, and (iv) calculate an intermediate value of feature point disparities of the input image and determine whether the intermediate value of feature point disparities exceeds a preset critical value of feature point disparities, and based on the calculated results, determine whether the reliability satisfies at least one preset condition.

2. 2. The method of claim 1, wherein the contamination detection system, using the contamination detection time determination unit, matches previous feature points extracted from the previous input image previously acquired by the camera with current feature points extracted from the current input image currently acquired, and if it is determined that a ratio of matched matching feature points is equal to or greater than the critical ratio of matching feature points, obtains feature point disparity vectors from positions and directions between the matched previous feature points and the current feature points, calculates a median value of a distribution of magnitudes of the feature point disparity vectors, and determines whether the median value of the distribution of magnitudes of the feature point disparity vectors exceeds a critical value of the feature point disparity.

3. The contamination detection system, using the contamination detection time determination unit, (i) compares the movement distance with the critical movement distance with reference to the behavior information acquired from the external sensor, and if it is determined that the movement distance exceeds the critical movement distance, performs step (b), and if it is determined that the movement distance is equal to or less than the critical movement distance, acquires the rotation angle from the external sensor; (ii) compares the rotation angle acquired from the external sensor with the critical rotation angle with reference to the behavior information acquired from the external sensor, and if it is determined that the rotation angle exceeds the critical rotation angle, performs step (b), and if it is determined that the rotation angle is equal to or less than the critical rotation angle, performs step (b).

3. The method of claim 2, further comprising: (iii) acquiring a previous feature point and the current feature point; (iii) performing step (b) if it is determined that a ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio; acquiring the feature point disparity if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature ratio; and (iv) performing step (b) if it is determined that the intermediate value of the feature point disparity magnitude distribution exceeds the critical value of the feature point disparity as a result of calculating the intermediate value; and terminating the contamination detection and determination process if it is determined that the intermediate value is equal to or less than the critical value of the feature point disparity.

4. In the step (b), 2. The method of claim 1, wherein the contamination detection system, in a state where n contamination states are set, which are arbitrary contamination states including at least a transparent state, a semi-transparent state, and an opaque state, acquires a contamination probability map indicating the contamination state for each detailed region using an output value acquired for each detailed region through the convolutional neural network operation using the contamination degree calculation unit, and derives the frame contamination degree for each detailed region by applying a weight vector individually preset for each of the contamination states to the contamination probability map.

5. 5. The method of claim 4, wherein the contamination detection system applies the weight vector to the contamination probability map using the contamination degree calculation unit to derive the frame contamination degree for each detailed region, in a state where the weight vector is preset to increase as the contamination degree approaches the opaque state among the n contamination states, which are in a range from the transparent state to the opaque state, and the contamination state among the n contamination states is preset to have a negative weight.

6. In the step (b), 2. The method of claim 1, wherein the contamination detection system acquires the frame sharpness for each detailed region by applying the edge map and an attenuation constant (the attenuation constant is a constant for adjusting the amount of change in the frame sharpness depending on the previous frame sharpness) for each detailed region to the previous frame sharpness derived for each detailed region for the previous input image by the sharpness calculation unit.

7. In the step (c), 2. The method of claim 1, wherein the contamination detection system uses the contamination determination unit to combine the frame contamination level and the frame clarity level obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculates an adjusted contamination level for each detailed region by referring to the previous contamination level and the current contamination level derived for each detailed region for the previous input image, and then determines whether the adjusted contamination level is greater than or equal to a predetermined value, and determines that the camera is contaminated if a certain number of first specific detailed regions are greater than or equal to the predetermined value.

8. 8. The method of claim 7, wherein, in a state where a region importance calculated by referring to at least a portion of the position importance of each detailed region, the class importance of the object, and the movement direction of the object is set for each detailed region of the camera, the contamination detection system determines, using the contamination determination unit, that the camera is contaminated if the sum of the region importances of second specific detailed regions whose corrected contamination level is equal to or greater than a predetermined value is equal to or greater than a predetermined critical contamination region importance.

9. The contamination detection system uses the definition calculation unit to output the first frame definition when a specific frame definition exceeds a predetermined first frame definition, to output the second frame definition when the specific frame definition is less than a predetermined second frame definition (the second frame definition is smaller than the first frame definition), and to output the specific frame definition when the specific frame definition has a value between the second frame definition and the first frame definition, 8. The method according to claim 7, wherein the contamination determination unit is configured to output the first corrected pollution level when a specific corrected pollution level exceeds a predetermined first corrected pollution level, to output the second corrected pollution level (the second corrected pollution level is a value smaller than the first corrected pollution level) when the specific corrected pollution level is less than a predetermined second corrected pollution level, and to output the specific corrected pollution level when the specific corrected pollution level has a value between the second corrected pollution level and the first corrected pollution level.

10. A computing device that determines in real time whether a camera is contaminated or not. at least one memory for storing instructions; and at least one processor configured to execute said instructions; The processor includes: (I) a process for determining, when an input image is generated by a camera, whether a reliability indicating whether the input image is suitable for contamination detection satisfies at least one preset condition using a contamination detection time determination unit; (II) if it is determined that the input image satisfies the preset condition, (i) providing the input image to a contamination level calculation unit, and then using the contamination level calculation unit to perform a convolutional neural network (CNNN) on the input image. (ii) a process of applying an operation of a network (Network) to classify the contamination state into predetermined detailed regions, which are regions generated when the input image is divided into a predetermined grid, and deriving a frame contamination degree for each detailed region based on the classification; (ii) a process of providing the input image to a sharpness calculation unit, and then using the sharpness calculation unit to obtain an edge map for each detailed region using a brightness difference between adjacent pixels in the input image, and deriving a frame clarity for each detailed region based on the edge map; and (iii) a process of using a contamination determination unit to determine whether the camera is contaminated by referring to the frame contamination degree and the frame clarity; In the process (I), and (iv) calculating a mean value of feature point disparities of the input image and determining whether the mean value of feature point disparities exceeds a predetermined critical value of feature point disparities, and determining whether the reliability satisfies at least one of the predetermined conditions.

11. 11. The computing device of claim 10, wherein the processor, using the contamination detection time determination unit, matches previous feature points extracted from a previous input image previously acquired by the camera with current feature points extracted from a currently acquired current input image, and if it is determined that a ratio of matched matching feature points is equal to or greater than the critical ratio of matching feature points, obtains a feature point disparity vector from positions and directions between the matched previous feature points and the current feature points, calculates a median value of a distribution of magnitudes of the feature point disparity vector, and determines whether the median value of the distribution of magnitudes of the feature point disparity vector exceeds a critical value of the feature point disparity.

12. The processor, using the contamination detection time determination unit, (i) refers to the behavior information acquired from the external sensor and compares the movement distance with the critical movement distance, and if it is determined that the movement distance exceeds the critical movement distance, performs the process (II). If it is determined that the movement distance is equal to or less than the critical movement distance, it obtains the rotation angle from the external sensor. (ii) refers to the behavior information acquired from the external sensor and compares the rotation angle acquired from the external sensor with the critical rotation angle, and if it is determined that the rotation angle exceeds the critical rotation angle, it performs the process (II). If it is determined that the rotation angle is equal to or less than the critical rotation angle, it performs the process (II). (iii) if it is determined that a ratio of matching feature points between the previous feature point and the current feature point is less than the critical matching ratio, perform the process (II); if it is determined that the ratio of matching feature points is equal to or greater than the critical matching feature point ratio, obtain the feature point disparity; (iv) if it is determined that the intermediate value of the feature point disparity magnitude distribution exceeds the critical value of the feature point disparity, perform the process (II); and if it is determined that the intermediate value is equal to or less than the critical value of the feature point disparity, terminate the contamination detection and determination process.

13. In the process (II), 11. The computing device of claim 10, wherein the processor sets n contamination states, which are arbitrary contamination states including at least a transparent state, a semi-transparent state, and an opaque state, and causes the contamination degree calculation unit to acquire a contamination probability map indicating the contamination state for each detailed region using output values ​​acquired for each detailed region through the convolutional neural network operation, and applies a weight vector individually preset for each of the contamination states to the contamination probability map to derive the frame contamination degree for each detailed region.

14. 14. The computing device of claim 13, wherein the processor is configured to increase the weight of the weight vector as the degree of contamination approaches the opaque state among the n contamination states that divide the range from the transparent state to the opaque state, and to assign a negative weight to a contamination state that is closer to the transparent state within a predetermined critical value among the n contamination states, and to cause the contamination degree calculation unit to apply the weight vector to the contamination probability map to derive the frame contamination degree for each detailed region.

15. In the process (II), 11. The computing device of claim 10, wherein the processor acquires the frame sharpness for each detailed region by applying the edge map and an attenuation constant (the attenuation constant is a constant for adjusting the amount of change in the frame sharpness depending on the previous frame sharpness) for each detailed region to the previous frame sharpness derived for each detailed region for the previous input image by the sharpness calculation unit.

16. In the process (III), 11. The computing device of claim 10, wherein the processor uses the contamination determination unit to combine the frame contamination level and the frame clarity level obtained for each detailed region to obtain a current contamination level calculated for each detailed region, calculate an adjusted contamination level for each detailed region by referring to the previous contamination level derived for each detailed region for the previous input image and the current contamination level, and then determine whether the adjusted contamination level is greater than or equal to a predetermined value. If a certain number of first specific detailed regions are greater than or equal to the predetermined value, the processor determines that the camera is contaminated.

17. 17. The computing device of claim 16, wherein the processor determines that the camera is contaminated when, with the contamination determination unit, a sum of the region importances of second specific detailed regions whose corrected contamination level is equal to or greater than a predetermined value is equal to or greater than a predetermined critical contamination region importance, while setting a region importance for each detailed region with respect to the camera, the region importance being calculated by referring to at least some of the position importance for each detailed region, the class importance of the object, and the moving direction of the object.

18. the processor, using the sharpness calculation unit, outputs the first frame sharpness when a specific frame sharpness exceeds a predetermined first frame sharpness, outputs the second frame sharpness (the second frame sharpness is a value smaller than the first frame sharpness) when the specific frame sharpness is less than a predetermined second frame sharpness, and outputs the specific frame sharpness when the specific frame sharpness has a value between the second frame sharpness and the first frame sharpness; 17. The computing device of claim 16, wherein the contamination determination unit is configured to output the first corrected contamination level when a specific corrected contamination level exceeds a predetermined first corrected contamination level, to output the second corrected contamination level (the second corrected contamination level is a value smaller than the first corrected contamination level) when the specific corrected contamination level is less than a predetermined second corrected contamination level, and to output the specific corrected contamination level when the specific corrected contamination level has a value between the second corrected contamination level and the first corrected contamination level.

Images