Pollution treatment method and system for vehicle-mounted camera lens
Through the deep learning model, the on-board camera lens is detected and processed, which solves the problem of image analysis function failure caused by on-board camera lens pollution, realizes efficient pollution detection and processing, and improves the function and reliability of on-board camera lenses.
Patent Information
- Application Number
- CN202411140378.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2019-01-30
- Publication Date
- 2024-11-29
AI Technical Summary
Vehicle camera lenses are susceptible to pollutants such as dust, sludge, sewage, etc., resulting in errors or failures in image analysis functions, and it is difficult for the existing technology to effectively detect and deal with lens pollution.
The trained deep learning model is used to detect the image frames collected by the on-board camera lens. The pollution probability is calculated by dividing the image sub-regions, the lens pollution signal is generated and sent to the associated application device for processing, including issuing pollution prompts, cleaning the lens or pausing it. Driver assistance function.
It realizes timely detection and treatment of on-board camera lens pollution, improves the accuracy and processing efficiency of pollution detection, reduces the impact caused by lens pollution, and enriches the functions of on-board camera lenses.
Smart Images

Figure CN119048462A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method and system for processing contamination of a vehicle-mounted camera lens. Background Art
[0002] With the development of computer vision technology, more and more scenes require capturing images of the scene through cameras and obtaining relevant information of the scene by analyzing the images.
[0003] For example, in-vehicle cameras are increasingly widely used, such as surround view systems, driving recorders, etc. In-vehicle camera lenses also play an extremely important role in ADAS (Advanced Driver Assistance System), and the operation of many intelligent applications depends on the image data captured by the in-vehicle camera lens. When the in-vehicle camera lens is directly exposed to the outside, it is easily affected by pollutants such as dust, silt, sewage, and leaves; when the in-vehicle camera lens is inside the car windshield, it is easily affected by the cleanliness of the windshield. These will cause errors or failures in image-based intelligent assistance functions. Summary of the invention
[0004] In view of this, the present application provides a method and system for treating contamination of a vehicle-mounted camera lens.
[0005] Specifically, the present application is implemented through the following technical solutions:
[0006] In a first aspect, the present application provides a method for treating contamination of a vehicle-mounted camera lens, the method comprising:
[0007] When the speed of the target vehicle exceeds a preset pollution detection start threshold, the trained deep learning model is used to perform pollution detection on the image frames captured by the vehicle-mounted camera lens to determine the pollution type when the vehicle-mounted camera lens is contaminated;
[0008] A lens contamination signal is generated according to the contamination type of the vehicle-mounted camera lens, and the lens contamination signal is sent to an application device associated with the vehicle-mounted camera lens, so that the application device performs contamination processing according to the lens contamination signal.
[0009] Preferably, the pollution treatment includes at least one or a combination of the following:
[0010] Issue pollution warning;
[0011] Cleaning the vehicle-mounted camera lens;
[0012] Driving assistance functions based on the vehicle's camera footage are suspended.
[0013] Preferably, the use of a trained deep learning model to perform pollution detection on the image frames collected by the vehicle-mounted camera lens to determine the pollution type when the vehicle-mounted camera lens is contaminated includes:
[0014] Inputting the image frame captured by the vehicle-mounted camera lens into a trained deep learning model, so that the deep learning model divides the image frame into a plurality of image sub-regions and calculates the contamination probability of each image sub-region;
[0015] According to the pollution probability of each image sub-region, it is determined that the vehicle-mounted camera lens is polluted when there is pollution.
[0016] Preferably, the determining, based on the pollution probability of each image sub-region, that the vehicle-mounted camera lens is contaminated as a pollution type when there is contamination, includes:
[0017] When the number of image frames input to the deep learning model reaches a preset number, the contamination probabilities of image sub-regions with the same regional positions in the preset number of image frames are averaged to obtain the regional contamination probability of each regional position;
[0018] If there is an area location where the area pollution probability is greater than or equal to the preset pollution threshold, it is determined that the vehicle-mounted camera lens is of the pollution type.
[0019] Preferably, the inter-frame interval of the image frames input to the deep learning model is determined according to the speed of the target vehicle.
[0020] In a second aspect, the present application provides a pollution treatment system for a vehicle-mounted camera lens, the system comprising:
[0021] A vehicle speed acquisition module, used to acquire the vehicle speed of the target vehicle detected by the vehicle speed sensor, and send the vehicle speed to the lens contamination detection module;
[0022] On-board camera lens, used to collect image frames;
[0023] A lens contamination detection module is used to obtain image frames captured by the vehicle-mounted camera lens when it is determined that the speed of the target vehicle exceeds a preset contamination detection start threshold, and use a trained deep learning model to perform contamination detection on the obtained image frames to determine the type of contamination when the vehicle-mounted camera lens is contaminated;
[0024] The signal generating module is used to generate a lens contamination signal according to the contamination type of the vehicle-mounted camera lens, and send the lens contamination signal to an application device associated with the vehicle-mounted camera lens, so that the application device performs contamination processing according to the lens contamination signal.
[0025] Preferably, the application device includes at least one or a combination of the following:
[0026] A prompting device, used for issuing a pollution prompt;
[0027] A cleaning device, used for cleaning the vehicle-mounted camera lens;
[0028] An advanced driver assistance ADAS device is used to suspend a driver assistance function based on the vehicle-mounted camera lens.
[0029] Preferably, the lens contamination detection module comprises:
[0030] A contamination probability determination submodule, used for inputting the image frame captured by the vehicle-mounted camera lens into a trained deep learning model, so that the deep learning model divides the image frame into a plurality of image sub-regions and calculates the contamination probability of each image sub-region;
[0031] The pollution type determination submodule is used to determine the pollution type when the vehicle-mounted camera lens is polluted according to the pollution probability of each image sub-area.
[0032] Preferably, the pollution type determination submodule is specifically used for:
[0033] When the number of image frames input to the deep learning model reaches a preset number, the contamination probabilities of image sub-regions with the same regional positions in the preset number of image frames are averaged to obtain the regional contamination probability of each regional position;
[0034] If there is an area location where the area pollution probability is greater than or equal to the preset pollution threshold, it is determined that the vehicle-mounted camera lens is of the pollution type.
[0035] Preferably, the inter-frame interval of the image frames input to the deep learning model is determined according to the speed of the target vehicle.
[0036] The technical solution provided by the embodiments of the present application may have the following beneficial effects:
[0037] In an embodiment of the present application, the vehicle-mounted camera lens can obtain the speed of the target vehicle. If it is detected that the speed of the target vehicle exceeds the preset pollution detection start threshold, the trained deep learning model is used to perform pollution detection on the image frames collected by the vehicle-mounted camera lens to determine the type of pollution when the vehicle-mounted camera lens is contaminated. By implementing pollution detection of the vehicle-mounted camera lens through the deep learning model, the pollution condition of the lens can be detected in time, the accuracy of pollution detection is improved, and the functions of the vehicle-mounted camera lens are enriched.
[0038] After the vehicle-mounted camera lens is identified as a contaminated type, a lens contamination signal is generated according to the contamination type of the vehicle-mounted camera lens, and the lens contamination signal is sent to an application device associated with the vehicle-mounted camera lens, so that the application device automatically performs contamination processing according to the lens contamination signal, thereby improving the efficiency of contamination processing and reducing the impact caused by lens contamination. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a flowchart of a method for treating contamination of a vehicle-mounted camera lens according to an exemplary embodiment of the present application;
[0040] Figure 2 is a flowchart of lens contamination detection steps shown in an exemplary embodiment of the present application;
[0041] Figure 3 is a schematic diagram showing an exemplary embodiment of the present application of dividing an image frame into multiple image sub-regions;
[0042] Figure 4 is a schematic diagram of fusing the pollution probabilities of each image sub-region of t image frames shown in an exemplary embodiment of the present application;
[0043] Figure 5 is a flowchart of deep learning model training steps shown in an exemplary embodiment of the present application;
[0044] Figure 6 is a schematic diagram of a pollution-free image sample acquired by a vehicle-mounted surround view camera, shown in an exemplary embodiment of the present application;
[0045] Figure 7 is a schematic diagram of a polluted image sample acquired by a vehicle-mounted surround view camera, shown in an exemplary embodiment of the present application;
[0046] Figure 8 is a schematic diagram of a truth value graph corresponding to a polluted image sample shown in an exemplary embodiment of the present application;
[0047] Fig. 9 It is a hardware structure diagram of the device in which the device of the present application is located;
[0048] Fig.10 It is a structural block diagram of an embodiment of a pollution treatment device for a vehicle-mounted camera lens shown in an exemplary embodiment of the present application;
[0049] Fig.11 It is a structural block diagram of an embodiment of a pollution treatment system for a vehicle-mounted camera lens shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0051] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0052] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0053] refer to Figure 1 , shows a flowchart of a method for treating contamination of a vehicle-mounted camera lens according to an exemplary embodiment of the present application, which may specifically include the following steps:
[0054] Step 101, when the speed of the target vehicle exceeds the set pollution detection start threshold, the trained deep learning model is used to perform pollution detection on the image frames collected by the vehicle-mounted camera lens to determine the pollution type when the vehicle-mounted camera lens is polluted;
[0055] In this embodiment, the vehicle-mounted camera lens can obtain the speed of the target vehicle in real time. In implementation, the real-time speed of the target vehicle can be obtained through a speed sensor, and the speed is transmitted to the vehicle-mounted camera lens.
[0056] After the vehicle camera lens obtains the speed of the target vehicle, it can further determine whether the real-time speed exceeds the set pollution detection start threshold. If the real-time speed exceeds the pollution detection start threshold, the pollution detection function of the vehicle camera lens is turned on. If the real-time speed does not exceed the pollution detection start threshold, the process ends.
[0057] After the pollution detection function of the vehicle camera lens is turned on, the vehicle camera lens can use the trained deep learning model to perform pollution detection on the image frames collected by the vehicle camera lens to determine the pollution type when the vehicle camera lens is contaminated.
[0058] In practice, in addition to the above pollution types, the deep learning model can also output the non-pollution type when there is no pollution in the vehicle camera lens. In implementation, the non-pollution type can be represented by a first value, such as a value of 0; the pollution type can be represented by a second value, such as a value of 1.
[0059] As an example, the vehicle-mounted camera lens may include but is not limited to a surround-view camera, a driving recorder, etc.
[0060] In a preferred embodiment of the present application, Figure 2 As shown in the lens contamination detection step flowchart, step 101 may include the following sub-steps:
[0061] Sub-step 101-1, inputting the image frame captured by the vehicle-mounted camera lens into a trained deep learning model, so that the deep learning model divides the image frame into multiple image sub-regions and calculates the contamination probability of each image sub-region;
[0062] When it is detected that the speed of the target vehicle exceeds the pollution detection start threshold, the image frame captured by the vehicle camera lens can be input into the trained deep learning model. In the deep learning model, the received image frame can be divided into multiple image sub-areas using a preset image division rule, and the pollution probability of each image sub-area is calculated.
[0063] In one embodiment, the preset image division rule may include the number of image sub-regions to be divided, and the size of each image sub-region may be determined according to the size of the input image frame and the number of image sub-regions to be divided. For example, assuming that the number of image sub-regions to be divided is n*n, and the width of the input image frame is w and the height is h, then the width of each image sub-region may be determined to be (w / n), the step length in the width direction is also (w / n), the height is (h / n), and the step length in the height direction is also (h / n), and finally the image frame may be divided into n*n image sub-regions. Figure 3 As shown, P0-Pn are image sub-regions divided for one image frame.
[0064] Of course, the embodiments of the present application are not limited to the above-mentioned image division rules, and different division rules can be set according to actual needs.
[0065] After dividing the image frame into multiple image sub-regions, the deep learning model can further perform pollution detection on each divided image sub-region and calculate the pollution probability of each image sub-region.
[0066] Sub-step 101 - 2 , determining the pollution type when the vehicle-mounted camera lens is polluted according to the pollution probability of each image sub-region.
[0067] In this embodiment, the contamination type of the vehicle camera lens can be determined based on the contamination probability of each image sub-area divided from the image frame captured by the vehicle camera lens. The contamination detection of the lens is performed based on image processing, which can improve the accuracy of lens contamination detection.
[0068] In a preferred embodiment of the present application, sub-step 101-2 may further include the following sub-steps:
[0069] When the number of image frames input into the deep learning model reaches a preset number, the contamination probabilities of image sub-regions with the same regional positions in the preset number of image frames are averaged to obtain the regional contamination probability of each regional position; if there is a regional position with a regional contamination probability greater than or equal to a preset contamination threshold, it is determined that the vehicle-mounted camera lens is of a contaminated type.
[0070] Specifically, when the number of image frames input to the deep learning model reaches a preset number, for example, when the number of image frames input to the deep learning model is t frames, the prediction results output by the deep learning model for the t frames can be fused to detect the pollution type of the vehicle camera lens. The embodiment of the present application fuses the prediction results of t frames of image frames to reduce the inaccuracy of the prediction results of a single image frame and improve the accuracy of lens pollution detection.
[0071] The above fusion method includes performing a mean operation on the contamination probabilities of the image sub-regions with the same regional position in the above t frames of image frames to obtain the regional contamination probability of each regional position. In practice, each image sub-region has a corresponding regional position in the image frame in which it is located, and the regional position distribution of the image sub-regions of each image frame obtained by dividing according to the same image division rule is the same. For example, Figure 4 As shown, in t image frames, the region position of the image sub-region at the PX position of each image frame is the same, where X=0, ..., n. Figure 4 In the method, the contamination probability of the image sub-region at the P0 position in each image frame is averaged to obtain the regional contamination probability at the P0 region; the contamination probability of the image sub-region at the P1 position in each image frame is averaged to obtain the regional contamination probability at the P1 region, and so on, until the regional contamination probability at the Pn region is calculated.
[0072] In a specific implementation, a pollution threshold value prob_th may be preset, wherein prob_th may be set according to an empirical value, and the embodiment of the present application does not limit it. When the regional pollution probability of a certain area position is greater than or equal to prob_th, it can be determined that the vehicle-mounted camera lens is of a polluted type. When the regional pollution probability of all area positions is less than prob_th, it can be determined that the vehicle-mounted camera lens is of a non-polluted type.
[0073] In one implementation, the inter-frame interval f of the image frames input to the deep learning model may be determined according to the vehicle speed V of the target vehicle. An embodiment of determining the inter-frame interval f may be:
[0074]
[0075] Among them, V1 is the pollution detection start threshold, V2 is the preset vehicle speed threshold corresponding to when f is minimum; fmin is the preset minimum inter-frame interval.
[0076] In a preferred embodiment of the present application, refer to Figure 5 The deep learning model training process can include the following steps:
[0077] Step 201, obtaining an image sample set, wherein the image sample set includes non-polluted image samples and polluted image samples with marked polluted areas;
[0078] In implementation, an image sample in the image sample set is an image captured by a vehicle-mounted camera lens, wherein the vehicle-mounted camera lens may include the vehicle-mounted camera lens subjected to pollution detection in this embodiment, or may be other vehicle-mounted camera lenses, and the embodiment of the present application does not limit this.
[0079] The image sample set of this embodiment includes a large number of non-polluted image samples and polluted image samples. For example, Figure 6 A schematic diagram of a pollution-free image sample obtained by a vehicle-mounted surround view camera is shown. Figure 6 In the figure, area A is the non-imaging area, and area B is the vehicle body area. Figure 7 A schematic diagram of a polluted image sample obtained by a vehicle-mounted surround view camera is shown. Figure 7 In the figure, area C is the polluted area.
[0080] In practice, the image samples in the image sample set may cover a variety of scenes such as daytime, nighttime, rainy days, tunnels, cities, highways, mountain roads, etc. The pollutants in the polluted image samples may cover a variety of common lens pollutants such as dust, silt, sewage, leaves, etc.
[0081] It should be noted that the image sample set obtained in step 201 is a training sample set. In addition to the training sample set, the sample set in this embodiment may also include a test sample set. For example, assuming that there are 100,000 samples in the sample set, of which the number of polluted image samples and the number of unpolluted image samples are half and half, and the division ratio of the training sample set and the test sample set is set to 8:2, then 80,000 samples can be randomly selected from the sample set to form the training sample set, and the remaining 20,000 samples form the test sample set.
[0082] After obtaining the image sample set, the image sample set can be input into the convolutional neural network CNN for training to generate a deep learning model.
[0083] Step 202, dividing each image sample in the image sample set into a plurality of first image sub-regions according to a preset size and step size through the data layer in the convolutional neural network, and for each image sample, identifying each first image sub-region in the image sample as a pollution type or a non-pollution type, and outputting the pollution type or non-pollution type of each first image sub-region of the image sample to the convolutional layer;
[0084] In a specific implementation, the convolutional neural network can receive the image sample set through the Data layer. After the data layer receives the image sample set, it can first divide each image sample in the image sample set into multiple first image sub-regions.
[0085] In actual division, the image samples can be divided according to a preset image division rule, wherein the image division rule may include the number of first image sub-regions to be divided for each image sample, and the image samples can be divided into regions in combination with the size of each image sample. For example, assuming that the width of the image sample is w, the height is h, and the number of first image sub-regions to be divided is n*n, then the width of each first image sub-region in each image sample can be determined to be (w / n), the step length in the width direction is also (w / n), the height is (h / n), and the step length in the height direction is also (h / n), and each image sample is finally divided into n*n first image sub-regions.
[0086] After the data layer divides each image sample into n*n first image sub-regions, it can also identify each first image sub-region as a pollution type or a non-pollution type.
[0087] In a preferred embodiment of the present application, the above step of identifying, for each image sample, each first image sub-region in the image sample as a pollution type or a non-pollution type may further include the following sub-steps:
[0088] Sub-step 202-1, for each polluted image sample, generating a truth map of the polluted image sample according to the marked polluted area in the polluted image sample, wherein the truth map is consistent with the size of the corresponding polluted image sample, the pixel values of the pixels included in the marked polluted area in the truth map are first preset pixel values, and the pixel values of the pixels outside the polluted area are second preset pixel values;
[0089] In a specific implementation, when marking samples, for each contaminated sample, the contaminated area of the contaminated sample can be marked. For example, the marked contaminated area can be Figure 7 According to the marked polluted area, the pollution type of each first image sub-area in each polluted image sample can be determined.
[0090] In this embodiment, the true value map of each polluted image sample can be generated according to the marked polluted area in each polluted image sample, wherein the true value map of each polluted image sample is consistent with the size of the corresponding polluted image sample.
[0091] In the true value map, the pixel values of the pixels included in the marked polluted area can be set to the first preset pixel value, and the pixel values of the pixels outside the polluted area can be set to the second preset pixel value. Figure 7 The polluted image sample is taken as follows: the first preset pixel value of the pixel points in the polluted area is set to 255, and the second preset pixel value of the pixel points outside the polluted area is set to 0, then a picture such as Figure 8 Black and white image shown, white indicates contaminated areas.
[0092] Sub-step 202-2, dividing the truth image into a plurality of second image sub-regions in accordance with a sub-region division method consistent with the corresponding contaminated image sample, wherein each second image sub-region in the truth image has a mapped first image sub-region in the corresponding contaminated image sample;
[0093] After the data layer of the convolutional neural network obtains the true value map of each contaminated image sample, the true value map can be divided into multiple second image sub-regions according to the sub-region division method consistent with the contaminated image sample. Since the image size of the true value map is consistent with that of the corresponding contaminated image sample, the image sub-regions obtained by using the consistent sub-region division method are also one-to-one corresponding, that is, each second image sub-region in the true value map has a mapped first image sub-region in the corresponding contaminated image sample.
[0094] Sub-step 202-3, calculates the proportion of pixel points in each second image sub-region whose pixel value is the first preset pixel value, and determines the first image sub-region mapped by the second image sub-region whose proportion is greater than or equal to the preset proportion threshold as a polluted type; and determines the first image sub-region mapped by the second image sub-region whose proportion is less than the preset proportion threshold as a non-polluted type.
[0095] Specifically, after each truth map is divided into a plurality of second image sub-regions, for each second image sub-region, the proportion of pixels in the second image sub-region whose pixel value is the first preset pixel value can be calculated, that is, the proportion = the number of pixels in the second image sub-region whose pixel value is the first preset pixel value / the total number of pixels in the second image sub-region. When the above proportion is greater than or equal to the preset proportion threshold, the second image sub-region can be determined to be a polluted sub-region, and the first image sub-region mapped by the second image sub-region can be further determined to be a polluted type. Correspondingly, when the above proportion is less than the preset proportion threshold, the second image sub-region can be determined to be a non-polluted sub-region, and the first image sub-region mapped by the second image sub-region can be further determined to be a non-polluted type.
[0096] In another preferred embodiment of the present application, the above step of identifying, for each image sample, each first image sub-region in the image sample as a pollution type or a non-pollution type may further include the following sub-steps:
[0097] For each pollution-free image sample, each first image sub-region of the pollution-free image sample is determined as a pollution-free type.
[0098] Specifically, for each pollution-free image sample, each first image sub-region of the pollution-free image sample may be directly determined as a pollution-free type.
[0099] After the data layer identifies that each first image sub-region in each image sample is a pollution type or a non-pollution type, the pollution type or non-pollution type of each first image sub-region of the image sample can be output to the convolution layer.
[0100] In step 203, the convolution layer learns the features of the first image sub-region of the pollution type based on the pollution type or non-pollution type of each first image sub-region of the received image samples, and generates a deep learning model after multiple iterative training of the learned features in combination with the pooling layer.
[0101] After the convolution layer receives the information that each first image sub-region in each image sample sent by the data layer is of a pollution type or a non-pollution type, it can extract the features of the first image sub-region of the pollution type, automatically learn the extracted features, and generate a deep learning model after multiple iterative training in combination with the pooling layer.
[0102] The generated deep learning model can be tested using a test sample set.
[0103] In an embodiment of the present application, when training a deep learning model, the image sample set used includes a large number of non-polluted image samples and contaminated image samples, so that the obtained deep learning model supports lens contamination detection in various scenarios and is more versatile.
[0104] When training the model, a convolutional neural network is used to divide each image sample in the image sample set to obtain multiple first image sub-regions, and each first image sub-region is identified as a polluted type or a non-polluted type, which is conducive to obtaining a deeper learning model with higher accuracy.
[0105] Step 102 , generating a lens contamination signal according to the contamination type of the vehicle-mounted camera lens, and sending the lens contamination signal to an application device associated with the vehicle-mounted camera lens, so that the application device performs contamination processing according to the lens contamination signal.
[0106] In a specific implementation, after the vehicle-mounted camera lens is detected for contamination and its pollution type is determined, a lens contamination signal can be generated according to the pollution type, and the lens contamination signal is input into an application device associated with the vehicle-mounted camera lens, and the application device performs pollution processing.
[0107] In one embodiment, the application device may include a cleaning device, and the pollution treatment may be cleaning the vehicle-mounted camera lens.
[0108] Specifically, after receiving the lens contamination signal, the cleaning device analyzes the lens contamination signal and cleans the vehicle-mounted camera lens.
[0109] If the signal received by the cleaning device is a non-contamination signal generated by determining that the lens is not contaminated, the cleaning device does not perform cleaning processing.
[0110] It should be noted that, in addition to cleaning the vehicle-mounted camera lens, if there is glass in front of the lens, such as a car windshield, the cleaning device can also clean the glass in front of the lens at the same time.
[0111] In practice, this embodiment can also establish a mapping relationship between each image sub-region divided for the image frame and the lens sub-region, and then locate the corresponding lens sub-region as the contaminated lens sub-region according to the pollution type of the image sub-region, and control the cleaning device to perform targeted cleaning on the contaminated lens sub-region, so as to improve the efficiency and effect of pollution treatment.
[0112] In another embodiment, the application device may include a prompting device, and the contamination processing may include issuing a contamination prompt.
[0113] Specifically, after receiving the lens contamination signal, the prompt device analyzes the lens contamination signal and issues a contamination prompt in voice or visual form to prompt the user of the contamination status of the lens.
[0114] If the signal received by the prompting device is a non-contamination signal generated by determining that the lens is not contaminated, the prompting device does not provide a prompt.
[0115] In another embodiment, the application device may include an advanced driver assistance ADAS device, and the pollution processing may be suspending the driving assistance function based on the vehicle-mounted camera lens.
[0116] Specifically, after receiving the lens contamination signal, the ADAS device analyzes the lens contamination signal and suspends the operation of auxiliary functions related to the lens. For example, the auxiliary functions may include camera-based vehicle detection, tracking and other functions, thereby preventing the auxiliary functions from erring or failing due to lens contamination.
[0117] If the signal received by the ADAS device is a non-contamination signal generated due to a determination that the lens is not contaminated, the ADAS device continues to run auxiliary functions related to the lens.
[0118] It should be noted that the embodiments of the present application are not limited to the above-mentioned application device, nor are they limited to the above-mentioned pollution treatment method. Those skilled in the art may adopt other methods for pollution treatment.
[0119] In practice, after the application device completes the related pollution treatment work, it can return a response message indicating that the treatment is completed, and then continue to execute step 101 until the vehicle-mounted camera lens is of a pollution-free type.
[0120] In an embodiment of the present application, the vehicle-mounted camera lens can obtain the speed of the target vehicle. If it is detected that the speed of the target vehicle exceeds the preset pollution detection start threshold, the trained deep learning model is used to perform pollution detection on the image frames collected by the vehicle-mounted camera lens to determine the type of pollution when the vehicle-mounted camera lens is contaminated. By implementing pollution detection of the vehicle-mounted camera lens through the deep learning model, the pollution condition of the lens can be detected in time, the accuracy of pollution detection is improved, and the functions of the vehicle-mounted camera lens are enriched.
[0121] After the vehicle-mounted camera lens is identified as a contaminated type, a lens contamination signal is generated according to the contamination type of the vehicle-mounted camera lens, and the lens contamination signal is sent to an application device associated with the vehicle-mounted camera lens, so that the application device automatically performs contamination processing according to the lens contamination signal, thereby improving the efficiency of contamination processing and reducing the impact caused by lens contamination.
[0122] Corresponding to the embodiments of the aforementioned methods, the present application also provides an embodiment of a pollution treatment device for a vehicle-mounted camera lens.
[0123] The embodiment of the pollution treatment device for the vehicle camera lens of the present application can be applied in vehicle-mounted equipment. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Fig. 9 As shown, it is a hardware structure diagram of the device where the device of this application is located, except Fig. 9 In addition to the processor, memory, network interface, and non-volatile memory shown, the device in the embodiment where the apparatus is located may also include other hardware according to the actual function of the apparatus, which will not be described in detail.
[0124] Please refer to Fig.10 , shows a structural block diagram of an embodiment of a pollution treatment device for a vehicle-mounted camera lens shown in an exemplary embodiment of the present application, which may specifically include the following modules:
[0125] The pollution detection module 1001, when the speed of the target vehicle exceeds a preset pollution detection start threshold, uses a trained deep learning model to perform pollution detection on the image frames collected by the vehicle-mounted camera lens to determine the pollution type when the vehicle-mounted camera lens is contaminated;
[0126] A pollution signal generating module 1002, used to generate a lens pollution signal according to the pollution type of the vehicle-mounted camera lens;
[0127] The pollution signal sending module 1003 is used to send the lens pollution signal to an application device associated with the vehicle-mounted camera lens, so that the application device performs pollution processing according to the lens pollution signal.
[0128] In a preferred embodiment of the present application, the pollution treatment includes at least one or a combination of the following:
[0129] Issue pollution warning;
[0130] Cleaning the vehicle-mounted camera lens;
[0131] Driving assistance functions based on the vehicle's camera footage are suspended.
[0132] In a preferred embodiment of the present application, the pollution detection module 1001 may further include the following submodules:
[0133] An image sub-region contamination detection sub-module, used for inputting the image frame captured by the vehicle-mounted camera lens into a trained deep learning model, so that the deep learning model divides the image frame into a plurality of image sub-regions and calculates the contamination probability of each image sub-region;
[0134] The pollution type determination submodule is used to determine the pollution type when the vehicle-mounted camera lens is polluted according to the pollution probability of each image sub-area.
[0135] In a preferred embodiment of the present application, the pollution type determination submodule is specifically used for:
[0136] When the number of image frames input to the deep learning model reaches a preset number, the contamination probabilities of image sub-regions with the same regional positions in the preset number of image frames are averaged to obtain the regional contamination probability of each regional position;
[0137] If there is an area location where the area pollution probability is greater than or equal to the preset pollution threshold, it is determined that the vehicle-mounted camera lens is of the pollution type.
[0138] In a preferred embodiment of the present application, the inter-frame interval of the image frames input to the deep learning model is determined according to the speed of the target vehicle.
[0139] Please refer to Fig.11 , shows a structural block diagram of an embodiment of a pollution treatment system for a vehicle-mounted camera lens shown in an exemplary embodiment of the present application, which may specifically include the following modules:
[0140] The vehicle speed acquisition module 1101 is used to acquire the vehicle speed of the target vehicle detected by the vehicle speed sensor and send the vehicle speed to the lens contamination detection module;
[0141] The vehicle-mounted camera lens 1102 is used to collect image frames;
[0142] The lens contamination detection module 1103 is used to obtain the image frames captured by the vehicle-mounted camera lens when it is determined that the speed of the target vehicle exceeds a preset contamination detection start threshold, and use the trained deep learning model to perform contamination detection on the obtained image frames to determine the contamination type when the vehicle-mounted camera lens is contaminated;
[0143] The signal generating module 1104 is used to generate a lens contamination signal according to the contamination type of the vehicle-mounted camera lens, and send the lens contamination signal to an application device associated with the vehicle-mounted camera lens, so that the application device performs contamination processing according to the lens contamination signal.
[0144] In a preferred embodiment of the present application, the application device 1105 includes at least one or a combination of the following:
[0145] A prompting device, used for issuing a pollution prompt;
[0146] A cleaning device, used for cleaning the vehicle-mounted camera lens;
[0147] An advanced driver assistance ADAS device is used to suspend a driver assistance function based on the vehicle-mounted camera lens.
[0148] In a preferred embodiment of the present application, the lens contamination detection module 1103 includes:
[0149] A contamination probability determination submodule, used for inputting the image frame captured by the vehicle-mounted camera lens into a trained deep learning model, so that the deep learning model divides the image frame into a plurality of image sub-regions and calculates the contamination probability of each image sub-region;
[0150] The pollution type determination submodule is used to determine the pollution type when the vehicle-mounted camera lens is polluted according to the pollution probability of each image sub-area.
[0151] In a preferred embodiment of the present application, the pollution type determination submodule is specifically used for:
[0152] When the number of image frames input to the deep learning model reaches a preset number, the contamination probabilities of image sub-regions with the same regional positions in the preset number of image frames are averaged to obtain the regional contamination probability of each regional position;
[0153] If there is an area location where the area pollution probability is greater than or equal to the preset pollution threshold, it is determined that the vehicle-mounted camera lens is of the pollution type.
[0154] In a preferred embodiment of the present application, the inter-frame interval of the image frames input to the deep learning model is determined according to the speed of the target vehicle.
[0155] As for the device and system embodiments, since they basically correspond to the method embodiments, the relevant parts may refer to the partial description of the method embodiments.
[0156] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. Ordinary technicians in this field can understand and implement it without creative work.
[0157] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the above method embodiment are implemented.
[0158] An embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method embodiment when executing the program.
[0159] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
[0160] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.
[0161] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a vehicle-mounted terminal, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, just to name a few.
[0162] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.
[0163] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.
[0164] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.
[0165] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.
[0166] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A method for treating contamination of a vehicle-mounted camera lens, characterized in that: The method comprises: When the speed of the target vehicle exceeds a preset pollution detection start threshold, the trained deep learning model is used to perform pollution detection on the image frames captured by the vehicle-mounted camera lens to determine the pollution type when the vehicle-mounted camera lens is contaminated; Generate a lens contamination signal according to the contamination type of the vehicle-mounted camera lens, and send the lens contamination signal to an application device associated with the vehicle-mounted camera lens, so that the application device performs contamination processing according to the lens contamination signal; The training process of the deep learning model includes: Acquire an image sample set, wherein the image sample set includes non-polluted image samples and polluted image samples with marked polluted areas; Each image sample in the image sample set is divided into a plurality of first image sub-regions, and for each image sample, based on the proportion of pixels contained in the marked polluted region, each first image sub-region in the image sample is identified as a polluted type or a non-polluted type, and the pollution type or non-polluted type of each first image sub-region of the image sample is output to the convolution layer; wherein, in the first image sub-region identified as a polluted type, the proportion of pixels contained in the marked polluted region is greater than or equal to a preset proportion threshold; and in the first image sub-region identified as a non-polluted type, the proportion of pixels contained in the marked polluted region is less than the preset proportion threshold; The convolution layer learns the features of the first image sub-region of the pollution type based on the pollution type or non-pollution type of each first image sub-region of the received image samples, and generates a deep learning model after multiple iterative training of the learned features in combination with the pooling layer.
2. The method according to claim 1, characterized in that The pollution treatment includes at least one or a combination of the following: Issue pollution warning; Cleaning the vehicle-mounted camera lens; Driving assistance functions based on the vehicle's camera footage are suspended.
3. The method according to claim 1 or 2, characterized in that: The using of the trained deep learning model to perform pollution detection on the image frames collected by the vehicle-mounted camera lens to determine the pollution type when the vehicle-mounted camera lens is polluted includes: Inputting the image frame captured by the vehicle-mounted camera lens into a trained deep learning model, so that the deep learning model divides the image frame into a plurality of image sub-regions and calculates the contamination probability of each image sub-region; Determining the pollution type when the vehicle-mounted camera lens is polluted according to the pollution probability of each image sub-region; The step of identifying, for each image sample, each first image sub-region in the image sample as a pollution type or a non-pollution type includes: For each polluted image sample, a truth map of the polluted image sample is generated according to the marked polluted area in the polluted image sample, wherein the truth map is consistent with the size of the corresponding polluted image sample, the pixel values of the pixels included in the marked polluted area in the truth map are the first preset pixel values, and the pixel values of the pixels outside the polluted area are the second preset pixel values; Dividing the truth map into a plurality of second image sub-regions in a sub-region division manner consistent with the corresponding contaminated image sample, wherein each second image sub-region in the truth map has a mapped first image sub-region in the corresponding contaminated image sample; Calculate the proportion of pixel points in each second image sub-region whose pixel value is the first preset pixel value, and determine the first image sub-region mapped by the second image sub-region whose proportion is greater than or equal to the preset proportion threshold as a pollution type; determine the first image sub-region mapped by the second image sub-region whose proportion is less than the preset proportion threshold as a non-pollution type.
4. The method according to claim 3, characterized in that The determining, based on the pollution probability of each image sub-region, that the vehicle-mounted camera lens is polluted as a pollution type when there is pollution, includes: When the number of image frames input to the deep learning model reaches a preset number, the contamination probabilities of image sub-regions with the same regional positions in the preset number of image frames are averaged to obtain the regional contamination probability of each regional position; If there is an area location where the area pollution probability is greater than or equal to the preset pollution threshold, it is determined that the vehicle-mounted camera lens is of the pollution type.
5. The method according to claim 4, characterized in that The inter-frame interval of the image frames input to the deep learning model is determined according to the speed of the target vehicle.
6. A pollution treatment system for a vehicle-mounted camera lens, characterized in that: The system comprises: A vehicle speed acquisition module, used to acquire the vehicle speed of the target vehicle detected by the vehicle speed sensor, and send the vehicle speed to the lens contamination detection module; On-board camera lens, used to collect image frames; A lens contamination detection module is used to obtain image frames captured by the vehicle-mounted camera lens when it is determined that the speed of the target vehicle exceeds a preset contamination detection start threshold, and use a trained deep learning model to perform contamination detection on the obtained image frames to determine the contamination type when the vehicle-mounted camera lens is contaminated; A signal generating module, configured to generate a lens contamination signal according to the contamination type of the vehicle-mounted camera lens, and send the lens contamination signal to an application device associated with the vehicle-mounted camera lens, so that the application device performs contamination processing according to the lens contamination signal; The training process of the deep learning model includes: Acquire an image sample set, wherein the image sample set includes non-polluted image samples and polluted image samples with marked polluted areas; By using a data layer in a convolutional neural network, according to a preset image division rule, the image division rule includes the number of first image sub-regions into which the image sample needs to be divided, and combining the size of each image sample, each image sample in the image sample set is divided into a plurality of first image sub-regions, and for each image sample, each first image sub-region in the image sample is identified as a pollution type or a non-pollution type, and the pollution type or non-pollution type of each first image sub-region of the image sample is output to the convolutional layer; Among them, in the first image sub-region identified as the pollution type, the proportion of pixels contained in the marked pollution area is greater than or equal to the preset proportion threshold; in the first image sub-region identified as the non-pollution type, the proportion of pixels contained in the marked pollution area is less than the preset proportion threshold; The convolution layer learns the features of the first image sub-region of the pollution type based on the pollution type or non-pollution type of each first image sub-region of the received image samples, and generates a deep learning model after multiple iterative training of the learned features in combination with the pooling layer.
7. The system according to claim 6, characterized in that The application device includes at least one or a combination of the following: A prompting device, used for issuing a pollution prompt; A cleaning device, used for cleaning the vehicle-mounted camera lens; An advanced driver assistance ADAS device is used to suspend a driver assistance function based on the vehicle-mounted camera lens.
8. The system according to claim 6 or 7, characterized in that: The lens contamination detection module comprises: A contamination probability determination submodule, used for inputting the image frame captured by the vehicle-mounted camera lens into a trained deep learning model, so that the deep learning model divides the image frame into a plurality of image sub-regions and calculates the contamination probability of each image sub-region; A pollution type determination submodule, used to determine the pollution type of the vehicle-mounted camera lens when there is pollution according to the pollution probability of each image sub-area; During the training of the deep learning model, for each image sample, identifying each first image sub-region in the image sample as a pollution type or a non-pollution type includes: For each polluted image sample, a truth map of the polluted image sample is generated according to the marked polluted area in the polluted image sample, wherein the truth map is consistent with the size of the corresponding polluted image sample, the pixel values of the pixels included in the marked polluted area in the truth map are the first preset pixel values, and the pixel values of the pixels outside the polluted area are the second preset pixel values; Dividing the truth map into a plurality of second image sub-regions in a sub-region division manner consistent with the corresponding contaminated image sample, wherein each second image sub-region in the truth map has a mapped first image sub-region in the corresponding contaminated image sample; Calculate the proportion of pixel points in each second image sub-region whose pixel value is the first preset pixel value, and determine the first image sub-region mapped by the second image sub-region whose proportion is greater than or equal to the preset proportion threshold as a pollution type; determine the first image sub-region mapped by the second image sub-region whose proportion is less than the preset proportion threshold as a non-pollution type.
9. The system according to claim 8, characterized in that The pollution type determination submodule is specifically used for: When the number of image frames input to the deep learning model reaches a preset number, the contamination probabilities of image sub-regions with the same regional positions in the preset number of image frames are averaged to obtain the regional contamination probability of each regional position; If there is an area location where the area pollution probability is greater than or equal to the preset pollution threshold, it is determined that the vehicle-mounted camera lens is of the pollution type.
10. The system according to claim 9, characterized in that The inter-frame interval of the image frames input to the deep learning model is determined according to the speed of the target vehicle.
Citation Information
Cited By
Pollution level estimation system, pollution level estimation method, and pollution level estimation program
US12802253B2
Pollution level estimation system, pollution level estimation method, and pollution level estimation program
US20240320848A1
