Unmanned aerial vehicle lens stain identification and cleaning method and system based on deep learning

Abstract

The invention relates to an unmanned aerial vehicle lens stain identification and cleaning method and system based on deep learning. The method comprises the following steps: generating an image acquisition instruction according to a set sampling period to obtain a to-be-detected image; the image acquisition instruction is used for instructing an unmanned aerial vehicle camera to perform image acquisition operation and feeding back in the form of a to-be-detected image; preprocessing the to-be-detected image to obtain a to-be-detected standard image; inputting a to-be-detected standard image into the stain detection model to obtain a stain identification result; the stain identification result comprises a central coordinate of the polluted area, bounding box data, a confidence coefficient score and a stain type label; generating a cleaning parameter and a cleaning instruction according to the stain identification result; the cleaning instruction is used for instructing the cleaning equipment to clean the unmanned aerial vehicle lens according to the cleaning parameters. By adopting the method, the problems of difficult stain identification and extensive cleaning decision in an existing unmanned aerial vehicle visual system can be solved, and the accuracy of stain detection and the intelligence of cleaning control are realized.

Description

Technical Field

[0001] The present invention belongs to the field of lens stain recognition and cleaning, and in particular relates to a method and system for identifying and cleaning unmanned aerial vehicle lens stains based on deep learning. Background Art

[0002] With the development of drone technology, visual perception technology has emerged, which usually relies on high-definition cameras to continuously collect external images and serve as the input source for functions such as navigation, recognition, and detection.

[0003] In order to ensure the quality of image acquisition, traditional camera lens cleaning technology mainly uses physical means or electronic control devices to remove contaminants on the lens surface to restore image clarity.

[0004] However, traditional cleaning technology cannot automatically determine whether the lens is contaminated. It often performs cleaning actions frequently when the lens is not contaminated, wasting energy and shortening the life of the equipment. In applications with high requirements for image quality, it is difficult to achieve intelligent identification of contamination and coordinated control of cleaning. Summary of the Invention

[0005] Based on this, it is necessary to provide a deep learning-based drone lens stain recognition and cleaning method and system that can identify pollution and adaptively control cleaning to address the above technical problems.

[0006] In the first aspect, the present application provides a method for identifying and cleaning stains on drone lenses based on deep learning, comprising:

[0007] Generate image acquisition instructions according to the set sampling period to obtain the image to be detected; the image acquisition instructions are used to instruct the drone camera to perform image acquisition operations and provide feedback in the form of the image to be detected;

[0008] Preprocess the image to be detected to obtain a standard image to be detected;

[0009] Input the standard image to be detected into the stain detection model to obtain the stain recognition result; the stain recognition result includes the center coordinates of the contaminated area, bounding box data, confidence score and stain type label;

[0010] Cleaning parameters and cleaning instructions are generated based on the stain identification results; the cleaning instructions are used to instruct the cleaning equipment to clean the drone lens according to the cleaning parameters.

[0011] In one embodiment, the stain detection model obtains the stain recognition result by the following method:

[0012] Perform multi-scale feature extraction on the standard image to be detected to obtain a multi-scale feature map; the multi-scale feature map includes multiple grid cells of detection candidates;

[0013] Predict candidate bounding boxes based on the grid cells to obtain a bounding box set; the bounding box set includes multiple candidate bounding boxes and their corresponding bounding box data and confidence scores;

[0014] Based on the non-maximum suppression algorithm, multiple candidate bounding boxes are filtered according to the confidence score to obtain a set of contaminated areas; the contaminated area set includes the center coordinates of the contaminated area, the size of the contaminated area bounding box, the stain type label, the confidence score and the total number of detected stain targets;

[0015] The contaminated area set is encapsulated in a structured form to obtain the stain recognition result.

[0016] In one embodiment, multi-scale feature extraction is performed on the standard image to be detected to obtain a multi-scale feature map, including:

[0017] Perform layer-by-layer feature extraction on the standard image to be detected to obtain an intermediate feature map; the intermediate feature map includes a small target feature map, a medium target feature map, and a large target feature map;

[0018] Perform channel attention enhancement on the intermediate feature map to obtain a channel enhanced feature map;

[0019] Perform spatial attention enhancement on the channel-enhanced feature map to obtain a multi-scale feature map;

[0020] The channel enhancement feature map is obtained through the following formula;

[0021]

[0022] Among them, P l is the intermediate feature map, P l ∈R C×H×W ; l is the target feature map, l∈(1,2,3); is the average pooling feature; is the maximum pooling feature; M c is the MLP shared fusion feature; σ is the activation function; P ' l Enhance the feature map for the channel;

[0023] The multi-scale feature map is obtained through the following formula:

[0024]

[0025] in, is a multi-scale feature map; is the spatial average pooling feature; is the spatial maximum pooling feature; M s Concatenated convolutional features; It is the channel enhanced feature map after feature enhancement.

[0026] In one embodiment, predicting candidate bounding boxes based on grid cells to obtain a bounding box set includes:

[0027] Predict multiple candidate bounding boxes for each grid cell and obtain a set of candidate bounding box regression parameters;

[0028] The activation function is used to activate the candidate bounding box regression parameter set and decode it to obtain a bounding box set; the bounding box set includes the bounding box center coordinates and bounding box size data.

[0029] In one embodiment, generating cleaning parameters based on the stain identification result includes:

[0030] According to the stain identification results, the heating and cleaning demand weight of each stain target is calculated to obtain the overall pollution intensity weight;

[0031] Based on the dynamic cleaning strategy, the heating time and power are obtained according to the overall pollution intensity weight;

[0032] The cleaning parameters are obtained according to the heating time and heating power;

[0033] The heating and cleaning demand weight is obtained through the following formula:

[0034] Q k =λ1·(w k ·h k )+λ2·s k

[0035]

[0036] Among them, Q k is the heating cleaning demand weight of a single stain target; λ1 and λ2 are weight coefficients; w k is the width of the bounding box of the contaminated area; h k is the height of the bounding box of the contaminated area; s k is the confidence score; k is the stain target index; N is the total number of stains; Q total is the overall pollution intensity weight;

[0037] The heating time and heating power are obtained by the following formula:

[0038] T=T0+γ·Q total

[0039] P=P0+η·normalize(Q total )

[0040] Among them, T is the heating time; T0 is the basic heating time; γ is the time adjustment coefficient; P is the heating power; P0 is the heater starting power; η is the power adjustment coefficient.

[0041] In one embodiment, the method further comprises:

[0042] Generate a verification image acquisition instruction to obtain a cleaned image; the verification image acquisition instruction is used to instruct the drone camera to perform an image acquisition operation and provide feedback in the form of a cleaned image;

[0043] Perform difference detection on the image to be detected and the corresponding cleaned image according to a preset time window length to obtain a joint loss function;

[0044] Incrementally train the stain detection model and optimize the model parameters based on the joint loss function;

[0045] The joint loss function is obtained through the following formula:

[0046] L total =L basic +β·L pair

[0047]

[0048] Among them, L pair is the joint loss function; L basic is the basic loss function of the model; L pair is the difference sensitivity loss function; β is the adjustment coefficient; N is the number of images; i is the image number index; is the image to be detected; is the cleaned image; f is the feature extraction function; ‖·‖ 2 is the Euclidean distance.

[0049] In one embodiment, the stain detection model is obtained by:

[0050] Obtain normal lens image samples, lens contamination image samples and their corresponding contamination area annotations, and perform data augmentation to obtain a training set; contamination area annotation includes bounding box annotation and stain type annotation;

[0051] Use the training set to train the YOLOv5-based pre-training model to obtain model parameters;

[0052] The model parameters with the best performance are calculated through loss function to form a stain detection model.

[0053] Secondly, this application also provides a drone lens stain recognition and cleaning system based on deep learning, including:

[0054] The image data acquisition module is used to generate image acquisition instructions according to the set sampling period to obtain the image to be detected;

[0055] An image preprocessing module is used to preprocess the image to be detected to obtain a standard image to be detected;

[0056] The stain recognition module is used to input the standard image to be detected into the stain detection model to obtain the stain recognition result;

[0057] The cleaning module is used to generate cleaning parameters and cleaning instructions based on the stain recognition results.

[0058] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of any of the above-mentioned deep learning-based drone lens stain recognition and cleaning methods.

[0059] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned deep learning-based drone lens stain recognition and cleaning methods.

[0060] The above-mentioned deep learning-based drone lens stain recognition and cleaning method and system, by constructing a drone lens stain recognition and cleaning process, can realize automatic detection of stains and cleaning response, ensuring the long-term clarity and reliability of the drone vision system. The deep learning-based stain detection model realizes high-precision recognition of various types of stains, and supports precise positioning of the position and size of the contaminated area, improving the granularity and accuracy of recognition. Through the process design of image acquisition, preprocessing, recognition and cleaning instruction generation, modularization and efficient collaboration are achieved, facilitating actual deployment and application. Automated cleaning control reduces manual intervention and maintenance costs, improves the operating stability and task completion rate of drones in complex environments, is suitable for a variety of drone platforms and application scenarios, and improves the intelligence level of drone vision equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 This is a flowchart of the deep learning-based drone lens stain recognition and cleaning method of the present invention;

[0063] Figure 2 Schematic diagram of the step-by-step process of step S103;

[0064] Figure 3 Schematic diagram of the step-by-step process of step S201;

[0065] Figure 4 This is a structural diagram of the drone lens stain recognition and cleaning system based on deep learning of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0067] In one embodiment, Figure 1 As shown, a method for identifying and cleaning stains on drone lenses based on deep learning is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0068] S101. Generate an image acquisition instruction according to a set sampling period to obtain an image to be detected. The image acquisition instruction is used to instruct the drone camera to perform an image acquisition operation and feedback in the form of an image to be detected.

[0069] Schematically, the image acquisition instruction can be sent to the camera module on the drone side via wired or wireless communication, driving it to acquire the current image frame as the image to be detected. By acquiring the image frame at the current moment, the environment in front of the lens and the potential pollution conditions can be directly reflected. Pollution such as water droplets, fog spots, oil films, etc. usually appear as bright spots, blurs, distortions or areas with reduced contrast in the image, and the visual features presented can be effectively identified by deep learning models. Optionally, image acquisition is not limited to fixed cycle control, but can also be activated by other triggering mechanisms, for example, when the image quality assessment value is lower than the threshold, a preset position in the flight path, a sudden weather change warning or manual remote triggering, etc. In complex application scenarios, the image acquisition strategy can be flexibly configured according to task priority and real-time requirements, thereby realizing multi-source perception and adaptive sampling.

[0070] S102 : Preprocess the image to be detected to obtain a standard image to be detected.

[0071] To improve the uniformity and recognizability of the input image and stabilize the prediction results of the subsequent neural network model, the image to be tested is preprocessed. Specifically, the image is resized to the input size used for model training (for example, 640×640 pixels) to avoid feature scale shifts caused by inconsistent sizes. Furthermore, image brightness and contrast are normalized to ensure that the image's lighting characteristics are uniformly distributed across different environments.

[0072] Optionally, if random noise exists in the sensor image, a noise reduction algorithm such as median filtering or bilateral filtering can be introduced to improve image clarity.

[0073] Optionally, if the model is trained using a specific color space such as HSV (Hue, Saturation, Value, color space) or Lab (color model), a color space conversion is performed to make pollution features such as water mark edges or oil film spots more prominent in a certain channel.

[0074] S103 , inputting the standard image to be detected into the stain detection model to obtain a stain recognition result; the stain recognition result includes the center coordinates of the contaminated area, bounding box data, confidence score and stain type label.

[0075] Schematically, the preprocessed standard image is input into a pre-trained stain detection model to perform identification, positioning and label determination of contaminated targets. Specifically, the stain detection model adopts the YOLOv5 (You Only LookOnce v5, a deep learning model for target detection) series network architecture, which has the characteristics of lightweight, fast inference and high detection accuracy. Inside the model, the standard image to be detected is extracted through a deep convolutional structure to form multiple candidate areas; several bounding boxes are predicted on each grid unit, and each candidate box outputs its center coordinates, width and height dimensions, contamination confidence, and stain category label information. Furthermore, the non-maximum suppression algorithm is used to remove redundant boxes and retain the contaminated area with the highest confidence, and finally a set of contaminated areas is obtained. The core elements contained in the contaminated area set include the center coordinates of each contaminated target, the bounding box size, the stain type label, the confidence score, and the total number of detected targets. Optionally, the contamination information corresponding to the contaminated area set is further encapsulated into a structured output JSON or other data format for downstream module calls.

[0076] S104. Generate cleaning parameters and cleaning instructions based on the stain identification result; the cleaning instructions are used to instruct the cleaning equipment to clean the drone lens according to the cleaning parameters.

[0077] Indicatively, cleaning control parameters and execution instructions are generated based on the pollution identification results, and the physical cleaning device is driven to complete the actual action. Specifically, the number, distribution range and confidence level of the stained areas are comprehensively considered to evaluate the severity of the pollution and generate cleaning parameters, such as cleaning time, cleaning power level or local execution location. Furthermore, the cleaning parameters are encapsulated into a control signal data packet and sent to the drone cleaning controller or lens heater, electric wiping device and other execution units. Optionally, the cleaning action can be carried out in a variety of ways such as thermal evaporation, electric wiping, liquid spraying, etc., which can be flexibly called according to the hardware configuration.

[0078] For example, a drone inspecting the facades of urban buildings in rain and fog was observed to have residual water droplets in front of its camera lens. Images were collected within a 5-second sampling period, pre-processed, and fed into the YOLOv5 model, which identified two contaminated targets with a confidence level exceeding 0.8. Based on the contaminated area and the confidence level, the model calculated a cleaning time of 3.5 seconds and a medium power level. The model then activated the transparent heating film in front of the camera to initiate the cleaning process.

[0079] The deep learning-based drone lens stain identification and cleaning method described above generates image acquisition instructions according to a set sampling cycle and drives the drone camera to capture images to be inspected, enabling real-time monitoring of the lens contamination status, ensuring timely and continuous detection. Standardized preprocessing of the acquired images improves image quality and consistency with model input data, enhancing the recognition stability and accuracy of the stain detection model. The stain detection model analyzes the preprocessed images to accurately identify the spatial location, boundary size, confidence level, and stain type of contaminated areas, achieving a refined representation of contamination information and providing reliable data support for subsequent cleaning control. Dynamic generation of cleaning parameters and cleaning instructions based on the stain identification results enables automatic linkage between stain identification and cleaning actions, effectively improving cleaning efficiency and avoiding resource waste. The cleaning instructions accurately instruct the cleaning equipment to perform targeted cleaning operations on the drone camera, enabling intelligent and automated lens maintenance, reducing manual intervention, and improving the autonomous operation of the cleaning method.

[0080] In one embodiment, Figure 2 As shown in the figure, the stain detection model obtains the stain recognition result through the following method:

[0081] S201 , performing multi-scale feature extraction on a standard image to be detected to obtain a multi-scale feature map; the multi-scale feature map includes a plurality of grid units of detection candidates.

[0082] Schematically, a multi-scale feature extraction operation is performed on the input standard image to be detected. This process is usually completed based on a deep convolutional neural network. For example, the CSPDarknet (backbone network) structure of YOLOv5 is used to process the image layer by layer through multi-layer convolution, activation and residual connection operations. Specifically, the backbone network will output feature maps of multiple scales. For example, three groups of feature maps of 80×80, 40×40 and 20×20 are output. Each group corresponds to the receptive field of targets of different sizes on the original image, covering a variety of possibilities such as tiny stains, medium pollution areas and large areas of stains. Each spatial unit of the feature map can be regarded as an encoded expression of a certain area of ​​the original image, that is, a grid unit, on which the prediction results of the candidate pollution targets will be further generated.

[0083] S202 : Predict candidate bounding boxes according to the grid cells to obtain a bounding box set; the bounding box set includes multiple candidate bounding boxes and their corresponding bounding box data and confidence scores.

[0084] Several candidate bounding boxes are predicted on each grid unit to form a bounding box set. Specifically, each grid position can generate prediction boxes with different aspect ratios based on multiple preset anchor boxes. Each box outputs the corresponding bounding box coordinate offset, target confidence, and stain category score. The bounding box data includes the relative position offset of the center point, the logarithmic transformation parameters of the width and height, and the confidence score reflects the probability that a real contaminated target exists in the current bounding box. After Sigmoid function activation and coordinate decoding, each candidate bounding box is converted into a rectangular box at the scale of the original image to form a complete set of candidate bounding boxes.

[0085] S203. Filter multiple candidate bounding boxes according to the confidence scores based on the non-maximum suppression algorithm to obtain a set of contaminated areas; the set of contaminated areas includes the center coordinates of the contaminated area, the size of the contaminated area bounding box, the stain type label, the confidence score, and the total number of detected stain targets.

[0086] Schematically, in order to eliminate repeated detection or interference from overlapping areas, the non-maximum suppression (NMS) algorithm is used to screen the bounding box set. The algorithm is based on the degree of overlap between candidate bounding boxes and the corresponding confidence scores. For example, the degree of overlap between candidate bounding boxes is calculated using IoU (intersection over union). Furthermore, bounding boxes with higher confidence are retained first, and low-confidence boxes that highly overlap with them are eliminated, thereby ensuring the uniqueness and accuracy of the output results, and obtaining a set of contaminated areas. Each contaminated area contains the center coordinates, bounding box size, stain type label, and corresponding confidence score. The stain type can be water droplets, fog spots, etc. Optionally, the total number of detected stain targets is counted as one of the input bases for the cleaning strategy.

[0087] S204: Encapsulate the pollution area set in a structured form to obtain a pollution identification result.

[0088] For example, the set of polluted areas is structured and packaged, and the identification results are output in a standard format, which can be expressed in the form of JSON, XML or database records, where each pollution target contains information such as location attributes, category attributes and score attributes.

[0089] In the above method, the stain detection model achieves sensitive perception of contaminated targets of different shapes and scales through multi-scale feature extraction, improves recognition accuracy and result uniqueness through candidate bounding box prediction and non-maximum suppression, and finally provides reliable control input for the cleaning module through structured encapsulation.

[0090] In one embodiment, Figure 3 As shown, multi-scale feature extraction is performed on the standard image to be detected to obtain a multi-scale feature map, including:

[0091] S301 , extracting features layer by layer from the standard image to be detected to obtain an intermediate feature map; the intermediate feature map includes a small target feature map, a medium target feature map, and a large target feature map.

[0092] A layer-by-layer convolutional feature extraction operation is performed on the standard image to construct an intermediate feature map. This process is typically based on the backbone architecture of a deep neural network, extracting spatial and semantic information from the image through successive convolutional layers, activation functions, and residual connections. During this process, the network extracts three representative intermediate feature maps at different downsampling rates: small object feature maps, medium object feature maps, and large object feature maps. For example, the downsampling rates can be 1 / 8, 1 / 16, and 1 / 32. The small object feature map retains high spatial resolution and is suitable for detecting tiny and fine contamination traces, such as water droplets or fog spots. The medium-scale feature map is suitable for detecting medium-sized stains. The large object feature map integrates contextual information from the entire image and is more suitable for detecting contamination scenarios such as large blur or fluid coverage. This hierarchical division provides the model with a multi-granular perspective, ensuring sufficient representation capabilities for multi-modal contamination problems.

[0093] S302: Perform channel attention enhancement on the intermediate feature map to obtain a channel enhanced feature map.

[0094] The channel enhancement feature map is obtained through the following formula;

[0095]

[0096] Among them, P l is the intermediate feature map, P l ∈R C×H×W; l is the target feature map, l∈(1,2,3); is the average pooling feature; is the maximum pooling feature; M c is the MLP shared fusion feature; σ is the activation function; P ' l Enhance the feature map for the channel;

[0097] To further improve the response strength of effective channels in the feature map, a channel attention mechanism is introduced to perform channel enhancement processing on the intermediate feature map. The channel attention mechanism enables the model to automatically identify which channels contain features related to the polluted target and assign them higher weights. This process usually performs global average pooling and maximum pooling on each feature map channel dimension, extracts a global description vector, and inputs the global description variable into a shared fully connected network to generate a channel weight vector. This weight vector is then multiplied channel by channel with the original feature map, i.e., the intermediate feature map, to achieve channel-selective suppression and enhancement. Through this mechanism, the model can actively suppress redundant background features and highlight key information such as color deviation, edge anomalies, and local brightness changes in the polluted area, thereby improving the positioning accuracy and category discrimination ability of subsequent target detection.

[0098] S303: Perform spatial attention enhancement on the channel enhancement feature map to obtain a multi-scale feature map.

[0099] The multi-scale feature map is obtained through the following formula:

[0100]

[0101] in, is a multi-scale feature map; is the spatial average pooling feature; is the spatial maximum pooling feature; M s Concatenated convolutional features; It is the channel enhanced feature map after feature enhancement.

[0102] Furthermore, a spatial attention mechanism is applied to the channel-enhanced feature map to complete the final multi-scale feature map construction. The introduction of spatial attention aims to reweight the spatial position dimension of the feature map, so that the model can identify which areas in the image are more likely to contain contaminated targets in terms of spatial position. Unlike channel attention, spatial attention forms a two-dimensional attention map by performing average pooling and maximum pooling on the enhanced feature map in the channel dimension, and generates a spatial weight matrix through convolution and Sigmoid activation. This matrix is ​​multiplied pixel by pixel with the channel-enhanced feature map, so that the response of the edge of the contaminated area or the light spot area in the feature map is further amplified, while the response of the invalid area is suppressed. It is suitable for solving the recognition problem of blurred contaminated areas, weak boundaries or low contrast.

[0103] In the above method, in an image with both background reflections and stains, both the reflective areas and the polluted areas in the original feature map may activate a high response of the model. However, after channel attention enhancement, the model can make semantic judgments that the pollution features are more significant. Through the spatial attention mechanism, the model further clarifies that the pollution appears in the dark area in the upper left corner of the image, and ultimately generates candidate bounding boxes only in the real polluted area, significantly improving the positioning accuracy and robustness of the stained targets.

[0104] In one embodiment, predicting candidate bounding boxes based on grid cells to obtain a bounding box set includes:

[0105] S41. Predict multiple candidate bounding boxes for each grid unit to obtain a set of candidate bounding box regression parameters.

[0106] Schematically, the multi-scale feature map is divided into multiple grid cells along the spatial dimension. Each grid cell represents a corresponding region in the original image and is responsible for detecting local contamination. Specifically, for each cell, the model predicts several candidate bounding boxes, which typically match the number of preset anchor boxes. For example, each grid cell corresponds to three candidate boxes.

[0107] Each candidate bounding box is represented by a set of regression parameters, called the regression parameter set, which mainly includes the offset of the box center point relative to the grid cell, the logarithmic scaling of the width and height, and the confidence that the box contains the target, accompanied by a set of classification scores for determining the type of contamination.

[0108] S42. Activate the candidate bounding box regression parameter set using an activation function and decode to obtain a bounding box set; the bounding box set includes bounding box center coordinates and bounding box size data.

[0109] The regression parameters of the candidate bounding box are nonlinearly transformed through the activation function, and the coordinate decoding is performed in combination with the anchor box template to obtain a set of bounding boxes at the standard image size. Specifically, the center coordinate offset is activated by the Sigmoid (activation function) function so that its output value is limited to between 0 and 1 to prevent the predicted center point from falling out of the current grid range; the width and height parameters are inversely solved by the exponential function to obtain the actual bounding box size ratio, and multiplied by the initial width and height of the corresponding anchor box to restore it to the original image scale. Furthermore, combined with the grid unit position and stride information, the normalized coordinates are restored to the absolute value coordinates in the image coordinate system to form a complete bounding box set. In the decoded bounding box set, each bounding box contains the center point position coordinates at the original image scale and the width and height of the bounding box.

[0110] For example, in a small object detection layer with an 80×80 feature map, the model predicts three bounding boxes per grid cell, outputting a total of 19,200 candidate bounding boxes. After activation, the center coordinates of each box are offset by approximately 0.1 to 0.9 relative to the grid cell, and the width and height have different scale responses due to the size of the pollutant. Exponential mapping, combined with the anchor boxes, can restore the true size of the polluted area.

[0111] The above method, through a candidate bounding box prediction and decoding mechanism guided by a grid structure, can achieve efficient, accurate, and structured preliminary positioning of polluted targets at a standard image scale, and provide a spatial location basis for subsequent pollution screening and identification result generation.

[0112] In one embodiment, generating cleaning parameters based on the stain identification result includes:

[0113] S51. Calculate the heating and cleaning demand weight for each stain target according to the stain identification result to obtain the overall pollution intensity weight.

[0114] For example, among various lens cleaning technologies, heating cleaning is widely used as an efficient, non-contact cleaning method due to its simple structure, fast response, and minimal damage to the equipment. In drones, heating devices are used to quickly evaporate water droplets, mist, or light stains from the lens surface, ensuring the continuity of the visual system while reducing maintenance frequency and manual intervention costs.

[0115] Heated cleaning uses a heated film or wire to evenly transfer heat to the lens surface, evaporating any attached water droplets or film, thereby quickly restoring lens clarity. Compared to traditional mechanical wiping or spray cleaning, heated cleaning avoids the risk of scratches caused by direct contact with the lens and does not rely on external liquids, making it suitable for continuous use in harsh environments. Furthermore, the heating device can be electronically controlled to precisely adjust power and duration, enabling customized cleaning strategies for varying levels of contamination and improving energy efficiency.

[0116] Schematically, by combining the results of deep learning recognition, the pollution intensity and cleaning demand weight are dynamically calculated, and the heating cleaning parameters are reasonably adjusted to give full play to the rapid response and energy-saving advantages of heating cleaning, thereby improving the intelligent level of lens pollution control. Based on the multiple pollution targets output by the stain recognition model, the heating cleaning demand weight corresponding to each target is calculated. This weight quantifies the overall severity of the pollution on the lens surface, comprehensively considering the area size and confidence score of the pollution area, and reflects the actual demand for cleaning resources for this target. For example, larger pollution areas or targets with higher confidence usually represent serious pollution and require more heating energy to be allocated.

[0117] S52. Based on the dynamic cleaning strategy, the heating time and heating power are obtained according to the overall pollution intensity weight. The heating cleaning demand weight is obtained by the following formula:

[0118] Q k =λ1·(w k ·h k )+λ2·s k

[0119]

[0120] Among them, Q k is the heating cleaning demand weight of a single stain target; λ1 and λ2 are weight coefficients; w k is the width of the bounding box of the contaminated area; h k is the height of the bounding box of the contaminated area; s k is the confidence score; k is the stain target index; N is the total number of stains; Q total is the overall pollution intensity weight;

[0121] Schematically, based on the dynamic cleaning strategy, the overall pollution intensity weight is mapped to the specific heating time and heating power. The strategy realizes adaptive heating control for different pollution levels through preset functions or empirical formulas, avoiding blind excessive or insufficient cleaning actions.

[0122] S53. Obtain cleaning parameters according to the heating time and heating power.

[0123] The heating time and heating power are obtained by the following formula:

[0124] T=T0+γ·Q total

[0125] P=P0+η·normalize(Q total )

[0126] Among them, T is the heating time; T0 is the basic heating time; γ is the time adjustment coefficient; P is the heating power; P0 is the heater starting power; η is the power adjustment coefficient.

[0127] Heating time and power are used as cleaning parameters, encapsulated into a command format that can be directly parsed and executed by the cleaning device's control unit. These cleaning parameters typically include specific time values, power levels, and necessary control flags. These commands are transmitted via a communication interface to the heating controller or related hardware, driving the heating film, heating wire, and other devices to perform targeted cleaning. Properly setting cleaning parameters ensures that the cleaning action effectively removes contaminants without causing equipment overheating or wasting power.

[0128] The above method, a dynamic cleaning parameter generation method based on stain recognition results, effectively realizes the intelligent linkage of pollution detection and cleaning actions, improves the adaptability and efficiency of drone lens cleaning methods, and is a key technical support for ensuring the continuous and stable operation of visual equipment.

[0129] In one embodiment, the method further comprises:

[0130] S61. Generate a verification image acquisition instruction to obtain a cleaning completed image. The verification image acquisition instruction is used to instruct the drone camera to perform an image acquisition operation and provide feedback in the form of a cleaning completed image.

[0131] In principle, to ensure that the cleaning effect meets the requirements, the image can be recaptured after cleaning and the pollution identification process can be executed again to determine whether the pollution has been completely removed. If there is still residual pollution, a second cleaning is automatically initiated or the action duration is extended to form a closed-loop feedback between pollution identification and cleaning control. Specifically, a special verification image acquisition instruction is generated to instruct the drone camera to promptly capture the cleaned image after the cleaning action is completed. This image serves as direct feedback on the pollution removal effect and truly reflects the cleaning status of the lens surface. By matching the verification image acquisition instruction with the initial image acquisition instruction to be detected, an image pair before and after cleaning is established to ensure the temporal and spatial consistency of the data.

[0132] S62 , performing difference detection on the image to be detected and the corresponding cleaned image according to a time window of a preset length, and obtaining a joint loss function.

[0133] The joint loss function is obtained through the following formula:

[0134] L total =L basic +β·L pair

[0135]

[0136] Among them, L pair is the joint loss function; L basic is the basic loss function of the model; L pair is the difference sensitivity loss function; β is the adjustment coefficient; N is the number of images; i is the image number index; is the image to be detected; is the cleaned image; f is the feature extraction function; ||·|| 2 is the Euclidean distance.

[0137] Within a time window of preset length, a difference detection is performed on each pair of images to be detected and their corresponding cleaned images. The difference detection process is based on the feature extraction mechanism within the deep learning model. The images before and after cleaning are mapped to the feature space respectively. The cleaning effect is quantified by calculating the difference measure between the features. This is achieved by designing a joint loss function, which combines the traditional detection error with the feature consistency between time series. This loss function not only considers the accuracy of contamination identification, but also strengthens the expression of state changes before and after cleaning. The standard loss of the stain detection model measures the accuracy of the predicted bounding box and category, prompting the model to learn not only to identify contaminated areas during incremental training, but also to perceive the visual differences caused by cleaning actions, thereby improving the model's sensitivity to cleaning effects and recognition accuracy.

[0138] S63. Perform incremental training on the stain detection model according to the joint loss function to optimize model parameters.

[0139] Schematically, the stain detection model is incrementally trained and parameter optimized based on a joint loss function. Incremental training utilizes online learning or periodic offline updates, taking newly acquired before-and-after cleaning image pairs as training inputs and fine-tuning the model parameters via backpropagation, combined with existing trained weights. This process eliminates the need for manual labeling of stains and leverages naturally occurring before-and-after cleaning pairs to fully implement weakly supervised learning, ensuring the model's continued adaptability in real-world deployments. Incremental training not only corrects for model drift caused by factors like ambient lighting changes and sensor noise, but also addresses variations in stain form and lens material, improving the robustness and generalization of stain detection.

[0140] The above method, cleaning verification and model adaptive optimization module builds a closed-loop mechanism based on real-time feedback, which can not only automatically identify and clean lens contamination, but also improve the recognition model performance through continuous learning and adapt to complex and changing application environments.

[0141] In one embodiment, the stain detection model is obtained by:

[0142] S71. Obtain normal lens image samples, contaminated lens image samples, and their corresponding contaminated area annotations, and perform data enhancement to obtain a training set; the contaminated area annotations include bounding box annotations and stain type annotations.

[0143] Schematically, two types of lens image samples are collected: normal lens image samples and contaminated lens image samples. Normal image samples reflect a clear view of the environment when the lens is uncontaminated, while contaminated image samples contain various forms of contamination, such as water droplets, oil films, and fog spots. For contaminated images, the bounding boxes of the contaminated areas and their corresponding stain type labels are manually or semi-automatically annotated to form a high-quality training dataset. To improve the model's generalization and anti-interference capabilities, the training set is also expanded through various data augmentation techniques, including random rotation, scaling, color transformation, and mosaic stitching, to ensure that the model can adapt to complex and changing real-world environments and diverse contamination patterns.

[0144] S72. Use the training set to train the pre-trained model based on YOLOv5 to obtain model parameters.

[0145] As an illustration, the above training set was used to iteratively train a pre-trained model based on YOLOv5. Leveraging YOLOv5's lightweight network structure and efficient multi-scale feature extraction capabilities, the model continuously adjusts network weight parameters through forward and backpropagation. During training, a composite objective function, including a cross-entropy loss function, a localization loss, and a confidence loss, was employed to jointly optimize the position, size, category, and presence probability of the object bounding box. Model weights were initialized using pre-trained parameters from a large-scale general object detection dataset to accelerate convergence and enhance initial detection capabilities. Hyperparameters such as the learning rate, batch size, and regularization parameter were adjusted during training to control training stability and prevent overfitting.

[0146] S73. Calculate the model parameters with the best performance through the loss function to form a stain detection model.

[0147] Model performance during training was evaluated using loss function metrics, and the optimal model parameters were selected as weights for the final stain detection model. The model's performance on the validation set met pre-determined metrics, including high accuracy, recall, and mean average precision (mAP), ensuring high accuracy and reliability in identifying contaminated areas on lenses in practical applications. After training, the model can be exported to a deployable format to support the real-time inference and detection needs of drone embedded devices.

[0148] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0149] Based on the same inventive concept, the embodiments of the present application also provide a deep learning-based drone lens stain recognition and cleaning system for implementing the deep learning-based drone lens stain recognition and cleaning method mentioned above. The implementation solution provided by this system is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more of the following embodiments of the deep learning-based drone lens stain recognition and cleaning system can be found in the above-mentioned limitations of the deep learning-based drone lens stain recognition and cleaning method, and will not be repeated here.

[0150] In an exemplary embodiment, Figure 4 As shown, a deep learning-based drone lens stain recognition and cleaning system is provided, including:

[0151] The image data acquisition module 401 is used to generate image acquisition instructions according to a set sampling period to obtain an image to be detected.

[0152] The image preprocessing module 402 is used to preprocess the image to be detected to obtain a standard image to be detected.

[0153] The stain recognition module 403 is used to input the standard image to be detected into the stain detection model to obtain a stain recognition result.

[0154] The cleaning module 404 is configured to generate cleaning parameters and cleaning instructions based on the stain identification result.

[0155] In one embodiment, it further includes:

[0156] A feature extraction module is used to extract multi-scale features from the standard image to be detected to obtain a multi-scale feature map;

[0157] The prediction module is used to predict candidate bounding boxes based on grid cells to obtain a set of bounding boxes;

[0158] The filtering module is used to filter multiple candidate bounding boxes according to the confidence scores based on the non-maximum suppression algorithm to obtain a set of contaminated areas;

[0159] The encapsulation module is used to encapsulate the contaminated area set in a structured form to obtain the stain recognition result.

[0160] In one embodiment, it further includes:

[0161] The sub-target extraction module is used to extract features layer by layer from the standard image to be detected and obtain an intermediate feature map;

[0162] The channel enhancement module is used to perform channel attention enhancement on the intermediate feature map to obtain a channel enhanced feature map;

[0163] The spatial enhancement module is used to perform spatial attention enhancement on the channel enhancement feature map to obtain a multi-scale feature map.

[0164] In one embodiment, the prediction module is further configured to predict a plurality of candidate bounding boxes for each grid unit to obtain a set of candidate bounding box regression parameters;

[0165] The prediction module is further used to activate the candidate bounding box regression parameter set using an activation function and decode to obtain a bounding box set; the bounding box set includes the bounding box center coordinates and bounding box size data.

[0166] In one embodiment, it further includes:

[0167] The stain weight allocation module is used to calculate the heating and cleaning demand weight for each stain target based on the stain identification results to obtain the overall pollution intensity weight;

[0168] The parameter generation module is used to obtain the heating time and heating power according to the overall pollution intensity weight based on the dynamic cleaning strategy.

[0169] The cleaning module 404 is further configured to obtain cleaning parameters according to the heating time and the heating power.

[0170] In one embodiment, it further includes:

[0171] A feedback module is used to generate a verification image acquisition instruction to obtain a cleaning completed image;

[0172] A comparison module is used to perform difference detection on the image to be detected and the corresponding cleaned image according to a preset time window length to obtain a joint loss function;

[0173] The optimization module is used to perform incremental training on the stain detection model and optimize the model parameters based on the joint loss function.

[0174] In one embodiment, it further includes:

[0175] The training set module is used to obtain normal lens image samples, lens contamination image samples and their corresponding contamination area annotations, and perform data enhancement to obtain the training set;

[0176] The training module is used to train the YOLOv5-based pre-training model using the training set to obtain model parameters;

[0177] The model building module is used to calculate the model parameters with the best performance to form a stain detection model through loss function calculation.

[0178] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0179] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0180] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0181] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A method for identifying and cleaning stains on drone lenses based on deep learning, characterized in that: The method comprises: Generate an image acquisition instruction according to a set sampling period to obtain an image to be detected; the image acquisition instruction is used to instruct the drone camera to perform an image acquisition operation and provide feedback in the form of an image to be detected; Preprocessing the image to be detected to obtain a standard image to be detected; Inputting the standard image to be detected into a stain detection model to obtain a stain recognition result; the stain recognition result includes the center coordinates of the contaminated area, bounding box data, confidence score and stain type label; Cleaning parameters and cleaning instructions are generated according to the stain identification results; the cleaning instructions are used to instruct the cleaning equipment to clean the drone lens according to the cleaning parameters.

2. The method according to claim 1, characterized in that The stain detection model obtains the stain recognition result by the following method: Performing multi-scale feature extraction on the standard image to be detected to obtain a multi-scale feature map; the multi-scale feature map includes a plurality of detection candidate grid units; Predicting candidate bounding boxes according to the grid units to obtain a bounding box set; the bounding box set includes a plurality of candidate bounding boxes and their corresponding bounding box data and confidence scores; Filtering the plurality of candidate bounding boxes according to the confidence scores based on a non-maximum suppression algorithm to obtain a set of contaminated areas; the set of contaminated areas includes the center coordinates of the contaminated areas, the size of the contaminated area bounding boxes, the stain type labels, the confidence scores, and the total number of detected stain targets; The contaminated area set is encapsulated in a structured form to obtain the contamination recognition result.

3. The method according to claim 2, characterized in that The step of performing multi-scale feature extraction on the standard image to be detected to obtain a multi-scale feature map includes: Performing layer-by-layer feature extraction on the standard image to be detected to obtain an intermediate feature map; the intermediate feature map includes a small target feature map, a medium target feature map, and a large target feature map; Performing channel attention enhancement on the intermediate feature map to obtain a channel enhanced feature map; Performing spatial attention enhancement on the channel enhanced feature map to obtain the multi-scale feature map; The channel enhancement feature map is obtained by the following formula: Among them, P l is the intermediate feature map, P l ∈R C×H×W ; l is the target feature map, l∈(1,2,3); is the average pooling feature; is the maximum pooling feature; M c is the MLP shared fusion feature; σ is the activation function; P' l Enhance the feature map for the channel; The multi-scale feature map is obtained by the following formula: in, is a multi-scale feature map; is the spatial average pooling feature; is the spatial maximum pooling feature; M s Concatenated convolutional features; It is the channel enhanced feature map after feature enhancement.

4. The method according to claim 2, characterized in that The step of predicting candidate bounding boxes according to the grid units to obtain a bounding box set includes: Predicting multiple candidate bounding boxes for each of the grid cells to obtain a set of candidate bounding box regression parameters; The candidate bounding box regression parameter set is activated using an activation function, and decoded to obtain the bounding box set; the bounding box set includes bounding box center coordinates and bounding box size data.

5. The method according to claim 1, wherein Generating cleaning parameters according to the stain identification result includes: Calculating the heating and cleaning demand weight for each stain target according to the stain identification result to obtain the overall pollution intensity weight; Based on the dynamic cleaning strategy, the heating time and heating power are obtained according to the overall pollution intensity weight; Obtaining the cleaning parameters according to the heating time and heating power; The heating and cleaning demand weight is obtained by the following formula: Q k =λ1·(w k ·h k )+λ2·s k Among them, Q k is the heating cleaning demand weight of a single stain target; λ1 and λ2 are weight coefficients; w k is the width of the bounding box of the contaminated area; h k is the height of the bounding box of the contaminated area; s k is the confidence score; k is the stain target index; N is the total number of stain targets; Q total is the overall pollution intensity weight; The heating time and heating power are obtained by the following formula: T=T0+γ·Q total P=P0+η·normalize(Q total ) Among them, T is the heating time; T0 is the basic heating time; γ is the time adjustment coefficient; P is the heating power; P0 is the heater starting power; η is the power adjustment coefficient.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Generate a verification image acquisition instruction to obtain a cleaning completion image; the verification image acquisition instruction is used to instruct the drone camera to perform an image acquisition operation and provide feedback in the form of a cleaning completion image; Performing difference detection on the image to be detected and the corresponding cleaned image according to a preset time window length to obtain a joint loss function; Performing incremental training on the stain detection model according to the joint loss function to optimize model parameters; The joint loss function is obtained by the following formula: L total =L basic +β·L pair Among them, L pair is the joint loss function; L basic is the basic loss function of the model; L pair is the difference sensitivity loss function; β is the adjustment coefficient; N is the number of images; i is the image number index; is the image to be detected; is the cleaned image; f is the feature extraction function; ||·|| 2 is the Euclidean distance.

7. The method according to claim 6, characterized in that The stain detection model is obtained by the following method: Obtaining normal lens image samples, lens contamination image samples, and their corresponding contamination area annotations, and performing data augmentation to obtain a training set; the contamination area annotations include bounding box annotations and stain type annotations; Using the training set to train a pre-trained model based on YOLOv5 to obtain model parameters; The model parameters with the best performance are used to form the stain detection model through loss function calculation.

8. A deep learning-based drone lens stain recognition and cleaning system, characterized in that: The system comprises: The image data acquisition module is used to generate image acquisition instructions according to the set sampling period to obtain the image to be detected; An image preprocessing module is used to preprocess the image to be detected to obtain a standard image to be detected; A stain recognition module, configured to input the standard image to be detected into a stain detection model to obtain a stain recognition result; The cleaning module is used to generate cleaning parameters and cleaning instructions according to the stain identification result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images