A machine vision-based unmanned aerial vehicle intelligent inspection system and method

Abstract

The application discloses a kind of unmanned aerial vehicle intelligent inspection system and method based on machine vision, it is related to the technical field of inspection.By introducing dynamic analysis mechanism to the attention distribution of visual recognition model in continuous inspection image, it can actively identify the target area that has not been fully focused by the model for a long time in the repeated structure and low change scene, and further combines local image change stability analysis and disturbance response verification, accurately distinguishes the true non-abnormal area from the visual fatigue area caused by model recognition inertia, so as to avoid the overconfidence and abnormal detection problem caused by the similar recognition result of the model in the prior art for a long time;At the same time, by triggering the corresponding reactivation inspection strategy for the recognition blunting area, potential risk areas can still be confirmed or reviewed in key without relying on model error or manual experience, effectively solving the technical problems that visual fatigue area is difficult to detect and timely expose.

Description

Technical Field

[0001] This invention relates to the field of inspection technology, specifically to a machine vision-based unmanned aerial vehicle (UAV) intelligent inspection system and method. Background Technology

[0002] In modern inspection scenarios, machine vision-based drone-based intelligent inspection has become an important alternative to traditional manual inspection. This method typically uses drones equipped with high-definition cameras or infrared / multispectral visual sensors to autonomously fly along a pre-set path, acquiring high-precision images of target areas or facilities. The acquired images are then input into a machine vision analysis system, undergoing image preprocessing, target detection, defect identification, and location annotation to automatically identify equipment status and analyze anomalies, ultimately generating a structured inspection report. This process not only significantly improves inspection efficiency and operational safety but also makes the detection of equipment anomalies more accurate and timely.

[0003] However, with the widespread deployment of this technology in large-scale scenarios such as power, transportation, and industrial facilities, some hidden problems that were not easily detected in early applications have gradually emerged. One particularly critical issue is the so-called "visual fatigue zone" effect: when faced with a large number of structurally repetitive areas with minimal image changes, visual models often become inert in recognizing these areas due to consistently outputting similar results. This can lead to the model ignoring or misjudging even genuine defects as normal. This problem does not trigger system errors and is difficult to detect during routine reviews, but it can mask potential faults in critical equipment over a long period, posing a systemic safety hazard. Therefore, it is urgent to conduct in-depth research and propose effective identification and response mechanisms. Summary of the Invention

[0004] The purpose of this invention is to solve the problems mentioned in the background art, and to propose a machine vision-based intelligent inspection system and method for unmanned aerial vehicles.

[0005] A first aspect of this invention provides a machine vision-based intelligent inspection method for unmanned aerial vehicles (UAVs), the method comprising: S1: Acquire the image sequence continuously collected by the UAV during the inspection process, identify the target area that has not received effective attention for a long time in multiple consecutive images based on the image sequence, and mark the target area as fatigue candidate area; S2: Based on fatigue candidate regions, extract their corresponding local image content in the image sequence, and select regions whose change amplitude has been stable for a long time as visually inert regions. S3: Introduce slight image perturbation to the visually inert region and re-perform visual recognition analysis. Based on the changes in recognition results before and after the perturbation, identify the regions where recognition sensitivity decreases as recognition passivation regions. S4: Based on the identified passivated area, trigger the reactivation inspection strategy for the corresponding target area, perform at least one re-shooting, attention reset, or manual review operation on the identified passivated area, and output the reactivation result to the inspection result.

[0006] Optionally, the step of identifying target regions that have not received effective attention for a long time in multiple consecutive images based on the image sequence, and marking the target regions as fatigue candidate regions, is as follows: During the inspection process, drones collect continuous image sequences along a preset path. The image sequences are then input into a visual recognition model that includes an attention mechanism. The model then extracts the attention heatmap generated by the model for the target area in each frame of the image. Based on the attention heatmap of the target region in each image frame, the distribution of attention weights at the same spatial location in the image sequence is statistically analyzed. If the attention weight of a certain region in consecutive images is lower than a set threshold and remains lower than a set number of frames, the region is identified as a region that has not received effective attention for a long time. The identified target areas that have not received effective attention for a long time are marked as fatigue candidate areas.

[0007] Optionally, the steps for selecting regions whose changes have remained stable over a long period as visually inert regions are as follows: The spatial location of fatigue candidate regions in the image sequence is located, and local image patches at the corresponding locations are extracted in each frame of the image to form a local image sequence; The difference between corresponding image patches in two adjacent frames of a local image sequence is calculated to obtain an image difference value sequence. The calculation of the image difference values ​​includes: For any pair of image blocks in adjacent frames, the image difference value is defined as the average absolute difference of the gray values ​​of the two image blocks; All difference values ​​are combined into an image difference value sequence; the standard deviation of the image difference value sequence within a set frame window is calculated. If the standard deviation is consistently less than a set stability threshold and all difference values ​​are lower than a set change threshold, then the fatigue candidate region is determined to be a region with a long-term stable change amplitude in the image sequence and is marked as a visually inert region.

[0008] Optionally, the step of determining the region where recognition sensitivity decreases as the recognition passivation region based on the changes in recognition results before and after the perturbation is as follows: For the previously identified visually inert regions, select the corresponding local image blocks in the image sequence, and construct at least one perturbation image in sequence. The perturbation image is the image obtained after introducing a slight intervention to the original image block. The intervention includes at least one: pixel grayscale fine adjustment, brightness enhancement, slight image rotation or contrast fine adjustment. The original image patch and the corresponding perturbation image are respectively input into the same visual recognition model to obtain the recognition result information before and after the perturbation. The sensitivity decrease index is calculated based on the recognition result information before and after the perturbation, and the region where the recognition sensitivity decreases is defined as the recognition passivation region based on the sensitivity decrease index and the preset threshold.

[0009] Optionally, the step of calculating the sensitivity decrease index based on the recognition results before and after the perturbation is as follows: The local feature consistency decay index and the multi-scale instability index are calculated based on the identification results before and after the perturbation. The sensitivity decrease index is obtained by subtracting the local feature consistency decay index from the multi-scale instability index.

[0010] Optionally, the calculation steps for the local feature consistency decay index are as follows: For the identified visually inert regions, extract the corresponding set of local feature points from the original image and the perturbed image; For each extracted feature point, its position coordinates and intensity value in the original image and the perturbed image are recorded respectively. The position coordinates include the pixel coordinates in the image, and the intensity value represents the local image intensity or gradient value of the feature point. Calculate the displacement of each feature point in the images before and after the disturbance. The displacement is the difference in position of the feature point in the images before and after the disturbance. Calculate the intensity change of each feature point before and after the perturbation. The intensity change is the absolute value of the difference between the intensity values ​​of the feature point in the image before and after the perturbation. The local feature consistency attenuation index is obtained by summing the absolute values ​​of the displacements and intensity changes of all feature points and dividing the sum by the total number of feature points.

[0011] Optionally, the calculation steps for the multi-scale instability index are as follows: The original image and the perturbed image are decomposed into multiple image layers of different resolutions, each representing image information at a different scale; the frequency components of each scale image are obtained through Fourier transform. Calculate the frequency component differences between the original image and the perturbed image at each scale; the difference represents the magnitude of the frequency change. Calculate the amplitude standard deviation of frequency variation at each scale, sum the positive square roots of the standard deviations at each scale, and divide the sum by the total number of scales to obtain the multiscale instability value for each image. The multiscale instability values ​​obtained by the synthesis are normalized to the range of 0 to 1 to obtain the corresponding multiscale instability index.

[0012] Optionally, the step of identifying regions with decreased sensitivity as passivation regions based on a sensitivity decrease index and a preset threshold is as follows: The sensitivity decrease index is compared with a preset threshold. If the sensitivity decrease index is not less than the preset threshold, the area where the sensitivity decreases is identified as the passivation area. If the sensitivity decrease index is less than the preset threshold, the area of ​​decreased sensitivity will not be identified as the passivation area.

[0013] A second aspect of this invention provides a machine vision-based intelligent inspection system for unmanned aerial vehicles (UAVs), the system comprising: Fatigue candidate module: acquires the image sequence continuously collected by the UAV during the inspection process, identifies the target area that has not received effective attention for a long time in multiple consecutive images based on the image sequence, and marks the target area as fatigue candidate area; Visual inertia module: Based on fatigue candidate regions, extract their corresponding local image content in the image sequence, and select regions whose change amplitude has been stable for a long time as visual inert regions. Passivation module: Introduces slight image perturbation into visually inert regions and re-executes visual recognition analysis. Based on the changes in recognition results before and after the perturbation, regions where recognition sensitivity decreases are identified as recognition passivation regions. Inspection module: Based on the identified passivated area, trigger the reactivation inspection strategy for the corresponding target area, perform at least one re-shooting, attention reset or manual review operation on the identified passivated area, and output the reactivation result to the inspection result.

[0014] The beneficial effects of this invention are: This invention proposes a machine vision-based intelligent inspection system and method for unmanned aerial vehicles (UAVs). By introducing a dynamic analysis mechanism for the attention distribution of the visual recognition model in continuous inspection images, it can proactively identify target areas that have not been sufficiently focused on by the model for a long time in repetitive structures and low-change scenes. Furthermore, by combining local image change stability analysis and disturbance response verification, it can accurately distinguish between truly anomaly-free areas and visual fatigue areas caused by model recognition inertia. This avoids the overconfidence and anomaly omission problems caused by the model outputting similar recognition results for a long time in existing technologies. At the same time, by triggering a targeted reactivation inspection strategy for recognition dulled areas, it is possible to reconfirm or focus on reviewing potential risk areas without relying on model errors or human experience. This effectively solves the technical problem of visual fatigue areas being difficult to detect and expose in a timely manner, and improves the reliability and safety of intelligent UAV inspection in large-scale, homogeneous scenarios. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a machine vision-based intelligent inspection method for unmanned aerial vehicles (UAVs) provided in an embodiment of the present invention; Figure 2This is a framework diagram of a machine vision-based unmanned aerial vehicle (UAV) intelligent inspection system provided in an embodiment of the present invention. Detailed Implementation

[0016] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on the specific implementation methods, structures, features and effects of the present invention.

[0017] This invention provides a machine vision-based intelligent inspection method for unmanned aerial vehicles (UAVs). See also... Figure 1 , Figure 1 A flowchart illustrating a machine vision-based intelligent inspection method for unmanned aerial vehicles (UAVs) is provided as an embodiment of the present invention. The method includes the following steps: S1: Acquire the image sequence continuously collected by the UAV during the inspection process, identify the target area that has not received effective attention for a long time in multiple consecutive images based on the image sequence, and mark the target area as fatigue candidate area; S2: Based on the fatigue candidate region, extract its corresponding local image content in the image sequence, analyze the changes of the local image content in continuous images, and select the region whose change amplitude is stable in a long-term state as the visual inert region. S3: Introduce slight image perturbation to the visually inert region and re-perform visual recognition analysis. Based on the changes in recognition results before and after the perturbation, identify the regions where recognition sensitivity decreases as recognition passivation regions. S4: Based on the identified passivated area, trigger the reactivation inspection strategy for the corresponding target area, perform at least one re-shooting, attention reset or manual review operation on the identified passivated area, and output the reactivation result to the inspection result to avoid abnormal missed detection caused by visual fatigue area.

[0018] The machine vision-based intelligent inspection method for drones provided in this invention introduces a dynamic analysis mechanism for the attention distribution of the visual recognition model in continuous inspection images. This mechanism can proactively identify target areas that have not been sufficiently noticed by the model in repetitive structures and low-change scenarios. Furthermore, by combining local image change stability analysis and disturbance response verification, it accurately distinguishes between truly anomaly-free areas and visually fatigued areas caused by model recognition inertia. This avoids the overconfidence and anomaly omission problems caused by the model outputting similar recognition results over a long period of time in existing technologies. At the same time, by triggering a targeted reactivation inspection strategy for recognition dulled areas, it is possible to reconfirm or focus on reviewing potential risk areas without relying on model errors or human experience. This effectively solves the technical problem of visually fatigued areas being difficult to detect and expose in a timely manner, and improves the reliability and safety of intelligent drone inspection in large-scale, homogeneous scenarios.

[0019] In one embodiment, S1: The step of acquiring a sequence of images continuously collected by the UAV during the inspection process, identifying target regions that have not received effective attention for a long time in multiple consecutive images based on the image sequence, and marking the target regions as fatigue candidate regions is as follows: During the inspection process, drones collect continuous image sequences along a preset path. The image sequences are then input into a visual recognition model that includes an attention mechanism. The model then extracts the attention heatmap generated by the model for the target area in each frame of the image. Based on the attention heatmap of the target region in each image frame, the distribution of attention weights at the same spatial location in the image sequence is statistically analyzed. If the attention weight of a certain region in consecutive images is lower than a set threshold and remains lower than a set number of frames, then the region is identified as a region that has not received effective attention for a long time. The identified target regions that have not received effective attention for a long time are marked as fatigue candidate regions, and their image positions, the number of frames continuously ignored, and the corresponding attention weight change trends are recorded for use in visual inertia identification and perturbation verification in subsequent steps.

[0020] It's important to note that "visual recognition models incorporating attention mechanisms" refer to deep neural network structures that introduce attention modules (such as self-attention or channel attention) to automatically determine which regions in an image are most important for the current recognition task and assign higher attention weights to these regions. Common models include YOLO, SwinTransformer, and SE-ResNet, all with attention mechanisms. When processing drone inspection images, these models generate an "attention heatmap" within each image, marking the distribution of model attention intensity in a two-dimensional space. Higher colors or values ​​indicate stronger model attention to that location. Specifically, when a drone continuously captures images of a target area along a preset flight path, the model extracts the corresponding attention heatmaps for these images and statistically analyzes the attention level of a fixed spatial location (e.g., a section of a photovoltaic panel or a component of a tower) across multiple frames. If a region consistently shows low attention across multiple frames, it can be preliminarily determined that the region has not been carefully analyzed by the model, potentially indicating recognition inertia or omission. For example, in a row of identical photovoltaic modules, the model might skip the analysis of the central area due to "familiarity," focusing only on the edges or shaded areas, thus overlooking minor damage or cracks in the middle. Therefore, identifying such long-term low-attention areas as "fatigue candidate regions" and recording their image coordinates, the number of ignored frames, and weight change trends can provide a reliable basis for subsequent analysis of whether visual inertia or recognition dulling exists in these areas. Through this process, without relying on manual judgment or explicit error reporting mechanisms, the potential "blind spots" of the model can be proactively discovered, providing a structured guarantee for improving inspection coverage and recognition accuracy.

[0021] In one embodiment, S2: Based on the fatigue candidate region, extract its corresponding local image content in the image sequence, analyze the changes of the local image content in continuous images, and select regions whose changes remain stable over a long period as visually inert regions. The spatial location of fatigue candidate regions in the image sequence is located, and local image patches at the corresponding locations are extracted in each frame of the image to form a local image sequence; The difference between corresponding image patches in two adjacent frames of a local image sequence is calculated to obtain an image difference value sequence. The formula for calculating the image difference value is as follows: ;I t (x,y) represents the pixel grayscale value located at coordinates (x,y) in the image block of frame t; t ₊1(x,y) represents the pixel grayscale value at coordinates (x,y) in the (t+1)th frame image patch; M and N represent the number of rows and columns of the image patch, respectively; D t This represents the average pixel difference between the image blocks in frame t and frame (t+1).

[0022] All difference values ​​D t The sequence of image difference values ​​{D1, D2, ..., D} k}, where k is the number of image frames in the image sequence minus one; Calculate the standard deviation of the image difference value sequence within a set frame window. If the standard deviation is consistently less than a set stability threshold and all difference values ​​are lower than a set change threshold, then the fatigue candidate region is determined to be a region with long-term stable change amplitude in the image sequence. It is marked as a visually inert region, and its image position and difference stability index are output for subsequent disturbance response verification steps.

[0023] It should be noted that after extracting the fatigue candidate region, its spatial location in the image sequence is determined, and corresponding local image patches are cropped from each frame of the image to form a set of image sequences for that region. Next, for any pair of adjacent frames in the sequence, denoted as the image patches of frame t and frame (t+1), the absolute value of the difference in grayscale values ​​is calculated by traversing each pixel position, and the average difference value D between the two image patches is obtained by averaging all pixel differences. t This operation reflects the degree of change in the current region over time; if the change is drastic, the value is large; if there is little change, the value is close to zero. The difference values ​​of all adjacent frame pairs are arranged in chronological order to form a complete difference value sequence {D1, D2, ..., D...}. k}, where k is the total number of image frames minus one. To further determine whether the changes in this region have long-term stability, the standard deviation σ of the difference sequence within a set sliding frame window (e.g., 5 or 10 consecutive frames) is calculated to measure the degree of fluctuation in the magnitude of the change; if the standard deviation is consistently less than a set threshold T, s And all difference values ​​D within the corresponding time window tIf all values ​​are below the absolute change threshold T_d, it can be determined that the region not only exhibits minimal change in the current sequence but also shows a stable trend, without any short-term abrupt changes or interference. This region is then formally marked as a visually inert region. Taking photovoltaic panel inspection as an example, if the average grayscale difference of a central component in 30 frames of images is consistently below 1, and the standard deviation does not exceed 0.3, it indicates that the image seen by the model shows almost no change. This type of region is precisely where "visual fatigue" most easily hides defects. This method allows for comprehensive verification of whether the visual state of the region has entered inertia from multiple time dimensions and pixel levels. It ensures that subsequent perturbation tests and reactivation strategies target regions that truly have a risk of recognition passivation, rather than areas with normal fluctuations or those temporarily overlooked. This design not only improves the accuracy of anomaly identification during inspection but also avoids misoperation or resource waste in normal areas, enhancing the robustness and discriminative ability of the entire intelligent inspection process.

[0024] In one embodiment, S3: Introducing a slight image perturbation into the visually inert region and re-performing the visual recognition analysis, and determining the region with decreased recognition sensitivity as the recognition passivation region based on the change in recognition results before and after the perturbation, is as follows: For the previously identified visually inert regions, select the corresponding local image blocks in the image sequence, and construct at least one perturbation image in sequence. The perturbation image is the image obtained after introducing a slight intervention to the original image block. The intervention includes at least one: pixel grayscale fine adjustment, brightness enhancement, slight image rotation or contrast fine adjustment. The original image patch and the corresponding perturbation image are respectively input into the same visual recognition model to obtain the recognition result information before and after the perturbation. The sensitivity decrease index is calculated based on the recognition result information before and after the perturbation, and the region where the recognition sensitivity decreases is defined as the recognition passivation region based on the sensitivity decrease index and the preset threshold.

[0025] It should be noted that the same visual recognition model refers to the model used in step S1, which is typically a deep neural network model capable of processing the input image and outputting recognition results. Common visual recognition models include convolutional neural networks (CNNs), deep convolutional neural networks (ResNet, VGG), and Transformer-based models (such as SwinTransformer). These models can automatically extract and learn features based on the input image features and generate prediction results.

[0026] There are various forms of perturbation intervention, the purpose of which is to simulate possible changes in image quality or environmental disturbances in reality, thereby testing the adaptability of visual recognition models to these changes. These intervention methods include: Pixel grayscale fine-tuning: This simulates changes in lighting or noise interference by slightly adjusting the grayscale value of each pixel. This allows you to test the model's performance under grayscale changes; for example, if a part of the original image is white, adjust it to be slightly gray.

[0027] Brightness enhancement: This simulates changes in ambient lighting by increasing or decreasing the overall brightness of the image. This method can be used to test the stability of a model under different lighting conditions.

[0028] Slight image rotation: Rotating the image by a small angle (e.g., 1-5 degrees) simulates the effect of viewpoint deviation or object movement on the image. This can test the model's ability to adapt to changes in spatial position.

[0029] Contrast fine-tuning: This modifies the contrast of an image, either enhancing or reducing the difference between light and dark areas. This operation simulates changes in light and shadow in a scene, examining the model's robustness under different contrast conditions.

[0030] In this step, firstly, corresponding local image patches are selected for the visually inert regions, and perturbation images are constructed sequentially. Next, the original image patches and each perturbation image are input into the same visual recognition model for recognition, yielding recognition results before and after perturbation. These results typically include the recognition category (e.g., "normal" or "defective") and confidence level (the model's confidence in the classification result). By comparing the recognition results before and after perturbation, a sensitivity reduction index can be calculated. This index quantifies the impact of the changes introduced by the perturbation on the model's recognition results.

[0031] This step accurately detects areas where the model's recognition ability weakens under minor disturbances or environmental changes, thus avoiding long-term neglect of potential problems in critical equipment or areas. This method avoids direct reliance on manual review or simple image changes; instead, it accurately identifies areas where recognition may be weakened through disturbance introduction and model self-evaluation, significantly improving the reliability and efficiency of inspections.

[0032] In one implementation, the step of calculating the sensitivity decrease index based on the recognition results before and after the perturbation is as follows: The local feature consistency decay index and the multi-scale instability index are calculated based on the identification results before and after the perturbation. The sensitivity decrease index is obtained by subtracting the local feature consistency decay index from the multi-scale instability index.

[0033] In one implementation, the calculation steps for the local feature consistency decay index are as follows: For the identified visually inert regions, the corresponding set of local feature points is extracted from the original image and the perturbed image. The local feature points are the edge, corner, or texture feature points in the image. The extracted feature points can be obtained by methods such as Harris corner detection and SIFT feature point detection. For each extracted feature point, its position coordinates and intensity value in the original image and the perturbed image are recorded respectively. The position coordinates include the pixel coordinates in the image, and the intensity value represents the local image intensity or gradient value of the feature point. Calculate the displacement of each feature point in the images before and after the perturbation. The displacement is the difference in position of the feature point in the images before and after the perturbation, which is represented as the composite value of the coordinate changes of the feature point in the horizontal and vertical directions. Calculate the intensity change of each feature point before and after the perturbation. The intensity change is the absolute value of the difference between the intensity values ​​of the feature point in the image before and after the perturbation. The local feature consistency decay index is obtained by summing the absolute values ​​of the displacement and intensity changes of all feature points and dividing the sum by the total number of feature points. The index represents a comprehensive measure of the displacement and intensity changes of all feature points. The larger the value of the local feature consistency decay index, the higher the degree of consistency decay of local features after the image is disturbed.

[0034] The local feature consistency decay index value is between 0 and 1, where 0 represents no decay and 1 represents complete decay. The final decay index is used to further determine the sensitivity change of the image after perturbation.

[0035] It should be noted that the data acquisition methods involved in calculating the local feature consistency decay index are as follows: First, local feature points (such as edges, corners, and texture feature points) in the original and perturbed images are obtained through feature extraction algorithms. Commonly used feature extraction methods include Harris corner detection and SIFT feature point detection. These methods can extract the most representative feature points of the image content, usually the edges, corners, or textured parts of the image, and have high local stability. Next, for each extracted feature point, its position coordinates and intensity value in the image need to be recorded. This information can be obtained through image pixel coordinates, which are usually the (x, y) positions of points in the image. The intensity value can be represented by the image's grayscale value or gradient value. The gradient value is the intensity of the most drastically changing area in the image, reflecting the texture or edge features in the image. The position and intensity value of the feature point are obtained by calculating the image's pixel values ​​and local gradients. Specifically, methods such as the Sobel operator and the Laplacian operator can be used to extract gradient information. Finally, the displacement and intensity change of each feature point are calculated. The displacement is obtained by comparing the coordinate changes of the feature points before and after the perturbation, while the intensity change is calculated by comparing the intensity differences of the feature points before and after the perturbation. These steps combined enable the final local feature consistency decay index to quantify the degree of image change after perturbation, further providing a basis for sensitivity change analysis.

[0036] The Local Feature Consistency Decay Index (LAND) measures the degree of decrease in consistency of local features (such as edges, corners, and textures) in an image after image perturbation (e.g., changes in illumination, rotation, noise). Specifically, it assesses the changes in the position and intensity of key feature points in an image before and after the perturbation; that is, whether feature points retain their original characteristics (e.g., relative position and intensity) in the perturbed image. A higher LAND indicates reduced stability of these key features (such as texture or edges), meaning the perturbation has a greater impact on the image, causing significant changes in the position and intensity of previously stable feature points. This indicates a lower sensitivity of the model to these changes. This situation implies a weaker ability to adapt to perturbations, potentially leading to misidentification or missed identification. For example, suppose an image of a building is rotated. If the edge or corner features in the image show almost no change before and after the perturbation, with minimal changes in displacement and intensity, the model is highly sensitive to such changes and can accurately identify features under perturbation; the LAND will then have a lower LAND. If, after rotation, the corner points or edge features in the image undergo significant displacement or intensity changes, it indicates that the model is insensitive to this perturbation and cannot effectively identify these changes, resulting in a high local feature consistency decay index. Therefore, a larger local feature consistency decay index indicates a more severe decrease in image feature consistency, and poorer model recognition stability and robustness. This typically means that in practical applications, the model may miss some important image features or misclassify them as other content, leading to inaccurate or unstable inspection results.

[0037] It's important to note that the advantage of calculating the local feature consistency decay index using the above method is its comprehensive assessment of changes in local features after perturbation. It considers not only pixel-level differences or overall image changes, but focuses on the stability of key feature points within the image. By calculating changes in the position and intensity of feature points, this method more accurately captures the specific impact of perturbations on image features, especially when dealing with complex textures or subtle changes, making it more representative than simple pixel differences. Compared to traditional calculation methods, pixel-level differences or global image changes often fail to effectively reflect changes in local details because they ignore the importance and stability of feature points, which typically represent the most critical structures and information in an image. This approach allows for a finer-grained evaluation of an image's robustness to perturbations, particularly regarding the sensitivity to changes in high-frequency features such as edges and corners. It avoids the limitations of relying solely on overall image differences and better reflects the stability of the visual model when handling perturbations. This method is particularly suitable for refined equipment monitoring and inspection tasks, effectively distinguishing feature decay under slight changes, thus more accurately identifying areas of decreased recognition capability and enhancing the model's practical application capabilities and reliability.

[0038] The advantage of calculating the local feature consistency decay index using the above method is that it comprehensively evaluates the changes in local features of an image after perturbation, focusing not only on pixel-level differences or overall image changes, but also on the stability of key feature points in the image. By calculating the positional and intensity changes of feature points, this method can more accurately capture the specific impact of perturbations on image features, especially when dealing with complex textures or subtle changes, making it more representative than simple pixel differences. Compared to traditional calculation methods, pixel-level differences or global image changes often fail to effectively reflect local detail changes in an image because they ignore the importance and stability of feature points, which typically represent the most critical structures and information in the image. This approach allows for a finer-grained evaluation of an image's robustness to perturbations, particularly regarding the sensitivity to changes in high-frequency features such as edges and corners, avoiding the limitations of relying solely on overall image differences and better reflecting the stability of the visual model when handling perturbations. This method is particularly suitable for refined equipment monitoring and inspection tasks, effectively distinguishing feature decay under slight changes, thus more accurately identifying areas of decreased recognition capability and enhancing the model's practical application capabilities and reliability.

[0039] In one implementation, the calculation steps for the multi-scale instability index are as follows: The original image and the perturbed image are decomposed into multiple image layers of different resolutions, each representing image information at a different scale; the frequency components of each scale image are obtained through Fourier transform. Calculate the frequency component differences between the original image and the perturbed image at each scale. The difference represents the magnitude of the frequency change; the frequency difference reflects the impact of the perturbation on the image.

[0040] Calculate the standard deviation of the frequency variation at each scale. The standard deviation measures the consistency of frequency variation and reflects the degree of fluctuation in the impact of perturbations on the image.

[0041] Calculate the amplitude standard deviation of frequency variation at each scale, sum the positive square roots of the standard deviations at each scale, and divide the sum by the total number of scales to obtain the multiscale instability value for each image. The multiscale instability values ​​obtained by the synthesis are normalized to the range of 0 to 1 to obtain the corresponding multiscale instability index.

[0042] In the calculation of the multi-scale instability index, the data is acquired as follows: First, multi-scale decomposition is performed using the image pyramid method to obtain image layers of different resolutions. The image pyramid generates image layers for each scale by downsampling the original image layer by layer. Each image layer represents a different scale, where low-resolution images capture global information, while high-resolution images retain more detailed information. Next, Fourier transform is used to obtain the frequency components of the image at each scale. By transforming the image from the spatial domain to the frequency domain, the frequency distribution of the image is obtained. Changes in frequency components reflect changes in high-frequency and low-frequency features of the image. The frequency difference is obtained by comparing the frequency components of the image before and after the perturbation, specifically by calculating the difference in frequency components before and after the perturbation to obtain the magnitude of the frequency change. Then, the standard deviation of the frequency change at each scale is calculated. The standard deviation measures the consistency of the perturbation on image stability by calculating the fluctuation amplitude of the frequency change at each scale. Finally, the standard deviation of each scale is processed and synthesized by weighted square root summation to obtain the multi-scale instability index, and then normalized to the range of 0 to 1 for subsequent analysis and comparison. These data are obtained step by step throughout the calculation process, including image pyramid decomposition, Fourier transform, frequency difference calculation, and standard deviation analysis.

[0043] The multiscale instability index is a metric used to measure the degree of stability variation of an image across multiple resolution scales. It assesses an image's sensitivity to perturbations (such as noise, illumination changes, rotation, etc.) by analyzing the changes in frequency components at different scale levels. Specifically, images display different levels of detail at multiple scales; high-resolution images contain more local details, while low-resolution images capture global features. When perturbations are applied to an image, details at different scales are affected in different ways. Low-resolution images may be less affected by the overall structure, while high-resolution images are more sensitive to the impact on details. Therefore, a higher multiscale instability index indicates a more severe decrease in the stability of image details or structure across multiple scale levels, and a greater impact of perturbations on the image. For example, when local details in an image (such as corners, edges, textures, etc.) are almost unaffected at low resolution but undergo significant changes at high resolution (such as blurred edges, corner shifts, etc.), the multiscale instability index will be high, indicating poor stability of the image at high-resolution scales and susceptibility to perturbations. Conversely, if the perturbation has a small impact on the image, especially if the changes are relatively consistent across all scales, the index will be low, indicating that the image is more robust to the perturbation. This method can effectively detect the magnitude of detail changes in an image after perturbation, especially the image's adaptability to subtle perturbations, and help identify which regions or image features are sensitive to perturbations, potentially leading to misclassification or missed detection. Therefore, the magnitude of the multi-scale instability index directly reflects the stability and sensitivity of an image across multiple scales. A larger index indicates poorer image stability and lower adaptability of the image model to perturbations, which may lead to misidentification.

[0044] The advantage of calculating the multi-scale instability index using the above method is that it comprehensively assesses the stability of an image under perturbation from different resolution levels, avoiding the limitations of focusing only on a single scale or simple pixel-level differences. First, multi-scale analysis can reveal changes in the global features and local details of an image at different scales. Low-resolution images primarily capture the overall structure of the image, while high-resolution images can sensitively capture changes in local details. For changes at different scales, it can more comprehensively reflect the impact of perturbation on various levels of the image. Second, by calculating frequency component differences rather than simple pixel differences, it can more accurately identify changes in the image in the frequency domain, especially in terms of details, texture, and structural changes, which are difficult to accurately capture using standard pixel comparison methods. More importantly, by calculating the standard deviation, the fluctuation of the perturbation's impact on image features can be quantified. The standard deviation reflects the stability of the changes, allowing the perturbation's impact on the image at various scales to be quantified and synthesized, thus ensuring a more stable and reliable assessment when facing different types of perturbations. The final normalization process ensures that the multi-scale instability index is within the range of 0 to 1, facilitating integration with other analytical indicators for unified evaluation.

[0045] In one embodiment, the step of identifying regions with decreased sensitivity as passivation regions based on a sensitivity decrease index and a preset threshold is as follows: The sensitivity decrease index is compared with a preset threshold. If the sensitivity decrease index is not less than the preset threshold, the area where the sensitivity decreases is identified as the passivation area. If the sensitivity decrease index is less than the preset threshold, the area of ​​decreased sensitivity will not be identified as the passivation area.

[0046] It's important to note that the sensitivity degradation index is compared to a preset threshold. The sensitivity degradation index quantifies the image's stability decay after a disturbance by measuring the changes in image recognition results before and after the disturbance; that is, the changes in the displacement and intensity of feature points under disturbance. The preset threshold is a value pre-set based on the model's stability requirements, determining whether an image meets the standard for being considered at risk of "recognition passivation." Next, the sensitivity degradation index is compared to the preset threshold. If the sensitivity degradation index is greater than or equal to the preset threshold, it indicates that the region is sufficiently sensitive to disturbances, potentially affecting recognition accuracy. In this case, we classify the region as a recognition passivation region. A recognition passivation region means that the region's stability deteriorates under disturbance, the model's ability to recognize that region decreases, and it may lead to misjudgments or missed judgments. This is similar to a "safety warning," indicating a potential risk of recognition failure in that region. For example, if in an inspection image of a power facility, a cable connection exhibits significant feature changes after disturbance, with the sensitivity degradation index exceeding the set threshold, it means that the features of that part may no longer be reliable for model recognition, leading to recognition passivation. Therefore, for these areas, a reactivation strategy or manual review will be triggered to ensure that no potential problems are missed. If the sensitivity decline index is less than a preset threshold, it indicates that the disturbance in that area has a small impact, and the model still maintains a high recognition ability for that area. Therefore, it does not need to be marked as a recognition passivation area and will be considered to have no recognition passivation risk, and will continue to be processed according to the regular inspection method. The core function of this step is to automatically identify areas where recognition ability may decline by comparing the sensitivity decline index with the preset threshold, thereby providing a basis for subsequent anomaly detection and risk control.

[0047] In one embodiment, S4: Based on the identification of the passivated region, a reactivation inspection strategy is triggered for the corresponding target region. At least one re-encoding, attention reset, or manual review operation is performed on the identified passivated region, and the reactivation result is output to the inspection result. The steps to avoid abnormal missed detections caused by visual fatigue areas are as follows: Once a visually dulled area is identified, the system will automatically trigger a reactivation inspection strategy, which includes performing at least one of the following operations: re-capture, attention reset, or manual review.

[0048] Re-capture operation: The system will re-capture the area where recognition has become dulled, ensuring that the area is re-identified and analyzed in the new image frame, thereby eliminating omissions or misidentifications caused by visual fatigue or recognition dulling. The re-captured image will be processed again by the visual recognition model to generate new recognition results, ensuring the accuracy of the area.

[0049] Attention Reset Operation: By resetting the attention mechanism of the visual recognition model, the model can re-evaluate the dulled regions instead of relying on previous low-attention results. This is achieved by adjusting the parameters of the attention mechanism or retraining the model's attention areas, ensuring the model re-focuses on the dulled regions and avoiding overlooking potential defects in those areas.

[0050] Manual review: For certain complex or high-risk recognition passivation areas, the system will automatically mark the area as an area to be manually reviewed. The human operator will manually check and confirm the area based on the new image or recognition results to ensure the accuracy and completeness of the final recognition results.

[0051] All the above operation results will be output in the inspection results, and the status of the processed areas will be updated in the inspection report to ensure that abnormal omissions caused by visual fatigue areas are avoided, and to ensure the completeness and efficiency of the inspection results.

[0052] It's important to note that upon identifying a visually dulled area, the system automatically triggers a reactivation inspection strategy. This strategy includes three operations: re-capturing, attention reset, and manual review. Each operation corresponds to a different processing method to ensure that abnormal omissions caused by visual fatigue areas are effectively corrected. The capture operation first re-acquires images of the dulled area, ensuring that the area is effectively captured and recognized in the new image frame. After image acquisition, the system compares the new image with the original image, inputs the new image into the same visual recognition model for processing, re-analyzes, and generates updated recognition results. This process eliminates misjudgments or omissions caused by the original image quality, viewing angle issues, or recognition dullness, ensuring that defects in the area are accurately identified. For example, suppose that during the inspection of a photovoltaic power station, damage to a module is not detected in time due to poor lighting. The new image obtained through the re-capturing operation allows the system to re-evaluate the area under better lighting conditions, thus obtaining an accurate recognition result. The attention reset operation resets the attention mechanism of the visual recognition model, forcing the model to re-evaluate the dulled area. Specifically, the system adjusts the weights and focus of each layer in the model, changing the parameters or training strategies of the model's focus areas to ensure it no longer relies on low-attention results previously caused by visual fatigue or long-term neglect. In this way, the system can enhance its sensitivity to identifying dulled areas, preventing previously overlooked potential problems from being ignored again. For example, in inspecting building exteriors, some similar-looking areas may not have received sufficient attention due to visual fatigue. After resetting attention, the model can re-evaluate these areas, ensuring they are not missed. Manual review is primarily for complex or high-risk dulled areas. The system marks these areas as requiring manual review and automatically sends relevant information to human operators for manual inspection based on new images or recognition results. Manual review provides final confirmation, ensuring that potential defects in the image are accurately verified and determined. For instance, in high-risk electrical equipment inspections, the system uses manual review to reconfirm blurry or difficult-to-determine areas to avoid misjudgments or omissions. The results of all these operations are recorded and output to the inspection report. The system updates the status of the processed areas in the inspection results to ensure the completeness and accuracy of the final report and avoid missing defects due to visual fatigue. In this way, the efficiency and reliability of the inspection task are significantly improved. The system can efficiently complete automatic inspections while minimizing the risk of omissions or misjudgments through a reactivation strategy.

[0053] Based on the same inventive concept, this invention also provides a machine vision-based intelligent inspection system for unmanned aerial vehicles (UAVs). See also Figure 2 , Figure 2 A schematic diagram of a machine vision-based unmanned aerial vehicle (UAV) intelligent inspection system provided in an embodiment of the present invention includes: Fatigue candidate module: acquires the image sequence continuously collected by the UAV during the inspection process, identifies the target area that has not received effective attention for a long time in multiple consecutive images based on the image sequence, and marks the target area as fatigue candidate area; Visual inertia module: Based on fatigue candidate regions, extract their corresponding local image content in the image sequence, analyze the changes of the local image content in continuous images, and select regions whose changes have remained stable over a long period of time as visual inert regions. Passivation module: Introduces slight image perturbation into visually inert regions and re-executes visual recognition analysis. Based on the changes in recognition results before and after the perturbation, regions where recognition sensitivity decreases are identified as recognition passivation regions. Inspection module: Based on the identification of the dulled area, the reactivation inspection strategy for the corresponding target area is triggered. At least one re-shooting, attention reset or manual review operation is performed on the identified dulled area, and the reactivation result is output to the inspection result to avoid abnormal missed detection caused by visual fatigue area.

[0054] The machine vision-based intelligent inspection system for drones provided in this invention introduces a dynamic analysis mechanism for the attention distribution of the visual recognition model in continuous inspection images. This mechanism can proactively identify target areas that have not been sufficiently focused on by the model for a long time in repetitive structures and low-change scenarios. Furthermore, by combining local image change stability analysis and disturbance response verification, it can accurately distinguish between truly anomaly-free areas and visually fatigued areas caused by model recognition inertia. This avoids the overconfidence and anomaly omission problems caused by the model outputting similar recognition results for a long time in the prior art. At the same time, by triggering a targeted reactivation inspection strategy for recognition dulled areas, it can still reconfirm or focus on reviewing potential risk areas without relying on model errors or human experience. This effectively solves the technical problem that visually fatigued areas are difficult to detect and expose in a timely manner, and improves the reliability and security of intelligent drone inspection in large-scale, homogeneous scenarios.

[0055] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the technical solution of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A machine vision-based intelligent inspection method for unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: S1: Acquire the image sequence continuously collected by the UAV during the inspection process, identify the target area that has not received effective attention for a long time in multiple consecutive images based on the image sequence, and mark the target area as fatigue candidate area; S2: Based on fatigue candidate regions, extract their corresponding local image content in the image sequence, and select regions whose change amplitude has been stable for a long time as visually inert regions. S3: Introduce slight image perturbation to the visually inert region and re-perform visual recognition analysis. Based on the changes in recognition results before and after the perturbation, identify the regions where recognition sensitivity decreases as recognition passivation regions. S4: Based on the identified passivated area, trigger the reactivation inspection strategy for the corresponding target area, perform at least one re-shooting, attention reset, or manual review operation on the identified passivated area, and output the reactivation result to the inspection result.

2. The machine vision-based intelligent inspection method for unmanned aerial vehicles according to claim 1, characterized in that, The steps for identifying target regions that have not received effective attention for a long time in multiple consecutive images based on image sequences, and marking these target regions as fatigue candidate regions, are as follows: During the inspection process, drones collect continuous image sequences along a preset path. The image sequences are then input into a visual recognition model that includes an attention mechanism. The model then extracts the attention heatmap generated by the model for the target area in each frame of the image. Based on the attention heatmap of the target region in each image frame, the distribution of attention weights at the same spatial location in the image sequence is statistically analyzed. If the attention weight of a certain region in consecutive images is lower than a set threshold and remains lower than a set number of frames, the region is identified as a region that has not received effective attention for a long time. The identified target areas that have not received effective attention for a long time are marked as fatigue candidate areas.

3. The machine vision-based intelligent inspection method for unmanned aerial vehicles according to claim 1, characterized in that, The steps to select regions whose changes have remained stable over a long period as visually inert regions are as follows: The spatial location of fatigue candidate regions in the image sequence is located, and local image patches at the corresponding locations are extracted in each frame of the image to form a local image sequence; The difference between corresponding image patches in two adjacent frames of a local image sequence is calculated to obtain an image difference value sequence. The calculation of the image difference values ​​includes: For any pair of image blocks in adjacent frames, the image difference value is defined as the average absolute difference of the gray values ​​of the two image blocks; All difference values ​​are combined into an image difference value sequence; the standard deviation of the image difference value sequence within a set frame window is calculated. If the standard deviation is consistently less than a set stability threshold and all difference values ​​are lower than a set change threshold, then the fatigue candidate region is determined to be a region with a long-term stable change amplitude in the image sequence and is marked as a visually inert region.

4. A machine vision-based intelligent inspection method for unmanned aerial vehicles according to claim 1, characterized in that, Based on the changes in recognition results before and after the disturbance, the steps to determine the regions where recognition sensitivity decreases as recognition passivation regions are as follows: For the previously identified visually inert regions, select the corresponding local image blocks in the image sequence, and construct at least one perturbation image in sequence. The perturbation image is the image obtained after introducing a slight intervention to the original image block. The intervention includes at least one: pixel grayscale fine adjustment, brightness enhancement, slight image rotation or contrast fine adjustment. The original image patch and the corresponding perturbation image are respectively input into the same visual recognition model to obtain the recognition result information before and after the perturbation. The sensitivity decrease index is calculated based on the recognition result information before and after the perturbation, and the region where the recognition sensitivity decreases is defined as the recognition passivation region based on the sensitivity decrease index and the preset threshold.

5. A machine vision-based intelligent inspection method for unmanned aerial vehicles according to claim 4, characterized in that, The steps for calculating the sensitivity decrease index based on the identification results before and after the perturbation are as follows: The local feature consistency decay index and the multi-scale instability index are calculated based on the identification results before and after the perturbation. The sensitivity decrease index is obtained by subtracting the local feature consistency decay index from the multi-scale instability index.

6. A machine vision-based intelligent inspection method for unmanned aerial vehicles according to claim 5, characterized in that, The calculation steps for the local feature consistency decay index are as follows: For the identified visually inert regions, extract the corresponding set of local feature points from the original image and the perturbed image; For each extracted feature point, its position coordinates and intensity value in the original image and the perturbed image are recorded respectively. The position coordinates include the pixel coordinates in the image, and the intensity value represents the local image intensity or gradient value of the feature point. Calculate the displacement of each feature point in the images before and after the disturbance. The displacement is the difference in position of the feature point in the images before and after the disturbance. Calculate the intensity change of each feature point before and after the perturbation. The intensity change is the absolute value of the difference between the intensity values ​​of the feature point in the image before and after the perturbation. The local feature consistency attenuation index is obtained by summing the absolute values ​​of the displacements and intensity changes of all feature points and dividing the sum by the total number of feature points.

7. A machine vision-based intelligent inspection method for unmanned aerial vehicles (UAVs) according to claim 5, characterized in that, The calculation steps for the multi-scale instability index are as follows: The original image and the perturbed image are decomposed into multiple image layers of different resolutions, each representing image information at a different scale; the frequency components of each scale image are obtained through Fourier transform. Calculate the frequency component differences between the original image and the perturbed image at each scale; the difference represents the magnitude of the frequency change. Calculate the amplitude standard deviation of frequency variation at each scale, sum the positive square roots of the standard deviations at each scale, and divide the sum by the total number of scales to obtain the multiscale instability value for each image. The multiscale instability values ​​obtained by the synthesis are normalized to the range of 0 to 1 to obtain the corresponding multiscale instability index.

8. A machine vision-based intelligent inspection method for unmanned aerial vehicles according to claim 4, characterized in that, The steps for identifying areas of decreased sensitivity as passivation areas based on the sensitivity decrease index and a preset threshold are as follows: The sensitivity decrease index is compared with a preset threshold. If the sensitivity decrease index is not less than the preset threshold, the area where the sensitivity decreases is identified as the passivation area. If the sensitivity decrease index is less than the preset threshold, the area of ​​decreased sensitivity will not be identified as the passivation area.

9. A machine vision-based intelligent inspection system for unmanned aerial vehicles (UAVs), used to implement the machine vision-based intelligent inspection method for UAVs as described in any one of claims 1-8, characterized in that, The system includes: Fatigue candidate module: acquires the image sequence continuously collected by the UAV during the inspection process, identifies the target area that has not received effective attention for a long time in multiple consecutive images based on the image sequence, and marks the target area as fatigue candidate area; Visual inertia module: Based on fatigue candidate regions, extract their corresponding local image content in the image sequence, and select regions whose change amplitude has been stable for a long time as visual inert regions. Passivation module: Introduces slight image perturbation into visually inert regions and re-executes visual recognition analysis. Based on the changes in recognition results before and after the perturbation, regions where recognition sensitivity decreases are identified as recognition passivation regions. Inspection module: Based on the identified passivated area, trigger the reactivation inspection strategy for the corresponding target area, perform at least one re-shooting, attention reset or manual review operation on the identified passivated area, and output the reactivation result to the inspection result.

Images