Oil leakage detection method and system based on region segmentation and contour feature point extraction

Abstract

The present application belongs to the field of wind power technology, and provides an oil leakage detection method and system based on region segmentation and contour feature point extraction. First, the image of the detection region is segmented into several regions, and the contour feature points of all regions are extracted. The picture block is extracted in the region near the contour feature point, and the picture block is coded as a non-local correlation feature. Then, the local feature of the image is extracted. The local feature and the non-local feature are fused to obtain the fused feature. Finally, the fused feature and the preset similarity loss function are used for oil leakage prediction. Based on region segmentation and contour feature point extraction, the non-local feature and the local feature are considered comprehensively, the problems of lack of local feature analysis capability and no consideration of extracting the contour feature of the oil leakage target are solved, and the detection of oil stains of different shapes can be adapted.

Description

Technical Field

[0001] This invention belongs to the field of wind power technology, and in particular relates to an oil leak detection method and system based on region segmentation and extraction of contour feature points. Background Technology

[0002] Currently, oil leak detection methods in the wind power field mainly include optical, acoustic, infrared, and negative pressure wave detectors. Among them, image and computer vision technologies have made breakthrough progress in oil leak detection and have received widespread attention. Computer vision technology is intuitive and convenient, and can monitor whether oil leaks occur during equipment operation at any time.

[0003] The inventors discovered that traditional oil leak detection methods, based on edge detection, are susceptible to noise affecting contour features, thus introducing noise and impacting contour judgment. The generated templates extract features from local oil spills, neglecting overall oil spill characteristics. Traditional manual feature analysis is ill-suited for detecting oil spills of varying shapes. Oil leaks with low contrast to the background exhibit weak localization capabilities and lack sufficient local feature analysis. Furthermore, YOLO neural network-based detection methods do not address the extraction of non-local image features or focus on learning features near shadows or the oil leak shape. They require mask annotation, increasing annotation time, and are not end-to-end oil leak localization and recognition methods. They only select difficult samples based on prediction results for further feature learning, without focusing on extracting and learning contour features such as the oil leak shape and shadows within the leak samples. Summary of the Invention

[0004] To address at least one of the problems in the background art, this invention proposes an oil leak detection method and system based on region segmentation and extraction of contour feature points. This invention comprehensively considers both non-local and local features, solving problems such as insufficient local feature analysis capabilities and the lack of consideration for extracting the contour features of the oil leak target.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the present invention provides an oil leak detection method based on region segmentation and extraction of contour feature points, comprising: Acquire an image of the detection area; The image is segmented into several regions, and the contour feature points of all regions are extracted; image patches are extracted in the regions near the contour feature points, and the image patches are encoded as non-local correlation features. Extract local features from the image; fuse local and non-local features to obtain fused features; Oil leaks are predicted using fusion features and a pre-defined similarity loss function.

[0006] Furthermore, when extracting the contour feature points of all regions, some scattered points within the regions were also extracted; the image was segmented into several regions based on brightness, color, and the size of the region blocks.

[0007] Furthermore, several feature points are extracted from the contour of each region, and several feature points are extracted within the region.

[0008] Furthermore, convolutional neural networks are used to extract local features.

[0009] Furthermore, the segmented region is divided into blocks, and the blocks containing feature sampling points are used to generate coded features. Non-local features are then calculated based on the coded features.

[0010] Furthermore, based on the fused features, the target bounding box and category of the oil leak are predicted, and the confidence level is given.

[0011] Furthermore, the offset between the selected feature points and the labeled bounding box is first calculated, and then the offset is moved to the predicted bounding box to find the position of the feature points on the predicted bounding box. Based on the positions of the feature points in the labeled bounding box and the predicted bounding box, patches containing the feature points are extracted, and the similarity loss between the patches is calculated. Finally, the similarity loss between the target patch and the predicted patch is calculated, including the calculation of brightness. Contrast and structural relationship :

[0012]

[0013]

[0014]

[0015] Where x represents the target tile; y represents the predicted tile; Calculate the average grayscale value of the two patches respectively. Calculate the variance of the two plots respectively; For covariance, , and This is a hyperparameter.

[0016] Secondly, the present invention also provides an oil leak detection system based on region segmentation and extraction of contour feature points, comprising: The data acquisition module is configured to acquire images of the detection area; The feature extraction module is configured to: segment the image into several regions and extract the contour feature points of all regions; extract image patches in the region near the contour feature points and encode the image patches into non-local correlation features; The feature fusion module is configured to: extract local features of the image; and fuse local and non-local features to obtain fused features. The prediction module is configured to predict oil leaks using fused features and a preset similarity loss function.

[0017] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the oil leak detection method based on region segmentation and extraction of contour feature points described in the first aspect.

[0018] Fourthly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the oil leak detection method based on region segmentation and extraction of contour feature points described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: In this invention, the image of the detection area is first segmented into several regions, and contour feature points of all regions are extracted. Image patches are extracted in the region near the contour feature points, and the image patches are encoded as non-local correlation features. Then, local features of the image are extracted. Local and non-local features are fused to obtain fused features. Finally, the fused features and a preset similarity loss function are used to predict oil leaks. Based on region segmentation and contour feature point extraction, this invention comprehensively considers both non-local and local features, solving problems such as insufficient local feature analysis capabilities and lack of consideration for extracting contour features of oil leak targets. It can adapt to the detection of oil stains of different shapes. Attached Figure Description

[0020] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0021] Figure 1 This is a route diagram of Embodiment 1 of the present invention; Figure 2 For each region block in Embodiment 1 of the present invention, a number of feature points on the contour and a number of feature points extracted inside the region are extracted. Figure 3 This is the convolutional network of Embodiment 1 of the present invention; Figure 4 This is the calculation of non-local correlation features based on encoding features in Embodiment 1 of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0024] Example 1: Image-based and computer vision-based oil leak detection techniques are divided into non-learning feature detection methods and learning feature detection methods. With the rise of deep learning, learning feature detection methods have enormous development potential, among which object detection frameworks are the fastest-growing computer vision application technology. Object detection transforms manual feature extraction into automatic target feature extraction via convolutional neural networks.

[0025] Object detection can be categorized into two-stage object detection (e.g., R-CNN (Region-CNN) and Faster R-CNN) and single-stage object detection (e.g., YOLO and SSD (Single Shot MultiBox Detector)) based on whether the bounding box detection and classification are completed in a single step. The features extracted by object detection are further classified into global, local, and hybrid feature extraction based on whether the feature originates from a large or small field of view. For the problem of learning from difficult oil leak samples during training, a recurrent training method has been proposed to better learn from these challenging samples. Image enhancement methods for removing shadows have also been proposed to address the difficulty of distinguishing between oil leaks and shadows.

[0026] Traditional oil spill detection methods, based on edge detection, suffer from easily affected contour features due to noise, which in turn impacts contour judgment. The generated templates extract local features from the oil spill, neglecting overall characteristics. Traditional manual feature analysis is ill-suited for detecting oil spills of varying shapes. Oil spills with low background contrast exhibit weak localization capabilities and lack sufficient local feature analysis. While YOLO neural network-based detection methods do not extract non-local image features or focus on learning features near shadows or the spill shape, they require masking, increasing annotation time and resulting in non-end-to-end oil spill localization and recognition. Furthermore, they only select difficult samples based on prediction results for further feature learning, without focusing on extracting and learning contour features such as the spill shape and shadows within the leak sample.

[0027] To address at least one of the aforementioned problems, this embodiment provides an oil leak detection method based on region segmentation and contour feature point extraction, comprising: Acquire an image of the detection area; The image is segmented into several regions, and the contour feature points of all regions are extracted; image patches are extracted in the regions near the contour feature points, and the image patches are encoded as non-local correlation features. Extract local features from the image; fuse local and non-local features to obtain fused features; Oil leaks are predicted using fusion features and a pre-defined similarity loss function.

[0028] The image of the detection area is segmented into several regions, and contour feature points of all regions are extracted. Image patches are extracted in the vicinity of the contour feature points, and the image patches are encoded as non-local correlation features. Then, local features of the image are extracted. Local and non-local features are fused to obtain fused features. Finally, the fused features and a preset similarity loss function are used to predict oil leaks. Based on region segmentation and contour feature point extraction, this method comprehensively considers both non-local and local features, solving problems such as insufficient local feature analysis capabilities and lack of consideration for extracting contour features of oil leak targets. It can adapt to the detection of oil stains of different shapes.

[0029] like Figure 1 As shown, the algorithm in this implementation adopts a dual-branch architecture. The algorithm also includes a feature fusion module, a feature point extraction module, and a similarity loss function. In the dual-branch architecture, the convolutional branch extracts the local correlation features of the image, and the attention branch extracts the non-local correlation features of the image. The feature point extraction module extracts several feature points based on the contour of the region after region fusion, and matches the extracted feature points with the template. The feature fusion module fuses the local and non-local correlation features calculated by the dual branches. The similarity loss function extracts sub-images based on the relative positional relationship between feature points and the center of the bounding box, and calculates the similarity of the sub-images.

[0030] Optionally, the specific steps of the method in this embodiment are as follows: S1. The oil leak image is processed by the region fusion module to form several regions, and contour feature points and some internal region scattered points are extracted from these regions. S2. The oil leak image also needs to go through a series of convolutional networks to extract local correlation features.

[0031] S3. Extract image patches in the region near the feature points and encode these image patches as non-local correlation features using an attention mechanism; S4. Merge local correlation features and non-local correlation features; optionally, this can be achieved by performing matrix addition on the local correlation feature matrix and the non-local correlation feature matrix. S5. Use the fused features for bounding box prediction and category prediction, where similarity loss function, intersection-union loss function and cross-entropy loss function are used as optimization objectives.

[0032] Step S1 includes: S1.1 Optionally, the image is divided into several regions based on information such as brightness, color, and size of the regions, and each region has similar properties as described above.

[0033] S1.1.1 Color similarity measurement considers the distribution histogram calculated for each region in each color channel; the intensity of each color channel is divided into 25 intervals, and then combined to obtain 75 bins (if RGB channels are used), as shown above. i area Color distribution histogram .use Norm normalization Color similarity is defined as the intersection of the histograms of two regions. Meanwhile, the color distribution histogram transfer relationship between the merging region and the merged region is defined as:

[0034] in, For the first i The size of the area or the region The number of pixels in the image; for j The size of the area or the region The number of pixels in the image.

[0035] S1.1.2. Use Scale-Invariant Feature Transform (IFT) feature results to represent texture and regions. Extract the distribution histograms of 10 bins in each direction and for each color channel. use Normalized distribution histogram, texture similarity index is defined as the intersection of the histograms of two regions. Meanwhile, the color distribution histogram transfer relationship between the merging region and the merged region is defined as follows:

[0036] S1.1.3 If two regions are relatively small, for example, smaller than a preset value, then the smaller region should be merged earlier. The size similarity is defined as:

[0037] in, This represents the area of ​​the original image or the number of pixels contained in the original image.

[0038] S1.1.4. The larger the gap between two small regions to be merged, the smaller the similarity should be. Define the minimum bounding box between the two regions. Then the gap similarity is defined as:

[0039] in, Minimum border containing two regions The area or the number of pixels of the image it contains.

[0040] S1.1.5, The final overall similarity is defined as follows: ; S1.1.6 The region aggregation process adopts a bottom-up approach. The initial region is obtained using the Felzenswalb method. Through a continuous combination process, regions of different scales are generated, and finally the entire image is transformed into a single region. The available algorithms are: Input: Image; Output: A set of target region locations; Get the initial region ; Initialize the similarity set ; for R and Adjacent do; Calculate the similarity between two regions ;

[0041] While do; Retrieve the maximum similarity value from set S. ; Merge the two regions corresponding to the maximum similarity value ; Remove identical items from set S. Calculated similarity value ; Remove identical items from set S. Calculated similarity value ; Calculate and merge the similarity values ​​of adjacent regions and add all values ​​to the set. ; The similarity set calculated from the fusion region is merged with the fusion region. middle:

[0042] from Extract the target region and obtain the minimum bounding box containing that region.

[0043] S1.2 Extract several feature points on the contour of each region block and extract several feature points within the region, such as... Figure 2 As shown.

[0044] The available algorithms are: Input: Regions in R at a certain scale and their corresponding bounding boxes; Output: Locations of feature points in each region; Initialize the boundary and interior sample sets of all regions. ; for in R do; Will The area within the outer border is converted into a mask (non-zero positions are set to 1); The Canny operator (Laplance operator) calculates the boundaries of the mask image; Extract non-zero positions and randomly or at equal intervals from the samples to place them into the set. ; exist Randomly select several pixel positions and put them into a set. :

[0045] In step S2, optionally, a convolutional neural network can be used to extract local correlation features, such as... Figure 3 As shown, the convolutional network involves feature pyramids, extended layer convergence network E-ELAN, cross-stage feature fusion SPPSCP, multi-scale feature stacking, convolution operation Conv, and downsampling reorg, among other networks. In step S3, optional options include, for example Figure 3 As shown, the segmented region is divided into blocks, and the blocks containing feature sampling points are used to generate coded features. Non-local correlation features are then calculated based on the coded features. Step 5 includes: S5.1 Predict the target bounding box and category based on the fusion features obtained in step S4 and provide the confidence level; S5.2 Extract feature points: In addition to the feature points in step S1.2, extract the feature points for template matching, and use SIFT (SURF, ORB, etc.) to calculate the feature points for template matching. S5.3 First, calculate the offset position between the selected feature points (feature points in the block fusion region and feature points matched with the template) and the labeled bounding box, then move the offset position to the predicted bounding box to find the position of the feature points on the predicted bounding box. The available algorithms are: Input: Set of feature point locations, set of target bounding boxes and set of predicted bounding boxes (there is an alignment function between the target bounding boxes and the predicted bounding boxes); Output: The set of image patches to which the target bounding box is calculated, and the set of image patches to which the predicted bounding box is calculated; Initialize two image tile sets ; Divide the image into a grid; for iterates through the target bounding box set; for iterates through the feature point set; if feature points Target border then; Calculate the offset between the feature point position and the bounding box center. Select the grid to which the feature point belongs as a target tile and add it to the target set; Filter the predicted bounding boxes corresponding to the target bounding boxes based on the alignment function; The predicted feature point positions are calculated based on the offset and the center of the predicted bounding box. Select the grid to which the predicted feature point belongs as a prediction tile and add it to the predict set.

[0046] S5.4 Based on the feature point positions in the labeled bounding box and the predicted bounding box calculated in step S5.2, extract the patches containing the feature points and calculate the similarity loss value between the patches; S5.5 Calculate the similarity loss between the target tile and the predicted tile. This includes calculating brightness, contrast, and structural relationships;

[0047]

[0048]

[0049]

[0050] in, x Represents the target block; y Represents a predicted plot; Calculate the average grayscale value of the two patches respectively; Calculate the variance of the two plots respectively; For covariance, , , , , and This is a hyperparameter.

[0051] Optionally, the target detection loss function is: +

[0052] in, This is the total loss function; To compare the losses; For category loss; To locate the loss; This is for similarity loss.

[0053] Calculate the total loss value between each grid cell and the target grid cells contained within the target bounding box in the image, and accumulate the loss values ​​of all grid cells. As the optimization objective of the neural network, the grid loss value is reduced, and the network parameters are updated through backpropagation so that the network prediction results are close to the manually labeled results.

[0054] The available algorithms are: Input: The set of image patches to which the target bounding box is calculated, and the set of image patches to which the predicted bounding box is calculated, with the patch indices in the two sets corresponding one-to-one; Output: The average loss value of the set of image patches; Set the Gaussian smoothing window size; Set the number of convolutional groups (channels); The for loop iterates through the target image block set; Gaussian smoothing is applied to each channel of the two input image tiles to obtain the average intensity values ​​E(x) and E(y) of the two image tiles; For each channel of the two input image tiles, first perform a quadratic calculation, then apply Gaussian smoothing to obtain the result. ; and the product calculation is then Gaussian smoothed to obtain E(xy); Using formula and Calculate the variance and covariance; Calculate the SSIM value according to step S5.5; Accumulate the SSIM values ​​for the entire tile set; Average the accumulated SSIM values.

[0055] Example 2: This embodiment provides an oil leak detection system based on region segmentation and contour feature point extraction, including: The data acquisition module is configured to acquire images of the detection area; The feature extraction module is configured to: segment the image into several regions and extract the contour feature points of all regions; extract image patches in the region near the contour feature points and encode the image patches into non-local correlation features; The feature fusion module is configured to: extract local features of the image; and fuse local and non-local features to obtain fused features. The prediction module is configured to predict oil leaks using fused features and a preset similarity loss function.

[0056] The working method of the system is the same as that of the oil leak detection method based on region segmentation and extraction of contour feature points in Example 1, and will not be repeated here.

[0057] Example 3: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the oil leak detection method based on region segmentation and extraction of contour feature points described in Embodiment 1.

[0058] Example 4: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the oil leak detection method based on region segmentation and extraction of contour feature points described in Embodiment 1.

[0059] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A method for detecting oil leaks based on region segmentation and extraction of contour feature points, characterized in that, include: Acquire an image of the detection area; The image is divided into several regions, and the contour feature points of all regions are extracted. Extract image patches near contour feature points and encode the image patches into non-local correlation features; divide the segmented region into blocks and generate encoded features from the image patches containing feature sampling points; calculate non-local features based on the encoded features. Extract local features from the image; fuse local and non-local features to obtain fused features; Oil leak prediction is performed using fused features and a pre-defined similarity loss function. First, the offset between selected feature points and the labeled bounding box is calculated. Then, the offset is moved to the predicted bounding box to find the position of the feature points on the predicted bounding box. Based on the positions of the feature points in the labeled bounding box and the predicted bounding box, patches containing the feature points are extracted, and the similarity loss value between the patches is calculated. Finally, the similarity loss between the target patch and the predicted patch is calculated. Includes calculating brightness Contrast and structural relationship : in, x Represents the target image block; y Represents a predicted plot; Calculate the average grayscale value of the two patches respectively; Calculate the variance of the two plots respectively; For covariance, , , , , and This is a hyperparameter.

2. The oil leak detection method based on region segmentation and contour feature point extraction as described in claim 1, characterized in that, When extracting the contour feature points of all regions, some scattered points within the regions were also extracted; the image was segmented into several regions based on brightness, color, and the size of the region blocks.

3. The oil leak detection method based on region segmentation and contour feature point extraction as described in claim 1, characterized in that, Extract several feature points on the contour of each region, and extract several feature points inside the region.

4. The oil leak detection method based on region segmentation and contour feature point extraction as described in claim 1, characterized in that, Local features are extracted using convolutional neural networks.

5. The oil leak detection method based on region segmentation and contour feature point extraction as described in claim 1, characterized in that, Predict the target bounding box and category of the oil leak based on fused features and provide the confidence level.

6. An oil leak detection system based on region segmentation and contour feature point extraction, characterized in that, include: The data acquisition module is configured to acquire images of the detection area; The feature extraction module is configured to: segment the image into several regions and extract contour feature points of all regions; extract image patches in the region near the contour feature points and encode the image patches into non-local correlation features; process the segmented regions into blocks and generate coded features from the image patches containing feature sampling points, and calculate non-local features based on the coded features; The feature fusion module is configured to: extract local features of the image; and fuse local and non-local features to obtain fused features. The prediction module is configured to: predict oil leaks using fused features and a preset similarity loss function; first, calculate the offset position between the selected feature points and the labeled bounding box, then move the offset position to the predicted bounding box to find the position of the feature points on the predicted bounding box; based on the feature point positions in the labeled bounding box and the predicted bounding box, extract patches containing the feature points and calculate the similarity loss value between the patches; finally, calculate the similarity loss between the target patch and the predicted patch. Includes calculating brightness Contrast and structural relationship : in, x Represents the target image block; y Represents a predicted plot; Calculate the average grayscale value of the two patches respectively; Calculate the variance of the two plots respectively; For covariance, , , , , and This is a hyperparameter.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps of the oil leak detection method based on region segmentation and extraction of contour feature points as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the oil leak detection method based on region segmentation and extraction of contour feature points as described in any one of claims 1-5.

Images