Oil leakage detection method and device, computer device, readable storage medium and product

Abstract

The application relates to an oil leakage detection method and device, computer equipment, a readable storage medium and a product. The method comprises the following steps: acquiring a to-be-detected image of a to-be-detected equipment surface; processing the to-be-detected image through a pre-trained target state model to obtain a state image corresponding to the to-be-detected image; the target state model is trained based on an observation image and a reference image; processing the to-be-detected image and the state image through a pre-trained state deviation perception model to obtain a state deviation probability graph or a state deviation mask; the state deviation perception model is trained based on the observation image, the reference image and oil leakage region labeling information; and determining an oil leakage region detection result of the to-be-detected equipment surface based on the state deviation probability graph or the state deviation mask. The method can reduce the false detection rate of the to-be-detected equipment surface oil leakage.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an oil leak detection method, apparatus, computer equipment, readable storage medium, and product. Background Technology

[0002] With the rapid development of computer vision and deep learning technologies, deep learning-based image recognition and segmentation methods have been widely applied in the field of industrial equipment defect detection. In the scenario of oil leak detection, existing technologies typically use semantic segmentation models to perform pixel-level recognition of the oil leak area, that is, to treat the oil leak area as a static target with specific appearance features and perform end-to-end segmentation processing.

[0003] In traditional techniques, oil leak detection methods involve collecting a large number of images of the equipment surface containing the oil leak area. Annotators then annotate the oil leak area pixel by pixel to build a training dataset. This dataset is then used to train a semantic segmentation model, which learns the appearance features of the oil leak area, such as color, texture, edge, and shape, thereby achieving automatic segmentation of the oil leak area.

[0004] However, the existing methods mentioned above rely on the appearance of the oil leak area to identify it. In real industrial scenarios, oil leak areas are often irregular in shape, have blurred boundaries, and are highly similar in color and texture to rust, shadows, oil stains, and other interfering substances on the equipment surface. This makes it difficult for detection methods that rely solely on appearance features to accurately distinguish the real oil leak area from the interfering substances, resulting in a high false detection rate. Summary of the Invention

[0005] Therefore, it is necessary to provide an oil leakage detection method, apparatus, computer equipment, readable storage medium, and product that can reduce the false detection rate of oil leakage on the surface of detection equipment, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides an oil leak detection method, including:

[0007] Acquire an image of the surface of the device to be inspected;

[0008] The target state model is pre-trained to process the image to be detected, and a state image corresponding to the image to be detected is obtained. The target state model is trained based on the observed image and the reference image. The reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0009] The pre-trained state deviation perception model processes the image to be detected and the state image to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image, and the annotation information of the oil leak area.

[0010] Based on the state deviation probability map or state deviation mask, the detection results of the oil leakage area on the surface of the device to be tested are determined.

[0011] Secondly, this application also provides an oil leak detection device, comprising:

[0012] The acquisition module is used to acquire the image of the surface of the device to be inspected.

[0013] The first processing module is used to process the image to be detected through a pre-trained target state model to obtain a state image corresponding to the image to be detected. The target state model is trained based on the observed image and the reference image. The reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0014] The second processing module is used to process the image to be detected and the state image through a pre-trained state deviation perception model to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image and the oil leak area annotation information.

[0015] The determination module is used to determine the detection results of the oil leakage area on the surface of the device under test based on the state deviation probability map or state deviation mask.

[0016] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0017] Acquire an image of the surface of the device to be inspected;

[0018] The target state model is pre-trained to process the image to be detected, and a state image corresponding to the image to be detected is obtained. The target state model is trained based on the observed image and the reference image. The reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0019] The pre-trained state deviation perception model processes the image to be detected and the state image to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image, and the annotation information of the oil leak area.

[0020] Based on the state deviation probability map or state deviation mask, the detection results of the oil leakage area on the surface of the device to be tested are determined.

[0021] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0022] Acquire an image of the surface of the device to be inspected;

[0023] The target state model is pre-trained to process the image to be detected, and a state image corresponding to the image to be detected is obtained. The target state model is trained based on the observed image and the reference image. The reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0024] The pre-trained state deviation perception model processes the image to be detected and the state image to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image, and the annotation information of the oil leak area.

[0025] Based on the state deviation probability map or state deviation mask, the detection results of the oil leakage area on the surface of the device to be tested are determined.

[0026] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0027] Acquire an image of the surface of the device to be inspected;

[0028] The target state model is pre-trained to process the image to be detected, and a state image corresponding to the image to be detected is obtained. The target state model is trained based on the observed image and the reference image. The reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0029] The pre-trained state deviation perception model processes the image to be detected and the state image to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image, and the annotation information of the oil leak area.

[0030] Based on the state deviation probability map or state deviation mask, the detection results of the oil leakage area on the surface of the device to be tested are determined.

[0031] The aforementioned oil leak detection method, apparatus, computer equipment, readable storage medium, and product generate a state image corresponding to the image to be detected through a pre-trained target state model. They then fuse the image to be detected, the state image, and the state deviation field through a state deviation perception model to locate the oil leak area. This ensures that oil leak detection always uses the reference appearance of the equipment in a non-leaking state as a benchmark for state deviation perception. Regardless of the inspection carrier from which the image to be detected originates, or how the imaging conditions change, the detection process does not rely on fixed appearance features such as the color, texture, or shape of the oil leak area. Instead, it uses the degree of abnormal deviation of the equipment surface state relative to the reference state as the criterion. Compared with traditional methods, this application constructs a state deviation field, fusing multi-dimensional information such as pixel deviation, edge deviation, texture deviation, and feature deviation to comprehensively characterize the state deviation caused by oil leaks from multiple levels. This avoids false detections caused by irregular shapes, blurred boundaries, or similar appearances to rust, shadows, or other interfering objects in the oil leak area. Therefore, this method significantly reduces the false detection rate of oil leak detection in complex industrial inspection scenarios, and effectively solves the technical problem that existing methods rely on the appearance features of oil leaks for identification, making it difficult to distinguish between oil leaks and similar-looking interference objects, resulting in a high false detection rate. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is an internal structural diagram of a computer device in one embodiment;

[0034] Figure 2 This is a flowchart illustrating an oil leak detection method in one embodiment;

[0035] Figure 3 This is a flowchart illustrating an oil leak detection method in another embodiment;

[0036] Figure 4 This is a flowchart illustrating an oil leak detection method in another embodiment;

[0037] Figure 5 This is a flowchart illustrating the oil leak detection method in another embodiment;

[0038] Figure 6 This is a flowchart illustrating an oil leak detection method in another embodiment;

[0039] Figure 7 This is a flowchart illustrating the oil leak detection method in another embodiment;

[0040] Figure 8 This is a flowchart illustrating the oil leak detection method in another embodiment;

[0041] Figure 9 This is a structural block diagram of an oil leak detection device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 1 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data during the oil leak detection process. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an oil leak detection method.

[0044] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0045] In one exemplary embodiment, such as Figure 2 As shown, an oil leak detection method is provided, which is applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 201 to 204. Wherein:

[0046] Step 201: Obtain the image of the surface of the device to be inspected.

[0047] The image to be inspected is an image containing the surface of the device to be inspected, used to characterize the visual state of the surface of the device at the current moment. The image to be inspected is a raw image or a pre-processed image containing the surface of the device to be inspected, acquired through drones, inspection robots, fixed cameras, handheld terminals, or video surveillance equipment. The image to be inspected can be a single static image or a video frame extracted from a video stream. The image to be inspected can be a color image (such as an RGB three-channel image) or a grayscale image.

[0048] The equipment to be tested includes industrial equipment such as transformers, circuit breakers, reactors, instrument transformers, bushings, on-load tap changers, valves, oil pipes, flanges, oil pumps, or oil storage devices. During long-term operation, these devices are prone to varying degrees of oil leakage, dripping, oil spread, or oil contamination due to factors such as aging of seals, vibration and impact, temperature changes, manufacturing and assembly errors, or prolonged high-load operation.

[0049] Equipment surface refers to the outer surface of the equipment under test that is exposed to the external environment, including the surface of the equipment housing, flange connections, valve interfaces, pipe joints, bushing roots, and other areas prone to oil leakage. Equipment surface can be a metal surface (such as steel plate, cast aluminum, cast iron), a ceramic surface (such as insulators, bushings), a composite insulating material surface, or a painted surface.

[0050] In this embodiment, the server acquires images of the surface of the equipment to be inspected, collected by a drone during a substation inspection. Specifically, the drone flies along a preset inspection route, carrying a visible light camera to photograph equipment such as transformers, circuit breakers, and instrument transformers within the substation. During flight, the drone transmits the collected images in real time to a ground server or cloud server via a wireless communication link. The server receives the image data transmitted by the drone, decodes and verifies the images, and obtains the image to be inspected. The server further preprocesses the image to be inspected: normalizing the image size to a preset size (e.g., 512×512 pixels or 1024×1024 pixels) to adapt to the input requirements of subsequent target state models and state deviation perception models; and performing brightness normalization processing on the image to reduce the impact of image brightness differences under different lighting conditions on the detection results. The server uses the preprocessed image as the image to be inspected for subsequent processing steps.

[0051] Step 202: The target state model is pre-trained to process the image to be detected to obtain a state image corresponding to the image to be detected. The target state model is trained based on the observed image and the reference image. The reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0052] Among them, the appearance attributes of the equipment refer to the set of visual features presented on the surface of the equipment in the image, including the main structure of the equipment, spatial layout, material texture, and imaging style. Specifically, the main structure of the equipment refers to the overall outline and geometry of the equipment; the spatial layout refers to the relative positional relationships between the various components of the equipment and the routing of pipelines; the material texture refers to the material quality, surface roughness, and pattern distribution of the equipment surface; and the imaging style refers to the overall visual style formed during image acquisition due to factors such as lighting conditions, shooting angle, and camera parameters.

[0053] The visual characteristics corresponding to an oil leak area refer to the abnormal visual appearance of the equipment surface in an image due to oil seepage, dripping, diffusion, or oil contamination. These include any one or more of the following: color characteristics of the oil contamination area (such as dark brown, yellowish-brown, black, etc., with a chromaticity differing from the equipment's body color); texture characteristics of the oil contamination area (such as radial, striped, or irregular patchy distributions formed by oil diffusion); edge characteristics of the oil contamination area (such as blurred edges, gradual transitions rather than sharp edges); and light reflection characteristics of the oil contamination area (such as gloss or reflection formed on the surface of the oil film). In practice, the visual characteristics corresponding to the oil leak area can also vary depending on the type of oil (insulating oil, hydraulic oil, lubricating oil, etc.) and the material of the equipment surface (metal, ceramic, composite insulating materials, etc.).

[0054] The reference image is used to characterize the surface state of the equipment under test under conditions free from oil leakage interference. It maintains the same appearance attributes as the image under test, but eliminates or weakens the visual features corresponding to the oil leakage area. The reference image is not a real oil-free image acquired at a historical moment, but rather a representation of the equipment's reference state obtained through image generation.

[0055] The target state model is an image generation model constructed using a deep learning network structure, used to generate corresponding reference images based on input observed images. The training samples for the target state model consist of image pairs composed of observed and reference images. The reference images are constructed through methods such as manual restoration, regularized texture filling, generative image editing, or semi-automatic assisted generation. During training, the target state model incorporates structure preservation constraints, ensuring that the model maintains consistency in the device's appearance attributes while eliminating visual features corresponding to the oil leak area; that is, the generated reference image remains consistent with the observed image in non-oil leak areas. The target state model employs an encoder-decoder network structure. The encoder extracts deep feature representations from the observed image, and the decoder reconstructs the reference image based on these feature representations.

[0056] In this embodiment, the server inputs the image to be detected captured by the UAV into a pre-trained target state model. The target state model adopts a generative network based on the U-Net architecture, including a downsampling encoder and an upsampling decoder. The encoder and decoder transmit multi-scale spatial detail information through skip connections. The server performs progressive downsampling processing on the input image to be detected through the encoder: first, shallow features are extracted through two convolutional operations, which contain device edges and texture details; then, the feature map size is reduced through max pooling, and then mid-level semantic features are extracted through multiple convolutions, which contain spatial layout information of device components; finally, deep semantic features are extracted through multiple downsampling operations, which contain information on the overall structure of the device and imaging style. The server passes the multi-scale features extracted by the encoder to the decoder through skip connections. The server performs progressive upsampling processing on the deep semantic features through the decoder: after each upsampling step, the server concatenates and fuses the upsampling result with the features of the corresponding layer of the encoder to restore the detail information of the image. The server outputs a reference image with the same size as the input image through the last layer of the decoder. The reference image maintains consistency with the image to be detected in terms of device outline, pipeline routing, connector location, background texture, and imaging style, but the visual features corresponding to the oil leak area are eliminated, presenting an ideal appearance of the device in a leak-free state. The server uses this reference image as input for subsequent state deviation field construction.

[0057] In another embodiment, the server inputs the image to be detected into a pre-trained target state model. The target state model employs a generative adversarial network (GAN)-based architecture, including a generator and a discriminator. During training, the generator and discriminator are jointly optimized through adversarial training to ensure that the reference image output by the generator maintains consistency with the image to be detected in terms of device appearance attributes. The generator uses a residual network structure, performing nonlinear transformations on the input image through multiple residual blocks. Each residual block includes a convolutional layer, a batch normalization layer, and an activation function layer. Within the residual block, skip connections directly superimpose the input onto the output to accelerate training convergence and preserve structural information. The server processes the image to be detected through the generator: first, the input image is mapped to a high-dimensional feature space through convolutional layers; then, features are extracted and transformed through multiple residual blocks; finally, the features are mapped back to the image space through deconvolutional layers, outputting a reference image. During training, the discriminator determines whether the input image is a real reference image (i.e., a manually constructed reference image) or a reference image generated by the generator. Adversarial loss forces the reference image output by the generator to be indistinguishable from the real reference image in terms of device appearance attributes. In this embodiment, the target state model further enhances the realism of the generated images in terms of material texture and imaging style through adversarial training, making it suitable for fine-grained detection scenarios with high image quality requirements. The server uses the reference image output by the generator as input for subsequent state offset field construction.

[0058] Step 203: The pre-trained state deviation perception model is used to process the image to be detected and the state image to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image and the oil leak area annotation information.

[0059] The state deviation probability map is a probability map with the same size as the image to be detected, and the pixel value of each pixel in the map represents the probability that the pixel belongs to the oil leakage area; the state deviation mask is a binary image or binary matrix with the same size as the image to be detected, wherein the first identifier value indicates that the pixel belongs to the oil leakage area, and the second identifier value indicates that the pixel belongs to the non-oil leakage area.

[0060] The state deviation perception model is a deep learning-based image segmentation or pixel-level classification model. It learns and outputs the state deviation distribution of the detected image relative to the state image within the oil leak area, based on the input image to be detected, the state image, and the state deviation field. The state deviation perception model includes an observation state feature extraction branch, a reference state feature extraction branch, a state deviation field feature extraction branch, and a state deviation fusion module. The observation state feature extraction branch extracts deep features from the image to be detected; the reference state feature extraction branch extracts deep features from the state image; the state deviation field feature extraction branch extracts deep features from the state deviation field; and the state deviation fusion module fuses the features from these three branches and outputs a pixel-level classification result.

[0061] The state deviation field is a multi-channel feature map or weight map used to characterize the degree of abnormal deviation of the device surface state reflected in the detection image from that reflected in the state image. The state deviation field is obtained by a weighted combination of one or more of the following components: pixel deviation component, edge deviation component, texture deviation component, and feature deviation component. The pixel deviation component refers to the color intensity difference between the detection image and the state image at each pixel location; the edge deviation component refers to the difference in edge gradient magnitude between the detection image and the state image; the texture deviation component refers to the difference in local texture patterns between the detection image and the state image; and the feature deviation component refers to the difference in high-level semantic features output by the pre-trained feature extraction network between the detection image and the state image. In practical applications, the pixel deviation component can capture the direct visual differences caused by color changes in oil stain areas; the edge deviation component can capture gradient changes caused by blurred or diffused oil stain edges; the texture deviation component can capture local pattern changes caused by abnormal textures in oil stain areas; and the feature deviation component can capture deep semantic feature changes caused by oil stains. Through multi-component weighted fusion, the state deviation field can comprehensively characterize the state deviation caused by oil leakage from multiple dimensions, improving the comprehensiveness and robustness of detection.

[0062] In one embodiment of this application, the server first constructs a state deviation field based on the image to be detected and the state image. Specifically, the server calculates the absolute difference between the pixel positions of the image to be detected and the state image in the three RGB color channels, and averages the differences of the three channels to obtain the pixel deviation component. The server performs the Sobel edge detection operator on the image to be detected and the state image respectively, calculates the gradient magnitude at each pixel position, and then calculates the absolute difference between the two gradient magnitude images to obtain the edge deviation component. The server inputs the image to be detected and the state image into a pre-trained feature extraction network (such as the intermediate layer of a VGG network) to extract high-level semantic features, and then calculates the Euclidean distance between the two feature maps to obtain the feature deviation component. The server obtains a preset first weight coefficient, a second weight coefficient, and a third weight coefficient, multiplies the pixel deviation component, the edge deviation component, and the feature deviation component by their respective weight coefficients, and then sums them to obtain the state deviation field. Regions with higher values ​​in this state deviation field indicate that the difference between the image to be detected and the state image at that position is large, i.e., there is a higher possibility of oil leakage.

[0063] Then, the server inputs the image to be detected, the state image, and the state deviation field into a pre-trained state deviation perception model. The observed state feature extraction branch of the state deviation perception model extracts features from the image to be detected using a multi-layer convolutional neural network to obtain an observed state feature map; the reference state feature extraction branch extracts features from the state image using a multi-layer convolutional neural network to obtain a reference state feature map; and the state deviation field feature extraction branch extracts features from the state deviation field using a multi-layer convolutional neural network to obtain a deviation field feature map. The state deviation fusion module concatenates these three feature maps along the channel dimension, then fuses the concatenated features through one or more convolutional layers, and finally outputs a state deviation probability map of the same size as the image to be detected through a classification layer (e.g., a 1x1 convolutional layer with a sigmoid activation function). Each pixel in this probability map has a value ranging from 0 to 1, representing the probability that the pixel belongs to the oil leak area. The server uses this state deviation probability map as input for subsequent oil leak area detection.

[0064] Step 204: Based on the state deviation probability map or state deviation mask, determine the detection results of the oil leakage area on the surface of the device to be tested.

[0065] The oil leak detection results include one or more of the following: oil leak area mask, oil leak area bounding box, oil leak area area, oil leak area percentage, and oil leak confidence score. The oil leak area mask is a binary image of the same size as the image to be detected, used to identify the pixel positions belonging to the oil leak area; the oil leak area bounding box is the coordinate information of the smallest rectangle that can define the oil leak area (such as the coordinates of the top left and bottom right corners, or the coordinates of the center point and its width and height); the oil leak area is the total number of pixels contained in the oil leak area or its converted physical area; the oil leak area percentage is the percentage of the oil leak area area to the total surface area of ​​the device or the total area of ​​the image to be detected; and the oil leak confidence score is a score reflecting the reliability of the oil leak detection, which can be calculated based on the average or maximum probability values ​​within the oil leak area.

[0066] In this embodiment, the server obtains the state deviation probability map output in step 203. The value of each pixel in the probability map represents the probability that the pixel belongs to the oil leak area. The server obtains a preset probability threshold (e.g., 0.5). The server traverses each pixel in the state deviation probability map and compares the probability value of each pixel with the probability threshold: if the probability value of the current pixel is greater than or equal to the probability threshold, the pixel is determined to be an oil leak pixel, and the corresponding position in the initial oil leak area mask is marked with a first identifier value (e.g., 1); if the probability value of the current pixel is less than the probability threshold, the pixel is determined to be a non-oil leak pixel, and the corresponding position in the initial oil leak area mask is marked with a second identifier value (e.g., 0). After the server traverses all pixels, it obtains the initial oil leak area mask.

[0067] Then, the server performs post-processing on the initial oil leak area mask. The server performs connected component analysis on the initial oil leak area mask: iterates through all pixels in the mask, clustering pixels belonging to the oil leak area and spatially adjacent (e.g., 4-connected or 8-connected) into the same connected component. Each connected component corresponds to a candidate oil leak area instance, resulting in a set of candidate oil leak area instances. The server obtains a preset area threshold, iterates through each candidate oil leak area instance in the set, and counts the number of pixels contained in the current candidate oil leak area instance as the area of ​​that instance. If the area of ​​the instance is less than the area threshold, the candidate oil leak area is determined to be noise or a false detection, and the instance is removed from the oil leak area; if the area of ​​the instance is greater than or equal to the area threshold, the oil leak area instance is retained. The server performs a morphological closing operation (dilation followed by erosion) on the filtered oil leak area mask to fill the small holes inside the oil leak area, making the boundary of the oil leak area more complete and continuous. The server performs boundary smoothing on the oil leak area mask after morphological closing operations to remove jagged edges and make the oil leak area boundary smoother and more natural. The server uses the binary image obtained after the above post-processing as the final oil leak area mask and outputs it as the oil leak area detection result. This post-processing process can effectively remove isolated false detection points caused by model prediction noise and improve the accuracy of the detection results.

[0068] In the aforementioned oil leak detection method, a state image corresponding to the image to be detected is generated through a pre-trained target state model. The oil leak area is located by fusing the image to be detected, the state image, and the state deviation field using a state deviation perception model. This ensures that oil leak detection always uses the reference appearance of the equipment in a non-leaking state as a benchmark for state deviation perception. Regardless of the inspection carrier from which the image to be detected originates, or how the imaging conditions change, the detection process does not rely on fixed appearance features such as the color, texture, or shape of the oil leak area. Instead, it uses the degree of abnormal deviation of the equipment surface state relative to the reference state as the criterion. Compared with traditional methods, this application constructs a state deviation field, fusing multi-dimensional information such as pixel deviation, edge deviation, texture deviation, and feature deviation to comprehensively characterize the state deviation caused by oil leaks from multiple levels. This avoids false detections caused by irregular shapes, blurred boundaries, or similar appearances to rust, shadows, or other interfering objects in the oil leak area. Therefore, this method significantly reduces the false detection rate of oil leak detection in complex industrial inspection scenarios and effectively solves the technical problem of existing methods relying on the appearance features of oil leaks for identification, making it difficult to distinguish between oil leaks and similar-looking interfering objects, resulting in a high false detection rate.

[0069] In one exemplary embodiment, such as Figure 3 As shown, the above-mentioned "processing the image to be detected and the state image using a pre-trained state deviation perception model to obtain a state deviation probability map or a state deviation mask" includes steps 301 to 302, wherein:

[0070] Step 301: Construct a state deviation field based on the image to be detected and the state image; the state deviation field is used to characterize the degree of abnormal deviation of the device surface state reflected by the image to be detected from the device surface state reflected by the state image.

[0071] The state deviation field is a multi-channel feature map, weight map, or numerical matrix used to characterize the degree of difference between the image to be detected and the state image. A larger value in the state deviation field indicates a greater difference between the image to be detected and the state image at that location, meaning a higher probability of oil leakage at that location.

[0072] The state deviation field is obtained by a weighted combination of one or more of the following components: pixel deviation, edge deviation, texture deviation, and feature deviation. The pixel deviation component refers to the difference in color or grayscale intensity between the image to be detected and the state image at each pixel location; the edge deviation component refers to the difference in edge gradient magnitude between the image to be detected and the state image; the texture deviation component refers to the difference in local texture patterns between the image to be detected and the state image; and the feature deviation component refers to the difference in high-level semantic features output by the pre-trained feature extraction network between the image to be detected and the state image. In practical applications, the pixel deviation component can capture the direct visual differences caused by color changes in oil stain areas; the edge deviation component can capture the gradient changes caused by blurred or diffused oil stain edges; the texture deviation component can capture the local pattern changes caused by texture anomalies in oil stain areas; and the feature deviation component can capture the deep semantic feature changes caused by oil stains. Through multi-component weighted fusion, the state deviation field can comprehensively characterize the state deviation caused by oil leaks from multiple dimensions, improving the comprehensiveness and robustness of detection.

[0073] In this embodiment, the server calculates the pixel deviation component, edge deviation component, and feature deviation component between the image to be detected and the state image, respectively. Specifically, the server converts the image to be detected and the state image from the RGB color space to the grayscale space, calculates the absolute difference between the two grayscale images at each pixel position, and obtains the pixel deviation component. The server performs the Sobel edge detection operator on the image to be detected and the state image respectively, calculates the gradient magnitude at each pixel position, and then calculates the absolute difference between the two gradient magnitude images to obtain the edge deviation component. The server inputs the image to be detected and the state image into a pre-trained feature extraction network (such as the intermediate convolutional layer of the VGG-16 network), extracts high-level semantic feature maps, and then calculates the Euclidean distance between the two feature maps to obtain the feature deviation component. The server obtains a preset first weight coefficient, a second weight coefficient, and a third weight coefficient, multiplies the pixel deviation component, the edge deviation component, and the feature deviation component by their respective weight coefficients, and then sums them to obtain the state deviation field.

[0074] Step 302: Input the image to be detected, the state image, and the state deviation field into the pre-trained state deviation perception model, and output the state deviation probability map or the state deviation mask.

[0075] The state deviation probability map is a probability map with the same size as the image to be detected, and the pixel value of each pixel in the map represents the probability that the pixel belongs to the oil leakage area; the state deviation mask is a binary image or binary matrix with the same size as the image to be detected, wherein the first identifier value indicates that the pixel belongs to the oil leakage area, and the second identifier value indicates that the pixel belongs to the non-oil leakage area.

[0076] In one embodiment of this application, the server inputs the image to be detected, the state image, and the state deviation field into three feature extraction branches of the state deviation perception model, respectively. The observation state feature extraction branch extracts features from the image to be detected using a multi-layer convolutional neural network to obtain an observation state feature map; the reference state feature extraction branch extracts features from the state image using a multi-layer convolutional neural network to obtain a reference state feature map; and the state deviation field feature extraction branch extracts features from the state deviation field using a multi-layer convolutional neural network to obtain a deviation field feature map. The state deviation fusion module concatenates the three feature maps along the channel dimension, then fuses the concatenated features through one or more convolutional layers, and finally outputs a state deviation probability map of the same size as the image to be detected through a classification layer (e.g., a 1×1 convolutional layer with a sigmoid activation function). Each pixel in this probability map has a value ranging from 0 to 1, representing the probability that the pixel belongs to an oil leak area. The server uses this state deviation probability map as input for subsequent oil leak area detection.

[0077] In one exemplary embodiment, such as Figure 4 As shown, the training process of the above target state model includes steps 401 to 403, wherein:

[0078] Step 401: Input the observed image into the state model to be trained to generate the training image; determine the structural consistency loss value between the observed image and the training image.

[0079] In this context, the training image is the prediction result generated by the model under test based on the input observed image. During the training phase, the model under test processes the observed image and outputs a training image, which is the model's prediction of the reference image. The training image and the reference image are identical in size.

[0080] The training state model is an image generation model built using a deep learning network structure. Its parameters are iteratively updated during training through backpropagation and optimization algorithms. The training state model employs an encoder-decoder network structure (such as U-Net), a generative adversarial network structure, or a diffusion model structure. In the initial training phase, the parameters of the training state model are either randomly initialized or initialized using pre-trained weights.

[0081] The structural consistency loss is a loss function value used to constrain the training image and the observed image to maintain consistent device appearance attributes within non-leaky areas. The structural consistency loss is obtained by a weighted combination of one or more of the following: reconstruction loss, structural preservation constraint loss, and state deviation enhancement constraint loss. The reconstruction loss constrains the training image and the reference image to maintain consistency at the pixel level; the structural preservation constraint loss constrains the training image and the observed image to maintain structural consistency within non-leaky areas; and the state deviation enhancement constraint loss enhances the separability of the leaky area between the observed image and the training image.

[0082] In this embodiment, the server extracts an observation image and its corresponding reference image from the training sample set as the current training samples. Both the observation image and the reference image are RGB three-channel color images with a preset size (e.g., 256×256 pixels or 512×512 pixels). The server inputs the observation image into the training state model. The training state model adopts a U-Net architecture, including a downsampling encoder and an upsampling decoder. Multi-scale spatial information is transmitted between the encoder and decoder via skip connections. The server progressively downsamples the observation image using the encoder to extract multi-scale image features; progressively upsampling the extracted features using the decoder, and combining the spatial information transmitted via skip connections to recover image details; the last layer of the decoder outputs a training image with the same size as the observation image. Then, the server determines the structural consistency loss value between the observation image and the training image. Specifically, the server calculates the pixel-level difference between the observation image and the training image to obtain a first difference value. The server obtains a pre-annotated oil leak area label mask, determines non-oil leak structural regions (i.e., regions with a mask value of 0) based on the oil leak area label mask, and calculates the feature difference between the observation image and the training image within the non-oil leak structural regions to obtain a second difference value. The server performs a weighted sum of the first and second difference values ​​to obtain the structural consistency loss value. This structural consistency loss value is used to update the parameters of the model to be trained in subsequent steps.

[0083] Step 402: Update the model parameters of the state model to be trained based on the structural consistency loss value.

[0084] In this embodiment, the server obtains the structural consistency loss value calculated in step 401. The server calculates the gradient of each model parameter in the training state model based on the structural consistency loss value. Specifically, the server uses the backpropagation algorithm: starting from the structural consistency loss value, it calculates the partial derivative (i.e., gradient) of the loss value with respect to each layer of model parameters in reverse order of the computation graph. The server updates the model parameters of the training state model based on the calculated gradients. Specifically, the server uses the Adam optimization algorithm to update the model parameters: first, it calculates the first moment estimate (i.e., the exponential moving average of the gradient) and the second moment estimate (i.e., the exponential moving average of the squared gradient); then, it corrects the bias of the first and second moment estimates; finally, it calculates the update amount of each model parameter based on the corrected first and second moment estimates, and adds the update amount to the current model parameters to obtain the updated model parameters. The server uses a preset learning rate (e.g., 0.001) to control the step size of the parameter updates. The server uses the updated model parameters as the current model parameters of the training state model for the next iteration of training.

[0085] In another embodiment, the server uses a stochastic gradient descent optimization algorithm to update the model parameters. Specifically, the server obtains the structural consistency loss value calculated in step 401 and uses the backpropagation algorithm to calculate the gradient of the loss value with respect to each model parameter. The server obtains a preset learning rate (e.g., 0.01) and momentum coefficient (e.g., 0.9). The server updates the model parameters according to the gradient descent direction: first, it calculates the exponential moving average of the current gradient (introducing a momentum term to accelerate convergence and suppress oscillations), then multiplies this moving average by the learning rate to obtain the parameter update amount, and finally subtracts the update amount from the current parameter value to obtain the updated model parameters. The server uses the updated model parameters as the current model parameters for the model to be trained.

[0086] Step 403: Repeat the steps of generating training images, calculating structural consistency loss values, and updating model parameters until the preset training stopping conditions are met to obtain the target state model. The preset training stopping conditions include one of the following: structural consistency loss is less than a preset threshold, the number of training rounds reaches a preset number of rounds, or the change in structural consistency loss in multiple consecutive iterations is less than a preset change threshold.

[0087] Wherein, the number of training rounds is the number of times the model to be trained completely traverses the entire training sample set, and one round of training means that the model completes one forward propagation and one backward propagation for all training samples; the target state model is the model obtained after training that can generate the corresponding reference image based on the input observation image.

[0088] In this embodiment, after updating the model parameters in each iteration, the server obtains the structural consistency loss value for the current iteration. The server compares the structural consistency loss value for the current iteration with a preset loss threshold (e.g., 0.05). If the structural consistency loss value for the current iteration is less than the preset loss threshold, the server determines that the training stopping condition is met, immediately terminates the training loop, and outputs the current training state model as the target state model. If the structural consistency loss value for the current iteration is greater than or equal to the preset loss threshold, the server determines that the training stopping condition is not met and continues to execute the next iteration.

[0089] In another embodiment, after updating the model parameters in each iteration, the server obtains the number of training epochs already executed. The server compares the number of training epochs already executed with a preset training epoch threshold (e.g., 100 epochs). If the number of training epochs already executed reaches the preset training epoch threshold, the server determines that the training stopping condition is met, immediately terminates the training loop, and outputs the current training state model as the target state model. If the number of training epochs already executed does not reach the preset training epoch threshold, the server determines that the training stopping condition is not met and continues to execute the next iteration.

[0090] In one exemplary embodiment, such as Figure 5 As shown, the above "determining the structural consistency loss value between the observed image and the training image" includes steps 501 to 503, wherein:

[0091] Step 501: Determine the reconstruction loss between the observed image and the training image.

[0092] Among them, the reconstruction loss is a pixel-level loss function used to constrain the training images and the observed images to maintain consistency in overall appearance, so as to avoid the model losing the appearance attribute information of the device during training.

[0093] In one embodiment of this application, the server acquires an observation image and a training image output by the model to be trained based on the observation image. Both the observation image and the training image are RGB three-channel color images with a preset size (e.g., 256×256 pixels). The server calculates the reconstruction loss between the observation image and the training image. Specifically, the server iterates through each pixel position and each color channel in the observation image and the training image, calculates the absolute value of the difference between the pixel values ​​at the same pixel position and in the same channel in the observation image and the training image, sums the absolute differences for all pixel positions and all channels, and then divides the sum by the product of the total number of pixels and the number of channels to obtain the reconstruction loss value. This reconstruction loss value is a scalar value used in subsequent steps to calculate the structural consistency loss value.

[0094] Step 502: Obtain the first weight coefficient and the second weight coefficient; calculate the first weighted result of the structure preservation constraint loss and the first weight coefficient; calculate the second weighted result of the state deviation enhancement constraint loss and the second weight coefficient.

[0095] The first weighting coefficient and the second weighting coefficient are preset values ​​used to adjust the proportion of the structure preservation constraint loss and the state deviation enhancement constraint loss in the structure consistency loss value. The structure preservation constraint loss is used to constrain the training image and the observation image to remain consistent in the non-oil leakage structure region, and the state deviation enhancement constraint loss is used to enhance the separability of the oil leakage region between the observation image and the training image.

[0096] The structure preservation constraint loss is a loss component used to ensure that the appearance properties of the equipment are consistent between the training images and the observed images within the non-leaking structural regions. By minimizing the structure preservation constraint loss, the model does not alter the equipment outline, pipeline orientation, connector positions, background texture, and imaging style in the non-leaking regions when generating training images.

[0097] The state deviation enhancement constraint loss is a loss component used to enhance the separability of the oil leak region between the observed image and the training image. By minimizing the state deviation enhancement constraint loss, the model actively eliminates or weakens the visual features corresponding to the oil leak region when generating training images, making the oil leak region significantly different between the observed image and the training image.

[0098] In one embodiment of this application, the server obtains a preset first weighting coefficient (e.g., 0.5) and a second weighting coefficient (e.g., 0.3). The server obtains the structure preservation constraint loss value and the state deviation enhancement constraint loss value. The server calculates the product of the structure preservation constraint loss and the first weighting coefficient to obtain a first weighted result. The server calculates the product of the state deviation enhancement constraint loss and the second weighting coefficient to obtain a second weighted result. Both the first weighted result and the second weighted result are scalar values, used in subsequent steps to calculate the structure consistency loss value.

[0099] In another embodiment, the server dynamically adjusts the first and second weight coefficients. Specifically, the server dynamically adjusts the values ​​of the first and second weight coefficients according to the current training epoch: In the early stages of training, the server sets the first weight coefficient to a larger value (e.g., 0.8) to enhance the supervision strength of the structure preservation constraint, ensuring that the model first learns the ability to preserve the device's apparent attributes; as the training epochs increase, the server gradually decreases the first weight coefficient while simultaneously increasing the second weight coefficient (e.g., the first weight coefficient linearly decreases from 0.8 to 0.3, and the second weight coefficient linearly increases from 0.1 to 0.5), so that the model pays more attention to feature elimination and state deviation enhancement of the leak area in the later stages of training. The server multiplies the first weight coefficient corresponding to the current training epoch with the structure preservation constraint loss to obtain the first weighted result; and multiplies the second weight coefficient corresponding to the current training epoch with the state deviation enhancement constraint loss to obtain the second weighted result.

[0100] Step 503: The reconstruction loss, the first weighted result, and the second weighted result are summed to obtain the structural consistency loss value.

[0101] Among them, the structural consistency loss is the total loss value used to train the model of the state to be trained. This loss value integrates information from three components: reconstruction loss, structural preservation constraint loss, and state deviation enhancement constraint loss.

[0102] In one embodiment of this application, the server obtains the reconstruction loss value determined in step 501, and the first weighted result and second weighted result calculated in step 502. The server sums the reconstruction loss value, the first weighted result, and the second weighted result, that is, adds the three scalar values ​​together to obtain the structural consistency loss value. This structural consistency loss value comprehensively reflects the performance of the training image in three aspects: pixel-level reconstruction quality, the degree of structural preservation in non-leaking areas, and the degree of state deviation enhancement in leaking areas. The server uses this structural consistency loss value as the total loss value of the model to be trained, and uses it to update the model parameters of the model to be trained in subsequent steps.

[0103] In another embodiment, the server applies different weight ratios to the reconstruction loss, the first weighted result, and the second weighted result for weighted summation. Specifically, the server obtains a preset third weight coefficient (e.g., 0.2), multiplies the reconstruction loss by the third weight coefficient to obtain the reconstruction weighted result, and then sums the reconstruction weighted result with the first and second weighted results obtained in step 502 to obtain the structural consistency loss value. This embodiment, by introducing a third weight coefficient to independently weight the reconstruction loss, can flexibly adjust the proportion of the reconstruction loss in the total loss to adapt to different training needs.

[0104] In one exemplary embodiment, such as Figure 6As shown, the process for determining the constraint loss of the structure includes steps 601 to 604, wherein:

[0105] Step 601: Obtain the oil leak area marking mask, and determine the non-oil leak structure area based on the oil leak area marking mask.

[0106] The oil leak area labeling mask is a binary image or binary matrix with the same size as the observed image, used to identify the pixel positions in the observed image that belong to the oil leak area; the non-oil leak structure area is the equipment surface area in the observed image other than the oil leak area, and this area should maintain the same appearance attributes of the equipment between the observed image and the training image.

[0107] The oil leak area annotation mask is a binary image or binary matrix with the same size as the observed image. A first identifier value (e.g., 1 or 255) indicates that the pixel belongs to the oil leak area, and a second identifier value (e.g., 0) indicates that the pixel belongs to a non-oil leak area. The oil leak area annotation mask is manually annotated by annotators based on the actual location of the oil leak area in the observed image. Annotation methods include polygon annotation, pen annotation, or intelligent semi-automatic annotation. Oil leak areas include different forms such as seepage, dripping, oil spread, or oil contamination.

[0108] The non-leaking structural area is the region corresponding to the pixel with a value of 0 in the oil leakage area marker mask, that is, the normal area on the device surface that is not affected by the oil leakage. The non-leaking structural area retains the original appearance attributes of the device, including the device outline, pipeline routing, connector positions, background texture, and imaging style.

[0109] In this embodiment, the server obtains a pre-annotated oil leak area label mask. This oil leak area label mask has the same size as the observed image and is a binary image, where the pixel value corresponding to the oil leak area is 1, and the pixel value corresponding to the non-oil leak area is 0. The server iterates through each pixel position in the oil leak area label mask, identifying the positions with pixel values ​​of 0 as non-oil leak structural regions, thus obtaining a label matrix for the non-oil leak structural regions. This label matrix has the same size as the observed image, where the pixel value corresponding to the non-oil leak structural regions is 1, and the pixel value corresponding to the oil leak area is 0. The non-oil leak structural regions are used in subsequent steps to constrain the training image to maintain consistency with the observed image within these regions.

[0110] In another embodiment, after obtaining the oil leak area annotation mask, the server performs morphological dilation processing on the oil leak area annotation mask to obtain a dilated oil leak area mask. Specifically, the server uses a structuring element of a preset size (such as a 3×3 pixel circular structuring element) to perform morphological dilation on the oil leak area annotation mask: the center point of the structuring element is traversed through each pixel position in the mask; if at least one oil leak pixel exists in the neighborhood covered by the structuring element, the center position is marked as the oil leak area. Morphological dilation processing can expand the boundary range of the oil leak area, incorporating the blurred transition area at the edge of the oil leak area into the oil leak area, avoiding incorrect constraints caused by deviations between the annotation boundary and the actual oil leak boundary in the structure maintaining constraint at the oil leak edge area. The server determines the non-oil leak structure area based on the dilated oil leak area mask, that is, the area with a pixel value of 0 in the dilated mask.

[0111] Step 602: Perform element-wise multiplication of the non-oil-leaking structural area with the observed image to obtain the first product result.

[0112] Element-wise multiplication refers to the operation of multiplying the values ​​at the same position in two images or two matrices. By using element-wise multiplication, non-oil-leaking structural regions can be applied as masks to the observed image, preserving the pixel information of non-oil-leaking structural regions in the observed image while suppressing the pixel information of oil-leaking regions.

[0113] In one embodiment of this application, the server obtains the identifier matrix of the non-oil-leaking structural region determined in step 601. This identifier matrix is ​​a binary matrix, where the pixel value at the corresponding position of the non-oil-leaking structural region is 1, and the pixel value at the corresponding position of the oil-leaking structural region is 0. The server obtains an observation image, which is an RGB three-channel color image with a preset size (e.g., 256×256 pixels). The server performs element-wise multiplication operations on the identifier matrix of the non-oil-leaking structural region with the red, green, and blue color channels of the observation image: for each pixel position, the pixel value of each channel of the observation image is multiplied by the pixel value (1 or 0) of the non-oil-leaking structural region identifier matrix at that position. For pixels within the non-oil-leaking structural region (identifier value 1), the product result retains the original pixel value of the pixel; for pixels within the oil-leaking region (identifier value 0), the product result is 0, and the pixel value of that region is set to black. The server merges the product results of the three channels to obtain a first product result. The first product result retains the complete pixel information of the observation image within the non-oil-leaking structural region, and the oil-leaking region is filled with black.

[0114] In another embodiment, after performing element-wise multiplication, the server normalizes the first product result. Specifically, the server performs statistical analysis on the pixel values ​​of non-oil-leaking structural regions in the first product result, calculating the mean and standard deviation of the pixel values. The server subtracts the mean from each pixel value and divides it by the standard deviation to obtain the normalized first product result. Normalization eliminates differences in pixel value distribution caused by variations in lighting conditions between different images, making the subsequent feature extraction process more stable.

[0115] Step 603: Perform element-wise multiplication of the non-oil-leaking structural region with the training image to obtain the second product result.

[0116] In this embodiment, the server obtains the identifier matrix of the non-oil-leaking structural region determined in step 601. This identifier matrix is ​​a binary matrix, where the pixel value at the corresponding position of the non-oil-leaking structural region is 1, and the pixel value at the corresponding position of the oil-leaking structural region is 0. The server obtains the training image output by the model to be trained. The training image is an RGB three-channel color image with the same size as the observed image (e.g., 256×256 pixels). The server performs element-wise multiplication operations on the identifier matrix of the non-oil-leaking structural region with the red, green, and blue color channels of the training image: for each pixel position, the pixel value of each channel of the training image is multiplied by the pixel value (1 or 0) of the non-oil-leaking structural region identifier matrix at that position. For pixels within the non-oil-leaking structural region (identifier value 1), the product result retains the original pixel value of the pixel; for pixels within the oil-leaking region (identifier value 0), the product result is 0, and the pixel value of that region is set to black. The server merges the product results of the three channels to obtain a second product result. The second product result retains the complete pixel information of the training image within the non-oil-leaking structural region, and the oil-leaking region is filled with black.

[0117] In another embodiment, after performing element-wise multiplication, the server normalizes the second product result. Specifically, the server performs statistical analysis on the pixel values ​​of non-leaking structural regions in the second product result, calculating the mean and standard deviation of the pixel values. The server subtracts the mean from each pixel value and divides by the standard deviation to obtain the normalized second product result. This normalization process ensures that the second product result and the first product result are on the same order of magnitude in terms of pixel value distribution, facilitating the calculation of the difference between the two in subsequent steps.

[0118] Step 604: Extract features from the first product result and the second product result respectively to obtain the first feature and the second feature; determine the absolute value of the difference between the first feature and the second feature as the structure preservation constraint loss.

[0119] In this embodiment, the server obtains the first product result obtained in step 602 and the second product result obtained in step 603. The server inputs the first product result into a pre-trained feature extraction network to obtain a first feature. The server inputs the second product result into the same feature extraction network to obtain a second feature. The feature extraction network adopts a VGG-16 network structure, which is pre-trained on the ImageNet dataset and is capable of extracting semantic features of images. Specifically, the server inputs the first product result into the VGG-16 network, and after multiple convolution and pooling operations, extracts the feature map output from the intermediate layer of the network (such as the third convolutional layer of the fourth convolutional block) as the first feature; the server inputs the second product result into the same VGG-16 network and extracts the feature map from the same layer as the second feature. Both the first and second features are multi-channel feature maps with a size smaller than the input image. The server calculates the difference between the first and second features: it iterates through the feature values ​​at each corresponding position and in each channel of the two feature maps, calculates the absolute value of the difference between the feature values, sums all the absolute differences, and divides by the total number of elements in the feature maps to obtain the feature difference value. The server determines this feature difference value as the structure preservation constraint loss. The smaller the structure preservation constraint loss value, the closer the training image is to the device appearance properties of the observed image in the non-leaking structure region.

[0120] In one exemplary embodiment, such as Figure 7 As shown, the process for determining the enhanced constraint loss due to state deviation includes steps 701 to 703, wherein:

[0121] Step 701: Obtain the oil leak area marking mask and determine the oil leak area based on the oil leak area marking mask.

[0122] The oil leak area labeling mask is a binary image or binary matrix with the same size as the observed image, used to identify the pixel positions in the observed image that belong to the oil leak area; the oil leak area is the pixel area in the observed image where oil seepage occurs.

[0123] The oil leak area annotation mask is a binary image or binary matrix with the same size as the observed image. A first identifier value (e.g., 1 or 255) indicates that the pixel belongs to the oil leak area, and a second identifier value (e.g., 0) indicates that the pixel belongs to a non-oil leak area. The oil leak area annotation mask is manually annotated by annotators based on the actual location of the oil leak area in the observed image. Annotation methods include polygon annotation, pen annotation, or intelligent semi-automatic annotation. Oil leak areas include different forms such as seepage, dripping, oil spread, or oil contamination.

[0124] In one embodiment of this application, the server obtains a pre-labeled oil leak area annotation mask. This oil leak area annotation mask has the same size as the observed image and is a binary image, where the pixel value corresponding to the oil leak area is 1, and the pixel value corresponding to the non-oil leak area is 0. The server iterates through each pixel position in the oil leak area annotation mask, identifying the positions with a pixel value of 1 as oil leak areas, thus obtaining an oil leak area identification matrix. This identification matrix has the same size as the observed image, where the pixel value corresponding to the oil leak area is 1, and the pixel value corresponding to the non-oil leak area is 0. The server counts the total number of pixels with a pixel value of 1 in the oil leak area identification matrix, denoted as the number of pixels in the oil leak area. The number of oil leak areas and the number of pixels in the oil leak area are used in subsequent steps to calculate the state deviation enhancement constraint loss.

[0125] Step 702: Calculate the pixel difference between the observed image and the training image at each pixel point in the oil leak area, and sum the pixel differences at each pixel point and divide by the number of pixels in the oil leak area to obtain the average pixel difference.

[0126] The average pixel difference measures the average difference between the observed image and the training image across all pixels within the oil leak area. A larger average pixel difference indicates a greater difference between the training image and the observed image within the oil leak area, meaning a more significant reduction or elimination of the visual features corresponding to the oil leak area by the training image.

[0127] In this embodiment, the server obtains the identifier matrix of the oil leak area and the number of pixels in the oil leak area determined in step 701. The server obtains the observation image and the training image output by the state model to be trained based on the observation image. Both the observation image and the training image are RGB three-channel color images with a preset size (e.g., 256×256 pixels). The server converts the observation image and the training image from the RGB color space to the grayscale space to obtain the observation grayscale image and the training grayscale image. The server iterates through each pixel position within the oil leak area, calculates the absolute value of the difference between the pixel value of the observation grayscale image and the training grayscale image at that position, and obtains the pixel difference value at that pixel position. The server sums the pixel difference values ​​in all oil leak areas to obtain the total pixel difference value. The server divides the total pixel difference value by the number of pixels in the oil leak area to obtain the average pixel difference. This average pixel difference is a scalar value used in subsequent steps to calculate the state deviation enhancement constraint loss.

[0128] Step 703: Take the negative exponent of the product of the average pixel difference and the preset adjustment coefficient to obtain the state deviation enhancement constraint loss.

[0129] The preset adjustment coefficient is a preset value used to adjust the influence of the average pixel difference on the state deviation enhancement constraint loss. The state deviation enhancement constraint loss is used to enhance the separability of the oil leak area between the observed image and the training image.

[0130] The preset adjustment coefficient is a pre-defined positive number used to adjust the scale of the average pixel difference, keeping the state deviation enhancement constraint loss within a reasonable range. The larger the preset adjustment coefficient, the more sensitive the state deviation enhancement constraint loss is to changes in the average pixel difference. The preset adjustment coefficient is pre-set based on the characteristics of the training data and the model structure (e.g., 0.1, 0.5, or 1.0).

[0131] The state deviation enhancement constraint loss is a loss component used to enhance the separability of the oil leak region between the observed and training images. When the average pixel difference of the oil leak region between the observed and training images is small, the state deviation enhancement constraint loss is large, applying a strong supervisory signal to the model and prompting it to increase the difference in the oil leak region. When the average pixel difference is large, the state deviation enhancement constraint loss approaches zero, indicating that the model has generated sufficient state deviation responses in the oil leak region. The state deviation enhancement constraint loss encourages the model to actively eliminate or weaken the visual features of the oil leak region during training, making the oil leak region significantly different between the observed and training images.

[0132] In one embodiment of this application, the server obtains the average pixel difference calculated in step 702. The server obtains a preset adjustment coefficient (e.g., 0.5). The server multiplies the average pixel difference by the preset adjustment coefficient to obtain the product result. The server takes the negative exponent of the product result with the natural constant e as the base, that is, it calculates the negative power of the product result of e to obtain the state deviation enhancement constraint loss. When the average pixel difference is small, the product result is small, the absolute value of the negative exponent is small, and the state deviation enhancement constraint loss is large (close to 1). This large loss value will drive the model to increase the pixel difference in the oil leak area in subsequent training. When the average pixel difference is large, the product result is large, the absolute value of the negative exponent is large, and the state deviation enhancement constraint loss is small (close to 0), indicating that the model has generated sufficient difference in the oil leak area and no further enhancement is needed. The server uses the calculated state deviation enhancement constraint loss to calculate the structural consistency loss value in subsequent steps.

[0133] In an exemplary embodiment, the above-mentioned "constructing a state deviation field based on the observed image and the reference image" includes:

[0134] The pixel deviation component, edge deviation component, texture deviation component, and feature deviation component between the observed image and the reference image are determined respectively; the pixel deviation component, edge deviation component, texture deviation component, and feature deviation component are weighted and summed to obtain the state deviation field.

[0135] The component comprises three parts: a pixel offset component, obtained by calculating the color or grayscale intensity difference between the observed and reference images at each pixel location, used to capture the direct visual deviation caused by color changes in the oil spill area; an edge offset component, obtained by extracting edge information from the observed and reference images and calculating their differences, used to capture gradient changes caused by blurred or diffused oil edges; a texture offset component, obtained by analyzing the local texture patterns of the observed and reference images and calculating their differences, used to capture texture changes in the oil spill area caused by oil seepage; and a feature offset component, obtained by inputting the observed and reference images into a pre-trained feature extraction network and calculating their differences in a high-level semantic feature space, used to capture deep semantic feature changes caused by the oil spill.

[0136] In one embodiment of this application, the server first determines the pixel deviation component, edge deviation component, and feature deviation component between the observed image and the reference image. For the pixel deviation component, the server converts the observed image and the reference image from the RGB color space to the grayscale space, calculates the absolute difference between the two grayscale images at each pixel position, and obtains a grayscale difference map as the pixel deviation component. For the edge deviation component, the server performs Sobel edge detection on the observed image and the reference image respectively, obtaining an observed edge image and a reference edge image. Then, it calculates the absolute difference between the two edge images at each pixel position, and obtains an edge difference map as the edge deviation component. For the feature deviation component, the server inputs the observed image and the reference image into a pre-trained VGG-16 network, extracts the feature maps output from the intermediate layers of the network, calculates the Euclidean distance between the two feature maps, and obtains a feature difference map as the feature deviation component. The server obtains preset first weight coefficients, second weight coefficients, and third weight coefficients, multiplies the pixel deviation component, edge deviation component, and feature deviation component by their respective weight coefficients, and then sums them to obtain the state deviation field. The region where the state deviates from the field with a higher value indicates a significant difference between the observed image and the reference image at that location, suggesting a higher probability of oil leakage.

[0137] In an exemplary embodiment, the aforementioned state deviation perception model includes an observed state feature extraction branch, a reference state feature extraction branch, a state deviation field feature extraction branch, and a state deviation fusion module. The phrase "processing the image to be detected and the state image using the pre-trained state deviation perception model to obtain a state deviation probability map or a state deviation mask" includes:

[0138] The detection image features are obtained by extracting features from the image to be detected through the observation state feature extraction branch; the state image features are obtained by extracting features from the state image through the reference state feature extraction branch; the state deviation field features are obtained by extracting features from the state deviation field through the state deviation field feature extraction branch; the detection image features, state image features, and state deviation field features are fused through the state deviation fusion module to obtain the fused deviation perception features; and the state deviation probability map or state deviation mask is output based on the fused deviation perception features.

[0139] The model comprises three sub-network modules: Observation State Feature Extraction Branch (a sub-network module within the State Deviation Perception Model) and Reference State Feature Extraction Branch (a sub-network module within the State Deviation Perception Model) and State Deviation Field Feature Extraction Branch (a sub-network module within the State Deviation Perception Model) and State Deviation Fusion Branch (a sub-network module within the State Deviation Perception Model) and State Deviation Fusion Branch (a sub-network module within the State Deviation Perception Model) and fusing the features extracted by the three sub-network modules. The State Deviation Probability Map is a probability map of the same size as the image to be detected, where the pixel value of each pixel represents the probability that the pixel belongs to an oil leak area. The State Deviation Mask is a binary image or binary matrix of the same size as the image to be detected, where the first identifier value indicates that the pixel belongs to an oil leak area, and the second identifier value indicates that the pixel belongs to a non-oil leak area.

[0140] In one embodiment of this application, the server inputs the image to be detected into the observation state feature extraction branch of the state deviation perception model. The observation state feature extraction branch employs a multi-layer convolutional neural network structure, including multiple convolutional layers, batch normalization layers, and activation function layers. The server performs progressive downsampling processing on the image to be detected through the observation state feature extraction branch, extracting multi-scale image features and finally outputting a detection image feature map. The server inputs the state image into the reference state feature extraction branch of the state deviation perception model. This branch has the same network structure as the observation state feature extraction branch (but different weight parameters), extracts features from the state image, and outputs a state image feature map. The server inputs the state deviation field into the state deviation field feature extraction branch of the state deviation perception model. This branch employs a convolutional neural network structure to extract features from the state deviation field and outputs a state deviation field feature map. The server inputs the detection image feature map, the state image feature map, and the state deviation field feature map into the state deviation fusion module. The state deviation fusion module concatenates the three feature maps along the channel dimension and then fuses the concatenated features through one or more convolutional layers to obtain a fused deviation perception feature map. The server outputs a state deviation probability map based on the fused deviation-aware feature map: a 1×1 convolutional layer maps the fused deviation-aware feature map into a single-channel output, and then a sigmoid activation function maps the output value to a probability range of 0 to 1, resulting in the state deviation probability map. The value of each pixel in this probability map represents the probability that the pixel belongs to the oil leak area.

[0141] In an exemplary embodiment, the above-mentioned "fusion processing of detection image features, state image features, and state deviation field features by the state deviation fusion module to obtain fused deviation perception features" includes:

[0142] The detected image features, state image features, and state deviation field features are concatenated to obtain concatenated features. The concatenated features are then processed with learnable parameters, and the result is normalized to obtain gating weights. The gating weights are then element-wise multiplied with the detected image features to obtain the first fusion component. The difference between the identity matrix and the gating weights is multiplied by the state image features to obtain the second fusion component. Finally, the first fusion component, the second fusion component, and the state deviation field features are summed to obtain the fused deviation perception features.

[0143] In this embodiment, the server acquires detection image features, state image features, and state deviation field features. All three feature maps are the same size (e.g., 32×32 pixels, 256 channels). The server concatenates the three feature maps along the channel dimension to obtain a concatenated feature map (32×32×768). The server performs function operations on the concatenated feature map and learnable parameters: a 1×1 convolutional layer (containing learnable parameters) is used to convolve the concatenated feature map, mapping 768 channels to 1 channel, outputting a gating weight map (32×32×1). The server normalizes the gating weight map: a Sigmoid activation function is used to map the values ​​of each pixel in the gating weight map to between 0 and 1, obtaining normalized gating weights (the value of each pixel represents the degree of confidence in the detection image features at that location; the closer the value is to 1, the more confident the detection image features are; the closer the value is to 0, the more confident the state image features are). The server performs element-wise multiplication of the normalized gating weights with the detection image features: the gating weight map is 32×32×1, and the detection image features are 32×32×256. The gating weight map is copied 256 times along the channel dimension and then element-wise multiplied with the detection image features at each channel and spatial location to obtain the first fusion component. The server calculates the difference between the identity matrix and the gating weights (1 minus the values ​​at each position in the gating weight map), copies the difference 256 times along the channel dimension, and then element-wise multiplies it with the state image features to obtain the second fusion component. The server then element-wise sums the first fusion component, the second fusion component, and the state deviation field features to obtain the fused deviation perception features. This fusion method adaptively adjusts the fusion ratio of the detection image features and the state image features at different spatial locations.

[0144] In one exemplary embodiment, such as Figure 8 As shown, the above-mentioned "determining the detection results of the oil leakage area on the surface of the device under test based on the state deviation probability map or state deviation mask" includes:

[0145] Step 801: Compare the probability values ​​of each pixel in the state deviation probability map with the probability threshold, and determine the pixels that are greater than or equal to the probability threshold as oil leakage pixels to obtain the initial oil leakage area mask.

[0146] In this embodiment, the server obtains a state deviation probability map output by the state deviation perception model. This probability map is a single-channel image with the same size as the image to be detected (e.g., 512×512 pixels). The value of each pixel is a floating-point number between 0 and 1, representing the probability that the pixel belongs to the oil leak area. The server obtains a preset probability threshold (e.g., 0.5). The server traverses each pixel position in the state deviation probability map and compares the probability value of each pixel with the probability threshold: if the probability value of the current pixel is greater than or equal to the probability threshold, the pixel is determined to be an oil leak pixel, and the corresponding position in the initial oil leak area mask is marked with a first identifier value (e.g., 255); if the probability value of the current pixel is less than the probability threshold, the pixel is determined to be a non-oil leak pixel, and the corresponding position in the initial oil leak area mask is marked with a second identifier value (e.g., 0). After traversing all pixels, the server obtains the initial oil leak area mask. The area with a pixel value of 255 in the initial oil leak area mask is the candidate oil leak area.

[0147] Step 802: Perform connected component filtering on the initial oil leak area mask to remove connected components with an area smaller than the area threshold, and obtain the filtered oil leak area mask.

[0148] Wherein, the connected region is a connected region composed of adjacent oil-leaking pixels in the initial oil-leaking region mask; the area threshold is a preset minimum number of pixels used to filter the connected region; the filtered oil-leaking region mask is a binary image or binary matrix after removing small noise areas.

[0149] A connected component is a region in a binary image that consists of adjacent (4-connected or 8-connected) leaking pixels. A 4-connected pixel is adjacent to pixels in its four directions (up, down, left, and right); an 8-connected pixel is adjacent to pixels in its eight directions (up, down, left, right, and four diagonal directions). Each connected component corresponds to an independent candidate leaking region instance.

[0150] The area threshold is a preset minimum number of pixels required to determine whether a connected component is a valid oil leak region. Connected components with an area smaller than the area threshold are usually noise points predicted by the model or minor false positives and should be discarded; connected components with an area greater than or equal to the area threshold are valid candidate oil leak regions.

[0151] In one embodiment of this application, the server obtains the initial oil leak region mask output in step 801. The server performs connected component analysis on the initial oil leak region mask: iterates through all pixels in the mask, clusters pixels belonging to the oil leak region and spatially adjacent (e.g., 8-connected) into the same connected component, and each connected component corresponds to a candidate oil leak region instance, thus obtaining a set of candidate oil leak region instances. The server obtains a preset area threshold (e.g., 100 pixels). The server iterates through each candidate oil leak region instance in the set of candidate oil leak region instances and counts the number of pixels contained in the current candidate oil leak region instance as the area of ​​the instance. The server compares the area of ​​the current instance with the area threshold: if the area of ​​the instance is less than the area threshold, the candidate oil leak region is determined to be noise or a false detection, and all pixel positions corresponding to the instance are marked with a second identifier value (0) in the mask, that is, the region is removed from the oil leak region; if the area of ​​the instance is greater than or equal to the area threshold, the oil leak region instance is retained. After the server has traversed all candidate oil leak region instances, it obtains the filtered oil leak region mask. The filtered oil leakage area mask retains oil leakage areas with an area greater than or equal to the area threshold, while removing small noise areas.

[0152] Step 803: Perform morphological closing operation and boundary smoothing on the selected oil leakage area mask in sequence to obtain the oil leakage area detection result; wherein, the oil leakage area detection result includes one or more of the following: oil leakage area mask, oil leakage area bounding box, oil leakage area area, oil leakage area proportion, and oil leakage confidence.

[0153] Among them, the morphological closing operation is used to fill the small holes and gaps inside the oil leak area, making the oil leak area more complete; the boundary smoothing process is used to remove the jagged burrs on the boundary of the oil leak area, making the boundary smoother and more natural.

[0154] Morphological closing is a fundamental operation in image morphology, consisting of two steps: dilation followed by erosion. Dilation expands the boundary of the leaking area, filling in small holes and gaps within it; erosion restores the area's size based on the dilated area while maintaining connectivity after filling. Through morphological closing, the internal voids of the leaking area are filled, improving the area's integrity.

[0155] Boundary smoothing is used to remove jagged burrs on the boundaries of the oil leak area. After thresholding and morphological operations, the boundaries of the oil leak area may have irregular jagged protrusions or depressions. Boundary smoothing can make the boundaries of the area smoother and more natural.

[0156] The bounding box of the oil leak area is the smallest rectangle that can define the oil leak area, which can be represented by the coordinates of the top left and bottom right corners or the coordinates of the center point and the width and height. The area of ​​the oil leak area is the total number of pixels contained in the oil leak area. The percentage of the oil leak area is the percentage of the oil leak area to the total surface area of ​​the device or the total area of ​​the image to be detected. The oil leak confidence score is a score reflecting the reliability of the oil leak detection.

[0157] In one embodiment of this application, the server obtains the filtered oil leak area mask output in step 802. The server performs a morphological closing operation on the filtered oil leak area mask: First, it uses a structuring element of a preset size (such as a 5×5 pixel rectangular structuring element) to perform a dilation operation on the mask. The center point of the structuring element is traversed through each pixel position in the mask. If there is at least one oil leak pixel in the neighborhood covered by the structuring element, the center position is marked as an oil leak area. The dilation operation can expand the boundary of the oil leak area and fill the small holes inside. Then, it uses the same structuring element to perform an erosion operation on the dilation result. The center point of the structuring element is traversed through the dilated mask. If all pixels in the neighborhood covered by the structuring element are oil leak areas, the center position is marked as an oil leak area. The erosion operation can restore the area size and maintain the connectivity after filling. After the morphological closing operation, the small holes and gaps inside the oil leak area are effectively filled. The server performs boundary smoothing on the mask after morphological closing operations: median filtering is used to smooth the mask boundaries, and the center pixel value is replaced with the median value of the neighboring pixels to remove jagged edges on the boundaries. The server uses the processed mask as the final mask for the oil leakage area.

[0158] In one exemplary embodiment, the above-described oil leak detection method further includes:

[0159] Step 1: Acquire the observation state image

[0160] First, acquire an image of the observed state of the device surface, denoted as:

[0161] in, Represents the observed state image. , , These represent the image height, width, and number of channels, respectively.

[0162] The observed status images can be from drone inspection images, inspection robot images, fixed camera images, handheld terminal images, or video frame images. The equipment can be transformers, circuit breakers, reactors, instrument transformers, bushings, on-load tap changers, valves, oil pipes, flanges, oil pumps, oil storage devices, etc.

[0163] In practice, preprocessing operations such as size normalization, brightness normalization, color normalization, noise suppression, image cropping, or target region extraction can be performed on the observed images to reduce the impact of different imaging conditions on subsequent detection.

[0164] Step 2: Construct observation state image-reference state image sample pairs

[0165] For each observed state image, a corresponding reference state image is constructed, denoted as:

[0166] in, This represents a reference state image.

[0167] The reference state image is not simply an image enhancement result, nor is it necessarily an oil-free image obtained from actual historical moments. Rather, it is an image after suppressing oil leakage characteristics while maintaining consistency in the main structure, spatial layout, material texture, and imaging style of the equipment. In other words, the reference state image represents the reference state of the equipment under conditions free from oil leakage interference.

[0168] The observed state image-reference state image sample pair is represented as follows:

[0169] in, Indicates the number of samples. Indicates the first A series of observation status images, This represents the corresponding reference state image.

[0170] In practice, the reference state image can be constructed in one or more of the following ways:

[0171] Manual repair method: The personnel marking the oil leak area manually edit or repair it to obtain a reference image of the equipment status;

[0172] Regularized texture filling method: The oil leak area is filled with texture based on the texture, color and structural information of the equipment surrounding the oil leak area;

[0173] Generative image editing method: Uses image generation model to reconstruct the state of the oil leak area and obtain a reference state image;

[0174] Semi-automatic assisted generation method: The initial reference image is generated by the automatic repair model, and then local correction is performed manually.

[0175] The sample pairs constructed in the above manner not only preserve the original structure and imaging conditions of the equipment, but also eliminate the characteristics of the oil leak state, providing supervision information for the subsequent training of the reference state generation model.

[0176] Step 3: Construct a training dataset for oil leak area annotation information and state deviation.

[0177] The oil leak area in the observed state image is labeled to obtain the oil leak area label mask, denoted as:

[0178]

[0179] in, Represents pixels This is an area with an oil leak. Represents pixels This area is not prone to oil leaks.

[0180] Based on the observed state image, the reference state image, and the labeled mask of the oil leak area, a state deviation training sample is constructed:

[0181] in, This indicates that the state has deviated from the training dataset.

[0182] Unlike traditional semantic segmentation datasets, the training dataset in this invention includes not only the image to be detected and the oil leak annotation, but also a reference state image with the same structure as the observed state image, thereby enabling explicit learning of the state deviation relationship between the observed state and the reference state.

[0183] Step 4: Train the reference state generation model

[0184] Construct a reference state generation model, denoted as:

[0185] The reference state generation model is used to generate a corresponding reference state image based on the observed state image, and its generation process is expressed as follows:

[0186] in, This represents the reference state image generated by the model.

[0187] The reference state generation model in this application employs a structurally consistent reference state generation mechanism. This mechanism is not merely used to obtain visually clearer images, but rather to generate reference state representations that can form an effective state comparison with the observed state images. Therefore, the following objectives must be met simultaneously during training:

[0188] The reference state image should be as close as possible to the actual constructed reference state image;

[0189] The reference condition image should maintain structural consistency with the observed condition image in non-leaking areas;

[0190] The reference state image should suppress the oil leak state characteristics in the oil leak area, so that the oil leak area forms a significant state deviation between the observed state and the reference state.

[0191] Step 4.1: Reconstructing the Loss

[0192] The reconstruction loss of the reference state generation model is expressed as:

[0193] in, This represents the generated reference state image. This represents the reference state image in a sample pair.

[0194] In another embodiment, the reconstruction loss may also include one or more of pixel loss, perceptual loss, structural similarity loss, or edge loss.

[0195] Step 4.2: Structural Preservation Constraint Loss

[0196] To prevent the reference state generation model from altering the main structure of the device, this invention introduces a structure preservation constraint loss. This loss is used to ensure that the generated image and the observed image remain consistent within the non-leaking structure region, and is expressed as follows:

[0197] in, The mask indicates the area of ​​the oil leak. Indicates the non-oil-leaking structural area. Represents the feature extraction function. This represents element-wise product.

[0198] By maintaining constraints through this structure, the reference state generation model can maintain the consistency of device outline, pipeline direction, connection parts and background texture when generating reference state images, thereby reducing false deviations in subsequent state deviation perception.

[0199] Step 4.3: State Deviation Enhancement Constraint Loss

[0200] To enhance the separability of the oil leak area between the observed state image and the reference state image, this invention introduces a state deviation enhancement constraint loss, which is expressed as:

[0201] in, This represents the adjustment coefficient. This indicates the number of pixels in the oil leakage area.

[0202] When the difference between the observed state image and the reference state image of the oil leak area is small Significantly large; when the differences in the oil leak areas are sufficiently widened, Reduce. By minimizing this loss, the state deviation response of the oil leak area can be enhanced.

[0203] Step 4.4: Total Loss of Reference State Generation Model

[0204] The total loss of the reference state generation model is expressed as:

[0205] in, and This represents the weighting coefficient.

[0206] Through the aforementioned joint loss constraints, the reference state generation model can generate a reference state image that maintains the consistency of the equipment structure while suppressing the characteristics of the oil leakage state.

[0207] Step 5: Construct the state deviation field

[0208] After obtaining the observed state image and the reference state image, a state deviation field is constructed, denoted as:

[0209] The state deviation field characterizes the degree of abnormal deviation of the current observed state relative to the reference state. The state deviation field can be obtained by a weighted combination of one or more of the following: pixel deviation component, edge deviation component, texture deviation component, and feature deviation component, and is expressed as:

[0210] in, Indicates pixel offset component Indicates edge deviation component, Indicates texture deviation component, Indicates the characteristic deviation component, This represents the weighting coefficient.

[0211] Pixel offset components are represented as:

[0212] The edge deviation component is represented as:

[0213] The characteristic deviation component is represented as:

[0214] in, This represents the edge extraction function. This represents a feature extraction network.

[0215] By constructing a state deviation field, this invention can distinguish the state deviation caused by oil leakage from non-target factors such as equipment structure, background texture, and lighting changes, thereby improving the robustness of oil leakage area location.

[0216] Step 6: Train the state deviation perception model

[0217] Construct a state deviation perception model, denoted as:

[0218] The state deviation sensing model is used to output a state deviation probability map based on the observed state image, the reference state image, and the state deviation field, which is expressed as follows:

[0219]

[0220] in, This represents a pixel-level state deviation probability map.

[0221] In practice, the state deviation perception model may include an observed state feature extraction branch, a reference state feature extraction branch, and a state deviation fusion module.

[0222] The observation state feature extraction branch is used to extract observation state image features:

[0223] The reference state feature extraction branch is used to extract features from the reference state image:

[0224] The state deviation field feature extraction branch is used to extract state deviation field features:

[0225]

[0226] in, , , These represent different feature extraction functions.

[0227] Step 6.1: State Deviation Gating Fusion

[0228] To adaptively fuse observed state features, reference state features, and state deviation field features, this invention designs a state deviation-gated fusion method, which is expressed as follows:

[0229] in, Indicates feature splicing, Indicates learnable parameters, This represents the Sigmoid function. Indicates the gating weight, This indicates the deviation from the perceived features after fusion.

[0230] By using state deviation gating fusion, the model can adaptively adjust the weights of observed state information, reference state information, and deviation field information according to different scenarios, thereby enhancing the response to the real oil leak area and suppressing interference from shadows, rust, reflections, background textures, and other factors.

[0231] Step 6.2: Loss function for the state deviation perception model

[0232] The training loss for the state deviation perception model can be binary cross-entropy loss, Dice loss, boundary loss, or a combination thereof. For example, the total loss is expressed as:

[0233] in, Represents the cross-entropy loss in binary classification. This indicates Dice's loss. This represents the weighting coefficient.

[0234] By minimizing this loss, the state deviation perception model can output a state deviation probability map that is consistent with the label mask of the oil leak area.

[0235] Step 7: Engineering Reasoning in the Detection Phase

[0236] During the detection phase, for any image of the observed state to be detected... First, input the reference state generation model to obtain the reference state image:

[0237] Then, a state deviation field is constructed based on the observed state image to be detected and the generated reference state image:

[0238] in, This represents the state deviation field construction function.

[0239] The observed state image to be detected, the reference state image, and the state deviation field are input into the state deviation perception model to obtain the state deviation probability map:

[0240] Threshold segmentation is performed on the state deviation probability map to obtain the initial oil leak area mask:

[0241]

[0242] in, This represents the threshold.

[0243] Subsequently, post-processing such as connected component screening, area filtering, morphological closing operation, and boundary smoothing is performed on the initial oil leak area mask to obtain the final oil leak area detection result.

[0244] The final output may include one or more of the following: oil leak area mask, oil leak area bounding box, oil leak area area, oil leak area percentage, and oil leak confidence level.

[0245] In an exemplary embodiment, the process of determining the reference image includes:

[0246] The oil leak area in the observation image is obtained; based on the appearance information of the equipment around the oil leak area, the oil leak area is subjected to regularized texture filling processing to obtain an intermediate image; the intermediate image is subjected to local correction processing to obtain a reference image.

[0247] In this embodiment, the server acquires the observed image and the location information of the oil leak area within it. The oil leak area location information is provided as an oil leak area annotation mask, which is a binary image of the same size as the observed image, where the pixel value corresponding to the oil leak area is 1, and the pixel value corresponding to the non-oil leak area is 0. The server determines the location of the oil leak area in the observed image based on the oil leak area annotation mask. The server extracts device appearance information within a preset neighborhood range around the oil leak area (e.g., a ring-shaped area extending 20 pixels outward from the boundary of the oil leak area), including the texture pattern, color statistical distribution, and structural orientation within this area. Based on the extracted device appearance information, the server performs regularized texture filling processing on the oil leak area: using a sample-based texture synthesis algorithm, texture blocks are sampled from the normal areas surrounding the oil leak area, and the sampled texture blocks are gradually filled into the oil leak area according to the principle of texture direction consistency, ensuring that the filled texture remains continuous with the surrounding area in direction, scale, and color. After completing texture filling for all oil leak areas, the server obtains an intermediate image. The server performs local correction processing on the intermediate image: it detects the boundary between the filled area and the surrounding area, performs color blending and edge feathering on pixels near the boundary to eliminate visual discontinuities at the boundary; and it adjusts the average brightness and contrast of the filled area to match those of the surrounding area. The server uses the locally corrected image as a reference image.

[0248] In one exemplary embodiment, the above method may further include:

[0249] Step 1: Obtain the oil leak area marking mask, and determine the non-oil leak structure area based on the oil leak area marking mask.

[0250] Step 2: Perform element-wise multiplication of the non-oil-leaking structural area with the observed image to obtain the first product result.

[0251] Step 3: Perform element-wise multiplication of the non-oil-leaking structural region with the training image to obtain the second product result.

[0252] Step 4: Extract features from the first product result and the second product result respectively to obtain the first feature and the second feature.

[0253] Step 5: Determine the absolute value of the difference between the first feature and the second feature as the structure preservation constraint loss.

[0254] Step 6: Obtain the oil leak area marking mask and determine the oil leak area based on the oil leak area marking mask.

[0255] Step 7: Calculate the pixel difference between the observed image and the training image at each pixel point in the oil leak area, and sum the pixel differences at each pixel point and divide by the number of pixels in the oil leak area to obtain the average pixel difference.

[0256] Step 8: Take the negative exponent of the product of the average pixel difference and the preset adjustment coefficient to obtain the state deviation enhancement constraint loss.

[0257] Step 9: Input the observed images into the state model to be trained to generate training images.

[0258] Step 10: Determine the reconstruction loss between the observed image and the training image.

[0259] Step 11: Obtain the first weight coefficient and the second weight coefficient.

[0260] Step 12: Calculate the first weighted result of the structure preservation constraint loss and the first weight coefficient.

[0261] Step 13: Calculate the second weighted result of the state deviation enhancement constraint loss and the second weight coefficient.

[0262] Step 14: The reconstruction loss, the first weighted result, and the second weighted result are summed to obtain the structural consistency loss value.

[0263] Step 15: Update the model parameters of the state model to be trained based on the structural consistency loss value.

[0264] Step 16: Repeat the steps of generating training images, calculating structural consistency loss, and updating model parameters until the preset training stopping conditions are met to obtain the target state model. The preset training stopping conditions include one of the following: structural consistency loss is less than a preset threshold, the number of training rounds reaches a preset number of rounds, or the change in structural consistency loss in multiple consecutive iterations is less than a preset change threshold.

[0265] Step 17: Obtain the image of the surface of the device to be inspected.

[0266] Step 18: Process the image to be detected using a pre-trained target state model to obtain a state image corresponding to the image to be detected; the target state model is trained based on the observed image and the reference image; the reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0267] Step 19: Determine the pixel offset component, edge offset component, texture offset component, and feature offset component between the observed image and the reference image, respectively.

[0268] Step 20: Perform weighted summation on the pixel deviation component, edge deviation component, texture deviation component, and feature deviation component to obtain the state deviation field; the state deviation field is used to characterize the degree of abnormal deviation of the device surface state reflected by the image to be detected from the device surface state reflected by the state image.

[0269] Step 21: Input the image to be detected, the state image, and the state deviation field into the pre-trained state deviation perception model, and output the state deviation probability map or the state deviation mask; the state deviation perception model is trained based on the observed image, the reference image, and the oil leak area annotation information.

[0270] Step 22: Based on the state deviation probability map or state deviation mask, determine the detection results of the oil leakage area on the surface of the device to be tested.

[0271] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0272] Based on the same inventive concept, this application also provides an oil leak detection device for implementing the oil leak detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the oil leak detection device provided below can be found in the limitations of the oil leak detection method described above, and will not be repeated here.

[0273] In one exemplary embodiment, such as Figure 9 As shown, an oil leak detection device is provided, comprising: an acquisition module 901, a first processing module 902, a second processing module 903, and a determination module 904, wherein:

[0274] The acquisition module 901 is used to acquire the image of the surface of the device to be inspected.

[0275] The first processing module 902 is used to process the image to be detected through a pre-trained target state model to obtain a state image corresponding to the image to be detected. The target state model is trained based on the observed image and the reference image. The reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image.

[0276] The second processing module 903 is used to process the image to be detected and the state image through a pre-trained state deviation perception model to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image and the oil leak area annotation information.

[0277] The determination module 904 is used to determine the detection results of the oil leakage area on the surface of the device to be tested based on the state deviation probability map or the state deviation mask.

[0278] In an exemplary embodiment, the second processing module 903 is specifically used to construct a state deviation field based on the image to be detected and the state image; the state deviation field is used to characterize the degree of abnormal deviation of the device surface state reflected by the image to be detected relative to the device surface state reflected by the state image; the image to be detected, the state image and the state deviation field are input into a pre-trained state deviation perception model, and the output state deviation probability map or state deviation mask is output.

[0279] In an exemplary embodiment, the above-described oil leak detection device is further configured to input the observed image into the state model to be trained to generate a training image; determine the structural consistency loss value between the observed image and the training image; update the model parameters of the state model to be trained based on the structural consistency loss value; and repeatedly execute the steps of generating training images, calculating structural consistency loss value, and updating model parameters until a preset training stopping condition is met to obtain the target state model; the preset training stopping condition includes one of the following: the structural consistency loss is less than a preset threshold, the number of training rounds reaches a preset number of rounds, or the change in structural consistency loss in consecutive iterations is less than a preset change threshold.

[0280] In an exemplary embodiment, the above-described oil leak detection device is further configured to determine the reconstruction loss between the observed image and the training image; obtain a first weighting coefficient and a second weighting coefficient; calculate a first weighted result of the structure preservation constraint loss and the first weighting coefficient; calculate a second weighted result of the state deviation enhancement constraint loss and the second weighting coefficient; and perform a weighted summation of the reconstruction loss, the first weighting result, and the second weighting result to obtain a structure consistency loss value.

[0281] In an exemplary embodiment, the above-mentioned oil leak detection device is further configured to acquire an oil leak area label mask, and determine a non-oil leak structure area based on the oil leak area label mask; perform element-wise multiplication of the non-oil leak structure area with the observed image to obtain a first product result; perform element-wise multiplication of the non-oil leak structure area with the training image to obtain a second product result; extract features from the first product result and the second product result respectively to obtain a first feature and a second feature; and determine the absolute value of the difference between the first feature and the second feature as the structure preservation constraint loss.

[0282] In an exemplary embodiment, the above-mentioned oil leak detection device is further configured to acquire an oil leak area label mask, and determine the oil leak area based on the oil leak area label mask; calculate the pixel difference between the observed image and the training image at each pixel point in the oil leak area, and sum the pixel differences at each pixel point and divide by the number of pixels in the oil leak area to obtain the average pixel difference; and take the negative exponent of the product of the average pixel difference and the preset adjustment coefficient to obtain the state deviation enhancement constraint loss.

[0283] In an exemplary embodiment, the second processing module 903 is specifically used to determine the pixel deviation component, edge deviation component, texture deviation component, and feature deviation component between the observed image and the reference image, respectively; and to perform weighted summation processing on the pixel deviation component, edge deviation component, texture deviation component, and feature deviation component to obtain the state deviation field.

[0284] In an exemplary embodiment, the state deviation perception model includes an observation state feature extraction branch, a reference state feature extraction branch, a state deviation field feature extraction branch, and a state deviation fusion module. The second processing module 903 is specifically used to extract features from the image to be detected through the observation state feature extraction branch to obtain detection image features; to extract features from the state image through the reference state feature extraction branch to obtain state image features; to extract features from the state deviation field through the state deviation field feature extraction branch to obtain state deviation field features; to fuse the detection image features, state image features, and state deviation field features through the state deviation fusion module to obtain fused deviation perception features; and to output a state deviation probability map or a state deviation mask based on the fused deviation perception features.

[0285] In an exemplary embodiment, the second processing module 903 is specifically used to perform feature concatenation processing on the detected image features, state image features, and state deviation field features to obtain concatenated features; perform function operation processing on the concatenated features and learnable parameters, and normalize the operation result to obtain gating weights; perform element-wise multiplication of the gating weights and the detected image features to obtain a first fusion component; multiply the difference between the identity matrix and the gating weights by the state image features to obtain a second fusion component; and sum the first fusion component, the second fusion component, and the state deviation field features to obtain the fused deviation perception features.

[0286] In an exemplary embodiment, the determining module 904 is specifically used to compare the probability value of each pixel in the state deviation probability map with a probability threshold, and determine the pixels with a probability value greater than or equal to the probability threshold as oil leakage pixels to obtain an initial oil leakage region mask; perform connected component filtering on the initial oil leakage region mask to remove connected components with an area smaller than an area threshold to obtain a filtered oil leakage region mask; and perform morphological closing operation and boundary smoothing on the filtered oil leakage region mask in sequence to obtain an oil leakage region detection result; wherein, the oil leakage region detection result includes one or more of the following: oil leakage region mask, oil leakage region bounding box, oil leakage region area, oil leakage region proportion, and oil leakage confidence.

[0287] Each module in the aforementioned oil leak detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0288] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0289] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0290] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0291] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0292] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0293] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting oil leaks, characterized in that, The method includes: Acquire an image of the surface of the device to be inspected; The image to be detected is processed by a pre-trained target state model to obtain a state image corresponding to the image to be detected; the target state model is trained based on the observed image and a reference image; the reference image is an image obtained by suppressing the visual features corresponding to the oil leakage area of ​​the device to be detected while maintaining the same appearance attributes of the device as the observed image. The image to be detected and the state image are processed by a pre-trained state deviation perception model to obtain a state deviation probability map or a state deviation mask; the state deviation perception model is trained based on the observed image, the reference image and the oil leak area annotation information. Based on the state deviation probability map or state deviation mask, the detection result of the oil leakage area on the surface of the device to be tested is determined; The step of processing the image to be detected and the state image using a pre-trained state deviation perception model to obtain a state deviation probability map or a state deviation mask includes: A state deviation field is constructed based on the image to be detected and the state image; the state deviation field is used to characterize the degree of abnormal deviation of the device surface state reflected by the image to be detected relative to the device surface state reflected by the state image. The image to be detected, the state image, and the state deviation field are input into a pre-trained state deviation perception model, which outputs the state deviation probability map or the state deviation mask. The state deviation perception model includes an observed state feature extraction branch, a reference state feature extraction branch, a state deviation field feature extraction branch, and a state deviation fusion module; the process of processing the image to be detected and the state image through the pre-trained state deviation perception model to obtain a state deviation probability map or a state deviation mask includes: The detection image features are obtained by extracting features from the image to be detected through the observation state feature extraction branch; The state image features are obtained by extracting features from the state image through the reference state feature extraction branch; The state deviation field features are extracted by the state deviation field feature extraction branch to obtain the state deviation field features; The state deviation fusion module fuses the detected image features, the state image features, and the state deviation field features to obtain the fused deviation perception features. The state deviation probability map or the state deviation mask is output based on the fused deviation perception features.

2. The method according to claim 1, characterized in that, The training process of the target state model includes: The observed images are input into the state model to be trained to generate training images; Determine the structural consistency loss value between the observed image and the training image; Update the model parameters of the training state model based on the structural consistency loss value; Repeat the steps of generating training images, calculating structural consistency loss, and updating model parameters until a preset training stopping condition is met to obtain the target state model. The preset training stopping condition includes one of the following: the structural consistency loss is less than a preset threshold, the number of training rounds reaches a preset number of rounds, or the change in structural consistency loss in multiple consecutive iterations is less than a preset change threshold.

3. The method according to claim 1, characterized in that, Determining the structural consistency loss value between the observed image and the training image includes: Determine the reconstruction loss between the observed image and the training image; Obtain the first weighting coefficient and the second weighting coefficient; The first weighted result of the structure preservation constraint loss and the first weight coefficient is calculated. Calculate the second weighted result of the state deviation enhancement constraint loss and the second weight coefficient; The reconstruction loss, the first weighted result, and the second weighted result are summed to obtain the structural consistency loss value.

4. The method according to claim 3, characterized in that, The process of determining the structural constraint loss includes: Obtain a mask indicating the oil leak area, and determine the non-oil leak structural area based on the mask indicating the oil leak area; The non-oil-leaking structural region is multiplied element-wise with the observed image to obtain the first product result; The non-oil-leaking structural region is multiplied element-wise with the training image to obtain a second product result. Feature extraction is performed on the first product result and the second product result respectively to obtain the first feature and the second feature; The absolute value of the difference between the first feature and the second feature is determined as the structure retention constraint loss.

5. The method according to claim 3, characterized in that, The process of determining the state deviation enhancement constraint loss includes: Obtain a mask indicating the oil leak area, and determine the oil leak area based on the mask. Calculate the pixel difference between the observed image and the training image at each pixel point in the oil leak area, and sum the pixel differences at each pixel point and divide by the number of pixels in the oil leak area to obtain the average pixel difference; The negative exponent of the product of the average pixel difference and the preset adjustment coefficient is used to obtain the state deviation enhancement constraint loss.

6. The method according to claim 1, characterized in that, The construction of the state deviation field based on the observed image and the reference image includes: The pixel offset component, edge offset component, texture offset component, and feature offset component between the observed image and the reference image are determined respectively. The state deviation field is obtained by performing a weighted summation on the pixel deviation component, the edge deviation component, the texture deviation component, and the feature deviation component.

7. The method according to claim 1, characterized in that, The process of fusing the detected image features, the state image features, and the state deviation field features through the state deviation fusion module to obtain the fused deviation perception features includes: The detected image features, the state image features, and the state deviation field features are concatenated to obtain concatenated features. The splicing features and learnable parameters are processed by function operation, and the operation results are normalized to obtain the gating weights; The first fusion component is obtained by element-wise multiplying the gating weights with the detected image features; The difference between the identity matrix and the gate weights is multiplied by the state image features to obtain the second fusion component; The first fusion component, the second fusion component, and the state deviation field feature are summed to obtain the fused deviation perception feature.

8. The method according to claim 1, characterized in that, The step of determining the detection result of the oil leakage area on the surface of the device under test based on the state deviation probability map or state deviation mask includes: The probability values ​​of each pixel in the state deviation probability map are compared with the probability threshold, and the pixels that are greater than or equal to the probability threshold are determined to be oil leakage pixels, thus obtaining the initial oil leakage area mask. The initial oil leak area mask is subjected to connected component filtering to remove connected components with an area smaller than the area threshold, resulting in a filtered oil leak area mask. The filtered oil leakage area mask is subjected to morphological closing operation and boundary smoothing processing in sequence to obtain the oil leakage area detection result; wherein, the oil leakage area detection result includes one or more of the following: oil leakage area mask, oil leakage area bounding box, oil leakage area, oil leakage area ratio, and oil leakage confidence level.

Images