A method and system for pipeline defect identification based on spiking neural networks

Abstract

This invention belongs to the field of computer vision technology, specifically relating to a method and system for pipeline defect recognition based on a spiking neural network. The method includes: acquiring and preprocessing an image of the pipeline's inner wall; converting it into a sparse pulse sequence using pulse differential encoding based on two-dimensional discrete cosine transform; extracting high-level semantic features using a lightweight spiking neural network containing pulse depth-separable convolution and a frequency-domain guided aggregation attention module; and finally, outputting the defect category and physical quantification indicators in parallel using a dual-head decoder. This achieves real-time, low-power detection of pipeline defects at resource-constrained edge computing environments, and simultaneously outputs quantification results such as crack length and corrosion area percentage, providing data support for precise pipeline maintenance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for pipeline defect identification based on spiking neural networks. Background Technology

[0002] Urban water supply and drainage pipelines are a crucial component of urban infrastructure. Affected by multiple factors such as geological subsidence, pipe aging, media corrosion, and external loads, pipelines are prone to defects such as cracks, corrosion, and root intrusion. Current pipeline inspection methods primarily rely on transmitting large amounts of video data back to ground workstations / cloud servers for processing and on real-time inspection using mobile robots equipped with vision sensors. However, the enclosed environment inside pipelines, unstable wireless signal transmission, and the large volume of high-resolution video data make it difficult to meet real-time requirements. Furthermore, the limited battery capacity of mobile robots makes it difficult to support long-term inference operations on high-power GPU servers. Therefore, how to acquire pipeline status information in real-time using low-power processors at the resource-constrained edge of a robot has become a critical problem that urgently needs to be solved.

[0003] After conducting research, the inventors discovered the following pain points in existing pipeline defect detection technologies:

[0004] 1. Traditional spiking neural networks directly convert the convolutional neural network structure into a spiking form, or simply use leakage integral-trigger neurons. This makes it difficult to effectively extract high-frequency edge features of fine cracks in low-contrast environments with uneven lighting and water mist interference inside pipes.

[0005] 2. Existing pulse attention mechanisms often use the Transformer architecture, retaining the calculation of the value matrix, resulting in high memory usage and failing to provide targeted frequency domain enhancement for defective textures.

[0006] 3. Current pipeline defect detection methods can mostly only perform qualitative analysis (determining whether there are defects or the type of defects), and cannot accurately give the severity of defects (such as crack length, corrosion area ratio), resulting in a lack of quantitative basis for maintenance decisions.

[0007] 4. Existing high-precision pipeline defect detection models typically rely on computationally intensive deep network architectures, resulting in a huge number of parameters and high inference latency. On embedded edge devices with severely limited computing power, power consumption, and storage space carried by pipeline robots, such models struggle to achieve real-time inference. Summary of the Invention

[0008] The purpose of this invention is to provide a pipeline defect identification method and system based on spiking neural networks, so as to solve the obvious deficiencies of existing technologies in terms of real-time performance, power consumption control, noise resistance accuracy and quantitative evaluation capability at the edge end.

[0009] The present invention achieves the above objectives through the following technical solutions:

[0010] In a first aspect, the present invention proposes a pipeline defect identification method, applied to an edge computing platform for mobile devices equipped with image sensors, the method comprising:

[0011] A grayscale image sequence of the inner wall of the pipe is obtained, and the grayscale image sequence is preprocessed by normalization to obtain a standardized image tensor.

[0012] The image tensor is subjected to pulse differential encoding based on two-dimensional discrete cosine transform to obtain a three-dimensional binary pulse tensor;

[0013] The three-dimensional binary pulse tensor is input into a pre-constructed feature extraction network for advanced semantic feature extraction, and an advanced semantic feature map is output. The feature extraction network includes a pulse depth separable convolution module and a frequency domain guided pulse aggregation self-attention module, which are used to enhance defect features and suppress background noise in the frequency domain.

[0014] The high-level semantic feature map is input into the decoder, and qualitative classification decoding and quantitative regression decoding are performed respectively, and the category label and physical quantitative index of pipeline defects are output in parallel; wherein, the physical quantitative index includes crack length or corrosion area ratio.

[0015] In a preferred embodiment, performing pulse differential coding based on two-dimensional discrete cosine transform on the image tensor includes:

[0016] The normalized grayscale image sequence is divided into multiple non-overlapping local patches;

[0017] Perform a two-dimensional discrete cosine transform on each local patch to obtain a spectrum matrix containing DC and AC components;

[0018] Calculate the energy proportion of high-frequency regions in the spectrum matrix of each patch, and dynamically set the firing threshold of neurons in the corresponding region based on the energy proportion through a reverse mapping strategy;

[0019] The adaptive threshold matrix is ​​used to drive the leakage integral-triggered neuron, which performs differential state updates by combining the current pixel input with the historical membrane potential. When the membrane potential exceeds the threshold, a pulse is emitted and a soft reset is performed, outputting a three-dimensional binary pulse tensor.

[0020] In a preferred embodiment, inputting the three-dimensional binary pulse tensor into a pre-constructed feature extraction network for advanced semantic feature extraction includes:

[0021] The three-dimensional binary pulse tensor is input into a backbone network consisting of a pulse initial layer and multiple cascaded pulse depth separable convolutional units for stepwise feature extraction.

[0022] The initial pulse layer performs preliminary convolution and spatial downsampling on the input pulse tensor to output a primary feature map.

[0023] The multiple cascaded pulse depth separable convolutional units sequentially receive the feature map output by the previous unit, extract spatial features through pulse depth convolution, fuse channel information through point convolution, and cooperate with step-by-step downsampling operations to finally output an intermediate feature map containing multi-scale semantic information.

[0024] The intermediate feature map is input into a frequency-domain guided pulse aggregation self-attention module for processing to obtain a high-level semantic feature map.

[0025] In a preferred embodiment, the step of inputting the intermediate feature map into a frequency-domain guided pulse aggregation self-attention module for processing includes:

[0026] The intermediate feature maps are mapped to generate query tensors and key-value tensors, respectively.

[0027] A two-dimensional discrete cosine transform is performed on the key value tensor, and the high-frequency coefficients of the corresponding defect texture features are enhanced in the frequency domain using a preset filter. Then, the key value tensor with frequency domain enhancement is obtained by inverse transformation.

[0028] Calculate the Hadamard product of the query tensor and the frequency domain enhanced key tensor, and generate a spatial attention weight map through global average pooling in the channel dimension;

[0029] The spatial attention weight map is applied to the intermediate feature map to generate the high-level semantic feature map, in which background noise is suppressed and defect features are enhanced in a targeted manner.

[0030] In a preferred embodiment, the high-level semantic feature map is input to the decoder to perform qualitative classification decoding, including:

[0031] Global spatiotemporal average pooling is performed on the high-level semantic feature map to calculate the average pulse firing rate of each feature channel in the time and space dimensions, thereby obtaining a one-dimensional feature vector;

[0032] The one-dimensional feature vector is input into a fully connected classification layer and processed by the Softmax function to output the probability distribution of pipeline defect categories;

[0033] Defect category labels are determined based on the probability distribution.

[0034] In a preferred embodiment, the high-level semantic feature map is input to the decoder to perform quantitative regression decoding, including:

[0035] The high-level semantic feature map is subjected to channel-weighted filtering to enhance the channel response sensitive to physical quantization, thereby obtaining a quantization-sensitive feature map;

[0036] Perform global pulse integration on the quantization-sensitive feature map and count the total effective pulse output at all time steps and spatial locations;

[0037] The total effective pulse output is input into the regression mapping layer and converted into a physical quantitative indicator reflecting the severity of the defect.

[0038] In a preferred embodiment, the feature extraction network and decoder are obtained through the following training steps:

[0039] Obtain a training dataset containing images of the pipe's inner wall, corresponding defect category labels, and true values ​​of physical quantification indicators;

[0040] A multi-task joint loss function is constructed, which is composed of a weighted sum of classification cross-entropy loss and regression mean square error loss, and is used to simultaneously constrain the defect category probability distribution and physical quantification index output by the decoder.

[0041] The surrogate gradient method is adopted to approximate the non-differentiable step of the neuron's spiking function in the spiking neural network with a smooth function, and combined with the backpropagation algorithm over time, the gradient of the joint loss function with respect to the parameters in the feature extraction network and the decoder is calculated.

[0042] Based on the gradient, the convolutional kernel weights, frequency domain enhancement coefficients, neuron dynamics parameters in the feature extraction network, and the regression mapping weights in the decoder are jointly optimized and iteratively updated until the model converges.

[0043] The trained network model is deployed to the edge computing unit of the mobile robot.

[0044] Secondly, this invention proposes a pipeline defect identification system, applied to the pipeline defect identification method described above. The system is installed on the edge computing platform of a pipeline inspection device and includes:

[0045] The image acquisition and preprocessing module is used to acquire a grayscale image sequence of the inner wall of the pipe and perform normalization preprocessing on the grayscale image sequence to obtain a standardized image tensor.

[0046] The pulse differential coding module is used to perform pulse differential coding based on two-dimensional discrete cosine transform on the image tensor to obtain a three-dimensional binary pulse tensor;

[0047] The feature extraction network module is used to input the three-dimensional binary pulse tensor into a pre-constructed feature extraction network for advanced semantic feature extraction and output an advanced semantic feature map. The feature extraction network includes a pulse depth separable convolution module and a frequency domain guided pulse aggregation self-attention module, which are used to enhance defect features and suppress background noise in the frequency domain.

[0048] A dual-head decoding module is used to input the high-level semantic feature map into the decoder, perform qualitative classification decoding and quantitative regression decoding respectively, and output the category label and physical quantitative index of the pipeline defect in parallel; wherein, the physical quantitative index includes crack length or corrosion area ratio;

[0049] The communication module is used to encapsulate the defect category label and the physical quantification index into a structured inspection report, and transmit it back to the ground control terminal in real time via a wireless communication link for display and alarm.

[0050] Thirdly, the present invention proposes a pipeline inspection device, comprising:

[0051] One or more processors;

[0052] Memory;

[0053] and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, the computer programs including instructions for performing the methods described above.

[0054] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0055] The beneficial effects of this invention are as follows:

[0056] This invention significantly reduces the amount of multiplication and accumulation operations in the network by combining pulse depthwise separable convolution with a lightweight aggregation attention mechanism that removes the value matrix. This results in a substantial reduction in computational power consumption and memory usage, enabling the model to run efficiently on resource-constrained edge computing platforms for pipeline robots and achieving real-time inference. Secondly, this invention introduces two-dimensional discrete cosine transform for frequency domain analysis and guidance, allowing the network to dynamically adjust its attention based on the high-frequency characteristics of defects in the frequency domain. This effectively suppresses interference from low-frequency environmental noise such as uneven lighting, water mist, and reflections within the pipeline, improving the ability to capture and detect defects such as fine cracks. Finally, this invention employs a dual-head pulse decoder to perform qualitative identification of defect categories and quantitative regression of physical scales in parallel within a single network. It can directly output quantitative indicators such as crack length and corrosion area ratio, overcoming the limitation of traditional methods that can only perform qualitative classification. Attached Figure Description

[0057] Figure 1 This is a schematic flowchart of a pipeline defect identification method based on a spiking neural network in a specific embodiment of the present invention;

[0058] Figure 2 This is an architecture diagram of the pipeline defect detection model in a specific embodiment of the present invention;

[0059] Figure 3 This is an architecture diagram of the pulse depth separable convolution module in a specific embodiment of the present invention;

[0060] Figure 4 This is a diagram of the pulse aggregation self-attention module architecture in a specific embodiment of the present invention;

[0061] Figure 5 This is a block diagram of a pipeline defect identification system based on a spiking neural network in a specific embodiment of the present invention. Detailed Implementation

[0062] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0063] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0064] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0065] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0066] The technical terms that may appear in the embodiments of this application are explained below.

[0067] Spiking Neural Network (SNN): A third-generation artificial neural network in which neurons communicate through discrete sequences of pulses (or spikes), simulating the dynamic behavior of biological neurons. It has event-driven and spatiotemporal information processing capabilities, and has advantages in low-power edge computing scenarios.

[0068] Two-dimensional discrete cosine transform (2D-DCT): A linear transform that converts an image from the spatial domain to the frequency domain, concentrating the energy of the image on a few low-frequency coefficients, making it easier to analyze and process the frequency components of the image.

[0069] Leaky Integrate-and-Fire (LIF) neuron model: A widely used computational model of spiking neurons whose membrane potential leaks (decays) over time. When the accumulated input causes its membrane potential to exceed a certain threshold, the neuron fires a spiking signal.

[0070] Pulse depthwise separable convolution: A lightweight convolution operation suitable for SNNs that decomposes standard convolution into two steps: depthwise convolution and pointwise convolution, which can significantly reduce computation and the number of parameters.

[0071] Surrogate Gradient: A method for training SNNs that approximates a non-differentiable step function by defining a smooth function that is differentiable at the moment of impulse firing, thus enabling gradient-based backpropagation algorithms to be applied to the training of SNNs.

[0072] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0073] Example 1

[0074] See Figure 1 and Figure 2 This invention provides a pipeline defect identification method based on a spiking neural network. This method can be applied to edge computing platforms (such as NVIDIA Jetson series modules) of pipeline inspection mobile robots or other mobile devices equipped with image sensors (such as CMOS cameras). Figure 1As shown, the method mainly includes the following steps S100 to S400.

[0075] S100: Obtain the grayscale image sequence of the inner wall of the pipe, and perform normalization preprocessing on the grayscale image sequence to obtain a standardized image tensor.

[0076] Specifically, a camera at the front end of a mobile robot captures real-time RGB image video streams of the pipe's inner wall. To adapt to the dim environment inside the pipe, a ring of LED fill lights is placed around the camera to provide uniform illumination. The camera has automatic exposure and white balance functions to cope with brightness changes caused by water reflection or corrosion on the pipe wall. The captured images are transmitted to an edge computing platform via a high-speed interface (such as MIPI CSI-2).

[0077] Within the edge computing module, the following basic preprocessing steps are performed on the raw video stream:

[0078] First, the input RGB image is processed and frames are extracted to adjust its resolution to 256×256 pixels. Considering that the characteristics of cracks and corrosion in the pipe inner wall image are mainly reflected in brightness differences rather than color differences, a weighted average method that conforms to the characteristics of human vision is used to convert the RGB three-channel image into a single-channel grayscale image. For the pixel at coordinates (x,y) in the image, its grayscale value G(x,y) is calculated as follows:

[0079]

[0080] In the formula, R(x,y), G(x,y), and B(x,y) represent the component values ​​of the red, green, and blue channels of the original image, respectively. This formula gives the green channel the highest weight, effectively preserving the texture details of the pipe wall while compressing the data volume to 1 / 3 of the original. Furthermore, to match the input voltage range of the leakage integral-trigger neuron in the subsequent spiking neural network and to eliminate the reference drift caused by differences in illumination intensity, a linear normalization operation is performed on the grayscale image, mapping the pixel intensity to the [0,1] interval. The normalized pixel value I... norm The formula for calculating (x, y) is as follows:

[0081]

[0082] In the formula, G min With G maxThese represent the grayscale values ​​of 0 and 255 in the current image frame, respectively. The normalized image tensor not only eliminates the influence of dimensions but also accelerates the convergence speed of Poisson coding in the inference phase of the SNN. The preprocessed, normalized image frame is directly loaded into the GPU memory, serving as the input tensor for subsequent discrete cosine transform frequency domain feature extraction and F-Mobile SNN spiking neural network inference, completing the entire process from physical acquisition to digital signal preparation.

[0083] S200: Perform pulse differential encoding based on two-dimensional discrete cosine transform (2D-DCT) on the image tensor to obtain a three-dimensional binary pulse tensor.

[0084] This step aims to convert continuous grayscale image frames into discrete pulse sequences suitable for spiking neural network processing and to enhance the signal-to-noise ratio using frequency domain information.

[0085] In a preferred embodiment, S200 includes the following sub-steps:

[0086] S210: Image segmentation and local frequency domain transformation. Receive the normalized grayscale image output from step S100. norm To capture local texture features, a sliding window strategy is used to divide the image into non-overlapping 8×8 local patches P(x, y). A two-dimensional discrete cosine transform is performed on each local patch P to convert it from the spatial domain to the frequency domain. The formula for calculating the frequency domain coefficients F(u, v) for pixel coordinates (i, j) within a patch is as follows:

[0087]

[0088] In the formula, u,v ∈ [0,7] are frequency coordinates, α() is the normalization coefficient, and when u=0, α(u)=1 / Otherwise, α(u) = 1 / 2. After the transformation, each tile generates an 8×8 spectrum matrix, where the top left corner F(0,0) represents the DC component (DC, corresponding to the average brightness of the background), and the rest are the AC components (AC, corresponding to texture and edge variations).

[0089] S220: Spectral Energy Analysis and Adaptive Threshold Generation. Based on the characteristic that pipeline defects (cracks, corrosion edges) exhibit significant high-frequency energy in the frequency domain, the high-frequency energy proportion E of each patch is calculated. ratio The region in the lower right corner of the spectrum matrix where u+v≥ξ is selected, where ξ is the frequency cutoff constant. In this embodiment, ξ=6 is chosen as the defect feature region. The calculation formula is as follows:

[0090]

[0091] Based on this energy ratio, an adaptive firing threshold V is generated for neurons within each patch.th (x, y). A reverse mapping strategy is adopted: if E ratio A high threshold indicates a high likelihood of cracks in the region; the threshold should be lowered to increase the pulse firing rate. If E... ratio A low threshold indicates a smooth pipe wall, and the threshold should be increased to suppress noise.

[0092]

[0093] In the formula, V base With a base threshold of 0.5, γ as the adjustment gain, and δ as the sensitivity bias, this step generates a threshold matrix V with the same size as the input image. th ∈R H×W .

[0094] S230: Differential Dynamics Encoding and Pulse Generation. The LIF neuron model is driven to generate pulses using the aforementioned adaptive threshold matrix. The time step is set to T (T=4 in this embodiment). At each time step t, the membrane potential V of the neuron with coordinates (x,y) is... mem The update of (t) follows the following difference equation:

[0095]

[0096] In the formula, β is the membrane potential decay factor (0.25 in this embodiment), I norm (x, y) represents the normalized pixel value input in step one (considered as the injected current). Pulse firing logic: When the membrane potential exceeds the adaptive threshold V calculated in step two... th When (x, y), the neuron fires a pulse S(t)=1 and performs a soft reset of the membrane potential as shown in the following formula; otherwise, S(t)=0.

[0097]

[0098]

[0099] The above formula means that if a neuron fires a pulse, the current membrane potential is updated to the remaining value after subtracting the threshold, achieving a soft reset to preserve residual information. If no pulse is fired, the membrane potential remains unchanged, waiting for the accumulation of the membrane potential described above at the next moment.

[0100] After the above processing, the original two-dimensional grayscale image is encoded into a three-dimensional binary pulse tensor S with a time dimension. out ∈{0,1} 4×1×256×256 In this tensor, the crack region is represented by a high-density cluster of pulses, while the background region is represented by sparse pulses, thereby enhancing the signal-to-noise ratio at the data input end and serving as the input data for the subsequent F-MobileSNN network.

[0101] S300: Input the three-dimensional binary pulse tensor into a pre-constructed feature extraction network for advanced semantic feature extraction and output an advanced semantic feature map.

[0102] This step utilizes a lightweight full-pulse neural network (F-MobileSNN) to map primary pulse signals to high-level semantic features. The feature extraction network includes a pulse depth-separable convolutional module and a frequency-domain guided pulse aggregation self-attention (F-SASA) module, used to enhance defect features and suppress background noise in the frequency domain.

[0103] In a preferred embodiment, S300 includes the following sub-steps:

[0104] S310: Shallow Feature Extraction and Spatial Dimensionality Reduction. (This refers to the process of extracting the impulse tensor S...) out The input is fed into the initial pulse layer, which uses a 3×3 convolution kernel with a stride of 2, padding of 1, and 32 output channels. Preliminary convolution and spatial downsampling (from 256×256 to 128×128) are performed to establish the initial feature space, outputting a primary feature map F. stem .

[0105] S320: Mid-layer feature extraction and contour capture. (The F...) stem Input to multiple cascaded pulse depth-separable convolutional units (structure as follows) Figure 3 Feature extraction is performed step-by-step in the backbone network (as shown).

[0106] Each pulse depth separable convolutional unit performs the following operations sequentially:

[0107] (1) Deep pulsed convolutional layer: F stem Grouped convolutions are performed using a 3×3 kernel and a stride of s. The simulated membrane potentials output from the convolutions are then passed through a batch normalization layer to adjust their distribution and accelerate convergence. The formula for calculating the batch normalization layer is shown below:

[0108]

[0109] In the formula, and These are the mean and variance of the current batch of data, respectively. and Here, ε is a learnable affine transformation parameter, and ε is a zero-prevention constant. The analog value output by BN is injected as the input current into the first-stage leakage integral-trigger neuron. The neuron performs an "accumulation-firing-reset" operation, quantizing the analog signal back into binary pulses. This step completes the extraction of spatial features and nonlinear pulse generation.

[0110] (2) Point-pulse convolutional layer

[0111] The system receives output pulses from the deep layer and performs full-channel convolution using a 1×1 kernel with a stride of 1 to expand the channel count. Next, it undergoes batch normalization. Finally, a second-level leak integral-triggered neuron is injected to fire pulses again. This step completes the information fusion between channels.

[0112] (3) Residual connection

[0113] The input and output pulses of the module are superimposed on the membrane potential to prevent gradient vanishing.

[0114] S330: Deep Feature Frequency Domain Enhancement and Attention Gating. The intermediate feature map output from S320 is input to the frequency domain-guided pulse aggregation self-attention (F-SASA) module (structure as follows). Figure 4 (As shown) perform core processing.

[0115] Specifically, it includes:

[0116] (1) Query and key generation: The intermediate feature map is passed through two independent 1×1 convolutional layers (followed by BN) to generate query tensor Q and key tensor K.

[0117] (2) Discrete Cosine Transform Frequency Domain Enhancement: The key value tensor K is divided into 8×8 local patches in spatial dimension, and a two-dimensional discrete cosine transform is performed on each patch. In the frequency domain, a preset mask M_mask is used to suppress DC and low-frequency coefficients (background), while preserving and enhancing mid-to-high frequency coefficients (defect texture). The filtering operation can be expressed as:

[0118]

[0119] The filtered spectrum is then restored to the spatial domain using a two-dimensional inverse discrete cosine transform to obtain K. freq And superimposed in the form of residuals:

[0120]

[0121] In the formula, λ represents the learnable frequency domain adjustment parameter.

[0122] (3) Aggregation of unvalued matrices and generation of weights: abandoning the QK in traditional attention T Matrix multiplication is performed using the less computationally intensive Hadamard product (element-wise multiplication). The Hadamard product of the query tensor Q and the augmented key tensor K' is calculated to obtain the attention map M.

[0123]

[0124] Then, global average pooling along the channel dimension is performed on M to generate a spatial attention graph A∈R. 4×1×32×32 :

[0125]

[0126] (4) Feature Gating and Re-pulsation: The Sigmoid function is used to map A to a gating weight σ(A) in the (0,1) interval. The normalized weights are broadcast multiplied with the original input intermediate feature map to obtain the weighted simulated feature Y. Finally, Y is injected into the adaptive LIF neuron and pulses are re-fired to obtain the final high-level semantic feature map F with background noise suppressed and defect features targeted enhanced. final (For example, the size remains 4×128×32×32).

[0127] Specifically, firstly, the Sigmoid function is used to map A to a gated weight σ(A) in the interval (0, 1), and then the normalized weight is broadcast multiplied with the original input pulse F6. Since the input is 0 / 1 and the weight is a floating-point number, the weighted feature Y obtained in this step is a simulated value.

[0128]

[0129]

[0130] Finally, to convert the simulated weighted feature Y back into binary impulses to fit the next layer of the network, it is injected into the adaptive leaky integral-triggered neuron to obtain the final binary high-level feature map F. final ∈{0,1} 4×128×32×32 After frequency domain gating by F-SASA, the pulse activity in the background region is suppressed to near zero, while the pulse firing rate in the crack region is significantly enhanced.

[0131] S400: Input the high-level semantic feature map into the decoder, perform qualitative classification decoding and quantitative regression decoding respectively, and output the category label and physical quantification index of pipeline defects in parallel.

[0132] This step uses a dual-head pulse decoder to transform the pulse-form feature map into intuitive decision indicators.

[0133] In a preferred embodiment, S400 includes the following two parallel branches:

[0134] S410: Qualitative Classification Decoding. This branch is used to identify defect categories (e.g., normal, cracks, corrosion, root intrusion).

[0135] (1) Global spatiotemporal pulse pooling

[0136] To obtain global feature description information, firstly, F... final Perform global average pooling in the spatial dimension and average integration in the temporal dimension. Compress the four-dimensional tensor into a one-dimensional feature vector V. cls∈R 128 The calculation formula is as follows:

[0137]

[0138] Among them, V cls (c) is the average pulse firing rate of the c-th channel within this time-space window, with a value range of [0,1], realizing the conversion from discrete pulses to continuous values.

[0139] (2) Fully connected mapping

[0140] The feature vector V cls The input is fed into a fully connected layer (without bias terms) and mapped to a class space. There are a total of 4 predefined defect categories (normal, crack, corrosion, and root intrusion), and the weight matrix is ​​W. cls ∈ R 128×4 :

[0141]

[0142] (3) Output of qualitative identification results

[0143] For the output logic value L logits By applying the Softmax function, the predicted probability distribution P for each defect category is obtained. class The category with the highest probability is selected as the final defect category label:

[0144]

[0145] S420: Quantitative Regression Decoding. This branch is used to assess the physical severity of defects (such as crack length and corrosion area ratio). Its core logic is that the more severe the defect, the greater the total number of pulses emitted in the corresponding characteristic region.

[0146] (1) Channel weighted filtering

[0147] Since different feature channels contribute differently to physical quantization, a 1×1 convolutional layer is first used to compress and map the 128 channels into a single quantized feature map M. reg ∈ R 4×1×32×32 .

[0148]

[0149] In the formula, w reg These are the learnable channel regression weights.

[0150] (2) Global pulse accumulation

[0151] The regression module uses a summation strategy to calculate M. reg Cumulative response value S at all time steps and spatial locations total :

[0152]

[0153] Here, ReLU is introduced to filter out the negative interference generated by convolution, retaining only the pulse intensity of positive activation.

[0154] (3) Output of quantitative indicators

[0155] The cumulative value S total This is mapped to the final physical quantification index Y through a linear scaling layer (or a single-neuron fully connected layer). metric :

[0156]

[0157] In the formula, α and β are regression parameters. When a crack is detected, Y metric The output shows the crack length in millimeters (mm); when corrosion is detected, the output shows the area percentage in percentage (%).

[0158] Preferably, the feature extraction network (F-MobileSNN) and decoder are obtained through the following training steps (network training phase):

[0159] S500: Obtain a training dataset containing images of the pipe's inner wall, corresponding defect category labels, and true values ​​of physical quantification indicators.

[0160] S510: Constructing the joint loss function L for multiple tasks total Used to simultaneously constrain classification and regression tasks:

[0161]

[0162] Among them, the classification loss function L cls By utilizing the predicted probability distribution P output from step four class Compared to the actual defect category label y true And the cross-entropy loss function is used to measure the difference in probability distributions:

[0163]

[0164] The aforementioned loss function can improve the network's accuracy in classifying defects such as cracks and corrosion. Regression loss function L reg By utilizing the physical quantization prediction value Y output from step four metric Compared with the actual physical measurement value y val (Manually calibrated crack length). The mean squared error loss function is used to measure quantization deviation:

[0165]

[0166] In the formula, M represents the number of samples in the current training batch. This term enables the network to learn a linear or nonlinear mapping relationship between the number of impulses in the feature map and the physical scale.

[0167] S520: A surrogate gradient method is used to address the non-differentiability of the impulse firing function. During backpropagation, the derivative of a smooth function (such as the arctangent function) is used to approximate the derivative of the step function. For the neuron membrane potential V... men The approximate derivative of its pulse firing Defined as:

[0168]

[0169] The derivative has a bell-shaped curve, which ensures that the gradient can be effectively transmitted near the threshold, while the gradient decays far from the threshold, which is consistent with the refractory period characteristics of biological neurons.

[0170] S530: Combine the backpropagation over time (BPTT) algorithm to calculate the total loss L. total The gradients of all learnable parameters in the network (including convolutional kernel weights, frequency domain augmentation coefficients λ, neuron parameters, regression weights, etc.) are iteratively updated using an optimizer (such as AdamW) until the model converges on the validation set.

[0171] S540: Deploy the trained F-MobileSNN network model to the edge computing unit of the pipeline inspection robot.

[0172] When the robot is inspecting the pipeline, the camera captures images of suspected defects. The system then performs real-time reasoning through the steps S100 to S400 described above, and transmits the defect category label and physical quantitative indicators back to the ground control terminal for display and alarm through the robot's communication module.

[0173] Example 2

[0174] See Figure 5 A pipeline defect identification system based on a spiking neural network is proposed. This system is applied to the pipeline defect identification method in Example 1 and can be integrated into the edge computing platform (e.g., NVIDIA Jetson series modules) of pipeline inspection robots, tracked inspection vehicles or other mobile inspection equipment.

[0175] The pipeline defect identification system specifically includes an image acquisition and preprocessing module 10, a pulse differential coding module 20, a feature extraction network module 30, a dual-head decoding module 40, and a communication module 50.

[0176] The image acquisition and preprocessing module 10 is used to acquire a grayscale image sequence of the inner wall of the pipe and perform normalization preprocessing on the grayscale image sequence to obtain a standardized image tensor.

[0177] Specifically, this module is responsible for driving the image sensor, performing automatic exposure and white balance adjustments, and writing image data into the shared memory of the edge computing platform through a high-speed data transmission interface (such as MIPI CSI-2) to provide input for subsequent processing in a low-latency manner.

[0178] The pulse differential coding module 20 is used to perform pulse differential coding based on two-dimensional discrete cosine transform on the image tensor to obtain a three-dimensional binary pulse tensor. Specifically, this module includes: image segmentation (8×8 local patches), performing two-dimensional discrete cosine transform, calculating the high-frequency energy ratio, generating an adaptive firing threshold matrix, and driving a leaky integral-triggered (LIF) neuron model for differential pulse coding. Finally, this module outputs a three-dimensional binary pulse tensor with spatiotemporal sparsity, which serves as the basic input for subsequent feature extraction.

[0179] Feature extraction network module 30 is used to input the three-dimensional binary pulse tensor into a pre-constructed feature extraction network for high-level semantic feature extraction and output a high-level semantic feature map; the feature extraction network includes a pulse depth-separable convolution module and a frequency domain-guided pulse aggregation self-attention module, used to enhance defect features and suppress background noise in the frequency domain; the network includes pulse depth-separable convolution units (structure as shown in the diagram). Figure 3 (as shown) and the frequency-domain guided pulse aggregation self-attention (F-SASA) module (structure as shown) Figure 4 (As shown). This module extracts multi-scale features step by step through pulse depth-separable convolutional units, and finally uses the F-SASA module to perform targeted enhancement of defect features and suppress background noise in the frequency domain, thereby generating a high-level semantic feature map with high discriminativeness.

[0180] The dual-head decoding module 40 is used to input the high-level semantic feature map into the decoder, perform qualitative classification decoding and quantitative regression decoding respectively, and output the category label and physical quantitative index of the pipeline defect in parallel; wherein, the physical quantitative index includes crack length or corrosion area ratio.

[0181] The communication module 50 is used to encapsulate the defect category label and the physical quantification index into a structured inspection report, and transmit it back to the ground control terminal in real time via a wireless communication link for display and alarm.

[0182] Example 3

[0183] This invention provides a pipeline inspection device 300, which can be an intelligent device that can move autonomously or remotely inside a pipeline, such as a pipeline crawling robot, a wheeled or tracked inspection vehicle.

[0184] A pipeline inspection device includes: one or more processors; and a memory;

[0185] And one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, the computer programs including instructions for performing the method in Embodiment 1.

[0186] Understandably, the computer program includes instructions for executing the pipeline defect identification method (i.e., steps S100 to S400) described in Embodiment 1. Specifically, when the computer program is loaded and run by the processor, it controls the following operations of the pipeline inspection equipment: driving its onboard image sensor to acquire images of the pipeline's inner wall; on the edge computing platform, calling the functional modules of the system described in Embodiment 2 to sequentially complete image preprocessing, pulse differential coding, neural network feature extraction, and dual-head decoding calculation; finally, controlling the communication reporting module to send the generated defect category label and physical quantification indicators to the ground control terminal.

[0187] Example 4

[0188] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of Embodiment 1.

[0189] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0190] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0191] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A pipeline defect identification method based on a spiking neural network, characterized in that, The method, applied to an edge computing platform for mobile devices with image sensors, includes: A grayscale image sequence of the inner wall of the pipe is obtained, and the grayscale image sequence is preprocessed by normalization to obtain a standardized image tensor. The image tensor is subjected to pulse differential encoding based on two-dimensional discrete cosine transform to obtain a three-dimensional binary pulse tensor; The three-dimensional binary pulse tensor is input into a pre-constructed feature extraction network for advanced semantic feature extraction, and an advanced semantic feature map is output. The feature extraction network includes a pulse depth separable convolution module and a frequency domain guided pulse aggregation self-attention module, which are used to enhance defect features and suppress background noise in the frequency domain. The high-level semantic feature map is input into the decoder, and qualitative classification decoding and quantitative regression decoding are performed respectively, and the category label and physical quantitative index of pipeline defects are output in parallel; wherein, the physical quantitative index includes crack length or corrosion area ratio; The step of performing pulse differential coding based on two-dimensional discrete cosine transform on the image tensor includes: The normalized grayscale image sequence is divided into multiple non-overlapping local patches; Perform a two-dimensional discrete cosine transform on each local patch to obtain a spectrum matrix containing DC and AC components; Calculate the energy proportion of high-frequency regions in the spectrum matrix of each patch, and dynamically set the firing threshold of neurons in the corresponding region based on the energy proportion through a reverse mapping strategy; The leakage integral-triggered neuron is driven by a threshold matrix that combines firing thresholds. It performs differential state updates by combining the current pixel input with the historical membrane potential. When the membrane potential exceeds the threshold, a pulse is fired and a soft reset is performed, outputting a three-dimensional binary pulse tensor.

2. The pipeline defect identification method based on a spiking neural network according to claim 1, characterized in that, The step of inputting the three-dimensional binary pulse tensor into a pre-constructed feature extraction network for high-level semantic feature extraction includes: The three-dimensional binary pulse tensor is input into a backbone network consisting of a pulse initial layer and multiple cascaded pulse depth separable convolutional units for stepwise feature extraction. The initial pulse layer performs preliminary convolution and spatial downsampling on the input pulse tensor to output a primary feature map. The multiple cascaded pulse depth separable convolutional units sequentially receive the feature map output by the previous unit, extract spatial features through pulse depth convolution, fuse channel information through point convolution, and cooperate with step-by-step downsampling operations to finally output an intermediate feature map containing multi-scale semantic information. The intermediate feature map is input into a frequency-domain guided pulse aggregation self-attention module for processing to obtain a high-level semantic feature map.

3. The pipeline defect identification method based on a spiking neural network according to claim 2, characterized in that, The step of inputting the intermediate feature map into a frequency-domain guided pulse aggregation self-attention module for processing includes: The intermediate feature maps are mapped to generate query tensors and key-value tensors, respectively. A two-dimensional discrete cosine transform is performed on the key value tensor, and the high-frequency coefficients of the corresponding defect texture features are enhanced in the frequency domain using a preset filter. Then, the key value tensor with frequency domain enhancement is obtained by inverse transformation. Calculate the Hadamard product of the query tensor and the frequency domain enhanced key tensor, and generate a spatial attention weight map through global average pooling in the channel dimension; The spatial attention weight map is applied to the intermediate feature map to generate the high-level semantic feature map, in which background noise is suppressed and defect features are enhanced in a targeted manner.

4. The pipeline defect identification method based on a spiking neural network according to claim 1, characterized in that, The high-level semantic feature map is input into the decoder to perform qualitative classification decoding, including: Global spatiotemporal average pooling is performed on the high-level semantic feature map to calculate the average pulse firing rate of each feature channel in the time and space dimensions, thereby obtaining a one-dimensional feature vector; The one-dimensional feature vector is input into a fully connected classification layer and processed by the Softmax function to output the probability distribution of pipeline defect categories; Defect category labels are determined based on the probability distribution.

5. The pipeline defect identification method based on a spiking neural network according to claim 1, characterized in that, The high-level semantic feature map is input into the decoder to perform quantitative regression decoding, including: The high-level semantic feature map is subjected to channel-weighted filtering to enhance the channel response sensitive to physical quantization, thereby obtaining a quantization-sensitive feature map; Perform global pulse integration on the quantization-sensitive feature map and count the total effective pulse output at all time steps and spatial locations; The total effective pulse output is input into the regression mapping layer and converted into a physical quantitative indicator reflecting the severity of the defect.

6. The pipeline defect identification method based on a spiking neural network according to claim 1, characterized in that, The feature extraction network and decoder are obtained through the following training steps: Obtain a training dataset containing images of the pipe's inner wall, corresponding defect category labels, and true values ​​of physical quantification indicators; A multi-task joint loss function is constructed, which is composed of a weighted sum of classification cross-entropy loss and regression mean square error loss, and is used to simultaneously constrain the defect category probability distribution and physical quantification index output by the decoder. The surrogate gradient method is adopted to approximate the non-differentiable step of the neuron's spiking function in the spiking neural network with a smooth function, and combined with the backpropagation algorithm over time, the gradient of the joint loss function with respect to the parameters in the feature extraction network and the decoder is calculated. Based on the gradient, the convolutional kernel weights, frequency domain enhancement coefficients, neuron dynamics parameters in the feature extraction network, and the regression mapping weights in the decoder are jointly optimized and iteratively updated until the model converges. The trained network model is deployed to the edge computing unit of a mobile device.

7. A pipeline defect identification system based on a spiking neural network, characterized in that, The system is applied to the pipeline defect identification method according to any one of claims 1-6, wherein the system is set on the edge computing platform of the pipeline inspection equipment, and the system includes: The image acquisition and preprocessing module is used to acquire a grayscale image sequence of the inner wall of the pipe and perform normalization preprocessing on the grayscale image sequence to obtain a standardized image tensor. The pulse differential coding module is used to perform pulse differential coding based on two-dimensional discrete cosine transform on the image tensor to obtain a three-dimensional binary pulse tensor; The feature extraction network module is used to input the three-dimensional binary pulse tensor into a pre-constructed feature extraction network for advanced semantic feature extraction and output an advanced semantic feature map. The feature extraction network includes a pulse depth separable convolution module and a frequency domain guided pulse aggregation self-attention module, which are used to enhance defect features and suppress background noise in the frequency domain. A dual-head decoding module is used to input the high-level semantic feature map into the decoder, perform qualitative classification decoding and quantitative regression decoding respectively, and output the category label and physical quantitative index of the pipeline defect in parallel; wherein, the physical quantitative index includes crack length or corrosion area ratio; The communication module is used to encapsulate the defect category label and the physical quantification index into a structured inspection report, and transmit it back to the ground control terminal in real time via a wireless communication link for display and alarm.

8. A pipeline inspection device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, the computer programs including instructions for performing the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images