Multi-modal feature pipeline defect image recognition method, device and server

Abstract

This invention provides a method, apparatus, and server for multimodal feature-based pipeline defect image recognition, relating to the technical field of image feature recognition. The method includes: using a deep network model for multimodal pipeline defects, performing local feature extraction processing on local image data and local texture descriptors in the multimodal feature data; extracting local detail feature vectors based on a late fully connected layer fusion strategy; using a multi-fusion layer network model for pipeline defects, performing global feature extraction processing on global image data and global texture descriptors in the multimodal feature data; extracting global scene feature vectors based on an early convolutional layer fusion strategy; calculating the cosine similarity between the local detail feature vectors and global scene feature vectors and the sample feature vectors respectively; and performing weighted summation on the target samples with the highest similarity to obtain a weighted voting score, which is used to determine the pipeline defect recognition result. This invention can significantly improve the accuracy of pipeline defect recognition.

Description

Technical Field

[0001] This invention relates to the technical field of image feature recognition, and in particular to a method, apparatus and server for multimodal feature pipeline defect image recognition. Background Technology

[0002] Image recognition of defects in municipal pipelines is a core technology for pipeline network operation and maintenance. By extracting defect features from pipeline images, it can identify problems such as cracks, corrosion, and deformation. Currently, related technologies propose that existing recognition methods are mainly divided into two categories: single-modal recognition and traditional multimodal recognition. Single-modal recognition methods mainly rely on the RGB data of a single pipeline image and use convolutional neural networks (CNNs) to extract global features to complete the recognition. However, in scenarios such as uneven lighting, stain coverage, and unclear local details of defects in pipelines, the recognition performance will drop significantly due to insufficient effective features. While traditional multimodal recognition can combine image and acoustic data for recognition, this approach often uses feature splicing or simple weight combination for fusion, which cannot fully explore the complementary information of different dimensions of data. In addition, its network architecture is fixed and does not design differentiated fusion strategies for the global scene and local details of pipeline defects. Furthermore, it has weak anti-interference capabilities, poor suppression of complex noises in pipelines (such as water reflection and pipe wall attachments), and insufficient robustness in the recognition of irregular defects. As a result, existing technologies cannot meet the requirements for high-precision and high-robustness defect recognition in complex pipeline environments. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, apparatus and server for multimodal feature pipeline defect image recognition, which can significantly improve the accuracy of pipeline defect recognition.

[0004] In a first aspect, embodiments of the present invention provide a method for multimodal feature pipeline defect image recognition. The method includes: acquiring multi-dimensional data of the target pipeline, and sequentially performing image alignment processing, noise suppression processing, and data normalization processing on the multi-dimensional data to obtain standardized multimodal feature data, wherein the multi-dimensional data includes: global image data, local image data, and texture descriptors; using a pipeline defect multi-feature deep network model, performing local feature extraction processing on the local image data and local texture descriptors in the multimodal feature data, extracting local detail feature vectors based on a late fully connected layer fusion strategy, and using a pipeline defect multi-fusion layer network model, performing global feature extraction processing on the global image data and global texture descriptors in the multimodal feature data, extracting global scene feature vectors based on an early convolutional layer fusion strategy; calculating the cosine similarity between the local detail feature vectors and global scene feature vectors and the feature vectors of each sample in a preset sample set, respectively, and performing weighted summation processing on a preset number of target samples with the highest similarity to obtain a weighted voting score, so as to determine the pipeline defect recognition result based on the weighted voting score.

[0005] In one embodiment, the step of noise suppression processing of multi-dimensional data includes: using a Gaussian filtering algorithm to filter global image data and local image data to remove random noise in the image, and using an adaptive threshold segmentation algorithm to binarize the filtered image to eliminate the occlusion of defect features by pipe wall stains, water reflections, and environmental deposits.

[0006] In one implementation, a deep network model for multi-feature pipeline defects is used to extract local features from local image data and local texture descriptors in multi-modal feature data. The step of extracting local detail feature vectors based on a late fully connected layer fusion strategy includes: inputting local image data and local texture descriptors into the first input stream and the second input stream of the deep network model for multi-feature pipeline defects, respectively; performing alternating operations through various convolutional layers and max pooling layers to determine the hierarchical features of local defects; and performing average fusion processing and max fusion processing on the hierarchical features of local defects to obtain local detail feature vectors.

[0007] In one implementation, a global feature extraction process is performed on global image data and global texture descriptors in multimodal feature data using a pipeline defect multi-fusion layer network model. The step of extracting global scene feature vectors based on an early convolutional layer fusion strategy includes: inputting global image data and global texture descriptors into the first and second convolutional layers of the pipeline defect multi-fusion layer network model, respectively, and performing double convolution fusion processing on the convolution output results to determine the activation mean and maximum activation value, so as to obtain the global scene feature vector based on the activation mean and maximum activation value.

[0008] In one implementation, the step of performing a weighted summation on a preset number of target samples with the highest similarity to obtain a weighted voting score includes: standardizing the local detail feature vector and the global scene feature vector respectively, and selecting a first target sample set and a second target sample set with the highest cosine similarity for the standardized local detail feature vector and the global scene feature vector respectively; summing the similarities of the first target sample set and the second target sample set respectively, and adding the summation results to obtain the weighted voting score.

[0009] In one implementation, the step of determining the pipeline defect identification result based on the weighted voting score includes: determining the sample label corresponding to the maximum weighted voting score as the pipeline defect identification result, wherein the pipeline defect identification result includes: defect type and severity.

[0010] In one implementation, after determining the sample label corresponding to the maximum weighted voting score as the pipeline defect identification result, the process includes: performing a consistency verification process on the pipeline defect identification result; if the weighted voting score corresponding to the pipeline defect identification result is not less than a preset consistency threshold, the consistency verification is determined to be successful, and the pipeline defect identification result is output.

[0011] Secondly, embodiments of the present invention also provide a multimodal feature pipeline defect image recognition device, the device comprising: a data acquisition and processing module, which acquires multi-dimensional data of the target pipeline and sequentially performs image alignment processing, noise suppression processing, and data normalization processing on the multi-dimensional data to obtain standardized multimodal feature data, wherein the multi-dimensional data includes: global image data, local image data, and texture descriptors; a dual-network feature extraction module, which performs local feature extraction processing on the local image data and local texture descriptors in the multimodal feature data through a pipeline defect multi-feature deep network model, extracts local detail feature vectors based on a later fully connected layer fusion strategy, and performs global feature extraction processing on the global image data and global texture descriptors in the multimodal feature data through a pipeline defect multi-fusion layer network model, extracts global scene feature vectors based on an early convolutional layer fusion strategy; and a weighted voting decision module, which calculates the cosine similarity between the local detail feature vectors and the global scene feature vectors and the feature vectors of each sample in a preset sample set, and performs weighted summation processing on a preset number of target samples with the highest similarity to obtain a weighted voting score, so as to determine the pipeline defect recognition result based on the weighted voting score.

[0012] Thirdly, embodiments of the present invention also provide a server, including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0014] The embodiments of the present invention bring the following beneficial effects: This invention provides a method, apparatus, and server for multimodal feature pipeline defect image recognition. First, multi-dimensional data of the target pipeline is acquired, and image alignment, noise suppression, and data normalization are sequentially performed on the multi-dimensional data to obtain standardized multimodal feature data. Then, a deep network model for multi-feature pipeline defects is used to extract local features from local image data and local texture descriptors in the multimodal feature data. Local detail feature vectors are extracted based on a later fully connected layer fusion strategy. Next, a multi-fusion layer network model for pipeline defects is used to extract global features from global image data and global texture descriptors in the multimodal feature data. Global scene feature vectors are extracted based on an early convolutional layer fusion strategy. Finally, the cosine similarity between the local detail feature vectors and global scene feature vectors and the feature vectors of each sample in a preset sample set is calculated. A weighted sum is then performed on a preset number of target samples with the highest similarity to obtain a weighted voting score. The pipeline defect recognition result is determined based on the weighted voting score. This invention can significantly improve the accuracy of pipeline defect recognition.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a multimodal feature pipeline defect image recognition method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another multimodal feature pipeline defect image recognition method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a multimodal feature pipeline defect image recognition device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, existing technologies suffer from three major problems: First, multi-feature fusion efficiency is low. Traditional methods simply splice data at the network end, failing to achieve deep collaboration between global and local data during feature learning, resulting in insufficient utilization of complementary features. Second, adaptability to complex scenarios is poor. When faced with uncontrolled factors such as fluctuating lighting, contamination, and irregular defect morphology within pipelines, single-modal features are prone to failure, and traditional multimodal networks lack targeted feature extraction and fusion mechanisms. Third, the decision-making logic is simplistic. The lack of a dynamic weight allocation mechanism for multi-dimensional features makes it difficult to adjust decision priorities based on the reliability of different features. In summary, existing technologies cannot meet the high-precision and robust defect identification requirements in complex pipeline environments. Therefore, it is urgent to construct a multimodal defect identification solution adapted to pipeline scenarios through differentiated fusion architectures, deep aggregation of multiple features, and intelligent decision-making strategies.

[0021] Based on this, the multimodal feature pipeline defect image recognition method, device, and server provided by this invention can achieve multimodal data acquisition and preprocessing, dual independent dual-stream convolutional neural networks, and weighted voting decision through three core modules. First, global / local RGB data, entropy, and HOG texture descriptors are collected by a CCTV robot. Then, dual networks PD-MDLN and PD-MFLN are constructed, and multi-feature deep aggregation is achieved by using late fully connected and early convolutional fusion strategies, respectively. Finally, the feature output is integrated through a weighted voting strategy to complete the identification and verification of defect type and severity. This invention improves feature fusion efficiency and adaptability to complex scenarios through differentiated fusion architecture, multi-feature aggregation, and intelligent decision-making, effectively reducing the false negative rate and false positive rate of defect identification, and solving the problems of low feature fusion efficiency, poor adaptability to complex scenarios, and simple decision logic in existing technologies.

[0022] See Figure 1 The diagram shows a flowchart of a multimodal feature pipeline defect image recognition method, which mainly includes the following steps S102 to S106: Step S102: Obtain multi-dimensional data of the target pipeline, and sequentially perform image alignment processing, noise suppression processing and data normalization processing on the multi-dimensional data to obtain standardized multi-modal feature data. The multi-dimensional data includes: global image data, local image data and texture descriptors.

[0023] In one implementation, multi-dimensional data of the target pipeline is first acquired. Global image data refers to the overall image of the pipeline's inner wall, collected by a pipeline inspection device moving at a constant speed along the pipeline axis. This image is used to capture the spatial relationship between defects and the pipeline background, as well as the overall environmental characteristics of the pipeline, such as the distance of the defect from the pipeline interface and the overall degree of corrosion on the pipe wall. Local image data refers to high-resolution close-up images obtained when the gray-level variance of a certain area in the global image exceeds a preset threshold. This triggers a local focusing acquisition mechanism, and the pipeline inspection device automatically adjusts its focus to magnify and capture the suspected area. This image is used to capture detailed information about the defects, such as crack width and corrosion spot morphology. Texture descriptors are supplementary features extracted from the acquired global and local images, specifically including entropy texture descriptors and histogram of oriented gradients (HARQ) texture descriptors. The entropy texture descriptor, based on information theory statistical methods, quantifies the uncertainty of pixels by calculating the gray-level probability distribution of image pixel regions. The gray-level distribution of defect areas is usually more uneven than that of background areas, resulting in higher entropy values, thus enhancing the distinction between defects and the background. The directional gradient histogram texture descriptor constructs a gradient distribution histogram by calculating the gradient direction and magnitude of local regions of an image. It can effectively capture the edge contour features of defects and is insensitive to changes in illumination.

[0024] After acquiring the aforementioned multi-dimensional data, image alignment, noise suppression, and data normalization are performed sequentially. Image alignment refers to using the inherent feature points of the pipe's inner wall as coordinate references and eliminating image position deviations caused by pipe inspection equipment movement and posture shifts through affine transformation, enabling subsequent feature extraction to be performed in a unified pixel coordinate system. Noise suppression includes two steps: first, a Gaussian filtering algorithm is used to filter the global and local image data, removing random salt-and-pepper noise and electronic noise; then, an adaptive threshold segmentation algorithm is used to binarize the filtered image, dynamically calculating the segmentation threshold based on the average gray value of the neighborhood around each pixel, thereby effectively eliminating the occlusion of defect features by pipe wall stains, water reflections, and environmental deposits. Data normalization involves mapping the values ​​of global image data, local image data, and texture descriptors to a uniform range of 0 to 1, eliminating dimensional differences between different types of data and enabling subsequent neural network models to stably process these data. After the above processing, standardized multimodal feature data is obtained.

[0025] Step S104: Using the deep network model of multi-feature pipeline defects, local feature extraction processing is performed on the local image data and local texture descriptors in the multi-modal feature data. Local detail feature vectors are extracted based on the late fully connected layer fusion strategy. Using the multi-fusion layer network model of pipeline defects, global feature extraction processing is performed on the global image data and global texture descriptors in the multi-modal feature data. Global scene feature vectors are extracted based on the early convolutional layer fusion strategy.

[0026] In one implementation, feature extraction is performed using two independent network models. The first network is a deep network model for multi-feature pipeline defects, used to process local detail features. This model employs a late-stage fully connected layer fusion strategy, and its processing is as follows: Local image data and local texture descriptors from the standardized multimodal feature data are input into the first and second input streams of the model, respectively. The local image data contains magnified close-up information of the defect region, while the local texture descriptor contains the entropy features and histogram of oriented gradients (HGP) features of the region. The two input streams are independently processed by alternating operations of multiple convolutional layers and max-pooling layers to progressively extract hierarchical features of the local defects. The convolutional layers are used to extract local features by sliding the convolutional kernel across the image, while the max-pooling layers are used to downsample the feature map to reduce dimensionality and retain salient features. After multiple layers of computation, the resulting hierarchical features include the texture details, morphological contours, and edge information of the defects. Subsequently, these hierarchical features are input into the first and second fully connected layers, respectively. The role of the fully connected layers is to map the distributed feature representations extracted by the convolutional and pooling layers to the sample label space. Then, a fusion operation is performed on the outputs of the first and second fully connected layers: the mean of the two outputs is calculated to perform average fusion, while the maximum value of the two outputs is selected to perform maximum fusion. Average fusion strengthens the representation of stable features, while maximum fusion highlights the key details of defects. The results of the two fusion operations are combined as a local detail feature vector.

[0027] The second network is a multi-fusion layer network model for pipe defects, used to process global scene features. This model employs an early convolutional layer fusion strategy, with the following process: Global image data and global texture descriptors from the standardized multimodal feature data are input into the model's first and second convolutional layers, respectively. The global image data contains overall scene information about the pipe's inner wall, while the global texture descriptor contains the global texture distribution of that scene. Immediately after the outputs of the first and second convolutional layers, a double convolutional fusion process is performed on the outputs, specifically calculating the mean and maximum activation values ​​of the two output feature maps. This early fusion method allows the network to learn the joint features of image and texture simultaneously at the lowest level. The fused feature map is then processed through subsequent convolutional layers, pooling layers, and fully connected layers to finally obtain the global scene feature vector.

[0028] Step S106: Calculate the cosine similarity between the local detail feature vector and the global scene feature vector and the feature vector of each sample in the preset sample set, respectively. Then, perform weighted summation on the preset number of target samples with the highest similarity to obtain a weighted voting score. Based on the weighted voting score, determine the pipeline defect identification result.

[0029] In one implementation, decision recognition is performed based on the local detail feature vectors and global scene feature vectors obtained above. First, the local detail feature vectors and global scene feature vectors are standardized. The standardization process uses a flexible maximum normalization method to convert the original feature vectors into probability distribution vectors, ensuring that each element's value is within the range of 0 to 1 and the sum of all elements is 1. This eliminates the difference in amplitude between the two network output feature vectors, making subsequent calculations more stable.

[0030] Next, the cosine similarity between the standardized feature vector and the feature vectors of each sample in the preset sample set is calculated. Cosine similarity is a measure of the consistency of the directions of two vectors, with a value ranging from -1 to 1. The closer the value is to 1, the more consistent the directions of the two vectors are, i.e., the more similar the features are. Since the feature vectors of this invention have been standardized, the actual cosine similarity value is within the range of 0 to 1. The preset sample set contains a large number of typical samples labeled with defect types and severity. Each sample has its local detail feature vector and global scene feature vector pre-stored.

[0031] For both local detail feature vectors and global scene feature vectors, a predetermined number of target samples with the highest cosine similarity are selected. For example, the top five samples with the highest similarity can be selected. For the local detail feature vector, the similarity of the five selected target samples is summed to obtain the local contribution score; similarly, for the global scene feature vector, the similarity of the five selected target samples is summed to obtain the global contribution score. Then, the local contribution score and the global contribution score are added together to obtain the weighted voting score. This method of summing separately and then adding together ensures that local features and global features contribute independently to the final score, eliminating the need for manually pre-setting their weights and achieving dynamic weight balance.

[0032] Finally, the pipeline defect identification result is determined based on the weighted voting score. Specifically, the sample label corresponding to the maximum weighted voting score is identified, and consistency verification is performed on this sample label: it is determined whether the weighted voting score is not less than a preset consistency threshold. If the verification passes, the sample label is determined as the pipeline defect identification result. This identification result includes the defect type and severity; for example, a level 3 crack indicates that the defect type is a crack and the severity is level 3.

[0033] The multimodal feature pipeline defect image recognition method provided in this embodiment of the invention designs a dual-stream convolutional neural network with early convolutional fusion (PD-MFLN) and late fully connected fusion (PD-MDLN) for the feature characteristics of local details and global scenes, respectively, to achieve deep aggregation of multimodal features. Furthermore, it combines entropy texture and HOG descriptors to enhance the richness and robustness of feature representation, adapting to complex noise interference and irregular defect scenes in pipelines. In addition, it dynamically optimizes the decision logic of defect recognition and verification by integrating the feature distance scores of the dual networks and dual modalities, thereby improving the recognition reliability in complex pipeline environments.

[0034] See Figure 2 The flowchart of another multimodal feature pipeline defect image recognition method is shown. This embodiment of the invention also provides an implementation method for multimodal feature pipeline defect image recognition. Based on the feature characteristics of local details of pipeline defects (such as crack width and corrosion spots) and global scene (such as defect location and overall pipeline environment), a differentiated fusion strategy is designed. The feature representation capability is enhanced by combining texture descriptors. The decision logic is optimized by weighted voting strategy to achieve accurate defect recognition in complex pipeline environments. See (1) to (3) below for details: (1) Multimodal data acquisition and preprocessing.

[0035] (1-1): Multimodal Data Acquisition. Pipeline inspection equipment (CCTV pipeline robot) is used to collect multi-dimensional data of the target pipeline. The collected data is divided into three categories, and the acquisition requirements and parameters for each type are specified as follows: (1-1-1) Global RGB data (i.e., global image data): Move at a constant speed (0.3m / s) along the pipeline axis, and collect one frame of global image of the pipeline inner wall every 0.5m, covering 360° of the pipeline circumference to ensure no visual blind spots; after collection, the image is uniformly cropped to 128×128 pixels (pixel density: 0.1mm / pixel) using an image cropping algorithm, focusing on the overall scene of the pipeline inner wall. The specific scene of the pipeline inner wall includes capturing the spatial relationship between defects and the pipeline background (such as the distance of the defect from the pipeline interface, the position of the defect on the pipeline circumference), and the overall environmental characteristics of the pipeline (such as the overall corrosion degree of the pipe wall, the distribution of attachments, etc.).

[0036] (1-1-2) Local RGB Data (i.e., Local Image Data): When the grayscale variance of a certain region in the global image is ≥50 (suspected defects are determined by real-time calculation of the local variance of the image), local focusing acquisition is triggered. The focal length of the CCTV robot lens is automatically adjusted (focal length range: 8mm~35mm), and 3 frames are continuously captured on the suspected area. The frame with the highest clarity is taken as the valid data. A region of interest (ROI) is constructed by expanding outward by 20 pixels from the suspected defect area, and cropped to 50×150 pixels (pixel density: 0.05mm / pixel) to ensure that no defect details are missed.

[0037] (1-1-3) Texture Descriptor Extraction: For the cropped global and local RGB data described above, two texture features are extracted as supplementary features, with fixed extraction rules: Entropy texture descriptor: Based on information theory statistical methods, it calculates the uncertainty of image pixel regions, enhancing the pixel distinction between defective areas and pipeline background. The specific calculation formula is as follows:

[0038] Where, E: entropy value (range: 0~8, the larger the value, the more uneven the gray-level distribution of the image pixels, and the higher the distinction between defects and background); g: gray value (0=pure black, 255=pure white); p(g): probability of gray value g (p(g)) = n g / N,n g (where g is the number of pixels with grayscale value g, and N is the total number of pixels in the image). Minimum value ( ), to avoid the logarithm being meaningless when p(g)=0.

[0039] Histogram of Oriented Gradients (HOG): Using 5×5 pixel cells and a 9-dimensional gradient histogram, the image gradient distribution is constructed in 2×2 cell blocks. First, gamma correction is performed on the grayscale image. The horizontal and vertical gradients are calculated, the image is divided into cell units, and histograms of gradient directions are calculated. Then, block units are divided and normalized. Finally, all block histograms are concatenated to obtain the HOG feature vector. The specific calculation formula is as follows: Gradient calculation (using the Sobel operator): Gx=I(x+1,y) I(x 1,y) Gy=I(x,y+1) I(x,y 1)

[0040]

[0041] Where I(x,y): pixel grayscale value at coordinates (x,y); Gx / Gy: horizontal / vertical gradient values; G: gradient magnitude (edge ​​intensity); θ: gradient direction (0°~180°, no directional gradient).

[0042] Histogram normalization (enhancing robustness):

[0043]

[0044]

[0045] Where Gx and Gy represent the horizontal and vertical gradient values, respectively; G: gradient magnitude (edge ​​strength, the larger the value, the clearer the edge); θ: gradient direction (range: 0°~180°, no directional gradient); v: gradient histogram vector of the block unit; ||v||2: vector L2 norm; Minimum value ( Fixed parameters: Cell = 5×5 pixels, Block = 2×2 cells, Gradient direction histogram bins = 9 (each 20° interval).

[0046] (1-2): Data Preprocessing. A series of processing steps are performed on the acquired RGB data and texture descriptors to eliminate interference from devices and the environment, and to achieve data standardization. The processing steps are executed in the following order: (1-2-1) Image Alignment: Using inherent feature points on the inner wall of the pipe (such as pipe joint seams, pipe wall markings, pipe bend feature points, etc.) as coordinate references, the pixel coordinate system of the global and local RGB images is fixed to eliminate image position deviations caused by CCTV robot movement and posture shifts; wherein, the coordinate transformation formula is based on the original image feature point coordinates (x i ,y i (i=1,2,...,n), the coordinates of the feature points in the target aligned image are ( , Alignment is achieved through affine transformation using standard coordinates based on the pipe axis.

[0047] Among them, (x i ,y i ): Pixel coordinates of feature points in the original image; , ): Pixel coordinates of the aligned feature points; The affine transformation matrix is ​​solved using the least squares method.

[0048] (1-2-2) Noise suppression: Gaussian filtering algorithm is used to filter global and local image data to remove random noise in the image. Adaptive threshold segmentation algorithm is used to binarize the filtered image to eliminate the occlusion of defect features by pipe wall stains, water reflection and environmental attachments.

[0049] A Gaussian filtering algorithm is used to filter the global and local RGB images to remove random salt-and-pepper noise and electronic noise. The Gaussian filtering calculation formula is as follows: For each pixel (x,y) in the image, its filtered gray value I′(x,y) is:

[0050] Among them, the Gaussian kernel function G( u , v ):

[0051] Where, I(x,y): original pixel grayscale value; I′(x,y): filtered pixel grayscale value; (u,v): relative coordinates within the Gaussian kernel; k: Gaussian kernel half-width (set to 1, i.e., a 3×3 Gaussian kernel, balancing noise reduction and detail preservation); σ: Gaussian kernel standard deviation (set to 0.8, adapting to the noise intensity of the pipeline image); G(u,v): Gaussian kernel weight (the sum of the 3×3 kernel weights is 1, ensuring that the image brightness remains unchanged).

[0052] The filtered image is binarized using an adaptive threshold segmentation algorithm to eliminate the occlusion of defect features by environmental interference such as pipe wall stains, water reflections, and pipe attachments. The specific calculation formula is as follows: For each pixel (x,y) in the image, the segmentation threshold T(x,y) is:

[0053] The segmented binary image B(x,y) is:

[0054] Where T(x,y): local threshold of pixel (x,y); μ(x,y): average gray value of the 3×3 neighborhood around pixel (x,y); C: threshold offset (set to 5, optimized according to the gray range of the pipeline image); I′(x,y): gray value after Gaussian filtering; B(x,y): binary image (255 for foreground defects, 0 for background).

[0055] (1-2-3) Normalization: The pixel values, entropy texture descriptors, and HOG texture descriptors of global / local RGB data are uniformly mapped to the [0,1] interval to eliminate the dimensional differences between different data types. The normalization calculation formula is:

[0056] Where x is the original data value, x min x max ...

[0057] (2) Dual Independent Dual-Stream Convolutional Neural Networks. Two independent dual-stream convolutional neural networks (PD-MDLN and PD-MFLN) are constructed to process local details and global scene data respectively. A differentiated fusion strategy is adopted to achieve deep aggregation of RGB data and texture descriptors.

[0058] (2-1): PD-MDLN (Deep Network for Multi-Feature Pipeline Defects) processes local details. Local image data and local texture descriptors are input into the first and second input streams of the PD-MDLN model, respectively. Alternating operations are performed through convolutional layers and max-pooling layers to determine the hierarchical features of local defects. Average fusion and max fusion are then performed on these hierarchical features to obtain the local detail feature vector. Considering the complex nature of local detail features in pipeline defects (such as gradual changes in crack width and scattered corrosion spots), a late-stage fully connected layer fusion strategy is adopted. The specific architecture and steps are detailed below: (2-1-1) Network architecture: It includes 14 convolutional layers, 8 max pooling layers, 2 fully connected layers (FC1, FC2, with an output dimension of 4096) and 2 fusion layers (Fuse_avg, Fuse_max); among them, the output dimension of FC1 and FC2 is 4096, and the kernel size of the convolutional layers is uniformly 3×3.

[0059] (2-1-2) Dual-stream input setting: The network has two input streams. The first input stream is the preprocessed local RGB data, and the second input stream is the preprocessed local texture descriptor (feature vector after concatenation of entropy and HOG).

[0060] (2-1-3) Hierarchical feature extraction: The dual-stream input data are input into the network separately, and the hierarchical features of local defects are extracted by alternating operations of convolutional layers and max pooling layers. The feature map dimension gradually evolves from the initial 64×50×150 to 512×6×18, and the core captures key information such as defect details, texture, shape contour, and edge features.

[0061] The core of a convolutional layer is to achieve non-linear feature mapping by convolving the weight matrix with local regions of the input feature map, as shown in the following formula:

[0062] Among them, Conv out (c out(h,w): Output feature map in channel c out Pixel value at coordinates (h, w); C in : The number of channels in the input feature map (e.g., the RGB stream C of Conv1) in =3, Texture Flow C in =2); k: kernel size (k=3); W(c out ,c in (i,j): Convolution kernel weights (c-th kernel) out The cth output channel in (Weight values ​​for each input channel and position (i,j)). Input feature map in channel c in ,coordinate Pixel value at; b(c out ): The cth out The bias term for each output channel (initial value is 0); : Activation function (ReLU(x)=max(0,x), introducing non-linear features).

[0063] (2-1-4), Post-processing fully connected layer fusion: The results of the hierarchical feature extraction are input into FC1 and FC2 respectively, and a double fusion operation is performed after the output of the fully connected layer. The fusion formula and rules are clear: Average fusion (Fuse) avg ): Calculate the mean of the activation values ​​of FC1 and FC2 to enhance the representation of stable features. The formula is:

[0064] Maximum fusion (Fuse_max): Selects the maximum activation value of the corresponding nodes of FC1 and FC2 to highlight key details of the defect (such as crack endpoints and severely corroded areas). The formula is: Fuse_ max =max[FC1(n),FC2(n)] (n is the node index, n∈[1,4096]).

[0065] Where, n: the index of the fully connected layer node (n∈[1,4096]); FC1(n): the output value of the RGB flow fully connected layer FC1 at the nth node (after ReLU activation, FC1(n)≥0); FC2(n): the output value of the texture flow fully connected layer FC2 at the nth node (similarly, FC2(n)≥0); square root operation: normalizes the summation result to avoid numerical overflow.

[0066] (2-1-5), Calculation of the loss function: The sum of cross-entropy losses is used, and the calculation formula is as follows:

[0067]

[0068] Where L is the cross-entropy loss function; Fuse Fusion features ( B: Batch size (B=32); c: Category label index (c∈[1,25]); y batch,c : The true label of the nth sample in the batch (one-hot encoded, correct class is 1, others are 0); Softmax( ): Probability normalization function; =10 8 Minimum value: avoids log(0) being meaningless; Double loss summation: simultaneously constrains the discriminative power of two fused features, improving network robustness.

[0069] (2-2): PD-MFLN (Pipeline Defect Multi-Fusing Layer Network) handles global scene processing. Global image data and global texture descriptors are input into the first and second convolutional layers of the pipeline defect multi-fusion layer network model, respectively. The convolutional outputs are then subjected to dual convolutional fusion processing to determine the mean and maximum activation values. Based on these values, a global scene feature vector is obtained. Considering the rich global scene features of pipelines (such as defect location and environmental interference distribution), an early convolutional layer fusion strategy is adopted. The specific architecture and steps are detailed below: (2-2-1) Network architecture: Fixed network layer structure, including 14 convolutional layers, 2 convolutional fusion layers (conv_avg, conv_max), 8 max pooling layers and 4 fully connected layers (FC1-FC4, output dimension 4096), and the convolutional kernel size of the convolutional layers is uniformly 3×3.

[0070] (2-2-2), Dual-stream input setting: The network has two input streams. The first input stream is the preprocessed global RGB data, and the second input stream is the preprocessed global texture descriptor (feature vector after concatenation of entropy and HOG).

[0071] (2-2-3), Early convolutional layer feature fusion: The dual-stream input data is fed into conv1 and conv2 layers respectively, and a dual convolutional fusion operation is performed after the outputs of the two layers. The fusion formula and rules are clear: Average convolutional fusion (conv) avg ): Calculate the activation mean of the conv1 and conv2 feature maps to balance the overall features of the global scene. The formula is:

[0072] in, w Let K be the weight matrix, K be the filter matrix, and b be the bias matrix.

[0073] Maximum Convolutional Fusion (Convmax): Selects the maximum activation value of the conv1 and conv2 feature maps to highlight key regions in the global scene (such as suspected defect locations or environmental interference sources). The formula is:

[0074] Where m is the pixel index of the feature map, c in / c out Input / output channel index (c in =64, c out =64); h / w: feature map coordinates (h∈[1,128], w∈[1,128]); conv1(c in ,h,w): RGB stream Conv1 in channel c in Output values ​​of coordinates (h, w); conv2(c in ,h,w): Texture flow Conv2 in channel c in Output values ​​of coordinates (h, w); w : Weight matrix (64×64, 64 input channels, 64 output channels); K: Convolution kernel (3×3, for local feature extraction); b: Bias vector (64, 1 bias per output channel).

[0075] (2-2-4), Formula for fully connected layer and output layer: After the fused features are deepened by subsequent convolutional and pooling layers, they are flattened into a 512-dimensional vector, which is then input into four fully connected layers (FC1-FC4) to extract high-order features. Finally, Softmax is used to output 25 probability distributions. The formula is the same as the logic of fully connected layer and output layer of PD-MDLN (FC3-FC4 correspond to texture fusion flow, with an output dimension of 4096).

[0076] (2-2-5), Loss function formula: The loss function of PD-MFLN is consistent with that of PD-MDL, employing multi-class cross-entropy loss and using the outputs of FC3 and FC4 as inputs to constrain the discriminative power of global features.

[0077] (3) Weighted Voting Decision. This module is the final decision-making unit of the entire recognition method. Its core objective is to integrate the dual network outputs of PD-MDLN (local detail features) and PD-MFLN (global scene features), and to achieve accurate identification and consistency verification of pipeline defect types and severity through a weighted voting strategy based on rank-K cosine similarity. The module's innovation lies in dynamic weight balancing—it does not require preset fixed weights, but automatically adapts the discrimination priority of local details and global scenes under different scenarios through Top-K aggregation of feature similarity, avoiding misjudgments caused by the failure of a single feature. The weighted voting process consists of four major steps: feature standardization, similarity calculation, rank-K weighted voting, and decision judgment, ensuring a logical closed loop.

[0078] (3-1): Feature Output and Standardization (Dual Network Feature Integration). After training, PD-MDLN and PD-MFLN output the original feature vectors, which need to be transformed into probability distribution vectors by Softmax standardization to eliminate the influence of differences in network output amplitudes. The feature standardization formula is: PD-MDLN (Local Details Network): This method combines the fused feature vectors of Fuse_avg and Fuse_max, then sums and normalizes the result.

[0079] PD-MFLN (Global Scene Features): The feature vectors of the FC3 and FC4 fully connected layers are summed and then normalized.

[0080] Softmax normalization formula (ensuring the sum of the eigenvector elements is 1 and the values ​​are ∈ [0,1]):

[0081] in, l : Standardized vector of local detail features (1×4096); g : Global scene feature normalized vector (1×4096); Fuse avg / Fuse max : Dual fusion feature vectors of PD-MDLN (both 1×4096); FC3 / FC4: Fully connected layer feature vectors of PD-MFLN (both 1×4096); vi: The i-th element of the original feature vector; e: Natural constant (≈2.71828).

[0082] (3-2): Cosine similarity calculation (feature matching). Cosine similarity is used to measure the directional consistency between the feature vector to be identified and the feature vectors in the sample set. The larger the value, the more similar the features (the better the match in defect type / severity). Cosine similarity calculation formula: Let vector A = [a1, a2, ..., an] and vector B = [b1, b2, ..., bn] (n = 4096), then the similarity is:

[0083] Where Sim(A,B): cosine similarity value (value range ∈ [-1,1], in this scheme, since the features have been standardized, it is actually ∈ [0,1]). Sim: The dot product of vectors A and B (reflecting the consistency of direction); ||A||2 / ||B||2: The L2 norm (magnitude, reflecting the magnitude of the vector) of vectors A and B; ai / bi: The i-th element of vectors A and B; Sim=1 when the two eigenvectors are perfectly matched; Sim≈0 when they are not perfectly matched.

[0084] (3-3): Rank-K weighted voting (dynamic weight aggregation) standardizes both local detail feature vectors and global scene feature vectors. For the standardized local detail feature vectors and global scene feature vectors, a preset number of first and second target sample sets with the highest cosine similarity are selected, respectively. The similarities of the first and second target sample sets are then summed separately, and the sums are added together to obtain the weighted voting score. The top-K highest similarity scores are summed, rather than a single similarity threshold, to avoid the influence of individual abnormal samples on the decision-making process, while automatically balancing the contributions of local and global features. First, the formula for calculating weighted voting scores: Recognition task: Feature vectors of the set to be recognized The feature vector of the j-th sample in the sample set Then the voting score is:

[0085] rank-K summation function definition: Simk(X,Y) is the similarity value of the k-th vector X to be identified among all vectors in the sample set after sorting their similarity (Top-K summation).

[0086] Where ω: weighted voting total score (value range ∈ [0, 2K], since the sum of the Top-K similarity of a single modality is ≤K, and the sum of the two modalities is ≤2K); Feature vectors of the set to be identified (including local features) global ); : The feature vector (including local features) of the j-th sample in the sample set global ); rankK( ): Top-K similarity summation function (K=5, summing the top 5 highest similarity values); Simk(X,Y): similarity value of the kth rank (k=1 is the highest similarity, k=5 is the 5th highest similarity).

[0087] (3-4): Decision-making (identification + verification dual tasks). The sample label corresponding to the maximum weighted voting score is determined as the pipeline defect identification result. Consistency verification is performed on the pipeline defect identification result. If the weighted voting score corresponding to the pipeline defect identification result is not less than the preset consistency threshold, the consistency verification is determined to be passed, and the pipeline defect identification result is output. The pipeline defect identification result includes: defect type and severity.

[0088] Defect identification task (type and severity determination): Objective: Assign unique labels (defect type and severity) to the pipeline data to be identified.

[0089] Where δ: the final identified label (one of the 25 categories); M: the number of samples in the sample set (in this scheme, M=1000, covering typical samples of all labels); argmax j Take the index j of the sample set corresponding to the maximum weighted voting score ω, and its label is the recognition result.

[0090] Defect verification task (consistency determination): Objective: To verify whether the defect to be identified is consistent with the standard defect (e.g., to review test results and match historical defects).

[0091] Where, v: verification result (1 = consistent, 0 = inconsistent); R : Reference set feature vector (standard defect data, such as the features of a known level 3 crack); Q : Query set feature vector (defect data to be verified); t: consistency threshold (t=8.2, after normalization, the upper limit of the 95% confidence interval of the original similarity sum); threshold setting logic: ensure that the verification pass rate of real matching pairs is ≥99%, and the false matching pair misjudgment rate is ≤1%.

[0092] In practical applications, a DN800 reinforced concrete rainwater pipe in a certain city was selected as the inspection object. This pipe has been in service for 12 years and has a daily flow rate of 1500 m³ / h. Previous manual inspections only found suspected cracks, without clarifying the specific type and severity of the defects. This study employs the identification method proposed in this invention, which utilizes three core modules: multimodal data acquisition and preprocessing, dual independent dual-stream convolutional neural networks, and weighted voting decision-making. Data on the inner wall of the pipe is collected by a CCTV pipeline robot and processed through a fully automated process, achieving accurate defect identification and consistency verification. This demonstrates the feasibility and superiority of this invention in real-world engineering scenarios.

[0093] Module 1: Demonstration Application of Multimodal Data Acquisition and Preprocessing Module. This module primarily completes the accurate acquisition, interference removal, and standardization of multi-dimensional pipeline defect data, providing high-quality input for the subsequent feature extraction module. Specific operations and results are as follows: (a) Actual operation and results of data collection.

[0094] 1. Global RGB Data Acquisition: A CCTV pipeline inspection robot (model XX-CCTV-001) was used to move along the pipeline axis at a constant speed of 0.3 m / s, acquiring one frame of the pipeline's inner wall global image every 0.5 m. After acquisition, the images were standardized to 128×128 pixels (pixel density 0.1 mm / pixel) using an image cropping algorithm. One frame of the image accurately covered the suspected defect area. The pipeline interface was located at the left 1 / 3 of the image. Measurements showed the defect was approximately 0.8 m from the pipeline interface. A small amount of brown stains were present on the pipe wall surface, but the suspected defect area was not completely obscured.

[0095] 2. Local RGB Data Acquisition: By calculating the local grayscale variance of the global image in real time, the grayscale variance of the suspected defect area is determined to be 58 (≥ preset threshold 50), triggering the local focusing acquisition mechanism. The robot lens focal length is automatically adjusted to 15mm, and three frames of images are continuously captured on the suspected defect area. The frame with the highest clarity and no motion blur is selected as the valid data. A region of interest (ROI) is constructed by extending 20 pixels outward from the defect area as the center, and cropped to 50×150 pixels (pixel density 0.05mm / pixel). After pixel conversion, the actual length of the defect is approximately 12mm (corresponding to 240 pixels in the image), and the width is gradually distributed (1-3 pixels, corresponding to 0.05~0.15mm in reality). The defect edge can be preliminarily identified.

[0096] 3. Texture Descriptor Extraction: For the cropped local RGB image, the entropy texture descriptor and the histogram of oriented gradients (HOG) texture descriptor are extracted respectively. The specific calculation process and results are as follows: 3-1. Entropy Texture Descriptor: The total number of pixels in the local image (N=7500) (50×150) is statistically analyzed using a grayscale histogram. The number of pixels in the defect area (grayscale value range 30~80) is ng=3200, and the number of pixels in the background area (grayscale value range 100~200) is ng=4300. The grayscale probabilities of the two types of regions are calculated as follows: p({defect})=3200 / 7500≈0.427, p{background}=4300 / 7500≈0.573; Substituting the data, the calculation is as follows:

[0097] The calculation results show that the entropy value of the defect area is significantly higher than that of the background area (the entropy value of the background area alone is about 0.3), which effectively enhances the distinction between defects and background.

[0098] 3-2. HOG Texture Descriptor: Features are extracted according to preset parameters (5×5 pixel cells, 2×2 cells as a block unit, 9-dimensional gradient histogram), and the horizontal and vertical gradients of the image are calculated using the Sobel operator. The core calculation formula is as follows: Gx=I(x+1,y) I(x 1,y) Gy=I(x,y+1) I(x,y 1)

[0099]

[0100] Where Gx and Gy are the gradient values ​​in the horizontal and vertical directions, respectively, and G is the gradient magnitude. Let I(x,y) be the pixel grayscale value at coordinates (x,y), and let G be the gradient direction. In this extraction, the gradient magnitude G in the defect edge region is concentrated between 30-50, and the gradient direction is... The features are mainly distributed between 0° and 20° (corresponding to the edge of a horizontal crack), and a 36-dimensional HOG feature vector is finally generated. Some components are: [0.08, 0.12, 0.95, 0.07, 0.05, ..., 0.04], which accurately captures the edge contour features of the defect.

[0101] (II) Actual operation and results of data preprocessing.

[0102] 1. Image Alignment: Due to a slight attitude shift (pitch angle of approximately 2°) during the movement of the CCTV robot within the pipe, the feature points in the original image are shifted. Coordinate calibration using the pipe's inherent feature points is necessary. The two endpoints of the pipe interface seam are extracted as feature points, with original coordinates of (22,30) and (105,30). The target alignment coordinates are set to (25,30) and (108,30) based on the pipe axis. The affine transformation matrix is ​​solved using the least squares method. The affine transformation formula is:

[0103] Where (xi,yi) are the original feature point coordinates, , Given the coordinates of the aligned feature points, the affine transformation matrix is ​​obtained by solving for them.

[0104] After alignment, the average offset error between the two feature points is 0.6 pixels, which meets the preset accuracy requirement of ≤1 pixel average offset error of feature points, ensuring the accuracy of subsequent feature extraction.

[0105] 2. Noise Suppression: To address interference from salt-and-pepper noise, pipe wall stains, and water reflection within the pipeline, a combined algorithm of Gaussian filtering and adaptive threshold segmentation is employed. The specific results are as follows: 2-1. Gaussian filtering: using a 3×3 Gaussian kernel =0.8, a Gaussian kernel is used to filter a local RGB image. The Gaussian kernel function and the filter calculation formula are as follows:

[0106]

[0107] Where (u,v) are the relative coordinates within the Gaussian kernel, k=1 (3×3 Gaussian kernel), I(x,y) are the original pixel grayscale values, and I'(x,y) are the filtered pixel grayscale values. The original image contained three salt-and-pepper noise pixels (grayscale values ​​of 0 or 255). After filtering, all noise pixels were eliminated, while the details of the defect edges were fully preserved without significant blurring.

[0108] 2-2. Adaptive Threshold Segmentation: To address the interference from stains caused by uneven illumination on the pipe wall, an adaptive threshold segmentation algorithm is adopted. The core calculation formula is as follows:

[0109]

[0110] Where T(x,y) is the local threshold of pixel (x,y). (x,y) represents the average gray value of the 3×3 neighborhood surrounding pixel (x,y), C=5 is the threshold offset, I'(x,y) is the gray value after Gaussian filtering, and B(x,y) is the pixel value of the segmented binary image. The average gray value of the 3×3 neighborhood in this local image is... (x,y)=125, therefore the local threshold T(x,y)=125-5=120; after processing, the brown stains on the pipe wall (grayscale value 130-150) are accurately identified as background and no longer interfere with the identification of defect areas.

[0111] 3. Data Normalization: To eliminate the differences in dimensions between different types of data, based on the statistical results of the training dataset, a max-min normalization algorithm is used. The calculation formula is as follows:

[0112] Where x is the original data value, x' is the normalized data value, and x' is the normalized data value. min x max These are the minimum and maximum values ​​of this type of data in the training dataset, respectively. The specific calculation is as follows: RGB pixel values: training set statistics x min =0, x max =255, the original value of a pixel in the defect area of ​​this local image is x=55, after normalization:

[0113] Entropy: Training set statistics x min =0.1, x max =2.0, the entropy value calculated this time is x=0.98, after normalization:

[0114] (3) HOG features: statistical analysis of a certain component x in the training set min =0, x max =10.0, the original value of this component x=4.8, after normalization:

[0115] After normalization, all data are mapped to the [0,1] interval, which satisfies the input requirements of subsequent neural network modules.

[0116] Module 2: Demonstration application of the dual independent dual-stream convolutional neural network module.

[0117] This module is the core feature extraction and fusion unit. It constructs two independent sub-networks: PD-MDLN (Depth Multi-Feature Network for Pipeline Defects) and PD-MFLN (Multi-Fuse Layer Network for Pipeline Defects). These sub-networks process local detail features and global scene features respectively, and output high-order features in parallel to support subsequent decision-making modules. Specific demonstration applications are as follows: (I) Demonstration Application of PD-MDLN (Local Detail Feature Processing). PD-MDLN adopts a late-stage fully connected layer fusion strategy, focusing on extracting local detailed features of pipeline defects (such as crack width and edge orientation). The specific operation and results are as follows: 1. Network input: Preprocessed local RGB data (3×50×150, 3 channels corresponding to the three RGB channels) and local texture descriptors (2×50×150, 2 channels corresponding to entropy texture and HOG texture respectively, after splicing). Both types of input data have been normalized.

[0118] 2. Feature Map Dimension Evolution: Taking Conv1~MaxPool1 as an example, the accuracy of the feature map dimension evolution is verified. The general calculation formula for the feature map dimension of convolutional and pooling layers is as follows:

[0119]

[0120] Conv1 parameter settings: pad=1, kernel_size=3, stride=1, input dimension (3, 50, 150), substitute into the formula for calculation:

[0121] The Conv1 output feature map has dimensions of (64, 50, 150); the MaxPool1 parameter is set to kernel_size=2 and stride=2. After calculation, the output feature map has dimensions of (64, 25, 75), which is completely consistent with the preset network architecture parameters, verifying the accuracy of the feature extraction process.

[0122] 3. Post-processing fusion and loss calculation: After alternating operations of 14 convolutional layers and 8 max-pooling layers, the local RGB stream and texture stream are input into fully connected layers FC1 and FC2 respectively, with an output dimension of 4096. Subsequently, feature aggregation is completed through average fusion (Fuse_avg) and maximum fusion (Fuse_max) strategies. The fusion formula is as follows:

[0123]

[0124] Where n is the node index of the fully connected layer (n∈[1,4096]), and FC1(n) and FC2(n) are the output values ​​of the nth node of the RGB stream and texture stream fully connected layers, respectively.

[0125] Simultaneously, the network training loss is calculated using the multi-class cross-entropy loss function, as shown in the following formula:

[0126]

[0127] The specific calculation results are as follows: Fully connected layer output: The output value of node 1000 of FC1 (RGB stream) is 0.92, and the output value of node 1000 of FC2 (texture stream) is 0.85.

[0128] Fusion feature calculation:

[0129]

[0130] Loss function calculation: The true label of this sample is crack level 3 (corresponding to the 13th class out of 25 labels, with the 13th bit of the one-hot encoding being 1). The Softmax output probability corresponding to Fuse_avg is 0.31, and the Softmax output probability corresponding to Fuse_max is 0.28. Substituting these values ​​into the formula, the loss function is calculated as follows:

[0131]

[0132]

[0133] The loss value is within a reasonable range in the early stages of training, indicating that the network is effectively learning the detailed features of local defects.

[0134] (II) Demonstration Application of PD-MFLN (Global Scene Feature Processing). PD-MFLN adopts an early convolutional layer fusion strategy, mainly to extract global scene features of pipeline defects (such as defect location and overall pipe wall environment). The specific operation and results are as follows: 1. Network Input: Preprocessed global RGB data (3×128×128) and global texture descriptor (2×128×128) from Module 1. Both types of input data have been normalized and cover the overall scene of the pipeline and global texture information.

[0135] 2. Early Convolutional Fusion: The global RGB stream and texture stream are input into Conv1 and Conv2 convolutional layers respectively. After extracting initial features, the early feature aggregation is completed through the average convolutional fusion (conv_avg) strategy. The core formula is as follows:

[0136] The output value of Conv1 (RGB stream) channel 1 at coordinate (50,50) (corresponding to the suspected defect edge area) is 0.76, and the output value of Conv2 (texture stream) at the same position is 0.68; the preset weight matrix w(1,1)=0.5, filter K(1,1)=0.8, and bias b(1)=0.1 are substituted into the formula for calculation:

[0137] The fused output value is 0.7, which effectively aggregates global RGB features and texture features, and strengthens the correlation between defects and the global environment.

[0138] 3. Fully Connected Layer Output: After advanced feature enhancement through subsequent convolutional and pooling layers, the fused feature map is flattened into a 512-dimensional vector, which is then input to fully connected layers FC3 and FC4 (both with an output dimension of 4096). Specifically, the output value of node 500 in FC3 (RGB fusion stream) is 0.89, and the output value of node 500 in FC4 (texture fusion stream) is 0.83, providing a high-quality global scene feature vector for subsequent weighted voting decisions.

[0139] Module 3: Weighted Voting Decision Module Demonstration Application. This module is the final decision-making unit, integrating the output features of PD-MDLN and PD-MFLN. Through a weighted voting strategy based on rank-K cosine similarity, it completes the identification of defect types and severity, as well as consistency verification. A specific demonstration application is as follows: (I) Feature Standardization. To eliminate the amplitude difference in the output features of the two networks, the output features of the two types of networks are subjected to Softmax standardization, which converts them into probability distribution vectors (both with a dimension of 1×4096). The Softmax normalization formula is as follows:

[0140] Where vi is the i-th element of the original feature vector, and e is the natural constant (e≈2.71828). The specific results are as follows: 1. PD-MDLN Output: After summing the fused feature vectors of Fuse_avg and Fuse_max, substitute them into the Softmax formula to obtain the local detail feature vectors. l The partial elements are: [0.003, 0.016, 0.009, 0.012, ..., 0.002], and the sum of all elements is 1, which meets the standardization requirements.

[0141] 2. PD-MFLN Output: After summing the feature vectors of the FC3 and FC4 fully connected layers, the global scene feature vector is obtained by Softmax normalization. gThe partial elements are: [0.006,0.010,0.007,0.011,...,0.001], which also meets the standardization requirements.

[0142] (II) Cosine Similarity Calculation. A preset sample set M=1000 samples, covering typical samples of 5 types of defects, each with 5 severity levels, including 40 samples of level 3 cracks. The 23rd level 3 crack sample (label 13) in the sample set is selected, and the cosine similarity between the feature vector of the target set and the feature vector of this sample is calculated using the following formula:

[0143] Where A and B are two feature vectors whose similarity is to be calculated, ai and bi are the i-th elements of vectors A and B respectively, and ||A||2 and ||B||2 are the L2 norms of vectors A and B respectively. The specific results are as follows: Local feature similarity:

[0144] 2. Global feature similarity:

[0145] The similarity values ​​are all close to 1, indicating that the feature to be identified is highly matched with the feature of the crack level 3 sample.

[0146] (iii) rank-K weighted voting.

[0147] With the preset rank-K parameter K=5 (experiments have shown that K=3 / sim7 yields the best recognition accuracy), the weighted voting score is calculated as follows:

[0148]

[0149] Where ω is the weighted total score. , These are the local details and global scene feature vectors of the set to be identified, respectively. Let j be the feature vector of the j-th sample in the sample set. Let be the global scene feature vector of the j-th sample set, rankK(.) be the Top-K similarity summation function, and Simk(X,Y) be the similarity value of the k-th rank.

[0150] The specific calculations are as follows: Local feature rank-K summation: The Top-5 similarity values ​​are 0.97, 0.95, 0.93, 0.91, and 0.89, respectively. The summation result is as follows:

[0151] Global feature rank-K summation: The Top-5 similarity values ​​are 0.94, 0.92, 0.90, 0.88, and 0.86, respectively. The summation result is:

[0152] Weighted total score:

[0153] (iv) Decision-making. Based on the weighted voting score, the two core tasks of defect identification and consistency verification were completed. The specific results are as follows: Defect identification: The argmax function is used to select the sample label corresponding to the maximum weighted voting score, as shown in the following formula:

[0154] in, To determine the final label, M=1000 represents the number of samples in the sample set. The weighted voting scores of all samples in the sample set are iterated over; the maximum value is 9.15, corresponding to the 23rd sample (label 13). Therefore, the final identification result is: ( =Label 13: Defect type = Crack (Class 1), Severity = Level 3).

[0155] Consistency verification: The preset consistency verification threshold is t=8.2, and the verification formula is as follows:

[0156] Where v is the verification result, The reference set feature vector (standard defect data). This is the feature vector of the query set (defect data to be verified). This is the reference set. The standard feature vector of crack level 3 (the mean feature value of 40 crack level 3 samples in the sample set), query set For the defect feature vector to be verified, calculate the weighted voting score between the two. =9.15≥t=8.2, therefore the verification result v=1→the defect to be identified is consistent with the standard crack level 3 defect.

[0157] In summary, the present invention has the following beneficial effects: 1. Differentiated fusion architecture improves feature aggregation efficiency: Early convolutional fusion and late fully connected fusion strategies are designed for the local details of pipeline defects and the feature characteristics of the global scene, respectively. This solves the problems of the traditional multimodal network fusion method being single and insufficient coordination between global and local features, and can make feature representation more targeted.

[0158] 2. Enhanced adaptability to complex scenes through multi-feature fusion: Combining RGB data with two texture descriptors (entropy and HOG) effectively reduces the impact of light fluctuations, dirt interference, and water reflection in the pipeline, while strengthening the capture of defect morphology and environmental features, thus improving the recognition accuracy of irregular defects.

[0159] 3. Intelligent decision-making strategy optimizes recognition reliability: The weighted voting strategy integrates the feature distance scores of dual networks and dual modalities, dynamically balances the decision weights of local details and global scenes, avoids recognition errors caused by the failure of a single modality, and thus significantly reduces the false negative rate and false positive rate.

[0160] Regarding the multimodal feature pipeline defect image recognition method provided in the foregoing embodiments, this invention provides a multimodal feature pipeline defect image recognition device, see [link to relevant documentation]. Figure 3 The diagram shows a multimodal feature pipeline defect image recognition device, which includes the following parts: The data acquisition and processing module 302 acquires multi-dimensional data of the target pipeline and sequentially performs image alignment processing, noise suppression processing and data normalization processing on the multi-dimensional data to obtain standardized multi-modal feature data. The multi-dimensional data includes: global image data, local image data and texture descriptors. The dual-network feature extraction module 304 performs local feature extraction processing on local image data and local texture descriptors in multimodal feature data through a deep network model of multi-feature pipeline defects, extracts local detail feature vectors based on a late fully connected layer fusion strategy, and performs global feature extraction processing on global image data and global texture descriptors in multimodal feature data through a multi-fusion layer network model of pipeline defects, extracts global scene feature vectors based on an early convolutional layer fusion strategy. The weighted voting decision module 306 calculates the cosine similarity between the local detail feature vector and the global scene feature vector and the feature vector of each sample in the preset sample set, and performs weighted summation on the preset number of target samples with the highest similarity to obtain a weighted voting score, so as to determine the pipeline defect identification result based on the weighted voting score.

[0161] The multimodal feature pipeline defect image recognition device provided in this application embodiment can significantly improve the accuracy of pipeline defect recognition.

[0162] In one embodiment, when performing the noise suppression processing of multi-dimensional data, the data acquisition and processing module 302 is further configured to: use a Gaussian filtering algorithm to filter the global image data and local image data to remove random noise in the image, and use an adaptive threshold segmentation algorithm to binarize the filtered image to eliminate the occlusion of defect features by pipe wall stains, water reflection and environmental attachments.

[0163] In one embodiment, during the steps of performing local feature extraction processing on local image data and local texture descriptors in multimodal feature data through a deep network model of multi-feature pipeline defects, and extracting local detail feature vectors based on a fusion strategy of a later fully connected layer, the aforementioned dual-network feature extraction module 304 is further configured to: input local image data and local texture descriptors into the first input stream and the second input stream of the deep network model of multi-feature pipeline defects, respectively; perform alternating operations through various convolutional layers and max pooling layers to determine the hierarchical features of local defects; and perform average fusion processing and max fusion processing on the hierarchical features of local defects to obtain local detail feature vectors.

[0164] In one embodiment, during the step of performing global feature extraction processing on global image data and global texture descriptors in multimodal feature data through a multi-fusion layer network model for pipeline defects, and extracting global scene feature vectors based on an early convolutional layer fusion strategy, the aforementioned dual-network feature extraction module 304 is further configured to: input global image data and global texture descriptors into the first and second convolutional layers of the multi-fusion layer network model for pipeline defects respectively, and perform dual convolutional fusion processing on the convolutional output results to determine the activation mean and maximum activation value, so as to obtain the global scene feature vector based on the activation mean and maximum activation value.

[0165] In one embodiment, when performing the step of weighted summation of a preset number of target samples with the highest similarity to obtain a weighted voting score, the weighted voting decision module 306 is further configured to: standardize the local detail feature vector and the global scene feature vector respectively, and select a first target sample set and a second target sample set with the highest cosine similarity for the standardized local detail feature vector and the global scene feature vector respectively; sum the similarities of the first target sample set and the second target sample set respectively, and add the summation results to obtain the weighted voting score.

[0166] In one embodiment, when performing the step of determining the pipeline defect identification result based on the weighted voting score, the weighted voting decision module 306 is further configured to: determine the sample label corresponding to the maximum value of the weighted voting score as the pipeline defect identification result, wherein the pipeline defect identification result includes: defect type and severity.

[0167] In one embodiment, after determining the sample label corresponding to the maximum value of the weighted voting score as the pipeline defect identification result, the weighted voting decision module 306 is further configured to: perform consistency verification processing on the pipeline defect identification result; if the weighted voting score corresponding to the pipeline defect identification result is not less than a preset consistency threshold, then the consistency verification is determined to be passed, and the pipeline defect identification result is output.

[0168] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0169] This invention provides a server, specifically, the server includes a processor and a storage device; the storage device stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0170] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0171] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0172] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0173] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0174] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0175] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0176] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for recognizing multimodal feature pipeline defects from images, characterized in that, The method includes: The target pipeline's multi-dimensional data is acquired, and image alignment, noise suppression, and data normalization are sequentially performed on the multi-dimensional data to obtain standardized multimodal feature data. The multi-dimensional data includes global image data, local image data, and texture descriptors. The local image data and local texture descriptor in the multimodal feature data are processed by a deep network model for pipeline defects. Local detail feature vectors are extracted based on a late fully connected layer fusion strategy. Global feature extraction is also performed on the global image data and global texture descriptor in the multimodal feature data by a multi-fusion layer network model for pipeline defects. Global scene feature vectors are extracted based on an early convolutional layer fusion strategy. The cosine similarity between the local detail feature vector and the global scene feature vector and the feature vector of each sample in the preset sample set is calculated respectively. The target samples with the highest similarity are then weighted and summed to obtain a weighted voting score. The pipeline defect identification result is determined based on the weighted voting score.

2. The multimodal feature pipeline defect image recognition method according to claim 1, characterized in that, The steps for noise suppression processing of the multi-dimensional data include: The global image data and the local image data are filtered using a Gaussian filtering algorithm to remove random noise from the images. The filtered images are then binarized using an adaptive threshold segmentation algorithm to eliminate the occlusion of defect features by pipe wall stains, water reflections, and environmental deposits.

3. The multimodal feature pipeline defect image recognition method according to claim 1, characterized in that, The step of extracting local features from the local image data and local texture descriptors in the multimodal feature data using a deep network model for multi-feature pipeline defects, and extracting local detail feature vectors based on a post-processing fully connected layer fusion strategy, includes: The local image data and the local texture descriptor are respectively input into the first input stream and the second input stream of the deep network model for multi-feature pipeline defects. The model is processed by alternating operations of various convolutional layers and max pooling layers to determine the hierarchical features of the local defects. The hierarchical features of the local defects are then subjected to average fusion processing and max fusion processing to obtain the local detail feature vector.

4. The multimodal feature pipeline defect image recognition method according to claim 1, characterized in that, The step of extracting global features from the global image data and global texture descriptors in the multimodal feature data using a multi-fusion layer network model for pipeline defects, and extracting global scene feature vectors based on an early convolutional layer fusion strategy, includes: The global image data and the global texture descriptor are respectively input into the first and second convolutional layers of the pipeline defect multi-fusion layer network model, and the convolution output is subjected to double convolution fusion processing to determine the activation mean and the maximum activation value, so as to obtain the global scene feature vector based on the activation mean and the maximum activation value.

5. The multimodal feature pipeline defect image recognition method according to claim 1, characterized in that, The step of performing a weighted summation on a preset number of target samples with the highest similarity to obtain a weighted voting score includes: The local detail feature vector and the global scene feature vector are standardized respectively. For the standardized local detail feature vector and the global scene feature vector, a first target sample set and a second target sample set with the highest cosine similarity are selected respectively. The similarity scores of the first target sample set and the second target sample set are summed respectively, and the sums are added together to obtain the weighted voting score.

6. The multimodal feature pipeline defect image recognition method according to claim 1, characterized in that, The step of determining the pipeline defect identification result based on the weighted voting score includes: The sample label corresponding to the maximum value of the weighted voting score is determined as the pipeline defect identification result, wherein the pipeline defect identification result includes: defect type and severity.

7. The multimodal feature pipeline defect image recognition method according to claim 6, characterized in that, After the step of determining the sample label corresponding to the maximum value of the weighted voting score as the pipeline defect identification result, the following steps are included: A consistency verification process is performed on the pipeline defect identification results. If the weighted voting score corresponding to the pipeline defect identification results is not less than a preset consistency threshold, the consistency verification is determined to be successful, and the pipeline defect identification results are output.

8. A multimodal feature pipeline defect image recognition device, characterized in that, The device includes: The data acquisition and processing module acquires multi-dimensional data of the target pipeline and sequentially performs image alignment processing, noise suppression processing, and data normalization processing on the multi-dimensional data to obtain standardized multi-modal feature data. The multi-dimensional data includes global image data, local image data, and texture descriptors. The dual-network feature extraction module performs local feature extraction processing on the local image data and local texture descriptors in the multimodal feature data through a deep network model of multi-feature pipeline defects, extracts local detail feature vectors based on a late fully connected layer fusion strategy, and performs global feature extraction processing on the global image data and global texture descriptors in the multimodal feature data through a multi-fusion layer network model of pipeline defects, extracting global scene feature vectors based on an early convolutional layer fusion strategy. The weighted voting decision module calculates the cosine similarity between the local detail feature vector and the global scene feature vector and the feature vectors of each sample in the preset sample set, and performs weighted summation on the target samples with the highest similarity to obtain a weighted voting score, so as to determine the pipeline defect identification result based on the weighted voting score.

9. A server, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images