Multimodal feature fusion method and device for pipeline defect images

Abstract

The present invention provides a multimodal feature fusion method and device for pipeline defect images, comprising: obtaining a defective pipeline image; extracting morphological features of the pipeline defects in the defective pipeline image to obtain pipeline defect morphological features; extracting local texture features of the pipeline defects in the defective pipeline image to obtain pipeline defect local texture features; and extracting statistical features of the defective pipeline image to obtain pipeline defect statistical features; constructing feature vectors corresponding to the pipeline defect morphological features, pipeline defect local texture features, and pipeline defect statistical features, respectively; and fusing features based on the feature vectors and the feature vectors to obtain a fusion vector. The method constructs feature vectors corresponding to the morphological features, local texture features, and statistical features, fuses the feature vectors to obtain a fusion vector, and trains a machine learning model based on the fusion vector to improve the model's recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of feature fusion technology, and in particular to a pipeline defect image multimodal feature fusion method and device. Background Art

[0002] Currently, the following technologies and their drawbacks exist for pipeline defect feature extraction: 1) Gray-Level Co-occurrence Matrix (GLCM) method, which calculates the grayscale and spatial relationships between pixels within an image by counting the number of pixel occurrences in a specific space. This method performs well in terms of classification accuracy, but increasing the number of grayscale levels significantly increases computational complexity and time consumption. 2) The improved gray-level co-occurrence matrix composite feature vector extraction method improves feature extraction efficiency compared to the aforementioned gray-level co-occurrence matrix method, but extraction time gradually increases as the number of partitions decreases. 3) Gabor filtering method, which uses filters of different scales and directions to extract features from an image. This method has a large feature dimension and requires dimensionality reduction as the scale and direction increase. Computational efficiency and results are significantly affected by scale and direction.

[0003] Furthermore, when multiple methods are used to extract features from pipeline defect images, there is a lack of corresponding feature fusion methods, which will affect the accuracy of the trained machine model in subsequent image recognition. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal feature fusion method and device for pipeline defect images to alleviate the technical problem that the existing technology lacks a corresponding feature fusion method, which will affect the accuracy of the trained machine model in subsequent image recognition, so as to achieve feature fusion and further improve the accuracy of image recognition.

[0005] In a first aspect, an embodiment of the present invention provides a multimodal feature fusion method for a pipeline defect image, comprising: acquiring a defective pipeline image; performing morphological feature extraction on the pipeline defect in the defective pipeline image to obtain a morphological feature of the pipeline defect; and performing local texture feature extraction on the pipeline defect in the defective pipeline image to obtain a local texture feature of the pipeline defect; and performing statistical feature extraction on the defective pipeline image to obtain a statistical feature of the pipeline defect; respectively constructing feature vectors corresponding to the morphological feature of the pipeline defect, the local texture feature of the pipeline defect, and the statistical feature of the pipeline defect; and performing feature fusion based on the feature vectors and preset weights corresponding to the feature vectors to obtain a fusion vector.

[0006] In a preferred embodiment of the present invention, after the step of performing feature fusion according to the above-mentioned feature vector and the preset weights corresponding to the above-mentioned feature vector to obtain the fusion vector, the above-mentioned method includes: training the initial machine learning model based on the above-mentioned fusion vector until the preset training conditions are reached to obtain a trained machine learning model.

[0007] In a preferred embodiment of the present invention, the step of extracting morphological features of the pipeline defect in the defective pipeline image to obtain the morphological features of the pipeline defect includes: calculating a first ratio of the defect portion area of ​​the pipeline defect to the pipeline area in the defective pipeline image using the following formula: Among them, a1 is the first proportion mentioned above, A defect is the area of ​​the defective part, and Apipe is the area of ​​the pipeline. The second ratio of the perimeter of the defective part in the pipeline defect to the perimeter of the pipeline in the defective pipeline image is calculated using the following formula: Among them, a2 is the second proportion mentioned above, P defect is the perimeter of the defect, P pipe is the circumference of the above-mentioned pipeline; the third ratio of the equivalent diameter of the defective part in the above-mentioned pipeline defect to the pipeline diameter in the above-mentioned defective pipeline image is calculated by the following formula: ; ; Among them, a3 is the third proportion mentioned above, D eq is the equivalent diameter of the above defect, D pip is the diameter of the above pipeline; the aspect ratio of the minimum area rectangle surrounding the above pipeline defect is calculated by the following formula: Where a4 is the aspect ratio of the rectangle, Width is the length of the minimum area rectangle, and Height is the width of the minimum area rectangle. The geometric characteristics of the minimum area shape surrounding the pipeline defect are calculated using the following formula: Wherein, a5 is the geometric characteristic. When the minimum area shape is a circle, the value of a5 is 1. The fourth ratio of the convex hull area of ​​the defective portion in the pipeline defect to the area of ​​the defective portion is calculated by the following formula: Among them, a6 is the fourth proportion mentioned above, A convexhull is the convex hull area; the first proportion, the second proportion, the third proportion, the fourth proportion, the rectangular aspect ratio and the geometric characteristics are extracted to obtain the morphological characteristics of the pipeline defect.

[0008] In a preferred embodiment of the present invention, the step of extracting local texture features of the pipeline defect in the defective pipeline image to obtain the local texture features of the pipeline defect includes: based on the following steps, performing texture analysis on the target window where each pixel point is located through a neighborhood image window with preset size parameters around each pixel point in the defective pipeline image to obtain the local texture features of the pipeline defect: Step 1, dividing the defective pipeline image into multiple image windows according to preset size parameters using the following formula; the multiple image windows overlap with each other; window=I(i:i+2, j:j+2), i={1,2,…,w-2}, j={1,2,…,h-2}; wherein window is the target window centered at the pixel coordinate point (i, j), and I(i:i+2, j:j+2) is the neighborhood image window of the target window; Step 2, comparing the center pixel of the target window with the neighborhood pixels of the neighborhood image window around the center pixel using the sign function of the following formula to generate a binary bit result; ; Signum(g,h) is 1 when the center pixel of the target window is greater than or equal to the neighboring pixel of the neighboring image window, otherwise it is 0, g is the center pixel, h is the neighboring pixel, step 3, convert the binary result into a decimal result using the following formula; ;in, is the decimal result of the neighborhood image window of the target window centered at the pixel coordinate point (i, j); step 4, counting the decimal results corresponding to each image window to generate a feature histogram; step 5, using the feature histogram to determine the local texture features of the pipeline defect.

[0009] In a preferred embodiment of the present invention, before the step of extracting statistical features from the defective pipeline image to obtain statistical features of the pipeline defect, the method includes: obtaining a non-defective pipeline image; and the step of extracting statistical features from the defective pipeline image to obtain statistical features of the pipeline defect includes: extracting statistical features from the non-defective pipeline image and the defective pipeline image to obtain statistical features of the pipeline defect.

[0010] In a preferred embodiment of the present invention, the step of extracting statistical features from the defect-free pipeline image and the defective pipeline image to obtain the statistical features of the pipeline defect comprises: dividing the defect-free pipeline image and the defective pipeline image into a plurality of sub-images according to a preset size; and statistically calculating the sum of pixel values ​​of each sub-image in the plurality of sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median value of the pixel value, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the sum of the above pixel values ​​and the standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum absolute value of the pixel values ​​and the square root of the sum of the squares of the pixel values, the value of the product of the maximum absolute value of the pixel values ​​of the image and the sum of the pixel values ​​divided by the total number of pixels, the maximum absolute value of the pixel values ​​of the image, the pixel value of the image The difference between the maximum value and the minimum value, the difference between the maximum value of the image pixel value and the average value of the pixel value; the sum of the above pixel values, the standard deviation of the above pixel values, the square root of the sum of the squares of the above pixel values, the entropy of the above pixel values, the weighted interaction sum of the above pixel intensity and position coordinate minus the square of the average value of the pixel value, the sum of the absolute values ​​of the above adjacent pixel value differences, the above third-order central moment based on pixel value, the above fourth-order central moment based on pixel value, the median value of the above pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the above pixel point with the highest brightness, the product of the sum of the above pixel values ​​and the standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum value of the absolute values ​​of the above pixel values ​​and the square root of the sum of the squares of the pixel values, the value of the product of the maximum value of the absolute values ​​of the above image pixel values ​​and the sum of the pixel values ​​divided by the total number of pixels, the maximum value of the absolute values ​​of the above image pixel values, the difference between the maximum value and the minimum value of the above image pixel values, and the difference between the maximum value of the image pixel value and the average value of the pixel value are determined as the statistical characteristics of the above pipeline defects.

[0011] In a preferred embodiment of the present invention, the following formula is used to calculate the sum of the pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel value, the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median value of the pixel value, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the sum of the above-mentioned pixel values ​​and the above-mentioned standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum absolute value of the pixel values ​​and the square root of the sum of the squares of the pixel values, the value of the product of the maximum absolute value of the pixel values ​​of the above-mentioned image and the sum of the pixel values ​​divided by the total number of pixels, the maximum absolute value of the pixel values ​​of the above-mentioned image, the difference between the maximum value and the minimum value of the pixel values ​​of the above-mentioned image, and the difference between the maximum value of the pixel value of the image and the average value of the pixel value:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] Where W and H represent the width and height of the above sub-image respectively, I w∗h is the pixel intensity at the preset position (w, h) of the above sub-image, Ij is the pixel intensity at the position j of the above sub-image; I ij is the pixel intensity at position (i, j) of the above sub-image; I +1 j is the pixel intensity at position (i+1, j) of the above sub-image; I, is the average value of all pixel intensities of the above sub-image, f(1) is the sum of the above pixel values, f(2) is the standard deviation of the above pixel values, f(3) is the square root of the sum of the squares of the above pixel values, f(4) is the entropy of the above pixel values, f(5) is the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, f(6) is the sum of the absolute values ​​of the differences between the above adjacent pixel values, f(7) is the third-order central moment based on the pixel value, f(8) is the fourth-order central moment based on the pixel value, f(9) is the median value of the above pixel value, and f(10) is the maximum brightness value of the above pixel value. The pixel value of the lowest pixel point, f (11) is the pixel value of the pixel point with the highest brightness, f (12) is the product of the above average value and the above standard deviation, f (13) is the root mean square of the sum of the squares of the above pixel values, f (14) is the product of the maximum absolute value of the above pixel value and the square root of the sum of the squares of the pixel values, f (15) is the product of the maximum absolute value of the above image pixel value and the sum of the pixel values ​​divided by the total number of pixels, f (16) is the maximum absolute value of the above image pixel value, f (17) is the difference between the maximum and minimum values ​​of the above image pixel value, and f (18) is the difference between the maximum value of the above image pixel value and the average value of the pixel value.

[0031] In a preferred embodiment of the present invention, the step of constructing a feature vector corresponding to the above-mentioned statistical feature includes: based on the sum of the above-mentioned pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the above-mentioned pixel values, the square root of the sum of the squares of the above-mentioned pixel values, the entropy of the above-mentioned pixel values, the weighted interaction of the above-mentioned pixel intensity and the position coordinate minus the square of the average value of the pixel value, the sum of the absolute values ​​of the above-mentioned adjacent pixel value differences, the above-mentioned third-order central moment based on the pixel value, the above-mentioned fourth-order central moment based on the pixel value, the median value of the above-mentioned pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the above-mentioned Redundancy analysis is performed on the product of the sum of the pixel values ​​and the standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum absolute value of the pixel values ​​and the square root of the sum of the squares of the pixel values, the product of the maximum absolute value of the image pixel values ​​and the sum of the pixel values ​​divided by the total number of pixels, the maximum absolute value of the image pixel values, the difference between the maximum and minimum values ​​of the image pixel values, the difference between the maximum value of the image pixel value and the average value of the pixel value, pixel intensity, contrast, brightness and saturation to obtain a redundant analysis result; and based on the redundant analysis result, a feature vector corresponding to the statistical feature is constructed.

[0032] In a preferred embodiment of the present invention, the step of performing feature fusion according to the above-mentioned feature vector and the preset weight corresponding to the above-mentioned feature vector to obtain a fused vector includes: taking the above-mentioned feature vector as the input of the initial support vector machine, taking the defect type corresponding to the above-mentioned feature vector as the output of the above-mentioned initial support vector machine, training the above-mentioned initial support vector machine until the preset training standard is reached to obtain an intermediate support vector machine; determining the accuracy corresponding to the above-mentioned feature vector based on the above-mentioned intermediate support vector machine; determining the preset weight of the above-mentioned feature vector according to the above-mentioned accuracy; and normalizing the above-mentioned feature vector according to the above-mentioned feature vector and the above-mentioned preset weight to obtain a fused feature vector.

[0033] In a second aspect, an embodiment of the present invention further provides a multimodal feature fusion device for pipeline defect images, comprising: a data acquisition module for acquiring a defective pipeline image; a feature extraction module for performing morphological feature extraction on the pipeline defect in the defective pipeline image to obtain a morphological feature of the pipeline defect; and performing local texture feature extraction on the pipeline defect in the defective pipeline image to obtain a local texture feature of the pipeline defect; and performing statistical feature extraction on the defective pipeline image to obtain a statistical feature of the pipeline defect; a feature vector construction module for respectively constructing feature vectors corresponding to the morphological feature of the pipeline defect, the local texture feature of the pipeline defect, and the statistical feature of the pipeline defect; and a feature fusion module for performing feature fusion according to the feature vectors and preset weights corresponding to the feature vectors to obtain a fusion vector.

[0034] The embodiments of the present invention have the following beneficial technical effects:

[0035] An embodiment of the present invention provides a method and device for multimodal feature fusion of pipeline defect images, comprising: obtaining a defective pipeline image; extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects; constructing feature vectors corresponding to the morphological features of the pipeline defects, the local texture features of the pipeline defects, and the statistical features of the pipeline defects; and performing feature fusion based on the feature vectors and preset weights corresponding to the feature vectors to obtain a fusion vector. The method constructs feature vectors corresponding to the morphological features, local texture features, and statistical features, fuses the feature vectors to obtain a fusion vector, and trains a machine learning model based on the fusion vector to improve the recognition accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A schematic diagram of a flow chart of a multimodal feature fusion method for pipeline defect images provided by an embodiment of the present invention;

[0038] Figure 2 A schematic flow chart of another multimodal feature fusion method for pipeline defect images provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of a divided image provided by an embodiment of the present invention;

[0040] Figure 4 A schematic structural diagram of a multimodal feature fusion device for pipeline defect images provided by an embodiment of the present invention.

[0041] Icons: 31-data acquisition module; 32-feature extraction module; 33-feature vector construction module; 34-feature fusion module. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0043] When multiple methods are used to extract features from pipeline defect images, there is a lack of corresponding feature fusion methods, which will affect the accuracy of the trained machine model in subsequent image recognition.

[0044] Based on this, embodiments of the present invention provide a method and apparatus for multimodal feature fusion of pipeline defect images. This method constructs feature vectors corresponding to morphological features, local texture features, and statistical features, fuses these feature vectors to obtain a fusion vector, and trains a machine learning model based on the fusion vector to improve the model's recognition accuracy. To facilitate understanding, a multimodal feature fusion method for pipeline defect images is first introduced.

[0045] Example 1

[0046] In an embodiment of the present invention, Figure 1 A flowchart of a multimodal feature fusion method for pipeline defect images provided by an embodiment of the present invention.

[0047] Depend on Figure 1 As can be seen, the method includes:

[0048] Step S101: Acquire a defective pipeline image.

[0049] Step S102: performing morphological feature extraction on the pipeline defect in the defective pipeline image to obtain morphological features of the pipeline defect; performing local texture feature extraction on the pipeline defect in the defective pipeline image to obtain local texture features of the pipeline defect; and performing statistical feature extraction on the defective pipeline image to obtain statistical features of the pipeline defect.

[0050] Step S103: constructing feature vectors corresponding to the pipeline defect morphological features, the pipeline defect local texture features, and the pipeline defect statistical features respectively.

[0051] Step S104: performing feature fusion according to the feature vectors and the preset weights corresponding to the feature vectors to obtain a fusion vector.

[0052] An embodiment of the present invention provides a multimodal feature fusion method for pipeline defect images, comprising: obtaining a defective pipeline image; extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects; constructing feature vectors corresponding to the morphological features of the pipeline defects, the local texture features of the pipeline defects, and the statistical features of the pipeline defects; and performing feature fusion based on the feature vectors and preset weights corresponding to the feature vectors to obtain a fusion vector. The method constructs feature vectors corresponding to the morphological features, local texture features, and statistical features, fuses the feature vectors to obtain a fusion vector, and trains a machine learning model based on the fusion vector to improve the recognition accuracy of the model.

[0053] Example 2

[0054] Based on the above embodiments, Figure 2 A schematic flow chart of another method for multimodal feature fusion of pipeline defect images provided by an embodiment of the present invention.

[0055] Depend on Figure 2 As can be seen, the method includes:

[0056] Step S201: Acquire a defective pipeline image.

[0057] Step S202: performing morphological feature extraction on the pipeline defect in the defective pipeline image to obtain morphological features of the pipeline defect; performing local texture feature extraction on the pipeline defect in the defective pipeline image to obtain local texture features of the pipeline defect; and performing statistical feature extraction on the defective pipeline image to obtain statistical features of the pipeline defect.

[0058] Step S203: constructing feature vectors corresponding to the pipeline defect morphological features, the pipeline defect local texture features, and the pipeline defect statistical features respectively.

[0059] Step S204: performing feature fusion according to the above feature vectors and the preset weights corresponding to the above feature vectors to obtain a fusion vector.

[0060] Step S205: Train the initial machine learning model based on the above fusion vector until the preset training conditions are met to obtain a trained machine learning model.

[0061] In an embodiment of the present invention, the step of extracting morphological features of the pipeline defect in the defective pipeline image to obtain the morphological features of the pipeline defect includes calculating a first ratio of the defect portion area of ​​the pipeline defect to the pipeline area in the defective pipeline image using the following formula:

[0062]

[0063] Among them, a1 is the first proportion mentioned above, A defect is the area of ​​the defective part, and Apipe is the area of ​​the pipeline;

[0064] The second ratio of the circumference of the defective part in the above pipeline defect to the circumference of the pipeline in the above defective pipeline image is calculated using the following formula:

[0065]

[0066] Among them, a2 is the second proportion mentioned above, P defect is the perimeter of the defect, P pipe is the circumference of the above-mentioned pipeline;

[0067] The third ratio of the equivalent diameter of the defective part in the above pipeline defect to the pipeline diameter in the above defective pipeline image is calculated using the following formula:

[0068] ;

[0069] ;

[0070] Among them, a3 is the third proportion mentioned above, D eq is the equivalent diameter of the above defect, D pip is the diameter of the above pipe;

[0071] The aspect ratio of the minimum area rectangle surrounding the above pipeline defect is calculated using the following formula:

[0072]

[0073] Wherein, a4 is the aspect ratio of the rectangle, Width is the length of the minimum area rectangle, and Height is the width of the minimum area rectangle;

[0074] The geometric characteristics of the minimum area shape that encloses the above pipeline defects are calculated using the following formula:

[0075]

[0076] in, a 5 is the above geometric characteristics. When the above minimum area shape is a circle, a The value of 5 is 1;

[0077] The fourth ratio of the convex hull area of ​​the defective part in the above pipeline defect to the area of ​​the defective part is calculated by the following formula:

[0078]

[0079] in, a 6 is the fourth proportion mentioned above, A convexhull is the area of ​​the convex hull mentioned above;

[0080] The first proportion, the second proportion, the third proportion, the fourth proportion, the rectangular aspect ratio, and the geometric characteristics are extracted to obtain the morphological characteristics of the pipeline defect.

[0081] Furthermore, the step of extracting local texture features of the pipeline defect in the defective pipeline image to obtain the local texture features of the pipeline defect includes: performing texture analysis on the target window where each pixel point is located through a neighborhood image window with preset size parameters around each pixel point in the defective pipeline image based on the following steps to obtain the local texture features of the pipeline defect:

[0082] Step 1: Divide the defective pipe image into multiple image windows according to preset size parameters using the following formula; the multiple image windows overlap each other;

[0083] window=I(i:i+2, j:j+2), i={1,2,…,w-2}, j={1,2,…,h-2};

[0084] Wherein, window is the target window centered at pixel coordinate point (i, j), and I(i:i+2, j:j+2) is the neighborhood image window of the target window.

[0085] Step 2: Compare the center pixel of the target window with the neighboring pixels of the neighborhood image window around the center pixel using the sign function according to the following formula to generate a binary bit result;

[0086] ;

[0087] Signum(g,h) is 1 when the center pixel of the target window is greater than or equal to the neighboring pixel of the neighboring image window, otherwise it is 0, g is the center pixel, and h is the neighboring pixel;

[0088] Step 3: Convert the binary result into decimal using the following formula:

[0089] ;

[0090] in, is the decimal result of the above neighborhood image window of the above target window centered at pixel coordinate point (i, j);

[0091] Step 4: Count the above decimal results corresponding to each image window to generate a feature histogram;

[0092] Step 5: Use the feature histogram to determine the local texture features of the pipeline defect.

[0093] Furthermore, before the step of extracting statistical features from the defective pipeline image to obtain statistical features of the pipeline defect, the method includes: obtaining a non-defective pipeline image; and extracting statistical features from the defective pipeline image to obtain statistical features of the pipeline defect, including: extracting statistical features from the non-defective pipeline image and the defective pipeline image to obtain statistical features of the pipeline defect.

[0094] Furthermore, the step of extracting statistical features from the above-mentioned defect-free pipeline image and the above-mentioned defective pipeline image to obtain the statistical features of the above-mentioned pipeline defects includes: dividing the above-mentioned defect-free pipeline image and the above-mentioned defective pipeline image into multiple sub-images according to a preset size; and statistically calculating the sum of pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel value, the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median value of the pixel value, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the sum of the above-mentioned pixel values ​​and the above-mentioned standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum absolute value of the pixel values ​​and the square root of the sum of the squares of the pixel values, the product of the maximum absolute value of the image pixel values ​​and the sum of the pixel values ​​divided by the total number of pixels, the maximum absolute value of the image pixel values, and the maximum value of the image pixel value. and the minimum value, the difference between the maximum value of the image pixel value and the average value of the pixel value; the sum of the above pixel values, the standard deviation of the above pixel values, the square root of the sum of the squares of the above pixel values, the entropy of the above pixel values, the weighted interaction sum of the above pixel intensity and position coordinates minus the square of the average value of the pixel value, the sum of the absolute values ​​of the above adjacent pixel value differences, the above third-order central moment based on pixel value, the above fourth-order central moment based on pixel value, the median value of the above pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the above pixel point with the highest brightness, the product of the sum of the above pixel values ​​and the standard deviation, the root mean square of the sum of the squares of the above pixel values, the product of the maximum value of the absolute value of the above pixel values ​​and the square root of the sum of the squares of the pixel values, the value of the product of the maximum value of the absolute value of the above image pixel values ​​and the sum of the pixel values ​​divided by the total number of pixels, the maximum value of the absolute value of the image pixel values, the difference between the maximum value and the minimum value of the above image pixel values, and the difference between the maximum value of the image pixel value and the average value of the pixel value are determined as the statistical characteristics of the above pipeline defects.

[0095] Furthermore, the following formula is used to calculate the sum of the pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel value, the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median value of the pixel value, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the sum of the above-mentioned pixel values ​​and the above-mentioned standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum absolute value of the pixel values ​​and the square root of the sum of the squares of the pixel values, the value of the product of the maximum absolute value of the pixel values ​​of the above-mentioned image and the sum of the pixel values ​​divided by the total number of pixels, the maximum absolute value of the image pixel values, the difference between the maximum value and the minimum value of the image pixel values, and the difference between the maximum value of the image pixel value and the average value of the pixel value:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114] Where W and H represent the width and height of the above sub-image respectively, I w∗h is the pixel intensity at the preset position (w, h) of the above sub-image, Ij is the pixel intensity at the position j of the above sub-image; I ij is the pixel intensity at position (i, j) of the above sub-image; I +1 j is the pixel intensity at position (i+1, j) of the above sub-image; I, is the average value of all pixel intensities of the above sub-image, f(1) is the sum of the above pixel values, f(2) is the standard deviation of the above pixel values, f(3) is the square root of the sum of the squares of the above pixel values, f(4) is the entropy of the above pixel values, f(5) is the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, f(6) is the sum of the absolute values ​​of the differences between the above adjacent pixel values, f(7) is the third-order central moment based on the pixel value, f(8) is the fourth-order central moment based on the pixel value, f(9) is the median value of the above pixel value, and f(10) is the maximum brightness value of the above pixel value. The pixel value of the lowest pixel point, f (11) is the pixel value of the pixel point with the highest brightness, f (12) is the product of the above average value and the above standard deviation, f (13) is the root mean square of the sum of the squares of the above pixel values, f (14) is the product of the maximum absolute value of the above pixel value and the square root of the sum of the squares of the pixel values, f (15) is the product of the maximum absolute value of the above image pixel value and the sum of the pixel values ​​divided by the total number of pixels, f (16) is the maximum absolute value of the above image pixel value, f (17) is the difference between the maximum and minimum values ​​of the above image pixel value, and f (18) is the difference between the maximum value of the above image pixel value and the average value of the pixel value.

[0115] Furthermore, the step of constructing a feature vector corresponding to the above statistical feature includes: based on the sum of the above pixel values ​​of each sub-image in the above multiple sub-images, the standard deviation of the above pixel values, the square root of the sum of the squares of the above pixel values, the entropy of the above pixel values, the weighted interaction of the above pixel intensity and the position coordinate minus the square of the average pixel value, the sum of the absolute values ​​of the differences between the above adjacent pixel values, the above third-order central moment based on the pixel value, the above fourth-order central moment based on the pixel value, the median value of the above pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the pixel value of the above pixel value Redundancy analysis is performed on the product of the sum and the above-mentioned standard deviation, the root mean square of the sum of the squares of the above-mentioned pixel values, the product of the maximum value of the absolute values ​​of the above-mentioned pixel values ​​and the square root of the sum of the squares of the pixel values, the product of the maximum value of the absolute values ​​of the above-mentioned image pixel values ​​and the sum of the pixel values ​​divided by the value of the total number of pixels, the maximum value of the absolute values ​​of the above-mentioned image pixel values, the difference between the maximum value and the minimum value of the above-mentioned image pixel values, the difference between the maximum value of the image pixel value and the average value of the pixel value, pixel intensity, contrast, brightness and saturation to obtain redundant analysis results; and based on the above-mentioned redundant analysis results, a feature vector corresponding to the above-mentioned statistical feature is constructed.

[0116] In this embodiment, the step of performing redundancy analysis on the above-mentioned f(1) to f(18) and the above-mentioned pixel intensity, contrast, brightness and saturation includes:

[0117] First, canonical correlation coefficients are calculated: these coefficients measure the strength of the linear relationship between two groups of variables. In redundancy analysis, the linear combination that maximizes the canonical correlation coefficient is sought. Then, regression analysis is performed on each pair of canonical variables: a canonical variable is a linear combination of the original variables that represents the relationship between the two groups of variables. Next, redundancy is calculated, which measures the proportion of variance in one variable group that can be explained by another. Finally, based on the canonical correlation coefficient and redundancy, each feature's contribution to the overall relationship is assessed, selecting the statistical features that have the highest positive correlation with the image's pixel intensity, contrast, brightness, and saturation.

[0118] Furthermore, the step of performing feature fusion according to the above-mentioned feature vector and the preset weight corresponding to the above-mentioned feature vector to obtain a fused vector includes: taking the above-mentioned feature vector as the input of the initial support vector machine, taking the defect type corresponding to the above-mentioned feature vector as the output of the above-mentioned initial support vector machine, training the above-mentioned initial support vector machine until the preset training standard is reached to obtain an intermediate support vector machine; determining the accuracy corresponding to the above-mentioned feature vector based on the above-mentioned intermediate support vector machine; determining the preset weight of the above-mentioned feature vector according to the above-mentioned accuracy; and normalizing the above-mentioned feature vector according to the above-mentioned feature vector and the above-mentioned preset weight to obtain a fused feature vector.

[0119] The steps of determining the preset weight of the feature vector according to the accuracy are as follows:

[0120] w1=A1 / (A1+A2+A3)

[0121] w2=A2 / (A1+A2+A3)

[0122] w3=A3 / (A1+A2+A3)

[0123] w1+w2+w3=1

[0124] Among them, A1 is the feature vector corresponding to the above-mentioned pipeline defect morphological feature, A2 is the feature vector corresponding to the above-mentioned pipeline defect local texture feature, A3 is the feature vector corresponding to the above-mentioned pipeline defect statistical feature, w1 is the first weight of the feature vector corresponding to the above-mentioned pipeline defect morphological feature, w2 is the second weight of the feature vector corresponding to the above-mentioned pipeline defect local texture feature, and w3 is the third weight of the feature vector corresponding to the above-mentioned pipeline defect statistical feature.

[0125] Furthermore, the above fused feature vector is expressed by the following formula:

[0126] F=[w1f1,w2f2,w3f3]

[0127] Among them, F is the fusion feature vector, f1 is the feature vector corresponding to the local texture feature of the above pipeline defect, f2 is the feature vector corresponding to the local texture feature of the above pipeline defect, and f3 is the feature vector corresponding to the statistical feature of the above pipeline defect.

[0128] An embodiment of the present invention provides a multimodal feature fusion method for pipeline defect images, comprising: obtaining a defective pipeline image; extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects; respectively constructing feature vectors corresponding to the morphological features of the pipeline defects, the local texture features of the pipeline defects, and the statistical features of the pipeline defects; performing feature fusion according to the feature vectors and the preset weights corresponding to the feature vectors to obtain a fusion vector; training an initial machine learning model based on the fusion vector until the preset training conditions are met to obtain a trained machine learning model. The method respectively constructs feature vectors corresponding to the morphological features, local texture features, and statistical features, fuses the feature vectors to obtain a fusion vector, and trains the machine learning model based on the fusion vector to improve the recognition accuracy of the model.

[0129] Example 3

[0130] For ease of understanding, this embodiment further details the steps of the method for implementing printing watermarks in specific operations based on the second embodiment.

[0131] Specifically, first, a defective pipeline image is obtained. Then, morphological features of the pipeline defects in the defective pipeline image are extracted to obtain morphological features of the pipeline defects. The morphological feature extraction includes steps a1 to a6 in Example 2.

[0132] The basic parameters of any two defective pipeline images are shown in Table 1 below:

[0133] Table 1

[0134]

[0135] Image 1: a1=0.30, a2=0.40, a3=0.60, a4=0.57, a5≈0.94, a6=1.17;

[0136] Image 2: a1=0.17, a2=0.25, a3=0.42, a4=0.60, a5≈1.13, a6=1.25;

[0137] Feature vector of image 1: F1=[0.3 0.4 0.6 0.57 0.94 1.17 ]

[0138] Among them, F1 is the feature vector of image 1.

[0139] Then, local texture features are extracted from the above images.

[0140] Specifically, the image is divided into 8×8 sub-images, and the value at the center of each image represents the pixel. For ease of understanding, Figure 3 A schematic diagram of a divided image provided by an embodiment of the present invention.

[0141] Furthermore, the steps for generating local texture features are as follows:

[0142] (1) Select the center pixel: In this 3x3 window, select the center pixel, let’s say it is 47.

[0143] (2) Compare neighborhood pixels: Compare the center pixel with the eight surrounding pixels to generate a binary pattern.

[0144] (3) Generate binary numbers: For each neighborhood pixel, if it is greater than or equal to the center pixel 47, assign it a binary value of 1, otherwise assign it a binary value of 0.

[0145] The comparison results of the center pixel 47 and the surrounding pixels are as follows:

[0146] 44<47, output 0; 45<47, output 0; 46<47, output 0; 55>47, output 1; 56>47, output 1; 48<47, output 0; 49<47, output 0; 50<47, output 0.

[0147] Therefore, the binary pattern of the center pixel 47 is: 00011100.

[0148] (4) Calculate decimal value: Convert the binary pattern to a decimal value.

[0149] The binary number 00011100 converted to decimal is 28.

[0150] (5) Generate histogram: Repeat the above steps for the entire image or a certain area of ​​the image, generate a local texture feature value for each pixel, and then count the frequency of occurrence of each local texture feature value to construct a histogram.

[0151] Assume that the local texture eigenvalues ​​and their frequencies of the entire image area are as follows: local texture eigenvalue 12 appears 5 times; local texture eigenvalue 22 appears 12 times; local texture eigenvalue 24 appears 7 times; local texture eigenvalue 36 appears 4 times; local texture eigenvalue 45 appears 8 times; and local texture eigenvalue 63 appears 12 times.

[0152] Concatenate the frequency values ​​in the histogram to form a feature vector F2=[5, 8, 7, 4, 8, 12].

[0153] The third feature extraction method is to first extract statistical features, which are specifically shown in Table 2 below:

[0154] Table 2

[0155]

[0156] Then, the typical correlation coefficients were calculated: the following typical correlation coefficients were obtained through RDA analysis, specifically: the typical correlation coefficient between pixel intensity and f(1): 0.95; the typical correlation coefficient between contrast and f(2): 0.90; the typical correlation coefficient between brightness and f(3): 0.85; the typical correlation coefficient between saturation and f(4): 0.80.

[0157] Then, regression analysis was performed. Specifically: pixel intensity has a significant positive effect on f(1); contrast has a significant positive effect on f(2); brightness has a significant positive effect on f(3); and saturation has a significant positive effect on f(4).

[0158] Then, repeating the above calculations for f(5) to f(18) can obtain the first six statistical features with the largest correlation coefficients with pixel intensity, contrast, and brightness, and we can get F3=[8 9 6 8 5 10]

[0159] Finally, image feature vector fusion is performed. Specifically, recognition is first performed using existing intelligent algorithms, such as commonly used support vector machine classification methods, based on feature vectors. The accuracy is calculated using a large number of images, with values ​​of 89%, 92%, and 94%, respectively. The present invention then uses w1, w2, and w3 to represent the weighted proportions of feature vector fusion.

[0160] w1 = A1 / (A1+A2+A3) = 0.32

[0161] w2 = A2 / (A1+A2+A3) = 0.33

[0162] w3 = A3 / (A1+A2+A3) = 0.35

[0163] w1+w2+w3=1

[0164] Then, the Laplace matrix is ​​calculated based on the eigenvectors F1, F2, and F3 obtained by the above three methods to obtain f1, f2, and f3. The calculation results are as follows:

[0165] f 1=

[0166] f 2=

[0167] f 3=

[0168] The feature vector F = w1f1+ w2f2+ w3f3 can be fused

[0169] Example 4

[0170] Based on the above embodiments, Figure 4 A schematic structural diagram of a multimodal feature fusion device for pipeline defect images provided by an embodiment of the present invention.

[0171] Depend on Figure 4 As can be seen, the device includes:

[0172] The data acquisition module 31 is used to acquire images of defective pipelines.

[0173] The feature extraction module 32 is configured to extract morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; extract local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extract statistical features of the defective pipeline image to obtain statistical features of the pipeline defects.

[0174] The feature vector construction module 33 is used to respectively construct feature vectors corresponding to the above-mentioned pipeline defect morphological features, the above-mentioned pipeline defect local texture features and the above-mentioned pipeline defect statistical features.

[0175] The feature fusion module 34 is configured to perform feature fusion according to the feature vectors and the preset weights corresponding to the feature vectors to obtain a fusion vector.

[0176] Among them, the data acquisition module 31, the feature extraction module 32, the feature vector construction module 33 and the feature fusion module 34 are connected in sequence.

[0177] The multimodal feature fusion device for pipeline defect images provided in the embodiments of the present invention shares the same technical features as the multimodal feature fusion method for pipeline defect images provided in the aforementioned embodiments, and thus solves the same technical problems and achieves the same technical effects. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating procedures of the device described above can be referenced to the corresponding procedures in the aforementioned method embodiments and will not be further elaborated here.

[0178] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" and the like indicate positions or locations based on the positions shown in the accompanying drawings, or the positions or locations in which the inventive product is typically placed when in use. These terms are intended solely to facilitate the description of the present invention and to simplify the description, and are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third," etc., are used solely to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0179] Furthermore, terms such as "horizontal," "vertical," and "overhanging" do not necessarily require that a component be absolutely horizontal or overhanging, but rather that it may be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but rather that it may be slightly tilted.

[0180] In the description of the present invention, it should also be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal feature fusion method for pipeline defect images, characterized in that: include: Acquire defective pipeline images; performing morphological feature extraction on the pipeline defect in the defective pipeline image to obtain pipeline defect morphological features; performing local texture feature extraction on the pipeline defect in the defective pipeline image to obtain pipeline defect local texture features; and performing statistical feature extraction on the defective pipeline image to obtain pipeline defect statistical features; Constructing feature vectors corresponding to the pipeline defect morphological feature, the pipeline defect local texture feature, and the pipeline defect statistical feature respectively; Perform feature fusion according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector; The step of extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects includes: Dividing the non-defective pipeline image and the defective pipeline image into a plurality of sub-images according to a preset size; performing redundancy analysis based on the sum of pixel values ​​of each sub-image in the multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median of the pixel values, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the sum of the pixel values ​​and the standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum absolute value of the pixel values ​​and the square root of the sum of the squares of the pixel values, the value of the product of the maximum absolute value of the image pixel values ​​and the sum of the pixel values ​​divided by the total number of pixels, the maximum absolute value of the image pixel values, the difference between the maximum and minimum values ​​of the image pixel values, the difference between the maximum value of the image pixel value and the average value of the pixel value, pixel intensity, contrast, brightness, and saturation, to obtain statistical characteristics of the pipeline defect; The steps of performing redundancy analysis include: Calculate the canonical correlation coefficient, which measures the strength of the linear relationship between two groups of variables; Calculate redundancy, which measures the proportion of variance in one variable group that can be explained by another variable group. Based on the canonical correlation coefficient and redundancy, the statistical features with the highest positive correlation with the pixel intensity, contrast, brightness and saturation of the image were screened out.

2. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: After the step of performing feature fusion according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector, the method includes: The initial machine learning model is trained based on the fusion vector until a preset training condition is reached to obtain a trained machine learning model.

3. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: The step of extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects includes: The first ratio of the defect area of ​​the pipeline defect to the pipeline area in the defective pipeline image is calculated using the following formula: Among them, a1 is the first proportion, A defect is the area of ​​the defective part, and Apipe is the area of ​​the pipeline; The second ratio of the circumference of the defective portion in the pipeline defect to the circumference of the pipeline in the defective pipeline image is calculated using the following formula: Among them, a2 is the second proportion, P defect is the perimeter of the defect, P pipe is the circumference of the pipeline; The third ratio of the equivalent diameter of the defective portion in the pipeline defect to the pipeline diameter in the defective pipeline image is calculated using the following formula: ； ； Among them, a3 is the third proportion, D eq is the equivalent diameter of the defect, D pip is the pipe diameter; The aspect ratio of the minimum area rectangle surrounding the pipeline defect is calculated by the following formula: Wherein, a4 is the aspect ratio of the rectangle, Width is the length of the minimum area rectangle, and Height is the width of the minimum area rectangle; The geometric characteristics of the minimum area shape surrounding the pipeline defect are calculated by the following formula: Wherein, a5 is the geometric characteristic. When the minimum area shape is a circle, the value of a5 is 1; The fourth ratio of the convex hull area of ​​the defective portion in the pipeline defect to the area of ​​the defective portion is calculated using the following formula: Among them, a6 is the fourth proportion, A convexhull is the convex hull area; The first proportion, the second proportion, the third proportion, the fourth proportion, the rectangular aspect ratio, and the geometric characteristics are extracted to obtain morphological characteristics of the pipeline defect.

4. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: The step of extracting local texture features of the pipeline defect in the defective pipeline image to obtain the local texture features of the pipeline defect comprises: Based on the following steps, texture analysis is performed on the target window where each pixel point is located using a neighborhood image window with preset size parameters around each pixel point in the defective pipeline image to obtain the local texture features of the pipeline defect: Step 1: Divide the defective pipe image into multiple image windows according to preset size parameters using the following formula; the multiple image windows overlap each other; window=I(i:i+2, j:j+2), i={1,2,…,w-2}, j={1,2,…,h-2}; Wherein, window is the target window centered at pixel coordinate point (i, j), and I(i:i+2, j:j+2) is the neighborhood image window of the target window; Step 2: Compare the center pixel of the target window with the neighboring pixels of the neighborhood image window around the center pixel through a sign function using the following formula to generate a binary bit result; ； Signum(g,h) is 1 when the center pixel of the target window is greater than or equal to the neighborhood pixel of the neighborhood image window, otherwise it is 0, g is the center pixel, h is the neighborhood pixel, Step 3: Convert the binary result into a decimal result using the following formula; ； in, is the decimal result of the neighborhood image window of the target window centered at pixel coordinate point (i, j); Step 4, counting the decimal results corresponding to each image window to generate a feature histogram; Step 5: Using the feature histogram, determine the local texture features of the pipeline defect.

5. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: Before the step of extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects, the method includes: Acquire defect-free pipeline images; The step of extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects includes: Statistical features are extracted from the non-defective pipeline image and the defective pipeline image to obtain statistical features of the pipeline defect.

6. The pipeline defect image multimodal feature fusion method according to claim 5, characterized in that: The following formula is used to calculate the sum of the pixel values ​​of each sub-image in the multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median value of the pixel value, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the sum of the pixel values ​​and the standard deviation, the root mean square of the sum of the squares of the pixel values, the product of the maximum absolute value of the pixel values ​​and the square root of the sum of the squares of the pixel values, the value of the product of the maximum absolute value of the image pixel values ​​and the sum of the pixel values ​​divided by the total number of pixels, the maximum absolute value of the image pixel values, the difference between the maximum and minimum values ​​of the image pixel values, and the difference between the maximum value of the image pixel values ​​and the average value of the pixel values: Wherein, W and H represent the width and height of the sub-image respectively, I w∗h is the pixel intensity at the preset position (w, h) of the sub-image, Ij is the pixel intensity at the position j of the sub-image; I ij is the pixel intensity of the position (i, j) of the sub-image; I +1 j is the pixel intensity at position (i+1, j) of the sub-image; I, is the average value of all pixel intensities of the sub-image, f(1) is the sum of the pixel values, f(2) is the standard deviation of the pixel values, f(3) is the square root of the sum of the squares of the pixel values, f(4) is the entropy of the pixel values, f(5) is the weighted interaction of the pixel intensity and the position coordinate minus the square of the average pixel value, f(6) is the sum of the absolute values ​​of the differences between adjacent pixel values, f(7) is the third-order central moment based on the pixel value, f(8) is the fourth-order central moment based on the pixel value, f(9) is the median value of the pixel value, and f(10) is the lowest brightness. The pixel value of the pixel point, f(11) is the pixel value of the pixel point with the highest brightness, f(12) is the product of the sum of the pixel values ​​and the standard deviation, f(13) is the root mean square of the sum of the squares of the pixel values, f(14) is the product of the maximum absolute value of the pixel value and the square root of the sum of the squares of the pixel values, f(15) is the value of the product of the maximum absolute value of the image pixel value and the sum of the pixel values ​​divided by the total number of pixels, f(16) is the maximum absolute value of the image pixel value, f(17) is the difference between the maximum and minimum values ​​of the image pixel value, and f(18) is the difference between the maximum value of the image pixel value and the average value of the pixel value.

7. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: The step of performing feature fusion according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector includes: Using the feature vector as the input of an initial support vector machine, using the defect type corresponding to the feature vector as the output of the initial support vector machine, training the initial support vector machine until a preset training standard is reached, and obtaining an intermediate support vector machine; Determining the accuracy corresponding to the feature vector based on the intermediate state support vector machine; Determining a preset weight of the feature vector according to the accuracy; The feature vector is normalized according to the feature vector and the preset weight to obtain a fused feature vector.

8. A multimodal feature fusion device for pipeline defect images, characterized in that: The method for implementing claim 1 comprises: A data acquisition module, used to acquire defective pipeline images; a feature extraction module configured to extract morphological features of the pipeline defect in the defective pipeline image to obtain morphological features of the pipeline defect; extract local texture features of the pipeline defect in the defective pipeline image to obtain local texture features of the pipeline defect; and extract statistical features of the defective pipeline image to obtain statistical features of the pipeline defect; A feature vector construction module, used to respectively construct feature vectors corresponding to the pipeline defect morphological features, the pipeline defect local texture features, and the pipeline defect statistical features; The feature fusion module is used to perform feature fusion according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector.

Images