Pipeline defect image multi-modal feature fusion method and device

Abstract

The invention provides a pipeline defect image multi-modal feature fusion method and device. The method comprises the following steps: acquiring a defect pipeline image; performing morphological feature extraction on pipeline defects of the defective pipeline image to obtain morphological features of the pipeline defects; local texture feature extraction is carried out on pipeline defects of the defective pipeline image, and local texture features of the pipeline defects are obtained; performing statistical feature extraction on the defective pipeline image to obtain pipeline defect statistical features; respectively constructing feature vectors corresponding to the pipeline defect morphological features, the pipeline defect local texture features and the pipeline defect statistical features; and performing feature fusion according to the feature vectors to obtain a fusion vector. According to the method, feature vectors corresponding to morphological features, local texture features and statistical features are constructed respectively, the feature vectors are fused to obtain a fusion vector, and a machine learning model is trained based on the fusion vector to improve the recognition precision of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of feature fusion, and in particular to a pipeline defect image multimodal feature fusion method and device. Background Art

[0002] At present, the existing pipeline defect feature extraction technologies have the following defects: 1) Gray-Level Co-occurrence Matrix (GLCM), that is, by calculating the number of pixel points in a specific space, the gray-level relationship between the pixels in the image and the space is obtained. This method performs well in terms of classification accuracy, but the increase in the number of gray levels will significantly increase the amount of calculation and time consumption; 2) Compared with the above-mentioned gray-level co-occurrence matrix method, the improved gray-level co-occurrence matrix composite feature vector extraction method has improved feature extraction efficiency, but as the partition decreases, the extraction time will gradually increase; 3) Gabor filtering method, that is, using filters of different scales and directions to extract features from the image. This method has a large feature dimension, and as the scale and direction increase, dimensionality reduction processing is required. The calculation efficiency and results are greatly affected by the scale and direction.

[0003] Furthermore, when multiple methods are used to extract features from pipeline defect images, there is a lack of corresponding feature fusion methods, which will affect the accuracy of the trained machine model in subsequent image recognition. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal feature fusion method and device for pipeline defect images to alleviate the technical problem that the prior art lacks a corresponding feature fusion method, which will affect the accuracy of the trained machine model in subsequent image recognition, so as to achieve feature fusion and further improve the accuracy of image recognition.

[0005] In a first aspect, an embodiment of the present invention provides a multimodal feature fusion method for a pipeline defect image, comprising: acquiring a defective pipeline image; performing morphological feature extraction on the pipeline defect of the defective pipeline image to obtain a morphological feature of the pipeline defect; and performing local texture feature extraction on the pipeline defect of the defective pipeline image to obtain a local texture feature of the pipeline defect; and performing statistical feature extraction on the defective pipeline image to obtain a statistical feature of the pipeline defect; respectively constructing feature vectors corresponding to the morphological feature of the pipeline defect, the local texture feature of the pipeline defect, and the statistical feature of the pipeline defect; and performing feature fusion according to the feature vector and the preset weights corresponding to the feature vector to obtain a fusion vector.

[0006] In a preferred embodiment of the present invention, after performing feature fusion according to the above-mentioned feature vector and the preset weights corresponding to the above-mentioned feature vector to obtain the fusion vector, the above-mentioned method includes: training the initial machine learning model based on the above-mentioned fusion vector until the preset training conditions are reached to obtain a trained machine learning model.

[0007] In a preferred embodiment of the present invention, the step of extracting morphological features of the pipeline defect in the defective pipeline image to obtain the morphological features of the pipeline defect includes: calculating a first ratio of the defective portion area of ​​the pipeline defect to the pipeline area in the defective pipeline image by the following formula: Among them, a1 is the first proportion mentioned above, A defect is the area of ​​the defective part, and Apipe is the area of ​​the pipeline. The second ratio of the perimeter of the defective part in the pipeline defect to the perimeter of the pipeline in the defective pipeline image is calculated by the following formula: Among them, a2 is the second proportion mentioned above, P defect is the circumference of the defect, P pipe is the circumference of the above pipeline; the third ratio of the equivalent diameter of the defective part in the above pipeline defect to the pipeline diameter in the above defective pipeline image is calculated by the following formula: ; ; Among them, a3 is the third proportion mentioned above, D eq is the equivalent diameter of the above defect, D pip is the diameter of the above pipeline; the rectangular aspect ratio of the minimum area rectangle surrounding the above pipeline defect is calculated by the following formula: Wherein, a4 is the aspect ratio of the rectangle, Width is the length of the minimum area rectangle, and Height is the width of the minimum area rectangle. The geometric characteristics of the minimum area shape surrounding the pipeline defect are calculated by the following formula: Wherein, a5 is the geometric characteristic. When the minimum area shape is a circle, the value of a5 is 1. The convex hull area of ​​the defective part in the above pipeline defect and the fourth ratio of the area of ​​the defective part are calculated by the following formula: Among them, a6 is the fourth proportion mentioned above, A convexhull is the convex hull area; the first proportion, the second proportion, the third proportion, the fourth proportion, the rectangular aspect ratio and the geometric characteristics are extracted to obtain the morphological characteristics of the pipeline defect.

[0008] In a preferred embodiment of the present invention, the step of extracting local texture features of the pipeline defects of the defective pipeline image to obtain local texture features of the pipeline defects includes: based on the following steps, texture analysis is performed on the target window where each pixel point is located through a neighborhood image window of a preset size parameter around each pixel point in the defective pipeline image to obtain local texture features of the pipeline defect: Step 1, dividing the defective pipeline image into multiple image windows according to a preset size parameter by the following formula; the multiple image windows overlap each other; window=I(i:i+2, j:j+2), i={1,2,…,w−2}, j={1,2,…,h−2}; wherein window is a target window centered on the pixel coordinate point (i, j), and I(i:i+2, j:j+2) is a neighborhood image window of the target window; Step 2, comparing the central pixel of the target window with the neighborhood pixels of the neighborhood image window around the central pixel by a sign function by the following formula to generate a binary bit result; ; Signum(g,h) is 1 when the center pixel of the target window is greater than or equal to the neighborhood pixel of the neighborhood image window, otherwise it is 0, g is the center pixel, h is the neighborhood pixel, step 3, convert the binary result into a decimal result by the following formula; ;in, is the decimal result of the above-mentioned neighborhood image window of the above-mentioned target window centered at the pixel coordinate point (i, j); step 4, counting the above-mentioned decimal results corresponding to each image window to generate a feature histogram; step 5, using the above-mentioned feature histogram to determine the local texture features of the above-mentioned pipeline defects.

[0009] In a preferred embodiment of the present invention, before the step of extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects, the method includes: acquiring a non-defective pipeline image; and extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects, including: extracting statistical features from the non-defective pipeline image and the defective pipeline image to obtain statistical features of the pipeline defects.

[0010] In a preferred embodiment of the present invention, the step of extracting statistical features from the above-mentioned defect-free pipeline image and the above-mentioned defective pipeline image to obtain the statistical features of the above-mentioned pipeline defects includes: dividing the above-mentioned defect-free pipeline image and the above-mentioned defective pipeline image into multiple sub-images according to a preset size; and statistically calculating the sum of pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the pixel values, the square root of the sum of squares of the pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the above-mentioned pixel values ​​minus the square of the above-mentioned average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel values, the fourth-order central moment based on the pixel values, the median value of the pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the product of the above-mentioned average value and the above-mentioned standard deviation, the ratio of the square root of the sum of squares of the pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of squares of the pixel values, and the value of the product of the above-mentioned image maximum value and the sum of pixel values ​​divided by the total number of pixels. , the maximum value of the above image, the image contrast, and the difference between the maximum value of the image and the average value; the sum of the above pixel values, the standard deviation of the above pixel values, the square root of the sum of the squares of the above pixel values, the entropy of the above pixel values, the sum of the products of the above pixel values ​​and the average value of the above pixel values ​​minus the square of the above average value, the sum of the absolute values ​​of the differences between the above adjacent pixel values, the third-order central moment based on the pixel values, the fourth-order central moment based on the pixel values, the median value of the above pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the product of the above average value and the standard deviation, the square root of the sum of the squares of the above pixel values, the ratio of the square root of the sum of the squares of the above pixel values ​​to the total number of pixels, the product of the above image maximum value and the square root of the sum of the squares of the pixel values, the product of the above image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the above image maximum value, the image contrast, and the difference between the image maximum value and the average value are determined as the statistical characteristics of the above pipeline defects.

[0011] In a preferred embodiment of the present invention, the sum of the pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the above-mentioned pixel values ​​minus the square of the above-mentioned average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median value of the pixel value, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the above-mentioned average value and the above-mentioned standard deviation, the ratio of the square root of the sum of the squares of the pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of the squares of the pixel values, the product of the above-mentioned image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the above-mentioned image maximum value, the image contrast, and the difference between the image maximum value and the average value are statistically calculated by the following formula:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] Where W and H represent the width and height of the above sub-image, respectively. w∗h is the pixel intensity at the preset position (w, h) of the above sub-image, and Ij is the pixel intensity at the position j of the above sub-image; I ij is the pixel intensity at position (i, j) of the above sub-image; I +1j is the pixel intensity at position (i+1, j) of the above sub-image; I, is the average value of all pixel intensities of the above sub-image, f(1) is the sum of the above pixel values, f(2) is the standard deviation of the above pixel values, f(3) is the square root of the sum of the squares of the above pixel values, f(4) is the entropy of the above pixel values, f(5) is the sum of the products of the above pixel values ​​and the average value of the above pixel values ​​minus the square of the above average value, f(6) is the sum of the absolute values ​​of the differences between the above adjacent pixel values, f(7) is the third-order central moment based on the pixel values, f(8) is the fourth-order central moment based on the pixel values, and f(9) is the median of the above pixel values. value, f(10) is the pixel value of the pixel with the lowest brightness, f(11) is the pixel value of the pixel with the highest brightness, f(12) is the product of the average value and the standard deviation, f(13) is the square root of the sum of squares of the pixel values, f(14) is the ratio of the square root of the sum of squares of the pixel values ​​to the total number of pixels, f(15) is the product of the maximum value of the image and the square root of the sum of squares of the pixel values, f(16) is the product of the maximum value of the image and the sum of pixel values ​​divided by the total number of pixels, f(17) is the maximum value of the image, and f(18) is the image contrast and the difference between the image maximum and the average value.

[0030] In a preferred embodiment of the present invention, the step of constructing a feature vector corresponding to the above-mentioned statistical feature includes: performing redundancy analysis based on the sum of the above-mentioned pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the above-mentioned pixel values, the square root of the sum of the squares of the above-mentioned pixel values, the entropy of the above-mentioned pixel values, the sum of the products of the above-mentioned pixel values ​​and the average value of the above-mentioned pixel values ​​minus the square of the above-mentioned average value, the sum of the absolute values ​​of the differences between the above-mentioned adjacent pixel values, the above-mentioned third-order central moment based on the pixel values, the above-mentioned fourth-order central moment based on the pixel values, the median value of the above-mentioned pixel values, the pixel value of the above-mentioned pixel point with the lowest brightness, the pixel value of the above-mentioned pixel point with the highest brightness, the product of the above-mentioned average value and the above-mentioned standard deviation, the square root of the sum of the squares of the above-mentioned pixel values, the ratio of the square root of the sum of the squares of the above-mentioned pixel values ​​to the total number of pixels, the product of the above-mentioned image maximum value and the square root of the sum of the squares of the pixel values, the value of the product of the above-mentioned image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the above-mentioned image maximum value, the above-mentioned image contrast and the difference between the image maximum value and the average value, pixel intensity, contrast, brightness and saturation to obtain a redundant analysis result; and constructing the feature vector corresponding to the above-mentioned statistical feature according to the above-mentioned redundant analysis result.

[0031] In a preferred embodiment of the present invention, the step of performing feature fusion according to the above-mentioned feature vector and the preset weight corresponding to the above-mentioned feature vector to obtain a fused vector includes: taking the above-mentioned feature vector as the input of an initial support vector machine, taking the defect type corresponding to the above-mentioned feature vector as the output of the above-mentioned initial support vector machine, training the above-mentioned initial support vector machine until a preset training standard is reached to obtain an intermediate support vector machine; determining the accuracy corresponding to the above-mentioned feature vector based on the above-mentioned intermediate support vector machine; determining the preset weight of the above-mentioned feature vector according to the above-mentioned accuracy; and normalizing the above-mentioned feature vector according to the above-mentioned feature vector and the above-mentioned preset weight to obtain a fused feature vector.

[0032] In a second aspect, an embodiment of the present invention further provides a multimodal feature fusion device for pipeline defect images, comprising: a data acquisition module, used to acquire a defective pipeline image; a feature extraction module, used to extract morphological features of the pipeline defects in the above-mentioned defective pipeline image to obtain morphological features of the pipeline defects; and, to extract local texture features of the pipeline defects in the above-mentioned defective pipeline image to obtain local texture features of the pipeline defects; and, to extract statistical features of the above-mentioned defective pipeline image to obtain statistical features of the pipeline defects; a feature vector construction module, used to respectively construct feature vectors corresponding to the above-mentioned pipeline defect morphological features, the above-mentioned pipeline defect local texture features and the above-mentioned pipeline defect statistical features; and a feature fusion module, used to perform feature fusion according to the above-mentioned feature vectors and preset weights corresponding to the above-mentioned feature vectors to obtain a fusion vector.

[0033] The embodiments of the present invention have the following beneficial technical effects: The embodiment of the present invention provides a method and device for multimodal feature fusion of pipeline defect images, including: acquiring a defective pipeline image; extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects; respectively constructing feature vectors corresponding to the morphological features of the pipeline defects, the local texture features of the pipeline defects, and the statistical features of the pipeline defects; and performing feature fusion according to the feature vectors and the preset weights corresponding to the feature vectors to obtain a fusion vector. The method respectively constructs feature vectors corresponding to the morphological features, local texture features, and statistical features, fuses the feature vectors to obtain a fusion vector, and trains a machine learning model based on the fusion vector to improve the recognition accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A schematic diagram of a flow chart of a pipeline defect image multimodal feature fusion method provided by an embodiment of the present invention; Figure 2 A schematic flow chart of another pipeline defect image multimodal feature fusion method provided by an embodiment of the present invention; Figure 3 A schematic diagram of a divided image provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a multimodal feature fusion device for pipeline defect images provided in an embodiment of the present invention.

[0036] Icons: 31-data acquisition module; 32-feature extraction module; 33-feature vector construction module; 34-feature fusion module. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0038] When multiple methods are used to extract features from pipeline defect images, there is a lack of corresponding feature fusion methods, which will affect the accuracy of the trained machine model in subsequent image recognition.

[0039] Based on this, an embodiment of the present invention provides a pipeline defect image multimodal feature fusion method and device, which respectively constructs feature vectors corresponding to morphological features, local texture features and statistical features, fuses the feature vectors to obtain a fusion vector, and trains a machine learning model based on the fusion vector to improve the recognition accuracy of the model. For ease of understanding, a pipeline defect image multimodal feature fusion method is first introduced.

[0040] Example 1 In an embodiment of the present invention, Figure 1 A schematic diagram of a flow chart of a pipeline defect image multimodal feature fusion method provided in an embodiment of the present invention.

[0041] Depend on Figure 1 As can be seen, the method includes: Step S101: Acquire a defective pipeline image.

[0042] Step S102: extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects.

[0043] Step S103: constructing feature vectors corresponding to the pipeline defect morphological features, the pipeline defect local texture features and the pipeline defect statistical features respectively.

[0044] Step S104: performing feature fusion according to the feature vectors and preset weights corresponding to the feature vectors to obtain a fusion vector.

[0045] The embodiment of the present invention provides a method for multimodal feature fusion of pipeline defect images, including: acquiring a defective pipeline image; extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects; respectively constructing feature vectors corresponding to the morphological features of the pipeline defects, the local texture features of the pipeline defects, and the statistical features of the pipeline defects; and performing feature fusion according to the feature vectors and the preset weights corresponding to the feature vectors to obtain a fusion vector. The method respectively constructs feature vectors corresponding to the morphological features, local texture features, and statistical features, fuses the feature vectors to obtain a fusion vector, and trains a machine learning model based on the fusion vector to improve the recognition accuracy of the model.

[0046] Example 2 Based on the above embodiments, Figure 2 A schematic flow chart of another pipeline defect image multimodal feature fusion method provided in an embodiment of the present invention.

[0047] Depend on Figure 2 As can be seen, the method includes: Step S201: Acquire a defective pipeline image.

[0048] Step S202: extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects.

[0049] Step S203: constructing feature vectors corresponding to the pipeline defect morphological features, the pipeline defect local texture features, and the pipeline defect statistical features, respectively.

[0050] Step S204: performing feature fusion according to the feature vectors and preset weights corresponding to the feature vectors to obtain a fusion vector.

[0051] Step S205: Train the initial machine learning model based on the above fusion vector until the preset training conditions are met to obtain a trained machine learning model.

[0052] In an embodiment of the present invention, the step of extracting morphological features of the pipeline defect in the defective pipeline image to obtain the morphological features of the pipeline defect includes: calculating a first ratio of the defect portion area of ​​the pipeline defect to the pipeline area in the defective pipeline image by the following formula:

[0053] Among them, a1 is the first proportion mentioned above, A defect is the area of ​​the defective part, Apipe is the area of ​​the pipeline; The second ratio of the circumference of the defective part in the above pipeline defect to the circumference of the pipeline in the above defective pipeline image is calculated by the following formula:

[0054] Among them, a2 is the second proportion mentioned above, P defect is the circumference of the defect, P pipe is the circumference of the above pipeline; The third ratio of the equivalent diameter of the defective part in the above pipeline defect to the pipeline diameter in the above defective pipeline image is calculated by the following formula: ; ; Among them, a3 is the third proportion mentioned above, D eq is the equivalent diameter of the above defect, D pip is the diameter of the above pipe; The rectangular aspect ratio of the minimum area rectangle surrounding the above pipeline defect is calculated by the following formula:

[0055] Wherein, a4 is the aspect ratio of the above rectangle, Width is the length of the above minimum area rectangle, and Height is the width of the above minimum area rectangle; The geometric properties of the minimum area shape that encloses the above pipeline defects are calculated by the following formula:

[0056] in, a 5 is the above geometric characteristic. When the above minimum area shape is a circle, a The value of 5 is 1; The fourth ratio of the convex hull area of ​​the defective part in the above pipeline defect to the area of ​​the defective part is calculated by the following formula:

[0057] in, a 6 is the fourth proportion mentioned above, A convexhull is the area of ​​the convex hull mentioned above; The first proportion, the second proportion, the third proportion, the fourth proportion, the rectangular aspect ratio and the geometric characteristics are extracted to obtain the morphological characteristics of the pipeline defect.

[0058] Furthermore, the step of extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects includes: based on the following steps, texture analysis is performed on the target window where each pixel point is located through a neighborhood image window with a preset size parameter around each pixel point in the defective pipeline image to obtain the local texture features of the pipeline defect: Step 1: Divide the defective pipeline image into a plurality of image windows according to preset size parameters using the following formula; the plurality of image windows overlap each other; window=I(i:i+2, j:j+2), i={1,2,…,w−2}, j={1,2,…,h−2}; Wherein, window is the target window centered at the pixel coordinate point (i, j), and I(i:i+2, j:j+2) is the neighborhood image window of the above target window; Step 2, comparing the central pixel of the target window with the neighboring pixels of the neighborhood image window around the central pixel through a sign function using the following formula to generate a binary bit result; ; Signum(g,h) is 1 when the central pixel of the target window is greater than or equal to the neighboring pixel of the neighboring image window, otherwise it is 0, g is the central pixel, and h is the neighboring pixel; Step 3, convert the above binary result into a decimal result by the following formula; ; in, is the decimal result of the above neighborhood image window of the above target window centered at the pixel coordinate point (i, j); Step 4, counting the above decimal results corresponding to each image window to generate a feature histogram; Step 5: The feature histogram is used to determine the local texture features of the pipeline defect.

[0059] Furthermore, before the step of extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects, the method includes: acquiring a non-defective pipeline image; extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects, including: extracting statistical features from the non-defective pipeline image and the defective pipeline image to obtain statistical features of the pipeline defects.

[0060] Furthermore, the step of extracting statistical features from the above-mentioned defect-free pipeline image and the above-mentioned defective pipeline image to obtain the statistical features of the above-mentioned pipeline defects includes: dividing the above-mentioned defect-free pipeline image and the above-mentioned defective pipeline image into multiple sub-images according to preset sizes; and statistically calculating the sum of pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the pixel values, the square root of the sum of squares of the pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the above-mentioned pixel values ​​minus the square of the above-mentioned average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel values, the fourth-order central moment based on the pixel values, the median value of the pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the product of the above-mentioned average value and the above-mentioned standard deviation, the square root of the sum of squares of the pixel values, the ratio of the square root of the sum of squares of the pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of squares of the pixel values, and the product of the above-mentioned image maximum value and the sum of pixel values. Divide by the value of the total number of pixels, the above-mentioned image maximum value, the image contrast, and the difference between the image maximum value and the average value; determine the sum of the above-mentioned pixel values, the standard deviation of the above-mentioned pixel values, the square root of the sum of the squares of the above-mentioned pixel values, the entropy of the above-mentioned pixel values, the sum of the products of the above-mentioned pixel values ​​and the average value of the above-mentioned pixel values ​​minus the square of the above-mentioned average value, the sum of the absolute values ​​of the differences between the above-mentioned adjacent pixel values, the above-mentioned third-order central moment based on the pixel value, the above-mentioned fourth-order central moment based on the pixel value, the median value of the above-mentioned pixel value, the pixel value of the above-mentioned pixel point with the lowest brightness, the pixel value of the above-mentioned pixel point with the highest brightness, the product of the above-mentioned average value and the above-mentioned standard deviation, the ratio of the square root of the sum of the squares of the above-mentioned pixel values ​​to the total number of pixels, the product of the above-mentioned image maximum value and the square root of the sum of the squares of the pixel values, the product of the above-mentioned image maximum value and the sum of the pixel values ​​divided by the value of the total number of pixels, the above-mentioned image maximum value, the above-mentioned image contrast, and the difference between the image maximum value and the average value as the statistical characteristics of the above-mentioned pipeline defects.

[0061] Furthermore, the following formula is used to calculate the sum of the pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the above-mentioned pixel values ​​minus the square of the above-mentioned average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel value, the fourth-order central moment based on the pixel value, the median value of the pixel value, the pixel value of the pixel with the lowest brightness, the pixel value of the pixel with the highest brightness, the product of the above-mentioned average value and the above-mentioned standard deviation, the ratio of the square root of the sum of the squares of the pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of the squares of the pixel values, the product of the above-mentioned image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the above-mentioned image maximum value, the image contrast, and the difference between the image maximum value and the average value:

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] Where W and H represent the width and height of the above sub-image, respectively. w∗h is the pixel intensity at the preset position (w, h) of the above sub-image, and Ij is the pixel intensity at the position j of the above sub-image; I ij is the pixel intensity at position (i, j) of the above sub-image; I +1 j is the pixel intensity at position (i+1, j) of the above sub-image; I, is the average value of all pixel intensities of the above sub-image, f(1) is the sum of the above pixel values, f(2) is the standard deviation of the above pixel values, f(3) is the square root of the sum of the squares of the above pixel values, f(4) is the entropy of the above pixel values, f(5) is the sum of the products of the above pixel values ​​and the average value of the above pixel values ​​minus the square of the above average value, f(6) is the sum of the absolute values ​​of the differences between the above adjacent pixel values, f(7) is the third-order central moment based on the pixel values, f(8) is the fourth-order central moment based on the pixel values, and f(9) is the median of the above pixel values. value, f(10) is the pixel value of the pixel with the lowest brightness, f(11) is the pixel value of the pixel with the highest brightness, f(12) is the product of the average value and the standard deviation, f(13) is the square root of the sum of squares of the pixel values, f(14) is the ratio of the square root of the sum of squares of the pixel values ​​to the total number of pixels, f(15) is the product of the maximum value of the image and the square root of the sum of squares of the pixel values, f(16) is the product of the maximum value of the image and the sum of pixel values ​​divided by the total number of pixels, f(17) is the maximum value of the image, and f(18) is the image contrast and the difference between the image maximum and the average value.

[0080] Furthermore, the step of constructing a feature vector corresponding to the above-mentioned statistical feature includes: performing redundancy analysis based on the sum of the above-mentioned pixel values ​​of each sub-image in the above-mentioned multiple sub-images, the standard deviation of the above-mentioned pixel values, the square root of the sum of the squares of the above-mentioned pixel values, the entropy of the above-mentioned pixel values, the sum of the products of the above-mentioned pixel values ​​and the average value of the above-mentioned pixel values ​​minus the square of the above-mentioned average value, the sum of the absolute values ​​of the differences between the above-mentioned adjacent pixel values, the above-mentioned third-order central moment based on the pixel values, the above-mentioned fourth-order central moment based on the pixel values, the median value of the above-mentioned pixel values, the pixel value of the above-mentioned pixel point with the lowest brightness, the pixel value of the above-mentioned pixel point with the highest brightness, the product of the above-mentioned average value and the above-mentioned standard deviation, the ratio of the square root of the sum of the squares of the above-mentioned pixel values ​​to the total number of pixels, the product of the above-mentioned image maximum value and the square root of the sum of the squares of the pixel values, the value of the product of the above-mentioned image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the above-mentioned image maximum value, the above-mentioned image contrast and the difference between the image maximum value and the average value, pixel intensity, contrast, brightness and saturation to obtain a redundant analysis result; and constructing the feature vector corresponding to the above-mentioned statistical feature according to the above-mentioned redundant analysis result.

[0081] In this embodiment, the step of performing redundancy analysis on the above f(1) to f(18) and the above pixel intensity, contrast, brightness and saturation includes: First, the canonical correlation coefficients are calculated: these coefficients measure the strength of the linear relationship between two groups of variables. In redundancy analysis, the linear combination that maximizes the canonical correlation coefficient is found. Then, regression analysis is performed on each pair of canonical variables: a canonical variable is a linear combination of the original variables that is used to represent the relationship between two groups of variables. Then, redundancy is calculated, where redundancy measures the proportion of variance in one group of variables that can be explained by another group of variables. Finally, based on the canonical correlation coefficient and redundancy, the contribution of each feature to the overall relationship is evaluated, and the statistical features with the highest positive correlation and correlation with the pixel intensity, contrast, brightness and saturation of the image are screened.

[0082] Furthermore, the step of performing feature fusion according to the above-mentioned feature vector and the preset weight corresponding to the above-mentioned feature vector to obtain a fused vector includes: taking the above-mentioned feature vector as the input of an initial support vector machine, taking the defect type corresponding to the above-mentioned feature vector as the output of the above-mentioned initial support vector machine, training the above-mentioned initial support vector machine until a preset training standard is reached to obtain an intermediate support vector machine; determining the accuracy corresponding to the above-mentioned feature vector based on the above-mentioned intermediate support vector machine; determining the preset weight of the above-mentioned feature vector according to the above-mentioned accuracy; and normalizing the above-mentioned feature vector according to the above-mentioned feature vector and the above-mentioned preset weight to obtain a fused feature vector.

[0083] The steps of determining the preset weight of the feature vector according to the accuracy are as follows: w1=A1 / (A1+A2+A3) w2=A2 / (A1+A2+A3) w3=A3 / (A1+A2+A3) w1+w2+w3=1 Among them, A1 is the feature vector corresponding to the above-mentioned pipeline defect morphological features, A2 is the feature vector corresponding to the above-mentioned pipeline defect local texture features, A3 is the feature vector corresponding to the above-mentioned pipeline defect statistical features, w1 is the first weight of the feature vector corresponding to the above-mentioned pipeline defect morphological features, w2 is the second weight of the feature vector corresponding to the above-mentioned pipeline defect local texture features, and w3 is the third weight of the feature vector corresponding to the above-mentioned pipeline defect statistical features.

[0084] Furthermore, the above fused feature vector is expressed by the following formula: F = [w1f1, w2f2, w3f3] Among them, F is the fusion feature vector, f1 is the feature vector corresponding to the local texture feature of the above pipeline defect, f2 is the feature vector corresponding to the local texture feature of the above pipeline defect, and f3 is the feature vector corresponding to the statistical feature of the above pipeline defect.

[0085] The embodiment of the present invention provides a method for multimodal feature fusion of pipeline defect images, including: obtaining a defective pipeline image; extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and extracting local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and extracting statistical features of the defective pipeline image to obtain statistical features of the pipeline defects; respectively constructing feature vectors corresponding to the morphological features of the pipeline defects, the local texture features of the pipeline defects, and the statistical features of the pipeline defects; performing feature fusion according to the feature vectors and the preset weights corresponding to the feature vectors to obtain a fusion vector; training the initial machine learning model based on the fusion vector until the preset training conditions are met to obtain a trained machine learning model. The method respectively constructs feature vectors corresponding to the morphological features, local texture features, and statistical features, fuses the feature vectors to obtain a fusion vector, and trains the machine learning model based on the fusion vector to improve the recognition accuracy of the model.

[0086] Example 3 For ease of understanding, this embodiment further details the steps of the method for implementing printing watermarks in specific operations based on Embodiment 2.

[0087] Specifically, firstly, a defective pipeline image is obtained. Then, morphological features of the pipeline defects in the defective pipeline image are extracted to obtain morphological features of the pipeline defects. The morphological feature extraction includes: a1 to a6 in Embodiment 2.

[0088] Among them, the basic parameters of any two defective pipeline images are shown in Table 1 below: Table 1

[0089] Image 1: a1=0.30, a2=0.40, a3=0.60, a4=0.57, a5≈0.94, a6=1.17; Image 2: a1=0.17, a2=0.25, a3=0.42, a4=0.60, a5≈1.13, a6=1.25; Feature vector of image 1: F1=[0.3 0.4 0.6 0.57 0.94 1.17 ] Among them, F1 is the feature vector of image 1.

[0090] Then, local texture features are extracted from the above images.

[0091] Specifically, the image is divided into 8×8 sub-images, and the value at the center of each image represents the pixel. For ease of understanding, Figure 3 A schematic diagram of a divided image provided by an embodiment of the present invention.

[0092] Furthermore, the steps of generating local texture features are as follows: (1) Select the center pixel: In this 3x3 window, select the center pixel, assuming it is 47.

[0093] (2) Compare neighborhood pixels: Compare the center pixel with the surrounding 8 pixels to generate a binary pattern.

[0094] (3) Generate binary numbers: For each neighborhood pixel, if it is greater than or equal to the central pixel 47, assign it a binary value of 1, otherwise assign it a binary value of 0.

[0095] The comparison results of the center pixel 47 and the surrounding pixels are as follows: 44<47, output 0; 45<47, output 0; 46<47, output 0; 55>47, output 1; 56>47, output 1; 48<47, output 0; 49<47, output 0; 50<47, output 0.

[0096] Therefore, the binary pattern of the center pixel 47 is: 00011100.

[0097] (4) Calculate decimal value: Convert the binary pattern to a decimal value.

[0098] The binary number 00011100 is converted to decimal 28.

[0099] (5) Generate histogram: Repeat the above steps for the entire image or a certain area of ​​the image, generate a local texture feature value for each pixel, and then count the frequency of occurrence of each local texture feature value to construct a histogram.

[0100] Assume that the local texture eigenvalues ​​and their frequencies of the entire image area are as follows: local texture eigenvalue 12 appears 5 times; local texture eigenvalue 22 appears 12 times; local texture eigenvalue 24 appears 7 times; local texture eigenvalue 36 appears 4 times; local texture eigenvalue 45 appears 8 times; local texture eigenvalue 63 appears 12 times.

[0101] Concatenate the frequency values ​​in the histogram to form a feature vector F2=[5, 8, 7, 4, 8, 12].

[0102] The third feature extraction is to first extract statistical features, which are specifically represented by the following Table 2: Table 2

[0103] Then, the typical correlation coefficients were calculated: the following typical correlation coefficients were obtained through RDA analysis, specifically: the typical correlation coefficient between pixel intensity and f(1): 0.95; the typical correlation coefficient between contrast and f(2): 0.90; the typical correlation coefficient between brightness and f(3): 0.85; the typical correlation coefficient between saturation and f(4): 0.80.

[0104] Then, regression analysis was performed. Specifically: pixel intensity has a significant positive effect on f(1); contrast has a significant positive effect on f(2); brightness has a significant positive effect on f(3); and saturation has a significant positive effect on f(4).

[0105] Then, repeating the above calculation for f(5)~f(18) can obtain the first six statistical features with the largest correlation coefficients with pixel intensity, contrast, and brightness, and we can get F3=[8 9 6 8 5 10] Finally, image feature vector fusion is performed. Specifically, first, recognition is performed using existing intelligent algorithms, such as commonly used support vector machine and other classification methods based on feature vectors, and the accuracy is calculated through a large number of pictures, which are expressed as 89%, 92% and 94% respectively. Then the present invention uses calculations w1, w2 and w3 to represent the weight ratio of feature vector fusion.

[0106] w1 = A1 / (A1+A2+A3) = 0.32 w2 = A2 / (A1+A2+A3) = 0.33 w3 = A3 / (A1+A2+A3) = 0.35 w1+w2+w3=1 Then, the Laplace matrix is ​​calculated based on the eigenvectors F1, F2 and F3 obtained by the above three methods to obtain f1, f2, f3. The calculation results are as follows: f 1=

[0107] f2=

[0108] f 3=

[0109] The feature vector can be fused F = w1f1+ w2f2+ w3f3 Example 4 Based on the above embodiments, Figure 4A schematic diagram of the structure of a multimodal feature fusion device for pipeline defect images provided in an embodiment of the present invention.

[0110] Depend on Figure 4 As can be seen, the device includes: The data acquisition module 31 is used to acquire defective pipeline images.

[0111] The feature extraction module 32 is used to extract morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and to extract local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and to extract statistical features of the defective pipeline image to obtain statistical features of the pipeline defects.

[0112] The feature vector construction module 33 is used to respectively construct feature vectors corresponding to the above pipeline defect morphological features, the above pipeline defect local texture features and the above pipeline defect statistical features.

[0113] The feature fusion module 34 is used to perform feature fusion according to the feature vectors and the preset weights corresponding to the feature vectors to obtain a fusion vector.

[0114] Among them, the data acquisition module 31, the feature extraction module 32, the feature vector construction module 33 and the feature fusion module 34 are connected in sequence.

[0115] The pipeline defect image multimodal feature fusion device provided in the embodiment of the present invention has the same technical features as the pipeline defect image multimodal feature fusion method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the above method embodiment, and will not be repeated here.

[0116] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0117] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not require that the components are absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0118] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pipeline defect image multimodal feature fusion method, characterized in that: include: Acquire defective pipeline images; Performing morphological feature extraction on the pipeline defect of the defective pipeline image to obtain morphological features of the pipeline defect; and, performing local texture feature extraction on the pipeline defect of the defective pipeline image to obtain local texture features of the pipeline defect; and, performing statistical feature extraction on the defective pipeline image to obtain statistical features of the pipeline defect; Constructing feature vectors corresponding to the pipeline defect morphological feature, the pipeline defect local texture feature, and the pipeline defect statistical feature respectively; Feature fusion is performed according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector.

2. The pipeline defect image multimodal feature fusion method according to claim 1 is characterized in that: After the step of performing feature fusion according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector, the method includes: The initial machine learning model is trained based on the fusion vector until a preset training condition is reached to obtain a trained machine learning model.

3. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: The step of extracting morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects includes: The first ratio of the defect area of ​​the pipeline defect to the pipeline area in the defective pipeline image is calculated by the following formula: Among them, a1 is the first proportion, A defect is the area of ​​the defective part, and Apipe is the area of ​​the pipeline; The second ratio of the circumference of the defective part in the pipeline defect to the circumference of the pipeline in the defective pipeline image is calculated by the following formula: Among them, a2 is the second proportion, P defect is the circumference of the defect, P pipe is the circumference of the pipeline; The third ratio of the equivalent diameter of the defective part in the pipeline defect to the pipeline diameter in the defective pipeline image is calculated by the following formula: ； ； Wherein, a3 is the third proportion, D eq is the equivalent diameter of the defect, D pip is the pipe diameter; The rectangular aspect ratio of the minimum area rectangle surrounding the pipeline defect is calculated by the following formula: Wherein, a4 is the aspect ratio of the rectangle, Width is the length of the minimum area rectangle, and Height is the width of the minimum area rectangle; The geometric properties of the minimum area shape surrounding the pipeline defect are calculated by the following formula: Wherein, a5 is the geometric characteristic, and when the minimum area shape is a circle, the value of a5 is 1; The fourth ratio of the convex hull area of ​​the defective part in the pipeline defect to the area of ​​the defective part is calculated by the following formula: Among them, a6 is the fourth proportion, A convexhull is the convex hull area; The first proportion, the second proportion, the third proportion, the fourth proportion, the rectangular aspect ratio and the geometric characteristics are extracted to obtain the morphological characteristics of the pipeline defect.

4. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: The step of extracting local texture features of the pipeline defect in the defective pipeline image to obtain local texture features of the pipeline defect comprises: Based on the following steps, texture analysis is performed on the target window where each pixel point is located through a neighborhood image window of preset size parameters around each pixel point in the defective pipeline image to obtain the local texture features of the pipeline defect: Step 1: dividing the defective pipeline image into multiple image windows according to preset size parameters using the following formula; the multiple image windows overlap each other; window=I(i:i+2, j:j+2), i={1,2,…,w−2}, j={1,2,…,h−2}; Wherein, window is the target window centered at the pixel coordinate point (i, j), and I(i:i+2, j:j+2) is the neighborhood image window of the target window; Step 2, comparing the central pixel of the target window with the neighboring pixels of the neighborhood image window around the central pixel through a sign function using the following formula to generate a binary bit result; ； Signum(g,h) is 1 when the central pixel of the target window is greater than or equal to the neighboring pixel of the neighborhood image window, otherwise it is 0, g is the central pixel, h is the neighborhood pixel, Step 3, converting the binary result into a decimal result by the following formula; ； in, is the decimal result of the neighborhood image window of the target window centered at the pixel coordinate point (i, j); Step 4, counting the decimal results corresponding to each image window to generate a feature histogram; Step 5: Determine the local texture features of the pipeline defect using the feature histogram.

5. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: Before the step of extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects, the method includes: Acquire defect-free pipeline images; The step of extracting statistical features from the defective pipeline image to obtain statistical features of pipeline defects includes: Statistical features are extracted from the non-defective pipeline image and the defective pipeline image to obtain statistical features of the pipeline defects.

6. The pipeline defect image multimodal feature fusion method according to claim 5, characterized in that: The step of extracting statistical features from the defect-free pipeline image and the defective pipeline image to obtain statistical features of the pipeline defect comprises: Dividing the non-defective pipeline image and the defective pipeline image into a plurality of sub-images according to a preset size; Counting the sum of pixel values ​​of each sub-image in the multiple sub-images, the standard deviation of the pixel values, the square root of the sum of squared pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the pixel values ​​minus the square of the average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel values, the fourth-order central moment based on the pixel values, the median value of the pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the product of the average value and the standard deviation, the ratio of the square root of the sum of squared pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of squared pixel values, the value of the product of the image maximum value and the sum of pixel values ​​divided by the total number of pixels, the image maximum value, the image contrast, and the difference between the image maximum value and the average value; The sum of the pixel values, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the pixel values ​​minus the square of the average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel values, the fourth-order central moment based on the pixel values, the median value of the pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the product of the average value and the standard deviation, the ratio of the square root of the sum of the squares of the pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of the squares of the pixel values, the value of the product of the image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the image maximum value, the image contrast, and the difference between the image maximum value and the average value are determined as the statistical characteristics of the pipeline defect.

7. The pipeline defect image multimodal feature fusion method according to claim 6, characterized in that: The following formula is used to calculate the sum of the pixel values ​​of each sub-image in the multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the pixel values ​​minus the square of the average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel values, the fourth-order central moment based on the pixel values, the median value of the pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the product of the average value and the standard deviation, the ratio of the square root of the sum of the squares of the pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of the squares of the pixel values, the value of the product of the image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the image maximum value, the image contrast, and the difference between the image maximum value and the average value: Wherein, W and H represent the width and height of the sub-image respectively, I w∗h is the pixel intensity at the preset position (w, h) of the sub-image, and Ij is the pixel intensity at the position j of the sub-image; I ij is the pixel intensity at position (i, j) of the sub-image; I +1 j is the pixel intensity at position (i+1, j) of the sub-image; I, is the average value of all pixel intensities of the sub-image, f(1) is the sum of the pixel values, f(2) is the standard deviation of the pixel values, f(3) is the square root of the sum of the squares of the pixel values, f(4) is the entropy of the pixel values, f(5) is the sum of the products of the pixel values ​​and the average value of the pixel values ​​minus the square of the average value, f(6) is the sum of the absolute values ​​of the differences between adjacent pixel values, f(7) is the third-order central moment based on the pixel value, f(8) is the fourth-order central moment based on the pixel value, and f(9) is the intermediate moment of the pixel value. value, f(10) is the pixel value of the pixel with the lowest brightness, f(11) is the pixel value of the pixel with the highest brightness, f(12) is the product of the average value and the standard deviation, f(13) is the square root of the sum of squares of the pixel values, f(14) is the ratio of the square root of the sum of squares of the pixel values ​​to the total number of pixels, f(15) is the product of the maximum value of the image and the square root of the sum of squares of the pixel values, f(16) is the product of the maximum value of the image and the sum of pixel values ​​divided by the total number of pixels, f(17) is the maximum value of the image, and f(18) is the image contrast and the difference between the image maximum value and the average value.

8. The pipeline defect image multimodal feature fusion method according to claim 7, characterized in that: The step of constructing a feature vector corresponding to the statistical feature comprises: Performing redundancy analysis based on the sum of the pixel values ​​of each sub-image in the multiple sub-images, the standard deviation of the pixel values, the square root of the sum of the squares of the pixel values, the entropy of the pixel values, the sum of the products of the pixel values ​​and the average value of the pixel values ​​minus the square of the average value, the sum of the absolute values ​​of the differences between adjacent pixel values, the third-order central moment based on the pixel values, the fourth-order central moment based on the pixel values, the median value of the pixel values, the pixel value of the pixel point with the lowest brightness, the pixel value of the pixel point with the highest brightness, the product of the average value and the standard deviation, the square root of the sum of the squares of the pixel values, the ratio of the square root of the sum of the squares of the pixel values ​​to the total number of pixels, the product of the image maximum value and the square root of the sum of the squares of the pixel values, the value of the product of the image maximum value and the sum of the pixel values ​​divided by the total number of pixels, the image maximum value, the image contrast and the difference between the image maximum value and the average value, pixel intensity, contrast, brightness and saturation to obtain a redundancy analysis result; According to the redundancy analysis result, a feature vector corresponding to the statistical feature is constructed.

9. The pipeline defect image multimodal feature fusion method according to claim 1, characterized in that: The step of performing feature fusion according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector includes: Using the feature vector as the input of an initial support vector machine, using the defect type corresponding to the feature vector as the output of the initial support vector machine, training the initial support vector machine until a preset training standard is reached, and obtaining an intermediate support vector machine; Based on the intermediate state support vector machine, determining the accuracy corresponding to the feature vector; Determining a preset weight of the feature vector according to the accuracy; According to the feature vector and the preset weight, the feature vector is normalized to obtain a fused feature vector.

10. A pipeline defect image multimodal feature fusion device, characterized in that: include: A data acquisition module, used to acquire defective pipeline images; A feature extraction module is used to extract morphological features of the pipeline defects in the defective pipeline image to obtain morphological features of the pipeline defects; and to extract local texture features of the pipeline defects in the defective pipeline image to obtain local texture features of the pipeline defects; and to extract statistical features of the defective pipeline image to obtain statistical features of the pipeline defects; A feature vector construction module, used to respectively construct feature vectors corresponding to the pipeline defect morphological features, the pipeline defect local texture features and the pipeline defect statistical features; The feature fusion module is used to perform feature fusion according to the feature vector and the preset weight corresponding to the feature vector to obtain a fusion vector.

Images