Industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion

Abstract

The invention relates to the technical field of computer vision, in particular to an industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal vision fusion, which comprises the following steps of: acquiring a multi-frame seam image to extract a texture dot matrix, analyzing displacement frequency to evaluate stability, calculating angle difference to identify a deformation structure surface, and performing multi-modal vision fusion. And detecting image and point cloud overlapping, screening abnormal blocks to generate a fusion set, identifying a spectrum hopping mapping image, positioning a boundary region, extracting a frequency analysis path repetition rate, identifying an abnormal focusing position, and generating a diagnosis list. According to the invention, through multi-frame image texture dot matrix displacement tracking, the dynamic abnormal region identification precision is improved, micro deformation is captured through three-dimensional point cloud angle difference, the structural anomaly detection is enhanced, the multi-modal anomaly judgment accuracy is improved, the anomaly characteristics, the inspection path frequency and anomaly aggregation analysis are refined, the high-frequency abnormal region is positioned, and the inspection resource configuration is optimized; risk prevention and control are enhanced, and limitation of equipment surface sensing and abnormal positioning is broken through.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an autonomous inspection and intelligent diagnosis method for industrial facilities based on multimodal vision fusion. Background Art

[0002] The field of computer vision encompasses technologies that use computers to simulate the human visual system to automatically acquire, process, analyze, and understand images and videos, and to generate decisions and provide feedback. Core areas within this field include image acquisition and preprocessing, feature extraction, image segmentation, object recognition, behavioral understanding, and 3D reconstruction. Computer vision is widely used in scenarios such as intelligent manufacturing, security monitoring, and autonomous driving. Its technological development relies on image sensors, deep learning algorithms, and large-scale data processing. By building perception systems, it enables accurate identification and analysis of objects and their states in complex scenarios, thereby assisting or replacing manual visual tasks.

[0003] Among them, the autonomous inspection and intelligent diagnosis method of industrial facilities refers to the means of using computer vision technology to acquire images, identify and analyze, and judge and diagnose the operating status of equipment in industrial environments. It covers the deployment of image acquisition devices in industrial facility scenarios to perform timed or real-time image acquisition of key equipment, and use image feature matching and change detection methods to identify equipment surface anomalies, such as cracks, rust, liquid leakage, etc., and use image classification and recognition methods to determine the type and location of anomalies. At the same time, it combines historical image sequences for comparative analysis to achieve the extraction of fault signs and the classification of abnormal patterns. This method uses image comparison methods under fixed rules, predefined image variation measurement mechanisms, and equipment surface image texture analysis methods to complete equipment status assessment and fault feature extraction.

[0004] Existing technologies often focus on static feature comparison within single-frame images, relying on fixed-rule image variation measurement and texture change detection. These methods ignore the dynamic temporal evolution of the target surface state, resulting in low sensitivity for identifying early signs of subtle anomalies. Regarding spatial analysis, traditional technologies lack the information support of three-dimensional point cloud structures. Based solely on texture variations in two-dimensional images, they struggle to accurately assess the geometric deformation characteristics of the device structure at different angles, and are prone to misjudgment or omission of detections in situations such as slight surface bumps and distortions. The independent processing of image and structural information results in signal fragmentation, reducing the confidence level of anomaly detection. Regarding spectral analysis, traditional methods rarely incorporate historical reflectance sequences at specific pixel locations for change analysis, failing to effectively detect spectral abrupt changes caused by material property changes. This limits their ability to discriminate heterogeneous areas such as oil stains, cracks, and oxidation. Inspection routes employ a uniform distribution of points, failing to analyze the distribution of high-frequency anomalies based on historical data. A lack of a mechanism for re-inspecting key anomaly locations results in insufficient attention to key areas, impacting the integrity of hazard detection and the efficient use of inspection resources. Summary of the Invention

[0005] In order to solve the technical problems existing in the prior art, the embodiment of the present invention provides an autonomous inspection and intelligent diagnosis method for industrial facilities based on multimodal visual fusion. The technical solution is as follows:

[0006] An autonomous inspection and intelligent diagnosis method for industrial facilities based on multimodal vision fusion, comprising:

[0007] S1: Obtain a multi-frame image sequence of the industrial pump shell joint area, extract the edge texture point array in each frame, identify the point array displacement frequency between adjacent frames, analyze the pixel motion amplitude to evaluate its texture stability, and generate a time series texture change map;

[0008] S2: Based on the key area corresponding to the time series texture change map, extract the three-dimensional point cloud data of the area, perform angle difference calculation on the direction change trend of the object surface at consecutive moments, identify the change segments of the spatial form, and obtain the time series point cloud deformation facet set;

[0009] S3: calling the image segments at the corresponding positions of the temporal point cloud deformation surface set, detecting the overlapping distribution of the image texture intensity and the point cloud mutation density in the spatial coordinate system, screening the pixel blocks with simultaneous texture disturbance and structural offset, and generating a multimodal abnormal fusion block set;

[0010] S4: Locate the multimodal anomaly fusion block set, identify the spectral reflectance sequence of the corresponding area in the specified band, perform gradient scanning on the spectral value of each pixel point, identify the boundary jump area, and obtain a local spectral jump map.

[0011] As a further solution of the present invention, the time series texture change map includes a texture lattice stability index, a pixel displacement frequency distribution map, and an inter-frame motion amplitude sequence; the time series point cloud deformation facet set includes a surface morphology change fragment set, an angle difference distribution map, and a three-dimensional facet evolution trend map; the multimodal anomaly fusion block set includes a texture intensity disturbance block, a structural mutation overlapping block, and a spatial anomaly fusion fragment; the local spectral jump mapping map includes a spectral gradient distribution map, a boundary jump feature map, and a pixel reflectivity change map.

[0012] As a further solution of the present invention, the steps of the time series texture change map are specifically as follows:

[0013] S101: Acquire a multi-frame image sequence captured at the seam area of the industrial pump housing, extract a single-frame pixel matrix, perform a directional difference operation on the grayscale gradient, extract pixel points in the local gradient area, construct an edge texture dot matrix based on coordinates, record the coordinate set frame by frame, and generate a seam texture dot matrix sequence;

[0014] S102: Based on the seam texture dot matrix sequence, performing differential displacement calculation on the dot matrix coordinates corresponding to adjacent frames, counting the number of times the displacement value exceeds the stable threshold pixel, summarizing the abnormal frequency by pixel point, and obtaining a texture offset frequency distribution value in the seam area;

[0015] S103: Call the texture offset frequency distribution value of the seam area, match the normal state frequency distribution of similar seams in the multimodal inspection library, identify the difference interval ratio and the proportion of exceeded points, mark the abnormal points, draw the corresponding time frame and abnormal point ratio curve, and generate a time series texture change map.

[0016] As a further solution of the present invention, the step of deforming the temporal point cloud into a facet set is specifically as follows:

[0017] S201: Based on the key area corresponding to the time series texture change map, extract the three-dimensional point cloud data of the area corresponding to the inspection time, identify the point coordinates and perform spatial matching between the time periods, analyze the three-dimensional offset vectors and direction changes of the matching points, and generate an inspection time series offset direction dataset;

[0018] S202: Call the inspection time series offset direction dataset, calculate the angle difference of spatial segments at consecutive moments, filter segments whose angle changes exceed a threshold, merge them based on connectivity and change continuity, extract regional feature contours, and obtain a time series point cloud deformation facet set.

[0019] As a further solution of the present invention, the angle difference of the spatial segments at consecutive moments is calculated using the formula:

[0020]

[0021] Among them, Δθ represents the angular difference of the space segments at consecutive moments, θ i represents the angle at time i, θ j represents the angle at time j, w k represents the weight of the moment segment k, d i,j Represents the distance between the spatial segments from time i to time j, and n represents the total number of time segments.

[0022] As a further solution of the present invention, the steps of the multimodal abnormal fusion block set are specifically as follows:

[0023] S301: Calling the image segments at the corresponding positions of the temporal point cloud deformation facet set, analyzing the relationship between the image grayscale gradient and the spatial coordinates, extracting the regional grayscale change value, and comparing it with the texture disturbance threshold, identifying the distribution of the exceeding limit area, and obtaining the texture abnormality response map;

[0024] S302: Based on the corresponding coordinate positions in the texture anomaly response map, extract the point cloud quantity change value in the same area of the point cloud facet set, identify the proportion of the number of mutation points and compare it with the point cloud mutation density benchmark value, screen the structural disturbance area and record the spatial label to obtain the structural variation coordinate set;

[0025] S303: Extract the coordinate index of the image and point cloud bimodal anomaly response based on the spatial overlap area of the structural variation coordinate set, aggregate the regional image texture value and the point cloud elevation difference, calculate the fusion feature contribution value, and generate a multimodal anomaly fusion block set.

[0026] As a further solution of the present invention, the fusion feature contribution value adopts the formula:

[0027]

[0028] Among them, F represents the fusion feature contribution value, T z Represents the value of the image texture value in the zth region, P z Represents the value of the point cloud elevation difference in the zth area, H z Represents the point cloud elevation of the zth region, D z represents the structural variation value of the zth region, and m represents the total number of spatially overlapping regions.

[0029] As a further solution of the present invention, the steps of mapping the local spectrum jump are specifically as follows:

[0030] S401: Locate the multimodal anomaly fusion block set, extract the spectral reflectance sequence of the industrial facility area in the specified band, identify the pixel spectral vector, perform passband difference scanning and amplitude judgment based on the structural boundary information, identify the reflectance jump point, and generate the facility structure boundary mutation position set;

[0031] S402: Cluster the jump points in the spatial dimension according to the facility structure boundary mutation position set, screen the jump groups with continuity and boundary contour characteristics, and map them to the original facility image area in combination with the spectral image spatial index to obtain a local spectral jump map.

[0032] As a further embodiment of the present invention, the method further comprises step S5:

[0033] S5: Based on the local spectral jump map, extract the distribution frequency and repetition rate in the inspection path sequence, analyze the frequency distribution of the pump facilities at the differentiated inspection points, identify the areas with abnormal frequency, count the key positions in the inspection path, analyze the repetition rate and abnormal characteristics of each position, and obtain a multimodal diagnosis focus distribution list for the pump facilities;

[0034] The multimodal diagnosis focus distribution list of the pump facility includes the distribution of high-frequency abnormal areas, inspection repetition rate analysis results, and abnormal feature focus points.

[0035] As a further solution of the present invention, the steps of the multimodal diagnosis focus distribution list of the pump facility are specifically as follows:

[0036] S501: Based on the local spectral jump map, extract the spatial coordinate points in the path image sequence, count the occurrence frequency, identify the jump frame frequency and amplitude, and generate a multimodal visual frequency distribution map;

[0037] S502: calling the multimodal visual frequency distribution map, identifying areas with higher frequencies than the average frequency of the facility section, extracting repeated regions of transition structures in image frames of multiple time periods, and analyzing them based on structural similarity and the degree of repetition of transition forms to establish an index table of abnormal repetition rate areas;

[0038] S503: According to the abnormal repetition rate area index table, the morphological overlap of the jump structure in the index area in RGB, thermal imaging and spectral images is compared, the jump structure position with high overlap features is extracted, and the corresponding coordinate points are screened to obtain a multimodal diagnosis focus distribution list of the pump facility.

[0039] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0040] By extracting edge texture points and tracking displacement frequency in multi-frame image sequences, the team helps accurately quantify local texture motion patterns, distinguishing stable regions from areas of abnormal disturbance and enhancing sensitivity to dynamic texture changes. By mapping key regions to a three-dimensional point cloud and combining angle differential analysis for spatial morphological trend analysis, the team improves the ability to capture subtle surface deformations and establishes a multi-time continuous observation mechanism for extracting abnormal structural changes. Overlapping image texture intensity and point cloud mutation density in a coordinate system enables co-occurrence screening between multimodal signals, significantly improving the accuracy and confidence of anomaly detection. Spectral reflectance sequence analysis, based on spatial synchronization, and gradient scanning to identify boundary jump regions, enhances responsiveness to material characterization mutations and forms a more hierarchical and in-depth anomaly detection system. Furthermore, the team analyzes the distribution statistics of frequency and repetition rates along the inspection path and, combined with the focus characteristics of abnormal pixels, enables precise localization of key monitoring points and automatic identification of high-frequency anomaly clusters, improving the efficiency of inspection resource allocation and the accuracy of risk prevention and control. Through time series analysis, spatial deformation modeling, signal overlap detection, and multimodal information fusion, the entire process effectively addresses the limitations of equipment surface status perception, achieves collaborative diagnosis between cross-dimensional data, and enhances the significance of local anomalies, providing a more directional and analytical map basis for inspection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0042] Figure 2 This is a detailed flow chart of S1 of the present invention;

[0043] Figure 3 This is a detailed flow chart of S2 of the present invention;

[0044] Figure 4 This is a detailed flow chart of S3 of the present invention;

[0045] Figure 5 This is a detailed flow chart of S4 of the present invention;

[0046] Figure 6 This is a detailed flow chart of S5 of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0048] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0049] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same.

[0050] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0051] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0052] See also Figure 1 The embodiment of the present invention provides an autonomous inspection and intelligent diagnosis method for industrial facilities based on multimodal visual fusion. The processing flow of the method may include the following steps:

[0053] S1: Obtain a multi-frame image sequence of the industrial pump shell joint area, extract the edge texture point array in each frame, identify the point array displacement frequency between adjacent frames, analyze the pixel motion amplitude to evaluate its texture stability, and generate a time series texture change map;

[0054] S2: Based on the key areas corresponding to the time series texture change map, the 3D point cloud data of the area is extracted, the directional change trend of the object surface at consecutive moments is calculated by angle difference, the change segments of the spatial form are identified, and the time series point cloud deformation surface set is obtained;

[0055] S3: Call the image segments at the corresponding positions of the temporal point cloud deformation surface set, detect the overlapping distribution of image texture intensity and point cloud mutation density in the spatial coordinate system, screen the pixel blocks with simultaneous texture disturbance and structural offset, and generate a set of multimodal abnormal fusion blocks;

[0056] S4: Locate the multimodal anomaly fusion block set, identify the spectral reflectance sequence of the corresponding area in the specified band, perform gradient scanning on the spectral value of each pixel, identify the boundary jump area, and obtain the local spectral jump map;

[0057] S5: Based on the local spectral jump map, the distribution frequency and repetition rate in the inspection path sequence are extracted, the frequency distribution of pump facilities at differentiated inspection points is analyzed, the areas with abnormal frequency are identified, the key positions in the inspection path are counted, the repetition rate and abnormal characteristics of each position are analyzed, and a multimodal diagnosis focus distribution list of pump facilities is obtained.

[0058] The time series texture change map includes texture lattice stability index, pixel displacement frequency distribution map, and inter-frame motion amplitude sequence. The time series point cloud deformation surface set includes surface morphology change fragment set, angle difference distribution map, and three-dimensional surface evolution trend map. The multimodal anomaly fusion block set includes texture intensity disturbance blocks, structural mutation overlapping blocks, and spatial anomaly fusion fragments. The local spectral jump mapping map includes spectral gradient distribution map, boundary jump feature map, and pixel reflectivity change map. The multimodal diagnosis focus distribution list of pump facilities includes high-frequency anomaly area distribution, inspection repetition rate analysis results, and anomaly feature focus points.

[0059] Specifically, if Figure 2 As shown in Figure 2, the steps of the time series texture change map are as follows:

[0060] S101: Acquire a multi-frame image sequence captured at the seam area of the industrial pump housing, extract a single-frame pixel matrix, perform a directional difference operation on the grayscale gradient, extract pixel points in the local gradient area, construct an edge texture dot matrix based on coordinates, record the coordinate set frame by frame, and generate a seam texture dot matrix sequence;

[0061] Image acquisition equipment should capture the joints of industrial pump casings in a stable and high-precision manner. Each frame of the image must be clear to avoid noise or blur that affects subsequent analysis. The pixel matrix in each frame of the image must be extracted. Each pixel in the image contains a certain grayscale value. The higher the image resolution, the larger the pixel matrix of each frame, reaching millions or even tens of millions of pixels. Directional differential operation is performed on the grayscale gradient of pixel points to identify the texture changes in the seam area. The operation process specifically includes calculating the difference in the grayscale values of adjacent pixels to obtain the gradient values in each direction. By analyzing the gradient direction and amplitude, the texture characteristics can be obtained. The local gradient area refers to the location in the seam area where the gradient change is more significant. The area is strongly associated with the physical characteristics of the seam (such as gaps or connection parts). The pixels of each local gradient area are organized into an edge texture lattice according to the coordinates, forming a lattice structure in a two-dimensional coordinate system. The lattice provides data support for subsequent seam status monitoring and change analysis. The coordinate set of the edge texture points is recorded frame by frame to form a seam texture lattice sequence. Subsequent analysis will rely on the timing characteristics of this lattice sequence to form continuous monitoring of seam texture changes.

[0062] S102: Based on the seam texture dot matrix sequence, differential displacement calculation is performed on the corresponding dot matrix coordinates of adjacent frames, the number of times the displacement value exceeds the stable threshold pixel is counted, the abnormal frequency is summarized by pixel point, and a texture offset frequency distribution value of the seam area is obtained;

[0063] A differential displacement calculation is performed on the coordinates of corresponding lattice points in adjacent frames. By comparing the displacement of the same pixel position in adjacent frames, the calculation can effectively identify changes in the seam texture. The differential displacement calculation first requires a precise set of reference coordinate points. Using the pixel coordinates in each frame, the change in position of the reference points between the two frames is calculated. Specifically, for each pixel, its coordinates in the first frame are first determined. The point's position is then relocated in the second frame, and the displacement between the two frames is calculated. The displacement value is represented by the difference in coordinates. If the displacement exceeds a preset stability threshold (e.g., 5 pixels), the texture offset at that location is considered out of range. The number of times the displacement exceeds the stability threshold is counted for all pixels. This threshold is set to reflect the range of texture variation expected under normal operating conditions, such as 5 pixels. The specific value varies depending on the material properties of the seam and operating conditions. Each pixel whose displacement exceeds the threshold is recorded as an outlier, and the frequency of the outlier is calculated. This value can reflect the stability and texture deviation degree of the joint area, further summarize the abnormal frequency and calculate the frequency distribution value of texture deviation in the joint area, which can be used as a basis for subsequent abnormality identification.

[0064] S103: Calling the frequency distribution value of the texture offset in the seam area, matching it with the normal state frequency distribution of similar seams in the multimodal inspection library, identifying the difference interval ratio and the proportion of exceeded points, marking abnormal points, and drawing a curve of the corresponding time frame and abnormal point ratio to generate a time series texture change map;

[0065] After calling the texture offset frequency distribution value of the seam area, it is matched with the normal state frequency distribution of similar seams in the multimodal inspection library. The step is based on the comparison of the normal seam texture offset pattern stored in the database. Specifically, by comparing the texture offset frequency distribution in the existing data with the distribution characteristics under the normal state, the difference interval is identified. For the abnormal texture offset frequency distribution value, the proportion of the difference interval and the proportion of exceeding the standard point are calculated according to the difference in distribution. The difference interval ratio refers to the range of the gap between the actual measured frequency distribution and the normal state distribution. If the difference is greater than a certain threshold (for example, 30%), it is considered that the joint has an abnormal change. The excess point ratio refers to the proportion of pixels whose displacement exceeds the preset threshold to all monitored pixels. If the excess point ratio exceeds the set threshold (for example, more than 20%), the joint is considered to be abnormal. Through the calculation of these two indicators, the abnormal position and severity of the joint area can be accurately located, and the abnormal points can be further marked and the corresponding time frame and abnormal point ratio curve can be drawn. The curve reflects the changing trend of the joint texture during the monitoring process, and finally a time series texture change map is generated, providing an effective data basis for equipment maintenance and repair.

[0066] Specifically, if Figure 3 As shown in the figure, the steps of temporal point cloud deformation facet set are as follows:

[0067] S201: Based on the key areas corresponding to the time series texture change map, extract the 3D point cloud data of the area corresponding to the inspection time, identify the point coordinates and perform spatial matching between the time periods, analyze the 3D offset vectors and direction changes of the matching points, and generate an inspection time series offset direction dataset;

[0068] Based on the location of key areas in the atlas, 3D point cloud data for that area is acquired. Point cloud data is a set of 3D coordinate points captured by a laser scanner or stereo vision system. Each point has a specific coordinate value (X, Y, Z) in space. The resolution and accuracy of the point cloud directly impact the accuracy of subsequent analysis. By adjusting the scanning frequency and resolution of the equipment, point cloud accuracy can be achieved to the millimeter level. Next, the coordinates of each point in the acquired 3D point cloud data need to be identified. First, the point cloud data is processed using point cloud segmentation technology, dividing the point cloud into different regions. Points in each region are classified based on characteristics such as distance and density. The coordinates of these points are spatially matched with key information from the time-series texture change atlas to determine how the spatial position of the points has changed at different times. The spatial matching process requires calculating the spatial relationship between the point cloud data and the locations of specific regions in the atlas. This is achieved by calculating the distance and relative position of the point cloud data in 3D space. This allows us to determine whether there is positional drift between points at different times. Through spatial matching between moments, the three-dimensional offset vector of each matching point can be obtained, that is, the displacement of the point between different time points can be calculated, and the deformation direction of the joint area can be identified by the change in the direction of the vector. The inspection time series offset direction dataset is generated. This dataset contains the displacement and direction of each point in the joint area at different time points, providing a basis for further deformation analysis.

[0069] S202: Calling the inspection time series offset direction dataset, calculating the angle difference of spatial segments at consecutive moments, screening segments whose angle changes exceed a threshold, merging them based on connectivity and change persistence, extracting regional feature contours, and obtaining a time series point cloud deformation facet set;

[0070] An angular difference operation is performed on each spatial segment at consecutive moments. This process involves calculating the angular change between the same spatial segments at consecutive moments. First, at each moment, the angle of each spatial segment is calculated from the point cloud data. A common angle calculation method uses the angle formula between vectors to calculate the offset direction between any two points in the point cloud. By comparing the change in angle between segments at consecutive moments, the angular difference of each spatial segment can be obtained. If the angle change exceeds a preset threshold (for example, set to 5 degrees), the segment is considered to have undergone significant deformation. Segments with angle changes exceeding this threshold are screened out. The screening process requires comparing the angle change value of each segment with the set threshold. Segments exceeding the threshold are marked as potential abnormal regions. Then, the spatial segments are merged based on their connectivity and the persistence of change. Connectivity assessment refers to determining whether adjacent segments in space have similar change trends to determine whether they belong to the same deformation region. If adjacent segments have consistent angle changes or similar change patterns, they are considered connected and can be grouped together. The evaluation of change persistence is to determine whether the segments need to be merged by detecting the duration of the change. If the angle change of a certain area lasts for more than a certain period of time (for example, more than 10 frames), the change is considered significant and should be merged into a group. The regional feature contours are extracted based on the information. The process analyzes the merged segments and identifies the deformation characteristics of the joints or key areas. The resulting time series point cloud deformation facet set can reflect the dynamic changes of the joints or areas in the time series and is further used for equipment health monitoring and maintenance prediction.

[0071] The angular difference of space segments at consecutive moments is calculated using the formula:

[0072]

[0073] Among them, Δθ represents the angular difference of the space segments at consecutive moments, θ i represents the angle at time i, θ j represents the angle at time j, w k represents the weight of the moment segment k, d i,j represents the distance between the spatial segments from time i to time j, and n represents the total number of time segments;

[0074] The angular difference of a spatial segment at consecutive moments refers to the degree of change in the orientation or direction of the spatial segment between adjacent time points. The angle of the spatial segment represents the deflection angle of the segment relative to a certain reference direction (such as the ground, geographic coordinate system, etc.). When the same spatial segment is observed at two different time points (such as time i and time j), its angle value will change. The angular difference is the difference in the angle of the spatial segment between the two moments. The size of the angular difference reflects the extent to which the direction of the segment has changed within the time span. It is used to analyze the movement or deformation of spatial objects or regions in the time dimension. The larger the angular difference, the more significant the change in the spatial segment, and vice versa.

[0075] Angle difference Δθ: This item represents the angle difference between time i and time j, angle θ i and θ j The spatial direction obtained by the sensor or monitoring device at the corresponding time point. The angle unit is degree (°) or radian (rad). Its value is based on the orientation data obtained by spatial sensors or image analysis;

[0076] Weight parameters This item represents the sum of the weights of consecutive moments. The weight parameter reflects the relative importance of each spatial segment in the calculation. The weight is obtained by considering the similarity and change continuity of each segment between moments. By comparing time series images or the change frequency of adjacent spatial segments, the segments with greater influence are selected and given higher weights. The unit of the weight is independent of the angle, but it will affect the calculation of the final angle difference. The weight range is generally 0 to 1, where the larger the weight, the greater the impact on the final result. This value is obtained through the data monitoring system or calculation model;

[0077] The absolute value of the weight This item is the absolute value sum of all weight values, which is used for normalization adjustment. Its purpose is to standardize the weights of all segments so that the sum of the weights of all segments is 1. The quantization of this value is obtained by taking the absolute value of the weight of the segment at each moment and then accumulating it.

[0078] Spatial distance |d i,j |: This item represents the spatial distance between time i and time j. The spatial distance is calculated by sensors or image monitoring systems. It is the physical distance between adjacent spatial segments. The unit of distance is meters (m). This value is obtained when the monitoring system or sensor is measured, and is obtained through the GPS system or image positioning.

[0079] Assumption: The angle at time i is θ i =45°, the angle at time j is θ j =30°, the weight values are w1=0.3, w2=0.5, w3=0.2, and the spatial distance between fragments |d i,j| = 10 m;

[0080] Calculate according to the formula: |θ i -θ j |=|45-30|=15°;

[0081] Weight sum:

[0082] Sum of absolute values of weights:

[0083] Substituting into the formula:

[0084] The result shows that the angle difference between time i and time j is 15.72°. The angle difference reflects the magnitude of spatial deformation between the two moments. Combined with the weight and spatial distance, its contribution to the final temporal point cloud deformation surface is further determined.

[0085] Specifically, if Figure 4 As shown in Figure 2, the steps of multimodal anomaly fusion block set are as follows:

[0086] S301: Calling the image segments at the corresponding positions of the temporal point cloud deformation facet set, analyzing the relationship between the image grayscale gradient and the spatial coordinates, extracting the regional grayscale change value, and comparing it with the texture disturbance threshold, identifying the distribution of the exceeding limit area, and obtaining the texture abnormality response map;

[0087] Image segments that match the point cloud data are located from the image library. These image segments contain grayscale information corresponding to the 3D point cloud region. The goal of image grayscale gradient analysis is to identify changes in grayscale values within the image region. These changes indicate minor defects or deformations on the object's surface. Specifically, the grayscale value of each pixel in the image is first extracted, and the grayscale gradient is obtained by calculating the grayscale difference between adjacent pixels. For seams or detailed areas, the grayscale gradient varies significantly. Analyzing the relationship between the grayscale gradient of a region and its spatial coordinates can reveal deformation trends or surface defects in the image. When extracting the grayscale change value of a region, it is compared with a pre-set texture perturbation threshold. This perturbation threshold is empirically derived from actual data and is set within a certain range, such as 0.1 to 0.5. If the grayscale change value of a region exceeds the preset threshold, it is considered to have a texture anomaly. By comparing the data of multiple image segments, the distribution of the exceeded areas is identified. Image analysis then generates a texture anomaly response map, which accurately displays the distribution of anomalies within the image region, providing a basis for subsequent analysis.

[0088] S302: Based on the corresponding coordinate positions in the texture anomaly response map, extract the point cloud quantity change value in the same area of the point cloud facet set, identify the proportion of the mutation points and compare them with the point cloud mutation density benchmark value, screen the structural disturbance area and record the spatial label to obtain the structural variation coordinate set;

[0089] The density and distribution of point cloud data are important information for describing the surface deformation of an object. The number of points in each point cloud reflects the level of detail in the area. When deformation or structural problems occur in an area, the density of the point cloud will change significantly, especially in cracks, joints, or structural deformations, where the distribution of the point cloud will become more sparse or concentrated. By comparing the changes in the number of point clouds in the same area over different time periods, potential deformation can be detected. The proportion of mutation points refers to the proportion of point cloud points in a given area whose point cloud number changes exceed a preset threshold. For example, if the change in the number of point clouds in the area exceeds a certain threshold (such as 30%), the area is considered to have undergone a mutation. By comparing the point cloud mutation density benchmark value (derived from historical data, such as an average change of 20%), the structural disturbance area can be screened out. If the point cloud number in a certain area suddenly changes beyond the benchmark value, it is considered that the area has undergone obvious deformation or damage. The process further summarizes the point cloud data, records the spatial labels, and obtains a set of structural variation coordinates. This set reflects the deformation characteristics of each key area during the monitoring period.

[0090] S303: Extracting coordinate indices of image and point cloud bimodal anomaly responses based on the spatial overlap of the structural variation coordinate set, aggregating regional image texture values and point cloud elevation differences, calculating fusion feature contribution values, and generating a multimodal anomaly fusion block set;

[0091] The image data and point cloud data are accurately matched, and the correlation between the two can be effectively determined by analyzing the spatial overlap area of the point cloud data and the image data. For example, when a certain area shows obvious deformation in the point cloud and abnormal grayscale changes in the image, it can be confirmed that there is an actual physical structural problem at that location. Aggregating the image texture value of the area and the point cloud elevation difference as fusion features can provide more detailed anomaly analysis. The image texture value represents the smoothness or defects of the object surface, while the point cloud elevation difference reveals the three-dimensional changes of the surface. By combining the abnormal data of the two, structural problems can be located more accurately. For example, at a certain point, if the texture change value of the image is 0.7 and the point cloud elevation difference is 5mm, it is believed that there is a more serious structural problem in the area. Generating a multimodal anomaly fusion block set can integrate the abnormal responses of the image and point cloud to provide a comprehensive monitoring perspective for further structural health assessment and maintenance prediction.

[0092] The fusion feature contribution value is calculated using the formula:

[0093]

[0094] Among them, F represents the contribution value of fusion features, T z Represents the value of the image texture value in the zth region, P z Represents the value of the point cloud elevation difference in the zth area, H z Represents the point cloud elevation of the zth region, D z represents the structural variation value of the zth region, and m represents the total number of spatially overlapping regions;

[0095] The fusion feature contribution value refers to the degree to which the difference between the image texture features and the elevation information in the point cloud data contributes to the final fusion result during the multimodal data fusion process. Specifically, by calculating the absolute value of the difference between the image texture features and the point cloud elevation, and weighting it according to the relationship between the point cloud elevation and the structural change value, it reflects the similarities and differences between the image and point cloud data in a specific area. This calculation method can assess the impact of different areas on the overall fusion result and help identify areas with abnormal changes in the image and point cloud data. The contribution value is an important indicator in the fusion algorithm, which helps to optimize the data fusion process and improve the accuracy and reliability of the final result.

[0096] Image texture value T z : Calculate the texture feature values of the image, such as contrast, energy, entropy, etc., through the gray level co-occurrence matrix (GLCM) method;

[0097] Point cloud elevation difference P z :Use the digital elevation model (DEM) difference method to calculate the elevation difference of the same location at different time points;

[0098] Point cloud elevation H z : Extract the average elevation value of each area from the point cloud data;

[0099] Structural variation value D z : Calculate the degree of structural changes in the image through the digital image correlation (DIC) method;

[0100] In practical applications, in order to unify the dimensions of each parameter, it is necessary to normalize them. For example, all parameters are normalized to the interval [0, 1] to eliminate the influence of different dimensions on the calculation results.

[0101] Specific numerical example: Assume that in a certain area z = 1, the following normalized parameter values have been obtained: image texture value T1 = 0.6, point cloud elevation difference P1 = 0.4, point cloud elevation H1 = 0.5, and structural variation value D1 = 0.3;

[0102] Substitute the value into the formula to calculate the fusion feature contribution value of the area:

[0103] The results show that in region z = 1, the contribution value of the fusion feature is approximately 0.3428. After similar calculations are performed on all m regions, the total fusion feature F is summed up to obtain the total fusion feature. This fusion feature is used to generate a set of multimodal abnormal fusion blocks to reflect the areas of abnormal changes in the image and point cloud data.

[0104] Specifically, if Figure 5 As shown, the steps of local spectrum jump mapping are as follows:

[0105] S401: Locate the multimodal anomaly fusion block set, extract the spectral reflectance sequence of the industrial facility area in the specified band, identify the pixel spectral vector, and perform passband difference scanning and amplitude judgment based on the structural boundary information to identify the reflectance jump point and generate the facility structure boundary mutation location set;

[0106] Extracting a spectral reflectance sequence from an industrial facility area at a specified wavelength requires the use of high-precision spectroscopic instruments to capture the reflectance characteristics of the facility's surface across multiple wavelengths. For example, laser scanning technology or spectral imaging equipment can be used to acquire reflectance data in different wavelengths, such as visible and infrared. After acquiring the reflectance sequence, the spectral vector of each pixel is analyzed—that is, the reflectance intensity of each pixel at different wavelengths. By comparing this data with the boundary information of the device structure, a passband difference scan is performed. This involves comparing spectral data from different wavelengths and calculating the reflectance differences between them to identify areas where structural boundary changes have occurred. Amplitude determination further assesses the significance of the change by calculating the magnitude of the reflectance difference. When the reflectance difference exceeds a predetermined threshold (e.g., 0.1 units), the area is considered to have undergone a structural change. Based on this determination, reflectance transition points are identified. These points indicate significant changes in the facility structure, such as crack expansion or joint deformation. These transition points are aggregated and formed into a set of facility structural boundary mutation locations, identifying locations where significant structural changes have occurred in the industrial facility during monitoring.

[0107] S402: Clustering the jump points in the spatial dimension based on the facility structure boundary mutation location set, screening the jump groups with continuity and boundary contour characteristics, and mapping them to the original facility image area in combination with the spectral image spatial index to obtain a local spectral jump map;

[0108] The jump points are clustered in the spatial dimension. The purpose of clustering is to classify adjacent jump points into the same group, and the points are located in the same structural variation area. When clustering, the distance between each jump point and the point is first calculated based on the spatial coordinates. If the distance between a jump point and the adjacent point is less than a preset threshold (for example, 5 mm), the two points are considered to belong to the same cluster group. Jump groups with continuity and boundary contour features are screened. Continuity refers to regions with similar structural changes in time and space dimensions, and the regions exhibit relatively stable deformation patterns. The boundary contour features are screened by calculating the geometric characteristics of the change region to identify the boundary. If the jump points in a certain area are arranged to form a closed boundary, the change in the area is considered to be more prominent. By combining the spectral image spatial index, the screened jump group is mapped to the original facility image area to form a spectral jump area corresponding to the original image, and a local spectral jump map is obtained. It can accurately display the area where the facility structure has changed, providing specific spatial information for subsequent structural evaluation.

[0109] Specifically, if Figure 6 As shown in the figure, the steps of the multimodal diagnosis focus distribution list of pump facilities are as follows:

[0110] S501: Based on the local spectral jump map, extract the spatial coordinate points in the path image sequence, count the occurrence frequency, identify the jump frame frequency and amplitude, and generate a multimodal visual frequency distribution map;

[0111] Spatial coordinate points are extracted from the path image sequence. The extraction process first identifies all pixels with significant changes in the image. The locations of these change points are determined by comparing spectral data. Each extracted coordinate point represents a region with the most significant change within the image frame, corresponding to the starting point of a structural defect or change. The extracted coordinate points are then counted. This process involves iterating through the spatial coordinates of all image frames and recording the number of occurrences of each coordinate point. A high frequency of occurrence for a particular coordinate point indicates frequent changes in that region. The relationship between the frame rate and amplitude of these changes is then analyzed. The frame rate of these changes refers to the frequency of changes in a specific region across multiple image frames, while the amplitude refers to the magnitude of the change in reflectivity or spectral parameters associated with each change. Large amplitude jumps indicate significant physical deformation or structural issues. This analysis generates a multimodal visual frequency distribution map, which displays the frequency changes of different regions over time, enabling accurate identification of potential fault areas or structural components requiring attention.

[0112] S502: Invoke the multimodal visual frequency distribution map to identify areas with higher frequencies than the average frequency of the facility section, extract the repeated regions of the transition structure in the multi-period image frames, and analyze them based on the structural similarity and the degree of repetition of the transition morphology to establish an index table of abnormal repetition rate regions;

[0113] Identify areas with frequencies above the average for the facility segment. This identification process compares the frequency of each region in the atlas with the average frequency of all regions within that facility segment. If the frequency of a region is significantly higher than the average, it indicates that it has repeatedly exhibited abnormal changes over multiple time periods, thus warranting attention. Extract repetitive regions of jump-like structures from multi-period image frames. Specifically, by comparing images at different time points, identify areas where structural changes occur repeatedly. These regions represent weak points in the equipment and indicate signs of structural fatigue or wear. Analysis is based on structural similarity and the degree of repetition of jump-like patterns. First, the shape similarity of each region is calculated to determine the similarity of their structural features. If a jump-like region exhibits similar morphology across multiple time periods, its pattern of structural change is considered relatively consistent. Second, the degree of repetition of the jump-like patterns is analyzed, specifically whether the reflectivity changes in that region have similar amplitudes or patterns across multiple time periods. This analysis creates an index table of regions with abnormal repetition rates. This table records areas of the equipment that frequently occur and exhibit consistent patterns of change, providing maintenance personnel with priority attention.

[0114] S503: Based on the abnormal repetition rate area index table, the morphological overlap of the jump structure in the index area in the RGB, thermal imaging image, and spectral image is compared, the jump structure position with high overlap features is extracted, and the corresponding coordinate points are selected to obtain a multimodal diagnosis focus distribution list for the pump body facility;

[0115] The morphological overlap of the jump structure in the index area in RGB, thermal imaging and spectral images is compared, and the structural morphology of each area in the RGB image is extracted. The changes in surface temperature or significant features are identified by analyzing color changes. The heat distribution in the thermal imaging image is analyzed to determine whether there is temperature anomaly in the area. The temperature change is used to determine whether the structure is affected by factors such as thermal expansion and contraction. The spectral image data is used to further verify whether there is anomaly in the spectral characteristics of the area. By comparing the data, the type of change in the area can be more accurately confirmed, the position of the jump structure with high overlap features can be extracted, the area with high overlap features can be screened out, and the corresponding coordinate points can be extracted. The coordinate points indicate the abnormal area that needs the most attention in the equipment, and a multimodal diagnostic focus distribution list of the pump facility is obtained. The list records all structural parts that show significant abnormalities in different modes, helping maintenance personnel to accurately locate equipment failures and perform effective repairs and maintenance.

[0116] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An autonomous inspection and intelligent diagnosis method for industrial facilities based on multimodal visual fusion, characterized by: The following steps are involved: S1: Obtain a multi-frame image sequence of the industrial pump shell joint area, extract the edge texture point array in each frame, identify the point array displacement frequency between adjacent frames, analyze the pixel motion amplitude to evaluate its texture stability, and generate a time series texture change map; S2: Based on the key area corresponding to the time series texture change map, extract the three-dimensional point cloud data of the area, perform angle difference calculation on the direction change trend of the object surface at consecutive moments, identify the change segments of the spatial form, and obtain the time series point cloud deformation facet set; S3: calling the image segments at the corresponding positions of the temporal point cloud deformation surface set, detecting the overlapping distribution of the image texture intensity and the point cloud mutation density in the spatial coordinate system, screening the pixel blocks with simultaneous texture disturbance and structural offset, and generating a multimodal abnormal fusion block set; S4: Locate the multimodal anomaly fusion block set, identify the spectral reflectance sequence of the corresponding area in the specified band, perform gradient scanning on the spectral value of each pixel point, identify the boundary jump area, and obtain a local spectral jump map.

2. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 1 is characterized in that: The time series texture change map includes a texture lattice stability index, a pixel displacement frequency distribution map, and an inter-frame motion amplitude sequence; the time series point cloud deformation facet set includes a surface morphology change fragment set, an angle difference distribution map, and a three-dimensional facet evolution trend map; the multimodal anomaly fusion block set includes a texture intensity disturbance block, a structural mutation overlapping block, and a spatial anomaly fusion fragment; the local spectral jump mapping map includes a spectral gradient distribution map, a boundary jump feature map, and a pixel reflectivity change map.

3. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 1 is characterized in that: The steps of the time series texture change atlas are specifically as follows: S101: Acquire a multi-frame image sequence captured at the seam area of the industrial pump housing, extract a single-frame pixel matrix, perform a directional difference operation on the grayscale gradient, extract pixel points in the local gradient area, construct an edge texture dot matrix based on coordinates, record the coordinate set frame by frame, and generate a seam texture dot matrix sequence; S102: Based on the seam texture dot matrix sequence, performing differential displacement calculation on the dot matrix coordinates corresponding to adjacent frames, counting the number of times the displacement value exceeds the stable threshold pixel, summarizing the abnormal frequency by pixel point, and obtaining a texture offset frequency distribution value in the seam area; S103: Call the texture offset frequency distribution value of the seam area, match the normal state frequency distribution of similar seams in the multimodal inspection library, identify the difference interval ratio and the proportion of exceeded points, mark the abnormal points, draw the corresponding time frame and abnormal point ratio curve, and generate a time series texture change map.

4. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 3 is characterized in that: The steps of deforming the temporal point cloud into a facet set are specifically as follows: S201: Based on the key area corresponding to the time series texture change map, extract the three-dimensional point cloud data of the area corresponding to the inspection time, identify the point coordinates and perform spatial matching between the time periods, analyze the three-dimensional offset vectors and direction changes of the matching points, and generate an inspection time series offset direction dataset; S202: Call the inspection time series offset direction dataset, calculate the angle difference of spatial segments at consecutive moments, filter segments whose angle changes exceed a threshold, merge them based on connectivity and change continuity, extract regional feature contours, and obtain a time series point cloud deformation facet set.

5. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 4 is characterized in that: The angular difference of the spatial segments at consecutive moments is calculated using the formula: Among them, Δθ represents the angular difference of the space segments at consecutive moments, θ i represents the angle at time i, θ j represents the angle at time j, w k represents the weight of the moment segment k, d i,j Represents the distance between the spatial segments from time i to time j, and n represents the total number of time segments.

6. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 4 is characterized in that: The steps of the multimodal abnormal fusion block set are specifically as follows: S301: Calling the image segments at the corresponding positions of the temporal point cloud deformation facet set, analyzing the relationship between the image grayscale gradient and the spatial coordinates, extracting the regional grayscale change value, and comparing it with the texture disturbance threshold, identifying the distribution of the exceeding limit area, and obtaining the texture abnormality response map; S302: Based on the corresponding coordinate positions in the texture anomaly response map, extract the point cloud quantity change value in the same area of the point cloud facet set, identify the proportion of the number of mutation points and compare it with the point cloud mutation density benchmark value, screen the structural disturbance area and record the spatial label to obtain the structural variation coordinate set; S303: Extract the coordinate index of the image and point cloud bimodal anomaly response based on the spatial overlap area of the structural variation coordinate set, aggregate the regional image texture value and the point cloud elevation difference, calculate the fusion feature contribution value, and generate a multimodal anomaly fusion block set.

7. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 6 is characterized in that: The fusion feature contribution value adopts the formula: Among them, F represents the contribution value of fusion features, T z Represents the value of the image texture value in the zth region, P z Represents the value of the point cloud elevation difference in the zth area, H z Represents the point cloud elevation of the zth region, D z represents the structural variation value of the zth region, and m represents the total number of spatially overlapping regions.

8. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 6 is characterized in that: The steps of mapping the local spectrum jump are specifically as follows: S401: Locate the multimodal anomaly fusion block set, extract the spectral reflectance sequence of the industrial facility area in the specified band, identify the pixel spectral vector, perform passband difference scanning and amplitude judgment based on the structural boundary information, identify the reflectance jump point, and generate the facility structure boundary mutation position set; S402: Cluster the jump points in the spatial dimension according to the facility structure boundary mutation position set, screen the jump groups with continuity and boundary contour characteristics, and map them to the original facility image area in combination with the spectral image spatial index to obtain a local spectral jump map.

9. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 1 is characterized in that: The method further comprises step S5: S5: Based on the local spectral jump map, extract the distribution frequency and repetition rate in the inspection path sequence, analyze the frequency distribution of the pump facilities at the differentiated inspection points, identify the areas with abnormal frequency, count the key positions in the inspection path, analyze the repetition rate and abnormal characteristics of each position, and obtain a multimodal diagnosis focus distribution list for the pump facilities; The multimodal diagnosis focus distribution list of the pump facility includes the distribution of high-frequency abnormal areas, inspection repetition rate analysis results, and abnormal feature focus points.

10. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multimodal visual fusion according to claim 9 is characterized in that: The steps of the multimodal diagnosis focus distribution list of the pump facility are specifically as follows: S501: Based on the local spectral jump map, extract the spatial coordinate points in the path image sequence, count the occurrence frequency, identify the jump frame frequency and amplitude, and generate a multimodal visual frequency distribution map; S502: calling the multimodal visual frequency distribution map, identifying areas with higher frequencies than the average frequency of the facility section, extracting repeated regions of transition structures in image frames of multiple time periods, and analyzing them based on structural similarity and the degree of repetition of transition forms to establish an index table of abnormal repetition rate areas; S503: According to the abnormal repetition rate area index table, the morphological overlap of the jump structure in the index area in RGB, thermal imaging and spectral images is compared, the jump structure position with high overlap features is extracted, and the corresponding coordinate points are screened to obtain a multimodal diagnosis focus distribution list of the pump facility.

Images