Industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion

Abstract

The present application relates to the technical field of computer vision, in particular to an industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion, comprising the following steps: acquiring multiple frames of joint images to extract texture dot matrix, analyzing displacement frequency to evaluate stability, calculating angle difference to identify deformation structure, detecting image and point cloud overlap to screen abnormal blocks to generate fusion set, identifying spectrum jump mapping to locate boundary area, extracting frequency to analyze path repetition rate to identify abnormal focus position to generate diagnosis list. In the present application, through multi-frame image texture dot matrix displacement tracking, dynamic abnormal area recognition accuracy is improved, three-dimensional point cloud angle difference captures micro deformation, structural anomaly detection is enhanced, multi-modal anomaly judgment accuracy is improved, abnormal features are refined, inspection path frequency and abnormal aggregation analysis are performed, high-frequency abnormal area is located, inspection resource allocation is optimized, risk prevention and control is strengthened, and the limitations of equipment surface perception and abnormal positioning are broken through.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to an industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion. BACKGROUND

[0002] The technical field of computer vision includes related technologies that simulate the human visual system through computers to achieve automatic acquisition, processing, analysis and understanding of images and videos, and make decisions and feedback. The core content of this technical field includes image acquisition and preprocessing, feature extraction, image segmentation, target recognition, behavior understanding and three-dimensional reconstruction. Computer vision is widely used in intelligent manufacturing, security monitoring, autonomous driving and other scenarios. Its technical development relies on image sensors, deep learning algorithms and large-scale data processing. By constructing a perception system, the target objects and their states in complex scenes can be accurately recognized and analyzed, thereby assisting or replacing manual visual tasks.

[0003] Among them, the industrial facility autonomous inspection and intelligent diagnosis method refers to the means of applying computer vision technology to image acquisition, recognition analysis and judgment diagnosis of the running state of equipment in industrial environment. It covers the following aspects: deploying image acquisition devices to collect images of key equipment at regular intervals or in real time in the industrial facility scene, using image feature matching and change detection methods to identify equipment surface abnormalities such as cracks, rust, liquid leaks, etc., using image classification and recognition methods to determine the type and location of abnormalities, and comparing with historical image sequences to extract fault signs and classify abnormal patterns. This method uses image comparison under fixed rules, pre-defined image variation measurement mechanism and equipment surface image texture analysis method to complete equipment state evaluation and fault feature extraction.

[0004] The prior art mainly compares the static features of single-frame images, relies on image variation measurement and texture change detection under fixed rules, ignores the dynamic evolution process of the target surface state in the time dimension, and thus has low sensitivity to the recognition of early micro-abnormal signs. In terms of spatial analysis, the traditional technology lacks information support of three-dimensional point cloud structure, and can only rely on two-dimensional image texture change, which is difficult to accurately evaluate the geometric deformation characteristics of the equipment structure under different angles, and is prone to misjudgment or omission in the case of slight surface protrusion, distortion and the like. The independent processing procedure between the image and the structure information causes signal fragmentation, which reduces the confidence level of abnormal recognition. In terms of spectral analysis, the traditional method rarely combines the historical reflectivity sequence of the specific pixel position for change analysis, and cannot effectively perceive the spectral mutation caused by the change of material physical properties, and has limited discrimination ability for heterogeneous feature areas such as oil stains and crack oxidation. The inspection path adopts the uniform point distribution mode, does not analyze the high-frequency abnormal distribution combined with the historical data, and lacks a re-inspection mechanism for key abnormal positions, which leads to insufficient attention to key areas, affects the integrity of hidden danger elimination, and affects the effective use of inspection resources. SUMMARY

[0005] To solve the technical problems existing in the prior art, embodiments of the present application provide an industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion. The technical solution is as follows:

[0006] The industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion comprises the following steps:

[0007] S1: acquiring a plurality of image sequences taken at a joint area of an industrial pump body shell, extracting edge texture points in each frame, identifying point array displacement frequency between adjacent frames, analyzing pixel motion amplitude to evaluate texture stability, and generating a time sequence texture change graph;

[0008] S2: based on the key area corresponding to the time sequence texture change graph, extracting three-dimensional point cloud data of the area, performing angle difference calculation on the direction change trend of the object surface at consecutive time points, identifying the change segment of the spatial form, and obtaining a time sequence point cloud deformation plane set;

[0009] S3: calling the image segment corresponding to the position of the time sequence point cloud deformation plane set, detecting the overlapping distribution of image texture intensity and point cloud mutation density in the spatial coordinate system, screening the pixel blocks with synchronous texture disturbance and structure deviation, and generating a multi-modal abnormal fusion block set;

[0010] S4: positioning the multi-modal abnormal fusion block set, identifying the spectral reflectivity sequence of the corresponding area under the specified waveband, performing gradient scanning on the spectral value of each pixel, identifying the boundary jump region, and obtaining a local spectral jump mapping graph.

[0011] As a further scheme of the present application, the time sequence texture change atlas comprises a texture point array stability index, a pixel displacement frequency distribution graph, and an interframe motion amplitude sequence, the time sequence point cloud deformation component set comprises a surface morphology change segment set, an angle difference distribution graph, and a three-dimensional component evolution trend graph, the multi-modal anomaly fusion block set comprises a texture intensity disturbance block, a structure mutation overlapping block, and a spatial anomaly fusion segment, and the local spectral jump mapping graph comprises a spectral gradient distribution graph, a boundary jump feature graph, and a pixel reflectivity change graph.

[0012] As a further scheme of the present application, the step of the time sequence texture change atlas is specifically:

[0013] S101: acquiring a plurality of image sequences photographed at a joint area of an industrial pump body shell, extracting a single-frame pixel matrix, performing directional difference operation on a gray gradient, extracting a local gradient area pixel point, assembling an edge texture point array according to coordinates, recording a coordinate set frame by frame, and generating a joint texture point array sequence;

[0014] S102: based on the joint texture point array sequence, performing difference displacement calculation on adjacent frame corresponding point array coordinates, counting the number of times that the displacement value exceeds a stable threshold value, aggregating abnormal frequencies according to pixel points, and obtaining a joint area texture displacement frequency distribution value;

[0015] S103: calling the joint area texture displacement frequency distribution value, matching a same type joint normal state frequency distribution in a multi-modal inspection library, identifying a difference interval proportion and an over-standard point proportion, marking an abnormal point, drawing a corresponding time frame and an abnormal point proportion curve, and generating a time sequence texture change atlas.

[0016] As a further scheme of the present application, the step of the time sequence point cloud deformation component set is specifically:

[0017] S201: based on a key area corresponding to the time sequence texture change atlas, extracting three-dimensional point cloud data of a corresponding area at an inspection time, identifying point coordinates and performing space matching between time points, analyzing three-dimensional displacement vectors and direction changes of matched points, and generating an inspection time sequence displacement direction data set;

[0018] S202: calling the inspection time sequence displacement direction data set, calculating an angle difference of a continuous time space segment, screening segments with an angle change exceeding a threshold value, merging in combination with connectivity and change persistence, extracting a regional feature contour, and obtaining a time sequence point cloud deformation component set.

[0019] As a further scheme of the present application, the angle difference of the continuous time space segment adopts a formula:

[0020]

[0021] Wherein, Δθ represents the angle difference of the spatial segment at continuous time, θ i represents the angle at time i, θ j represents the angle at time j, w k represents the weight of time segment k, d i,j represents the distance between time i and time j, and n represents the total number of time segments.

[0022] As a further scheme of the present application, the step of the multi-modal anomaly fusion block set is specifically:

[0023] S301: Call the image segment of the corresponding position of the time point cloud deformation profile set, analyze the relationship between image gray gradient and spatial coordinates, extract the regional gray value, compare it with the texture disturbance threshold, identify the distribution of out-of-limit area, and obtain the texture anomaly response graph;

[0024] S302: Based on the corresponding coordinate position in the texture anomaly response graph, extract the point cloud number change value of the same region in the point cloud profile set, identify the proportion of mutation points and compare the point cloud mutation density reference value, screen the structure disturbance region and record the space label, and obtain the structure variation coordinate set;

[0025] S303: According to the spatial overlapping region of the structure variation coordinate set, extract the coordinate index of the image and point cloud double-modal anomaly response, aggregate the regional image texture value and point cloud elevation difference, calculate the fusion feature contribution value, and generate a multi-modal anomaly fusion block set.

[0026] As a further scheme of the present application, the fusion feature contribution value adopts the formula:

[0027]

[0028] Wherein, F represents the fusion feature contribution value, T z represents the value of image texture value in the zth region, P z represents the value of point cloud elevation difference in the zth region, H z represents the point cloud elevation of the zth region, D z represents the structure variation value of the zth region, and m represents the total number of spatial overlapping regions.

[0029] As a further scheme of the present application, the step of the local spectrum jump mapping is specifically:

[0030] S401: Position the multi-modal anomaly fusion block set, extract the spectral reflectance sequence of the industrial facility region in the specified wave band, identify the pixel spectrum vector, and perform passband difference value scanning and amplitude judgment through structure boundary information, identify the reflectance jump point, and generate a facility structure boundary mutation position set;

[0031] S402: According to the facility structure boundary mutation position set, the jump points are clustered in the spatial dimension, the jump group of the continuity and boundary contour features is screened, the local spectral jump mapping diagram is obtained by combining the spectral image space index mapping to the original facility image region.

[0032] As a further scheme of the present application, the method further comprises a step S5:

[0033] S5: Based on the local spectral jump mapping diagram, the distribution frequency and repetition rate in the inspection path sequence are extracted, the frequency distribution of the pump body facility at the differential inspection points is analyzed, the frequency abnormal area is identified, the key positions in the inspection path are counted, the repetition rate and abnormal features of each position are analyzed, and a pump body facility multi-modal diagnosis focusing distribution list is obtained.

[0034] The pump body facility multi-modal diagnosis focusing distribution list includes high-frequency abnormal area distribution, inspection repetition rate analysis result, and abnormal feature focusing point.

[0035] As a further scheme of the present application, the step of the pump body facility multi-modal diagnosis focusing distribution list is specifically:

[0036] S501: Based on the local spectral jump mapping diagram, the spatial coordinate points in the path image sequence are extracted, the occurrence frequency is counted, the jump frame frequency and amplitude are identified, and a multi-modal visual frequency distribution atlas is generated.

[0037] S502: The multi-modal visual frequency distribution atlas is called, the area higher than the average frequency of the facility section is identified, the jump structure repeated area in the multi-time period image frame is extracted, and the abnormal repetition rate area index table is established based on the structure similarity and jump form repetition degree.

[0038] S503: According to the abnormal repetition rate area index table, the coincidence degree of the jump structure in the index area in RGB, thermal imaging and spectral image is compared, the jump structure position of high coincidence feature is extracted, and the corresponding coordinate points are screened, and a pump body facility multi-modal diagnosis focusing distribution list is obtained.

[0039] The technical scheme provided by the embodiment of the present application has at least the following beneficial effects:

[0040] By extracting edge texture points from multi-frame image sequences and tracking displacement frequencies, the motion patterns of local textures can be accurately quantified, thus distinguishing stable regions from anomalous disturbance regions and enhancing the sensitivity to dynamic texture changes. Mapping key regions to 3D point clouds and combining this with angular difference analysis for spatial morphological trend analysis improves the ability to capture minute surface deformations, establishing a multi-moment continuous observation mechanism for extracting anomalous structural changes. Overlapping and comparing image texture intensity with point cloud mutation density in a coordinate system enables co-occurrence filtering among multimodal signals, significantly improving the accuracy and confidence of anomaly detection. Introducing spectral reflectance sequence analysis based on spatial synchronization and identifying boundary transition regions through gradient scanning enhances the response to abrupt changes in material characterization, forming a more hierarchical and in-depth anomaly identification system. Furthermore, statistical analysis of frequency and repetition rates along the inspection path, combined with the focusing characteristics of anomalous pixels, enables precise positioning of key monitoring points and automatic identification of high-frequency anomaly clusters, improving the efficiency of inspection resource allocation and the accuracy of risk control. The entire process effectively addresses the limitations of equipment surface condition perception through time series analysis, spatial deformation modeling, signal overlap detection, and multimodal information fusion. It achieves collaborative diagnosis between cross-dimensional data and enhances the significance of local anomalies, providing more targeted and analytical spectral support for inspection tasks. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the workflow of the present invention;

[0042] Figure 2 This is a detailed flowchart of S1 of the present invention;

[0043] Figure 3 This is a detailed flowchart of the S2 process of the present invention;

[0044] Figure 4 This is a detailed flowchart of the S3 process of the present invention;

[0045] Figure 5 This is a detailed flowchart of the S4 process of the present invention;

[0046] Figure 6 This is a detailed flowchart of S5 of the present invention. Detailed Implementation

[0047] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0048] In the embodiments of the present application, the words such as "example", "for example" are used to represent an example, illustration, or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0049] In the embodiments of the present application, "image" and "picture" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "corresponding" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0050] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0051] In order to make the technical problems, technical schemes and advantages to be solved by the present application more clear, the following will be described in detail in conjunction with the drawings and specific embodiments.

[0052] Please refer to Figure 1 The embodiments of the present application provide an industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion, and the processing flow of the method can include the following steps:

[0053] S1: Obtain a plurality of image sequences shot at a joint area of an industrial pump body shell, extract edge texture points in each frame, identify point array displacement frequency between adjacent frames, analyze pixel motion amplitude to evaluate texture stability, and generate a time sequence texture change graph;

[0054] S2: Based on the key area corresponding to the time sequence texture change graph, extract three-dimensional point cloud data of the area, perform angle difference calculation on the direction change trend of the object surface at consecutive time points, identify the change segment of the spatial form, and obtain a time sequence point cloud deformation surface set;

[0055] S3: Call the image segment corresponding to the position of the time sequence point cloud deformation surface set, detect the overlapping distribution of image texture intensity and point cloud mutation density in the spatial coordinate system, filter the pixel blocks with synchronous texture disturbance and structure deviation, and generate a multi-modal abnormal fusion block set;

[0056] S4: Position the multi-modal anomaly fusion block set, identify the spectral reflectance sequence of the corresponding area under the specified waveband, perform gradient scanning on each pixel point spectral value, identify the boundary jump area, and obtain the local spectral jump mapping diagram;

[0057] S5: Based on the local spectral jump mapping diagram, extract the distribution frequency and repetition rate in the inspection path sequence, analyze the frequency distribution of the pump body facility at the differential inspection points, identify the frequency abnormal area, count the key positions in the inspection path, analyze the repetition rate and abnormal characteristics of each position, and obtain the multi-modal diagnosis focusing distribution list of the pump body facility.

[0058] The time series texture change graph includes a texture point array stability index, a pixel displacement frequency distribution graph, and a frame motion amplitude sequence. The time series point cloud deformation component set includes a surface morphology change segment set, an angle difference distribution graph, and a three-dimensional component evolution trend graph. The multi-modal anomaly fusion block set includes a texture intensity disturbance block, a structure mutation overlap block, and a spatial anomaly fusion segment. The local spectral jump mapping diagram includes a spectral gradient distribution graph, a boundary jump feature graph, and a pixel reflectivity change graph. The multi-modal diagnosis focusing distribution list of the pump body facility includes a high-frequency abnormal area distribution, an inspection repetition rate analysis result, and an abnormal characteristic focusing point.

[0059] Specifically, as shown in Figure 2 The steps of the time series texture change graph are as follows:

[0060] S101: Obtain a plurality of image sequences captured by the industrial pump body shell joint area, extract a single frame pixel matrix, perform directional difference operation on the gray gradient, extract local gradient area pixels, group edge texture points according to coordinates, record coordinate sets frame by frame, and generate a joint texture point array sequence;

[0061] The image acquisition device should shoot the industrial pump body shell joint in a stable and high-precision manner, each frame of image should ensure that the image is clear, avoid affecting the subsequent analysis due to noise or blur, extract the pixel matrix in each frame of image, each pixel point in the image contains a certain gray value, the higher the image resolution, the larger the pixel matrix of each frame, reaching millions or even tens of millions of pixel points. The direction difference operation is performed on the gray gradient of the pixel point to identify the texture change of the joint area. The operation process specifically includes calculating the difference between the gray values of adjacent pixel points to obtain the gradient value in each direction. Through the analysis of the gradient direction and amplitude, the texture feature can be obtained. The local gradient area refers to the position where the gradient change is more significant in the joint area. The area has a strong correlation with the physical characteristics of the joint (such as the gap or connection part). The pixel points in each local gradient area are organized into an edge texture dot matrix according to the coordinates to form a dot matrix structure under a two-dimensional coordinate system. The dot matrix provides data support for subsequent joint state monitoring and change analysis. The coordinates of the edge texture dots are recorded frame by frame, and a joint texture dot matrix sequence is formed. The subsequent analysis will rely on the time sequence characteristics of the dot matrix sequence to form continuous monitoring of the joint texture change.

[0062] S102: Based on the joint texture dot matrix sequence, difference displacement calculation is performed on the corresponding dot matrix coordinates of adjacent frames, the number of displacement values exceeding the stable threshold value is counted, the abnormal frequency is summarized according to the pixel points, and the joint area texture displacement frequency distribution value is obtained;

[0063] The difference displacement calculation is performed on the corresponding dot matrix coordinates of adjacent frames. The calculation process can effectively identify the change of the joint texture by comparing the displacement of the same pixel point position in adjacent frames. The difference displacement calculation first requires setting an accurate reference coordinate point set. The position change of the reference point in two frames of images is calculated through the pixel point coordinates in each frame of image. Specifically, for each pixel point, first determine its coordinate in the first frame, then reposition the position of the point in the second frame image, calculate the displacement value of the point between the two frames, and the displacement value is represented by the coordinate difference. If the displacement exceeds the preset stable threshold value (for example, set to 5 pixels), it is considered that the texture displacement of the position exceeds the normal range. Count the number of times the displacement of all pixel points exceeds the stable threshold value. The threshold value is set to reflect the texture change range of the device under normal working conditions, such as 5 pixels. The specific value will be adjusted according to the material characteristics and working conditions of the joint. Each time the pixel point exceeds the threshold value is recorded as an abnormal value, and the abnormal frequency value is counted. The value can reflect the stability and texture displacement degree of the joint area. Further, the abnormal frequency is summarized and the joint area texture displacement frequency distribution value is calculated. Based on this, subsequent abnormal identification is performed.

[0064] S103: Call the joint area texture offset frequency distribution value, match the same type of joint normal state frequency distribution in the multi-modal inspection library, identify the difference interval proportion and the over-standard point proportion, mark the abnormal point, draw the corresponding time frame and abnormal point proportion curve, and generate a time sequence texture change map;

[0065] After calling the joint area texture offset frequency distribution value, match the same type of joint normal state frequency distribution in the multi-modal inspection library, the step is to compare the normal joint texture offset mode stored in the database, specifically by comparing the texture offset frequency distribution in the existing data with the distribution characteristics in the normal state, to identify the difference interval, for the abnormal texture offset frequency distribution value, according to the difference of the distribution, calculate the difference interval proportion and the over-standard point proportion. The difference interval proportion refers to the difference range between the actual measured frequency distribution and the normal state distribution, such as the difference degree is greater than a certain threshold (for example, 30%), it is considered that the joint has abnormal change, the over-standard point proportion refers to the proportion of pixel points whose displacement exceeds the preset threshold in all monitored pixel points, if the over-standard point proportion exceeds the set threshold (for example, more than 20%), it is considered that the joint appears abnormal, through the calculation of the two indexes, the abnormal position and severity of the joint area can be accurately located, further mark the abnormal point and draw the corresponding time frame and abnormal point proportion curve, which reflects the change trend of the joint texture in the monitoring process, finally generate a time sequence texture change map, which provides effective data basis for the maintenance and repair of the equipment.

[0066] Specifically, as shown in Figure 3 The steps of the time sequence point cloud deformation surface set are specifically:

[0067] S201: Based on the key area corresponding to the time sequence texture change map, extract the three-dimensional point cloud data of the corresponding area at the inspection time, identify the point coordinates and perform space matching between time points, analyze the three-dimensional offset vector and direction change of the matched points, and generate a time sequence offset direction data set;

[0068] According to the position of the key region in the atlas, three-dimensional point cloud data of the region is obtained. The point cloud data is a set of three-dimensional coordinate points obtained by a laser scanner or a stereo vision system, and each point has a specific coordinate value (X, Y, Z) in space. The resolution and accuracy of the point cloud directly affect the accuracy of subsequent analysis. By adjusting the scanning frequency and resolution of the device, the accuracy of the point cloud can reach millimeter level. Then, for the obtained three-dimensional point cloud data, the coordinates of each point need to be identified. First, the point cloud data is processed by point cloud segmentation technology to divide the point cloud into different regions. The point positions in each region are classified according to distance, density and other characteristics. The coordinates of the point positions are spatially matched with the key information in the time series texture change atlas to determine how the spatial position of the point changes at different times. The spatial matching process requires calculating the spatial relationship between the point cloud data and the specific region position in the atlas. This is done by calculating the distance and relative position relationship of the point cloud data in three-dimensional space. In this way, it is determined whether there is a position offset between points at different time points. Through spatial matching between time points, the three-dimensional offset vector of each matching point can be obtained, that is, the displacement amount of the point between different time points is calculated, and the deformation direction of the joint region is identified by the direction change of the vector. The inspection time sequence offset direction data set is generated, which contains the displacement amount and direction of each point in the joint region at different time points, and provides a basis for further deformation analysis.

[0069] S202: Call the inspection time sequence offset direction data set, calculate the angle difference of the spatial segments at consecutive time points, filter the segments with angle change exceeding the threshold, merge them according to connectivity and change persistence, extract the regional feature contour, and obtain the time sequence point cloud deformation surface set;

[0070] Angle difference calculations are performed on each spatial segment at consecutive time points. This process includes calculating the angle changes between the same spatial segments at consecutive time points. First, at each time point, the angle of each spatial segment is calculated using point cloud data. A common angle calculation method is to use the formula for the angle between vectors to calculate the offset direction between any two points in the point cloud. By comparing the amount of angle change between segments at consecutive time points, the angle difference of each spatial segment can be obtained. If the angle change exceeds a preset threshold (e.g., a threshold of 5 degrees), the segment is considered to have undergone significant deformation. Segments with angle changes exceeding this threshold are filtered out. The filtering process requires comparing the angle change value of each segment with the set threshold. Segments exceeding the threshold are marked as potential abnormal regions. Then, a merging operation is performed by combining the connectivity and persistence of changes of spatial segments. Connectivity assessment refers to determining whether adjacent segments belong to the same deformation region by whether their change trends are similar. If adjacent segments have consistent angle changes or similar change patterns, they are considered connected and can be grouped together. The assessment of change persistence is to determine whether segments need to be merged by detecting the duration of the change. If the angle change of a certain area lasts for more than a certain period of time (e.g., more than 10 frames), the change is considered to be significant and should be merged into a group. Based on the information, the feature contour of the area is extracted. The process analyzes the merged segments to identify the deformation features of seams or key areas, and obtains a time-series point cloud deformation surface set, which can reflect the dynamic changes of seams or areas in the time series, and can be further used for equipment health monitoring and maintenance prediction.

[0071] The angular difference between spatial segments at consecutive time points is expressed by the formula:

[0072]

[0073] Where Δθ represents the angular difference of a spatial segment at consecutive time points, θ i θ represents the angle at time i. j w represents the angle at time j. k d represents the weight of time segment k. i,j represents the distance between spatial segments from time i to time j, and n represents the total number of time segments;

[0074] The angle difference of the spatial segment at consecutive time points refers to the degree of change in the orientation or direction of the spatial segment between adjacent time points. The angle of the spatial segment represents the deflection angle of the segment relative to a certain reference direction (e.g., the ground, a geographic coordinate system, etc.). When the same spatial segment is observed at two different time points (e.g., time i and time j), the angle value changes, and the angle difference is the difference between the angles of the spatial segment at the two time points. The size of the angle difference reflects the degree of directional change of the segment within the time span, which is used to analyze the motion or deformation of spatial objects or regions in the time dimension. The larger the angle difference value, the more significant the change of the spatial segment, and vice versa;

[0075] Angle difference Δθ: This term represents the angle difference between time i and time j, and the angle θ i and θ j is obtained from the spatial direction of the sensor or monitoring device at the corresponding time point. The unit of the angle is degrees (°) or radians (rad), and the value is based on the orientation data obtained by spatial sensors or image analysis;

[0076] Weight parameter This term represents the weight sum of the segments at consecutive time points. The weight parameter reflects the relative importance of each spatial segment in the calculation. The weight is obtained by considering the similarity and change persistence of the segments between time points, through comparison of time series images or change frequency of adjacent spatial segments, and selecting segments with greater impact to give higher weight. The unit of the weight is not related to the angle, but it will affect the calculation of the final angle difference. The weight range is generally 0 to 1, and the greater the weight of the segment, the greater the impact on the final result. This value is obtained through a data monitoring system or a calculation model;

[0077] Absolute value sum of weight This term is the absolute value sum of all weight values, which is used for normalization adjustment. The purpose is to standardize the weights of all segments so that the weight sum of all segments is 1. The quantification of this value is obtained by taking the absolute value of the weight of each time segment and then accumulating it;

[0078] Spatial distance |d i,j : This term represents the spatial distance between time i and time j. The spatial distance is calculated by a sensor or image monitoring system, which is the physical distance between adjacent spatial segments. The unit of the distance is meters (m), and this value is obtained by measuring the monitoring system or sensor at the time, through the GPS system or image positioning;

[0079] Assumption: the angle at time i is θ i = 45°, the angle at time j is θ j = 30°, the weight values are w1 = 0.3, w2 = 0.5, w3 = 0.2, and the spatial distance between segments is |d i,j| = 10 meters;

[0080] According to the formula: | θ i - θ j | = | 45 - 30 | = 15°;

[0081] Weight sum:

[0082] Weight absolute value sum:

[0083] Substitute the formula:

[0084] The result shows that the angle difference between time i and time j is 15.72°, and the angle difference value reflects the size of the spatial deformation between the two times, which further determines its contribution to the final time sequence point cloud deformation plane in combination with the weight sum and spatial distance.

[0085] Specifically, as Figure 4 indicated, the steps of the multi-modal anomaly fusion block set are specifically:

[0086] S301: Call the image segment corresponding to the position of the time sequence point cloud deformation plane set, analyze the relationship between image gray gradient and spatial coordinates, extract the regional gray value change, and compare it with the texture disturbance threshold, identify the distribution of the over-limit area, and obtain the texture anomaly response atlas;

[0087] From the image library, the image segment matching the point cloud data is located, and the image segment contains the gray information corresponding to the three-dimensional point cloud area. The target of image gray gradient analysis is to identify the change of gray value in the image area, and the change indicates the existence of small defects or deformation on the surface of the object. In the specific steps, first, the gray value of each pixel point in the image is extracted, and the gray gradient is obtained by calculating the gray difference of adjacent pixel points. For the joint or detail area, the gray gradient changes greatly. By analyzing the relationship between the gray gradient of the analysis area and the spatial coordinates, the deformation trend or surface defect in the image can be revealed. When extracting the regional gray value change, the value is compared with the pre-set texture disturbance threshold. This disturbance threshold is obtained through actual data experience and is set within a certain range, such as 0.1 to 0.5. If the gray value of the region exceeds the pre-set threshold, it is considered that there is a texture anomaly in the region. By comparing the data of multiple image segments, the distribution of the over-limit area is identified, and the texture anomaly response atlas is obtained through image analysis. The atlas can accurately display the abnormal distribution in the image area, providing a basis for subsequent analysis.

[0088] S302: Based on the corresponding coordinate position in the texture anomaly response atlas, the point cloud quantity change value of the same region in the point cloud profile set is extracted, the mutation point quantity proportion is identified and compared with the point cloud mutation density benchmark value, the structure disturbance region is screened and the space label is recorded, and the structure variation coordinate set is obtained;

[0089] The density and distribution of point cloud data are important information for describing the deformation of the surface of an object. The number of each point cloud point reflects the detail level of the region. When the region deforms or has a structure problem, the density of the point cloud will change significantly, especially at the crack, joint or structure deformation. By comparing the point cloud quantity change of the same region in different time periods, potential deformation can be detected. The identification of the mutation point quantity proportion refers to the proportion of point cloud points whose point cloud quantity change exceeds a preset threshold in a given region. For example, if the point cloud quantity change in the region exceeds a certain threshold (such as 30%), it is considered that the region has mutated. By comparing the point cloud mutation density benchmark value (obtained from historical data, such as an average change of 20%), the structure disturbance region can be screened. If the point cloud quantity mutation of a region exceeds the benchmark value, it is considered that the region has obvious deformation or damage. The process further summarizes the point cloud data, records the space label, and obtains the structure variation coordinate set, which reflects the deformation characteristics of each key region in the monitoring period.

[0090] S303: According to the spatial coincidence region of the structure variation coordinate set, the coordinate index of the image and point cloud dual-mode abnormal response is extracted, the regional image texture value and point cloud elevation difference are aggregated, the fusion feature contribution value is calculated, and the multi-modal abnormal fusion block set is generated;

[0091] The image data and point cloud data are accurately matched. By analyzing the spatial coincidence region of the point cloud data and the image data, the correlation between the two can be effectively determined. For example, when a region shows obvious deformation in the point cloud, and at the same time, the region also has abnormal gray scale change in the image, it can be confirmed that there is an actual physical structure problem at this position. Aggregating the regional image texture value and the point cloud elevation difference as a fusion feature can provide more detailed abnormal analysis. The image texture value represents the smoothness or defects of the surface of the object, and the point cloud elevation difference reveals the three-dimensional change of the surface. By combining the abnormal data of the two, the structure problem can be more accurately located. For example, at a certain point, if the texture change value of the image is 0.7 and the point cloud elevation difference is 5mm, it is considered that the region has a more serious structure problem. The multi-modal abnormal fusion block set can integrate the abnormal responses of the image and the point cloud, provide a comprehensive monitoring perspective, and be used for further structure health assessment and maintenance prediction;

[0092] The fusion feature contribution value adopts the formula:

[0093]

[0094] where F represents the fusion feature contribution value, T z represents the image texture value in the zth region, P z represents the point cloud elevation difference in the zth region, H z represents the point cloud elevation in the zth region, D z represents the structure variation value in the zth region, and m represents the total number of spatially overlapping regions.

[0095] The fusion feature contribution value refers to the degree of contribution of the difference between the image texture feature and the elevation information in the point cloud data to the final fusion result in the multi-modal data fusion process. Specifically, by calculating the absolute value of the texture feature of the image and the point cloud elevation difference, and weighting according to the relationship between the point cloud elevation and the structure variation value, the similarities and differences between the image and the point cloud data in a specific region are reflected. Through this calculation method, the influence degree of different regions on the overall fusion result can be evaluated, and the regions with abnormal changes in the image and the point cloud data can be identified. The contribution value is an important indicator in the fusion algorithm, which helps to optimize the data fusion process and improve the accuracy and reliability of the final result.

[0096] Image texture value T z : The texture feature value of the image is calculated by the gray level co-occurrence matrix (GLCM) method, such as contrast, energy, entropy, etc.

[0097] Point cloud elevation difference P z : The elevation difference value at the same position at different time points is calculated by the digital elevation model (DEM) difference method.

[0098] Point cloud elevation H z : The average elevation value of each region is extracted from the point cloud data.

[0099] Structure variation value D z : The degree of structure variation in the image is calculated by the digital image correlation (DIC) method.

[0100] In practical applications, in order to unify the dimensions of various parameters, normalization processing is required, for example, all parameters are normalized to the interval [0, 1] to eliminate the influence of different dimensions on the calculation results.

[0101] Specific numerical example: Assuming that in a certain region z = 1, the following normalized parameter values have been obtained, image texture value T1 = 0.6, point cloud elevation difference P1 = 0.4, point cloud elevation H1 = 0.5, and structure variation value D1 = 0.3.

[0102] Substitute the values into the formula to calculate the fusion feature contribution value of the region:

[0103] The results show that in the region z = 1, the contribution value of the fusion feature is about 0.3428, and after similar calculation is performed on all m regions, the total fusion feature F is obtained by summation, which is used to generate a multi-modal anomaly fusion block set, reflecting the regions of abnormal changes in the image and point cloud data.

[0104] Specifically, as shown in Figure 5 The steps of the local spectral jump mapping diagram are specifically:

[0105] S401: Positioning the multi-modal anomaly fusion block set, extracting the spectral reflectance sequence of the industrial facility region in the specified waveband, identifying the pixel spectral vector, and performing passband difference scanning and amplitude judgment through the structure boundary information, identifying the reflectance jump point, and generating the facility structure boundary mutation position set;

[0106] Extract the spectral reflectance sequence of the industrial facility region in the specified waveband, and the extraction process requires the use of high-precision spectral instruments to capture the reflection characteristics of the industrial facility surface in multiple wavebands. For example, through laser scanning technology or spectral imaging equipment to obtain reflection data in different wavebands such as visible light, infrared light, etc. After obtaining the reflectance sequence, analyze the spectral vector of each pixel point, that is, the reflection intensity of each pixel at different wavelengths. By comparing with the boundary information of the device structure, passband difference scanning is performed, which means comparing the spectral data of different wavebands to calculate the reflectance difference between different wavebands, so as to judge the area where the structure boundary changes. The amplitude judgment is to further evaluate the significance of the change by calculating the amplitude of the reflectance difference, when the reflectance difference exceeds the predetermined threshold (for example, 0.1 units), it is considered that the region has structural changes, according to the judgment, the reflectance jump point is identified, which indicates that the facility structure has significant changes, such as the expansion of cracks or the deformation of joints, the jump points are collected and formed into a facility structure boundary mutation position set, which identifies the positions of major structural changes of the industrial facility during the monitoring process.

[0107] S402: According to the facility structure boundary mutation position set, clustering the jump points in the spatial dimension, screening the jump groups of continuity and boundary contour features, and combining the spectral image space index to map to the original facility image region to obtain a local spectral jump mapping diagram;

[0108] The jump points are clustered in the spatial dimension, and the purpose of clustering is to classify adjacent jump points into the same group, and the points are located in the same structural variation region. When clustering, first calculate the distance between each jump point and the point according to the spatial coordinates, and if the distance between a jump point and the adjacent point is less than a predetermined threshold (for example, 5mm), it is considered that the two points belong to the same cluster group. The jump group is screened for continuity and boundary profile characteristics. Continuity refers to the region with similar change trend in time and space dimensions, and the region shows a relatively stable deformation mode. The boundary profile characteristics are identified by calculating the geometric characteristics of the variation region. If the jump points of a region form a closed boundary, it is considered that the variation of the region is more prominent. By combining the spectral image space index, the screened jump group is mapped to the original facility image region to form a spectral jump region corresponding to the original image, and a local spectral jump mapping map is obtained, which can accurately display the region where the facility structure changes and provide specific spatial information for subsequent structure evaluation.

[0109] Specifically, as shown in Figure 6 the pump body facility multi-modal diagnosis focuses on the distribution list, and the steps are specifically:

[0110] S501: Based on the local spectral jump mapping map, extract the spatial coordinate points in the path image sequence, count the occurrence frequency, identify the jump frame frequency and amplitude, and generate a multi-modal visual frequency distribution atlas;

[0111] Extract the spatial coordinate points in the path image sequence. In the process of extracting spatial coordinate points, first identify all the pixel points with significant changes from the image. The change point is determined by comparing the spectral data, and each extracted coordinate point represents a region with the most significant change in the image frame, which corresponds to the starting point of structural defects or changes. The frequency of the extracted coordinate points is counted. The counting process is to traverse all the spatial coordinates in the image frame and record the number of times each coordinate point appears. If a coordinate point appears frequently, it means that the region changes frequently, and the relationship between the jump frame frequency and the amplitude is analyzed. The jump frame frequency refers to the change frequency of a specific region in multiple image frames, and the amplitude refers to the variation amplitude of the reflectivity or spectral parameter of the region in each change. Large amplitude jump indicates that the region has large physical deformation or structural problems. Through analysis, a multi-modal visual frequency distribution atlas is generated, which shows the frequency change of different regions in the time sequence and can accurately identify potential fault regions or structural parts that need attention.

[0112] S502: Call the multi-modal visual frequency distribution atlas, identify the region higher than the average frequency of the facility section, extract the jump structure repeated region in the multi-period image frame, and analyze based on the structural similarity and jump form repetition degree to establish an abnormal repetition rate region index table;

[0113] Identify the area higher than the average frequency of the facility section, the identification process by comparing the frequency of each area in the atlas with the average frequency of all areas in the facility section, when the frequency of a certain area is significantly higher than the average, which indicates that the area repeatedly appears abnormal changes in multiple time periods, therefore, it needs to be paid attention to. Extract the jump structure repeated area in the multi-time image frame, specifically, compare the images at different time points, identify the area where the structural change position repeatedly appears, which is the weak point of the equipment and the manifestation of structural fatigue or wear. Based on the analysis of structural similarity and jump morphology repetition degree, first, determine the similarity of the structural characteristics of each area by calculating the shape similarity, if a jump area shows similar morphology in multiple time periods, it is considered that the pattern of structural change is more consistent. Secondly, analyze the repetition degree of jump morphology, that is, whether the reflectivity change of the area in multiple time periods has similar amplitude or pattern. Through analysis, an abnormal repetition rate area index table is established, which records the areas that appear frequently and have consistent change patterns in the equipment, and provides them for maintenance personnel to pay attention to first.

[0114] S503: According to the abnormal repetition rate area index table, compare the morphology coincidence degree of the jump structure in the index area in RGB, thermal imaging and spectral images, extract the jump structure position with high coincidence characteristics, and select the corresponding coordinate points to obtain the pump body facility multi-modal diagnosis focusing distribution list;

[0115] Compare the morphology coincidence degree of the jump structure in the index area in RGB, thermal imaging and spectral images, extract the structure morphology of each area in the RGB image, identify the change of surface temperature or significant features by analyzing color change, analyze the heat distribution in the thermal imaging image, judge whether there is temperature abnormality in the area, judge whether the structure is affected by factors such as thermal expansion and cold contraction through temperature change, use spectral image data to further verify whether there is abnormality in spectral characteristics of the area, through the comparison of data, the change type of the area can be more accurately confirmed, the jump structure position with high coincidence characteristics is extracted, the area with high coincidence characteristics is selected, and the corresponding coordinate points are extracted. The coordinate points indicate the abnormal area in the equipment that needs to be paid most attention to, and the pump body facility multi-modal diagnosis focusing distribution list is obtained, which records all the structure parts that show significant abnormalities in different modalities, helping maintenance personnel to accurately locate equipment failure and effectively repair and maintain.

[0116] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An industrial facility autonomous inspection and intelligent diagnosis method based on multi-modal visual fusion, characterized in that, It comprises the following steps: S1: acquire a plurality of image sequences shot at the joint area of the industrial pump body shell, extract edge texture points in each frame, identify point array displacement frequency between adjacent frames, analyze pixel motion amplitude to evaluate texture stability, and generate a time sequence texture change graph; The step of the time sequence texture change graph is specifically: S101: acquire a plurality of image sequences shot at the joint area of the industrial pump body shell, extract a single frame pixel matrix, perform directional difference operation on the gray gradient, extract local gradient area pixel points, group edge texture points according to coordinates, record coordinate sets frame by frame, and generate a joint texture point array sequence; S102: based on the joint texture point array sequence, perform difference displacement calculation on the corresponding point array coordinates of adjacent frames, count the number of times that the displacement value exceeds the stable threshold value, aggregate abnormal frequency according to pixel points, and obtain joint area texture displacement frequency distribution value; S103: call the joint area texture displacement frequency distribution value, match the normal state frequency distribution of the same type of joint in the multi-modal inspection library, identify the difference interval proportion and the over-standard point proportion, mark the abnormal points, draw the corresponding time frame and abnormal point proportion curve, and generate a time sequence texture change graph; S2: based on the key area corresponding to the time sequence texture change graph, extract the three-dimensional point cloud data of the area, perform angle difference calculation on the direction change trend of the object surface at consecutive time, identify the change segment of the spatial form, and obtain a time sequence point cloud deformation surface set; The step of the time sequence point cloud deformation surface set is specifically: S201: based on the key area corresponding to the time sequence texture change graph, extract the three-dimensional point cloud data of the corresponding area at the inspection time, identify the point coordinates and perform space matching between time, analyze the three-dimensional displacement vector and direction change of the matching points, and generate an inspection time sequence displacement direction data set; S202: call the inspection time sequence displacement direction data set, calculate the angle difference of the space segment at consecutive time, filter the segments with angle change exceeding the threshold value, merge them according to connectivity and change persistence, extract the area feature contour, and obtain a time sequence point cloud deformation surface set; S3: call the image segment corresponding to the time sequence point cloud deformation surface set, detect the overlap distribution of image texture intensity and point cloud mutation density in the spatial coordinate system, filter the pixel blocks with synchronous texture disturbance and structure displacement, and generate a multi-modal abnormal fusion block set; S4: locate the multi-modal abnormal fusion block set, identify the spectral reflectance sequence of the corresponding area under the specified waveband, perform gradient scanning on the spectral value of each pixel, identify the boundary jump area, and obtain a local spectral jump mapping.

2. The method of claim 1, wherein, The time sequence texture change graph comprises a texture point array stability index, a pixel displacement frequency distribution graph, and a frame-to-frame motion amplitude sequence. The time sequence point cloud deformation surface set comprises a surface form change segment set, an angle difference distribution graph, and a three-dimensional surface evolution trend graph. The multi-modal abnormal fusion block set comprises a texture intensity disturbance block, a structure mutation overlap block, and a spatial abnormal fusion segment. The local spectral jump mapping comprises a spectral gradient distribution graph, a boundary jump feature graph, and a pixel reflectance change graph.

3. The method of claim 1, wherein, The step of fusing the multi-modal anomaly block set is specifically: S301: Call the image segment corresponding to the position of the time point cloud deformation aspect set, analyze the relationship between image gray gradient and spatial coordinates, extract the regional gray value, compare it with the texture disturbance threshold, identify the distribution of over-limit areas, and obtain the texture anomaly response graph; S302: Based on the corresponding coordinate position in the texture anomaly response graph, extract the point cloud quantity change value of the same region in the point cloud aspect set, identify the mutation point quantity proportion and compare it with the point cloud mutation density reference value, screen the structure disturbance region and record the space label, and obtain the structure variation coordinate set; S303: According to the spatial coincidence area of the structure variation coordinate set, extract the coordinate index of the image and point cloud double-modal anomaly response, aggregate the regional image texture value and point cloud elevation difference, calculate the fusion feature contribution value, and generate a multi-modal anomaly fusion block set.

4. The method of claim 3, wherein, The steps of the local spectral jump mapping diagram are specifically: S401: Position the multi-modal anomaly fusion block set, extract the spectral reflectance sequence of the industrial facility region in the specified waveband, identify the pixel spectral vector, and perform passband difference value scanning and amplitude judgment through structure boundary information to identify the reflectance jump point and generate the facility structure boundary mutation position set; S402: According to the facility structure boundary mutation position set, cluster the jump points in the spatial dimension, screen the jump groups with continuity and boundary contour features, and map the spectral image space index to the original facility image region to obtain a local spectral jump mapping diagram.

5. The method for autonomous inspection and intelligent diagnosis of industrial facilities based on multi-modal visual fusion according to claim 1, characterized in that, The method further comprises the S5 step: S5: Based on the local spectral jump mapping diagram, extract the distribution frequency and repetition rate in the inspection path sequence, analyze the frequency distribution of the pump body facility at the differential inspection points, identify the frequency anomaly area, count the key positions in the inspection path, analyze the repetition rate and abnormal features of each position, and obtain a pump body facility multi-modal diagnosis focusing distribution list; The pump body facility multi-modal diagnosis focusing distribution list includes high-frequency abnormal area distribution, inspection repetition rate analysis result, and abnormal feature focusing point.

6. The method of claim 5, wherein, The steps of the pump body facility multi-modal diagnosis focusing distribution list are specifically: S501: Based on the local spectral jump mapping diagram, extract the spatial coordinate points in the path image sequence, count the occurrence frequency, identify the jump frame frequency and amplitude, and generate a multi-modal visual frequency distribution graph; S502: Call the multi-modal visual frequency distribution graph, identify the area higher than the average frequency of the facility section, extract the jump structure repetition area in the multi-time period image frame, and analyze based on the structure similarity and jump form repetition degree to establish an abnormal repetition rate area index table; S5 03: According to the abnormal repetition rate area index table, compare the morphological coincidence degree of the jump structure in RGB, thermal imaging graph and spectral graph in the index area, extract the jump structure position with high coincidence features, and screen the corresponding coordinate points to obtain a pump body facility multi-modal diagnosis focusing distribution list.

Images