Substation Autonomous Inspection and Monitoring System Based on Industrial Vision
By combining multimodal inspection and image fusion technology with the Mumford-Shah anomaly segmentation model and the Split Bregman algorithm, the problems of accuracy and continuity in anomaly boundary identification in substation visual inspection were solved, and efficient monitoring of anomaly areas was achieved.
Patent Information
- Application Number
- CN202611267767.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing visual inspection technologies for substations are insufficient to accurately identify abnormal areas under conditions such as complex equipment structures, uneven lighting, heat source interference, and discontinuous abnormal boundaries. They are also prone to false alarms and missed detections.
A method is adopted, which involves constructing multimodal inspection frames, fusing visible light and infrared image coordinates, segmenting anomaly boundaries, solving iteratively by splitting, correcting temporal continuity, and implementing linkage monitoring and control. By using the Mumford-Shah anomaly segmentation model and the Split Bregman algorithm, stable anomaly boundaries are generated and linkage monitoring is performed.
It improves the accuracy and continuity of abnormal boundary identification, reduces the occurrence of misjudgments and false alarms, and improves the accuracy and efficiency of inspections.
Smart Images

Figure CN123117522A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial visual monitoring technology, and in particular to a substation autonomous inspection and monitoring system based on industrial vision. Background Technology
[0002] With the advancement of smart grids, unmanned inspections, and digital operation and maintenance of substations, autonomous inspection and monitoring of substations based on industrial vision has become an important application scenario for power equipment condition monitoring. Currently, substation inspections typically collect visible light images, infrared thermal images, and pose data of equipment through inspection robots, fixed camera devices, or mobile inspection equipment. The equipment's appearance defects, abnormal heating, and operational risks are then assessed through image recognition, temperature analysis, and alarm rules.
[0003] Existing visual inspection technologies for substations still have significant limitations under conditions of complex equipment structures, varying shooting angles, uneven lighting, heat source interference, and discontinuous abnormal boundaries. On the one hand, traditional image segmentation or edge detection methods often rely on a single visible light image or a single infrared thermal image, making it difficult to uniformly constrain equipment structural boundaries, visible light edges, and infrared temperature changes. This can easily lead to misjudging normal structural edges as abnormal boundaries or missing abnormal areas due to insignificant temperature gradients. On the other hand, existing anomaly identification methods typically lack multi-frame continuous correction and linkage confirmation mechanisms. They struggle to determine the stability of abnormal areas based on boundary positions, abnormal areas, and edge intensity changes in adjacent inspection frames. This can easily trigger false alarms due to single-frame noise, pose deviations, or instantaneous thermal interference. Furthermore, there is a lack of effective linkage between alarm results and video transmission, inspection path adjustments, and maintenance work order generation, affecting the accuracy and efficiency of autonomous substation inspection and monitoring.
[0004] Therefore, how to provide an autonomous inspection and monitoring system for substations based on industrial vision is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose an autonomous inspection and monitoring system for substations based on industrial vision. This invention achieves autonomous identification and dynamic monitoring of abnormal areas of substation equipment through multimodal inspection frame construction, visible light and infrared image coordinate fusion, abnormal boundary segmentation, splitting iterative solution, temporal continuity correction, and linkage monitoring and control. It has the advantages of high accuracy in abnormal boundary identification, strong stability in continuous abnormal judgment, and high efficiency in inspection linkage response.
[0006] The substation autonomous inspection and monitoring system based on industrial vision according to an embodiment of the present invention includes the following steps: The inspection frame construction module collects visible light images, infrared thermal images, inspection pose data, and acquisition time of the target device, and constructs a multimodal inspection frame sequence for the same device object; The coordinate fusion module maps visible light images and infrared thermal images to a unified equipment coordinate domain based on the inspection pose data, generating the main equipment area, visible light edge field, and infrared temperature gradient field. The anomaly modeling module receives the main area of the device, the visible light edge field, and the infrared temperature gradient field, constructs a Mumford-Shah anomaly segmentation model that embeds the device structure prior and the temperature gradient coupling, and generates the initial solution of the anomaly boundary. The splitting solution module uses the Split Bregman algorithm to split variables in the Mumford-Shah anomaly segmentation model, generating region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables. The iterative convergence module alternately updates the region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables based on the initial solution of the anomaly boundary to generate the anomaly boundary convergence result. The timing correction module continuously corrects the convergence results of the abnormal boundary based on the boundary position offset, abnormal area change and edge intensity change of adjacent frames in the multimodal inspection frame sequence, and generates a stable abnormal region. The linkage monitoring module generates anomaly boundary linkage energy values based on the area, boundary curvature, visible light edge intensity, and infrared temperature gradient of the stable anomaly area. It then triggers monitoring alarms, video transmission, inspection path adjustment, and maintenance work order generation based on the anomaly boundary linkage energy values.
[0007] Optionally, constructing a multimodal inspection frame sequence includes: After the inspection equipment reaches the preset shooting point corresponding to the target equipment, it reads the target equipment number, the spatial coordinates of the inspection equipment, the attitude angle, the lens orientation and the current acquisition time, and starts the visible light camera and the infrared thermal imager respectively under the same trigger signal. The visible light image and infrared thermal image acquired for the same trigger signal are written into the same buffer unit, and the target device number, acquisition time, spatial coordinates, attitude angle and lens orientation are written into the frame header field of the buffer unit to form a single multimodal inspection frame. According to the ascending order of acquisition time, single multimodal inspection frames with the same target device number are written into the same device frame queue, and single multimodal inspection frames with missing target device number, missing visible light image, missing infrared thermal image or missing inspection pose data are removed. Calculate the time difference and pose difference between two adjacent single multimodal inspection frames in the device frame queue; When the time difference is within the preset time range and the pose difference is within the preset pose range, the corresponding single multimodal inspection frame is retained; when the time difference exceeds the preset time range or the pose difference exceeds the preset pose range, the corresponding single multimodal inspection frame is marked as a discontinuous frame. The continuously retained single multimodal inspection frames are sorted according to the acquisition time to generate a multimodal inspection frame sequence corresponding to the target device, and the discontinuous frames are output as sequence segmentation markers.
[0008] Optionally, the generation of the main body region of the device, the visible light edge field, and the infrared temperature gradient field includes: Read a single multimodal inspection frame from the multimodal inspection frame sequence, parse the target device number, spatial coordinates, attitude angle, lens orientation and acquisition time in the frame header field, and read the visible light image and infrared thermal image in the same buffer unit. Call the device reference coordinate template corresponding to the target device number, and project the pixels in the visible light image onto the device reference coordinate template one by one based on the spatial coordinates, attitude angle and lens orientation, and record the coordinate position of each pixel in the device reference coordinate template; The temperature points in the infrared thermal image are projected onto the device reference coordinate template one by one using the same projection method, and the visible light pixel value and infrared temperature value that fall into the same coordinate position of the template after projection are written into the same coordinate unit. Traverse the coordinate cells in the device reference coordinate template, mark the coordinate cells that simultaneously write visible light pixel values and infrared temperature values as effective fusion cells, mark the coordinate cells that lack visible light pixel values or infrared temperature values as empty cells, and retain the empty mark for empty cells. Based on the pre-stored equipment outer contour coordinate range in the equipment reference coordinate template, read the effective fusion units within the target equipment outer contour and connect consecutive adjacent effective fusion units to form the main body area of the equipment. The visible light grayscale values of adjacent coordinate units are read line by line within the main area of the device. When the grayscale difference between adjacent coordinate units is greater than the preset grayscale difference threshold, the corresponding coordinate unit is marked as an edge unit, and the edge units are connected according to the coordinate adjacency relationship to generate a visible light edge field. The infrared temperature values of adjacent coordinate units are read row by row within the main body area of the device. When the temperature difference between adjacent coordinate units is greater than the preset temperature difference threshold, the corresponding coordinate unit is marked as a temperature gradient unit, and the temperature gradient units are connected according to the coordinate adjacency relationship to generate an infrared temperature gradient field. The main body area of the equipment, the visible light edge field, and the infrared temperature gradient field are bound to the single multimodal inspection frame number.
[0009] Optionally, generating initial solutions for anomalous boundaries includes: Read the main body area of the device, the visible light edge field and the infrared temperature gradient field, and write them into the same anomaly modeling cache area according to the single multimodal inspection frame number; Call the prior template of the equipment structure corresponding to the target equipment number, and map the equipment outline area, axial arrangement area and boundary constraint area in the prior template of the equipment structure to the coordinate position corresponding to the equipment body area; Traverse the coordinate cells in the main body area of the device, mark the coordinate cells located within the device outline area as main body area cells, and mark the coordinate cells located outside the device outline area as background area cells, and retain the visible light grayscale value, edge mark value and infrared temperature value corresponding to the main body area cells. Read the edge marker values and infrared temperature values in the main region unit, and write the coordinate units that simultaneously meet the edge continuity condition and temperature change condition into the abnormal candidate unit set. Based on the axial arrangement region and the boundary constraint region, the abnormal candidate unit set is compared in position, and the coordinate units that fall within the normal structural boundary of the equipment and do not form gray-scale abrupt changes or temperature abrupt changes are deleted from the abnormal candidate unit set. The main equipment region, the set of candidate anomalies, the visible light edge field, the infrared temperature gradient field, and the prior template of the equipment structure are used as the input data of the Mumford-Shah anomaly segmentation model, and the set of anomaly boundary coordinates and the anomaly region mask are used as the output data of the Mumford-Shah anomaly segmentation model. Based on the set of anomaly boundary coordinates output by the Mumford-Shah anomaly segmentation model, the coordinates of adjacent anomaly boundaries in the anomaly region mask are connected to generate an initial solution for the anomaly boundaries, and the initial solution for the anomaly boundaries is output.
[0010] Optional, Mumford-Shah anomaly segmentation models include: The system reads the main body region of the device, the set of candidate anomalies, the visible light edge field, the infrared temperature gradient field, and the prior template of the device structure, and constructs a discrete solution domain using the effective fusion units within the main body region of the device. ,Will The visible light gray value of each coordinate unit is denoted as The infrared temperature value is recorded as The edge markers of the visible light edge field are denoted as ,in Index for coordinate cells; The equipment structure prior template is in Upper rasterization yields the structural mask. With structural boundary point sequence And generate the structural tangential direction for each structural boundary point according to the order of adjacent points in the structural boundary point sequence. At the same time, the points adjacent to the structural boundary points in the normal direction will be... Coordinate cells are written into the structural neighborhood table; Candidate boundary units are read from the set of abnormal candidate units and connected according to their coordinate adjacency to form a set of candidate boundary segments. And calculate the visible light gradient direction at each point within each candidate boundary segment. and corresponding normal direction ,in Take and Vertical unit direction; right For each candidate boundary segment, a structural proof of contradiction is performed point by point. Specifically, the structural neighborhood table is searched for points that are related to the candidate boundary point. Nearest structural boundary point ,when arrive When the distance between the candidate boundary point and the structure is less than the preset pixel distance, and the angle between the tangent direction of the candidate boundary point and the tangent direction of the structure is less than the preset angle, the candidate boundary point is added to the structure counter-evidence set. For each candidate boundary segment, the statistics of falling into the boundary are calculated. The percentage of points is determined by the threshold. When the percentage exceeds a preset ratio, the entire candidate boundary segment is written into the structural proof boundary segment table and then... Delete; denote the set of candidate boundary segments that have not been deleted as the set of allowable structural boundaries. and will As the sole candidate source for subsequent fracture sets; Set of allowable boundaries for structures Perform infrared penetration verification, specifically: Each candidate boundary point Along its normal direction Take two sampling points with a fixed offset distance on both sides respectively. and Read and And calculate the temperature difference Simultaneously read the infrared temperature gradient field at gradient label at ; For each candidate boundary point Perform consistency determination during the traversal, specifically: when It is a temperature gradient unit and When the preset temperature difference threshold is exceeded, Write to infrared cross-traverse set For each candidate boundary segment, the number of segments falling within it is statistically analyzed. The length of the continuous point chain is set. When the length of the continuous point chain exceeds the preset length, the boundary segment is added to the forced break set. ; The set of allowed boundary conditions for structures that have not entered However, the remaining boundary segments are written into the optional fracture set. And establish constraints: the break set must contain Furthermore, it must not contain any boundary segment from the structural proof boundary segment table; exist The initial markers for the internal region division are established, specifically: using a forced fracture set. As the initial fracture boundary, Divided into two initial regions on either side of the fracture boundary and Calculate separately and Inside The region grayscale representative value is written into the region fitting table; The variables and interfaces for constructing the Mumford-Shah anomaly segmentation model are as follows: Specifically, defining the region function. exist The upper segment is smooth, and the fracture set is defined. for The set of boundary coordinates within; configure the model input interface to receive... , , , , Structural mask Forced fracture set With optional fracture set Configure the model output interface to output the set of anomaly boundary coordinates. Masking of abnormal regions; The energy of Mumford-Shah anomaly segmentation is defined as a combination of a region fitting term, a region smoothing term, and a fracture length term, and the boundary gating relationship obtained is solidified in the fracture set constraint. The energy is expressed as: ; in, satisfy ,and ; Will Structural mask , structural tangential direction Structural allowable boundary set Forced fracture set Optional fracture set Region fitting table and energy Write the parameters into the model parameter area to form a Mumford-Shah anomaly segmentation model with embedded structure counter-evidence elimination and infrared penetration consistency gating, and output the set of anomaly boundary coordinates and anomaly region mask.
[0011] Optionally, the variables for generating region fitting, boundary regularization, temperature coupling, and structural constraint variables include: Read the Mumford-Shah anomaly segmentation model and extract the discrete solution domain from the model parameter region. Visible light grayscale field Region functions Candidate fracture sets , Structural proof set Infrared Crossing Collection and will Initialize to the initial region partitioning results corresponding to the region fitting table; exist Establish a discrete gradient operator with divergence operator The gradient variable corresponding to the region smoothing term is introduced into the auxiliary variable. Introduce the indicator variable of the fracture boundary into the boundary variable. Structural constraints and temperature travel constraints are introduced into the constraint variables. ,in Record whether the coordinate unit satisfies the gated state of non-structural proof by contradiction and infrared crossing is valid; The solution to the Mumford-Shah anomaly segmentation model is incorporated into the Split Bregman framework to construct split consistency constraints: And write the boundary variable constraints: , At the same time, the temperature crossing constraint is written: when The corresponding coordinate unit must belong to ; for constraints Introducing Bregman variables Introducing Bregman variables for boundary constraints ; In each iteration, the boundary variable update is defined as a two-stage process of solving first and then projecting, specifically: first in... Obtain intermediate boundary variables Then Project onto the feasible set that satisfies the set constraints. The projection rule is: project all... Set the corresponding boundary variables to mandatory boundaries, and select all... The corresponding boundary variables are set to non-boundary, and the remaining boundary variables are restricted to... Within the corresponding coordinate set, a set of coordinates that satisfies the hard constraints is formed. The projection result is then written to the boundary regularization variable cache. The solution process is broken down into four subproblems and updated alternately in a fixed order: For the region fitting subproblem, fix , , , and Update the region function And write it into the region fitting variable table; For gradient-consistent subproblems, fix Compared with other variables, based on and Difference update And write it into the region smoothing variable table; For the boundary regular subproblem, fix , Together with the other variables, generate intermediate boundary variables. And perform the above projection solidification update to obtain And write it into the boundary regularization variable table; For constrained gating subproblems, fixed , , With the other variables, prove the set by contradiction based on the structure. With infrared crossing collection Determine the boundary coordinates point by point to generate And the boundary points that do not satisfy the non-structural proof by contradiction and for which infrared crossing is valid are removed from... Delete the corresponding coordinate set and write the updated boundary coordinate set back to the boundary regularization variable table; Structural constraints and temperature coupling constraints are written into two separate gated channels and the Bregman variables are updated accordingly. Specifically, when a boundary point triggers a structural contradiction condition, only the Bregman variable is updated. The structural channels remain unchanged while the temperature channels remain constant; when the boundary point triggers a temperature crossing condition that does not hold, only the structure is updated. The temperature channel is updated while the structural channel remains unchanged; when the boundary point simultaneously satisfies both types of gating conditions, the two channels are updated synchronously; and the update results of the two channels are written to the boundary Bregman variable buffer. After each round of four subproblems is completed, the Bregman variable is updated according to the following rules: , And calculate the energy convergence criterion value, which is expressed as: ; in, Indicates the first The abnormal segmentation energy value corresponding to the round iteration; when The iteration stops when the energy is less than a preset energy threshold and the boundary coordinate set remains unchanged for two consecutive iterations, and the final region fitting variables are output. Regional smoothing variables Boundary regularization variables and gated constraint variables ; Boundary regularization variables The corresponding coordinate set is written into the abnormal boundary convergence set, and the region fitting variables are... The corresponding region divisions are written into the region convergence table, and the abnormal boundary convergence set and the region convergence table are output.
[0012] Optionally, generating convergence results for anomalous boundaries includes: Read the initial solution of the outlier boundary and read the region fitting variables. Boundary regularization variables Temperature coupling variables and structural constraint variables Write the boundary coordinates of the initial solution of the abnormal boundary into the current boundary table. Write to the current table in the region; The current boundary table is connected into a boundary chain according to the coordinate adjacency relationship, and the boundary chain is standardized. The standardization process includes rearranging the boundary chain order according to the fixed starting point rule, deleting duplicate boundary points, and filling the gaps of single-point breaks to obtain a standard boundary chain. Boundary topology fingerprint locking generates boundary topology fingerprints for standard boundary chains. The boundary topological fingerprint consists of the boundary chain length, endpoint coordinate sequence, and closure state encoding, and... Write to the fingerprint history table; Perform alternating updates in a fixed order: based on standard boundary chains and Update the current table of the region According to the updated In the boundary normal neighborhood, recalculate the gray-scale difference and temperature difference on both sides of the boundary, write back the temperature coupling variable and update the structural constraint variable synchronously; delete the boundary points that violate the gating condition from the current boundary table based on the updated constraint variable, and reconnect the remaining boundary points to generate a new standard boundary chain; When the newly generated standard boundary chain does not cover the coordinates corresponding to the forced break set, the current boundary table is rolled back to the previous round of standard boundary chain, and deletion or connection operations are only allowed for the coordinates corresponding to the optional break set. After forming a boundary chain that covers the forced break set, the next round of update will begin. After each round of updates, the convergence criterion is calculated and written into the convergence record table. The convergence criterion is represented as: ; in, Indicates the first Round region fitting results, Indicates the first Wheel boundary topological fingerprint, This indicates a fingerprint inconsistency indicator. when When the preset threshold is met and the boundary topology fingerprint remains consistent within a preset number of consecutive iterations, the abnormal boundary convergence result of the corresponding standard boundary chain is output, and the abnormal boundary convergence result is output along with the abnormal region mask corresponding to the current region table.
[0013] Optionally, generating stable anomalous regions includes: Read the multimodal inspection frame sequence, and read the convergence results of the abnormal boundary and the abnormal region mask of each inspection frame. Write them into the frame boundary table and the frame region table according to the frame number. For the Frame and the The frame boundary table is used to calculate the boundary centroid coordinate difference to obtain the boundary position offset. The change in abnormal area is obtained by statistically analyzing the difference in the number of pixels in abnormal regions in the frame region table. ; Along the first Frame and the The edge response values of the visible light edge field are read point by point from the boundary coordinates of the frames and summed to obtain the edge intensities of the two frames, and the change in edge intensity is calculated. ; With three consecutive frames As gated windows, respectively for , , Perform threshold determination, count the number of indicators that meet the threshold, and when the number of indicators that meet the threshold in the determination of adjacent frames in the same frame reaches a preset number, then the [frame name] will be [set to a certain threshold]. Frame boundaries are determined to be continuous boundaries; otherwise, the first... Frame boundaries are identified as transition boundaries; When the When the frame boundary is determined to be a continuous boundary, the first... The abnormal region mask of the frame is directly written into the stable frame table; when the first frame... When a frame boundary is determined to be a jump boundary, the first frame in the stable frame table is read. Frame-stable anomalous boundaries are translated along the direction of boundary centroid offset. Generate a prediction boundary, and use the region enclosed by the prediction boundary to evaluate the first... The frame anomaly region mask performs cropping, retaining anomalous pixels within the predicted boundary area to form candidate stable regions; For each frame of stable anomaly boundaries, a boundary segment identifier is generated. The boundary segment identifier consists of the boundary closure state and the number of boundary coordinates encoded; when the... The boundary segment identifier corresponding to the candidate stable region of the frame and the first When the boundary segment identifiers of frame stability anomaly boundaries are inconsistent, the first segment will be... Frame instability anomaly region rollback to the first Frame stability anomaly boundary The enclosed area after translation, and the first The original anomaly boundary convergence result is marked as a discontinuous boundary. The first The final stable abnormal region of the frame is written into the stable region sequence and output.
[0014] The beneficial effects of this invention are: This invention constructs a Mumford-Shah anomaly segmentation model that embeds the prior knowledge of the equipment structure and the coupling relationship of the infrared temperature gradient. This enables the anomaly boundary segmentation process to combine the equipment structure contour, visible light edge changes, and infrared temperature changes for joint constraints. This avoids the problem that traditional single edge segmentation methods easily misjudge normal equipment structure boundaries as anomaly areas, and reduces the impact of local illumination changes, shadow interference, and thermal noise on anomaly identification results, thereby improving the accuracy and continuity of anomaly boundary extraction in complex substation environments.
[0015] This invention employs the Split Bregman algorithm to split variables and iteratively solve the Mumford-Shah anomaly segmentation model, enabling region fitting, boundary regularization, temperature coupling, and structural constraints to be updated independently while maintaining synergistic constraint relationships. This reduces the overall difficulty of solving complex anomaly segmentation models, improves the model's solution efficiency and convergence stability in large-scale inspection images, and reduces local oscillations and discontinuities that occur during anomaly boundary iteration. Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the substation autonomous inspection and monitoring system based on industrial vision proposed in this invention; Figure 2 This is a flowchart illustrating the construction process of the Mumford-Shah anomaly segmentation model proposed in this invention. Detailed Implementation
[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0018] refer to Figures 1-2 The substation autonomous inspection and monitoring system based on industrial vision includes the following steps: The inspection frame construction module collects visible light images, infrared thermal images, inspection pose data, and acquisition time of the target device, and constructs a multimodal inspection frame sequence for the same device object; The coordinate fusion module maps visible light images and infrared thermal images to a unified equipment coordinate domain based on the inspection pose data, generating the main equipment area, visible light edge field, and infrared temperature gradient field. The anomaly modeling module receives the main area of the device, the visible light edge field, and the infrared temperature gradient field, constructs a Mumford-Shah anomaly segmentation model that embeds the device structure prior and the temperature gradient coupling, and generates the initial solution of the anomaly boundary. The splitting solution module uses the Split Bregman algorithm to split variables in the Mumford-Shah anomaly segmentation model, generating region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables. The iterative convergence module alternately updates the region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables based on the initial solution of the anomaly boundary to generate the anomaly boundary convergence result. The timing correction module continuously corrects the convergence results of the abnormal boundary based on the boundary position offset, abnormal area change and edge intensity change of adjacent frames in the multimodal inspection frame sequence, and generates a stable abnormal region. The linkage monitoring module generates anomaly boundary linkage energy values based on the area, boundary curvature, visible light edge intensity, and infrared temperature gradient of the stable anomaly area. It then triggers monitoring alarms, video transmission, inspection path adjustment, and maintenance work order generation based on the anomaly boundary linkage energy values.
[0019] In this embodiment, constructing the multimodal inspection frame sequence includes: After the inspection equipment reaches the preset shooting point corresponding to the target equipment, it reads the target equipment number, the spatial coordinates of the inspection equipment, the attitude angle, the lens orientation and the current acquisition time, and starts the visible light camera and the infrared thermal imager respectively under the same trigger signal. The visible light image and infrared thermal image acquired for the same trigger signal are written into the same buffer unit, and the target device number, acquisition time, spatial coordinates, attitude angle and lens orientation are written into the frame header field of the buffer unit to form a single multimodal inspection frame. According to the ascending order of acquisition time, single multimodal inspection frames with the same target device number are written into the same device frame queue, and single multimodal inspection frames with missing target device number, missing visible light image, missing infrared thermal image or missing inspection pose data are removed. Calculate the time difference and pose difference for two adjacent single multimodal inspection frames in the device frame queue. The pose difference is expressed as: ; in, Indicates the first The pose difference between a single multimodal inspection frame and the previous frame. Indicates the first The spatial coordinates corresponding to a single multimodal inspection frame Indicates the first The attitude angle corresponding to a single multimodal inspection frame. Indicates the attitude angle weighting coefficient; When the time difference is within the preset time range and the pose difference is within the preset pose range, the corresponding single multimodal inspection frame is retained; when the time difference exceeds the preset time range or the pose difference exceeds the preset pose range, the corresponding single multimodal inspection frame is marked as a discontinuous frame. The continuously retained single multimodal inspection frames are sorted according to the acquisition time to generate a multimodal inspection frame sequence corresponding to the target device, and the discontinuous frames are output as sequence segmentation markers.
[0020] In this embodiment, the generation of the main device area, the visible light edge field, and the infrared temperature gradient field includes: Read a single multimodal inspection frame from the multimodal inspection frame sequence, parse the target device number, spatial coordinates, attitude angle, lens orientation and acquisition time in the frame header field, and read the visible light image and infrared thermal image in the same buffer unit. Call the device reference coordinate template corresponding to the target device number, and project the pixels in the visible light image onto the device reference coordinate template one by one based on the spatial coordinates, attitude angle and lens orientation, and record the coordinate position of each pixel in the device reference coordinate template; The temperature points in the infrared thermal image are projected onto the device reference coordinate template one by one using the same projection method, and the visible light pixel value and infrared temperature value that fall into the same coordinate position of the template after projection are written into the same coordinate unit. Traverse the coordinate cells in the device reference coordinate template, mark the coordinate cells that simultaneously write visible light pixel values and infrared temperature values as effective fusion cells, mark the coordinate cells that lack visible light pixel values or infrared temperature values as empty cells, and retain the empty mark for empty cells. Based on the pre-stored equipment outer contour coordinate range in the equipment reference coordinate template, read the effective fusion units within the target equipment outer contour and connect consecutive adjacent effective fusion units to form the main body area of the equipment. The visible light grayscale values of adjacent coordinate units are read line by line within the main area of the device. When the grayscale difference between adjacent coordinate units is greater than the preset grayscale difference threshold, the corresponding coordinate unit is marked as an edge unit, and the edge units are connected according to the coordinate adjacency relationship to generate a visible light edge field. The infrared temperature values of adjacent coordinate units are read row by row within the main body area of the device. When the temperature difference between adjacent coordinate units is greater than the preset temperature difference threshold, the corresponding coordinate unit is marked as a temperature gradient unit, and the temperature gradient units are connected according to the coordinate adjacency relationship to generate an infrared temperature gradient field. The main body area of the equipment, the visible light edge field, and the infrared temperature gradient field are bound to the single multimodal inspection frame number.
[0021] In this embodiment, generating the initial solution for the abnormal boundary includes: Read the main body area of the device, the visible light edge field and the infrared temperature gradient field, and write them into the same anomaly modeling cache area according to the single multimodal inspection frame number; Call the prior template of the equipment structure corresponding to the target equipment number, and map the equipment outline area, axial arrangement area and boundary constraint area in the prior template of the equipment structure to the coordinate position corresponding to the equipment body area; Traverse the coordinate cells in the main body area of the device, mark the coordinate cells located within the device outline area as main body area cells, and mark the coordinate cells located outside the device outline area as background area cells, and retain the visible light grayscale value, edge mark value and infrared temperature value corresponding to the main body area cells. Read the edge marker values and infrared temperature values in the main region unit, and write the coordinate units that simultaneously meet the edge continuity condition and temperature change condition into the abnormal candidate unit set. Based on the axial arrangement region and the boundary constraint region, the abnormal candidate unit set is compared in position, and the coordinate units that fall within the normal structural boundary of the equipment and do not form gray-scale abrupt changes or temperature abrupt changes are deleted from the abnormal candidate unit set. The main equipment region, the set of candidate anomalies, the visible light edge field, the infrared temperature gradient field, and the prior template of the equipment structure are used as the input data of the Mumford-Shah anomaly segmentation model, and the set of anomaly boundary coordinates and the anomaly region mask are used as the output data of the Mumford-Shah anomaly segmentation model. Based on the set of anomaly boundary coordinates output by the Mumford-Shah anomaly segmentation model, the coordinates of adjacent anomaly boundaries in the anomaly region mask are connected to generate an initial solution for the anomaly boundaries, and the initial solution for the anomaly boundaries is output.
[0022] In this embodiment, the Mumford-Shah anomaly segmentation model includes: The system reads the main body region of the device, the set of candidate anomalies, the visible light edge field, the infrared temperature gradient field, and the prior template of the device structure, and constructs a discrete solution domain using the effective fusion units within the main body region of the device. ,Will The visible light gray value of each coordinate unit is denoted as The infrared temperature value is recorded as The edge markers of the visible light edge field are denoted as ,in Index for coordinate cells; The equipment structure prior template is in Upper rasterization yields the structural mask. With structural boundary point sequence And generate the structural tangential direction for each structural boundary point according to the order of adjacent points in the structural boundary point sequence. At the same time, the points adjacent to the structural boundary points in the normal direction will be... Coordinate cells are written into the structural neighborhood table; Candidate boundary units are read from the set of abnormal candidate units and connected according to their coordinate adjacency to form a set of candidate boundary segments. And calculate the visible light gradient direction at each point within each candidate boundary segment. and corresponding normal direction ,in Take and Vertical unit direction; right For each candidate boundary segment, a structural proof of contradiction is performed point by point. Specifically, the structural neighborhood table is searched for points that are related to the candidate boundary point. Nearest structural boundary point ,when arrive When the distance between the candidate boundary point and the structure is less than the preset pixel distance, and the angle between the tangent direction of the candidate boundary point and the tangent direction of the structure is less than the preset angle, the candidate boundary point is added to the structure counter-evidence set. For each candidate boundary segment, the statistics of falling into the boundary are calculated. The percentage of points is determined by the threshold. When the percentage exceeds a preset ratio, the entire candidate boundary segment is written into the structural proof boundary segment table and then... Delete; denote the set of candidate boundary segments that have not been deleted as the set of allowable structural boundaries. and will As the sole candidate source for subsequent fracture sets; Set of allowable boundaries for structures Perform infrared penetration verification, specifically: Each candidate boundary point Along its normal direction Take two sampling points with a fixed offset distance on both sides respectively. and Read and And calculate the temperature difference Simultaneously read the infrared temperature gradient field at gradient label at ; For each candidate boundary point Perform consistency determination during the traversal, specifically: when It is a temperature gradient unit and When the preset temperature difference threshold is exceeded, Write to infrared cross-traverse set For each candidate boundary segment, the number of segments falling within it is statistically analyzed. The length of the continuous point chain is set. When the length of the continuous point chain exceeds the preset length, the boundary segment is added to the forced break set. ; The set of allowed boundary conditions for structures that have not entered However, the remaining boundary segments are written into the optional fracture set. And establish constraints: the break set must contain Furthermore, it must not contain any boundary segment from the structural proof boundary segment table; exist The initial markers for the internal region division are established, specifically: using a forced fracture set. As the initial fracture boundary, Divided into two initial regions on either side of the fracture boundary and Calculate separately and Inside The region grayscale representative value is written into the region fitting table; The variables and interfaces for constructing the Mumford-Shah anomaly segmentation model are as follows: Specifically, defining the region function. exist The upper segment is smooth, and the fracture set is defined. for The set of boundary coordinates within; configure the model input interface to receive... , , , , Structural mask Forced fracture set With optional fracture set Configure the model output interface to output the set of anomaly boundary coordinates. Masking of abnormal regions; The energy of Mumford-Shah anomaly segmentation is defined as a combination of a region fitting term, a region smoothing term, and a fracture length term, and the boundary gating relationship obtained is solidified in the fracture set constraint. The energy is expressed as: ; in, satisfy ,and ; Will Structural mask , structural tangential direction Structural allowable boundary set Forced fracture set Optional fracture set Region fitting table and energy Write the parameters into the model parameter area to form a Mumford-Shah anomaly segmentation model with embedded structure counter-evidence elimination and infrared penetration consistency gating, and output the set of anomaly boundary coordinates and anomaly region mask of the model.
[0023] In this embodiment, the generation of region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables includes: Read the Mumford-Shah anomaly segmentation model and extract the discrete solution domain from the model parameter region. Visible light grayscale field Region functions Candidate fracture sets , Structural proof set Infrared Crossing Collection and will Initialize to the initial region partitioning results corresponding to the region fitting table; exist Establish a discrete gradient operator with divergence operator The gradient variable corresponding to the region smoothing term is introduced into the auxiliary variable. Introduce the indicator variable of the fracture boundary into the boundary variable. Structural constraints and temperature travel constraints are introduced into the constraint variables. ,in Record whether the coordinate unit satisfies the gated state of non-structural proof by contradiction and infrared crossing is valid; The solution to the Mumford-Shah anomaly segmentation model is incorporated into the Split Bregman framework to construct split consistency constraints: And write the boundary variable constraints: , At the same time, the temperature crossing constraint is written: when The corresponding coordinate unit must belong to ; for constraints Introducing Bregman variables Introducing Bregman variables for boundary constraints ; In each iteration, the boundary variable update is defined as a two-stage process of solving first and then projecting, specifically: first in... Obtain intermediate boundary variables Then Project onto the feasible set that satisfies the set constraints. The projection rule is: project all... Set the corresponding boundary variables to mandatory boundaries, and select all... The corresponding boundary variables are set to non-boundary, and the remaining boundary variables are restricted to... Within the corresponding coordinate set, a set of coordinates that satisfies the hard constraints is formed. The projection result is then written to the boundary regularization variable cache. The solution process is broken down into four subproblems and updated alternately in a fixed order: For the region fitting subproblem, fix , , , and Update the region function And write it into the region fitting variable table; For gradient-consistent subproblems, fix Compared with other variables, based on and Difference update And write it into the region smoothing variable table; For the boundary regular subproblem, fix , Together with the other variables, generate intermediate boundary variables. And perform the above projection solidification update to obtain And write it into the boundary regularization variable table; For constrained gating subproblems, fixed , , With the other variables, prove the set by contradiction based on the structure. With infrared crossing collection Determine the boundary coordinates point by point to generate And the boundary points that do not satisfy "non-structural proof by contradiction and infrared crossing is valid" will be removed from... Delete the corresponding coordinate set and write the updated boundary coordinate set back to the boundary regularization variable table; The Bregman dual-channel update for gated constraints writes structural constraints and temperature coupling constraints into two separate gated channels and updates the Bregman variables accordingly. Specifically, when a boundary point triggers a structural contradiction condition, only the Bregman variables are updated. The structural channels remain unchanged while the temperature channels remain constant; when the boundary point triggers a temperature crossing condition that does not hold, only the structure is updated. The temperature channel is updated while the structural channel remains unchanged; when the boundary point simultaneously satisfies both types of gating conditions, the two channels are updated synchronously; and the update results of the two channels are written to the boundary Bregman variable buffer. After each round of four subproblems is completed, the Bregman variable is updated according to the following rules: , And calculate the energy convergence criterion value, which is expressed as: ; in, Indicates the first The abnormal segmentation energy value corresponding to the round iteration; when The iteration stops when the energy is less than a preset energy threshold and the boundary coordinate set remains unchanged for two consecutive iterations, and the final region fitting variables are output. Regional smoothing variables Boundary regularization variables and gated constraint variables ; Boundary regularization variables The corresponding coordinate set is written into the abnormal boundary convergence set, and the region fitting variables are... The corresponding region divisions are written into the region convergence table, and the abnormal boundary convergence set and the region convergence table are output.
[0024] In this embodiment, generating the convergence result of the abnormal boundary includes: Read the initial solution of the outlier boundary and read the region fitting variables. Boundary regularization variables Temperature coupling variables and structural constraint variables Write the boundary coordinates of the initial solution of the abnormal boundary into the current boundary table. Write to the current table in the region; The current boundary table is connected into a boundary chain according to the coordinate adjacency relationship, and the boundary chain is standardized. The standardization process includes rearranging the boundary chain order according to the fixed starting point rule, deleting duplicate boundary points, and filling the gaps of single-point breaks to obtain a standard boundary chain. Boundary topology fingerprint locking generates boundary topology fingerprints for standard boundary chains. The boundary topological fingerprint consists of the boundary chain length, endpoint coordinate sequence, and closure state encoding, and... Write the fingerprint history table to determine the topological stability of the boundary chain during iteration; Perform alternating updates in a fixed order: based on standard boundary chains and Update the current table of the region According to the updated In the boundary normal neighborhood, recalculate the gray-scale difference and temperature difference on both sides of the boundary, write back the temperature coupling variable and update the structural constraint variable synchronously; delete the boundary points that violate the gating condition from the current boundary table based on the updated constraint variable, and reconnect the remaining boundary points to generate a new standard boundary chain; Forced break rollback verification: When the newly generated standard boundary chain does not cover the coordinates corresponding to the forced break set, the current boundary table is rolled back to the previous round of standard boundary chain, and only deletion or connection operations are allowed on the coordinates corresponding to the optional break set. After forming a boundary chain that covers the forced break set, the next round of update is then entered. After each round of updates, the convergence criterion is calculated and written into the convergence record table. The convergence criterion is represented as: ; in, Indicates the first Round region fitting results, Indicates the first Wheel boundary topological fingerprint, This indicates a fingerprint inconsistency indicator. when When the preset threshold is met and the boundary topology fingerprint remains consistent within a preset number of consecutive iterations, the abnormal boundary convergence result of the corresponding standard boundary chain is output, and the abnormal boundary convergence result is output along with the abnormal region mask corresponding to the current region table.
[0025] In this embodiment, generating stable anomaly regions includes: Read the multimodal inspection frame sequence, and read the convergence results of the abnormal boundary and the abnormal region mask of each inspection frame. Write them into the frame boundary table and the frame region table according to the frame number. For the Frame and the The frame boundary table is used to calculate the boundary centroid coordinate difference to obtain the boundary position offset. The change in abnormal area is obtained by statistically analyzing the difference in the number of pixels in abnormal regions in the frame region table. ; Along the first Frame and the The edge response values of the visible light edge field are read point by point from the boundary coordinates of the frames and summed to obtain the edge intensities of the two frames, and the change in edge intensity is calculated. ; With three consecutive frames As gated windows, respectively for , , Perform threshold determination, count the number of indicators that meet the threshold, and when the number of indicators that meet the threshold in the determination of adjacent frames in the same frame reaches a preset number, then the [frame name] will be [set to a certain threshold]. Frame boundaries are determined to be continuous boundaries; otherwise, the first... Frame boundaries are identified as transition boundaries; When the When the frame boundary is determined to be a continuous boundary, the first... The abnormal region mask of the frame is directly written into the stable frame table; when the first frame... When a frame boundary is determined to be a jump boundary, the first frame in the stable frame table is read. Frame-stable anomalous boundaries are translated along the direction of boundary centroid offset. Generate a prediction boundary, and use the region enclosed by the prediction boundary to evaluate the first... The frame anomaly region mask performs cropping, retaining anomalous pixels within the predicted boundary area to form candidate stable regions; For each frame of stable anomaly boundaries, a boundary segment identifier is generated. The boundary segment identifier consists of the boundary closure state and the number of boundary coordinates encoded; when the... The boundary segment identifier corresponding to the candidate stable region of the frame and the first When the boundary segment identifiers of frame stability anomaly boundaries are inconsistent, the first segment will be... Frame instability anomaly region rollback to the first Frame stability anomaly boundary The enclosed area after translation, and the first The original anomaly boundary convergence result is marked as a discontinuous boundary. The first The final stable anomaly region of the frame is written into the stable region sequence and output, where the continuity determination quantity is represented as: ; in, , and These are fixed weighting coefficients.
[0026] In this embodiment, the method for generating abnormal boundary linkage energy values and triggering linkage control includes: Read the set of coordinates of stable anomaly regions and their corresponding anomaly boundaries, and count the number of pixels within the stable anomaly regions to obtain the anomaly area. ; Calculate the rotation angles of adjacent boundary segments in coordinate order along the abnormal boundary coordinate set and accumulate them to obtain the boundary curvature. The edge response values of the visible light edge field are read point by point at the coordinates of the anomalous boundary and summed to obtain the visible light edge intensity. Within the stable anomalous region, the gradient values of the infrared temperature gradient field are read point by point and summed to obtain the infrared temperature gradient amount. ; right , , and Each value is proportionally converted according to a fixed reference interval to obtain the corresponding area ratio, curvature ratio, edge ratio, and temperature ratio. The four ratio values are then weighted and summed according to preset fixed weights to generate the abnormal boundary linkage energy value. ,in It is obtained by multiplying the area ratio by the area weight, the curvature ratio by the curvature weight, the edge ratio by the edge weight, and the temperature ratio by the temperature weight, and then summing them up. In calculation , , and Previously, boundary points falling within the coordinate range of the structural counter-evidence set from the abnormal boundary coordinate set were removed from the statistical set, and boundary points that did not meet the infrared penetration condition were removed from the temperature statistical set. Then, based on the removed boundaries and regions, [further steps were taken]. , , and Calculation; Linkage energy value Compared with the preset grading threshold, when When the value is below the first threshold, only abnormal information is recorded; when When the first threshold is reached, a monitoring alarm is triggered and video transmission is initiated; when When the second threshold is reached, an inspection path adjustment instruction is generated and directed to the current stable anomaly area; when When the third threshold is reached, an operation and maintenance work order is generated, and the abnormal boundary coordinate set and abnormal area are recorded. Boundary curvature quantity and infrared temperature gradient Write the work order details; Read the linkage energy value of the same target device in consecutive adjacent inspection frames. When the energy value within a consecutive preset number of frames all exceed the same threshold and the stable abnormal area remains continuous in spatial position, confirm the trigger result. When only a single frame exceeds the threshold and adjacent frames do not reach the threshold, do not perform path adjustment and work order generation, only retain alarm records, and output the finally confirmed trigger result to the corresponding execution unit.
[0027] Example: To verify the feasibility of this invention, it was applied to an autonomous inspection and monitoring system for a large substation. This substation houses various high-voltage power equipment, including main transformers, circuit breakers, disconnectors, cable joints, busbar connectors, bushings, and surge arresters. This equipment operates continuously for extended periods, and some connecting components are prone to overheating and surface cracks due to increased contact resistance, insulation aging, surface contamination, or localized moisture. Existing inspection methods primarily rely on inspection robots to collect single visible light images or infrared thermal images, which are then used by a backend system for anomaly identification through edge detection, temperature threshold judgment, and manual verification. In actual operation, factors such as reflective metal casings, uneven nighttime lighting, background heat source interference, and changes in the inspection robot's posture can all lead to normal equipment outlines being misjudged as crack boundaries, or actual overheating areas being obscured by background heat diffusion, resulting in false alarms, missed detections, and discontinuous anomaly boundaries.
[0028] In this embodiment, the inspection robot is equipped with a visible light industrial camera, an infrared thermal imager, a pose positioning unit, and an edge computing terminal. After the robot arrives at the preset shooting point corresponding to the target device, the system reads the target device number, robot spatial coordinates, attitude angle, lens orientation, and acquisition time, and simultaneously starts the visible light industrial camera and infrared thermal imager under the same trigger signal. The synchronously acquired visible light image and infrared thermal image are written to the same buffer unit, and the target device number, acquisition time, spatial coordinates, attitude angle, and lens orientation are written to the frame header field of the buffer unit, thereby forming a single multimodal inspection frame. The system then writes multiple single multimodal inspection frames of the same target device into the same device frame queue in ascending order of acquisition time, and removes invalid frames that lack device number, visible light image, infrared thermal image, or pose data, forming a multimodal inspection frame sequence that can be used for subsequent analysis.
[0029] During image fusion processing, the system retrieves the corresponding device reference coordinate template based on the target device number in the frame header field. Then, according to the robot's spatial coordinates, attitude angle, and lens orientation, it projects pixels from the visible light image onto the device reference coordinate template point by point. Simultaneously, it projects temperature points from the infrared thermal image onto the same template using the same mapping rules. For visible light pixel values and infrared temperature values falling into the same template coordinate position, the system writes them into the same coordinate unit; for coordinate units lacking visible light pixel values or infrared temperature values, the system retains a missing marker. Subsequently, based on the pre-stored device outer contour coordinate range in the device reference coordinate template, the system extracts effective fusion units within the target device's outer contour and connects consecutive adjacent effective fusion units to form the main device area. The system reads the visible light grayscale values of adjacent coordinate units line by line within the main body area of the device. When the grayscale difference exceeds the preset grayscale difference threshold, the corresponding coordinate unit is marked as an edge unit and connected to generate a visible light edge field. The system also reads the infrared temperature values of adjacent coordinate units line by line within the main body area of the device. When the temperature difference exceeds the preset temperature difference threshold, the corresponding coordinate unit is marked as a temperature gradient unit and connected to generate an infrared temperature gradient field.
[0030] In anomaly modeling, the system reads the main body region of the equipment, the visible light edge field, and the infrared temperature gradient field, and calls the prior template of the equipment structure corresponding to the target equipment number. The prior template of the equipment structure pre-stores the equipment outline region, axial arrangement region, and boundary constraint region. The system maps these structural regions to the corresponding coordinate positions of the main body region of the equipment and reads the edge marker values and infrared temperature values in the main body region units. When certain coordinate units simultaneously satisfy the edge continuity condition and the temperature change condition, the system adds them to the anomaly candidate unit set; when a candidate unit falls within the normal structural boundary and does not form a grayscale abrupt change or temperature abrupt change, the system deletes it from the anomaly candidate unit set, thereby reducing the possibility of normal metal outlines, flange edges, and sleeve structure lines being misjudged as anomaly boundaries.
[0031] In this embodiment, the system constructs a Mumford-Shah anomaly segmentation model that embeds the coupling relationship between device structure priors and temperature gradients. A discrete solution domain is constructed using effective fusion units within the main device area. The visible light grayscale value, infrared temperature value, and edge marker value of each coordinate unit are written into the model parameter area. The device structure prior template is rasterized to obtain a structural mask and a sequence of structural boundary points. For candidate boundary segments, the system calculates the distance between the boundary point and the nearest structural boundary point, as well as the angle between their tangential directions. When the distance is less than a preset pixel distance and the angle is less than a preset angle, the candidate boundary point is written into the structural counter-evidence set. When the proportion of points in a candidate boundary segment falling into the structural counter-evidence set exceeds a preset proportion, the entire candidate boundary segment is deleted. For the retained candidate boundary segments, the system reads the infrared temperature values of sampling points on both sides along the boundary normal direction and performs cross-traversal consistency verification in conjunction with the infrared temperature gradient field. When a temperature gradient exists at a boundary point and the temperature difference between the two sides exceeds a preset temperature difference threshold, the boundary point is written into the infrared crossing set; when the length of the chain of points that continuously meet the infrared crossing conditions exceeds a preset length, the corresponding boundary segment is written into the forced break set, thereby forming the initial solution of the abnormal boundary.
[0032] To improve model solution efficiency, the system employs the Split Bregman algorithm to split variables in the Mumford-Shah anomaly segmentation model, writing the region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables into their respective buffers. In each iteration, the system fixes some variables and alternately updates others. For the region fitting subproblem, it updates the region function; for the boundary regularization subproblem, it generates intermediate boundary variables and projects boundary variables from the forced fracture set to mandatory boundaries, while projecting boundary variables from the structural proof set to non-boundaries. The system also writes structural constraints and temperature coupling constraints into two gated channels. When a boundary point triggers a structural proof condition, the structural channel is updated; when a boundary point does not meet the temperature crossing condition, the temperature channel is updated. This avoids invalid expansion, drift, or fracture of the boundaries during iteration.
[0033] In the abnormal boundary convergence processing, the system connects the boundary coordinates in the current boundary table into boundary chains according to their coordinate adjacency relationships, and performs standardization processing on the boundary chains, including rearranging the boundary chain order according to a fixed starting point, deleting duplicate boundary points, and filling single-point break gaps. The system generates a boundary topological fingerprint based on the standard boundary chains. This fingerprint consists of the boundary chain length, endpoint coordinate sequence, and closure state code, used to determine the topological stability of the boundary chains during the iteration process. When the newly generated standard boundary chain does not cover the coordinates corresponding to the forced break set, the system rolls back the current boundary table to the previous round of standard boundary chains and only allows deletion or connection operations on the coordinates corresponding to the optional break set. Once the boundary topological fingerprint remains consistent and the convergence determination quantity meets a preset threshold, the system outputs the abnormal boundary convergence result.
[0034] During continuous inspection, the system reads the convergence results of abnormal boundaries and abnormal region masks from adjacent inspection frames, and calculates the boundary position offset, abnormal area change, and edge intensity change between adjacent frames. The system uses three consecutive frames as a gating window to perform consistency checks on these three indicators. When an abnormal region appears only in a single frame and no continuous boundary is formed in adjacent frames, the system classifies it as a jump boundary and generates a predicted boundary based on the stable abnormal boundary of the previous frame, thus cropping the abnormal region mask of the current frame. If the boundary segment identifier of the candidate stable region is inconsistent with the stable abnormal boundary of the previous frame, the system rolls back the stable abnormal region of the current frame to the area enclosed by the translated stable abnormal boundary of the previous frame, thereby suppressing false triggers caused by single-frame noise, reflections, and pose disturbances.
[0035] In the coordinated monitoring process, the system reads the coordinates of stable abnormal regions and their corresponding abnormal boundaries, counts the number of pixels within the stable abnormal region as the abnormal area, calculates the turning angles of adjacent boundary segments sequentially along the abnormal boundary coordinates and accumulates the boundary curvature, reads the visible light edge response values at the abnormal boundary coordinates and sums them to obtain the visible light edge intensity, and reads the infrared temperature gradient values within the stable abnormal region and sums them to obtain the infrared temperature gradient. Before calculating the above indicators, the system first deletes boundary points falling into the structural counter-evidence set and removes temperature statistics points that do not meet the infrared crossing conditions. Then, it converts the area, curvature, edge intensity, and temperature gradient into proportional values according to a fixed reference interval and generates an abnormal boundary coordinated energy value according to preset weights. When the coordinated energy value reaches the first threshold, the system triggers a monitoring alarm and starts video transmission; when it reaches the second threshold, the system generates an inspection path adjustment instruction to guide the inspection robot to re-inspect the current stable abnormal region; when it reaches the third threshold, the system generates a maintenance work order and writes the abnormal boundary coordinates, abnormal area, boundary curvature, and infrared temperature gradient into the work order content.
[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. An industrial vision-based autonomous inspection monitoring system for a substation, characterized in that, include: The inspection frame construction module collects visible light images, infrared thermal images, inspection pose data, and acquisition time of the target device, and constructs a multimodal inspection frame sequence for the same device object; The coordinate fusion module maps visible light images and infrared thermal images to a unified equipment coordinate domain based on the inspection pose data, generating the main equipment area, visible light edge field, and infrared temperature gradient field. The anomaly modeling module receives the main area of the device, the visible light edge field, and the infrared temperature gradient field, constructs a Mumford-Shah anomaly segmentation model that embeds the device structure prior and the temperature gradient coupling, and generates the initial solution of the anomaly boundary. The splitting solution module uses the Split Bregman algorithm to split variables in the Mumford-Shah anomaly segmentation model, generating region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables. The iterative convergence module alternately updates the region fitting variables, boundary regularization variables, temperature coupling variables, and structural constraint variables based on the initial solution of the anomaly boundary to generate the anomaly boundary convergence result. The timing correction module continuously corrects the convergence results of the abnormal boundary based on the boundary position offset, abnormal area change and edge intensity change of adjacent frames in the multimodal inspection frame sequence, and generates a stable abnormal region. The linkage monitoring module generates anomaly boundary linkage energy values based on the area, boundary curvature, visible light edge intensity, and infrared temperature gradient of the stable anomaly area. It then triggers monitoring alarms, video transmission, inspection path adjustment, and maintenance work order generation based on the anomaly boundary linkage energy values.
2. The industrial vision based substation autonomous inspection monitoring system according to claim 1, wherein, Constructing a multimodal inspection frame sequence includes: After the inspection equipment reaches the preset shooting point corresponding to the target equipment, it reads the target equipment number, the spatial coordinates of the inspection equipment, the attitude angle, the lens orientation and the current acquisition time, and starts the visible light camera and the infrared thermal imager respectively under the same trigger signal. The visible light image and infrared thermal image acquired for the same trigger signal are written into the same buffer unit, and the target device number, acquisition time, spatial coordinates, attitude angle and lens orientation are written into the frame header field of the buffer unit to form a single multimodal inspection frame. According to the ascending order of acquisition time, single multimodal inspection frames with the same target device number are written into the same device frame queue, and single multimodal inspection frames with missing target device number, missing visible light image, missing infrared thermal image or missing inspection pose data are removed. Calculate the time difference and pose difference between two adjacent single multimodal inspection frames in the device frame queue; When the time difference is within the preset time range and the pose difference is within the preset pose range, the corresponding single multimodal inspection frame is retained; when the time difference exceeds the preset time range or the pose difference exceeds the preset pose range, the corresponding single multimodal inspection frame is marked as a discontinuous frame. The continuously retained single multimodal inspection frames are sorted according to the acquisition time to generate a multimodal inspection frame sequence corresponding to the target device, and the discontinuous frames are output as sequence segmentation markers.
3. The industrial vision based substation autonomous inspection monitoring system according to claim 1, wherein, The main body area of the generating device, the visible light edge field, and the infrared temperature gradient field include: Read a single multimodal inspection frame from the multimodal inspection frame sequence, parse the target device number, spatial coordinates, attitude angle, lens orientation and acquisition time in the frame header field, and read the visible light image and infrared thermal image in the same buffer unit. Call the device reference coordinate template corresponding to the target device number, and project the pixels in the visible light image onto the device reference coordinate template one by one based on the spatial coordinates, attitude angle and lens orientation, and record the coordinate position of each pixel in the device reference coordinate template; The temperature points in the infrared thermal image are projected onto the device reference coordinate template one by one using the same projection method, and the visible light pixel value and infrared temperature value that fall into the same coordinate position of the template after projection are written into the same coordinate unit. Traverse the coordinate cells in the device reference coordinate template, mark the coordinate cells that simultaneously write visible light pixel values and infrared temperature values as effective fusion cells, mark the coordinate cells that lack visible light pixel values or infrared temperature values as empty cells, and retain the empty mark for empty cells. Based on the pre-stored equipment outer contour coordinate range in the equipment reference coordinate template, read the effective fusion units within the target equipment outer contour and connect consecutive adjacent effective fusion units to form the main body area of the equipment. The visible light grayscale values of adjacent coordinate units are read line by line within the main area of the device. When the grayscale difference between adjacent coordinate units is greater than the preset grayscale difference threshold, the corresponding coordinate unit is marked as an edge unit, and the edge units are connected according to the coordinate adjacency relationship to generate a visible light edge field. The infrared temperature values of adjacent coordinate units are read row by row within the main body area of the device. When the temperature difference between adjacent coordinate units is greater than the preset temperature difference threshold, the corresponding coordinate unit is marked as a temperature gradient unit, and the temperature gradient units are connected according to the coordinate adjacency relationship to generate an infrared temperature gradient field. The main body area of the equipment, the visible light edge field, and the infrared temperature gradient field are bound to the single multimodal inspection frame number.
4. The industrial vision-based substation autonomous inspection monitoring system of claim 1, wherein, Generating initial solutions for anomalous boundaries includes: Read the main body area of the device, the visible light edge field and the infrared temperature gradient field, and write them into the same anomaly modeling cache area according to the single multimodal inspection frame number; Call the prior template of the equipment structure corresponding to the target equipment number, and map the equipment outline area, axial arrangement area and boundary constraint area in the prior template of the equipment structure to the coordinate position corresponding to the equipment body area; Traverse the coordinate cells in the main body area of the device, mark the coordinate cells located within the device outline area as main body area cells, and mark the coordinate cells located outside the device outline area as background area cells, and retain the visible light grayscale value, edge mark value and infrared temperature value corresponding to the main body area cells. Read the edge marker values and infrared temperature values in the main region unit, and write the coordinate units that simultaneously meet the edge continuity condition and temperature change condition into the abnormal candidate unit set. Based on the axial arrangement region and the boundary constraint region, the abnormal candidate unit set is compared in position, and the coordinate units that fall within the normal structural boundary of the equipment and do not form gray-scale abrupt changes or temperature abrupt changes are deleted from the abnormal candidate unit set. The main equipment region, the set of candidate anomalies, the visible light edge field, the infrared temperature gradient field, and the prior template of the equipment structure are used as the input data of the Mumford-Shah anomaly segmentation model, and the set of anomaly boundary coordinates and the anomaly region mask are used as the output data of the Mumford-Shah anomaly segmentation model. Based on the set of anomaly boundary coordinates output by the Mumford-Shah anomaly segmentation model, the coordinates of adjacent anomaly boundaries in the anomaly region mask are connected to generate an initial solution for the anomaly boundaries, and the initial solution for the anomaly boundaries is output.
5. The industrial vision-based substation autonomous inspection monitoring system of claim 4, wherein, The Mumford-Shah anomaly segmentation model includes: read the device body region, the abnormal candidate unit set, the visible light edge field, the infrared temperature gradient field and the device structure prior template, and construct a discrete solution domain with the effective fusion units in the device body region , the visible light gray value of each coordinate unit in the visible light image is recorded as , the infrared temperature value is recorded as , the edge mark of the visible light edge field is recorded as , the infrared temperature value is recorded as , wherein is the coordinate unit index; The equipment structure prior template is in Upper rasterization yields the structural mask. With structural boundary point sequence And generate the structural tangential direction for each structural boundary point according to the order of adjacent points in the structural boundary point sequence. At the same time, the points adjacent to the structural boundary points in the normal direction will be... Coordinate cells are written into the structural neighborhood table; Candidate boundary units are read from the set of abnormal candidate units and connected according to their coordinate adjacency to form a set of candidate boundary segments. And calculate the visible light gradient direction at each point within each candidate boundary segment. and corresponding normal direction ,in Take and Vertical unit direction; right For each candidate boundary segment, a structural proof of contradiction is performed point by point. Specifically, the structural neighborhood table is searched for points that are related to the candidate boundary point. Nearest structural boundary point ,when arrive When the distance between the candidate boundary point and the structure is less than the preset pixel distance, and the angle between the tangent direction of the candidate boundary point and the tangent direction of the structure is less than the preset angle, the candidate boundary point is added to the structure counter-evidence set. For each candidate boundary segment, the statistics of falling into the boundary are calculated. The percentage of points is determined by the threshold. When the percentage exceeds a preset ratio, the entire candidate boundary segment is written into the structural proof boundary segment table and then... Delete; denote the set of candidate boundary segments that have not been deleted as the set of allowable structural boundaries. and will As the sole candidate source for subsequent fracture sets; Set of allowable boundaries for structures Perform infrared penetration verification, specifically: Each candidate boundary point Along its normal direction Take two sampling points with a fixed offset distance on both sides respectively. and Read and And calculate the temperature difference Simultaneously read the infrared temperature gradient field at gradient label at ; For each candidate boundary point Perform consistency determination during the traversal, specifically: when It is a temperature gradient unit and When the preset temperature difference threshold is exceeded, Write to infrared cross-traverse set For each candidate boundary segment, the number of segments falling within it is statistically analyzed. The length of the continuous point chain is set. When the length of the continuous point chain exceeds the preset length, the boundary segment is added to the forced break set. ; The set of allowed boundary conditions for structures that have not entered However, the remaining boundary segments are written into the optional fracture set. And establish constraints: the break set must contain Furthermore, it must not contain any boundary segment from the structural proof boundary segment table; exist The initial markers for the internal region division are established, specifically: using a forced fracture set. As the initial fracture boundary, Divided into two initial regions on either side of the fracture boundary and Calculate separately and Inside The region grayscale representative value is written into the region fitting table; The variables and interfaces for constructing the Mumford-Shah anomaly segmentation model are as follows: Specifically, defining the region function. exist The upper segment is smooth, and the fracture set is defined. for The set of boundary coordinates within; configure the model input interface to receive... , , , , Structural mask Forced fracture set With optional fracture set Configure the model output interface to output the set of anomaly boundary coordinates. Masking of abnormal regions; The energy of Mumford-Shah anomaly segmentation is defined as a combination of a region fitting term, a region smoothing term, and a fracture length term, and the boundary gating relationship obtained is solidified in the fracture set constraint. The energy is expressed as: ; in, satisfy ,and ; Will Structural mask , structural tangential direction Structural allowable boundary set Forced fracture set Optional fracture set Region fitting table and energy Write the parameters into the model parameter area to form the Mumford-Shah anomaly segmentation model, and output the set of anomaly boundary coordinates and anomaly region mask.
6. The substation autonomous inspection and monitoring system based on industrial vision according to claim 1, characterized in that, The variables generated for region fitting, boundary regularization, temperature coupling, and structural constraint include: Read the Mumford-Shah anomaly segmentation model and extract the discrete solution domain from the model parameter region. Visible light grayscale field Region functions Candidate fracture sets , Structural proof set Infrared Crossing Collection and will Initialize to the initial region partitioning results corresponding to the region fitting table; exist Establish a discrete gradient operator with divergence operator The gradient variable corresponding to the region smoothing term is introduced into the auxiliary variable. Introduce the indicator variable of the fracture boundary into the boundary variable. Structural constraints and temperature travel constraints are introduced into the constraint variables. ,in Record whether the coordinate unit satisfies the gated state of non-structural proof by contradiction and infrared crossing is valid; The solution to the Mumford-Shah anomaly segmentation model is incorporated into the Split Bregman framework to construct split consistency constraints: And write the boundary variable constraints: , At the same time, the temperature crossing constraint is written: when The corresponding coordinate unit must belong to ; for constraints Introducing Bregman variables Introducing Bregman variables for boundary constraints ; In each iteration, the boundary variable update is defined as a two-stage process of solving first and then projecting, specifically: first in... Obtain intermediate boundary variables Then Project onto the feasible set that satisfies the set constraints. The projection rule is: project all... Set the corresponding boundary variables to mandatory boundaries, and select all... The corresponding boundary variables are set to non-boundary, and the remaining boundary variables are restricted to... Within the corresponding coordinate set, a set of coordinates that satisfies the hard constraints is formed. The projection result is then written to the boundary regularization variable cache. The solution process is broken down into four subproblems and updated alternately in a fixed order: For the region fitting subproblem, fix , , , and Update the region function And write it into the region fitting variable table; For gradient-consistent subproblems, fix Compared with other variables, based on and Difference update And write it into the region smoothing variable table; For the boundary regular subproblem, fix , Together with the other variables, generate intermediate boundary variables. And perform the above projection solidification update to obtain And write it into the boundary regularization variable table; For constrained gating subproblems, fixed , , With the other variables, prove the set by contradiction based on the structure. With infrared crossing collection Determine the boundary coordinates point by point to generate And the boundary points that do not satisfy the non-structural proof by contradiction and for which infrared crossing is valid are removed from... Delete the corresponding coordinate set and write the updated boundary coordinate set back to the boundary regularization variable table; Structural constraints and temperature coupling constraints are written into two separate gated channels and the Bregman variables are updated accordingly. Specifically, when a boundary point triggers a structural contradiction condition, only the Bregman variable is updated. The structural channels remain unchanged while the temperature channels remain constant; when the boundary point triggers a temperature crossing condition that does not hold, only the structure is updated. The temperature channel is updated while the structural channel remains unchanged; when the boundary point simultaneously satisfies both types of gating conditions, the two channels are updated synchronously; and the update results of the two channels are written to the boundary Bregman variable buffer. After each round of four subproblems is completed, the Bregman variable is updated according to the following rules: , And calculate the energy convergence criterion value, which is expressed as: ; in, Indicates the first The abnormal segmentation energy value corresponding to the round iteration; when The iteration stops when the energy is less than a preset energy threshold and the boundary coordinate set remains unchanged for two consecutive iterations, and the final region fitting variables are output. Regional smoothing variables Boundary regularization variables and gated constraint variables ; Boundary regularization variables The corresponding coordinate set is written into the abnormal boundary convergence set, and the region fitting variables are... The corresponding region divisions are written into the region convergence table, and the abnormal boundary convergence set and the region convergence table are output.
7. The substation autonomous inspection and monitoring system based on industrial vision according to claim 1, characterized in that, The convergence results for generating anomalous boundaries include: Read the initial solution of the outlier boundary and read the region fitting variables. Boundary regularization variables Temperature coupling variables and structural constraint variables Write the boundary coordinates of the initial solution of the abnormal boundary into the current boundary table. Write to the current table in the region; The current boundary table is connected into a boundary chain according to the coordinate adjacency relationship, and the boundary chain is standardized. The standardization process includes rearranging the boundary chain order according to the fixed starting point rule, deleting duplicate boundary points, and filling the gaps of single-point breaks to obtain a standard boundary chain. Boundary topology fingerprint locking generates boundary topology fingerprints for standard boundary chains. The boundary topological fingerprint consists of the boundary chain length, endpoint coordinate sequence, and closure state encoding, and... Write to the fingerprint history table; Perform alternating updates in a fixed order: based on standard boundary chains and Update the current table of the region According to the updated In the boundary normal neighborhood, recalculate the gray-scale difference and temperature difference on both sides of the boundary, write back the temperature coupling variable and update the structural constraint variable synchronously; delete the boundary points that violate the gating condition from the current boundary table based on the updated constraint variable, and reconnect the remaining boundary points to generate a new standard boundary chain; When the newly generated standard boundary chain does not cover the coordinates corresponding to the forced break set, the current boundary table is rolled back to the previous round of standard boundary chain, and deletion or connection operations are only allowed for the coordinates corresponding to the optional break set. After forming a boundary chain that covers the forced break set, the next round of update will begin. After each round of updates, the convergence criterion is calculated and written into the convergence record table. The convergence criterion is represented as: ; in, Indicates the first Round region fitting results, Indicates the first Wheel boundary topological fingerprint, This indicates a fingerprint inconsistency indicator. when When the preset threshold is met and the boundary topology fingerprint remains consistent within a preset number of consecutive iterations, the abnormal boundary convergence result of the corresponding standard boundary chain is output, and the abnormal boundary convergence result is output along with the abnormal region mask corresponding to the current region table.
8. The substation autonomous inspection and monitoring system based on industrial vision according to claim 1, characterized in that, The generation of stable anomalous regions includes: Read the multimodal inspection frame sequence, and read the convergence results of the abnormal boundary and the abnormal region mask of each inspection frame. Write them into the frame boundary table and the frame region table according to the frame number. For the Frame and the The frame boundary table is used to calculate the boundary centroid coordinate difference to obtain the boundary position offset. The change in abnormal area is obtained by statistically analyzing the difference in the number of pixels in abnormal regions in the frame region table. ; Along the first Frame and the The edge response values of the visible light edge field are read point by point from the boundary coordinates of the frames and summed to obtain the edge intensities of the two frames, and the change in edge intensity is calculated. ; With three consecutive frames As gated windows, respectively for , , Perform threshold determination, count the number of indicators that meet the threshold, and when the number of indicators that meet the threshold in the determination of adjacent frames in the same frame reaches a preset number, then the [frame name] will be [set to a certain threshold]. Frame boundaries are determined to be continuous boundaries; otherwise, the first... Frame boundaries are identified as transition boundaries; When the When the frame boundary is determined to be a continuous boundary, the first... The abnormal region mask of the frame is directly written into the stable frame table; when the first frame... When a frame boundary is determined to be a jump boundary, the first frame in the stable frame table is read. Frame-stable anomalous boundaries are translated along the direction of boundary centroid offset. Generate a prediction boundary, and use the region enclosed by the prediction boundary to evaluate the first... The frame anomaly region mask performs cropping, retaining anomalous pixels within the predicted boundary area to form candidate stable regions; For each frame of stable anomaly boundaries, a boundary segment identifier is generated. The boundary segment identifier consists of the boundary closure state and the number of boundary coordinates encoded; when the... The boundary segment identifier corresponding to the candidate stable region of the frame and the first When the boundary segment identifiers of frame stability anomaly boundaries are inconsistent, the first segment will be... Frame instability anomaly region rollback to the first Frame stability anomaly boundary The enclosed area after translation, and the first The original anomaly boundary convergence result is marked as a discontinuous boundary. The first The final stable abnormal region of the frame is written into the stable region sequence and output.
