Power inspection multi-modal data analysis method and system based on deep learning

Abstract

The application discloses a power inspection multi-modal data analysis method and system based on deep learning, and belongs to the technical field of power equipment inspection. The method comprises the following steps: acquiring a space registration transformation matrix of a visible light camera and an infrared thermal imager which is calibrated in advance; generating a time-space aligned multi-modal image pair; processing the time-space aligned multi-modal image pair, monitoring power equipment state information output by the visible light image, and determining temperature change characteristics of a heating area; obtaining a credibility score of a state jump event; and confirming the state jump event as a real state change and generating a state change record. The application verifies the state change recognized by the visible light by introducing temperature data collected by the infrared thermal imager, and solves the problem that in the prior art, only the visible light image is used for state recognition, which leads to detection jitter, state misjudgment and invalid alarm flooding under light interference.

Description

Technical Field

[0001] This invention relates to the field of power equipment inspection technology, and in particular to a method and system for multimodal data analysis of power inspection based on deep learning. Background Technology

[0002] With the continuous expansion of the power grid and the increasing demands for operation and maintenance, power grid inspection is gradually shifting from traditional manual inspection to intelligent inspection based on deep learning. Utilizing drones and robots equipped with sensors such as visible light cameras and infrared thermal imagers, combined with deep learning target detection models, to automatically identify and analyze the status of power equipment has become an important means to improve inspection efficiency and ensure power grid safety.

[0003] Existing power line inspection systems typically acquire images of equipment using visible light cameras and run pre-trained target detection models to identify equipment status, such as determining the open / closed position of disconnect switches and reading instrument readings. However, in actual outdoor substation or transmission line inspection scenarios, lighting conditions are extremely unstable. Backlighting in the early morning, strong direct sunlight at noon causing reflections on the surface of metal equipment, and moving light and shadows caused by inspection vehicle lights at night can all cause vision-based target detection models to exhibit a "flickering" phenomenon. Specifically, the same device may be stably identified in the first few frames, but when the angle of light changes abruptly, causing glare, the model may suddenly lose the target, resulting in the target bounding box shaking violently or disappearing briefly.

[0004] In existing technologies, time-series state machines relying solely on visible light images cannot distinguish between genuine changes in device state and light interference when receiving jittery target frames. They are prone to misinterpreting brief loss and re-capture of targets as a sudden change in device state, thus triggering incorrect state change records. When the system receives a large amount of repetitive state change data caused by light and shadow interference within a short period, it generates a massive number of invalid alarms. This not only undermines the trust of maintenance personnel in the alarm system but, more importantly, masks genuine state changes caused by equipment overheating or mechanical failures, posing a serious threat to the safe operation of the power grid. Summary of the Invention

[0005] This application provides a deep learning-based multimodal data analysis method and system for power inspection, which solves the problems in the prior art where relying solely on visible light images for state recognition easily leads to detection jitter, misjudgment of state, and rampant invalid alarms under light interference. It achieves the technical effect of accurately identifying real equipment state changes and effectively filtering out false alarms caused by environmental interference based on multimodal data consistency verification.

[0006] This application provides a deep learning-based multimodal data analysis method for power line inspection, including: obtaining a pre-calibrated spatial registration transformation matrix between a visible light camera and an infrared thermal imager; Acquire visible light image sequences and infrared thermal image sequences at the same timestamp to generate spatiotemporally aligned multimodal image pairs; The spatiotemporally aligned multimodal image pairs are processed to identify the status information of power equipment in visible light images and the heat-generating areas and temperature distribution data in infrared thermal images, respectively. Monitor the status information of power equipment output by visible light images. When a status change occurs, store the status change event, equipment identifier, timestamp, and the corresponding index of visible light image sequence and infrared thermal image sequence into the status change candidate queue. Based on the timestamp of the state transition event, extract the infrared thermal image sequence within the time window to determine the temperature change characteristics of the heat-generating area; The credibility score of the state transition event is obtained by comparing the temperature change characteristics with the expected correspondence between the state changes and temperature changes of power equipment in the physical consistency rule base. Based on the comparison between the credibility score and the preset credibility threshold, the state transition event is confirmed as a real state change and a state change record is generated, or the state transition event is discarded.

[0007] Furthermore, the steps of acquiring visible light image sequences and infrared thermal image sequences at the same timestamp to generate spatiotemporally aligned multimodal image pairs include: Analyze the intrinsic parameter matrix of a visible light camera and an infrared thermal imager; Extract the rotation matrix and translation vector from the spatial registration transformation matrix; The coordinates of each infrared pixel in the infrared thermal image are projected onto a three-dimensional spatial coordinate system to obtain the three-dimensional spatial coordinates. By using rotation matrices and translation vectors to transform the three-dimensional spatial coordinates, the three-dimensional coordinates of the target in the visible light camera coordinate system are obtained. The target's three-dimensional coordinates, combined with the intrinsic parameter matrix of the visible light camera, are projected onto the pixel plane of the visible light image to obtain the mapped pixel coordinates. Calculate the distance between the mapped pixel coordinates and the discrete pixel grid points in the visible light image; The grayscale values ​​of the mapped pixel coordinates are resampled based on the distance using a bilinear interpolation algorithm. The resampled infrared thermal image data is seamlessly stitched with the visible light image data at the same timestamp by channel to generate a spatiotemporally aligned multimodal image pair data tensor.

[0008] Furthermore, the steps for processing spatiotemporally aligned multimodal image pairs to identify power equipment status information in visible light images and heat-generating areas and temperature distribution data in infrared thermal images include: Input the spatiotemporally aligned multimodal image pair data tensor into the edge computing unit; Extract visible light feature maps from the visible light detection branch of the edge computing unit, and output the bounding box coordinates of the visible light target and the first classification confidence and first state identification label of the power equipment; Infrared feature maps are extracted from the infrared detection branch of the edge computing unit, and the coordinates of the infrared target bounding box and the second classification confidence of the heat-generating area are output. Calculate the intersection-union ratio (IoU) of the bounding box coordinates of the visible light target and the bounding box coordinates of the infrared target; When the crossover ratio is greater than the preset spatial overlap threshold, it is confirmed that the visible light target bounding box and the infrared target bounding box point to the same power equipment. Assign a unique device identifier to the bounding boxes of visible light targets and infrared targets pointing to the same power equipment; Extract the absolute temperature values ​​of all pixels within the infrared target bounding box and construct a temperature distribution data matrix; The first state identification tag is output as the power equipment state information.

[0009] Furthermore, when a state transition occurs, the step of storing the state transition event, device identifier, timestamp, and the corresponding indexes of the visible light image sequence and infrared thermal image sequence into the state change candidate queue includes: Read the power equipment status information associated with the device identifier sequentially according to time order; Set a status observation time sliding window to compare the power equipment status information at the current timestamp with the power equipment status information at the previous timestamp. When the power equipment status information at the current timestamp is inconsistent with the power equipment status information at the previous timestamp, and the number of frames in which the inconsistent state persists within the state observation time window is less than the preset anti-jitter frame number threshold, a state transition is determined to have occurred, and a state transition event is generated. Establish a blank candidate queue data structure for state changes; Write the state label before and after the state transition event into the fields of the state change candidate queue data structure; Extract the current timestamp corresponding to the generated state transition event; Locate the first physical address of the multimodal image pair data tensor corresponding to the current timestamp in the storage area, and the second physical address of the multimodal image pair data tensor of the preceding and following frames in the storage area; The first physical address and the second physical address are used as indexes and stored in the state change candidate queue data structure.

[0010] Furthermore, the steps of extracting the infrared thermal image sequence within the time window and determining the temperature change characteristics of the heat-generating area include: Based on the first physical address and the second physical address, retrieve the temperature distribution data matrix corresponding to the infrared thermal image sequence within the time window; Calculate the highest and average temperature values ​​of the temperature distribution data matrix for each frame within the time window; The highest temperature values ​​are combined into a one-dimensional highest temperature sequence according to the order of timestamps. The average temperature values ​​are combined into a one-dimensional average temperature sequence. ; Let the start time of the time window be... The end time is ; Calculate the slope of the highest temperature change within the time window for the highest temperature sequence: ; Calculate the average temperature change of the average temperature series within the time window: ; Extract the instantaneous highest temperature value corresponding to the timestamp of the state transition event; The slope of the highest temperature change, the amplitude of the average temperature change, and the instantaneous highest temperature value are concatenated according to a preset feature vector format to generate a temperature change feature vector of the heating area.

[0011] Furthermore, the steps to obtain a confidence score for a state transition event include: Read the standard maximum temperature slope range, standard average temperature amplitude range, and standard instantaneous high temperature upper limit value set for different types of power equipment in the physical consistency rule base; Extract the power equipment type corresponding to the device identifier in the state transition event; Based on the type of power equipment, the corresponding target standard maximum temperature slope range, target standard average temperature range, and target standard instantaneous high temperature upper limit value are retrieved from the physical consistency rule base. Calculate the first absolute error value of the slope of the maximum temperature change that deviates from the target standard maximum temperature slope range; Calculate the second absolute error value of the deviation of the average temperature change from the target standard average temperature range; Calculate the third absolute error value between the instantaneous maximum temperature value and the target standard instantaneous high temperature limit value; Preset reciprocal weighting coefficients are assigned to the first absolute error value, the second absolute error value, and the third absolute error value, respectively; The credibility score of the state transition event is obtained by multiplying the first absolute error value, the second absolute error value, and the third absolute error value by their respective reciprocal weight coefficients and then summing them.

[0012] Furthermore, the state transition event can be confirmed as a real state change and a state change record can be generated, or the state transition event can be discarded: Set a high confidence threshold and a low confidence threshold, with the high confidence threshold being greater than the low confidence threshold; The credibility score of the state transition event is compared with the high credibility threshold and the low credibility threshold in turn. If the credibility score is greater than or equal to the high credibility threshold, then the temperature change characteristics of the state transition event are determined to conform to the physical heating law of this type of equipment. This state transition event has been confirmed as a real state change at the physical level. Package the device identifier, the state label after the state transition event, and the current timestamp to generate a standardized state change record file and write it to the backend database. If the credibility score is less than the low credibility threshold, the state transition event is determined to be a temporal state machine erroneous triggering event caused by the loss of visual target detection due to ambient light noise. The state transition event is marked as invalid in the state transition candidate queue, and the memory space occupied by the state transition event is discarded directly.

[0013] Furthermore, the steps of packaging the device identifier, the state label after the state transition event, and the current timestamp to generate a standardized state change log file and writing it to the backend database include: Retrieve all historical temperature change feature vectors of the same device identifier that were confirmed as real state changes within the historical operating cycle from the backend database. Calculate the Euclidean distance between the newly generated temperature change feature vector and all historical temperature change feature vectors; When the Euclidean distance is greater than the preset out-of-distribution drift threshold, the power equipment is determined to be in an unrecorded special operating environment. Extract the slope of the highest temperature change and the amplitude of the average temperature change under this special working condition; An adaptive smoothing filter algorithm is used to smooth the data of the slope of the highest temperature change and the amplitude of the average temperature change to eliminate transient noise interference. The slope of the highest temperature change after smoothing and the magnitude of the average temperature change are used as new expected correspondence parameters. Insert the new expected correspondence parameters into the physical consistency rule base and update the target standard maximum temperature slope range and target standard average temperature amplitude range for the type corresponding to the device identifier.

[0014] This application provides a deep learning-based multimodal data analysis system for power line inspection, used to implement a deep learning-based multimodal data analysis method for power line inspection, including: Matrix acquisition module, sequence acquisition module, image processing module, information monitoring module, feature determination module, score acquisition module, and record generation module; The matrix acquisition module is used to acquire the pre-calibrated spatial registration transformation matrix between the visible light camera and the infrared thermal imager. The sequence acquisition module is used to acquire visible light image sequences and infrared thermal image sequences at the same timestamp, and generate spatiotemporally aligned multimodal image pairs; The image processing module is used to process spatiotemporally aligned multimodal image pairs to identify power equipment status information in visible light images and heat-generating areas and temperature distribution data in infrared thermal images, respectively. The information monitoring module is used to monitor the status information of power equipment output by visible light images. When a status change occurs, the status change event, equipment identifier, timestamp, and the corresponding indexes of visible light image sequence and infrared thermal image sequence are stored in the status change candidate queue. The feature determination module is used to extract the infrared thermal image sequence within the time window based on the timestamp of the state transition event, and determine the temperature change characteristics of the heating area. The scoring acquisition module is used to compare the temperature change characteristics with the expected correspondence between the power equipment state changes and temperature changes in the physical consistency rule base to obtain the credibility score of the state jump event. The record generation module is used to confirm the state transition event as a real state change and generate a state change record based on the comparison result between the credibility score and the preset credibility threshold, or to discard the state transition event.

[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By introducing temperature data collected by an infrared thermal imager to verify the state changes of visible light recognition, when sudden changes in illumination cause target jitter or loss in visible light detection, the visual results are not directly accepted. Instead, the temperature change trend at the same moment is retrieved for comparison to determine whether the state has really changed, thereby effectively filtering out false state jumps caused by light and shadow interference.

[0016] Furthermore, when confirming state changes, the direction and magnitude of temperature changes are compared to see if they conform to the physical heating law of the equipment itself. This ensures that only state changes that truly conform to physical logic can pass the verification, avoiding false triggering of the timing state machine due to the temporary loss and re-capture of the target, and allowing the state judgment to return to the actual working condition of the equipment itself.

[0017] Furthermore, since all status change records reported to the backend have undergone dual verification of temperature change characteristics and physical rule base, the number of invalid alarms caused by ambient light noise has been reduced. Maintenance personnel are no longer disturbed by a large number of false alarms and can focus their energy on real equipment overheating or mechanical failures, avoiding the problem of real risks being overwhelmed and ensuring the reliability and safety of power grid inspection. Attached Figure Description

[0018] Figure 1 A flowchart of a deep learning-based multimodal data analysis method for power line inspection provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of a deep learning-based multimodal data analysis system for power line inspection provided in an embodiment of this application. Detailed Implementation

[0019] This application provides a deep learning-based multimodal data analysis method and system for power inspection, which solves the problems of detection jitter, misjudgment of status, and rampant invalid alarms caused by relying solely on visible light images for status recognition in the prior art under light interference. By introducing temperature data collected by an infrared thermal imager to verify the status changes identified by visible light, the extracted temperature change features are compared with the expected correspondence between equipment status changes and temperature changes in the physical consistency rule base. Only status changes that conform to the physical heating law are confirmed as real changes. This achieves the technical effect of accurately identifying real equipment status changes and effectively filtering out false alarms caused by environmental interference based on multimodal data consistency verification.

[0020] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0021] like Figure 1 The diagram shown is a flowchart of a deep learning-based multimodal data analysis method for power line inspection provided in this application embodiment. This method is applied to a deep learning-based multimodal data analysis system for power line inspection. The method includes the following steps: obtaining a pre-calibrated spatial registration transformation matrix between a visible light camera and an infrared thermal imager. Acquire visible light image sequences and infrared thermal image sequences at the same timestamp to generate spatiotemporally aligned multimodal image pairs; The spatiotemporally aligned multimodal image pairs are processed to identify the status information of power equipment in visible light images and the heat-generating areas and temperature distribution data in infrared thermal images, respectively. Monitor the status information of power equipment output by visible light images. When a status change occurs, store the status change event, equipment identifier, timestamp, and the corresponding index of visible light image sequence and infrared thermal image sequence into the status change candidate queue. Based on the timestamp of the state transition event, extract the infrared thermal image sequence within the time window to determine the temperature change characteristics of the heat-generating area; The credibility score of the state transition event is obtained by comparing the temperature change characteristics with the expected correspondence between the state changes and temperature changes of power equipment in the physical consistency rule base. Based on the comparison between the credibility score and the preset credibility threshold, the state transition event is confirmed as a real state change and a state change record is generated, or the state transition event is discarded.

[0022] Furthermore, the steps of acquiring visible light image sequences and infrared thermal image sequences at the same timestamp to generate spatiotemporally aligned multimodal image pairs include: Analyze the intrinsic parameter matrix of a visible light camera and an infrared thermal imager; Extract the rotation matrix and translation vector from the spatial registration transformation matrix; The coordinates of each infrared pixel in the infrared thermal image are projected onto a three-dimensional spatial coordinate system to obtain the three-dimensional spatial coordinates. By using rotation matrices and translation vectors to transform the three-dimensional spatial coordinates, the three-dimensional coordinates of the target in the visible light camera coordinate system are obtained. The target's three-dimensional coordinates, combined with the intrinsic parameter matrix of the visible light camera, are projected onto the pixel plane of the visible light image to obtain the mapped pixel coordinates. Calculate the distance between the mapped pixel coordinates and the discrete pixel grid points in the visible light image; The grayscale values ​​of the mapped pixel coordinates are resampled based on the distance using a bilinear interpolation algorithm. The resampled infrared thermal image data is seamlessly stitched with the visible light image data at the same timestamp by channel to generate a spatiotemporally aligned multimodal image pair data tensor.

[0023] The specific steps include: during the movement of the inspection robot, a synchronous trigger signal is used to cause the visible light camera and the infrared thermal imager to simultaneously acquire images at the same frame rate, thus obtaining visible light images at the same time $t$. and infrared thermal images Visible light images are matrices of brightness values, while infrared thermal images are matrices of temperature values ​​(unit: degrees Celsius).

[0024] Perform geometric correction on each pair of images: adjust the coordinates of each pixel in the infrared image. Using internal references The points are converted to 3D points in the camera coordinate system, then transformed back to the visible light camera coordinate system using a rotation matrix R and a translation vector t. Finally, the visible light intrinsic parameters are used. Projecting onto the visible light image plane yields the mapped coordinates. .because Typically not integer pixels, grayscale values ​​are sampled from the infrared image using bilinear interpolation to generate a new infrared image geometrically aligned with the visible light image. .Will and The data is stitched together along the channel dimension to form a six-channel multimodal image pair data tensor T.

[0025] Furthermore, the steps for processing spatiotemporally aligned multimodal image pairs to identify power equipment status information in visible light images and heat-generating areas and temperature distribution data in infrared thermal images include: Input the spatiotemporally aligned multimodal image pair data tensor into the edge computing unit; Extract visible light feature maps from the visible light detection branch of the edge computing unit, and output the bounding box coordinates of the visible light target and the first classification confidence and first state identification label of the power equipment; Infrared feature maps are extracted from the infrared detection branch of the edge computing unit, and the coordinates of the infrared target bounding box and the second classification confidence of the heat-generating area are output. Calculate the intersection-union ratio (IoU) of the bounding box coordinates of the visible light target and the bounding box coordinates of the infrared target; When the crossover ratio is greater than the preset spatial overlap threshold, it is confirmed that the visible light target bounding box and the infrared target bounding box point to the same power equipment. Assign a unique device identifier to the bounding boxes of visible light targets and infrared targets pointing to the same power equipment; Extract the absolute temperature values ​​of all pixels within the infrared target bounding box and construct a temperature distribution data matrix; The first state identification tag is output as the power equipment state information.

[0026] After the data tensor is input into the edge computing unit, visible light and infrared data are processed separately through a dual-branch network. The visible light detection branch outputs the device bounding box, classification confidence score, and status label, while the infrared detection branch outputs the heating area bounding box and classification confidence score. A key step involves calculating the intersection-union ratio (IUGR) of the bounding boxes output by the two branches. When this ratio exceeds a preset threshold, it is determined that both points to the same physical device, and a unique device identifier is assigned to this association pair. Subsequently, the temperature values ​​of all pixels within the infrared bounding box are extracted to construct a temperature distribution data matrix, and the status label identified by visible light is used as the current state of the device. This process unifies the identity of the same device in both modalities, allowing subsequent analysis to associate the "state" identified by visible light with the "temperature" measured by infrared light onto the same object. The spatial association method using IUGR is simple and effective, avoiding complex data matching logic and ensuring the accuracy of multimodal data fusion.

[0027] Furthermore, when a state transition occurs, the step of storing the state transition event, device identifier, timestamp, and the corresponding indexes of the visible light image sequence and infrared thermal image sequence into the state transition candidate queue includes: Read the power equipment status information associated with the device identifier sequentially according to time order; Set a status observation time sliding window to compare the power equipment status information at the current timestamp with the power equipment status information at the previous timestamp. When the power equipment status information at the current timestamp is inconsistent with the power equipment status information at the previous timestamp, and the number of frames that the inconsistent state lasts within the state observation time window is less than the preset anti-jitter frame number threshold, a state transition is determined to have occurred, and a state transition event is generated. Establish a blank candidate queue data structure for state changes; Write the state label before and after the state transition event into the fields of the state change candidate queue data structure; Extract the current timestamp corresponding to the generated state transition event; Locate the first physical address of the multimodal image pair data tensor corresponding to the current timestamp in the storage area, and the second physical address of the multimodal image pair data tensor of the preceding and following frames in the storage area; The first physical address and the second physical address are used as indexes and stored in the state change candidate queue data structure.

[0028] By monitoring the continuous status of each device chronologically, a state observation time window is set to compare the state information of adjacent timestamps. Only when a state change is detected, and the number of frames this change lasts within the window is less than a preset anti-jitter frame threshold, is it considered a valid state transition event and an event record is generated. This design introduces an anti-jitter mechanism that filters out transient state changes caused by detection jitter in a single frame or a very small number of frames, preventing these obviously unreasonable transitions from being directly sent to subsequent verification processes. Subsequently, the state labels before and after the transition, the device identifier, the timestamp, and the physical storage address of the corresponding multimodal data are all stored in a state change candidate queue. Storing the physical address rather than the data itself is an efficient memory management method, preserving the possibility of tracing back to the original data while avoiding unnecessary memory consumption. Through this step, potential state change events are temporarily "detained," awaiting more sufficient evidence for adjudication.

[0029] Furthermore, the steps of extracting the infrared thermal image sequence within the time window and determining the temperature change characteristics of the heat-generating area include: Based on the first physical address and the second physical address, retrieve the temperature distribution data matrix corresponding to the infrared thermal image sequence within the time window; Calculate the highest and average temperature values ​​of the temperature distribution data matrix for each frame within the time window; The highest temperature values ​​are combined into a one-dimensional highest temperature sequence according to the order of timestamps. The average temperature values ​​are combined into a one-dimensional average temperature sequence. ; Let the start time of the time window be... The end time is ; Calculate the slope of the highest temperature change within the time window for the highest temperature sequence: ; Calculate the average temperature change of the average temperature series within the time window: ; Extract the instantaneous highest temperature value corresponding to the timestamp of the state transition event; The slope of the highest temperature change, the amplitude of the average temperature change, and the instantaneous highest temperature value are concatenated according to a preset feature vector format to generate a temperature change feature vector of the heating area.

[0030] Furthermore, the steps to obtain a confidence score for a state transition event include: Read the standard maximum temperature slope range, standard average temperature amplitude range, and standard instantaneous high temperature upper limit value set for different types of power equipment in the physical consistency rule base; Extract the power equipment type corresponding to the device identifier in the state transition event; Based on the type of power equipment, the corresponding target standard maximum temperature slope range, target standard average temperature range, and target standard instantaneous high temperature upper limit value are retrieved from the physical consistency rule base. Calculate the first absolute error value of the slope of the maximum temperature change that deviates from the target standard maximum temperature slope range; Calculate the second absolute error value of the deviation of the average temperature change from the target standard average temperature range; Calculate the third absolute error value between the instantaneous maximum temperature value and the target standard instantaneous high temperature limit value; Preset reciprocal weighting coefficients are assigned to the first absolute error value, the second absolute error value, and the third absolute error value, respectively; The credibility score of the state transition event is obtained by multiplying the first absolute error value, the second absolute error value, and the third absolute error value by their respective reciprocal weight coefficients and then summing them.

[0031] Furthermore, the state transition event can be confirmed as a real state change and a state change record can be generated, or the state transition event can be discarded: Set a high confidence threshold and a low confidence threshold, with the high confidence threshold being greater than the low confidence threshold; The credibility score of the state transition event is compared with the high credibility threshold and the low credibility threshold in turn. If the credibility score is greater than or equal to the high credibility threshold, then the temperature change characteristics of the state transition event are determined to conform to the physical heating law of this type of equipment. This state transition event has been confirmed as a real state change at the physical level. Package the device identifier, the state label after the state transition event, and the current timestamp to generate a standardized state change record file and write it to the backend database. If the credibility score is less than the low credibility threshold, the state transition event is determined to be a temporal state machine erroneous triggering event caused by the loss of visual target detection due to ambient light noise. The state transition event is marked as invalid in the state transition candidate queue, and the memory space occupied by the state transition event is discarded directly.

[0032] Furthermore, the steps of packaging the device identifier, the state label after the state transition event, and the current timestamp to generate a standardized state change log file and writing it to the backend database include: Retrieve all historical temperature change feature vectors of the same device identifier that were confirmed as real state changes within the historical operating cycle from the backend database. Calculate the Euclidean distance between the newly generated temperature change feature vector and all historical temperature change feature vectors; When the Euclidean distance is greater than the preset out-of-distribution drift threshold, the power equipment is determined to be in an unrecorded special operating environment. Extract the slope of the highest temperature change and the amplitude of the average temperature change under this special working condition; An adaptive smoothing filter algorithm is used to smooth the data of the slope of the highest temperature change and the amplitude of the average temperature change to eliminate transient noise interference. The slope of the highest temperature change after smoothing and the magnitude of the average temperature change are used as new expected correspondence parameters. Insert the new expected correspondence parameters into the physical consistency rule base and update the target standard maximum temperature slope range and target standard average temperature amplitude range for the type corresponding to the device identifier.

[0033] like Figure 2 The diagram shown is a structural schematic of a deep learning-based multimodal data analysis system for power line inspection provided in this embodiment of the application. The deep learning-based multimodal data analysis system for power line inspection provided in this embodiment of the application includes: a matrix acquisition module, a sequence acquisition module, an image processing module, an information monitoring module, a feature determination module, a scoring acquisition module, and a record generation module. The matrix acquisition module is used to acquire the pre-calibrated spatial registration transformation matrix between the visible light camera and the infrared thermal imager. The sequence acquisition module is used to acquire visible light image sequences and infrared thermal image sequences at the same timestamp, and generate spatiotemporally aligned multimodal image pairs; The image processing module is used to process spatiotemporally aligned multimodal image pairs to identify power equipment status information in visible light images and heat-generating areas and temperature distribution data in infrared thermal images, respectively. The information monitoring module is used to monitor the status information of power equipment output by visible light images. When a status change occurs, the status change event, equipment identifier, timestamp, and the corresponding indexes of visible light image sequence and infrared thermal image sequence are stored in the status change candidate queue. The feature determination module is used to extract the infrared thermal image sequence within the time window based on the timestamp of the state transition event, and determine the temperature change characteristics of the heating area. The scoring acquisition module is used to compare the temperature change characteristics with the expected correspondence between the power equipment state changes and temperature changes in the physical consistency rule base to obtain the credibility score of the state jump event. The record generation module is used to confirm the state transition event as a real state change and generate a state change record based on the comparison result between the credibility score and the preset credibility threshold, or to discard the state transition event.

[0034] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0035] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0036] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0037] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0038] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0039] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based multimodal data analysis method for power line inspection, characterized in that, Includes the following steps: Obtain the pre-calibrated spatial registration transformation matrix between the visible light camera and the infrared thermal imager; Acquire visible light image sequences and infrared thermal image sequences at the same timestamp to generate spatiotemporally aligned multimodal image pairs; The spatiotemporally aligned multimodal image pairs are processed to identify the status information of power equipment in visible light images and the heat-generating areas and temperature distribution data in infrared thermal images, respectively. Monitor the status information of power equipment output by visible light images. When a status change occurs, store the status change event, equipment identifier, timestamp, and the corresponding index of visible light image sequence and infrared thermal image sequence into the status change candidate queue. Based on the timestamp of the state transition event, extract the infrared thermal image sequence within the time window to determine the temperature change characteristics of the heat-generating area; The credibility score of the state transition event is obtained by comparing the temperature change characteristics with the expected correspondence between the state changes and temperature changes of power equipment in the physical consistency rule base. Based on the comparison between the credibility score and the preset credibility threshold, the state transition event is confirmed as a real state change and a state change record is generated, or the state transition event is discarded.

2. The deep learning-based multimodal data analysis method for power line inspection as described in claim 1, characterized in that, The steps for acquiring visible light image sequences and infrared thermal image sequences at the same timestamp and generating spatiotemporally aligned multimodal image pairs include: Analyze the intrinsic parameter matrix of a visible light camera and an infrared thermal imager; Extract the rotation matrix and translation vector from the spatial registration transformation matrix; The coordinates of each infrared pixel in the infrared thermal image are projected onto a three-dimensional spatial coordinate system to obtain the three-dimensional spatial coordinates. By using rotation matrices and translation vectors to transform the three-dimensional spatial coordinates, the three-dimensional coordinates of the target in the visible light camera coordinate system are obtained. The target's three-dimensional coordinates, combined with the intrinsic parameter matrix of the visible light camera, are projected onto the pixel plane of the visible light image to obtain the mapped pixel coordinates. Calculate the distance between the mapped pixel coordinates and the discrete pixel grid points in the visible light image; The grayscale values ​​of the mapped pixel coordinates are resampled based on the distance using a bilinear interpolation algorithm. The resampled infrared thermal image data is seamlessly stitched with the visible light image data at the same timestamp by channel to generate a spatiotemporally aligned multimodal image pair data tensor.

3. The deep learning-based multimodal data analysis method for power line inspection as described in claim 1, characterized in that, The steps for processing spatiotemporally aligned multimodal image pairs to identify power equipment status information in visible light images and heat-generating areas and temperature distribution data in infrared thermal images include: Input the spatiotemporally aligned multimodal image pair data tensor into the edge computing unit; Extract visible light feature maps from the visible light detection branch of the edge computing unit, and output the bounding box coordinates of the visible light target and the first classification confidence and first state identification label of the power equipment; Infrared feature maps are extracted from the infrared detection branch of the edge computing unit, and the coordinates of the infrared target bounding box and the second classification confidence of the heat-generating area are output. Calculate the intersection-union ratio (IoU) of the bounding box coordinates of the visible light target and the bounding box coordinates of the infrared target; When the crossover ratio is greater than the preset spatial overlap threshold, it is confirmed that the visible light target bounding box and the infrared target bounding box point to the same power equipment. Assign a unique device identifier to the bounding boxes of visible light targets and infrared targets pointing to the same power equipment; Extract the absolute temperature values ​​of all pixels within the infrared target bounding box and construct a temperature distribution data matrix; The first state identification tag is output as the power equipment state information.

4. The deep learning-based multimodal data analysis method for power line inspection as described in claim 1, characterized in that, When a state transition occurs, the steps of storing the state transition event, device identifier, timestamp, and the corresponding indexes of the visible light image sequence and infrared thermal image sequence into the state transition candidate queue include: Read the power equipment status information associated with the device identifier sequentially according to time order; Set a status observation time sliding window to compare the power equipment status information at the current timestamp with the power equipment status information at the previous timestamp. When the power equipment status information at the current timestamp is inconsistent with the power equipment status information at the previous timestamp, and the number of frames in which the inconsistent state persists within the state observation time window is less than the preset anti-jitter frame number threshold, a state transition is determined to have occurred, and a state transition event is generated. Establish a blank candidate queue data structure for state changes; Write the state label before and after the state transition event into the fields of the state change candidate queue data structure; Extract the current timestamp corresponding to the generated state transition event; Locate the first physical address of the multimodal image pair data tensor corresponding to the current timestamp in the storage area, and the second physical address of the multimodal image pair data tensor of the preceding and following frames in the storage area; The first physical address and the second physical address are used as indexes and stored in the state change candidate queue data structure.

5. The deep learning-based multimodal data analysis method for power line inspection as described in claim 1, characterized in that, The steps for extracting infrared thermal image sequences within a time window and determining the temperature change characteristics of the heat-generating region include: Based on the first physical address and the second physical address, retrieve the temperature distribution data matrix corresponding to the infrared thermal image sequence within the time window; Calculate the highest and average temperature values ​​of the temperature distribution data matrix for each frame within the time window; The highest temperature values ​​are combined into a one-dimensional highest temperature sequence according to the order of timestamps. The average temperature values ​​are combined into a one-dimensional average temperature sequence. ; Let the start time of the time window be... The end time is ; Calculate the slope of the highest temperature change within the time window for the highest temperature sequence: ； Calculate the average temperature change of the average temperature series within the time window: ； Extract the instantaneous highest temperature value corresponding to the timestamp of the state transition event; The slope of the highest temperature change, the amplitude of the average temperature change, and the instantaneous highest temperature value are concatenated according to a preset feature vector format to generate a temperature change feature vector of the heating area.

6. The deep learning-based multimodal data analysis method for power line inspection as described in claim 1, characterized in that, The steps to obtain a confidence score for a state transition event include: Read the standard maximum temperature slope range, standard average temperature amplitude range, and standard instantaneous high temperature upper limit value set for different types of power equipment in the physical consistency rule base; Extract the power equipment type corresponding to the device identifier in the state transition event; Based on the type of power equipment, the corresponding target standard maximum temperature slope range, target standard average temperature range, and target standard instantaneous high temperature upper limit value are retrieved from the physical consistency rule base. Calculate the first absolute error value of the slope of the maximum temperature change that deviates from the target standard maximum temperature slope range; Calculate the second absolute error value of the deviation of the average temperature change from the target standard average temperature range; Calculate the third absolute error value between the instantaneous maximum temperature value and the target standard instantaneous high temperature limit value; Preset reciprocal weighting coefficients are assigned to the first absolute error value, the second absolute error value, and the third absolute error value, respectively; The credibility score of the state transition event is obtained by multiplying the first absolute error value, the second absolute error value, and the third absolute error value by their respective reciprocal weight coefficients and then summing them.

7. The deep learning-based multimodal data analysis method for power line inspection as described in claim 1, characterized in that, Alternatively, recognize the state transition event as a real state change and generate a state change record, or discard the state transition event: Set a high confidence threshold and a low confidence threshold, with the high confidence threshold being greater than the low confidence threshold; The credibility score of the state transition event is compared with the high credibility threshold and the low credibility threshold in turn. If the credibility score is greater than or equal to the high credibility threshold, then the temperature change characteristics of the state transition event are determined to conform to the physical heating law of this type of equipment. This state transition event has been confirmed as a real state change at the physical level. Package the device identifier, the state label after the state transition event, and the current timestamp to generate a standardized state change record file and write it to the backend database. If the credibility score is less than the low credibility threshold, the state transition event is determined to be a temporal state machine erroneous triggering event caused by the loss of visual target detection due to ambient light noise. The state transition event is marked as invalid in the state transition candidate queue, and the memory space occupied by the state transition event is discarded directly.

8. The deep learning-based multimodal data analysis method for power line inspection as described in claim 7, characterized in that, The steps involved in packaging the device identifier, the state label after the state transition event, and the current timestamp to generate a standardized formatted state change log file and writing it to the backend database include: Retrieve all historical temperature change feature vectors of the same device identifier that were confirmed as real state changes within the historical operating cycle from the backend database. Calculate the Euclidean distance between the newly generated temperature change feature vector and all historical temperature change feature vectors; When the Euclidean distance is greater than the preset out-of-distribution drift threshold, the power equipment is determined to be in an unrecorded special operating environment. Extract the slope of the highest temperature change and the amplitude of the average temperature change under this special working condition; An adaptive smoothing filter algorithm is used to smooth the data of the slope of the highest temperature change and the amplitude of the average temperature change to eliminate transient noise interference. The slope of the highest temperature change after smoothing and the magnitude of the average temperature change are used as new expected correspondence parameters. Insert the new expected correspondence parameters into the physical consistency rule base and update the target standard maximum temperature slope range and target standard average temperature amplitude range for the type corresponding to the device identifier.

9. A deep learning-based multimodal data analysis system for power line inspection, used to implement the deep learning-based multimodal data analysis method for power line inspection as described in any one of claims 1-8, characterized in that, include: Matrix acquisition module, sequence acquisition module, image processing module, information monitoring module, feature determination module, score acquisition module, and record generation module; The matrix acquisition module is used to acquire the pre-calibrated spatial registration transformation matrix between the visible light camera and the infrared thermal imager. The sequence acquisition module is used to acquire visible light image sequences and infrared thermal image sequences at the same timestamp, and generate spatiotemporally aligned multimodal image pairs; The image processing module is used to process spatiotemporally aligned multimodal image pairs to identify power equipment status information in visible light images and heat-generating areas and temperature distribution data in infrared thermal images, respectively. The information monitoring module is used to monitor the status information of power equipment output by visible light images. When a status change occurs, the status change event, equipment identifier, timestamp, and the corresponding indexes of visible light image sequence and infrared thermal image sequence are stored in the status change candidate queue. The feature determination module is used to extract the infrared thermal image sequence within the time window based on the timestamp of the state transition event, and determine the temperature change characteristics of the heating area. The scoring acquisition module is used to compare the temperature change characteristics with the expected correspondence between the power equipment state changes and temperature changes in the physical consistency rule base to obtain the credibility score of the state jump event. The record generation module is used to confirm the state transition event as a real state change and generate a state change record based on the comparison result between the credibility score and the preset credibility threshold, or to discard the state transition event.

Images