Power transmission line detection method based on multi-modal data fusion and related equipment

Abstract

The invention provides a multi-modal data fusion-based power transmission line detection method and related equipment. The method comprises the following steps: performing scene segmentation on a space region of a power transmission line based on geographic information and remote sensing image data to obtain a plurality of scene regions with space continuity; obtaining original multi-modal data of the power transmission line; screening from the original multi-modal data based on a preset distance condition to obtain target multi-modal data of the target area in the target time period; obtaining a graph structure representing a relationship between the target multi-modal data based on the target multi-modal data and the corresponding position information; performing feature extraction on the target multi-modal data to obtain target multi-modal features; performing feature fusion on the target multi-modal features based on the graph structure to obtain global fusion features; and determining the operation state of the target position of the power transmission line at the target moment based on the global fusion feature. The operation state of the power transmission line can be accurately and reliably determined.

Description

Technical Field

[0001] The present disclosure relates to the field of line detection, and in particular to a transmission line detection method based on multimodal data fusion and related equipment. Background Art

[0002] With the exponential growth of power grids, transmission networks have evolved into three-dimensional systems spanning complex geographical environments. Traditional manual inspection models, limited by temporal and spatial coverage and subjective experience, struggle to meet the demands of dynamically changing line perception. Transmission line anomalies often go undetected due to low manual inspection efficiency, resulting in low accuracy and reliability in transmission circuit detection. Summary of the Invention

[0003] The present disclosure proposes a transmission line detection method and related equipment based on multimodal data fusion to solve the above-mentioned technical problems to a certain extent.

[0004] In a first aspect of the present disclosure, a transmission line detection method based on multimodal data fusion is provided, comprising: Performing scene segmentation on the spatial region of the transmission line based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity; Acquire original multimodal data of the power transmission line; wherein the original multimodal data includes sensor data and drone data having location information of the power transmission line; Filtering the original multimodal data based on a preset distance condition to obtain target multimodal data of a target area in a target time period; wherein the target area includes a first scene area in the scene area where the target position is located and a second scene area adjacent to the first scene area, and the target time period includes a target time; Obtaining a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding position information; Performing feature extraction on the target multimodal data to obtain target multimodal features; Performing feature fusion on the target multimodal features based on the graph structure to obtain a global fusion feature; An operating state of the target location of the transmission line at the target time is determined based on the global fusion feature.

[0005] In some embodiments, obtaining a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding location information includes: Determining a node of the graph structure based on each target multimodal data; In response to the spatial distance between the nodes being less than a preset threshold, constructing an edge between the nodes, and determining an edge weight of the edge based on the data type of the nodes and the spatial distance; The nodes and the edges are traversed based on a preset weight rule to update the node weights of the nodes and / or the edge weights of the edges.

[0006] In some embodiments, the edge weights are: , where w ij For nodes i and neighbor nodes j The edge weights of For nodes i and neighbor nodes j The data type relationship, ; For nodes i and neighbor nodes j The spatial distance between is a parameter that controls the speed of weight decay.

[0007] In some embodiments, performing feature fusion on the target multimodal features based on the graph structure to obtain a global fusion feature includes: Determining an attention coefficient of the node based on the node weight, the edge weight, and the target multimodal feature; Performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature; Performing global optimization on the node fusion features to obtain the global fusion features; The attention coefficient includes: , , , in, is the attention coefficient of node i and neighbor node j, exp is the exponential function, is the set of neighbor nodes of node i, j and k are the serial numbers of neighbor nodes, is the activation function, is the attention vector, W is the weight matrix, is the node weight of node i and the weighted multimodal features of the target, is the node weight of neighbor node j and the weight of target multimodal features, h k weighted is the node weight of neighbor node k and the weighted multimodal features of the target, is the feature splicing operation, is the edge weight between node i and neighbor node j, w ik is the edge weight between node i and its neighbor node k.

[0008] In some embodiments, performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature includes: , in, is the node fusion feature of node i, is a non-linear activation function.

[0009] In some embodiments, performing global optimization on the node fusion features to obtain the global fusion features includes: Initializing the label distribution of the node; Updating the label distribution based on message transmission between the node and neighboring nodes until a preset condition is met to obtain the global fusion feature; Wherein, updating the label distribution based on the message transmission between the node and the neighboring nodes until a preset condition is satisfied, and obtaining the global fusion feature further includes: The node transmits messages to neighbor nodes, where the message between node i and neighbor node j is , is the label of node i, is the label of neighbor node j, is the transfer item between node i and its neighbor node j, is the label distribution of node i; Updating the label distribution based on the message includes: ,in, is the updated label distribution of node i, is the node item of node i, Z i is the normalization constant; Obtaining the global fusion feature based on the updated label distribution includes: ',in, is the global fusion feature of node i.

[0010] In some embodiments, the spatial region of the power transmission line is segmented based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity, including: Determining a first penalty value based on vertical distances between two pixel points in the remote sensing image data and the corresponding nearest transmission lines, and whether the nearest transmission lines corresponding to the two pixel points belong to the same transmission line network segment; determine a second penalty value based on vertical distances of two pixel points in the remote sensing image data to corresponding nearest towers and whether the two pixel points belong to the same tower; determine a priori information distance based on a sum of the first penalty value and the second penalty value; obtain distance metric based on the a priori information distance, color distance and spatial distance of the two pixel points, wherein the spatial distance is obtained based on geographic information of the two pixel points; determine pixel points belonging to the same scene region based on the distance metric.

[0011] In a second aspect, the present disclosure provides a power transmission line detection device based on multi-modal data fusion, comprising: a region division module configured to perform scene segmentation on spatial regions of the power transmission line based on geographic information and remote sensing image data to obtain a plurality of scene regions with spatial continuity; a data acquisition module configured to acquire original multi-modal data of the power transmission line, wherein the original multi-modal data comprises sensor data and unmanned aerial vehicle data with location information of the power transmission line; a data screening module configured to screen target multi-modal data of a target region in a target time period from the original multi-modal data based on a preset distance condition, wherein the target region comprises a first scene region in which a target location is located and a second scene region adjacent to the first scene region, and the target time period comprises a target time point; a graph structure module configured to obtain a graph structure representing relationships between the target multi-modal data based on the target multi-modal data and corresponding location information; a feature extraction module configured to perform feature extraction on the target multi-modal data to obtain target multi-modal features; a feature fusion module configured to perform feature fusion on the target multi-modal features based on the graph structure to obtain global fusion features; a state detection module configured to determine an operating state of the target location of the power transmission line at the target time point based on the global fusion features.

[0012] In some embodiments, obtaining a graph structure representing relationships between the target multi-modal data based on the target multi-modal data and corresponding location information comprises: determining nodes of the graph structure based on each of the target multi-modal data; in response to a spatial distance between the nodes being less than a preset threshold, constructing an edge between the nodes and determining an edge weight of the edge based on a data type of the nodes and the spatial distance; The nodes and the edges are traversed based on a preset weight rule to update the node weights of the nodes and / or the edge weights of the edges.

[0013] In some embodiments, the edge weights are: , where w ij For nodes i and neighbor nodes j The edge weights of For nodes i and neighbor nodes j The data type relationship, ; For nodes i and neighbor nodes j The spatial distance between is a parameter that controls the speed of weight decay.

[0014] In some embodiments, performing feature fusion on the target multimodal features based on the graph structure to obtain a global fusion feature includes: Determining an attention coefficient of the node based on the node weight, the edge weight, and the target multimodal feature; Performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature; Performing global optimization on the node fusion features to obtain the global fusion features; The attention coefficient includes: , , , in, is the attention coefficient of node i and neighbor node j, exp is the exponential function, is the set of neighbor nodes of node i, j and k are the serial numbers of neighbor nodes, is the activation function, is the attention vector, W is the weight matrix, is the node weight of node i and the weighted multimodal features of the target, is the node weight of neighbor node j and the weight of target multimodal features, h k weighted is the node weight of neighbor node k and the weighted multimodal features of the target, is the feature splicing operation, is the edge weight between node i and neighbor node j, w ik is the edge weight between node i and its neighbor node k.

[0015] In some embodiments, performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature includes: , in, is the node fusion feature of node i, is a non-linear activation function.

[0016] In some embodiments, performing global optimization on the node fusion features to obtain the global fusion features includes: Initializing the label distribution of the node; Updating the label distribution based on message transmission between the node and neighboring nodes until a preset condition is met to obtain the global fusion feature; Wherein, updating the label distribution based on the message transmission between the node and the neighboring nodes until a preset condition is satisfied, and obtaining the global fusion feature further includes: The node transmits messages to neighbor nodes, where the message between node i and neighbor node j is , is the label of node i, is the label of neighbor node j, is the transfer item between node i and its neighbor node j, is the label distribution of node i; Updating the label distribution based on the message includes: ,in, is the updated label distribution of node i, is the node item of node i, Z i is the normalization constant; Obtaining the global fusion feature based on the updated label distribution includes: ',in, is the global fusion feature of node i.

[0017] In some embodiments, the spatial region of the power transmission line is segmented based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity, including: Determining a first penalty value based on vertical distances between two pixel points in the remote sensing image data and the corresponding nearest transmission lines, and whether the nearest transmission lines corresponding to the two pixel points belong to the same transmission line network segment; Determining a second penalty value based on vertical distances between two pixel points and corresponding nearest towers in the remote sensing image data, and whether the two pixel points and the corresponding nearest towers belong to the same tower; Determining a priori information distance based on the sum of the first penalty value and the second penalty value; A distance metric is obtained based on the prior information distance, the color distance of the two pixels, and the spatial distance; wherein the spatial distance is obtained based on the geographic information of the two pixels; Pixel points belonging to the same scene area are determined based on the distance metric.

[0018] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors are caused to execute the method described in the first aspect.

[0020] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0021] As can be seen from the foregoing, the present disclosure provides a transmission line detection method and related equipment based on multimodal data fusion. This method divides the transmission line into regions and, based on preset distance conditions, selects target multimodal data from the original multimodal data for a target region within a target time period. This data includes sensor data with location information and drone data. A graph structure representing the relationships between the target multimodal data and its corresponding location information is then constructed. Feature extraction is then performed on the multimodal data to obtain multimodal features. These multimodal features are then fused using the graph structure to obtain global fused features. Finally, the operating status of the target transmission line location at the target time is determined based on the global fused features. By fusing multimodal data, the advantages of different data sources can be comprehensively utilized, overcoming the limitations of a single data source, and more comprehensively and accurately reflecting the operating status of the transmission line. Using a graph structure for feature fusion effectively mines correlation information between multimodal data, improving the effectiveness of feature fusion. The transmission line operating status determined based on the global fused features is more accurate and reliable, helping to promptly identify potential problems with the transmission line and ensure its safe and stable operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 Schematic diagram of the architecture of a transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure.

[0024] Figure 2 Schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0025] Figure 3 Schematic diagram of the process of a transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure.

[0026] Figure 4 Schematic diagram of a transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure.

[0027] Figure 5 Schematic diagram of a feature extraction network for sensor data according to an embodiment of the present disclosure.

[0028] Figure 6 Schematic diagram of a feature extraction network for image data according to an embodiment of the present disclosure.

[0029] Figure 7 Schematic diagram of a feature extraction network for radar data according to an embodiment of the present disclosure.

[0030] Figure 8 This is an accuracy curve of the transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure.

[0031] Figure 9 This is an accuracy curve of the transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure.

[0032] Figure 10 Schematic diagram of a transmission line detection device based on multimodal data fusion according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0034] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0035] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0036] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0037] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0038] Figure 1 FIG2 is a schematic diagram showing the architecture of a transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure. Figure 1 The transmission line detection method architecture 100 based on multimodal data fusion may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0039] Terminal 120 can be implemented in hardware or software. For example, when implemented in hardware, terminal 120 can be any electronic device with a display screen that supports page display, including but not limited to smartphones, tablet computers, e-book readers, laptop computers, and desktop computers. When terminal 120 is implemented in software, it can be installed in the electronic devices listed above; it can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it can be implemented as a single software or software module, and no specific limitations are given here.

[0040] It should be noted that the power transmission line detection method based on multimodal data fusion provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks, and servers in the embodiment is for illustration only and is not intended to limit the number of terminals, networks, and servers.

[0041] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, electronic device 200 may include: processor 202, memory 204, network module 206, peripheral interface 208 and bus 210. Processor 202, memory 204, network module 206 and peripheral interface 208 are connected to each other through bus 210 in communication with each other within electronic device 200.

[0042] The processor 202 may be a central processing unit (CPU), a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. The processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, the processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, the processor 202 may include multiple processors, namely a first processor 202a, a second processor 202b, and a third processor 202c.

[0043] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2As shown, the data stored by the memory 204 can include program instructions (e.g., program instructions for implementing the power line detection method based on multi-modal data fusion of the embodiments of the present disclosure) and data to be processed (e.g., the memory can store configuration files of other modules, etc.). The processor 202 can also access the program instructions and data stored by the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 can include volatile storage or non-volatile storage. In some embodiments, the memory 204 can include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid state disk (SSD), flash memory, memory stick, etc.

[0044] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It can be understood that the type of network is not limited to the specific examples described above. In some embodiments, the network module 206 can include any combination of any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc.

[0045] The peripheral interface 208 can be configured to connect the electronic device 200 with one or more peripheral devices to enable information input and output. For example, the peripheral devices can include input devices such as keyboards, mice, touchpads, touchscreens, microphones, various sensors, etc., and output devices such as displays, speakers, vibrators, indicator lights, etc.

[0046] The bus 210 can be configured to transmit information between various components (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208) of the electronic device 200, such as internal buses (e.g., processor-memory buses), external buses (USB ports, PCI-E buses), etc.

[0047] It should be noted that although the architecture of the electronic device 200 described above only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208, and the bus 210, in the specific implementation process, the architecture of the electronic device 200 can also include other components necessary for normal execution. In addition, those skilled in the art can understand that the architecture of the electronic device 200 described above can also only include components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0048] Transmission line condition perception and monitoring primarily rely on single-modal data. For example, monitoring models are constructed solely using sensor data, or condition perception is performed solely based on drone-collected imagery. However, using single-modal data for condition perception presents certain challenges. First, using only single-modal data can result in incomplete information. For example, while sensors can detect changes in conductor sag, they cannot identify surrounding external damage hazards (such as fallen trees). While drone imagery can capture information about surrounding obstacles, it lacks parameters such as conductor load. Therefore, relying solely on single-modal data can easily lead to incomplete or biased condition perception, making it difficult to reflect the true operating status of transmission lines. Second, various sensors and drone-mounted equipment are subject to failure, resulting in erroneous or missing data, which can severely impact perception results. Furthermore, using single-modal data for condition perception struggles to fully capture the complex operating conditions of transmission lines.

[0049] Despite the emergence of transmission line status monitoring technologies based on multimodal data, achieving multimodal data fusion remains challenging. First, multimodal data differs in temporal dimensions: sensors sample data at the second or minute level, while drones conduct hourly inspections, resulting in a time-frequency misalignment. Second, multimodal data differs significantly in structure and has complex relationships. Simple fusion methods alone cannot capture these relationships and their impact on monitoring results, making it difficult to directly fuse multimodal data for joint status perception. For example, some methods construct a three-dimensional finite element model of the tower-line system and infer the tilt angle or ice thickness based on the tower leg stress values ​​to perform transmission line status perception. However, these models may deviate from the actual situation, resulting in inaccurate perception results. Furthermore, relying solely on tower leg stress data fails to fully capture the complex operating conditions of transmission lines. Other methods integrate waveform and time data into a coordinate system to calculate the specific location of transmission line faults. However, relying solely on waveform data can only identify limited fault types and is prone to misjudgment. Fault diagnosis is impossible if inaccurate data is present during waveform acquisition or if the waveform acquisition equipment malfunctions. Some methods rely on image recognition to diagnose transmission line faults. However, since image recognition is not sensitive to faults such as tower tilt and the image acquisition frequency is low, it is impossible to perform continuous and high-precision fault detection.

[0050] Therefore, how to improve the accuracy and efficiency of transmission line detection has become a technical problem that needs to be solved urgently.

[0051] In light of this, embodiments of the present disclosure provide a transmission line detection method and related equipment based on multimodal data fusion. The method divides the transmission line into regions and, based on preset distance conditions, filters target multimodal data from the original multimodal data for a target region within a target time period. This data includes both sensor data with location information and drone data. A graph structure is then constructed based on the target multimodal data and its corresponding location information to represent the relationships between the multimodal data. Feature extraction is then performed on the multimodal data to obtain multimodal features. These multimodal features are then fused using the graph structure to obtain global fused features. Finally, the operating status of the target transmission line location at the target time is determined based on the global fused features. By fusing multimodal data, the advantages of different data sources can be comprehensively utilized, overcoming the limitations of a single data source, and providing a more comprehensive and accurate representation of the transmission line's operating status. Using the graph structure for feature fusion effectively mines correlation information between multimodal data, improving the effectiveness of feature fusion. The transmission line operating status determined based on the global fused features is more accurate and reliable, helping to promptly identify potential problems with the transmission line and ensure its safe and stable operation.

[0052] See also Figure 3 , Figure 3 A schematic flow chart of a power transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure is shown. The power transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure can be deployed on a terminal or a server. Figure 3 In the embodiment, the transmission line detection method 300 based on multimodal data fusion may further include the following steps.

[0053] In step S310, scene segmentation is performed on the spatial region of the power transmission line based on geographic information and remote sensing image data to obtain a plurality of scene regions with spatial continuity.

[0054] Transmission lines can be conductors and their associated infrastructure used to transmit electrical energy within a power system. Geographic information can refer to the basic geographic features of the area where the transmission line is located, such as topographic relief and the location of surrounding cities and villages. Remote sensing image data is a record of information about the Earth's surface or atmosphere acquired through remote sensing technology. Specifically, sensors can be used to detect the Earth's surface from the air or space, capturing information about electromagnetic waves reflected or emitted by surface objects and converting this information into image data that can reveal features such as the shape, size, color, and texture of surface objects. For transmission lines, remote sensing image data can clearly reveal information such as the line's direction, surrounding topography, vegetation cover, and the presence of obstacles (such as buildings and trees), providing an intuitive and rich data source for scene segmentation. The spatial area of ​​a transmission line refers to the area occupied by the transmission line in geographic space. It includes not only the area where the transmission line itself (such as conductors and towers) is located, but also areas within a certain range around the line that may affect its operation and safety. Scene segmentation is the process of dividing a scene into several sub-areas with specific meanings or characteristics. In power transmission line scene segmentation, the spatial region of the transmission line is divided into different sections based on the characteristics reflected by geographic information and remote sensing image data. These sections can be divided based on different criteria, such as mountainous areas, plains, and hilly areas based on topography; farmland areas, forest areas, and residential areas based on land use type; or safe areas and potentially dangerous areas based on the relationship between the transmission line and surrounding objects. Spatial continuity refers to the fact that the segmented scene regions are interconnected and uninterrupted in geographic space. In other words, there are no obvious gaps or jumps between adjacent scene regions, and they together constitute a complete transmission line spatial region.

[0055] In some embodiments, scene segmentation is performed on the spatial region of the transmission line based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity, including: Determining a first penalty value based on vertical distances between two pixel points in the remote sensing image data and the corresponding nearest transmission lines, and whether the nearest transmission lines corresponding to the two pixel points belong to the same transmission line network segment; Determining a second penalty value based on vertical distances between two pixel points and corresponding nearest towers in the remote sensing image data, and whether the two pixel points and the corresponding nearest towers belong to the same tower; Determining a priori information distance based on the sum of the first penalty value and the second penalty value; A distance metric is obtained based on the prior information distance, the color distance of the two pixels, and the spatial distance; wherein the spatial distance is obtained based on the geographic information of the two pixels; Pixel points belonging to the same scene area are determined based on the distance metric.

[0056] In remote sensing image data, power transmission lines appear as linear features with a specific direction and distribution. By calculating the vertical distance between each pixel and its nearest transmission line, the pixel's proximity to the transmission line can be measured. The smaller the vertical distance between the pixel and its nearest transmission line, the stronger the correlation between the pixel and the nearest transmission line. Furthermore, if the nearest transmission lines corresponding to the two pixels belong to the same transmission line segment, the first penalty value can be set to 0. If the nearest transmission lines corresponding to the two pixels do not belong to the same transmission line segment, the first penalty value can be determined based on the vertical distance between the two pixels and the nearest transmission line. The closer the pixels are to their nearest transmission lines, the stronger their relationship with their respective line segments, and they should be classified into different segments. In this case, a larger penalty value is assigned to reduce the likelihood of them being classified into the same scene area; conversely, a smaller penalty value is assigned to increase the probability of them belonging to the same scene area. For example, consider two pixels A and B. The vertical distance from point A to the nearest transmission line L_A is 5 meters, and the vertical distance from point B to the nearest transmission line L_B is 10 meters. Therefore, the nearest transmission lines L_A and L_B are in different network segments. For two pixels C and D, the vertical distance from point C to the nearest transmission line L_C is 20 meters, and the vertical distance from point D to the nearest transmission line L_D is 40 meters. Therefore, the nearest transmission lines L_C and L_D are in different network segments. Based on the preset rule, the calculated first penalty values ​​for pixels A and B will be greater than the first penalty values ​​for pixels C and D.

[0057] The tower is an important supporting structure of the power transmission line, and its distribution and position have an important influence on the spatial area division of the power transmission line. The vertical distance from two pixel points to their respective nearest tower reflects their relative position relationship with the tower. The smaller the vertical distance from the pixel point to the nearest tower, the stronger the correlation between the pixel point and the nearest tower. If the nearest tower corresponding to the two pixel points belongs to the same tower, the second penalty value can be set to 0. If the nearest tower corresponding to the two pixel points does not belong to the same tower, the second penalty value can be determined based on the vertical distance from the two pixel points to the nearest tower. The closer the pixel point is to the nearest tower, the stronger the relationship between them and the respective tower, and they should be divided into different network segments. At this time, a larger penalty value is given to reduce the possibility of being divided into the same scene area; otherwise, the penalty value is smaller, increasing the probability of belonging to the same scene area. For example, two pixel points A' and B', the vertical distance from A' to the nearest tower T_A is 1 meter, and the vertical distance from B' to the nearest tower T_B is 5 meters. The nearest tower T_A and the nearest tower T_B are different towers. And two pixel points C' and D', the vertical distance from C' to the nearest tower T_C is 10 meters, and the vertical distance from D' to the nearest tower T_D is 15 meters. The nearest tower T_C and the nearest tower T_D are different towers. According to the preset rule, the second penalty value of pixel points A' and B' will be greater than that of pixel points C' and D'.

[0058] The sum of the first penalty value and the second penalty value is used to determine the prior information distance. The prior information distance comprehensively considers the position relationship between the pixel points and the power transmission line and the tower. The sum of the first penalty value and the second penalty value is obtained. The greater the value, the greater the difference between the two pixel points in terms of power transmission line network segment and tower association, and the less likely they belong to the same scene area; otherwise, the smaller the prior information distance, the greater the possibility that they belong to the same scene area. For example, if the first penalty value of the pixel points A and B mentioned earlier is 10, and the second penalty value is 8, then the prior information distance of A and B is 18; the first penalty value of the pixel points C and D is 3, and the second penalty value is 2, and the prior information distance is 5. Obviously, the prior information distance of A and B is greater, and the possibility that they belong to the same scene area is lower.

[0059] Distance metrics can be derived based on prior information distance, the color distance between two pixels, and the spatial distance. Color distance can be used to represent the different color characteristics of different features in remote sensing imagery data. The color distance between two pixels reflects the degree of color similarity. A smaller color distance indicates closer colors, and they are more likely to belong to the same feature type or scene area. A larger color distance indicates less likely they belong to the same scene area. Spatial distance can be calculated based on the geographic information (such as latitude and longitude) between two pixels. A smaller spatial distance indicates geographic proximity and a higher likelihood of belonging to the same scene area; a larger spatial distance indicates the opposite. A comprehensive distance metric combines the prior information distance, color distance, and spatial distance to produce a distance metric. This metric takes into account the pixel's positional relationship with the transmission line and tower, its color characteristics, and its geographic location, providing a more comprehensive measure of whether two pixels belong to the same scene area. For example, suppose pixels E and F have a prior information distance of 4, a color distance of 2, and a spatial distance of 3. Using pre-set weights and a calculation formula, their distance metric can be derived.

[0060] In some implementations, pixels whose distance metric is less than a metric threshold can be determined to be in the same scene area. The metric threshold is a pre-set standard value used to determine whether two pixels belong to the same scene area. If the distance metric of two pixels is less than the metric threshold, it means that their comprehensive differences in prior information, color, and spatial position are small, and they are determined to be in the same scene area; conversely, if the distance metric is greater than the metric threshold, they are considered not to belong to the same scene area. For example, if the set metric threshold is 8, the distance metric value of the aforementioned pixels E and F is calculated to be 7, which is less than the metric threshold, so E and F are determined to be in the same scene area; and the distance metric value of pixels A and B is assumed to be 20, which is greater than the metric threshold, so they do not belong to the same scene area. In this way, the spatial area of ​​the transmission line can be segmented to obtain multiple scene areas with spatial continuity, providing a basis for subsequent analysis and management of the transmission line.

[0061] In other implementations, a superpixel segmentation algorithm based on Simple Linear Iterative Clustering (SLIC) can be used to divide transmission line corridors into spatially continuous scene units. Due to the wide distribution of transmission lines, the large monitoring area, and the complex environment surrounding the lines, environmental conditions vary significantly in different areas, significantly affecting the line's operating status and the monitoring process.

[0062] Specifically, the distance metric Defining pixels and pixels The similarities between The smaller the value, the more similar the two pixels are, and the greater the probability of them being divided into the same area. Calculation method: The specific calculation formula is as follows: ,in, and is an adjustable weight parameter.

[0063] in, , is the distance between the RGB color representations of two pixels, the more similar the colors are The smaller; , is the spatial distance between two pixels; is the distance between two pixels based on the tower coordinates and the prior information of the line direction. The calculation method is as follows: First, determine whether two pixels belong to the same transmission line segment. The line segment closest to pixel i is , the vertical distance from i to the line is ; The nearest line segment to pixel j is , the vertical distance from j to the line is .like , then the line segments to which the two pixels belong are considered to be different, and the penalty value is set , that is, the closer the two pixels are to their respective lines, the stronger the correlation between the pixels and different line segments, and the weaker the correlation between the two pixels; if , then the two pixels are considered to belong to the same line segment, and the penalty value is set .therefore .

[0064] Secondly, determine whether the two pixels belong to the same tower. The tower closest to pixel i is , the distance is The nearest tower to pixel j is , the distance is .like , then the two pixels belong to different towers, and the penalty value for the tower is set. , that is, the closer two pixels are to their respective towers, the stronger the correlation between different towers, and the weaker the correlation between pixels. , then the two pixels belong to the same tower, and the penalty value .therefore .

[0065] Add the penalties for transmission line segment differences and tower differences to obtain the final prior information distance: .

[0066] Through the above calculation method of pixel similarity and the SLIC superpixel segmentation algorithm, the transmission line corridor in the monitoring range is divided into multiple regions with spatial continuity. Each region presents its own characteristics due to different geographical elements, environmental conditions and transmission line directions.

[0067] The spatial neighborhood knowledge base is used to describe the formation of a region with obvious characteristics due to the interaction of environmental elements in adjacent spatial units. It can represent the environmental characteristics of the internal space. Therefore, a spatial neighborhood knowledge base with regional characteristics is first constructed to reflect the characteristics of each monitoring region. In order to realize the labeled management of different regions of the transmission line corridor, a pre-defined label system and software analysis method are used to add labels to each region. The labels are divided into several categories such as terrain, vegetation cover, and meteorological environment, and each category has multiple different labels corresponding to different regional characteristics. Through the analysis of GIS data, remote sensing image data and other information, the extracted regional characteristics are matched with the pre-defined label system to automatically add corresponding labels to different regions. For example, if the analysis result shows that the average elevation of a certain region is high and the slope is large, the "mountainous area" label is added; if the analysis result shows that the humidity of a certain region is large, the "high humidity" label is added. Through the above method, the transmission line corridor in the monitoring range is divided into different regions, each region has its own characteristics, and corresponding labels are added according to the regional characteristics, and finally the construction of the spatial neighborhood knowledge base is completed.

[0068] In step S320, the original multi-modal data of the transmission line is obtained; wherein the original multi-modal data includes sensor data and unmanned aerial vehicle data with position information of the transmission line.

[0069] Among them, the original multi-modal data refers to data sets from different data acquisition devices or with different characteristics, such as including sensor data and unmanned aerial vehicle data with position information. Sensor data can be obtained by sensors installed on the transmission line or related facilities, which can monitor various physical quantities (such as temperature, humidity, current, voltage, etc.), and transmit these data together with position information to the detection system. Unmanned aerial vehicle data refers to data obtained by unmanned aerial vehicles (unmanned aerial vehicles) flying near or above the transmission line. Unmanned aerial vehicles can carry various sensors and cameras to take pictures, videos or obtain other related data of the transmission line, providing intuitive information about the appearance, structure or operating state of the transmission line.

[0070] Specifically, a variety of sensors are deployed on the power transmission line to continuously collect data, including wind deviation sensors, tower inclination sensors, tension sensors, ice-coating sensors, temperature sensors, galloping sensors, current and voltage sensors, etc. Some types of sensors are only deployed on the tower, such as tower inclination sensors; some types of sensors are only deployed on the power transmission line, such as galloping sensors; and some sensors can be deployed on both the tower and the power transmission line, such as ice-coating sensors. Once the sensors are deployed, their positions remain unchanged, so the position information of the sensors can be obtained directly when they are deployed, and all the collected data also has the position information of the sensors. The unmanned aerial vehicle is equipped with visible light cameras, laser radars and other devices to conduct regular inspection of the entire power transmission line. During the inspection, the unmanned aerial vehicle follows the pre-set flight route to take pictures and scan at fixed points, ensuring that the information of each tower and power transmission line on the inspection route can be obtained. At the same time, the unmanned aerial vehicle is equipped with a GPS module, which can obtain GPS information in real time during the inspection. Therefore, each image and radar data collected by the unmanned aerial vehicle has GPS information of the shooting position. According to the method of the embodiments of the present disclosure, sensor data, images and radar data collected by the unmanned aerial vehicle are used as input. The sensor data contains the position information of the sensors, and the images and radar data collected by the unmanned aerial vehicle contain the position information of the collection time, so that all the input data have accurate position information.

[0071] In step S330, target multi-modal data of a target region in a target time period is selected from the original multi-modal data based on a preset distance condition; wherein the target region includes a first scene region where a target position in the scene region is located and a second scene region adjacent to the first scene region, and the target time period includes a target time.

[0072] The preset distance condition can include the spatial distance between locations corresponding to the original multimodal data being less than a preset distance. During transmission line status monitoring, due to the widespread distribution of data acquisition terminals, the large amount of data, and the regional correlation of transmission line operating conditions, multimodal data with strong correlation with the status of monitoring points must be selected as the data source for transmission line status perception to more accurately reflect the true status of the line. By combining adjacent region screening and distance threshold screening, data sources with strong correlation with the status of monitoring points are screened. First, based on the spatial topological relationships of each region in a constructed spatial proximity knowledge base, all regions directly adjacent to the region containing the monitoring point are identified. The region containing the monitoring point and all directly adjacent regions are considered candidate regions. Data from all candidate regions collected within the time t' before the current monitoring time t are selected, and the spatial distance between these data and the monitoring point is calculated. If the distance is less than the preset distance threshold, the data is used as the monitoring data source; if the distance is greater than the threshold, the data is not considered in the monitoring process for that point. The spatial topological relationships of regions reflect the spatial adjacency between regions, and adjacent regions generally have high status correlation. The introduction of distance thresholds further constrains the spatial scope of data sources, avoiding the inclusion of data that is too far from monitoring points, thereby ensuring the spatial locality and relevance of data sources. By comprehensively considering regional adjacency and data source distance constraints, data sources that are spatially closely related to monitoring points can be effectively screened, providing high-quality data support for subsequent transmission line status monitoring.

[0073] See also Figure 4 , Figure 4 A schematic diagram of a transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure is shown. Figure 4 In the algorithm, we can first filter the data source based on spatial location prior, then build a graph structure based on location information and data relationships, then extract multimodal data features based on deep learning, and finally perform cross-modal feature fusion and perception enhancement judgment based on graph attention network and conditional random field.

[0074] The sensor has a fixed location when it is deployed. When collecting data, the drone will obtain the current location information based on the GPS module it carries. Therefore, all collected data has specific location coordinates. The coordinate information of the data includes latitude, longitude and altitude information. Since the position of the sensor is fixed, the coordinates of the sensor data are marked as ; Since the drone is constantly moving during the inspection process, the data collection location is constantly changing, and the coordinates of the drone data are marked as , that is, the data collected by the drone at the location at time t. Assume that the state perception system needs to monitor the location The line operation status at time t can be used to calculate the distance between all data sources and the monitoring point. The formula is as follows: , in, Represent the coordinate positions of the monitoring points, Respectively represent the coordinate position of the data.

[0075] Based on the above formula, the distance between all data and monitoring points is calculated and the selected The data is used as the data source for state perception. is the distance threshold parameter. After filtering out the adjacent data, take all the data collected within Before The data within the time period is the data source of the state perception, so the data source of the state perception is the data in the vicinity of the detection location. The data collected during the time is the time threshold parameter. Because different sensors have different acquisition frequencies and the drone's radar scanning target continuously changes, each data point has a different shape. A neural network will subsequently be used to extract data of varying shapes into uniformly sized features for further feature fusion and state perception. Based on preset spatial distance conditions, sensor data and drone data (i.e., target multimodal data) covering the first scene area where the target location is located and its adjacent second scene area (i.e., target area) within a specific target time period are filtered from the original multimodal data to enable precise, focused analysis of the transmission line status within that specific area and time period.

[0076] In step S340 , a graph structure representing the relationship between the target multimodal data is obtained based on the target multimodal data and the corresponding position information.

[0077] The graph structure associates sensor data and drone data with location information based on their corresponding locations. Each multimodal data point is considered a node, and each node contains information from different modalities (sensor data, drone data). Edges are constructed based on the proximity of nodes. Closer nodes are more likely to influence each other, thus establishing edge connections.

[0078] Traditional data processing methods can often only process sensor data or drone data individually, making it difficult to discover potential connections between different modal data. Graph-based methods, however, can comprehensively consider multiple modal data and, through the relationships between nodes and edges, uncover the inherent connections between data at different locations and of different types. For example, they can discover a correlation between anomalies in sensor data at a certain location and changes in the appearance of a power line captured by a nearby drone. Through graph-guided feature fusion, the resulting global fusion features contain richer information and can more accurately characterize the operating status of transmission lines. Compared to simple feature concatenation or addition, graph-based fusion features have stronger expressive power and discrimination, providing more comprehensive information for transmission line operating status detection, reducing the errors and uncertainties introduced by a single data source, thereby improving detection accuracy and enhancing generalization capabilities.

[0079] In some embodiments, obtaining a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding location information includes: Determining a node of the graph structure based on each target multimodal data; In response to the spatial distance between the nodes being less than a preset threshold, constructing an edge between the nodes, and determining an edge weight of the edge based on the data type of the nodes and the spatial distance; The nodes and the edges are traversed based on a preset weight rule to update the node weights of the nodes and / or the edge weights of the edges.

[0080] Each data point in the target multimodal data is mapped to a node in a graph structure, with the node carrying its corresponding location information and data type. If the spatial distance between two nodes is less than a preset threshold, an edge is constructed between them, and the edge weight is calculated based on whether the nodes have the same data type and the spatial distance. Finally, the graph structure is iteratively optimized using preset weighting rules, dynamically adjusting node weights and / or edge weights to construct a spatiotemporal graph network capable of representing the complex relationships between multimodal data.

[0081] Specifically, after completing the screening of the data source, a topological map of the data source is constructed by combining the information of the data itself and the regional label information to characterize the relationship between the data. The present disclosure proposes a method for constructing a data source topological map based on predefined rules of a spatial proximity knowledge base, aiming to more effectively integrate the semantic information in the transmission line monitoring data. Each node in the topological map represents a data source, and the edge between two nodes indicates that there is a correlation between the two data sources, and the weight of the edge indicates the strength of the correlation between the data sources. Each node contains the following attributes: device ID; data type; collection time; location information, i.e., GPS coordinates; regional ID, used to identify the area where the node is located; regional labels, such as "mountainous area", "high humidity", etc.; and node weight , used to adjust the importance of the node feature in the subsequent graph attention network, with the initial value set to 1.0.

[0082] First, a set of rules is predefined to describe how factors like data type and region labels affect edge and node weights. These rules are stored in a rule base. Each rule includes the following elements: the rule generation condition; the target (i.e., whether it affects node weights or edge weights); the weight adjustment method and the weight adjustment magnitude. For example, "If the region label contains 'forest' and the data type is 'visible light image,' then the node weight is multiplied by 0.7." and "If the number of region labels in data 1 and data 2 that are identical is greater than 2, then the edge weight is multiplied by 1.1."

[0083] When constructing a topological graph, each data point is first treated as a node. If the spatial distance between two nodes is less than a threshold, an edge is established between the two nodes, indicating that there is a correlation between the two data points. If the distance is greater than the threshold, there is no edge between the two nodes, indicating that there is no correlation between the two data points. The node weights of all nodes are initialized to 1.0, and the initial edge weights are calculated based on the spatial distance between the two nodes and the data type.

[0084] In some embodiments, determining the edge weight of the edge based on the data type of the node and the spatial distance includes: ,in, For nodes i and nodes j The data type relationship, ; For nodes i and nodes j The spatial distance between ; is a parameter that controls the speed of weight decay.

[0085] Among them, x i 、y i 、z i is the coordinate of node i, x j 、y j 、z j The coordinates of node j are given by . Next, each rule in the rule base is traversed to check whether any nodes or edges meet the rule conditions. If so, the weights of the corresponding nodes or edges are updated according to the rule. This rule-based topology graph construction method allows for the flexible integration of semantic information into the topology graph, improving the graph's ability to represent data.

[0086] In step S350, feature extraction is performed on the target multimodal data to obtain target multimodal features.

[0087] After the data source screening and graph structure construction are completed, different neural networks can be used to extract features from the multi-modal data. According to the characteristics of sensor data, unmanned aerial vehicle image data, and radar point cloud data, a special feature extraction network can be designed, and feature standardization and dimension alignment are performed to ensure that the features of different modalities have the same shape and size, thereby providing a consistent feature space for subsequent cross-modal fusion.

[0088] A DNN composed of stacked autoencoders can be used to extract features from sensor data. Since different types of sensors have different sampling frequencies, for example, a dancing sensor samples at a second level to monitor the dancing of a power transmission line in a short period of time, while a tower tilt sensor samples at a hourly level to monitor the slow changes of a tower over a long period of time. In addition, different sensors have different change rules and characteristics. Therefore, different DNN networks are designed for different sensors to adapt to different sampling frequencies and data characteristics. Referring to Figure 5 , Figure 5 FIG. 7 shows a schematic diagram of a feature extraction network for sensor data according to an embodiment of the present disclosure. Figure 5 In the embodiment, for a sensor with a sampling frequency of , the DNN network can be set as follows: the neural network is composed of 3 layers of encoders and 3 layers of decoders (the decoder is used for unsupervised pre-training, and only the encoder part is used in the actual feature extraction process), and the shape of the input data is , where is the length of the time window (in seconds), is the number of channels of the sensor (for example, the data collected by a vibration sensor has 3 channels, corresponding to the vibration data in direction); and the output is a feature vector of dimensions. According to the sampling frequency f, a specific network structure is designed to extract different sensor data into feature vectors of the same size.

[0089] Referring to Figure 6 , Figure 6 FIG. 8 shows a schematic diagram of a feature extraction network for image data according to an embodiment of the present disclosure. Since a convolutional neural network (CNN) has advantages in image processing, a CNN is used to extract features from the RGB image data collected by an unmanned aerial vehicle. The images collected by the unmanned aerial vehicle are usually of the same size, and it is assumed that the image size is , where is the image width, i.e., the number of horizontal pixels; is the image height, i.e., the number of vertical pixels; is the number of channels, which is 3 in an RGB image. A CNN network is designed to extract image features for images of this size. The CNN network is composed of 4 convolutional blocks and 1 global average pooling layer, and the input shape is The image output Through the above CNN network, effective features can be extracted from the high-resolution images taken by the drone.

[0090] See also Figure 7 , Figure 7 A schematic diagram of a feature extraction network for radar data according to an embodiment of the present disclosure is shown. Since the number of point clouds in radar point cloud data is usually variable, depending on factors such as the scanning range, object density, radar resolution, and environmental conditions, a dynamic graph convolutional network (DGCNN) can effectively process radar data with different point cloud numbers without the need to train a separate network for each input size. At the same time, it can capture the local geometric structure and global features of the point cloud. Therefore, a dynamic graph convolutional network is used to extract features from radar point cloud data. Assuming that the point cloud data collected by the radar contains N points, the shape of the input data is N×3, and the output is Through the above dynamic graph convolutional network, effective features can be extracted from the three-dimensional point cloud data collected by the radar.

[0091] In step S340 , feature fusion is performed on the target multimodal features based on the graph structure to obtain a global fusion feature.

[0092] Among them, after completing the feature extraction of multimodal data, the feature vectors of sensor data, UAV image data and UAV radar point cloud data are obtained, which are 、 and In order to achieve cross-modal feature fusion while considering the correlation between data, a graph attention network (GAT) is used for feature fusion, and conditional random fields are used for further optimization to make the fusion results globally consistent. Finally, Softmax is used for state judgment.

[0093] In some embodiments, performing feature fusion on the target multimodal features based on the graph structure to obtain a global fusion feature includes: Determining an attention coefficient of the node based on the node weight, the edge weight, and the target multimodal feature; Performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature; The node fusion features are globally optimized to obtain the global fusion features.

[0094] Specifically, in the graph attention network, based on the constructed graph structure, each node aggregates the information of its neighboring nodes through the attention mechanism. For node i and its neighbor node j, the attention coefficient includes: , , , in, is the attention coefficient of node i and neighbor node j, exp is the exponential function, is the set of neighbor nodes of node i, j and k are the serial numbers of neighbor nodes, is the activation function, is the attention vector, W is the weight matrix, is the node weight of node i and the weighted multimodal features of the target, is the node weight of neighbor node j and the weight of target multimodal features, h k weighted is the node weight of neighbor node k and the weighted multimodal features of the target, is the feature splicing operation, is the edge weight between node i and neighbor node j, w ik is the edge weight between node i and its neighbor node k.

[0095] Among them, through the attention coefficient , the updated node fusion feature of node i can be calculated for: ,in, is a non-linear activation function.

[0096] In order to capture higher-order relationships between nodes, multiple layers of GAT are stacked, and the output features of each layer are used as the input of the next layer, and finally the fusion features of each node are obtained. .

[0097] After completing the cross-modal feature fusion based on the graph attention network, the fusion feature of each node is obtained In order to further optimize the fusion results and make them globally consistent, the conditional random field (CRF) is introduced to ensure the continuity of the fusion features in space and time by modeling the dependencies between nodes, and provide more reliable input for subsequent state judgment.

[0098] The core of the conditional random field is through the energy function Model the joint probability distribution of node labels y. The energy function consists of two parts: node term and transfer term. The node term represents the contribution of each node's own characteristics to the label, and the calculation formula is: ,in, It is the label probability distribution calculated by the fully connected layer and the Softmax function. The calculation formula is: ,in, and is a learnable parameter, and K is the number of label categories. The transfer term represents the dependency between nodes and encourages spatially adjacent nodes to have similar labels. The calculation formula is: ,in, is the label compatibility function, usually defined as: , is the feature similarity function, usually using a Gaussian kernel: ,in is an adjustable parameter that controls the similarity decay.

[0099] Combining the node term and the transfer term, the energy function can be defined as: In order to minimize the energy function , use the mean field approximation for inference, the specific steps are as follows: Initialize the label distribution of each node , each node i and its neighbor node j transmit messages, the message , combined with the node item and message passing results. Update the label distribution of node i: ,in Is a normalization constant. Repeat the message passing and label distribution update until convergence or the maximum number of iterations is reached. After CRF optimization, the label distribution of each node Not only its own features are considered, but also the contextual information of neighboring nodes is incorporated. In order to facilitate subsequent state judgment, the fusion features of all nodes are globally averaged and pooled to obtain the global fusion feature. , the fusion formula is as follows: .

[0100] Ultimately, As the output of global consistency optimization, it is used for subsequent status judgment.

[0101] In step S350, the operating state of the target location of the transmission line at the target time is determined based on the global fusion feature.

[0102] Among them, after completing the feature fusion based on the graph attention network and the global consistency optimization based on the conditional random field, the global fusion feature is obtained. Next, we use this information to accurately judge the state of the transmission line through classification tasks. The state judgment network can be composed of three parts: input layer, fully connected layer and output layer. The input layer is the global feature , further extract global features in the fully connected layer: ,in , Finally, the Softmax function is used to output the probability distribution of each category of running status: ,in, , , C is the number of fault categories. The operating status category with the highest probability is the operating status judgment result of the current time and current location.

[0103] See also Figure 8 , Figure 8 The accuracy curve of the transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure is shown. When the inter-node adjacency threshold in the graph structure is 25 meters, the state perception judgment accuracy and computational efficiency are set to 0.1 km to 1.5 km based on the spatial location prior data screening radius. It can be found that when the inter-node adjacency threshold is 25 meters, the transmission line state perception can be completed with high accuracy when the data screening radius is greater than 0.4 km; when the data screening radius is less than 0.7 km, the transmission line state perception calculation can be completed with high efficiency.

[0104] See also Figure 9 , Figure 9 The accuracy curve of the transmission line detection method based on multimodal data fusion according to an embodiment of the present disclosure is shown. When the data screening radius is 0.7 km, the state perception accuracy and computational efficiency are shown when the node adjacency threshold in the graph structure is set to 10 meters to 50 meters. It can be found that when the data screening radius is 0.7 km and the node adjacency threshold is set between 20 meters and 30 meters, transmission line fault detection can be completed with high accuracy; when the node adjacency threshold is less than 40 meters, transmission line state perception calculation can be completed with high efficiency.

[0105] As can be seen, according to the method of the embodiment of the present disclosure, based on data location information and interdependencies, a model combining a graph attention network and a conditional random field is used to dynamically perceive the operating status of the transmission line, providing strong state perception support for fault detection, thereby performing high-accuracy transmission line fault diagnosis. The method proposed in the present invention can effectively solve the problems of weak perception and low accuracy of methods that rely solely on single-modal data in transmission line state perception, as well as the heterogeneity of multi-source data. At the same time, it can complete accurate assessment of line status in the absence of some sensor data, insufficient drone inspection coverage, and severe weather interference. It can filter and fuse multimodal data based on location information, ultimately completing enhanced state perception and judgment of the transmission line. In this transmission line perception and judgment method, a multimodal data source screening method based on spatial location priors is proposed to address the data source issue. A multimodal data feature extraction method based on multiple deep neural networks is proposed to address the heterogeneity of different modal data. A cross-modal feature fusion and perception enhancement judgment method based on a graph attention network and conditional random fields is proposed to address the multimodal feature fusion problem. Simulation experiments show that the method proposed in this invention can still complete the accurate assessment of line status even when some sensor data is missing, drone inspection coverage is insufficient, and there is interference from severe weather, and has a good transmission line status perception effect.

[0106] In summary, the method of the embodiment of the present disclosure achieves regional division of the transmission line corridor through a pixel point distance measurement calculation method for the transmission line scene. The multimodal data source screening method based on the spatial proximity knowledge base selects data with strong correlation with the monitoring point location, providing a high-quality data source for transmission line status perception. The graph structure construction method based on location information and data relationships reflects the spatial, temporal, and attribute relationships between data through the graph structure. The multimodal data feature extraction method based on deep learning designs targeted neural networks for data of different modalities, extracts features of the same size and shape from multimodal data, and solves the heterogeneous problem of multimodal data. The multimodal feature fusion and perception enhancement judgment method based on graph attention network and conditional random field efficiently fuses the features of multimodal data based on multiple relationships between data and performs global consistency optimization, providing accurate results for transmission line status perception and fault judgment. Even in the case of missing sensor data, insufficient drone inspection coverage, and severe weather interference, it can still complete an accurate assessment of the line status, with good transmission line status perception effect.

[0107] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0108] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] Based on the same technical concept, corresponding to any of the above embodiments and methods, the present disclosure also provides a transmission line detection device based on multimodal data fusion, see Figure 10 The transmission line detection device based on multimodal data fusion comprises: A region division module is used to segment the spatial region of the transmission line based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity; A data acquisition module, configured to acquire original multimodal data of the power transmission line; wherein the original multimodal data includes sensor data and drone data having location information of the power transmission line; a data screening module, configured to screen the original multimodal data based on a preset distance condition to obtain target multimodal data of a target area in a target time period; wherein the target area includes a first scene area where a target position is located in the scene area and a second scene area adjacent to the first scene area, and the target time period includes a target time; A graph structure module, configured to obtain a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding position information; A feature extraction module, configured to extract features from the target multimodal data to obtain target multimodal features; A feature fusion module, configured to fuse the target multimodal features based on the graph structure to obtain a global fusion feature; A state detection module is used to determine the operating state of the target position of the transmission line at the target time based on the global fusion feature.

[0110] In some embodiments, obtaining a graph structure representing relationships between the target multi-modal data based on the target multi-modal data and corresponding position information comprises: determining nodes of the graph structure based on each of the target multi-modal data; in response to a spatial distance between the nodes being less than a preset threshold, constructing an edge between the nodes and determining an edge weight of the edge based on a data type of the nodes and the spatial distance; traversing the nodes and the edges based on a preset weight rule to update a node weight of the nodes and / or the edge weight of the edges.

[0111] In some embodiments, the determining the edge weight of the edge based on the data type of the nodes and the spatial distance comprises: wherein w ij is an edge weight of a node i and a neighbor node j , is a data type relationship of a node i and a neighbor node j , ; is a spatial distance between a node i and a neighbor node j , and p is a parameter controlling a weight decay speed.

[0112] In some embodiments, performing feature fusion on the target multi-modal features based on the graph structure to obtain a global fusion feature comprises: determining an attention coefficient of the nodes based on the node weight, the edge weight, and the target multi-modal features; performing feature fusion on the target multi-modal features of the nodes based on the attention coefficient to obtain a node fusion feature; performing global optimization on the node fusion feature to obtain the global fusion feature; wherein the attention coefficient comprises: , , , wherein is an attention coefficient of a node i and a neighbor node j, exp is an exponential function, is a neighbor node set of the node i, j and k are serial numbers of the neighbor nodes, is an activation function, is an attention vector, and W is a weight matrix, is the node weight of node i and the weighted value of the target multimodal feature, is the node weight of neighbor node j and the weight of target multimodal features, h k weighted is the node weight of neighbor node k and the weighted multimodal features of the target, is the feature splicing operation, is the edge weight between node i and neighbor node j, w ik is the edge weight between node i and its neighbor node k.

[0113] In some embodiments, performing feature fusion on the target multimodal feature of the node based on the attention coefficient to obtain a node fusion feature includes: , in, is the node fusion feature of node i, is a non-linear activation function.

[0114] In some embodiments, performing global optimization on the node fusion features to obtain the global fusion features includes: Initializing the label distribution of the node; Updating the label distribution based on message transmission between the node and neighboring nodes until a preset condition is met to obtain the global fusion feature; Wherein, updating the label distribution based on the message transmission between the node and the neighboring nodes until a preset condition is satisfied, and obtaining the global fusion feature further includes: The node transmits messages to neighbor nodes, where the message between node i and neighbor node j is , is the label of node i, is the label of neighbor node j, is the transfer item between node i and its neighbor node j, is the label distribution of node i; Updating the label distribution based on the message includes: ,in, is the updated label distribution of node i, is the node item of node i, Z i is the normalization constant; Obtaining the global fusion feature based on the updated label distribution includes: ',in, is the global fusion feature of node i.

[0115] In some embodiments, scene segmentation is performed on the spatial region of the transmission line based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity, including: Determining a first penalty value based on vertical distances between two pixel points in the remote sensing image data and the corresponding nearest transmission lines, and whether the nearest transmission lines corresponding to the two pixel points belong to the same transmission line network segment; Determining a second penalty value based on vertical distances between two pixel points and corresponding nearest towers in the remote sensing image data, and whether the two pixel points and the corresponding nearest towers belong to the same tower; Determining a priori information distance based on the sum of the first penalty value and the second penalty value; A distance metric is obtained based on the prior information distance, the color distance of the two pixels, and the spatial distance; wherein the spatial distance is obtained based on the geographic information of the two pixels; Pixel points belonging to the same scene area are determined based on the distance metric.

[0116] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0117] The apparatus of the above embodiment is used to implement the corresponding transmission line detection method based on multimodal data fusion in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0118] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the transmission line detection method based on multimodal data fusion as described in any of the above embodiments.

[0119] The computer-readable media of this embodiment includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0120] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the transmission line detection method based on multimodal data fusion as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0121] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0122] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of the present disclosure, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. Furthermore, devices may be shown in block diagram form to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the implementation details of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (i.e., such details should be fully understood by those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without or with variations in these specific details. Accordingly, these descriptions should be considered illustrative rather than restrictive.

[0123] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the discussed embodiments.

[0124] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A transmission line detection method based on multimodal data fusion, characterized in that: include: Performing scene segmentation on the spatial region of the transmission line based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity; Acquire original multimodal data of the power transmission line; wherein the original multimodal data includes sensor data and drone data having location information of the power transmission line; Filtering the original multimodal data based on a preset distance condition to obtain target multimodal data of a target area in a target time period; wherein the target area includes a first scene area in the scene area where the target position is located and a second scene area adjacent to the first scene area, and the target time period includes a target time; Obtaining a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding position information; Performing feature extraction on the target multimodal data to obtain target multimodal features; Performing feature fusion on the target multimodal features based on the graph structure to obtain a global fusion feature; An operating state of the target location of the transmission line at the target time is determined based on the global fusion feature.

2. The method according to claim 1, characterized in that Obtaining a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding position information, including: Determining a node of the graph structure based on each target multimodal data; In response to the spatial distance between the nodes being less than a preset threshold, constructing an edge between the nodes, and determining an edge weight of the edge based on the data type of the nodes and the spatial distance; The nodes and the edges are traversed based on a preset weight rule to update the node weights of the nodes and / or the edge weights of the edges.

3. The method according to claim 2, characterized in that The edge weights are: , where w ij For nodes i and neighbor nodes j The edge weights of For nodes i and neighbor nodes j The data type relationship, ; For nodes i and neighbor nodes j The spatial distance between is a parameter that controls the speed of weight decay.

4. The method according to claim 3, characterized in that Performing feature fusion on the target multimodal features based on the graph structure to obtain a global fusion feature, including: Determining an attention coefficient of the node based on the node weight, the edge weight, and the target multimodal feature; Performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature; Performing global optimization on the node fusion features to obtain the global fusion features; The attention coefficient includes: ， ， ， in, is the attention coefficient of node i and neighbor node j, exp is the exponential function, is the set of neighbor nodes of node i, j and k are the serial numbers of neighbor nodes, is the activation function, is the attention vector, W is the weight matrix, is the node weight of node i and the weighted multimodal features of the target, is the node weight of neighbor node j and the weight of target multimodal features, h k weighted is the node weight of neighbor node k and the weighted multimodal features of the target, is the feature splicing operation, is the edge weight between node i and neighbor node j, w ik is the edge weight between node i and its neighbor node k.

5. The method according to claim 4, characterized in that Performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature, including: ， in, is the node fusion feature of node i, is a non-linear activation function.

6. The method according to claim 5, characterized in that Performing global optimization on the node fusion features to obtain the global fusion features includes: Initializing the label distribution of the node; Updating the label distribution based on message transmission between the node and neighboring nodes until a preset condition is met to obtain the global fusion feature; Wherein, updating the label distribution based on the message transmission between the node and the neighboring nodes until a preset condition is satisfied, and obtaining the global fusion feature further includes: The node transmits messages to neighbor nodes, where the message between node i and neighbor node j is , is the label of node i, is the label of neighbor node j, is the transfer item between node i and its neighbor node j, is the label distribution of node i; Updating the label distribution based on the message includes: ,in, is the updated label distribution of node i, is the node item of node i, Z i is the normalization constant; Obtaining the global fusion feature based on the updated label distribution includes: ',in, is the global fusion feature of node i.

7. The method according to claim 1, characterized in that The spatial region of the transmission line is segmented based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity, including: Determining a first penalty value based on vertical distances between two pixel points in the remote sensing image data and the corresponding nearest transmission lines, and whether the nearest transmission lines corresponding to the two pixel points belong to the same transmission line network segment; Determining a second penalty value based on vertical distances between two pixel points and corresponding nearest towers in the remote sensing image data, and whether the two pixel points and the corresponding nearest towers belong to the same tower; Determining a priori information distance based on the sum of the first penalty value and the second penalty value; A distance metric is obtained based on the prior information distance, the color distance of the two pixels, and the spatial distance; wherein the spatial distance is obtained based on the geographic information of the two pixels; Pixel points belonging to the same scene area are determined based on the distance metric.

8. A transmission line detection device based on multimodal data fusion, characterized in that: include: A region division module is used to segment the spatial region of the transmission line based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity; A data acquisition module, configured to acquire original multimodal data of the power transmission line; wherein the original multimodal data includes sensor data and drone data having location information of the power transmission line; a data screening module, configured to screen the original multimodal data based on a preset distance condition to obtain target multimodal data of a target area in a target time period; wherein the target area includes a first scene area where a target position is located in the scene area and a second scene area adjacent to the first scene area, and the target time period includes a target time; A graph structure module, configured to obtain a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding position information; A feature extraction module, configured to extract features from the target multimodal data to obtain target multimodal features; A feature fusion module, configured to fuse the target multimodal features based on the graph structure to obtain a global fusion feature; A state detection module is used to determine the operating state of the target position of the transmission line at the target time based on the global fusion feature.

9. The device according to claim 8, characterized in that Obtaining a graph structure representing a relationship between the target multimodal data based on the target multimodal data and corresponding position information, including: Determining a node of the graph structure based on each target multimodal data; In response to the spatial distance between the nodes being less than a preset threshold, constructing an edge between the nodes, and determining an edge weight of the edge based on the data type of the nodes and the spatial distance; The nodes and the edges are traversed based on a preset weight rule to update the node weights of the nodes and / or the edge weights of the edges.

10. The device according to claim 9, characterized in that The edge weights are: , where w ij For nodes i and neighbor nodes j The edge weights of For nodes i and neighbor nodes j The data type relationship, ; For nodes i and neighbor nodes j The spatial distance between is a parameter that controls the speed of weight decay.

11. The device according to claim 10, characterized in that Performing feature fusion on the target multimodal features based on the graph structure to obtain a global fusion feature, including: Determining an attention coefficient of the node based on the node weight, the edge weight, and the target multimodal feature; Performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature; Performing global optimization on the node fusion features to obtain the global fusion features; The attention coefficient includes: ， ， ， in, is the attention coefficient of node i and neighbor node j, exp is the exponential function, is the set of neighbor nodes of node i, j and k are the serial numbers of neighbor nodes, is the activation function, is the attention vector, W is the weight matrix, is the node weight of node i and the weighted multimodal features of the target, is the node weight of neighbor node j and the weight of target multimodal features, h k weighted is the node weight of neighbor node k and the weighted multimodal features of the target, is the feature splicing operation, is the edge weight between node i and neighbor node j, w ik is the edge weight between node i and its neighbor node k.

12. The device according to claim 11, characterized in that Performing feature fusion on the target multimodal features of the node based on the attention coefficient to obtain a node fusion feature, including: ， in, is the node fusion feature of node i, is a non-linear activation function.

13. The device according to claim 12, characterized in that Performing global optimization on the node fusion features to obtain the global fusion features includes: Initializing the label distribution of the node; Updating the label distribution based on message transmission between the node and neighboring nodes until a preset condition is met to obtain the global fusion feature; Wherein, updating the label distribution based on the message transmission between the node and the neighboring nodes until a preset condition is satisfied, and obtaining the global fusion feature further includes: The node transmits messages to neighbor nodes, where the message between node i and neighbor node j is , is the label of node i, is the label of neighbor node j, is the transfer item between node i and its neighbor node j, is the label distribution of node i; Updating the label distribution based on the message includes: ,in, is the updated label distribution of node i, is the node item of node i, Z i is the normalization constant; Obtaining the global fusion feature based on the updated label distribution includes: ',in, is the global fusion feature of node i.

14. The device according to claim 8, characterized in that The spatial region of the transmission line is segmented based on geographic information and remote sensing image data to obtain multiple scene regions with spatial continuity, including: Determining a first penalty value based on vertical distances between two pixel points in the remote sensing image data and the corresponding nearest transmission lines, and whether the nearest transmission lines corresponding to the two pixel points belong to the same transmission line network segment; Determining a second penalty value based on vertical distances between two pixel points and corresponding nearest towers in the remote sensing image data, and whether the two pixel points and the corresponding nearest towers belong to the same tower; Determining a priori information distance based on the sum of the first penalty value and the second penalty value; A distance metric is obtained based on the prior information distance, the color distance of the two pixels, and the spatial distance; wherein the spatial distance is obtained based on the geographic information of the two pixels; Pixel points belonging to the same scene area are determined based on the distance metric.

15. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

16. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

Images