Component segmentation method and system based on Hough decomposition and multi-modal fusion

Abstract

The invention discloses a part segmentation method and system based on Hough decomposition and multi-modal fusion, and the method consists of five core steps: multi-modal data collection and preprocessing, Hough decomposition-driven division and quantization feature extraction, component-level structured multi-modal fusion, structural sensing component collaborative modeling and segmentation output. According to the method, the Hough decomposition theory is introduced into a multi-modal segmentation network for the first time, each modal characteristic field is orthogonally decomposed into a gradient component, a rotation component and a harmonic component, and deep collaboration and dynamic fusion of different modal information under the same component are realized in a component level structure space for different component characteristics. And finally obtaining an electric power part segmentation result with accurate boundary, clear structure and strong space consistency. The prior advantages of physical structure components in multi-mode perception are fully exerted, and the component identification precision, robustness and interpretability can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent segmentation of power components, and specifically to a component segmentation method and system based on Hodge decomposition and multimodal fusion, especially for power component segmentation and component defect detection in industrial scenarios. Background Art

[0002] The current mainstream technology solutions for multimodal segmentation of power components are usually based on structures such as deep convolutional neural networks or Transformers, and use multi-source data such as visible light, infrared images, and lidar point clouds as input. Each modal data is first preliminarily represented by the backbone feature extraction network. Then, in the intermediate layer or segmentation head stage of the segmentation network, the multimodal features are comprehensively utilized through feature splicing, weighted fusion, channel attention, spatial attention, cross attention, etc. The fused features are then processed through decoding, upsampling, and other processes to finally output pixel-level segmentation results. The advantage of this type of solution is that it can fully utilize the complementary information of each modal data and improve the segmentation accuracy and robustness in complex environments. Therefore, it is widely used in practical scenarios such as power inspection and intelligent identification.

[0003] In recent years, in order to further improve the ability of modal collaborative perception, some methods have introduced multi-level, multi-scale attention mechanisms or Transformer structures into the network structure to achieve deeper feature interaction. However, regardless of the fusion strategy adopted, the existing methods are essentially still based on the linear superposition of features in high-dimensional space or the adjustment of self-attention weights. The fusion process fails to distinguish different types of structural information within the multimodal features, especially at the physical level such as component boundaries, rotational structures, and global consistency. There is a lack of targeted structural modeling. In general, the current fusion method of multimodal segmentation networks is still mainly based on overall feature mixing, which makes it difficult to achieve effective decoupling of the physical level and component-level collaboration for complex structural semantics, becoming the main technical bottleneck for further improving segmentation performance. That is, since these existing multimodal segmentation methods only use traditional methods such as feature splicing, weighted fusion or attention allocation, they often find it difficult to fully distinguish and express the physical structure level within multimodal data.

[0004] In actual power inspection scenarios, however, the boundaries between components often become blurred due to lighting variations, noise interference, and complex backgrounds. Local structural details and overall spatial relationships are easily weakened or even lost during the fusion process. Furthermore, common multimodal cross-attention mechanisms are mostly limited to information interaction between modal entities and lack targeted modeling of specific structural features. This makes segmentation models less than ideal when dealing with complex morphologies and multi-source noise, limiting the application of multimodal segmentation methods to power components.

[0005] With the continuous improvement of the demand for automation and intelligent inspection of power systems, higher requirements are placed on the high-precision intelligent segmentation of power components in complex scenarios. However, in practical applications, due to factors such as the complex on-site environment, diverse component structures and significant multimodal data noise, the existing multimodal segmentation methods based on deep learning still have obvious deficiencies in structural semantic decoupling, fine boundary recognition and spatial consistency. Specifically, in the process of multimodal data fusion, the local boundaries and complex structures of components are easily disturbed by inter-modal noise and redundant information, resulting in blurred boundaries, missing structural information or misjudgment in the segmentation results, which makes it difficult to meet the actual needs of high-precision identification and anomaly detection of key components in power inspections. Therefore, how to effectively distinguish and coordinate various structural semantics in a multimodal environment to improve the accuracy, robustness and structural interpretability of power component segmentation is a core technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The present invention aims to address the aforementioned power component segmentation accuracy issues, particularly the lack of structural interpretability, by providing a component segmentation method and system based on Hodge decomposition and multimodal fusion. This method is not limited to power component segmentation but can be extended to other fields and scenarios. The present invention's technical solution is applicable to any segmentation task involving multimodal image input and sensitive to structural information. For example, component defect detection in industrial scenarios, organ or lesion segmentation in medical images, or the extraction of structures such as roads and buildings in remote sensing images can all be achieved using component-level modeling and fusion methods. Specifically, the present invention proposes incorporating Hodge decomposition into a multimodal feature fusion and segmentation network. Using learnable differential operators and structure-aware convolutional kernels, the fused multimodal features are decomposed into distinct components with clear physical meaning, corresponding to the target's structural boundaries, local variations, and global spatial information. Furthermore, combined with a component-level multimodal cross-attention mechanism, the method achieves precise coordination and dynamic fusion of various structural features, significantly improving the segmentation network's ability to perceive complex power component structures, spatial relationships, and boundaries, effectively overcoming the shortcomings of existing methods in terms of structural representation and noise suppression.

[0007] To this end, the present invention provides the following technical solutions:

[0008] A component segmentation method based on Hodge decomposition and multimodal fusion includes the following steps:

[0009] Multimodal data acquisition, i.e. collecting multimodal images of the target scene for segmentation;

[0010] Hodge decomposition-driven quantitative feature extraction: For each modal image, three parallel gradient component extraction subnets, a curl component extraction subnet, and a harmonic component extraction subnet are used to extract a type of Hodge decomposition feature corresponding to each modal image. The Hodge decomposition feature includes gradient component, curl component, and harmonic component.

[0011] Component-level structured multimodal fusion and structure perception;

[0012] The component-level structured multimodal fusion is to fuse the same type of Hodge decomposition features of different modalities to obtain the fusion features corresponding to the gradient component, the curl component, and the harmonic component respectively; and the structural perception is to perform structural perception on the features of the gradient component, the curl component, and the harmonic component;

[0013] Feature fusion, fusing current features to obtain comprehensive structure-aware features;

[0014] Segmentation output: using the comprehensive structural perception feature as the input of the segmentation head to obtain the component segmentation result of the segmentation target scene.

[0015] Further optionally, the mathematical model of the gradient component extraction subnet is expressed as:

[0016]

[0017] Where, is the gradient component feature map, is the gradient component extraction operator, is the gradient convolution weight matrix, is the bias term of the mathematical model corresponding to the gradient component; represents the input image, that is, the modal image corresponding to the input, Representing modal information, wherein the first convolution kernel of the gradient component extraction subnet is initialized with the goal of focusing on the structural features corresponding to the gradient component;

[0018] The mathematical model of the curl component extraction subnet is expressed as:

[0019]

[0020] Where, is the curl component feature map, is the curl component extraction operator, is the curl convolution weight matrix, is the bias term of the mathematical model corresponding to the curl component, wherein the first convolution kernel of the curl component extraction subnet is initialized with the goal of focusing on the structural features corresponding to the curl component

[0021] The mathematical model of the harmonic component extraction subnet is expressed as:

[0022]

[0023] Where, is the harmonic component characteristic diagram, is the harmonic component extraction operator, is the harmonic convolution weight matrix, It is a global average pooling, in which the first convolution kernel of the harmonic component extraction subnet is initialized with the goal of focusing on the structural features corresponding to the harmonic components.

[0024] Optionally, a component-level Selective Attention fusion mechanism is introduced to fuse the same type of Hodge decomposition features of different modalities using the SelectiveAttention operator to obtain the fusion features corresponding to the gradient component, curl component, and harmonic component respectively.

[0025] The mathematical model of the Selective Attention operator is expressed as:

[0026]

[0027] Where, represents the Selective Attention operator for a class of Hodge decomposition feature j, and Represents the mode and modal The corresponding type of Hodge decomposition feature j; j is the category label of the Hodge decomposition feature, corresponding to the gradient component, curl component, and harmonic component; 、 Both are learnable linear transformation matrices, corresponding to the gradient components of the two modes respectively; Represents feature splicing; 、 are all attention mechanism parameters; σ represents nonlinear activation.

[0028] Further optionally, if component-level structured multimodal fusion is performed first, and then structure perception and feature fusion are performed, the generation process of the comprehensive structure perception feature is:

[0029] After component-level structured multimodal fusion, the fusion features of the gradient component, the curl component, and the harmonic component are respectively input into the structure perception branch network corresponding to each type of Hodge decomposition feature for structure perception and feature extraction, which is expressed as:

[0030]

[0031] Where, 、 、 They represent the features extracted after the fusion features of the gradient component, the curl component, and the harmonic component are passed through the structure-aware branch network. 、 、 Represent the fusion features of gradient component, curl component and harmonic component respectively; 、 、 They represent structure-aware branch networks respectively;

[0032] The features extracted by the three types of structure-aware branch networks are then weighted and fused according to the learnable weights to obtain the comprehensive structure-aware features, which can be expressed as:

[0033]

[0034] in, 、 、 are all adaptive fusion weights obtained through end-to-end training. It is a comprehensive structural perception feature.

[0035] Further optionally, the component segmentation method pre-constructs a sample set of the target scene to be segmented, and performs model training on the network model based on the sample set and a designed loss function, thereby segmenting the target scene to be segmented;

[0036] Among them, in addition to measuring the predicted segmentation mask and the true label In addition to the loss index of similarity between the two, the loss function also includes structure-aware loss , used to constrain the expressive power of fractional features on spatial structure, the structure-aware loss The component loss is used to ensure that the three types of structural components, namely gradient, curl and harmonic, maintain physical consistency and orthogonality in spatial distribution and expression, and prevent structural aliasing.

[0037] Alternatively, the structure-aware loss Expressed as:

[0038]

[0039]

[0040] Where, is the structural orthogonality loss, is the energy attributable loss, , are the Hodge decomposition features of the i-th and j-th categories after component-level structured multimodal fusion, where i, j are grad, curl, and harm, respectively, representing the gradient component, the curl component, and the harmonic component; represents the square of the L2 norm (Euclidean norm), which is used to measure the distance or difference between two features. To represent the original overall features of the input image, that is, the undecomposed overall fusion features before the fractional feature extraction driven by Hodge decomposition, 、 、 They represent the features extracted after the fusion features of the gradient component, curl component, and harmonic component are passed through the structure-aware branch network.

[0041] Further optionally, the modal image is divided into a visible light image and an infrared image; the components are power components, parts of industrial scenes, roads or buildings in remote sensing images, and organs or lesions in medical images.

[0042] In addition, the technical solution of the present invention also provides a system based on the component segmentation method, comprising a multimodal data acquisition module, a quantitative feature extraction module, a component-level structured multimodal fusion module, a structure-aware component fusion module, and a segmentation output module, which are sequentially or mutually connected;

[0043] Multimodal data acquisition module, used to acquire multimodal images of the segmented target scene;

[0044] A quantitative feature extraction module is used to extract a type of Hodge decomposition feature corresponding to each modal image through three parallel gradient component extraction subnets, curl component extraction subnets, and harmonic component extraction subnets, respectively. The Hodge decomposition feature includes a gradient component, a curl component, and a harmonic component;

[0045] The component-level structured multimodal fusion module is used to fuse the same type of Hodge decomposition features of different modalities to obtain the fusion features corresponding to the gradient component, curl component, and harmonic component respectively;

[0046] The structure-aware component fusion module is used to perform structural perception on the features of gradient components, curl components, and harmonic components;

[0047] The structure-aware component fusion module is also used for feature fusion, that is, fusing current features to obtain comprehensive structure-aware features;

[0048] The segmentation output module is used to use the comprehensive structural perception feature as the input of the segmentation head to obtain the component segmentation result of the segmentation target scene.

[0049] The technical solution of the present invention further provides a computer device, comprising:

[0050] one or more processors and memory storing one or more computer programs;

[0051] The processor calls the computer program to implement: a step of a component segmentation method based on Hodge decomposition and multimodal fusion.

[0052] The technical solution of the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is called by a processor to implement: steps of a component segmentation method based on Hodge decomposition and multimodal fusion.

[0053] Beneficial effects

[0054] 1. The technical solution of the present invention proposes a component segmentation method based on Hodge decomposition and multimodal fusion. Its core formula is to decouple and collaboratively model the structural information in the multimodal fusion feature field at the physical component level. Among them, the multimodal features are decomposed into gradient components, curl components and harmonic components, which correspond to the boundary structure, local deformation and global consistency information of the segmented components respectively. Subsequently, for different component features, deep collaboration and dynamic fusion of different modal information under the same type of components are achieved in the component-level structural space, effectively enhancing the expression ability of key structural information. Furthermore, the three types of structural features are deeply modeled separately through the structure-aware branch network, and component-level fusion is achieved in the output stage, and finally the segmentation results of the segmented components with precise boundaries, clear structures and strong spatial consistency are obtained. The overall solution breaks through the limitations of traditional multimodal segmentation methods that only rely on feature splicing or overall attention fusion, gives full play to the a priori advantages of physical structural components in multimodal perception, and can significantly improve the component recognition accuracy, robustness and interpretability in complex scenarios of power systems.

[0055] 2. In practical applications, the technical solution of this invention has broad potential for promotion and outstanding results. It is not limited to power component segmentation but is also applicable to any segmentation task that uses multimodal images as input and is sensitive to structural information. In actual power inspection scenarios, the boundaries between components are often blurred due to lighting changes, noise interference, and complex backgrounds. Local structural details and overall spatial relationships are easily weakened or even lost during the fusion process. To this end, the technical solution of the present invention introduces the Hodge decomposition theory into the multimodal segmentation network for the first time. For each modal feature field, it is orthogonally decomposed into gradient component, curl component and harmonic component. The gradient component corresponds to the edge and local texture structure of the image, the curl component corresponds to the structural features such as rotation, bending, and winding in the image, and the harmonic component represents spatial structures such as global consistency and background redundancy. Based on this, the technical solution of the present invention uses three components to realize structured modeling at the physical level such as component boundaries, rotation structures and global consistency, thereby improving structural interpretability. Finally, the model can accurately identify key structures such as the boundaries of power components, local deformation of insulators, and wire winding in substation scenarios under complex backgrounds and multi-source noise interference conditions, and achieve high-precision pixel-level segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of the overall architecture of the bimodal component-level structured segmentation network based on Hodge decomposition.

[0057] Figure 2 This is a diagram showing the connection relationship of the hardware components of the computer device provided by the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to illustrate the present invention and are not intended to limit the present invention. The technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0059] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] In order to achieve high-precision structured segmentation of power components in complex power scenarios, this paper proposes a multimodal component-level structured segmentation network based on Hodge decomposition. The overall structure of the dual-modal component-level structured segmentation network, taking visible light images and infrared images as examples, is shown in the figure. Figure 1 As shown in the figure, it consists of five core steps, and the technical ideas are as follows: Step S1 is multimodal data acquisition and preprocessing, Step S2 is quantitative feature extraction driven by Hodge decomposition, Step S3 is component-level structured multimodal fusion, Step S4 is structure-aware component collaborative modeling, and Step S5 is segmentation output. It should be noted that the structure diagram is a single-layer structure. In actual application, the modules of Steps S2 to S4 can be stacked into a multi-layer progressive structure according to task requirements to further improve the model's multi-scale structure modeling capabilities and segmentation accuracy; and the execution order of Steps S3 and S4 can also be adaptively adjusted.

[0062] The present invention will be described in detail below with reference to specific embodiments, wherein power components are used as an example for segmentation. It should be understood that the technical concept of the present invention is not limited to the application field of segmentation objects only in the power field.

[0063] Example 1

[0064] Step S1: Multimodal data acquisition: The purpose of this step is to obtain multimodal image data for segmenting the target scene.

[0065] In this embodiment, the multimodal image data acquired by the on-site multi-sensor platform are visible light images and infrared images. Conventional image preprocessing methods such as scaling, cropping, and normalization are used for the two types of images originally collected to uniformly adjust them to the same spatial size and pixel distribution. After preprocessing, a visible light image with strict one-to-one correspondence between pixels in space is obtained. and infrared images , H is the height of the image, W is the width of the image, The above processing provides a standardized input basis for subsequent feature component extraction and multimodal collaborative fusion, effectively ensuring the consistency and comparability of the data.

[0066] In other feasible embodiments, multimodal image data is not limited to the aforementioned image types, and the specific preprocessing techniques are not limited to those implemented in this embodiment. In actual applications, the modal image type and preprocessing techniques can be set according to application requirements, and the present invention does not impose specific limitations on this.

[0067] Step S2: Quantitative feature extraction based on Hodge decomposition.

[0068] In two-dimensional space, the Hodge decomposition theorem states that any vector field Can be uniquely decomposed into gradient component (no rotation), curl component (no dispersion) and harmonic component (no rotation and no dispersion), and its mathematical expression is:

[0069]

[0070] in, is the gradient component, corresponding to the irrotational structure of the field, reflecting local boundary and texture changes; is the curl component, is the curl potential function or vector potential field, The divergence-free structure of the corresponding field characterizes deformations such as local rotation and winding. During the feature extraction phase, the present invention proposes a Hodge decomposition-driven quantitative feature extraction subnetwork for the first time. This involves constructing a three-way parallel component subnetwork for each modality after the original image input, directly extracting the three characteristic components of the corresponding physical structure. It should be understood that any component subnetwork capable of extracting the corresponding gradient, curl, and harmonic components will meet the requirements of the present invention.

[0071] In order to encourage the three component sub-networks to focus on the characteristic components of the corresponding physical structures, the embodiment of the present invention preferably sets the initialization method of the first-layer convolution kernel to guide the sub-networks to pay attention to the characteristics of different physical structures. For example, for the gradient component, the curl component and the harmonic component, the differential kernel, the rotation kernel and the smoothing kernel are used to guide the attention to different structures respectively. This method can effectively achieve functional structural decoupling; secondly, it is also preferred to add structural loss and orthogonality constraints during training to guide each sub-network to focus on extracting the corresponding structural components. At this time, there are no rigid requirements on the structure of the three types of sub-networks. For example, a layered convolution stacking structure can be used. Specific explanation can be found below.

[0072] S2.1: Gradient component extraction subnet. The purpose of this gradient component extraction subnet is to extract the gradient component in the Hodge decomposition features of each modality image.

[0073] The gradient component corresponds to the edge and local texture structure of the image, which can be expressed as:

[0074]

[0075] Where, is the gradient component feature map, is the gradient component extraction operator, is the gradient convolution weight matrix, is the bias term of the mathematical model corresponding to the gradient component. represents the input image, that is, the modal image corresponding to the input, Indicates modal information. In this embodiment, m can be or , representing visible light or infrared, and then expanded into:

[0076]

[0077] Where, , They are the gradient component feature maps corresponding to visible light and infrared modality images, , They are the gradient component extraction operators corresponding to visible light and infrared modal images, , Respectively, visible light and infrared modality images; , are the gradient convolution weight matrices corresponding to visible light and infrared modality images respectively; , are the bias terms corresponding to the gradient components of visible light and infrared modality images respectively.

[0078] In this embodiment, a first-order difference form of learnable convolution kernel is used to approximate the gradient operator of the continuous field. The reason is that the first-order difference form can better simulate the gradient changes in the image, especially the edge structure. Initialize as a first-order difference convolution mask, such as in the horizontal direction:

[0079]

[0080] Or vertically:

[0081]

[0082] Where, and They are convolution kernels in the horizontal and vertical directions respectively, that is, the horizontal and vertical convolution kernels are set in the same gradient subnet at the same time to simultaneously extract the horizontal and vertical edge features in the image.

[0083] It should be understood that the technical solution of the present invention does not limit the initial value of the weight It must be in the form of first-order differences. In principle, as long as the convolution kernel can effectively converge to the directional features that express image edges or texture changes during training, the requirements can be met. Because this component essentially extracts the irrotational (gradient) information in the image, it focuses on areas with drastic structural changes such as component boundaries and insulation damage. For example, in the technical solution of the present invention for this subnet, its first-layer convolution kernel is initialized to a gradient directional difference kernel (such as the Sobel form), which guides the subnet to respond to image edges and detailed structures in the early stage of the network. Secondly, in other embodiments, it is also preferred to enhance its response strength in areas such as boundaries and structural mutations through guidance in the loss function.

[0084] In this embodiment, according to and Assign weights to the convolution kernel , subsequent parameter updates can be adaptively optimized through end-to-end training to best approximate the physical gradient structure of the modal image. Among them, end-to-end training adaptive optimization means that the gradient subnet is not trained separately, but as part of the entire backbone network, it participates in forward propagation and reverse gradient updates simultaneously during the overall network training process. In other words, the initial convolution kernel is set to the first-order difference form only to provide a priori guidance on the physical structure. In the actual training process, its weight parameters and bias Both are learnable and are jointly optimized and updated through the main loss function.

[0085] S2.2 Curl Component Extraction Subnet. The purpose of this curl component extraction subnet is to extract the curl component from the Hodge decomposition features of each modal image. The core design of the curl subnet is to introduce a loop difference kernel or a convolution kernel with a rotationally symmetric structure during the initialization phase to guide the network to respond to information such as local rotation, curvature, and texture distortion in the image. For example, commonly used Laplace rings and Sobel cross kernels are both curl-sensitive.

[0086] The curl component corresponds to the structural features of the image, such as rotation, bending, and winding, and is expressed as:

[0087]

[0088] Where, is the curl component feature map, is the curl component extraction operator, is the curl convolution weight matrix, is the bias term of the mathematical model corresponding to the curl component. This embodiment is for visible light or infrared images, and is further expanded as:

[0089]

[0090] Where, , They are the curl component feature maps corresponding to visible light and infrared modal images, , They are the curl component extraction operators corresponding to visible light and infrared modal images respectively; , are the curl convolution weight matrices corresponding to visible light and infrared modal images respectively; , are the bias terms corresponding to the downward rotation components of visible light and infrared modality images, respectively.

[0091] In this embodiment, a learnable curl convolution kernel is used in the sub-network to approximate the two-dimensional curl operator. Initialize to templates such as loop differences, convolution groups, rotationally symmetric filters, such as the curl approximation operator for a two-dimensional field or a higher-order loop difference kernel. Here, you can initialize to the following 3×3 discrete curl operator to perceive the local rotation and loop structure of the image:

[0092]

[0093] Where, This is the convolution kernel for the curl component; subsequent weights can be adaptively trained. This component essentially extracts non-divergent but rotational structure, focusing on local rotational deformations such as cable bending and component winding. It should be understood that the above convolution kernel design is a heuristic prior guidance method, aimed at helping the subnet quickly focus on rotational structure or local curvature information in the image during the initial training phase. This is merely an example from the perspective of effectiveness, and the technical solution of the present invention is not limited to this type of convolution kernel design.

[0094] S2.3 Harmonic Component Extraction Subnet. The purpose of this harmonic component extraction subnet is to extract the harmonic components from the Hodge decomposition features of each modal image. The harmonic components mainly correspond to the background consistency and slowly changing areas in the image. The present invention guides the subnet to focus on low-frequency and smooth area features by initializing with a large-size mean convolution kernel or a Gaussian kernel and combining it with global average pooling. During training, the harmonic subnet is decoupled from the gradient and curl subnets. It extracts the residual part after removing the edge and rotation information, thereby achieving effective modeling of the harmonic features.

[0095] The harmonic component represents the spatial structure of global consistency and background redundancy. In theory, it is the residual part that is neither irrotational nor divergent, and is expressed as:

[0096]

[0097] Where, is the harmonic component characteristic diagram, is the harmonic component extraction operator, is the harmonic convolution weight matrix, This is global average pooling. This embodiment is for visible light or infrared images, and is further expanded as follows:

[0098]

[0099] Where, , They are the harmonic component feature maps corresponding to visible light and infrared modal images, , They are the harmonic component extraction operators corresponding to visible light and infrared modal images respectively; , They are the harmonic convolution weight matrices corresponding to visible light and infrared modality images respectively.

[0100] In this embodiment, global average pooling and large receptive field low-pass convolution kernel are used for joint extraction. Initialized to a large-size mean kernel or Gaussian kernel, which can be trained later; global pooling ensures consistent response to the global space. This component focuses on the global scene structure and low-frequency background. It should be understood that in other feasible embodiments, it is not necessary to set it to a large-size mean kernel or Gaussian kernel. This embodiment is a design adopted to enhance the response to low-frequency and background consistency areas. In principle, as long as the convolution kernel or subnet structure can effectively extract the low-frequency and smoothly changing features in the image and weaken the edge and local rotation information, the modeling requirements of the harmonic component can be met. Therefore, other forms of smoothing kernels or global context modeling structures (such as long-distance attention, low-pass filters, etc.) can also be used instead.

[0101] The above three components respectively approximate the physical gradient component (no rotation, reflecting the structural boundary), the curl component (no divergence, representing the local rotation change), and the harmonic component (no rotation and no divergence, capturing the global consistency) in the Hodge decomposition. Through structured learnable convolution weights, end-to-end spatial physical component extraction is achieved, giving each branch of the network clear physical semantics and structural interpretability. It should be noted that in obtaining 、 and After the Hodge decomposition features of these three pairs of infrared and visible light are obtained, they already possess basic gradient, curl, and harmonic characteristics, and can proceed to step S3 for fusion. However, to obtain deeper features, it is often necessary to further enhance the features of each pair of features using the corresponding component extraction subnet. This allows the fusion results of more layers of Hodge decomposition to be obtained in step S3. This embodiment first illustrates the technical solution of the present invention by taking only one layer of extraction as an example.

[0102] Step S3: Component-level structured multimodal fusion

[0103] This phase innovatively proposes a component-level Selective Attention fusion mechanism. Specifically, each type of physical structure component (such as gradient, curl, and harmonics) interacts and fuses only with components of the same type in different modalities, thereby achieving precise alignment, complementarity, and enhancement of structural information.

[0104] This embodiment takes the gradient component as an example, and the features after the gradient components of different modes are fused are Expressed as:

[0105]

[0106] in, The Selective Attention operator designed for the gradient component is implemented as follows:

[0107]

[0108] 、 Both are learnable linear transformation matrices, used for the gradient components of the two modes respectively. 、 It is just a conventional fully connected layer weight matrix, which is used to linearly map the input features to facilitate cross-modal feature alignment and attention fusion. 、 As long as the basic linear transformation ability, learnability and dimension matching requirements are met, the use requirements of this method can be met. Represents feature splicing; 、 is the attention mechanism parameter; σ represents nonlinear activation (such as Sigmoid or Softmax).

[0109] The curl component and the harmonic component use the same structure, and only the input features need to be replaced. The mathematical model is expressed as:

[0110]

[0111] Where, and They are the features after fusion of the curl components of different modes and the features after fusion of the harmonic components of different modes; and These are SelectiveAttention operators designed for the curl component and harmonic component, respectively. It should be understood that each Selective Attention structure can be set as a set of independent parameters, or it can share some weights to enhance structural consistency. The essence of component-level Selective Attention is to perform weighted interaction only on the information between similar physical components, avoid the aliasing of different physical structure features, and enhance structural interpretability. The fusion method is not limited to simple weighted splicing, but can also use modules such as deep separable convolution and structure-aware gating to enhance the dynamic modeling capability of information. This mechanism significantly improves the complementary recognition capability of complex structures in multimodal scenarios, and can achieve efficient and accurate alignment and enhancement of power component boundaries, rotational deformations, and global backgrounds.

[0112] It should be understood that Selective Attention falls under the category of attention mechanisms. The corresponding structure of the mathematical model described above differs from existing mainstream approaches such as multi-head self-attention, channel attention, spatial attention, gated fusion, or graph attention. Instead, it is a customized fusion structure based on feature concatenation and linear mapping, used to achieve fine alignment and dynamic enhancement of modal features at the physical structure component level. In other feasible embodiments, the fusion mechanism in step S3 may also adopt other attention structures, such as multi-head self-attention, spatial / channel attention, gated fusion, graph convolution, etc., or employ operators such as feature concatenation and weighted averaging for information fusion.

[0113] Step S4: Component-aware structure modeling and co-segmentation.

[0114] The three fused physical components (gradient, curl, and harmonics) are fed into dedicated structure-aware branch networks to further extract and enhance their respective structural representations. Each structure-aware branch can include deep structures such as multi-layer local convolution, pyramid pooling, and Transformer blocks to capture complex spatial relationships from fine-grained to global scales.

[0115]

[0116] in, 、 、 The following structures are used for implementation: stacked local convolutional blocks (such as ResBlock and DilatedConv) to extract spatial details and boundary information; modules such as spatial pyramid pooling (SPP) and / or pyramid attention (PAM) to enhance multi-scale semantic perception capabilities; and structured Transformer modules (such as SwinTransformer and ViT) to strengthen long-range dependencies and structural globality. After the structure-aware branch, the three component features respectively complete high-level semantic aggregation and structural enhancement, fully capturing boundary changes, rotational deformation, and background consistency in complex scenes. It should be understood that deep structures such as multi-layer local convolution, pyramid pooling, and Transformer blocks and their connection relationships can be selectively selected, replaced, or adjusted based on the application object and accuracy requirements.

[0117] Finally, the above three types of structure-aware components are weighted and fused according to learnable weights to obtain the final comprehensive structure-aware feature expression:

[0118]

[0119] in, 、 、 All are adaptive fusion weights obtained through end-to-end training, which automatically balance the contribution of each structural component in different tasks and scenarios.

[0120] The above comprehensive perception characteristics This serves as the input to the segmentation head, which outputs the final pixel-level segmentation mask for the power components. This collaborative, structure-aware modeling and fusion of components not only improves the ability to express and distinguish various physical structural information, but also enhances the segmentation model's interpretability and adaptability to different scenarios.

[0121] It should be noted that some implementation processes (such as component-level fusion and structural perception) can be performed in parallel, cascaded, or integrated according to actual engineering requirements, and the process sequence can be flexibly adjusted. For example, in some embodiments, structural perception is performed first, followed by multimodal fusion. It is also important to emphasize that component extraction and physical attribution are always based on Hodge decomposition theory. Therefore, in actual engineering implementation, in addition to using Hodge decomposition for core component modeling, other alternative solutions can all be engineering implementations and structural expansions based on Hodge decomposition and fall within the scope of protection of this invention.

[0122] Step S5: Segmentation output and optimization

[0123] The fused structure-aware features Input to the segmentation head (such as U-Net, DeepLab, Mask R-CNN and other mainstream segmentation decoder structures), through multi-scale upsampling and pixel-level classification, and finally output the segmentation mask of the power component .

[0124] During model training, the loss function not only uses the conventional pixel-level cross entropy loss , to measure the difference between the predicted segmentation mask and the true label The similarity between them is also combined with the introduction of structure-aware loss , which is specifically used to constrain the expressive power of fractional features in spatial structure. Ultimately, the joint optimization objective can be expressed as:

[0125]

[0126] Among them, structure-aware loss It is a component loss, which is used to ensure that the three types of structural components, namely gradient, curl, and harmonic, maintain physical consistency and orthogonality in spatial distribution and expression, and prevent structural aliasing. 、 These are weight coefficients that can be adjusted according to specific tasks and datasets, taking into account both pixel-level accuracy and structural component representation capabilities.

[0127] In this embodiment, during the structure perception and decoding phase, a dedicated structure perception branch is designed for each component (multi-layer convolution, pooling, Transformer, etc. can be used), so that each type of physical component can obtain multi-scale structure enhancement from local to global. Finally, the three-component features are fused as the input of the segmentation head, specifically introducing the structural orthogonal loss. and energy attributable losses , to maintain the orthogonality between components and the uniqueness of component attribution, specifically:

[0128]

[0129] Finally, the structural loss is obtained:

[0130]

[0131] Where, , They are the iii-th and jj-th Hodge decomposition features after component-level structured multimodal fusion, where i and j can be "grad" (gradient component), "curl" (curl component), and "harm" (harmonic component), respectively; represents the square of the L2 norm (Euclidean norm), which is used to measure the distance or difference between two features. To represent the original overall features of the input image, that is, the undecomposed overall fusion features before the fractional feature extraction driven by Hodge decomposition. Structural orthogonal loss and energy attributable losses All of these belong to component losses. It should be understood that structural orthogonality loss, energy attribution loss, structural consistency loss, etc. can be implemented using a variety of metrics such as inner product minimization, cosine similarity, and contrast loss. All loss designs that can effectively constrain the physical independence of component structures fall within the scope of protection of this invention.

[0132] Through these physical mechanisms and loss constraints, the model can achieve spatial attribution and de-redundancy of physical components, ensure the uniqueness and interpretability of structural expression, and effectively suppress noise and feature aliasing.

[0133] The solution of the present invention can be widely used in intelligent inspection and component identification scenarios of power systems. Similarly, it can also be applied to any type of segmentation task that takes multimodal images as input and is sensitive to structural information. Taking the on-site inspection of substations as an example, the system synchronously collects visible light images and infrared images of power equipment through a multi-sensor platform. After preprocessing such as scaling, cropping, and normalization, a multimodal input with one-to-one pixel correspondence in space is obtained. Subsequently, the quantitative feature modeling network based on Hodge decomposition described in the present invention is used to decouple the physical structure of the gradient component, curl component, and harmonic component of the input image, and realize multimodal Selective Attention fusion at the component level. Ultimately, the model can accurately identify key structures such as the boundaries of power components, local deformation of insulators, and winding of wires in substation scenarios under complex backgrounds and multi-source noise interference conditions, and achieve high-precision pixel-level segmentation. In actual application, the method of the present invention demonstrates deep collaboration and adaptive modeling capabilities for inter-modal structural information, which can effectively improve the recognition accuracy, structural interpretability and spatial consistency of segmentation results of power components in harsh environments, and provide a reliable structured perception foundation for downstream tasks such as power operation and maintenance, equipment detection and safety monitoring.

[0134] Example 2

[0135] An embodiment of the present invention provides a system based on the above-mentioned component segmentation method, including a multimodal data acquisition module, a quantitative feature extraction module, a component-level structured multimodal fusion module, a structure-aware component fusion module and a segmentation output module that are connected in sequence or interconnected.

[0136] Among them, the multimodal data acquisition module is used to acquire multimodal images of the segmented target scene.

[0137] The quantitative feature extraction module is used to extract a type of Hodge decomposition feature corresponding to each modal image through three parallel gradient component extraction subnets, curl component extraction subnets, and harmonic component extraction subnets. The Hodge decomposition feature includes gradient component, curl component, and harmonic component.

[0138] The component-level structured multimodal fusion module is used to fuse the same type of Hodge decomposition features of different modalities to obtain the fusion features corresponding to the gradient component, curl component, and harmonic component respectively.

[0139] The structure-aware component fusion module is used to perform structural perception on the features of gradient component, curl component and harmonic component; the structure-aware component fusion module is also used for feature fusion, that is, fusing current features to obtain comprehensive structure-aware features.

[0140] The segmentation output module is used to use the comprehensive structure perception feature as the input of the segmentation head to obtain the segmentation result of the segmentation target scene.

[0141] It should also be understood that the specific implementation process of each module please refer to the above method content, the present invention will not go into details here, and the division of the above functional modules is only for example illustration. In some embodiments, some functional modules can be merged, and some functional modules can be split. Each functional module can be implemented in software or hardware or a combination of software and hardware. Among them, software and hardware equipment include but are not limited to general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0142] Example 3

[0143] An embodiment of the present invention provides a computer device, comprising: one or more processors and a memory storing one or more computer programs;

[0144] The processor calls the computer program to implement the steps of a component segmentation method based on Hodge decomposition and multimodal fusion. For the specific implementation process of each step, please refer to the description of the embodiment of the component segmentation method based on Hodge decomposition and multimodal fusion.

[0145] In some embodiments, as Figure 2 As shown, the electronic components of computer equipment include:

[0146] Processor 1600 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. Processor 1600 is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0147] Memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in memory 1700, and the processor 1600 calls and executes the algorithm program of the component segmentation method based on Hodge decomposition and multimodal fusion according to the embodiment of the present invention.

[0148] The input / output interface 1800 is used to implement information input and output.

[0149] The communication interface 1900 is used to realize communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0150] The bus 2000 transmits information between various components of the device (eg, the processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 ).

[0151] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0152] Example 4

[0153] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is called by a processor to implement: steps of a component segmentation method based on Hodge decomposition and multimodal fusion.

[0154] For the specific implementation process of each step, please refer to the description of the aforementioned embodiment of the component segmentation method based on Hodge decomposition and multimodal fusion.

[0155] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard disk equipped on the controller, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Furthermore, the readable storage medium can also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0156] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0157] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is a flow chart according to the method, device (system), and computer program product of the embodiment of the present application and / or the instructions executed by the processor to generate a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product comprising an instruction device, which realizes the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0158] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A component segmentation method based on Hodge decomposition and multimodal fusion, characterized by: The following steps are involved: Multimodal data acquisition, i.e. collecting multimodal images of the target scene for segmentation; Hodge decomposition-driven quantitative feature extraction: For each modal image, three parallel gradient component extraction subnets, a curl component extraction subnet, and a harmonic component extraction subnet are used to extract a type of Hodge decomposition feature corresponding to each modal image. The Hodge decomposition feature includes gradient component, curl component, and harmonic component. Component-level structured multimodal fusion and structure perception; The component-level structured multimodal fusion is to fuse the same type of Hodge decomposition features of different modalities to obtain the fusion features corresponding to the gradient component, the curl component, and the harmonic component respectively; and the structural perception is to perform structural perception on the features of the gradient component, the curl component, and the harmonic component; Feature fusion, fusing current features to obtain comprehensive structure-aware features; Segmentation output: using the comprehensive structural perception feature as the input of the segmentation head to obtain the component segmentation result of the segmentation target scene.

2. The component segmentation method according to claim 1, wherein: The mathematical model of the gradient component extraction subnet is expressed as: ； Where, is the gradient component feature map, is the gradient component extraction operator, is the gradient convolution weight matrix, is the bias term of the mathematical model corresponding to the gradient component; represents the input image, that is, the modal image corresponding to the input, Representing modal information, wherein the first convolution kernel of the gradient component extraction subnet is initialized with the goal of focusing on the structural features corresponding to the gradient component; The mathematical model of the curl component extraction subnet is expressed as: ； Where, is the curl component feature map, is the curl component extraction operator, is the curl convolution weight matrix, is the bias term of the mathematical model corresponding to the curl component, wherein the first convolution kernel of the curl component extraction subnet is initialized with the goal of focusing on the structural features corresponding to the curl component; The mathematical model of the harmonic component extraction subnet is expressed as: ； Where, is the harmonic component characteristic diagram, is the harmonic component extraction operator, is the harmonic convolution weight matrix, It is a global average pooling, in which the first convolution kernel of the harmonic component extraction subnet is initialized with the goal of focusing on the structural features corresponding to the harmonic components.

3. The component segmentation method according to claim 1, wherein: A component-level Selective Attention fusion mechanism is introduced, and the Selective Attention operator is used to fuse the same type of Hodge decomposition features of different modes to obtain the fusion features corresponding to the gradient component, curl component, and harmonic component respectively; The mathematical model of the Selective Attention operator is expressed as: ； Where, represents the Selective Attention operator for a class of Hodge decomposition feature j, and Represents the mode and modal The corresponding type of Hodge decomposition feature j; j is the category label of the Hodge decomposition feature, corresponding to the gradient component, curl component, and harmonic component; 、 Both are learnable linear transformation matrices, corresponding to the gradient components of the two modes respectively; Represents feature splicing; 、 are all attention mechanism parameters; σ represents nonlinear activation.

4. The component segmentation method according to claim 1, wherein: If component-level structured multimodal fusion is performed first, followed by structure perception and feature fusion, the generation process of the comprehensive structure perception feature is as follows: After component-level structured multimodal fusion, the fusion features of the gradient component, the curl component, and the harmonic component are respectively input into the structure perception branch network corresponding to each type of Hodge decomposition feature for structure perception and feature extraction, which is expressed as: ； Where, 、 、 They represent the features extracted after the fusion features of the gradient component, the curl component, and the harmonic component are passed through the structure-aware branch network. 、 、 Represent the fusion features of gradient component, curl component and harmonic component respectively; 、 、 They represent structure-aware branch networks respectively; The features extracted by the three types of structure-aware branch networks are then weighted and fused according to the learnable weights to obtain the comprehensive structure-aware features, which can be expressed as: ； in, 、 、 are all adaptive fusion weights obtained through end-to-end training. It is a comprehensive structural perception feature.

5. The component segmentation method according to claim 1, wherein: The component segmentation method pre-builds a sample set of the target scene to be segmented, and trains the network model based on the sample set and the designed loss function, thereby segmenting the target scene to be segmented; Among them, in addition to measuring the predicted segmentation mask and the true label In addition to the loss index of similarity between the two, the loss function also includes structure-aware loss , used to constrain the expressive power of fractional features on spatial structure, the structure-aware loss It is a component loss, which is used to ensure that the three types of structural components, namely gradient, curl, and harmonic, maintain physical consistency and orthogonality in spatial distribution and expression.

6. The component segmentation method according to claim 5, wherein: Structure-aware loss Expressed as: ； ； Where, is the structural orthogonality loss, is the energy attributable loss, , are the Hodge decomposition features of the i-th and j-th categories after component-level structured multimodal fusion, where i, j are grad, curl, and harm, respectively, representing the gradient component, the curl component, and the harmonic component; represents the square of the L2 norm, which is used to measure the distance or difference between two features. To represent the original overall features of the input image, that is, the undecomposed overall fusion features before the fractional feature extraction driven by Hodge decomposition, 、 、 They represent the features extracted after the fusion features of the gradient component, curl component, and harmonic component are passed through the structure-aware branch network.

7. The component segmentation method according to claim 1, wherein: The modal images are divided into visible light images and infrared images; the components are power components, parts of industrial scenes, roads or buildings in remote sensing images, and organs or lesions in medical images.

8. A system based on the component segmentation method according to any one of claims 1 to 7, characterized in that: It includes a multimodal data acquisition module, a quantitative feature extraction module, a component-level structured multimodal fusion module, a structure-aware component fusion module, and a segmentation output module that are sequentially or mutually connected; Multimodal data acquisition module, used to acquire multimodal images of the segmented target scene; A quantitative feature extraction module is used to extract a type of Hodge decomposition feature corresponding to each modal image through three parallel gradient component extraction subnets, curl component extraction subnets, and harmonic component extraction subnets, respectively. The Hodge decomposition feature includes a gradient component, a curl component, and a harmonic component; The component-level structured multimodal fusion module is used to fuse the same type of Hodge decomposition features of different modalities to obtain the fusion features corresponding to the gradient component, curl component, and harmonic component respectively; The structure-aware component fusion module is used to perform structural perception on the features of gradient components, curl components, and harmonic components; The structure-aware component fusion module is also used for feature fusion, that is, fusing current features to obtain comprehensive structure-aware features; The segmentation output module is used to use the comprehensive structural perception feature as the input of the segmentation head to obtain the component segmentation result of the segmentation target scene.

9. A computer device, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: The steps of the component segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of the component segmentation method according to any one of claims 1 to 7.

Images