Transformer-based power multi-modal full-factor sample fusion labeling method and system

Abstract

The application relates to a Transformer-based power multi-modal full-element sample fusion labeling method and system, which comprises the following steps: S1: obtaining multi-modal original data of a production operation site, and obtaining a preprocessed multi-modal data set through preprocessing; S2: performing multi-modal feature coding and Transformer representation modeling according to the preprocessed multi-modal data, and obtaining a fusion feature representation through cross-modal feature fusion and information completion; S3: acquiring a multi-modal semantic alignment label set through semantic consistency detection and a semantic mapping mechanism according to the fusion feature representation and a corresponding candidate label set; S4: constructing a label system structure tree according to the multi-modal semantic alignment label set; and S5: constructing an automatic labeling module to generate preliminary labeling based on the fusion feature representation and the label system structure tree, and directly inferring labels based on similar samples and category probabilities. The application realizes efficient automatic label generation and sample dynamic classification.

Description

A Transformer-based method and system for fusion and annotation of full-element samples in multimodal power systems. Technical Field

[0001] This invention relates to the field of data annotation, and in particular to a Transformer-based method and system for fusion annotation of multimodal full-element samples in the power industry. Background Technology

[0002] With the accelerating digitalization and intelligentization of power systems, enterprises have accumulated a wealth of diverse and rich field data in their daily production, operation and maintenance, equipment repair, accident handling, and safety management activities. This data includes not only traditional text data such as operation tickets and work records, but also images and videos generated at the equipment operation site, as well as structured parameters and environmental information collected by various sensors. In-depth mining and high-quality annotation of multimodal data have become the core foundation driving power AI applications such as intelligent operation and maintenance, intelligent analysis, and knowledge graphs.

[0003] However, power production operation data exhibits strong modal heterogeneity, with significant differences in data representation, information granularity, and structural patterns. This makes it difficult for traditional single-method annotation to accommodate various data characteristics, easily leading to problems such as information loss, label ambiguity, or low annotation efficiency. Meanwhile, the power industry has extremely stringent requirements for data accuracy, demanding not only comprehensive and standardized annotation results but also semantic consistency and traceability. This places higher demands on the intelligent annotation process for multimodal data. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a Transformer-based method and system for fusion and labeling of multimodal power samples, enabling efficient automated label generation and dynamic sample classification.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The Transformer-based method for fusion and annotation of full-element samples of multimodal power systems includes the following steps:

[0007] S1: Acquire multimodal raw data from the production site and preprocess it to obtain a preprocessed multimodal dataset;

[0008] S2: Based on the preprocessed multimodal data, perform multimodal feature encoding and Transformer representation modeling, and obtain the fused feature representation through cross-modal feature fusion and information completion;

[0009] S3: Based on the fused feature representation and its corresponding candidate label set, obtain the multimodal semantic alignment label set through semantic consistency detection and semantic mapping mechanism;

[0010] S4: Align the tag set according to multimodal semantics and construct the tag architecture tree;

[0011] S5: Based on the fusion of feature representation and label architecture tree, an automated annotation module is built to generate preliminary annotations, and labels are directly inferred based on similar samples and class probabilities.

[0012] Furthermore, the multimodal raw data includes text data, image data, video data, and structured data; the text data includes work instructions, safety regulations, accident reports, and inspection records; the image data includes equipment status photos, work site images, and infrared thermal images; the video data includes work process recordings, monitoring videos, and training videos; and the structured data includes sensor data, equipment parameters, and environmental monitoring data.

[0013] Further preprocessing is performed as follows: First, a multi-source data access standard is established, and text data is encoded uniformly, special characters are processed, and terminology is standardized to ensure the parsability of the text content; for image data, noise reduction, brightness correction, resolution unification, and background interference removal are performed; for video data, frame rate unification, key frame extraction, and timestamp calibration are performed, and intelligent segmentation is carried out according to the characteristics of the work scenario; a unified metadata standard is established, and attribute tags such as timestamp, location information, data source, and quality score are added to each data sample.

[0014] Furthermore, for the video data, frame rate unification, keyframe extraction, and timestamp calibration are performed, as follows:

[0015] First, perform frame rate normalization on all videos. Let the original video be V. i The frame rate is f i The unified target frame rate is f t Then the uniform frame interval Δt is defined as:

[0016] ;

[0017] For any frame sequence {F1,F2,...,F... n If f i >f t Then uniform downsampling is performed:

[0018] ;

[0019] Among them, F k This represents the k-th frame in the original video frame sequence; f represents the k-th frame obtained after frame rate conversion; i f is the frame rate of the original video. t is the target frame rate; n is the total number of frames in the original video;

[0020] Indicates the floor function;

[0021] Conversely, if f i <f t Then, linear interpolation is used to insert virtual frames on the time axis;

[0022] ;

[0023] in, This represents a virtual frame generated using linear interpolation, located between the k-th and (k+1)-th frames; α is the interpolation coefficient, which determines the position of the new frame between the two original frames, with a value range of [0,1].

[0024] Output video sequence with unified frame rate Combining the inter-frame difference method with depth feature similarity evaluation, let the video frame set be... Then the difference metric D between the two frames k Defined as:

[0025] ;

[0026] Where H and W are the image height and width, respectively;

[0027] To improve robustness, a feature similarity S for convolutional networks is introduced. k The cosine distance is used for calculation:

[0028] ;

[0029] Among them, S k Indicates the kth and Inter-frame depth feature dissimilarity; is the high-dimensional feature vector extracted from the k-th frame by a deep neural network; This represents the dot product of two vectors. The norm of a vector;

[0030] The keyframe scoring function is defined by combining spatial difference and semantic difference:

[0031] Q k =λ1D k +λ2S k ;

[0032] Among them, Q k λ represents the score of the k-th frame; λ1 and λ2 represent the weighting coefficients;

[0033] When Q k ≥τ key ,τ key Based on a preset threshold, this frame is identified as a keyframe;

[0034] Output keyframe set: ; This is the keyframe for the km-th frame;

[0035] The device timestamp is T dev The synchronization reference time is T. ref The time offset is then defined as:

[0036] ;

[0037] The calibrated video frame timestamps are :

[0038] ;

[0039] Among them, T k For video frame timing before calibration;

[0040] When video equipment has time drift error Linear interpolation correction is used:

[0041] ;

[0042] Where a and b are the drift parameters obtained by sample pairing fitting; The time drift error of the k-th frame; t k Given the absolute timestamps of k frames; output a sequence of video frames with globally standardized timestamps. .

[0043] Furthermore, intelligent segmentation is performed based on the characteristics of the work scenario, as follows:

[0044] Intelligent video segmentation decomposes continuous video streams into segments with complete semantics or action units based on the semantics of power operation scenarios. Segmentation is based on factors including visual change rate, action detection results, and scene semantic similarity, and a comprehensive segmentation judgment function is defined:

[0045] ;

[0046] Among them, D t For inter-frame pixel difference intensity; S t For inter-frame semantic similarity; P t The overlap of the confidence level of the target action category; α, β, γ are weighting coefficients, satisfying α+β+γ=1;

[0047] When J tExceeding the threshold τ seg At that time, it is determined that this is a scene boundary, and a video segment index is automatically generated; an LSTM model is introduced to process the inter-frame change sequence {J}. t The model captures the transition trends of action stages from a global temporal perspective, achieving semantically consistent segmentation results and outputting a set of intelligent video segments.

[0048] Seg={(S1,E1),(S2,E2),...,(Sn,En)};

[0049] Where (Sn, En) represent the start and end frame indices or time range of the nth video segment, respectively.

[0050] Furthermore, multimodal feature encoding and Transformer representation modeling are performed, and fused feature representation is obtained through cross-modal feature fusion and information completion, as follows: semantic vectors of text modality are extracted through language Transformer model; spatial-temporal feature encoding of image and video modality is achieved through visual Transformer; structured data is transformed into time series feature expression through embedding mapping layer, and finally cross-modal attention association is constructed in fusion layer to obtain fused feature representation.

[0051] Furthermore, based on the fused feature representation and its corresponding candidate label set, a multimodal semantically aligned label set is obtained through semantic consistency detection and semantic mapping mechanisms, as follows:

[0052] A semantic consistency detection mechanism is introduced. By calculating the semantic similarity at the attribute level, sentence vector level, and concept level, the deviation of different modalities in tag semantics is automatically detected and corrected, thereby achieving the unification of multi-source tag semantics.

[0053] The semantic consistency detection is based on the fusion feature vector E of each sample. i and candidate tag set

[0054] L cand ={l1,l2,…,l m};

[0055] Among them, l m Let m be the label of the m-th sample;

[0056] Evaluate each modal sub-feature E a To determine the semantic matching degree of candidate labels, first calculate the modal semantic embedding vector v. mod With the semantic embedding vector v of the label l Cosine similarity:

[0057] ;

[0058] Among them, Ea Let l be the fused feature vector of the a-th multimodal data sample; b For the b-th candidate label; v mod Let be the semantic embedding feature vector of the 'a'-th sample under modality mod; For fusion feature E a With candidate label l b Cosine similarity under modal mod;

[0059] Through a multimodal weighting strategy:

[0060] ;

[0061] Where M is the set of all involved modes; α mod These are the modal weighting coefficients; To synthesize the multimodal weighted semantic consistency score, the overall semantic consistency score between each fused sample and the candidate label is obtained.

[0062] After obtaining the semantic matching degree between the fused samples and candidate labels, the semantic mapping stage begins. This involves constructing a multimodal semantic graph to achieve semantic-level mapping alignment between labels. The semantic graph is defined as follows:

[0063] G=(V,E),V={v fusion ,v ab ,...,v AB};

[0064] Where node v fusion Represents the semantic center of sample fusion, node v ab The semantic vector representing the candidate label ab; v AB The semantic vector representing the candidate label ab;

[0065] To further enhance the robustness of the mapping, a semantic projection matrix W is introduced. map Mapping the label embedding space to the sample feature space:

[0066] ;

[0067] in, Labels projected onto the sample feature space b semantic vector; W is the semantic vector of the original label lb; map The following is obtained by training while minimizing semantic alignment loss:

[0068] ;

[0069] That is, to maximize semantic similarity while minimizing feature distance, thereby ensuring alignment between label semantics and sample feature space. For multimodal semantic alignment loss; The trade-off weight hyperparameter for the loss term;

[0070] Using the maximum weight matching algorithm of semantic graphs, the optimal mapping is selected from all candidate labels. :

[0071] ;

[0072] Among them, L cand For the candidate tag set; w ab The semantic association weight between the a-th sample and the b-th label.

[0073] Furthermore, by combining semantic hierarchical clustering and knowledge graph mapping, a tag structure tree is constructed. Hierarchical connections between tags are established using semantic similarity, hierarchical relationships, and scene co-occurrence criteria, enabling semantic tags to form a semantic hierarchy from concept to task, as detailed below:

[0074] First, the semantic embedding vector v of each tag is... l The mapping matrix W learned during the semantic alignment stage map Its unified semantic representation is obtained:

[0075] ;

[0076] Any two tags p and l q The semantic similarity between them is defined as:

[0077] ;

[0078] in, Label l p and l q A unified semantic representation;

[0079] In the similarity matrix Sim(l p ,l q Based on this, through cluster analysis, the tags are automatically divided into several semantic clusters, with each cluster corresponding to one level of the tree;

[0080] To determine the hierarchical relationship between tags, a semantic inclusiveness metric is introduced:

[0081] ;

[0082] Where Context(l) represents the semantic context set of label l; when C(l) p ,lq )>τ c And Sim(l p ,l q )>τ s At that time, it was believed that l p It is l q The higher-level concept node is used to establish a directed edge l. p →l q , where τ c , τ s The preset threshold is used; after iterative merging and edge optimization, a tree structure T=(V,E) is formed, where V is the set of nodes and E is the set of semantically dependent edges.

[0083] After hierarchical aggregation and relational constraints, a preliminary label semantic tree T0 is obtained. Semantic consistency optimization and hierarchical weight update mechanisms are used for dynamic adjustment, and the tree structure scoring function is defined as follows:

[0084] ;

[0085] Where ω1 and ω2 are weight parameters; edge set E represents all parent-child semantic mapping relationships;

[0086] By using a greedy optimization algorithm, S(T) is maximized to obtain the final optimal semantic tree.

[0087] Furthermore, an automated annotation module is constructed to generate preliminary annotations, and labels are directly inferred based on similar samples and class probabilities, as detailed below:

[0088] Tag-based architecture tree Based on the structural relationships in the code, establish a set of rule triples R={(lp,r,lq)}, where r represents the logical relation;

[0089] For input sample features E a When the semantic feature quantification index meets the predefined conditions, the system automatically triggers the corresponding rule to generate a tag set. :

[0090] ;

[0091] Where, Φ r For rule triggering function; E a Let be the fusion feature representation of the a-th sample;

[0092] Based on the rule results, a supervised model is introduced to predict label probabilities for E. a The predicted probability distribution for each label category is as follows:

[0093] P(l b |E a )=fθ (E a );

[0094] Where, f θ For the trained prediction model; determine the initial labels based on the class probability maximization rule or threshold screening strategy:

[0095] ;

[0096] in, The predicted label with the highest probability for the a-th sample;

[0097] When multiple labels have similar probabilities and are adjacent in semantic level, use fusion to determine the appropriate label.

[0098] ;

[0099] in, Let P(l) be the set of labels predicted by the model for the a-th sample, with high confidence and similar level to the main label; b |E a ) represents the a-th sample belonging to label l b The predicted probability; τ p Depth(l) is the threshold for label confidence. b ) for label l b Hierarchical depth in the tag architecture tree; The hierarchy depth of the main tag in the tag structure tree; ∧ represents a logical AND relationship.

[0100] After integrating the above annotation results, a comprehensive tag set is generated:

[0101] ;

[0102] in, This represents a fusion function based on confidence weighting and semantic hierarchy constraints;

[0103] Label confidence is determined by the following formula:

[0104]

[0105] Where η1 and η2 represent weighting coefficients; P rule (l b The label l is obtained based on rule-based reasoning. b Confidence level;

[0106] P pred (l b ) represents the label predicted by the model. b The probability score;

[0107] The final automated annotation result set output by the system is as follows:

[0108] .

[0109] The Transformer-based power multimodal full-element sample fusion annotation system includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the Transformer-based power multimodal full-element sample fusion annotation method as described above.

[0110] The present invention has the following beneficial effects:

[0111] 1. This invention integrates multimodal data from the production operation site, including multi-source information such as images, voice, text, and sensors. Through systematic preprocessing and feature encoding, it achieves comprehensive perception of all elements and multiple perspectives of the power operation scenario. Compared with traditional single-modal or weak fusion methods, it can effectively capture multi-dimensional information that affects the power operation process and safety judgment, and greatly improve the richness and representation ability of sample data.

[0112] 2. This invention adopts an end-to-end modeling mechanism based on Transformer to perform deep representation and cross-modal fusion of multimodal features, which effectively improves the information complementarity and missing information completion capabilities between different modalities. At the same time, by introducing semantic consistency detection and cross-modal label semantic mapping mechanism, it solves the problems of label ambiguity, poor naming consistency and difficulty in multi-label fusion, and ensures the accuracy and interpretability of multimodal semantic alignment label set.

[0113] 3. This invention introduces a label architecture tree based on semantic alignment results, and combines it with a rule engine, intelligent prediction, and similarity inference to form a hierarchical, traceable, and self-growing label management system. The standardized label tree structure can adapt to the increasingly complex and dynamically changing labeling needs of power operations, while realizing efficient automated label generation and dynamic sample classification. The design of the automated labeling module not only significantly reduces the cost of manual labeling, but also improves the intelligence level of label generation and knowledge management through empty sample mining and confidence fusion mechanisms. Attached Figure Description

[0114] Figure 1 is a flowchart of the method of the present invention. Detailed Implementation

[0115] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0116] Referring to Figure 1, in this embodiment, a Transformer-based method for fusion and annotation of full-element samples of multimodal power systems is provided, including the following steps:

[0117] S1: Acquire multimodal raw data from the production site and preprocess it to obtain a preprocessed multimodal dataset;

[0118] S2: Based on the preprocessed multimodal data, perform multimodal feature encoding and Transformer representation modeling, and obtain the fused feature representation through cross-modal feature fusion and information completion;

[0119] S3: Based on the fused feature representation and its corresponding candidate label set, obtain the multimodal semantic alignment label set through semantic consistency detection and semantic mapping mechanism;

[0120] S4: Align the tag set according to multimodal semantics and construct the tag architecture tree;

[0121] S5: Based on the fusion of feature representation and label architecture tree, an automated annotation module is built to generate preliminary annotations, and labels are directly inferred based on similar samples and class probabilities.

[0122] In this embodiment, the multimodal raw data includes text data, image data, video data, and structured data; the text data includes work instructions, safety procedures, accident reports, and inspection records; the image data includes equipment status photos, work site images, and infrared thermal images; the video data includes work process recordings, monitoring videos, and training videos; and the structured data includes sensor data, equipment parameters, and environmental monitoring data.

[0123] In this embodiment, the preprocessing is as follows: First, a multi-source data access standard is established, and text data is encoded uniformly, special characters are processed, and terminology is standardized to ensure the parsability of the text content; noise reduction, brightness correction, resolution unification, and background interference removal are performed on image data; for video data, frame rate unification, key frame extraction, and timestamp calibration are performed, and intelligent segmentation is carried out according to the characteristics of the work scenario; a unified metadata standard is established, and attribute tags such as timestamp, location information, data source, and quality score are added to each data sample.

[0124] In this embodiment, the video data undergoes frame rate unification, keyframe extraction, and timestamp calibration, as detailed below:

[0125] Due to significant differences in frame rates among on-site video acquisition devices (such as head-mounted cameras, fixed surveillance cameras, and drone cameras), to ensure temporal alignment and model input consistency, all videos are first subjected to frame rate normalization. Let the original video V... i The frame rate is f i The unified target frame rate is f t Then the uniform frame interval Δt is defined as:

[0126] ;

[0127] For any frame sequence {F1,F2,...,F... n If f i >f t Then uniform downsampling is performed:

[0128] ;

[0129] Among them, F k This represents the k-th frame in the original video frame sequence; f represents the k-th frame obtained after frame rate transformation (downsampling / resampling); i f is the frame rate of the original video. t is the target frame rate; n is the total number of frames in the original video;

[0130] Indicates the floor function;

[0131] Conversely, if f i <f t Then, linear interpolation is used to insert virtual frames on the time axis;

[0132] ;

[0133] in, This represents a virtual frame generated using linear interpolation, located between the k-th and (k+1)-th frames; α is the interpolation coefficient, which determines the position of the new frame between the two original frames, with a value range of [0,1].

[0134] Output video sequence with unified frame rate Ensure that all videos have the same temporal resolution and inter-frame interval;

[0135] Keyframe extraction aims to remove redundant frames and retain representative frames representing moments of significant changes in action or scene. The method combines inter-frame difference analysis with deep feature similarity evaluation. Let the video frame set be... Then the difference metric D between the two frames k Defined as:

[0136] ;

[0137] Where H and W are the image height and width, respectively; Let be the feature value of the (k-1)th frame at position (x, y);

[0138] To improve robustness, a feature similarity S for convolutional networks is introduced. k The cosine distance is used for calculation:

[0139] ;

[0140] Among them, S k Indicates the kth and Inter-frame depth feature dissimilarity; This is the high-dimensional feature vector extracted from the k-th frame by a deep neural network (using a convolutional network); This represents the dot product of two vectors. The norm of a vector;

[0141] The keyframe scoring function is defined by combining spatial difference and semantic difference:

[0142] Q k =λ1D k +λ2S k ;

[0143] Among them, Q k λ represents the score of the k-th frame; λ1 and λ2 represent the weighting coefficients;

[0144] When Q k ≥τ key ,τ key Based on a preset threshold, this frame is identified as a keyframe;

[0145] Output keyframe set: ; This is the keyframe for the km-th frame;

[0146] Different acquisition devices and sensor data exhibit time drift and clock errors. To achieve multimodal synchronization, a unified time reference needs to be established, with the device timestamp being T. dev The synchronization reference time (such as SCADA system or GPS standard time) is T. ref The time offset is then defined as:

[0147] ;

[0148] The calibrated video frame timestamps are :

[0149] ;

[0150] Among them, T k For video frame timing before calibration;

[0151] When video equipment has time drift error Linear interpolation correction is used:

[0152] ;

[0153] Where a and b are the drift parameters obtained by sample pairing fitting; The time drift error of the k-th frame; t k The absolute time of k frames; the calibrated video can be precisely aligned in time with modal data such as sensor data and text reports; output a video frame sequence with a global standard timestamp. This provides a unified time reference for multimodal feature encoding.

[0154] In this embodiment, intelligent segmentation is performed based on the characteristics of the work scenario, as detailed below:

[0155] Intelligent video segmentation decomposes continuous video streams into segments with complete semantics or action units based on the semantics of power operation scenarios. Segmentation is based on factors including visual change rate, action detection results, and scene semantic similarity, and a comprehensive segmentation judgment function is defined:

[0156] ;

[0157] Among them, D t S represents the inter-frame pixel difference intensity (characterizing the rate of change in the image); t For inter-frame semantic similarity (calculated by extracting features from the Transformer); P t The overlap of the confidence level of the target action category; α, β, γ are weighting coefficients, satisfying α+β+γ=1;

[0158] When J t Exceeding the threshold τ seg At that time, it is determined that this is a scene boundary, and a video segment index is automatically generated; an LSTM model is introduced to process the inter-frame change sequence {J}. t The model captures the transition trends of action stages from a global temporal perspective, achieving semantically consistent segmentation results and outputting a set of intelligent video segments.

[0159] Seg={(S1,E1),(S2,E2),...,(Sn,En)};

[0160] Where (Sn, En) represent the start and end frame indices or time range of the nth video segment, respectively. Each segment contains a clear temporal context and semantic topic, and is independently input into the Transformer model for subsequent feature extraction and annotation. The LSTM model adopts a two-layer LSTM stacked structure: the first layer captures local inter-frame dynamics (short-term mode); the second layer aggregates global semantic trends (long-term dependencies); J is input at each time step. t After passing through two LSTM layers, the hidden state h is output. t Then, a feedforward neural network (FFN) is used to output the stage change probability p for each frame. t That is, whether the frame is a potential segmentation point.

[0161] In this embodiment, multimodal feature encoding and Transformer representation modeling are performed, and fused feature representation is obtained through cross-modal feature fusion and information completion, as follows: semantic vectors of text modality are extracted through language Transformer model; spatial-temporal feature encoding of image and video modality is achieved through visual Transformer; structured data is transformed into time series feature expression through embedding mapping layer, and finally cross-modal attention association is constructed in fusion layer to obtain fused feature representation.

[0162] In this embodiment, based on the fused feature representation and its corresponding candidate label set, a multimodal semantically aligned label set is obtained through semantic consistency detection and semantic mapping mechanisms, as follows:

[0163] In the obtained fusion feature representation Efusion, text, image, video and structured data modalities have been connected to a unified high-dimensional semantic space through Transformer features. However, the differences in content and context dimensions of each modality may still cause semantic drift of labels (such as "device failure" vs "abnormal state"). To this end, a semantic consistency detection mechanism is introduced. By calculating the semantic similarity at the attribute level, sentence vector level and concept level, the mechanism automatically detects and corrects the deviation of different modalities in label semantics, so as to achieve the unification of multi-source label semantics.

[0164] The semantic consistency detection is based on the fusion feature vector E of each sample. i and candidate tag set

[0165] L cand ={l1,l2,…,l m};

[0166] Among them, l m Let m be the label of the m-th sample;

[0167] Evaluate each modal sub-feature E a To determine the semantic matching degree of candidate labels, first calculate the modal semantic embedding vector v. mod With the semantic embedding vector v of the label l Cosine similarity:

[0168] ;

[0169] Among them, E a Let l be the fused feature vector of the a-th multimodal data sample; b For the b-th candidate label; v mod Let be the semantic embedding feature vector of the 'a'-th sample under modality mod; For fusion feature E a With candidate label l bCosine similarity under modal mod;

[0170] Through a multimodal weighting strategy:

[0171] ;

[0172] Where M is the set of all involved modes; α mod These are the modal weighting coefficients; To synthesize the multimodal weighted semantic consistency score, the overall semantic consistency score between each fused sample and the candidate label is obtained.

[0173] After obtaining the semantic matching degree between the fused samples and candidate labels, the semantic mapping stage begins. This involves constructing a multimodal semantic graph to achieve semantic-level mapping alignment between labels. The semantic graph is defined as follows:

[0174] G=(V,E),V={v fusion ,v ab ,...,v AB};

[0175] Where node v fusion Represents the semantic center of sample fusion, node v ab The semantic vector representing the candidate label ab; v AB The semantic vector representing the candidate label ab;

[0176] To further enhance the robustness of the mapping, a semantic projection matrix W is introduced. map Mapping the label embedding space to the sample feature space:

[0177] ;

[0178] in, Labels projected onto the sample feature space b semantic vector; For the original label l b semantic vectors; W map The following is obtained by training while minimizing semantic alignment loss:

[0179] ;

[0180] That is, to maximize semantic similarity while minimizing feature distance, thereby ensuring alignment between label semantics and sample feature space. For multimodal semantic alignment loss; The trade-off weight hyperparameter for the loss term;

[0181] Using the maximum weight matching algorithm of semantic graphs, the optimal mapping is selected from all candidate labels. :

[0182] ;

[0183] Among them, L cand For the candidate tag set; w ab The semantic association weight between the a-th sample and the b-th label.

[0184] In this embodiment, a tag system architecture tree is constructed by combining semantic hierarchical clustering and knowledge graph mapping. Hierarchical connections between tags are established using semantic similarity, hierarchical relationships, and scene co-occurrence criteria, enabling semantic tags to form a semantic hierarchy from the concept level to the task level. For example, "equipment operation" serves as the upper-level node, and "circuit breaker closing" and "insulator inspection" serve as lower-level nodes, thus forming an interpretable hierarchical structure. The constructed tag system can support automatic tag expansion and guide semantic reasoning and knowledge fusion, as detailed below:

[0185] First, the semantic embedding vector v of each tag is... l The mapping matrix W learned during the semantic alignment stage map Its unified semantic representation is obtained:

[0186] ;

[0187] Any two tags p and l q The semantic similarity between them is defined as:

[0188] ;

[0189] in, Label l p and l q A unified semantic representation;

[0190] In the similarity matrix Sim(l p ,l q Based on this, through cluster analysis (such as hierarchical clustering, spectral clustering, or threshold-based KNN semantic graphs), the labels are automatically divided into several semantic clusters, with each cluster corresponding to one level of the tree;

[0191] To determine the hierarchical relationship between tags, a semantic inclusiveness metric is introduced:

[0192] ;

[0193] Where Context(l) represents the semantic context set of label l (which can be extracted from co-occurring samples, text descriptions, or defined terms); when C(l) p ,lq )>τ c And Sim(l p ,l q )>τ s At that time, it was believed that l p It is l q The higher-level concept node is used to establish a directed edge l. p →l q , where τ c , τ s The preset threshold is used; after iterative merging and edge optimization, a tree structure T=(V,E) is formed, where V is the set of nodes (labels) and E is the set of semantic dependency edges (up-down relationship).

[0194] After hierarchical aggregation and relational constraints, a preliminary label semantic tree T0 is obtained. To improve the semantic rationality and generalization ability of the hierarchical structure, a semantic consistency optimization and hierarchical weight update mechanism are used for dynamic adjustment, and the tree structure scoring function is defined as follows:

[0195] ;

[0196] Where ω1 and ω2 are weight parameters; edge set E represents all parent-child semantic mapping relationships;

[0197] By using a greedy optimization algorithm (such as minimum generation of hierarchical graph), S(T) is maximized to obtain the final optimal semantic tree.

[0198] In this embodiment, an automated annotation module is constructed to generate preliminary annotations, and the labels are directly inferred based on similar samples and class probabilities, as detailed below:

[0199] Tag-based architecture tree Based on the structural relationships in the input sample features, establish a set of rule triples R={(lp,r,lq)}, where r represents the logical relationship (such as "containment", "cause and effect", "before and after"). a When the semantic feature quantification indicators meet predefined conditions (such as action intensity, text keywords, sensor thresholds), the system automatically triggers the corresponding rules to generate a tag set. :

[0200] ;

[0201] Where, Φ r For rule triggering function; E a Let be the fusion feature representation of the a-th sample;

[0202] Based on the rule-based results, a supervised model (such as Transformer classifier head, XGBoost, BERT-MLP, etc.) is introduced to predict label probabilities for E.a The predicted probability distribution for each label category is as follows:

[0203] P(l b |E a )=f θ (E a );

[0204] Where, f θ For the trained prediction model; determine the initial labels based on the class probability maximization rule or threshold screening strategy:

[0205] ;

[0206] in, The predicted label with the highest probability for the a-th sample;

[0207] When multiple labels have similar probabilities and are adjacent in semantic hierarchy (by...) When constrained, use fusion decision:

[0208] ;

[0209] in, Let P(l) be the set of labels predicted by the model for the a-th sample, with high confidence and similar level to the main label; b |E a ) represents the a-th sample belonging to label l b The predicted probability; τ p Depth(l) is the threshold for label confidence. b ) for label l b Hierarchical depth in the tag architecture tree; The hierarchy depth of the main tag in the tag structure tree; ∧ represents a logical AND relationship.

[0210] After integrating the above annotation results, a comprehensive tag set is generated:

[0211] ;

[0212] in, This represents a fusion function based on confidence weighting and semantic hierarchy constraints;

[0213] Label confidence is determined by the following formula:

[0214]

[0215] Where η1 and η2 represent weighting coefficients; P rule (l b The label l is obtained based on rule-based reasoning. b Confidence level;

[0216] P pred (l b ) represents the label predicted by the model. b The probability score;

[0217] The final automated annotation result set output by the system is as follows:

[0218] .

[0219] The Transformer-based power multimodal full-element sample fusion and annotation system includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the Transformer-based power multimodal full-element sample fusion and annotation method described above.

[0220] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0222] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0223] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0224] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A Transformer-based method for fusion and annotation of multimodal power data samples, characterized in that, The process includes the following steps: S1: Acquire multimodal raw data from the production site and preprocess it to obtain a preprocessed multimodal dataset; the multimodal raw data includes text data, image data, video data, and structured data; S2: Based on the preprocessed multimodal data, perform multimodal feature encoding and Transformer representation modeling, and obtain a fused feature representation through cross-modal feature fusion and information completion; S3: Based on the fused feature representation and its corresponding candidate label set, obtain a multimodal semantically aligned label set through semantic consistency detection and semantic mapping mechanisms; S4: Construct a label architecture tree based on the multimodal semantically aligned label set; S5: Based on the fused feature representation and label architecture tree, construct an automated annotation module to generate preliminary annotations, and directly infer labels based on similar samples and category probabilities; The process of obtaining a multimodal semantically aligned label set based on the fused feature representation and its corresponding candidate label set through semantic consistency detection and semantic mapping mechanisms is as follows: A semantic consistency detection mechanism is introduced, which automatically detects and corrects the deviations in label semantics of different modalities by calculating the semantic similarity at the attribute level, sentence vector level, and concept level, thereby achieving the unification of multi-source label semantics; The semantic consistency detection is based on the fused feature vector E of each sample. i and candidate label set L cand :L cand ={l1,l2,…,l m }; where l m For the m-th label in the candidate label set; evaluate each modal sub-feature E a To determine the semantic matching degree of candidate labels, first calculate the modal semantic embedding vector v. mod With the semantic embedding vector v of the label l Cosine similarity: Among them, E a Let l be the fused feature vector of the a-th multimodal data sample; b For the b-th candidate label; v mod Let be the semantic embedding feature vector of the 'a'-th sample under modality mod; For fusion feature E a With candidate label l b Cosine similarity under modality mod; using a multimodal weighting strategy: Where M is the set of all involved modes; α mod These are the modal weighting coefficients; To synthesize the multimodal weighted semantic consistency score, the overall semantic consistency score between each fused sample and candidate label is obtained. After obtaining the semantic matching degree between the fused sample and candidate label, the semantic mapping stage is entered. By constructing a multimodal semantic graph, semantic-level mapping alignment between labels is achieved. The semantic graph is defined as: G=(V,E), V={v fusion ,v ab ,...,v AB }; where node v fusion Represents the semantic center of sample fusion, node v ab The semantic vector representing the candidate label ab; v AB The semantic vectors of candidate labels A and B are represented; to further enhance the robustness of the mapping, a semantic projection matrix W is introduced. map Mapping the label embedding space to the sample feature space: ;in, Labels projected onto the sample feature space b semantic vector; For the original label l b semantic vector; W map The following is obtained by training while minimizing semantic alignment loss: That is, to maximize semantic similarity while minimizing feature distance, thereby ensuring the alignment of label semantics with sample feature space. For multimodal semantic alignment loss; The weighted hyperparameters represent trade-offs for the loss term; the optimal mapping is selected from all candidate labels using the maximum weight matching algorithm of the semantic graph. ： Among them, L cand For the candidate tag set; w ab The semantic association weight between the a-th sample and the b-th label.

2. The Transformer-based multimodal full-element sample fusion and annotation method for power systems according to claim 1, characterized in that, The text data includes work instructions, safety regulations, accident reports, and inspection records; the image data includes equipment status photos, work site images, and infrared thermal images; the video data includes work process recordings, monitoring videos, and training videos; and the structured data includes sensor data, equipment parameters, and environmental monitoring data.

3. The Transformer-based multimodal full-element sample fusion and annotation method for power systems according to claim 2, characterized in that, The preprocessing is as follows: First, a multi-source data access standard is established, and text data is encoded uniformly, special characters are processed, and terminology is standardized to ensure the parsability of the text content; noise reduction, brightness correction, resolution unification, and background interference removal are performed on image data; for video data, frame rate unification, key frame extraction, and timestamp calibration are performed, and intelligent segmentation is carried out according to the characteristics of the work scenario; a unified metadata standard is established, and timestamps, location information, data source, and quality score attribute tags are added to each data sample.

4. The Transformer-based multimodal full-element sample fusion and annotation method for power systems according to claim 3, characterized in that, For video data, frame rate unification, keyframe extraction, and timestamp calibration are performed as follows: First, all videos undergo frame rate normalization processing, assuming the original video V... i The frame rate is f i The unified target frame rate is f t Then the uniform frame interval Δt is defined as: For any frame sequence {F1, F2, ..., F...} n If f i >f t Then uniform downsampling is performed: ;in, F k This represents the k-th frame in the original video frame sequence; f represents the k-th frame obtained after frame rate conversion; i f is the frame rate of the original video. t is the target frame rate; n is the total number of frames in the original video; Indicates the floor function; conversely, if f i <f t Then, linear interpolation is used to insert virtual frames on the time axis; ;in, This represents a virtual frame generated using linear interpolation, located between frame k and frame (k+1); α is the interpolation coefficient, determining the position of the new frame between the two original frames, with a value range of [0,1]; the output is a video sequence with a unified frame rate. Combining the inter-frame difference method with depth feature similarity evaluation, let the video frame set be... Then the difference metric D between the two frames k Defined as: Where H and W are the image height and width, respectively; Let be the feature value of the k-th frame at position (x, y); The feature value at position (x, y) in the (k-1)th frame is used. To improve robustness, convolutional network feature similarity is introduced, calculated using cosine distance. Among them, S k This represents the dissimilarity of depth features between frames k and k-1; is the high-dimensional feature vector extracted from the k-th frame by a deep neural network; This represents the dot product of two vectors. The norm of a vector is used to represent the keyframe scoring function, which combines spatial difference and semantic difference. k =λ1D k +λ2S k Among them, Q k Represents the score of the k-th frame; λ1 and λ2 represent the weighting coefficients; when Q k ≥τ key ,τ key Based on a preset threshold, this frame is determined to be a keyframe; the keyframe set K is output: ； This is the keyframe for frame km; the device timestamp is T. dev The synchronization reference time is T. ref The time offset is then defined as: ΔT = T ref -T dev The calibrated video frame timestamps are: ： Among them, T k To calibrate the video frame time; when the video device has time drift error. Linear interpolation correction is used: Where a and b are the drift parameters obtained by sample pairing fitting; The time drift error of the k-th frame; t k Given the absolute timestamps of k frames; output a sequence of video frames with globally standardized timestamps. 。 5. The Transformer-based multimodal full-element sample fusion and annotation method for power systems according to claim 4, characterized in that, The intelligent segmentation based on the characteristics of the work scenario is as follows: Based on the semantics of the power work scenario, the continuous video stream is decomposed into segments with complete semantics or action units. The segmentation criteria include visual change rate, action detection results, and scene semantic similarity. A comprehensive segmentation judgment function J is defined. t : ; where D t For inter-frame pixel difference intensity; U t For inter-frame semantic similarity; P t The overlap of the target action category confidence; α, β, γ are weight coefficients, satisfying α + β + γ = 1; when J t Exceeding the threshold τ seg At that time, it is determined that this is a scene boundary, and a video segment index is automatically generated; an LSTM model is introduced to process the inter-frame change sequence {J}. t The model captures the transition trend of action stages from a global time perspective, achieves semantically consistent segmentation results, and outputs a set of intelligent video segments: Seg={(S1,E1),(S2,E2),...,(Sn,En)}; where (Sn,En) represent the start and end frame indices or time range of the nth video segment, respectively.

6. The method for fusion and annotation of multimodal power samples based on Transformer according to claim 1, characterized in that, The process involves multimodal feature encoding and Transformer representation modeling, followed by cross-modal feature fusion and information completion to obtain a fused feature representation. Specifically, text modalities are extracted using a language Transformer model to extract semantic vectors; image and video modalities are encoded using a visual Transformer model to achieve spatial-temporal feature encoding. Structured data is transformed into time-series feature representations through an embedding mapping layer, and finally cross-modal attention associations are constructed in the fusion layer to obtain fused feature representations.

7. The method for fusion and annotation of multimodal power samples based on Transformer according to claim 1, characterized in that, Combining semantic hierarchical clustering and knowledge graph mapping, a tag architecture tree is constructed. Hierarchical connections between tags are established using semantic similarity, hierarchical relationships, and scene co-occurrence criteria, enabling semantic tags to form a semantic hierarchy from concept to task. Specifically: First, the semantic embedding vector v of each tag is... l The mapping matrix W learned during the semantic alignment stage map Its unified semantic representation is obtained: Any two tags p and l q The semantic similarity between them is defined as: ;in, Label l p and l q A unified semantic representation; in the similarity matrix Sim(l p ,l q Based on this, cluster analysis is used to automatically divide the tags into several semantic clusters, with each cluster corresponding to one level of the tree; to determine the hierarchical relationship between tags, a semantic inclusiveness metric C(l) is introduced. p ,l q ): ; where Context(l) represents the semantic context set of label l; when C(l) p ,l q )>τ c And Sim(l p ,l q )>τ s At that time, it was believed that l p It is l q The higher-level concept node is used to establish a directed edge l. p →l q , where τ c , τ s A preset threshold is set; after iterative merging and edge optimization, a tree structure T=(V,E) is formed, where V is the node set and E is the semantic dependency edge set; after hierarchical aggregation and relation constraints, a preliminary label semantic tree T0 is obtained, which is dynamically adjusted using semantic consistency optimization and hierarchical weight update mechanisms. The tree structure scoring function S(T) is defined as follows: Where ω1 and ω2 are weight parameters; edge set E represents all parent-child semantic mapping relationships; by using a greedy optimization algorithm, S(T) is maximized to obtain the final optimal semantic tree.

8. The Transformer-based multimodal full-element sample fusion and annotation method for power systems according to claim 7, characterized in that, The automated annotation module generates preliminary annotations and directly infers labels based on similar samples and class probabilities, specifically as follows: based on the label architecture tree T Based on the structural relationships in the input sample features, establish a set of rule triples R={(lp,r,lq)}, where r represents the logical relation; a When the semantic feature quantification index meets the predefined conditions, the system automatically triggers the corresponding rule to generate a tag set. ： ; where Φ r For rule triggering function; E a Let E be the fusion feature representation of the a-th sample; based on the rule results, a supervised model is introduced to predict the label probability, for E a Predict the probability distribution P(l) of each label category b |E a ) is: P(l b |E a )=f θ (E a ); where, f θ For the trained prediction model; determine the initial labels based on the class probability maximization rule or threshold screening strategy: ;in, The predicted label with the highest probability for the a-th sample; when multiple labels have similar probabilities and are semantically adjacent, a fusion decision is used: ;in, Let P(l) be the set of labels predicted by the model for the a-th sample, with high confidence and similar level to the main label; b |E a ) represents the a-th sample belonging to label l b The predicted probability; τ p Depth(l) is the threshold for label confidence. b ) for label l b Hierarchical depth in the tag architecture tree; The hierarchy depth of the main tag in the tag structure tree; ∧ represents the logical AND relationship; after combining the above annotation results, a comprehensive tag set is generated. ： ;in, This represents a fusion function based on confidence weighting and semantic hierarchy constraints; label confidence. Determined by the following formula: Where η1 and η2 represent weighting coefficients; P rule (l b The label l is obtained based on rule-based reasoning. b confidence level; P pred (l b ) represents the label predicted by the model. b The probability score; the final automated annotation result set output by the system. for: 。 9. A Transformer-based multimodal full-element sample fusion and annotation system for electricity, characterized in that, It includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the Transformer-based power multimodal full-element sample fusion and annotation method as described in any one of claims 1-8.

Images