Multi-modal large model electric power knowledge training method for digital anchor

Abstract

The invention discloses a multi-modal large model electric power knowledge training method for a digital anchor. The method comprises the following steps: obtaining multi-modal electric power data; forming an electric power knowledge hierarchical system; constructing a knowledge dependence graph to obtain a knowledge dependence tree; constructing a multi-modal sample set; obtaining a first stage model; adopting an InfoNCE and semantic center regularization combined mechanism to obtain a second-stage model; introducing a distillation and expression anchoring method to obtain a multi-modal large model; and digital anchor-oriented customized fine tuning is carried out on the multi-modal large model to obtain a final multi-modal large model, so that hierarchical organization and multi-modal consistent learning of electric power knowledge are realized, and the method is suitable for various digital anchor application scenes of electric power science popularization, operation and maintenance training and accident recovery.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and power systems, and more particularly to a method for training power knowledge in a multimodal large model for digital broadcasters. Background Technology

[0002] With the rapid development of artificial intelligence and virtual digital human technologies, digital anchors are increasingly being widely used in media dissemination, science popularization, corporate training, and intelligent services. In the power industry, the deepening digital transformation has created a significant demand for digital anchors with professional power knowledge in application scenarios such as intelligent dispatching, smart operation and maintenance, and power training systems. However, as one of the core capabilities of digital anchors, the understanding, explanation, and multimodal presentation of power knowledge involves multiple knowledge dimensions, including the basic principles of power systems, equipment composition, system operation, dispatching strategies, and emergency response. Current technologies for intelligent explanation in the power field mainly rely on traditional text-based language models or single-modal recognition models, lacking systematic modeling of the power knowledge system and the ability to deeply integrate multimodal power data.

[0003] Existing technologies typically use general pre-trained large models or general visual models as a foundation, fine-tuning them with limited text samples to enable digital anchors to generate a certain level of explanation of power-related content. However, because they fail to incorporate the hierarchical structure and prerequisite relationships of power-related knowledge, the models often cannot accurately understand the logical system of power knowledge. For example, there is a strict upstream-downstream dependency between the fundamental principles of "voltage, current, and power" in a power system and "power flow distribution and operating modes," and the structural relationship between "primary equipment and secondary equipment" directly affects the understanding of subsequent operating modes. Summary of the Invention

[0004] One objective of this invention is to propose a multimodal large-scale power knowledge training method for digital broadcasters. This invention fully utilizes the power knowledge hierarchy system, knowledge dependency tree, and progressive multimodal joint training technology, and introduces the InfoNCE and semantic center regularization joint mechanism and distillation and expression anchoring methods to train large-scale models specifically for the power field. It has the advantages of strong professionalism, good multimodal fusion effect, and rich application scenarios.

[0005] The multimodal large-scale power knowledge training method for digital broadcasters according to embodiments of the present invention includes the following steps:

[0006] Acquire and preprocess power knowledge corpus to obtain multimodal power data;

[0007] Terms and knowledge points are extracted from the standardized power text data of multimodal power data to obtain power knowledge points and form a power knowledge hierarchy system;

[0008] Based on the chapter order of textbooks, causal descriptions in industry materials, co-occurrence relationships of knowledge points, and expert annotation results, the prerequisite relationships between various power knowledge points are obtained, a knowledge dependency graph is constructed, and a knowledge dependency tree is obtained.

[0009] Based on the power knowledge hierarchy and knowledge dependency tree, each power knowledge point is bound to corresponding multimodal data to construct a multimodal sample set;

[0010] Input the samples bound to text modality and tabular modality data from the multimodal sample set into the multimodal large model, perform task training, update the parameters of the multimodal large model, and obtain the first-stage model;

[0011] Based on the first-stage model, the text and image features of the same knowledge node are jointly trained, and the parameters of the multimodal large model are updated by using the InfoNCE and semantic center regularization joint mechanism to obtain the second-stage model.

[0012] Based on the second-stage model, the multimodal features of the same knowledge node are jointly trained, and distillation and expression anchoring methods are introduced to update the parameters of the multimodal large model and obtain the multimodal large model.

[0013] Based on the multimodal large model, the multimodal large model is customized and fine-tuned for digital anchors to obtain the final multimodal large model.

[0014] Optionally, the power knowledge corpus includes power industry standards, regulations, operation manuals, equipment manuals, accident review reports, and training materials. The preprocessing includes sentence segmentation, word segmentation, terminology standardization, unit unification, and chart structuring. The multimodal power data includes standardized power text data, tabular data, image data, and video data.

[0015] Optionally, the structure of the power knowledge hierarchy specifically includes:

[0016] The standardized power text data of multimodal power data is used to extract terms and knowledge points. The standardized power text data is segmented into sentences and words, and term recognition is performed in combination with the power field terminology list, part-of-speech tagging results and dependency relationship results. Terms related to basic power concepts and fundamental laws are tagged, as are terms related to the structural relationships of power generation equipment, substation equipment, transmission lines, distribution devices and their components. Terms related to power system operation modes, changes in operating conditions and adjustment processes of operating parameters are also tagged. Candidate power knowledge points are formed, extracted and deduplicated, and the power knowledge points are obtained.

[0017] The basic concepts and fundamental laws of electricity in the knowledge points of electricity are classified into the basic principles layer; the structural relationships of power generation equipment, power transformation equipment, transmission lines, power distribution devices and their components in the knowledge points of electricity are classified into the equipment composition layer; and the operating modes, changes in operating conditions and adjustment processes of operating parameters of power systems in the knowledge points of electricity are classified into the system operation layer.

[0018] The power knowledge points are divided into the dispatching strategy layer, which includes dispatching plans, power allocation, dispatching instructions, and operation strategy adjustment. The power knowledge points are divided into the emergency handling layer, which includes the accident occurrence, fault propagation, isolation and handling, and operation restoration process. This forms a power knowledge hierarchical system consisting of the basic principle layer, equipment composition layer, system operation layer, dispatching strategy layer, and emergency handling layer.

[0019] Optionally, the formation of the knowledge dependency tree specifically includes:

[0020] The textbook chapters, power industry technical data, and expert annotation data are traversed to record the chapter number of each power knowledge point in the textbook, the paragraph number in the industry data, and the annotation number in the expert annotation results, forming a chapter order list, a cause-and-effect description list, and an expert annotation list indexed by power knowledge points;

[0021] Based on the chapter order of the textbook, two electrical knowledge points are compared. When the chapter number of one electrical knowledge point is less than that of the other, the former is regarded as a candidate prerequisite knowledge point for the latter, forming a set of candidate prerequisite relationships based on the chapter order of the textbook.

[0022] Based on the causal descriptions in industry data, power knowledge points in the cause and effect positions are extracted from sentences containing causal trigger words in industry data. Knowledge points in the cause position are taken as prerequisite knowledge points, and knowledge points in the effect position are taken as subsequent knowledge points, forming a set of candidate prerequisite relationships based on causal descriptions.

[0023] Based on standardized power text data and exercises and case studies, the co-occurrence of power knowledge points is statistically analyzed. The ratio of the number of co-occurrences within a fixed window to the total number of windows is used as the co-occurrence degree. When the co-occurrence degree is greater than a preset threshold, the priority direction is obtained according to the order of knowledge points in the textbook, forming a set of candidate priority relationships based on co-occurrence relationships.

[0024] Based on the expert annotation results, all the prerequisite knowledge points directly provided by experts are combined into an expert-annotated prerequisite relation set. The candidate prerequisite relation sets obtained based on the textbook chapter order, causal description, and co-occurrence relationship are then merged with the expert-annotated prerequisite relation set to obtain the initial prerequisite relation set.

[0025] Using electrical knowledge points as nodes and an initial set of prerequisite relations as directed edges, a directed graph is constructed. Loop detection is performed on the directed graph. When a directed cycle consisting of prerequisite relations is detected, a directed edge is deleted from the cycle. This loop detection is repeated until no directed cycles exist in the graph, resulting in a cycle-free directed graph. In this cycle-free directed graph, knowledge points without predecessors are set to the first level, and knowledge points with predecessors are set to the level of the maximum value among all predecessor levels plus one level. The level indices of all knowledge points are obtained. Based on these level indices, the longest path is constructed from knowledge points without predecessors as the main prerequisite path. Directed relations that meet the level requirements are used as supplementary prerequisite relations, forming a knowledge dependency tree.

[0026] Optionally, the construction of the multimodal sample set specifically includes:

[0027] Based on the power knowledge hierarchy system and knowledge dependency tree, each power knowledge point is associated with the hierarchical information of its respective basic principle layer, equipment composition layer, system operation layer, dispatch strategy layer, and emergency handling layer, as well as the node identifier, predecessor node, and successor node relationship in the knowledge dependency tree, forming a set of power knowledge nodes with hierarchical attributes and dependency relationship attributes.

[0028] Each basic principle layer knowledge node in the power knowledge node set is bound to its corresponding text and tabular data to generate a multimodal knowledge node sample of the basic principle layer.

[0029] Each device component layer knowledge node in the power knowledge node set is bound to the corresponding image data to generate a multimodal knowledge node sample of the device component layer.

[0030] The system operation layer knowledge nodes of the power knowledge node set are bound to the corresponding curve diagrams, power flow diagrams and operation parameter data, and the scheduling strategy layer knowledge nodes of the power knowledge node set are bound to the corresponding scheduling table data, generating multimodal knowledge node samples of the system operation layer and multimodal knowledge node samples of the scheduling strategy layer.

[0031] Each emergency processing layer knowledge node in the power knowledge node set is bound to the corresponding video data to generate a multimodal knowledge node sample for the emergency processing layer.

[0032] All multimodal knowledge node samples generated from the basic principle layer, equipment composition layer, system operation layer, scheduling strategy layer, and emergency handling layer are uniformly organized according to the order of their prerequisite relationships in the knowledge dependency tree and their hierarchical information in the power knowledge hierarchy system to construct a multimodal sample set.

[0033] Optionally, obtaining the first-stage model specifically includes:

[0034] The first-stage training sample set is formed by selecting power knowledge point samples containing text data and tabular data from the multimodal sample set, and configuring target outputs for three types of tasks: power concept question answering, definition explanation and table parsing for each sample;

[0035] The first-stage training sample set is divided into training batches according to a preset batch size. The text data in the training batches is organized into a text input sequence, and the table data is organized into a table input sequence.

[0036] Input the text input sequence into the text encoding channel of the multimodal large model, and input the table input sequence into the table encoding channel of the multimodal large model to obtain the text encoding result and the table encoding result, and generate the power concept question answering prediction result, definition explanation prediction result and table parsing prediction result respectively;

[0037] For each sample in the training batch, the prediction results of the electricity concept question answering, definition explanation, and table parsing are compared with the corresponding target output to obtain the electricity concept question answering loss, definition explanation loss, and table parsing loss, forming the total loss of the training batch.

[0038] Based on the total loss of the training batches, update the parameters of the text encoding channel, table encoding channel, and task output module in the multimodal large model, and repeat the training for all training batches until the preset iteration conditions are met to obtain the first-stage model.

[0039] Optionally, obtaining the second-stage model specifically includes:

[0040] Based on the first-stage model, power knowledge point samples containing image data are selected from the multimodal sample set to form the second-stage training sample set, and target outputs for three types of tasks—image content recognition, image-text matching, and image explanation generation—are configured for each sample.

[0041] The second-stage training sample set is divided into training batches according to a preset batch size. The text data in each training batch is organized into a text input sequence, and the image data is organized into an image input sequence.

[0042] The text input sequence is input into the text encoding channel of the multimodal large model, and the image input sequence is input into the image encoding channel of the multimodal large model to obtain text features and image features, and respectively generate image content recognition prediction results, image-text matching prediction results and image explanation generation prediction results;

[0043] The image content recognition prediction results, image-text matching prediction results, and image explanation generation prediction results are compared with the corresponding target outputs to obtain the image content recognition loss, image-text matching loss, and image explanation generation loss, respectively. An InfoNCE and semantic center regularization joint mechanism is adopted. The InfoNCE and semantic center regularization joint mechanism constructs negative sample pairs based on the text features and image features of different power knowledge point samples. The InfoNCE contrast loss that brings the positive sample pairs closer and separates the negative sample pairs is calculated in a contrastive learning manner. According to the multimodal feature distribution of the same power knowledge point in different samples, the corresponding semantic center representation is calculated, and the multimodal features of each sample are constrained to shrink towards the semantic center to obtain the semantic center regularization loss, forming the total loss of the training batch.

[0044] Based on the total loss of the training batches, update the parameters corresponding to the text encoding channel, image encoding channel, and related task output modules in the multimodal large model. Repeat the training for all training batches until the preset iteration conditions are met to obtain the second-stage model.

[0045] Optionally, obtaining the multimodal large model specifically includes:

[0046] Based on the second-stage model, power knowledge point samples containing video modal data are selected from the multimodal sample set to form the third-stage training sample set, and target outputs for three types of tasks—operation process explanation, accident process explanation, and multimodal alignment—are configured for each sample.

[0047] The training sample set of the third stage is divided into training batches according to the preset batch size. The text data in each training batch is organized into a text input sequence, the image data is organized into an image input sequence, and the video frame sequence is organized into a video input sequence.

[0048] The text input sequence is input into the text encoding channel of the multimodal large model, the image input sequence is input into the image encoding channel of the multimodal large model, and the video input sequence is input into the video encoding channel of the multimodal large model to obtain text features, image features, and video features, and respectively generate the operation process explanation prediction results, accident process explanation prediction results, and multimodal alignment prediction results;

[0049] The prediction results of the operation process explanation, the prediction results of the accident process explanation, and the prediction results of the multimodal alignment are compared with the corresponding target outputs to obtain the operation process explanation loss, the accident process explanation loss, and the multimodal alignment loss. A distillation and expression anchoring mechanism is introduced, which compares the current model output with the preset teacher model output to obtain the distillation loss. A semantic anchoring representation is constructed based on the multimodal feature distribution of the same power knowledge point in different training samples, and the multimodal features of the current training sample are constrained to shrink towards the semantic anchoring representation to obtain the expression anchoring loss. The operation process explanation loss, the accident process explanation loss, the multimodal alignment loss, the distillation loss, and the expression anchoring loss are combined to form the total loss of the training batch.

[0050] Based on the total loss of the training batches, update the parameters corresponding to the text encoding channel, image encoding channel, video encoding channel, and the output modules of the three types of tasks in the multimodal large model. Repeat the training for all training batches until the preset iteration conditions are met to obtain the multimodal large model.

[0051] Optionally, the customized fine-tuning refers to loading the digital anchor's voice features, lip-sync driving parameters, and facial expression parameters. The multimodal large model generates voice, lip-sync, and facial expression control signals corresponding to the power knowledge content while outputting the narration text.

[0052] The beneficial effects of this invention are:

[0053] This invention constructs a hierarchical knowledge system and knowledge dependency tree for the power industry, organizing knowledge such as basic power principles, equipment composition, system operation, dispatching strategies, and emergency handling in a hierarchical and graded manner according to their prerequisite relationships. Based on multi-source evidence, it calculates the prerequisite relationships between various knowledge points, ensuring that the multimodal large model strictly follows the cognitive order of the power industry during training. This avoids the problems of knowledge jumps, causal misalignment, and conceptual confusion caused by mixed training of knowledge points in existing technologies, thus making the digital anchor's explanations in the power field more coherent, rigorous, professional, and credible.

[0054] In terms of multimodal fusion, this invention adopts a progressive training strategy that introduces text, image, and video modalities. During the joint text-image training phase, it utilizes cross-modal consistency constraints, combining InfoNCE with semantic center regularization to align and aggregate features of the same power knowledge point across different modalities, significantly reducing semantic conflicts between them. Furthermore, this invention employs cross-stage consistency loss, combining distillation constraints with representation anchoring mechanisms to suppress catastrophic forgetting while introducing new modalities, maintaining the stability of existing power knowledge representations. Based on multimodal samples constructed at the knowledge point granularity, this invention supports the synchronous and linked output of text, charts, images, and videos. This enables digital broadcasters to deliver progressively ordered explanations and synchronize semantics with visuals when conducting power science popularization, operational training, and accident debriefing, significantly enhancing its practicality and application value in the power industry. Attached Figure Description

[0055] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0056] Figure 1 This is an overall flowchart of the multimodal large-scale power knowledge training method for digital broadcasters proposed in this invention;

[0057] Figure 2 This is a schematic diagram of the structure of the second-stage model of the multimodal large-scale power knowledge training method for digital broadcasters proposed in this invention;

[0058] Figure 3 This is a schematic diagram of the structure of the multimodal large model for training power knowledge in a multimodal large model for digital broadcasters proposed in this invention. Detailed Implementation

[0059] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0060] refer to Figures 1-3 A reinforcement learning-based intelligent control method for multi-station loading and unloading of conveyor lines includes the following steps:

[0061] Acquire and preprocess power knowledge corpus to obtain multimodal power data;

[0062] Terms and knowledge points are extracted from the standardized power text data of multimodal power data to obtain power knowledge points and form a power knowledge hierarchy system;

[0063] Based on the chapter order of textbooks, causal descriptions in industry materials, co-occurrence relationships of knowledge points, and expert annotation results, the prerequisite relationships between various power knowledge points are obtained, a knowledge dependency graph is constructed, and a knowledge dependency tree is obtained.

[0064] Based on the power knowledge hierarchy and knowledge dependency tree, each power knowledge point is bound to corresponding multimodal data to construct a multimodal sample set;

[0065] Input the samples bound to text modality and tabular modality data from the multimodal sample set into the multimodal large model, perform task training, update the parameters of the multimodal large model, and obtain the first-stage model;

[0066] Based on the first-stage model, the text and image features of the same knowledge node are jointly trained, and the parameters of the multimodal large model are updated by using the InfoNCE and semantic center regularization joint mechanism to obtain the second-stage model.

[0067] Based on the second-stage model, the multimodal features of the same knowledge node are jointly trained, and distillation and expression anchoring methods are introduced to update the parameters of the multimodal large model and obtain the multimodal large model.

[0068] Based on the multimodal large model, the multimodal large model is customized and fine-tuned for digital anchors to obtain the final multimodal large model.

[0069] In this embodiment, the power knowledge corpus includes power industry standards, regulations, operation manuals, equipment manuals, accident review reports, and training materials. The preprocessing includes sentence segmentation, word segmentation, terminology standardization, unit unification, and chart structuring. The multimodal power data includes standardized power text data, tabular data, image data, and video data.

[0070] In this embodiment, the power knowledge hierarchy system specifically includes the following components:

[0071] The standardized power text data of multimodal power data is used to extract terms and knowledge points. The standardized power text data is segmented into sentences and words, and term recognition is performed in combination with the power field terminology list, part-of-speech tagging results and dependency relationship results. Terms related to basic power concepts and fundamental laws are tagged, as are terms related to the structural relationships of power generation equipment, substation equipment, transmission lines, distribution devices and their components. Terms related to power system operation modes, changes in operating conditions and adjustment processes of operating parameters are also tagged. Candidate power knowledge points are formed, extracted and deduplicated, and the power knowledge points are obtained.

[0072] The basic concepts and fundamental laws of electricity in the knowledge points of electricity are classified into the basic principles layer; the structural relationships of power generation equipment, power transformation equipment, transmission lines, power distribution devices and their components in the knowledge points of electricity are classified into the equipment composition layer; and the operating modes, changes in operating conditions and adjustment processes of operating parameters of power systems in the knowledge points of electricity are classified into the system operation layer.

[0073] The power knowledge points are divided into the dispatching strategy layer, which includes dispatching plans, power allocation, dispatching instructions, and operation strategy adjustment. The power knowledge points are divided into the emergency handling layer, which includes the accident occurrence, fault propagation, isolation and handling, and operation restoration process. This forms a power knowledge hierarchical system consisting of the basic principle layer, equipment composition layer, system operation layer, dispatching strategy layer, and emergency handling layer.

[0074] In this embodiment, the formation of the knowledge dependency tree specifically includes:

[0075] The textbook chapters, power industry technical data, and expert annotation data are traversed to record the chapter number of each power knowledge point in the textbook, the paragraph number in the industry data, and the annotation number in the expert annotation results, forming a chapter order list, a cause-and-effect description list, and an expert annotation list indexed by power knowledge points;

[0076] Based on the chapter order of the textbook, two electrical knowledge points are compared. When the chapter number of one electrical knowledge point is less than that of the other, the former is regarded as a candidate prerequisite knowledge point for the latter, forming a set of candidate prerequisite relationships based on the chapter order of the textbook.

[0077] Based on the causal descriptions in industry data, power knowledge points in the cause and effect positions are extracted from sentences containing causal trigger words in industry data. Knowledge points in the cause position are taken as prerequisite knowledge points, and knowledge points in the effect position are taken as subsequent knowledge points, forming a set of candidate prerequisite relationships based on causal descriptions.

[0078] Based on standardized power text data and exercises and case studies, the co-occurrence of power knowledge points is statistically analyzed. The ratio of the number of co-occurrences within a fixed window to the total number of windows is used as the co-occurrence degree. When the co-occurrence degree is greater than a preset threshold, the priority direction is obtained according to the order of knowledge points in the textbook, forming a set of candidate priority relationships based on co-occurrence relationships.

[0079] Based on the expert annotation results, all the prerequisite knowledge points directly provided by experts are combined into an expert-annotated prerequisite relation set. The candidate prerequisite relation sets obtained based on the textbook chapter order, causal description, and co-occurrence relationship are then merged with the expert-annotated prerequisite relation set to obtain the initial prerequisite relation set.

[0080] Using electrical knowledge points as nodes and an initial set of prerequisite relations as directed edges, a directed graph is constructed. Loop detection is performed on the directed graph. When a directed cycle consisting of prerequisite relations is detected, a directed edge is deleted from the cycle. This loop detection is repeated until no directed cycles exist in the graph, resulting in a cycle-free directed graph. In this cycle-free directed graph, knowledge points without predecessors are set to the first level, and knowledge points with predecessors are set to the level of the maximum value among all predecessor levels plus one level. The level indices of all knowledge points are obtained. Based on these level indices, the longest path is constructed from knowledge points without predecessors as the main prerequisite path. Directed relations that meet the level requirements are used as supplementary prerequisite relations, forming a knowledge dependency tree.

[0081] In this embodiment, the construction of the multimodal sample set specifically includes:

[0082] Based on the power knowledge hierarchy system and knowledge dependency tree, each power knowledge point is associated with the hierarchical information of its respective basic principle layer, equipment composition layer, system operation layer, dispatch strategy layer, and emergency handling layer, as well as the node identifier, predecessor node, and successor node relationship in the knowledge dependency tree, forming a set of power knowledge nodes with hierarchical attributes and dependency relationship attributes.

[0083] Each basic principle layer knowledge node in the power knowledge node set is bound to its corresponding text and tabular data to generate a multimodal knowledge node sample of the basic principle layer.

[0084] Each device component layer knowledge node in the power knowledge node set is bound to the corresponding image data to generate a multimodal knowledge node sample of the device component layer.

[0085] The system operation layer knowledge nodes of the power knowledge node set are bound to the corresponding curve diagrams, power flow diagrams and operation parameter data, and the scheduling strategy layer knowledge nodes of the power knowledge node set are bound to the corresponding scheduling table data, generating multimodal knowledge node samples of the system operation layer and multimodal knowledge node samples of the scheduling strategy layer.

[0086] Each emergency processing layer knowledge node in the power knowledge node set is bound to the corresponding video data to generate a multimodal knowledge node sample for the emergency processing layer.

[0087] All multimodal knowledge node samples generated from the basic principle layer, equipment composition layer, system operation layer, scheduling strategy layer, and emergency handling layer are uniformly organized according to the order of their prerequisite relationships in the knowledge dependency tree and their hierarchical information in the power knowledge hierarchy system to construct a multimodal sample set.

[0088] In this embodiment, obtaining the first-stage model specifically includes:

[0089] The first-stage training sample set is formed by selecting power knowledge point samples containing text data and tabular data from the multimodal sample set, and configuring target outputs for three types of tasks: power concept question answering, definition explanation and table parsing for each sample;

[0090] The first-stage training sample set is divided into training batches according to a preset batch size. The text data in the training batches is organized into a text input sequence, and the table data is organized into a table input sequence.

[0091] Input the text input sequence into the text encoding channel of the multimodal large model, and input the table input sequence into the table encoding channel of the multimodal large model to obtain the text encoding result and the table encoding result, and generate the power concept question answering prediction result, definition explanation prediction result and table parsing prediction result respectively;

[0092] For each sample in the training batch, the prediction results of the electricity concept question answering, definition explanation, and table parsing are compared with the corresponding target output to obtain the electricity concept question answering loss, definition explanation loss, and table parsing loss, forming the total loss of the training batch.

[0093] Based on the total loss of the training batches, update the parameters of the text encoding channel, table encoding channel, and task output module in the multimodal large model, and repeat the training for all training batches until the preset iteration conditions are met to obtain the first-stage model.

[0094] In this embodiment, obtaining the second-stage model specifically includes:

[0095] Based on the first-stage model, power knowledge point samples containing image data are selected from the multimodal sample set to form the second-stage training sample set, and target outputs for three types of tasks—image content recognition, image-text matching, and image explanation generation—are configured for each sample.

[0096] The second-stage training sample set is divided into training batches according to a preset batch size. The text data in each training batch is organized into a text input sequence, and the image data is organized into an image input sequence.

[0097] The text input sequence is input into the text encoding channel of the multimodal large model, and the image input sequence is input into the image encoding channel of the multimodal large model to obtain text features and image features, and respectively generate image content recognition prediction results, image-text matching prediction results and image explanation generation prediction results;

[0098] The image content recognition prediction results, image-text matching prediction results, and image explanation generation prediction results are compared with the corresponding target outputs to obtain the image content recognition loss, image-text matching loss, and image explanation generation loss, respectively. An InfoNCE and semantic center regularization joint mechanism is adopted. The InfoNCE and semantic center regularization joint mechanism constructs negative sample pairs based on the text features and image features of different power knowledge point samples. The InfoNCE contrast loss that brings the positive sample pairs closer and separates the negative sample pairs is calculated in a contrastive learning manner. According to the multimodal feature distribution of the same power knowledge point in different samples, the corresponding semantic center representation is calculated, and the multimodal features of each sample are constrained to shrink towards the semantic center to obtain the semantic center regularization loss, forming the total loss of the training batch.

[0099] Based on the total loss of the training batches, update the parameters corresponding to the text encoding channel, image encoding channel, and related task output modules in the multimodal large model. Repeat the training for all training batches until the preset iteration conditions are met to obtain the second-stage model.

[0100] In this embodiment, obtaining the multimodal large model specifically includes:

[0101] Based on the second-stage model, power knowledge point samples containing video modal data are selected from the multimodal sample set to form the third-stage training sample set, and target outputs for three types of tasks—operation process explanation, accident process explanation, and multimodal alignment—are configured for each sample.

[0102] The training sample set of the third stage is divided into training batches according to the preset batch size. The text data in each training batch is organized into a text input sequence, the image data is organized into an image input sequence, and the video frame sequence is organized into a video input sequence.

[0103] The text input sequence is input into the text encoding channel of the multimodal large model, the image input sequence is input into the image encoding channel of the multimodal large model, and the video input sequence is input into the video encoding channel of the multimodal large model to obtain text features, image features, and video features, and respectively generate the operation process explanation prediction results, accident process explanation prediction results, and multimodal alignment prediction results;

[0104] The prediction results of the operation process explanation, the prediction results of the accident process explanation, and the prediction results of the multimodal alignment are compared with the corresponding target outputs to obtain the operation process explanation loss, the accident process explanation loss, and the multimodal alignment loss. A distillation and expression anchoring mechanism is introduced, which compares the current model output with the preset teacher model output to obtain the distillation loss. A semantic anchoring representation is constructed based on the multimodal feature distribution of the same power knowledge point in different training samples, and the multimodal features of the current training sample are constrained to shrink towards the semantic anchoring representation to obtain the expression anchoring loss. The operation process explanation loss, the accident process explanation loss, the multimodal alignment loss, the distillation loss, and the expression anchoring loss are combined to form the total loss of the training batch.

[0105] Based on the total loss of the training batches, update the parameters corresponding to the text encoding channel, image encoding channel, video encoding channel, and the output modules of the three types of tasks in the multimodal large model. Repeat the training for all training batches until the preset iteration conditions are met to obtain the multimodal large model.

[0106] In this embodiment, the customized fine-tuning refers to loading the voice features, lip-sync driving parameters, and facial expression parameters of the digital anchor. The multimodal large model generates voice, lip-sync, and facial expression control signals corresponding to the power knowledge content while outputting the narration text.

[0107] Example 1:

[0108] At a provincial power company training center, newly hired dispatchers are required to master a core knowledge system ranging from basic power principles to accident handling within two weeks. Traditional digital broadcasters rely on pre-set scripts and can only linearly read textbook content. They cannot organize explanations according to the logical order of power knowledge, nor can they connect and explain main wiring diagrams, power flow diagrams, and accident videos. This results in frequent instances of logical jumps, discrepancies between text and images, and asynchrony between video and audio. Statistics from the training center show that in the past three training sessions, trainees' average accuracy rate on power flow analysis questions was less than 70%, their accuracy rate on accident handling questions was less than 65%, and their first-time pass rate for accident drills was even lower than 55%, making it difficult to support the high professional knowledge requirements of power dispatching positions.

[0109] In this scenario, the training center uses textbook chapters, operating procedures, accident reports, operating record tables, power flow diagrams, main wiring diagrams, and accident replay videos as multimodal power data. This method automatically extracts power knowledge points, constructing a five-layer power knowledge system encompassing basic principles, equipment composition, system operation, dispatching strategies, and emergency handling, and generating a complete knowledge dependency tree. For example, "short-circuit current" is prerequisite knowledge for "relay protection settings," "main transformer wiring group" is the foundation of "power flow distribution," and "power flow distribution" is a prerequisite for "dispatch strategies" and "accident handling plans." Through the knowledge dependency tree, this method restructures the originally scattered, cross-document power knowledge into a strictly sequential, logically linked chain structure.

[0110] Subsequently, each knowledge point is associated with text, tables, images, or videos. For example, "power flow distribution" is associated with a power flow diagram, "fault isolation process" with an accident replay video, and "main transformer wiring method" with a structural diagram. The model then enters a phased training process: the first phase learns from text and tables, enabling the model to understand power concepts and regulations; the second phase introduces images, using InfoNCE and semantic center regularization to achieve consistency between text and images, improving the "image-based explanation" ability; the third phase introduces videos, using operational process explanations, accident process explanations, and multimodal alignment training to enable the model to acquire temporal inference capabilities, while distillation and expression anchoring prevent the forgetting of knowledge from the first two phases.

[0111] After the model training was completed, the training center conducted a comparative test on the two types of digital anchors: a control group using the traditional method and an experimental group using this method. The two groups consisted of 120 trainees, with identical course content, duration, and instructor guidance. The test data is shown in the table below.

[0112] Table 1. Performance Comparison of Multimodal Digital Anchors in Power Training Scenarios

[0113] Project Category Indicator Name Traditional solution This method Theoretical mastery Average score of closed-book test (100 points) 73.2 86.7 Accuracy rate (%) for trend analysis questions 68.5 88.1 Accuracy rate (%) of emergency response questions 62.4 84.3 Practical performance Accident drill first-time pass rate (%) 54.0 79.5 Explanation quality Image-text consistency score (1–10) 6.1 8.7 Video and commentary are rated simultaneously (1–10) 5.8 8.9 stability The percentage of text-based questions decreased after the addition of videos. -7.4 -1.3 Student Experience Overall student satisfaction (1–10) 7.0 9.0

[0114] Data analysis reveals that this method achieves significant improvements across multiple key indicators. The average score in theoretical tests increased from 73.2 in the traditional approach to 86.7, a rise of 18.4%. The accuracy rate for power flow analysis questions improved by nearly 20 percentage points, and for emergency response questions by over 22 percentage points, indicating that training based on knowledge dependency trees helps the model present a clearer logical chain to learners. The accident drill pass rate increased from 54.0% to 79.5%, demonstrating that video-modal temporal explanation training significantly improves accident comprehension. Regarding explanation quality, the text-image consistency score improved from 6.1 to 8.7, and video synchronization improved from 5.8 to 8.9, validating the effectiveness of InfoNCE and semantic center regularization in cross-modal alignment.

[0115] More importantly, after adding 30 accident videos, the accuracy rate of text-based questions in the traditional approach decreased by 7.4 percentage points, while this method only decreased by 1.3 percentage points. This demonstrates that distillation and expression anchoring mechanisms can effectively suppress catastrophic forgetting, allowing the model to maintain the stability of existing knowledge without loss when expanding to new modalities. Ultimately, the overall student satisfaction rate increased from 7.0 to 9.0, indicating that this method not only improved professional learning outcomes but also achieved a better user experience.

[0116] In summary, the proposed method significantly improved the logic, consistency, and multimodal synchronization of knowledge explanations in practical applications at the power training center, effectively enhancing trainees' learning outcomes and accident handling capabilities, and verifying the engineering practicality and significant application value of the method.

Claims

1. A multi-modal large model power knowledge training method for digital anchors, characterized in that, The method comprises the following steps: acquiring and preprocessing power knowledge corpus to obtain multi-modal power data; extracting terms and knowledge points from normalized power text data of the multi-modal power data to obtain power knowledge points and form a power knowledge hierarchy system; obtaining pre-requisite relationships between the power knowledge points based on chapter sequences of teaching materials, causal descriptions in industry materials, knowledge point co-occurrence relationships and expert annotation results, constructing a knowledge dependency graph and obtaining a knowledge dependency tree; binding corresponding multi-modal data to each power knowledge point according to the power knowledge hierarchy system and the knowledge dependency tree, and constructing a multi-modal sample set; inputting samples binding text modality and table modality data in the multi-modal sample set into a multi-modal large model, performing task training, updating multi-modal large model parameters, and obtaining a first-stage model; based on the first-stage model, jointly training text and image features of the same knowledge node, adopting an InfoNCE and semantic center regular joint mechanism, updating multi-modal large model parameters, and obtaining a second-stage model; based on the second-stage model, jointly training multi-modal features of the same knowledge node, introducing a distillation and expression anchoring method, updating multi-modal large model parameters, and obtaining a multi-modal large model; based on the multi-modal large model, customizing and fine-tuning the multi-modal large model for digital anchors, and obtaining a final multi-modal large model.

2. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, characterized in that, The power knowledge corpus comprises power industry standards, regulation documents, operation regulation manuals, equipment manuals, accident review reports and training courseware, and the preprocessing comprises sentence segmentation, word segmentation, term normalization, unit unification and table structuring processing, and the multi-modal power data comprises normalized power text data, table data, image data and video data.

3. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, characterized in that, The power knowledge hierarchy system comprises the following steps: extracting terms and knowledge points from normalized power text data of the multi-modal power data, performing sentence segmentation and word segmentation on the normalized power text data, and combining power domain term tables, part-of-speech annotation results and dependency relationship results to perform term recognition, annotating power basic concepts and basic law relationship terms, annotating power generation equipment, power transformation equipment, power transmission lines, power distribution devices and their component structure relationship terms, annotating power system operation mode, operation condition change and operation parameter adjustment process terms, forming candidate power knowledge points for extraction and deduplication, and obtaining power knowledge points; dividing power basic concepts and basic law relationship terms in the power knowledge points to the basic principle layer, dividing power generation equipment, power transformation equipment, power transmission lines, power distribution devices and their component structure relationship terms in the power knowledge points to the equipment composition layer, and dividing power system operation mode, operation condition change and operation parameter adjustment process terms in the power knowledge points to the system operation layer; dividing dispatching plan, power distribution, dispatching instruction and operation strategy adjustment process terms in the power knowledge points to the dispatching strategy layer, dividing accident occurrence, fault expansion, isolation disposal and recovery operation process terms in the power knowledge points to the emergency handling layer, and forming the power knowledge hierarchy system comprising the basic principle layer, the equipment composition layer, the system operation layer, the dispatching strategy layer and the emergency handling layer.

4. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, characterized in that, The knowledge dependency tree comprises the following steps: The chapter order list, the cause-effect description list and the expert annotation list are formed by taking the power knowledge points as indexes, the chapter number of each power knowledge point in the textbook, the paragraph number of each power knowledge point in the industry data and the annotation number of each power knowledge point in the expert annotation result are recorded. The two power knowledge points are compared based on the chapter order of the textbook, when the chapter number of one power knowledge point is less than that of another power knowledge point, the former is taken as the candidate prerequisite knowledge point of the latter, and a candidate prerequisite relationship set based on the chapter order of the textbook is formed. The power knowledge points in the cause position and the result position are extracted from the sentences containing the cause-effect relationship trigger words in the industry data, the knowledge point in the cause position is taken as the prerequisite knowledge point, and the knowledge point in the result position is taken as the subsequent knowledge point, and a candidate prerequisite relationship set based on the cause-effect description is formed. The co-occurrence of the power knowledge points is counted based on the normalized power text data and the exercise and case data, the ratio of the co-occurrence number in a fixed window to the total number of windows is taken as the co-occurrence degree, when the co-occurrence degree is greater than a preset threshold, the prerequisite direction is obtained according to the order of the knowledge points in the textbook, and a candidate prerequisite relationship set based on the co-occurrence relationship is formed. The expert annotation prerequisite relationship set is composed of all the prerequisite knowledge point pairs directly provided by the expert, and the candidate prerequisite relationship sets based on the chapter order of the textbook, the cause-effect description and the co-occurrence relationship are combined with the expert annotation prerequisite relationship set to obtain an initial prerequisite relationship set. The directed graph is constructed by taking the power knowledge points as nodes and the initial prerequisite relationship set as directed edges, loop detection is performed on the directed graph, when it is detected that there is a directed loop composed of prerequisite relationships, one directed edge is deleted from the loop, the loop detection is repeated until there is no directed loop in the graph, and an acyclic directed graph is obtained, in the acyclic directed graph, the knowledge point without predecessor is set as the first level, the level of the knowledge point with predecessor is set as the maximum of all predecessor levels plus one level, the level numbers of all knowledge points are obtained, and the longest path starting from the knowledge point without predecessor is constructed as the main prerequisite path according to the level numbers, the directed relationship meeting the level requirement is taken as the supplementary prerequisite relationship, and the knowledge dependency tree is formed.

5. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, characterized in that, The construction of the multi-modal sample set specifically includes: Based on the power knowledge hierarchy system and the knowledge dependency tree, the level information of each power knowledge point belonging to the basic principle layer, the equipment composition layer, the system operation layer, the dispatching strategy layer and the emergency handling layer, and the node identifier, the predecessor node and the successor node relationship in the knowledge dependency tree are corresponded, and a power knowledge node set with level attribute and dependency relationship attribute is formed; Each basic principle layer knowledge node of the power knowledge node set is bound with the corresponding text data and table data to generate a basic principle layer multi-modal knowledge node sample; Each equipment composition layer knowledge node of the power knowledge node set is bound with the corresponding image data to generate an equipment composition layer multi-modal knowledge node sample; and Each system operation layer knowledge node of the power knowledge node set is bound with the corresponding text data and table data to generate a system operation layer multi-modal knowledge node sample. The system operation layer knowledge nodes of the power knowledge node set are bound with corresponding curve graphs, power flow graphs and operation parameter type data, and the dispatching strategy layer knowledge nodes of the power knowledge node set are bound with corresponding dispatching type table data, to generate system operation layer multi-modal knowledge node samples and dispatching strategy layer multi-modal knowledge node samples; Each emergency treatment layer knowledge node of the power knowledge node set is bound with corresponding video type data, to generate emergency treatment layer multi-modal knowledge node samples; All multi-modal knowledge node samples generated by the basic principle layer, the equipment composition layer, the system operation layer, the dispatching strategy layer and the emergency treatment layer are uniformly arranged according to the prerequisite relationship in the knowledge dependency tree and the hierarchical information in the power knowledge hierarchical system, to construct a multi-modal sample set.

6. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, characterized in that, The first stage model is obtained specifically as follows: Power knowledge point samples containing text data and table data are selected from the multi-modal sample set to form a first stage training sample set, and each sample is configured with target outputs of three types of tasks of power concept question answering, definition explanation and table analysis; The first stage training sample set is divided into training batches according to a preset batch size, the text data in the training batches is arranged into text input sequences, and the table data is arranged into table input sequences; The text input sequences are input into a text encoding channel of the multi-modal large model, the table input sequences are input into a table encoding channel of the multi-modal large model, text encoding results and table encoding results are obtained, and power concept question answering prediction results, definition explanation prediction results and table analysis prediction results are respectively generated; For each sample in the training batch, the power concept question answering prediction results, the definition explanation prediction results and the table analysis prediction results are compared with the corresponding target outputs respectively, to obtain power concept question answering losses, definition explanation losses and table analysis losses, and form a total loss of the training batch; According to the total loss of the training batch, an update operation is performed on parameters in the multi-modal large model related to the text encoding channel, the table encoding channel and the task output module, and the training is repeatedly performed on all training batches until a preset iteration condition is met, to obtain the first stage model.

7. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, characterized in that, The second stage model is obtained specifically as follows: Based on the first stage model, power knowledge point samples containing image data are selected from the multi-modal sample set to form a second stage training sample set, and each sample is configured with target outputs of three types of tasks of image content recognition, image-text matching and image explanation generation; The second stage training sample set is divided into training batches according to a preset batch size, the text data in each training batch is arranged into text input sequences, and the image data is arranged into image input sequences; The text input sequences are input into a text encoding channel of the multi-modal large model, the image input sequences are input into an image encoding channel of the multi-modal large model, text features and image features are obtained, and image content recognition prediction results, image-text matching prediction results and image explanation generation prediction results are respectively generated; The image content recognition prediction result, the image-text matching prediction result and the image explanation generation prediction result are compared with the corresponding target outputs respectively to obtain an image content recognition loss, an image-text matching loss and an image explanation generation loss, and an InfoNCE and semantic center regularization joint mechanism is adopted, the InfoNCE and semantic center regularization joint mechanism is to construct a negative sample pair based on the text features and the image features of different power knowledge point samples, calculate the InfoNCE contrast loss of the close positive sample pair and the far negative sample pair in the manner of contrast learning, calculate the corresponding semantic center representation according to the multi-modal feature distribution of the same power knowledge point in different samples, and constrain the multi-modal features of each sample to shrink to the semantic center to obtain a semantic center regularization loss, and form the total loss of the training batch; According to the total loss of the training batch, the parameters corresponding to the text encoding channel, the image encoding channel and the related task output module in the multi-modal large model are updated, and all training batches are repeatedly trained until the preset iteration condition is met, to obtain the second stage model.

8. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, characterized in that, The multi-modal large model is obtained specifically including: Based on the second stage model, the power knowledge point samples containing video modal data are selected from the multi-modal sample set to form a third stage training sample set, and the target outputs of the three types of tasks of running process explanation, accident process explanation and multi-modal alignment are configured for each sample; The third stage training sample set is divided into training batches according to the preset batch size, the text data in each training batch is arranged into a text input sequence, the image data is arranged into an image input sequence, and the video frame sequence is arranged into a video input sequence; The text input sequence is input into the text encoding channel of the multi-modal large model, the image input sequence is input into the image encoding channel of the multi-modal large model, and the video input sequence is input into the video encoding channel of the multi-modal large model to obtain text features, image features and video features, and generate running process explanation prediction results, accident process explanation prediction results and multi-modal alignment prediction results respectively; The running process explanation prediction result, the accident process explanation prediction result and the multi-modal alignment prediction result are compared with the corresponding target output respectively to obtain a running process explanation loss, an accident process explanation loss and a multi-modal alignment loss, and a distillation and expression anchoring mechanism is introduced, the distillation and expression anchoring mechanism is to compare the current model output with the preset teacher model output to obtain a distillation loss, construct a semantic anchoring representation according to the multi-modal feature distribution of the same power knowledge point in different training samples, and constrain the multi-modal features of the current training sample to shrink to the semantic anchoring representation to obtain an expression anchoring loss, and combine the running process explanation loss, the accident process explanation loss, the multi-modal alignment loss, the distillation loss and the expression anchoring loss to form the total loss of the training batch; According to the total loss of the training batch, the parameters corresponding to the text encoding channel, the image encoding channel, the video encoding channel and the three types of task output modules in the multi-modal large model are updated, and all training batches are repeatedly trained until the preset iteration condition is met, to obtain the multi-modal large model.

9. The multi-modal large model electricity knowledge training method for digital anchors according to claim 1, wherein, The customized fine-tuning refers to loading the voice features, mouth shape driving parameters and expression action parameters of the digital anchor, and the multi-modal large model generates the voice, mouth shape and expression action control signals corresponding to the power knowledge content while outputting the interpretation text.

Images