Construction and intelligent retrieval method and system for uninterruptible power operation multi-mode resource library

Abstract

The invention discloses a construction and intelligent retrieval method and system for an uninterruptible power operation multi-mode resource library, and the method comprises the following steps: building a cross-resource semantic alignment relationship between core units, and forming a structured associated resource set; based on a domain term system and an operation procedure, performing joint knowledge extraction on the core unit to obtain common operation entities, technical actions and risk points in the core unit, and constructing a unified knowledge graph by utilizing an extraction result; the extracted features are fused with related context structure information in the unified knowledge graph, and a corresponding knowledge enhancement type multi-mode representation vector is generated; receiving a multi-modal query formed by a natural language, a schematic diagram or a video clip by a user; and fusing the retrieval result based on the map reasoning and the retrieval result based on the vector matching, and carrying out comprehensive sorting and personalized pushing on candidate resources. The training efficiency and the operation safety level can be remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of power system multimodal resource database construction and operation support technology, specifically to a method and system for constructing and intelligently retrieving a multimodal resource database for uninterrupted power operation. Background Technology

[0002] Live-line working is a key technology for ensuring reliable power supply and improving the quality of power services. As work projects become increasingly complex and technical standards are constantly updated, higher demands are placed on the skill levels of workers. Currently, the industry widely uses various forms of digital resources for training and guidance, including written procedures, instructional videos, and 3D simulations.

[0003] However, existing technologies have the following prominent problems: 1) Various resources (text, video, VR) are stored independently, resulting in resource isolation, modal fragmentation, and a lack of effective semantic connections and alignment. When learning, students find it difficult to accurately match the operations in the video with the clauses of the textual procedures or the device structure in the VR, leading to fragmented knowledge acquisition.

[0004] 2) Resource retrieval relies heavily on keyword matching or simple classification, resulting in primitive and inefficient retrieval methods. When trainees describe problems using natural language (such as "how to prevent wires from falling off when replacing pin insulators under live conditions") or upload a picture of a field tool, the existing system cannot perform semantic understanding and cross-modal matching, making it difficult to accurately locate the required resources.

[0005] 3) The resource library is merely a passive storage system. The knowledge is rigid and lacks intelligence. It cannot obtain the necessary highly relevant learning materials or risk warnings based on the learner's skill gaps, current assignment, or real-time operation scenario.

[0006] 4) Existing systems only process resources at the level of metadata annotation, failing to delve into their inherent visual, spatial, and semantic information, and are unable to deeply integrate with domain-specific knowledge systems, thus limiting content-based intelligent applications. Summary of the Invention

[0007] To address the above issues, this invention provides a method and system for constructing and intelligently retrieving a multimodal resource library for live-line work. This solution significantly improves training efficiency and operational safety through multimodal resource alignment, knowledge graph construction, and deep semantic fusion.

[0008] On the one hand, this invention provides a method for constructing and intelligently retrieving a multimodal resource library for uninterrupted power supply operations, comprising the following steps: S1: Collect the following resources in the field of live-line work: technical documents, courseware, operation videos, VR simulation programs and historical cases; for the set target work theme, extract the core units of the text, visual and interactive content of the relevant resources from the collected resources, and establish cross-resource semantic alignment relationships between the core units to form a structured set of related resources; S2: Based on the domain terminology system and operating procedures, perform joint knowledge extraction on the core units of text, visual and interactive content in the structured associated resource set to obtain common operating entities, technical actions and risk points, and use the extraction results to construct a unified knowledge graph; S3: For the structured set of associated resources, extract its textual semantic features, visual spatial features, and interactive behavior features respectively; using the unified knowledge graph as a semantic guidance framework, fuse the extracted features with the relevant contextual structure information in the unified knowledge graph to generate the corresponding knowledge-enhanced multimodal representation vector. S4: Receive multimodal queries from users consisting of natural language, diagrams, or video clips; The unified knowledge graph is used to perform semantic parsing and intent expansion on the query to obtain retrieval results based on graph reasoning; The query is transformed into a query vector, and a similarity search is performed in the vector space composed of the knowledge-enhanced multimodal representation vectors to obtain retrieval results based on vector matching. By integrating the retrieval results based on graph reasoning and the retrieval results based on vector matching, candidate resources are comprehensively ranked and personalized.

[0009] Furthermore, establishing cross-resource semantic alignment relationships between the core units in step S1 specifically includes: For the same task, instructional videos and VR simulation programs are time-series aligned using motion recognition and timestamps; For VR simulation programs and technical documents, establish a spatial-semantic alignment mapping between 3D scene models and text descriptions; Case-standard alignment and association are performed by linking on-site data from historical cases with standard procedure clauses.

[0010] Furthermore, the unified knowledge graph constructed in step S2 includes "personnel skill profile" entity nodes. These entity nodes are connected to "skill items", "training resources", and "historical cases" entity nodes through the relationships of "mastered", "needs to learn", and "previously applied", which are used to quantitatively assess skill levels and recommend personalized learning paths.

[0011] Furthermore, step S3 is performed for each structured set of associated resources, specifically including: S31: Extract the associated subgraphs of the corresponding nodes of the structured associated resource set in the unified knowledge graph, and use graph neural network encoding to generate the graph context feature vector of the structured associated resource set; S32: Using a pre-trained model, extract the text semantic feature vector, visual spatial feature vector, and interactive behavior feature vector from the core units of text, visual, and interactive content contained in the resource set, respectively. S33: Input the graph context feature vector, text semantic feature vector, visual spatial feature vector, and interaction behavior feature vector into a multimodal fusion Transformer, perform adaptive weighted fusion through a cross-attention mechanism, and output the final knowledge-enhanced multimodal representation vector.

[0012] Furthermore, step S4 specifically includes: S41: Entity and relation recognition is performed on the multimodal query, and path traversal and logical reasoning are performed in the unified knowledge graph to obtain the first candidate resource list and its graph relevance score; S42: Transform the multimodal query into a query vector, and perform an approximate nearest neighbor search in the vector space composed of the knowledge-enhanced multimodal representation vectors to obtain a second candidate resource list and its vector similarity score; S43: Merge and deduplicate the first candidate resource list and the second candidate resource list, and calculate the comprehensive ranking score based on the weighted fusion algorithm, combining the graph relevance score and vector similarity score of each resource, to generate a comprehensive ranking list; S44: After personalizing and pushing the comprehensive sorting list by combining user profiles and real-time operation scenarios, the list is pushed out.

[0013] Furthermore, it also includes step S45: establishing a feedback optimization closed loop, recording user interaction data on pushed resources and subsequent simulated operation performance, and jointly iteratively optimizing the multimodal fusion representation model and the retrieval ranking model, wherein: The multimodal fusion representation model is used to execute step S3 to generate the knowledge-enhanced multimodal representation vector. The retrieval and ranking model is used to execute the logic in step S4, which involves merging and ranking the candidate resource list and then pushing personalized recommendations.

[0014] On the other hand, the present invention also provides a system for constructing and intelligently retrieving a multimodal resource library for live-line work, used to implement the above-described method for constructing and intelligently retrieving a multimodal resource library for live-line work, including: The multimodal resource alignment processing module is used to perform step S1, constructing and managing the structured associated resource set; The knowledge graph construction and management module is used to execute step S2, store, update and provide query and reasoning services for the unified knowledge graph; A multimodal semantic fusion computing module is used to execute step S3, generating and maintaining the knowledge-enhanced multimodal representation vector; The hybrid intelligent retrieval and push engine is used to execute step S4, receive multimodal queries, coordinate the invocation of the knowledge graph construction and management module and the multimodal semantic fusion calculation module, and complete the fusion sorting and push. The user interaction and context-aware interface module is used to provide multimodal query entry points, resource display, and access to external systems to obtain real-time operational context data.

[0015] Furthermore, the multimodal resource alignment processing module integrates a time sequence alignment unit, a spatial semantic alignment unit, and a case standard alignment unit, which respectively realize cross-resource semantic alignment between the core units.

[0016] Furthermore, the knowledge graph construction and management module includes an automated incremental learning unit that can monitor newly added resources, automatically initiate the knowledge extraction process, and incrementally update the unified knowledge graph after manual review, thereby realizing the dynamic evolution of the knowledge base.

[0017] Furthermore, the hybrid intelligent retrieval and push engine internally includes: The knowledge graph semantic retrieval unit is used to call the knowledge graph construction and management module to perform logical reasoning retrieval. The vector similarity search unit is used to call the multimodal semantic fusion calculation module to perform an approximate nearest neighbor search; The hybrid result fusion and sorting unit is used to perform the weighted fusion and sorting logic as described above.

[0018] Furthermore, the user interaction and context awareness interface module is connected to the VR training platform or on-site operation terminal, which can obtain the status of the operation object and environmental parameters in the virtual or real operation scenario in real time, and trigger the immediate and proactive push of safety specifications, operation demonstration videos or VR simulation examples that are strongly related to the current high-risk operation or complex steps.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: 1) By using structured sets of related resources and a unified knowledge graph, modal barriers are broken down, enabling deep integration and understanding of resources, and making resources such as text, video, and VR form an organic whole at the spatiotemporal and semantic levels.

[0020] 2) The proposed knowledge-enhanced multimodal representation vector not only includes the content features of the resource itself, but also incorporates its structural information within the domain knowledge system. Current multimodal retrieval research largely focuses on visual-text fusion, while this invention introduces a unified knowledge graph as a "semantic guidance framework" to generate "knowledge-enhanced multimodal representation vectors." This is essentially a multimodal representation method that deeply injects domain knowledge, not only improving the semantic depth of resource representation, but also having more significant advantages in addressing professional domains and resolving semantic sparsity and ambiguity issues, thereby enabling computers to understand resources more closely to the level of human experts.

[0021] 3) This invention employs a hybrid retrieval mechanism that integrates knowledge graph reasoning and vector similarity search. These two approaches are not simply parallel but rather complementary. Knowledge graph retrieval excels in interpretability, logical rigor, and relational reasoning, ensuring that search results conform to domain rules and causal logic (such as security specifications), but may be limited by the graph's coverage. Vector retrieval excels in semantic flexibility, fuzzy matching, and cross-modal content discovery, capturing deep similarities in unstructured content, but may overlook logical constraints. This invention achieves a balance between "logical correctness" and "semantic relevance" through a weighted fusion hybrid retrieval system, providing more accurate and flexible intelligent retrieval. It ensures both the rigor of professional logic and enables fuzzy semantic matching and cross-modal retrieval, significantly improving recall and precision.

[0022] 4) The system can proactively push personalized resource combinations based on user skill profiles and real-time operation scenarios. It can also provide context-based real-time guidance and risk warnings in VR simulations or on-site operations, achieving a leap from passive retrieval to proactive intelligent services and truly realizing intelligent intervention. Especially for real-time risk warnings in high-risk operation scenarios, this not only significantly improves efficiency but also represents a qualitative leap in proactive safety protection capabilities, transforming "people seeking knowledge" into "knowledge finding people + risk intervention."

[0023] 5) Through feedback optimization and the incremental learning mechanism of knowledge graph, the system can continuously optimize its retrieval, push effect and knowledge coverage as new resources and cases are added and user behavior feedback is received. It has the ability to evolve on its own and thus has a longer lifespan. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is a flowchart illustrating the construction and intelligent retrieval method for a multimodal resource library for uninterrupted power supply operations provided by the present invention.

[0026] Figure 2 This is a schematic diagram of the system architecture of a multimodal resource library construction and intelligent retrieval system for uninterrupted power supply operations provided by the present invention. Detailed Implementation

[0027] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The embodiments of the present invention mainly include method embodiments and system embodiments. In order to better describe the technical solutions of the present invention, the method embodiments and / or system embodiments include one or more subdivided embodiments, which are respectively represented by "Embodiment 1", "Embodiment 2"... "Embodiment N".

[0028] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Method Implementation Examples

[0029] refer to Figure 1 This invention provides a method for constructing and intelligently retrieving a multimodal resource library for uninterrupted power supply operations. The method mainly includes the following steps: S1: Collect the following resources in the field of live-line work: technical documents, courseware, operation videos, VR simulation programs and historical cases; for the set target work theme, extract the core units of the text, visual and interactive content of relevant resources from the collected resources, and establish cross-resource semantic alignment relationships between core units to form a structured set of related resources; S2: Based on the domain terminology system and operating procedures, joint knowledge extraction is performed on the core units of text, visual and interactive content in the structured related resource set to obtain common operating entities, technical actions and risk points, and a unified knowledge graph is constructed using the extraction results; S3: For structured sets of related resources, extract their textual semantic features, visual spatial features, and interactive behavior features respectively; using a unified knowledge graph as a semantic guidance framework, fuse the extracted features with the relevant contextual structure information in the unified knowledge graph to generate corresponding knowledge-enhanced multimodal representation vectors. S4: Receive multimodal queries from users consisting of natural language, diagrams, or video clips; By using a unified knowledge graph to perform semantic parsing and intent expansion on queries, retrieval results based on graph reasoning are obtained. The query is transformed into a query vector, and a similarity search is performed in the vector space composed of knowledge-enhanced multimodal representation vectors to obtain retrieval results based on vector matching. By integrating retrieval results based on graph reasoning and retrieval results based on vector matching, candidate resources are comprehensively ranked and personalized.

[0030] In this context, a work topic refers to a technical unit with complete semantic boundaries, defined within the field of live-line work, used to represent content such as specific work tasks, key technical aspects, or core knowledge domains. Various work topics can be pre-defined, and each of these topics can be used as a target work topic to execute the method provided in this embodiment of the invention.

[0031] S1: Multimodal resource alignment and structuring.

[0032] This step involves integrating the original, heterogeneous, multimodal resources collected in the field of live-line work into a structured, logically interconnected resource set (referred to as the resource set). Specifically, for technical documents, courseware, operation videos, VR simulation programs, and historical work cases in the field of live-line work, for each target work topic, the core units of its textual, visual, and interactive content are extracted from the collected resources, and cross-resource semantic alignment relationships are established between the core units, forming a structured, interconnected resource set that includes text description segments, key visual frames / fragments, VR interactive nodes, and case data fields.

[0033] In other words, the goal of step S1 is to extract meaningful core units from the original multi-source, heterogeneous resources and establish semantic bridges between these core units. The extraction of core units includes: 1) Extraction of technical documents / courseware (text modality) primarily employs deep analysis using natural language processing techniques. This includes discourse analysis (e.g., identifying document chapters, clauses, step lists, etc.) and key sentence / paragraph extraction. For key sentence / paragraph extraction, text summarization or key information extraction techniques are mainly used to extract sentences or paragraphs describing core operations, safety regulations, and technical parameters. For example, "using an electroscope to test the voltage of each phase of the conductor" can be extracted as a core unit of the text from the procedure.

[0034] 2) Extraction of operational video (visual + temporal modality), including shot segmentation and keyframe extraction, action recognition and segment localization. Shot segmentation and keyframe extraction primarily utilize video processing libraries (such as OpenCV, FFmpeg) or deep learning-based scene segmentation models to divide long videos into semantically coherent shots or segments. Action recognition and segment localization mainly use temporal action detection models to automatically identify and locate the start and end times of standard technical actions such as "electricity testing," "masking," and "installation" in the video. These action segments are the core units of the visual content.

[0035] 3) Extraction of VR simulation programs (interactive + spatial modal) mainly includes scene graph analysis and interactive node extraction. That is, by analyzing the runtime data or configuration files of the VR program, its scene graph (a graph structure describing all objects and their relationships in three-dimensional space) and predefined interactive nodes (such as "grabable tools", "clickable switches", and "trigger-based teaching prompts") are extracted. These nodes and objects constitute the core units of interactive and spatial content.

[0036] 4) Historical Cases (Structured / Semi-structured Data): This mainly involves the extraction of information fields. Using rule templates or information extraction models, structured fields such as work items, risk levels, tools used, and exception records are extracted from case reports (which may be text, tables, or database records). Each meaningful field or combination thereof can be considered a core data unit.

[0037] After extracting the core units, cross-resource semantic alignment relationships are established between the core units. Preferably, establishing cross-resource semantic alignment relationships between the core units in step S1 specifically includes: i) temporal alignment (video↔VR): For teaching videos and VR simulation programs of the same task, by comparing the timestamps and logical order of the video action recognition results and the VR interaction nodes, the video segments describing the same operation steps are aligned with the VR interaction nodes on the timeline. ii) spatial-semantic alignment (VR↔text): The 3D model (with semantic labels, such as "insulator string") in the VR scene diagram is linked with the text paragraphs in the technical document describing the component. This alignment process can be achieved through entity linking technology, such as calculating the semantic similarity between the entity names mentioned in the text and the VR model labels (using word vectors or sentence vectors), or using a predefined device name mapping table. iii) case-standard alignment (case↔ procedure): The "abnormal record" field extracted from historical cases (such as "incomplete shielding") is associated with the corresponding "standard clauses" in the standard operating procedure (such as "insulation shielding should be tight and secure"). This is typically based on keyword matching, semantic similarity calculation, or a manually built rule base (e.g., mapping “incomplete masking” to all clauses in the procedure that require “masking”).

[0038] Taking the "live-line disconnection" operation as an example, the following multimodal resources are processed: "Live-line Working Procedures (Connection of Leads)" (PDF), a standard instructional video (MP4), a VR training module (.exe), and several historical work order records from the PMS system: 1) Video and text time sequence alignment. For example, use video action recognition models (such as those based on Open Pose or I3D) to automatically identify key operation segments in the video, such as "voltage testing," "insulation shielding," "wire stripping," and "connection." Use Natural Language Processing (NLP) technology to perform chapter structure analysis and extract key step statements from the "Live-Line Working Procedures (Connecting Leads)" (PDF). Align the teaching video segments with the step description paragraphs in the "Live-Line Working Procedures (Connecting Leads)" (PDF) using Dynamic Time Warping (DTW) algorithms or matching based on key action tags. Finally, generate structured data: {"Step Name": "Insulation Shielding", "Text Paragraph ID": "P_2.3.1", "Video Start and End Timestamps": "[00:12:05, 00:15:30]"}.

[0039] 2) VR and Text Space - Semantic Alignment. Parse the scene tree and asset list of the VR program, extracting all 3D model objects (e.g., "insulating gloves," "insulating blankets," "drainage wires"). Define semantic labels and spatial bounding boxes for each model object. Using rules or few-shot learning, link the model labels to the device and tool names appearing in the specification text. Record key interactive events in VR (e.g., "the user picks up the insulating blanket and covers the A-phase conductor") and the triggered program states. Associate this event with the clause in the specification that "insulation shielding should be provided for all phase-current conductors and grounding components." The final mapping table is: {"VR Model ID": "model_insulating_blanket_01", "Semantic Label": "insulating blanket", "Corresponding Specification Clause": "Clause_4.5", "Typical VR Interaction Event": "cover_conductor"}.

[0040] 3) Extract the following structured fields from historical work orders: work time, weather, work supervisor, list of tools used, and recorded anomalies (e.g., "incomplete shielding," "torque not meeting standard"). Establish a link between "anomalies" and "standard procedures." For example, in the case study, "incomplete shielding" is linked to the clause "insulation shielding should be tight and secure" in the procedure and marked as a "violation case"; "use of new quick-locking clamps" is linked to "new tool application cases." Encapsulate the entire case study into a resource set, indexing its internal fields with corresponding standard clauses and related video / VR resources.

[0041] Finally, through the extraction and association described above, step S1 does not produce a simple list of resources, but rather a structured set of associated resources. Within this set, a text paragraph, a video, a VR node, and a case data field, all associated with the same core concept (such as "insulation shielding"), together form a composite entity rich in semantic relationships that can be understood and processed by a computer.

[0042] S2: Construct a unified knowledge graph across modalities.

[0043] Based on a predefined ontology of the power operation domain (including core concepts such as equipment, tools, operation items, technical actions, risk points, and safety regulations), joint knowledge extraction is performed on the core units, including text, visual, and interactive data, contained in the structured associated resource set constructed in step S1. Common operation entities, technical actions, and risk points are identified and associated to construct a unified knowledge graph that can be referenced and reasoned across modalities. This graph specifically includes a "Personnel Skill Profile" entity node, which dynamically associates personnel's skills, learning resources, and operational history. For example, this "Personnel Skill Profile" entity node is connected to "Skill Items," "Training Resources," and "Historical Cases" entity nodes through relationships such as "Mastered," "Needs to Learn," and "Previously Applied To," used to quantitatively assess skill levels and recommend personalized learning paths.

[0044] This step aims to extract core domain concepts and their relationships from the structured, associated resource set formed in step S1, and construct a unified knowledge graph that is machine-understandable and can be referenced and reasoned across modalities. Its innovation lies in cross-modal joint knowledge extraction, that is, collaboratively learning and supplementing knowledge from text, visual (video / VR), and structured data. The specific process is as follows, mainly including four stages: domain ontology design and structured guidance, cross-modal joint knowledge extraction and recognition, cross-modal verification and knowledge fusion, and graph construction. a) Domain Ontology Design and Structured Guidance. Based on the domain terminology system and standard operating procedures for live-line electrical work, a structured "domain concept dictionary" or "seed ontology" is pre-constructed. This ontology clearly defines the core concept categories to be identified and their relationships, such as: a1) Work Entities: insulated bucket truck, tension insulator, shielding suit, wire tensioner; a2) Technical Actions: voltage detection, shielding, cutting, connection, tightening; a3) Risk Points: insufficient safety distance, tool failure, conductor detachment, induced electric shock; a4) Relationships: belongs to, applies to, needs to be prevented, violates, is a prerequisite. This domain ontology provides prior knowledge anchors for the subsequent automatic extraction model, ensuring that the extraction direction conforms to domain specifications, rather than performing general semantic analysis.

[0045] b) Joint knowledge extraction and recognition across multiple modalities. The recognition work is based on the above ontology guidance and is carried out collaboratively on text, visual (video / VR) and structured data, and the alignment relationship established in step S1 is used for mutual verification and supplementation.

[0046] b1) Deep semantic parsing of text modalities: For texts such as technical documents, courseware, and case reports, use domain-fine-tuned pre-trained language models (such as models based on the BERT architecture and further trained on power operation corpora) to perform named entity recognition (NER) and relation extraction (RE).

[0047] b2) Semantic conversion and alignment recognition of visual and interactive modalities: For operation videos, the core visual units (action segments) extracted in step S1 are used first.

[0048] For keyframes, a natural language description (e.g., "An electrician is tightening bolts on an insulator with a wrench") is generated using a visual language model (e.g., "An electrician is tightening bolts on an insulator with a wrench"). This descriptive text is then fed into the text extraction process described above, thereby indirectly identifying entities and actions contained in the video. Simultaneously, the action labels directly output by the video action recognition model (e.g., tightening, installing) can serve as strong signals of technical action entities.

[0049] For VR simulation programs, their embedded structured information is utilized. Object tags in the program scene tree (such as object_type: “insulator”) can be directly mapped to task entities (such as “insulator”) in the knowledge graph. Events recorded in the interaction log (such as event: “install_tool”, tool: “torque_wrench”) can be directly or parsed through simple rules into technical actions (“installation tool”) and the entities used (“torque wrench”).

[0050] b3) Direct mapping of structured data: The structured fields extracted in step S1 from historical cases are directly converted into knowledge triples through predefined mapping rules. For example, the field "Hazard type: Incomplete concealment" is converted to "(Case X)-[Hazard exists]->(Incomplete concealment)"; the field "Tool used: Torque wrench" is converted to "(Case X)-[Tool used]->(Torque wrench)".

[0051] c) Cross-modal verification and knowledge fusion. The same entity or action may appear in multiple modalities. Through the alignment relationship established in step S1, recognition results from different modalities can be associated and verified. For example, the "insulation masking" action described in text, the corresponding operation segment in a video, and the interactive node in VR all point to the same concept. If multiple modalities recognize the concept with high confidence, the confidence level for its final inclusion in the knowledge graph is significantly increased. This fusion of multimodal evidence effectively improves the accuracy and robustness of the recognition results and can correct ambiguities or errors that may arise from single-modal recognition.

[0052] d) Knowledge graph construction. The knowledge triples extracted and validated across modalities are input into a graph database (such as Neo4j) for storage and relational connections, forming a unified knowledge graph rich in semantic relationships.

[0053] Preferably, this unified knowledge graph specifically includes "Personnel Skill Profile" entity nodes. These nodes are dynamically linked to entity nodes such as "Skill Items," "Training Resources," and "Historical Cases" through relationships like "Mastered," "Needs to Learn," and "Previously Applied To." For example, relationships such as "(Zhang San) - [Mastered] -> (Insulation and Masking Skills)" and "(Insulation and Masking Skills) - [Passed Course] -> (Course C001)" can be constructed. This design enables the graph to not only describe static domain knowledge but also quantitatively assess personnel skill levels and provide direct data support for recommending personalized learning paths, achieving a crucial leap from static storage to dynamic empowerment of the knowledge base.

[0054] Finally, a unified knowledge graph covering elements such as equipment, operations, risks, personnel, resources, and skills is output. This unified knowledge graph becomes the core semantic storage and reasoning engine of the entire system.

[0055] S3: Knowledge-Enhanced Multimodal Semantic Fusion Representation The goal of this step is to generate a dense vector representation (embedding) for each structured set of related resources. This vector should not only contain its multimodal content information, but also incorporate its contextual semantic information in the unified knowledge graph, i.e., generate a knowledge-enhanced multimodal representation vector.

[0056] Preferably, step S3 specifically includes: S31: Extract the associated subgraphs of the corresponding nodes of the structured associated resource set in the unified knowledge graph, and use graph neural network encoding to generate the graph context feature vector of the structured associated resource set; S32: Using a pre-trained model, extract text semantic feature vectors, visual spatial feature vectors, and interactive behavior feature vectors from the core units of text, visual, and interactive content contained in the resource set, respectively. S33: Input the graph context feature vector, text semantic feature vector, visual spatial feature vector, and interaction behavior feature vector into a multimodal fusion Transformer, perform adaptive weighted fusion through a cross-attention mechanism, and output the final knowledge-enhanced multimodal representation vector.

[0057] The connection between the contextual semantic information of the graph (short for Unified Knowledge Graph) and the graph contextual feature vector lies in the fact that the graph contextual semantic information is encoded by a graph neural network to generate the graph contextual feature vector.

[0058] For example, the input is a structured set of associated resources for "energized disconnected leads" and its associated entity subgraphs in a knowledge graph. The processing flow includes: 1) Intramodal feature extraction. All text associated with the resource (procedure paragraphs, case descriptions, video subtitles) are concatenated and input into a pre-trained language model (such as RoBERTa, an optimized and enhanced version of the BERT model). The output at the [CLS] position (CLS is the classification token of the RoBERTa model) is taken as the text feature vector. All key video frames and VR scene screenshots associated with the resource are extracted and input into a visual base model (such as CLIP's visual encoder or DINOv2). The features of multiple images are averaged or aggregated using attention to obtain a visual feature vector. For VR resources, their user operation logs (event sequences, such as: pick_up(tool_A) -> move_to(location_B) -> perform(action_C)) are encoded. Recurrent neural networks (RNNs) or Transformer encoders can be used to encode the time series to generate behavioral feature vectors. 2) Knowledge graph-based context encoding. From the unified knowledge graph, a K-hop adjacency subgraph centered on the current core resource entity (e.g., "live-line disconnection operation") is extracted. This subgraph is encoded using a graph neural network (GNN), such as Graph SAGE or Graph Attention Network (GAT). The GNN aggregates neighbor information through message passing, ultimately generating an embedding representation for each node in the subgraph (including the current resource node). The GNN output vector of the current resource node is taken as the graph context feature vector (K). This vector encapsulates the location and association information of the resource within the professional knowledge network. 3) Deep feature fusion. The four feature vectors [T, V, B, K] are input into a custom multimodal fusion Transformer (a preferred specific form of the multimodal fusion representation model). In this Transformer, the initial embedding of the [CLS] token is obtained by concatenating or adding the four feature vectors. This multimodal fusion representation model employs multi-head self-attention and cross-attention mechanisms to allow text features to focus on relevant visual segments. These visual features are then used to query the corresponding knowledge graph entity descriptions. The knowledge graph context feature vector (K) serves as a global semantic guide, participating in the interaction and integration of all modal features. After computation through several Transformer layers, the final embedding of the [CLS] token is taken as the knowledge-enhanced multimodal representation vector (F) for the resource.

[0059] Finally, the output is a high-dimensional, dense knowledge-enhanced multimodal representation vector F. This vector F is stored in a dedicated vector database (such as Milvus, Pinecone) for subsequent efficient similarity retrieval.

[0060] At this point, the system has generated a corresponding knowledge-enhanced multimodal representation vector for each structured set of associated resources and stored it in the vector database, forming a multimodal semantic vector space for the field of uninterrupted power supply operations, which supports subsequent efficient and intelligent retrieval.

[0061] S4: Semantic retrieval and contextualized push based on a hybrid graph and vector approach.

[0062] It can receive multimodal queries from users, consisting of natural language, diagrams, or video clips.

[0063] For this multimodal query, preferably, step S4 specifically includes: S41: Perform entity and relation recognition on multimodal queries, perform path traversal and logical reasoning in the unified knowledge graph, and obtain the first candidate resource list and its graph relevance score; S42: Transform the multimodal query into a query vector, and perform an approximate nearest neighbor search in the vector space composed of knowledge-enhanced multimodal representation vectors to obtain the second candidate resource list and its vector similarity score; S43: Merge and deduplicate the first and second candidate resource lists, and calculate the comprehensive ranking score based on the weighted fusion algorithm, combining the graph relevance score and vector similarity score of each resource, to generate a comprehensive ranking list; S44: Based on user profiles and real-time operational scenarios, the comprehensive sorting list is personalized and then pushed out.

[0064] For this multimodal query, in another preferred embodiment, step S4 specifically includes: By using a unified knowledge graph, semantic parsing and logical reasoning are performed on the query to obtain a set of related graph nodes and their reasoning weights; The query is transformed into a query vector, which is then searched in a vector space composed of knowledge-enhanced multimodal representation vectors to obtain a set of similar vectors and their similarity scores. The graph nodes obtained by graph reasoning retrieval and the similar vectors obtained by vector similarity search are mapped to their corresponding structured associated resource sets according to the preset correspondence relationship, so as to obtain two candidate resource set lists based on graph reasoning and vector matching and their respective relevance scores. The two candidate resource sets obtained after mapping are deduplicated, and the two types of relevance scores of each resource set are weighted and fused to obtain a comprehensive ranking list. The ranking list is then calibrated by combining user profiles and real-time operation scenarios, and finally personalized push is performed.

[0065] Preferably, the method further includes a step of optimizing the models used in the above steps through feedback (e.g., S45): establishing a feedback optimization closed loop, recording user interaction data with pushed resources and subsequent simulated operation performance, and jointly iteratively optimizing the multimodal fusion representation model and the retrieval ranking model. Wherein: the multimodal fusion representation model is used to execute step S3 to generate knowledge-enhanced multimodal representation vectors. The retrieval ranking model is used to execute the logic in step S4 from the fusion and ranking of the candidate resource list to personalized push notifications.

[0066] In summary, the retrieval process in step S4 employs a hybrid retrieval strategy: on the one hand, a unified knowledge graph is used to perform semantic parsing, entity linking, and intent expansion of the query, inferring relevant resources through graph relationship paths (logical retrieval); on the other hand, the query is transformed into a vector, and an approximate nearest neighbor search (a type of similarity search) is performed in a vector database composed of knowledge-enhanced multimodal representation vectors of all resources. Furthermore, a weighted fusion algorithm is designed to fuse and reorder the two sets of results. Finally, by combining user roles, real-time operational scenarios (obtained from VR training platforms or on-site terminals), and "personnel skill profiles" or user profiles, the ranking results are personalized, proactively pushing the most relevant multimodal resource combinations, and enabling real-time warnings and knowledge prompts for high-risk steps.

[0067] Example 1 To better illustrate the embodiments of the present invention, let's take a scenario where a trainee encounters difficulty tightening the conductor during a VR simulation training operation of "replacing a tension pole insulator string." The trainee asks via voice, "I always feel insufficient force when tightening the conductor, what should I do?" Simultaneously, the context-aware interface captures real-time events sent by the VR program: "user_strain_during_tightening (the user is experiencing difficulty tightening)" and the current tool status: "tensioner_in_use (the tensioner is being used)." The system converts the speech to text and integrates it with the contextual events to generate an enhanced query representation Q: "How to handle the problem of insufficient force when tightening the conductor? [Context: tension pole operation, using a tensioner]."

[0068] The system performs a hybrid search. During graph semantic search, entity links are established for Q (the problem), identifying "tightening conductors," "insufficient strength," "tension bar," and "tensioner." In the unified knowledge graph, starting from the "tightening conductors" node, the system traverses and reasons along relational paths such as [Common problems] -> (insufficient strength), [Possible causes] -> (inappropriate tool selection), and [Solutions] -> (using a force-saving pulley block). Graph search returns a list of resource nodes, List_A, strongly related to "how to use a force-saving pulley block," "tensioner specification verification," and "tension bar stress analysis." During vector similarity search, the query text Q is transformed into a query vector Q_vec using the same text encoder (consistent with S3). In the vector database, an approximate nearest neighbor (ANN) search is performed to find resources corresponding to the knowledge-enhanced multimodal representation vector with the highest cosine similarity to Q_vec. Vector search returns a list of resources, List_B, that are semantically similar, potentially including videos explaining "conductor tightening techniques" and VR modules demonstrating "pulley block installation."

[0069] Next, the results are merged and sorted. For example, the merged and sorted score is designed as follows: Final_Score = α*Graph_Score + β*Vector_Score + γ*Personalization_Score. Here, Graph_Score (graph association score) is calculated based on the resource's ranking in List_A and its path distance to the query entity in the graph. Vector_Score (vector similarity score) is based on the similarity score returned by the ANN search. Personalization_Score (personalization adaptation score) is based on the student's skill profile; if their proficiency in the "tool usage" skill is low, resources related to "tool usage" will receive bonus points; simultaneously, considering the current "VR training" scenario, VR / video resources are prioritized. After deduplicating List_A and List_B, they are sorted by Final_Score to generate the final resource list.

[0070] Furthermore, the system doesn't simply return a list; instead, it proactively intervenes based on the context. When it detects that a learner is encountering difficulties in the VR simulation, the system uses a context-aware interface to instantly push the top-ranked "Effort-Saving Pulley Assembly Installation Demonstration" VR micro-lesson module into the VR environment via picture-in-picture or voice prompts, providing timely and precise guidance.

[0071] Finally, there's feedback optimization. The system records whether students viewed the pushed resources, the viewing duration, and whether they successfully completed the tightening operation in the VR simulation after viewing. This positive / negative feedback data is used to: a) optimize the retrieval / ranking model: as training data, adjust the weights (α, β, γ) in the fusion ranking algorithm.

[0072] b) Optimize the representation model: If a resource is frequently clicked and subsequent operations are successful, its vector representation should be closer to other successful resources; otherwise, it should be further away. This allows for fine-tuning of the multimodal fusion model in step S3 through comparative learning and other methods.

[0073] Example 2 Taking the "replacing insulators on a straight pole under energized conditions on a 10kV line" project as an example, the implementation steps of the method of the present invention are explained.

[0074] Step S1: Resource alignment and structure.

[0075] Obtain the "Standardized Operation Instruction Manual" (text), on-site teaching videos, VR simulation program, and 3 historical case reports for this task.

[0076] Time alignment: Through action recognition algorithms, the time periods of key steps in the video, such as "installing the insulating shield" and "using the insulated bucket truck to enter the working position", are matched and associated with the corresponding interactive nodes in the VR simulation.

[0077] Spatial-semantic alignment: In the VR scene, 3D model components such as "insulator", "crossarm" and "conductor" are labeled and mapped to the text descriptions and technical parameters of these components in the work instructions.

[0078] Case Study - Standard Alignment: Extract the data "tighten bolts to 50 N·m using a torque wrench" from a historical case study, associate it with the "bolt tightening torque standard" clause in the work instruction, and mark the case as either "standard execution" or "minor deviation".

[0079] Ultimately, a structured set of related resources on "replacing insulators on straight poles" is generated, which internally links relevant text paragraphs, video clips, VR scene nodes, and case data fields.

[0080] Step S2: Knowledge graph construction.

[0081] From the above-mentioned resources, entities such as "insulated bucket truck (equipment)", "insulated shielding (technical operation)", "conductor detachment (risk point)", and "Zhang San (operator)" are jointly extracted.

[0082] Establish relationships: (Insulated bucket truck) - [Used for] -> (Insulator replacement operation); (Insulator replacement operation) - [Needs to prevent] -> (Risk of conductor falling off); (Zhang San) - [Master] -> (Insulation shielding skills).

[0083] The extracted triples are integrated into a unified knowledge graph. In the graph, the node "Replace straight pole insulator" will be associated with the resource set generated in step 1.

[0084] Step S3: Generate fusion representation.

[0085] For the above resource set, extract the semantic vector of the text, the visual vector of the video keyframes, and the behavioral vector of the VR operation sequence.

[0086] Extract a second-degree-of-fact subgraph (containing nodes such as equipment, risk, and skill) centered on "replacing insulators on straight poles" from the knowledge graph, and encode this subgraph using a graph neural network to obtain the graph context vector.

[0087] Text vectors, visual vectors, behavioral vectors, and knowledge graph context vectors are input into a multimodal fusion Transformer. After training, the model learns, for example, that the visual features of "insulation shield" should be strongly correlated with the "insulation protection" node in the text description and the "safety specifications" node in the knowledge graph. Finally, the model outputs a knowledge-enhanced multimodal representation vector of the resource, which is stored in a vector database.

[0088] Step S4: Hybrid search and push notifications.

[0089] Scenario 1 (Training Retrieval): A trainee inputs a voice question into the system: "How far should a boom lift maintain from live conductors during operation?" The system first parses the entities "insulated boom lift" and "safe distance" from the knowledge graph, and then uses reasoning to link them to relevant regulations and video clips related to this point. Simultaneously, the system converts the voice into text and quantizes it, finding safety regulations documents and video resource vectors containing footage of boom lift operation in the vector library. The system merges these two results and prioritizes pushing the corresponding chapters of the "Safety Regulations" and VR simulation training modules marked with "safe distance reminders."

[0090] Scenario 2 (Assignment Support): During VR simulation training, students perform an "insulator replacement" operation. When the virtual controller is about to touch the simulated conductor, the scenario awareness interface captures this "high-risk approach" scenario in real time. The system immediately triggers a search, quickly locates the "conductor detachment" risk node in the knowledge graph and related handling case videos, and actively pops up the warning case clip in a picture-in-picture format in the VR interface to achieve real-time risk warning.

[0091] System Implementation Examples In this embodiment, the present invention discloses a system for constructing and intelligently retrieving a multimodal resource library for live-line work, used to implement the aforementioned method for constructing and intelligently retrieving a multimodal resource library for live-line work. Figure 2 As shown, the construction and intelligent retrieval system for the multimodal resource library for uninterrupted power supply operations of the present invention mainly includes: a multimodal resource alignment processing module, a knowledge graph construction and management module, a multimodal semantic fusion calculation module, and a hybrid intelligent retrieval and push engine user interaction and context-aware interface module.

[0092] The multimodal resource alignment processing module is used to construct and manage a structured set of associated resources, including text description segments, key visual frames / fragments, VR interaction nodes, and case data fields. More specifically, the multimodal resource alignment processing module integrates time-series alignment units, spatial semantic alignment units, and case standard alignment units to respectively realize the time-series alignment, spatial semantic alignment mapping, and case-standard alignment association of the structured set of associated resources.

[0093] The knowledge graph construction and management module is used to build a cross-modal unified knowledge graph, store and maintain (update and optimize) the unified knowledge graph, and provide knowledge query, reasoning, and incremental update services (including automated joint knowledge extraction and manual review units). Preferably, the knowledge graph construction and management module includes an automated incremental learning unit that can monitor newly added resources, automatically initiate the joint knowledge extraction process, and incrementally update the unified knowledge graph after manual review, thereby realizing the dynamic evolution of the knowledge base.

[0094] The multimodal semantic fusion computation module is used to generate knowledge-enhanced multimodal representation vectors. More specifically, by integrating feature extraction models, graph neural networks, and multimodal fusion Transformers, it generates and maintains knowledge-enhanced multimodal representation vectors for each resource set.

[0095] The hybrid intelligent retrieval and push engine receives multimodal queries and coordinates the invocation of the knowledge graph construction and management module and the multimodal semantic fusion calculation module to perform semantic retrieval and contextualized push based on a hybrid graph and vector approach. Specifically, the hybrid intelligent retrieval and push engine includes a graph semantic retrieval unit, a vector similarity search unit, and a hybrid result fusion and ranking unit, processing multimodal queries and generating push results. The graph semantic retrieval unit invokes the knowledge graph construction and management module for logical reasoning retrieval. The vector similarity search unit invokes the multimodal semantic fusion calculation module for approximate nearest neighbor search (a type of similarity search). The hybrid result fusion and ranking unit executes a weighted fusion algorithm as described above.

[0096] The user interaction and context-aware interface module provides a multimodal query input interface and resource display interface, and connects to external training or operation systems to obtain real-time contextual data to trigger contextualized push notifications. Preferably, this user interaction and context-aware interface module connects to a VR training platform or on-site operation terminal, enabling it to obtain the status of operational objects and environmental parameters in virtual or real operation scenarios in real time, and trigger immediate and proactive push notifications of safety specifications, operation demonstration videos, or VR simulation examples strongly related to the current high-risk operation or complex steps.

[0097] Preferably, the system of the present invention can be deployed on a cloud platform or enterprise intranet using a microservice architecture. The front end provides users with a web portal and VR / mobile applications, integrating user interaction and context-aware interface modules. The backend service cluster includes: a multimodal resource alignment processing service (calling FFmpeg, OpenCV, etc. to process audio and video), a knowledge graph engine service (based on graph databases such as Neo4j), a multimodal semantic fusion computing service (based on model inference using PyTorch / TensorFlow), a hybrid intelligent retrieval service (vector retrieval service, based on dedicated vector databases such as FAISS and Milvus), and a push service. All services communicate through an API gateway, and data storage uses a hybrid database (relational database for metadata, graph database for knowledge graph, and vector database for vector representation).

[0098] For example, the workflow of the system of the present invention is as follows: An administrator uploads a new operation video through the web backend, triggering the resource alignment service to automatically align the video with existing procedures in terms of timing, and triggering the knowledge graph service to extract incremental knowledge. After review, the knowledge graph is updated. Student Li logs into the mobile app to learn about "Live-line Operations in Complex Terrain." The system queries his skill profile and finds that his "Terrain Risk Assessment" skill is weak. Therefore, through the intelligent retrieval service, it proactively pushes relevant typical case videos and assessment formula documents (personalized push) to his learning homepage. Student Wang trains in "Operations in Strong Winds" in a VR simulation cabin. When the simulated wind speed exceeds the safety threshold, the scenario awareness interface immediately captures the event. The intelligent retrieval service is triggered, quickly retrieving the "Prohibition of Operations in Strong Winds" procedure clause and a related accident case, and proactively pushes them in the VR interface with a red warning box and voice broadcast until the student pauses operation.

[0099] It should be noted that in this paper, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply these relationships. There is no such actual relationship or order between entities or operations. Furthermore, the terms "including" and "package" do not apply. The word "comprise" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0101] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0102] In particular, the device embodiments are basically similar to the method embodiments, so they are described in a simpler way. For relevant details, please refer to the description of the method embodiments.

[0103] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this invention, the functions of each unit / module can be implemented in one or more software and / or hardware.

[0104] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for constructing and intelligently retrieving a multimodal resource database for uninterrupted power supply operations, characterized in that, Includes the following steps: S1: Collect the following resources in the field of live-line work: technical documents, courseware, operation videos, VR simulation programs and historical cases; for the set target work theme, extract the core units of the text, visual and interactive content of the relevant resources from the collected resources, and establish cross-resource semantic alignment relationships between the core units to form a structured set of related resources; S2: Based on the domain terminology system and operating procedures, perform joint knowledge extraction on the core units of text, visual and interactive content in the structured associated resource set to obtain common operating entities, technical actions and risk points, and use the extraction results to construct a unified knowledge graph; S3: For the structured set of associated resources, extract its textual semantic features, visual spatial features, and interactive behavior features respectively; Using the unified knowledge graph as a semantic guidance framework, the extracted features are fused with the relevant contextual structure information in the unified knowledge graph to generate a corresponding knowledge-enhanced multimodal representation vector. S4: Receive multimodal queries from users consisting of natural language, diagrams, or video clips; The unified knowledge graph is used to perform semantic parsing and intent expansion on the query to obtain retrieval results based on graph reasoning; The query is transformed into a query vector, and a similarity search is performed in the vector space composed of the knowledge-enhanced multimodal representation vectors to obtain retrieval results based on vector matching. By integrating the retrieval results based on graph reasoning and the retrieval results based on vector matching, candidate resources are comprehensively ranked and personalized.

2. The method according to claim 1, characterized in that, The step S1, which establishes cross-resource semantic alignment relationships between the core units, specifically includes: For the same task, instructional videos and VR simulation programs are time-series aligned using motion recognition and timestamps; For VR simulation programs and technical documents, establish a spatial-semantic alignment mapping between 3D scene models and text descriptions; Case-standard alignment and association are performed by linking on-site data from historical cases with standard procedure clauses.

3. The method according to claim 1, characterized in that, The unified knowledge graph constructed in step S2 includes "personnel skill profile" entity nodes. These entity nodes are connected to "mastered", "needs to learn", and "previously applied" relationships with "skill items", "training resources", and "historical cases" entity nodes, which are used to quantitatively assess skill levels and recommend personalized learning paths.

4. The method according to claim 1, characterized in that, Step S3 is executed for each structured set of associated resources, specifically including: S31: Extract the associated subgraphs of the corresponding nodes of the structured associated resource set in the unified knowledge graph, and use graph neural network encoding to generate the graph context feature vector of the structured associated resource set; S32: Using a pre-trained model, extract the text semantic feature vector, visual spatial feature vector, and interactive behavior feature vector from the core units of text, visual, and interactive content contained in the resource set, respectively. S33: Input the graph context feature vector, text semantic feature vector, visual spatial feature vector, and interaction behavior feature vector into a multimodal fusion Transformer, perform adaptive weighted fusion through a cross-attention mechanism, and output the final knowledge-enhanced multimodal representation vector.

5. The method according to claim 4, characterized in that, Step S4 specifically includes: S41: Entity and relation recognition is performed on the multimodal query, and path traversal and logical reasoning are performed in the unified knowledge graph to obtain the first candidate resource list and its graph relevance score; S42: Transform the multimodal query into a query vector, and perform an approximate nearest neighbor search in the vector space composed of the knowledge-enhanced multimodal representation vectors to obtain a second candidate resource list and its vector similarity score; S43: Merge and deduplicate the first candidate resource list and the second candidate resource list, and calculate the comprehensive ranking score based on the weighted fusion algorithm, combining the graph relevance score and vector similarity score of each resource, to generate a comprehensive ranking list; S44: After personalizing and pushing the comprehensive sorting list by combining user profiles and real-time operation scenarios, the list is pushed out.

6. The method according to claim 5, characterized in that, It also includes step S45: establishing a feedback optimization closed loop, recording user interaction data with pushed resources and subsequent simulated operation performance, and jointly iteratively optimizing the multimodal fusion representation model and the retrieval ranking model, wherein: The multimodal fusion representation model is used to execute step S3 to generate the knowledge-enhanced multimodal representation vector. The retrieval and ranking model is used to execute the logic in step S4, which involves merging and ranking the candidate resource list and then pushing personalized recommendations.

7. A system for constructing and intelligently retrieving a multimodal resource database for live-line work, used to implement the method for constructing and intelligently retrieving a multimodal resource database for live-line work as described in any one of claims 1-6, characterized in that, include: The multimodal resource alignment processing module is used to perform step S1, constructing and managing the structured associated resource set; The knowledge graph construction and management module is used to execute step S2, store, update and provide query and reasoning services for the unified knowledge graph; A multimodal semantic fusion computing module is used to execute step S3, generating and maintaining the knowledge-enhanced multimodal representation vector; The hybrid intelligent retrieval and push engine is used to execute step S4, receive multimodal queries, coordinate the invocation of the knowledge graph construction and management module and the multimodal semantic fusion calculation module, and complete the fusion sorting and push. The user interaction and context-aware interface module is used to provide multimodal query entry points, resource display, and access to external systems to obtain real-time operational context data.

8. The system according to claim 7, characterized in that, The multimodal resource alignment processing module integrates a time-series alignment unit, a spatial semantic alignment unit, and a case standard alignment unit, which respectively realize cross-resource semantic alignment between the core units; the knowledge graph construction and management module includes an automated incremental learning unit that can monitor newly added resources, automatically start the knowledge extraction process, and incrementally update the unified knowledge graph after manual review, thereby realizing the dynamic evolution of the knowledge base.

9. The system according to claim 7, characterized in that, The hybrid intelligent search and push engine includes: The knowledge graph semantic retrieval unit is used to call the knowledge graph construction and management module to perform logical reasoning retrieval. The vector similarity search unit is used to call the multimodal semantic fusion calculation module to perform an approximate nearest neighbor search; The mixed result fusion and sorting unit is used to execute the weighted fusion and sorting logic in the method of claim 5.

10. The system according to claim 7, characterized in that, The user interaction and context awareness interface module is connected to the VR training platform or on-site operation terminal, and can obtain the status of the operation object and environmental parameters in the virtual or real operation scenario in real time, triggering the immediate and proactive push of safety specifications, operation demonstration videos or VR simulation examples that are strongly related to the current high-risk operation or complex steps.

Images