Transformer fault maintenance virtual training method and system based on multi-mode cooperation

Abstract

The invention provides a transformer fault maintenance virtual training method and system based on multi-modal cooperation. The method comprises the following steps: constructing a multi-modal knowledge graph of transformer fault data; pre-testing the trainee by adopting a preset collaborative training system, determining the initial proficiency of the trainee, and determining an eye movement triggering threshold of the trainee by adopting a preset fault complexity optimization algorithm and a preset eye movement triggering threshold algorithm; according to the eye movement data of the trainee and the eye movement triggering threshold value of the trainee, fault associated data in the multi-mode knowledge graph are called, a preset virtual-real feedback intensity algorithm is adopted, fault interaction data of the trainee and the collaborative training system are obtained, and then a personalized training scheme is generated. According to the invention, trainees are assisted to construct a complete fault maintenance knowledge framework, knowledge fragmentation is avoided, association understanding is enhanced, the problem of missing practical operation is solved, a personalized training scheme is generated, the training pertinence and efficiency are improved, and the skill improvement period is shortened.

Description

Technical Field

[0001] This application relates to the field of power equipment maintenance training technology, specifically to a virtual training method and system for transformer fault maintenance based on multimodal collaboration. Background Technology

[0002] As a core piece of equipment in the power system, the stable operation of transformers directly determines the reliability and security of power supply. Once a fault occurs, it will cause power outages, affecting industrial production, commercial operations and residents' lives, and even causing significant economic losses. Therefore, training professional transformer fault repair personnel is crucial to ensuring the stability of the power system.

[0003] With the continuous development of the power industry, transformer equipment structures are becoming increasingly complex, and fault types are becoming more diverse. The skill requirements for maintenance personnel have evolved from simple fault handling to a comprehensive capability encompassing "identification, analysis, processing, and mechanism understanding." Currently, technologies such as virtual reality, multimodal interaction, and knowledge graphs are gradually penetrating the field of industrial training. These technologies can overcome the physical limitations of traditional training, transforming abstract fault mechanisms and complex equipment structures into intuitive and perceptible content. This provides the necessary technical foundation for building an efficient and safe transformer fault maintenance training system, and promotes the development of maintenance training towards intelligence and immersion.

[0004] Traditional transformer fault diagnosis and repair training mainly relies on theoretical lectures and limited hands-on demonstrations. Theoretical knowledge is primarily conveyed through text and images, making it difficult for trainees to intuitively understand abstract fault characteristic descriptions and internal equipment structures, resulting in low knowledge absorption efficiency. Furthermore, it fails to establish a connection between fault types, equipment components, and mechanisms. Using actual transformers for practical training not only requires significant costs for equipment purchase and maintenance but also poses safety risks associated with high-voltage operation. It also makes it difficult to simulate various typical fault scenarios and allows trainees insufficient time to practice fault handling procedures, thus limiting the breadth and depth of the training.

[0005] Furthermore, traditional training lacks consideration for individual differences among trainees, adopts uniform training content and pace, and ignores the learning needs of trainees with different backgrounds. This results in those with weak foundations struggling to keep up with the pace, while those with better foundations struggle to improve. At the same time, the lack of systematic integration in the knowledge transfer process easily leads to fragmented knowledge among trainees, making it difficult to respond quickly and accurately in actual fault handling. Summary of the Invention

[0006] In view of one of the deficiencies in the prior art, the purpose of this application is to provide a virtual training method and system for transformer fault repair based on multimodal collaboration.

[0007] The first aspect of this application provides a virtual training method for transformer fault repair based on multimodal collaboration, comprising: Collect real transformer fault sample data and construct a multimodal knowledge graph with component-fault-multimodal features-mechanism semantic hierarchy; A pre-set collaborative training system is used to conduct pre-tests on trainees to determine their initial proficiency and initialize the training scenario; Based on the trainee's initial proficiency, a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm are used to determine the trainee's eye-tracking trigger threshold. Based on the trainee's eye movement data and the trainee's eye movement trigger threshold, the fault association data in the multimodal knowledge graph is called, and a preset virtual-real feedback intensity algorithm is used to obtain the fault interaction data between the trainee and the virtual transformer scenario in the preset collaborative training system. A personalized training plan is generated based on the trainee's eye movement data and the fault interaction data.

[0008] Optionally, the preset collaborative training system includes an MR headset, a vibration simulator, a force feedback handle, and a voice interaction unit. The MR headset includes an eye-tracking module, a cross-modal fault feature injection module, and a virtual-real interaction feedback module. The preset collaborative training system is used to create the virtual transformer scene and to interact with the trainee in virtual reality.

[0009] Optionally, the step of collecting real transformer fault sample data and constructing a multimodal knowledge graph at the component-fault-multimodal feature-mechanism semantic level includes: Collect real transformer fault sample data, the types of faulty components included in the real transformer fault sample data include radiators, breather pipes and bushings; The real fault sample data of the transformer is cleaned and labeled to determine the transformer type, transformer component type, fault type, multimodal characteristic parameters, maintenance steps and mechanism analysis information corresponding to each real fault sample data of the transformer. Based on the transformer type, transformer component type, fault type, multimodal characteristic parameters, maintenance steps, and mechanism analysis information corresponding to each actual transformer fault sample data, the core nodes of the multimodal knowledge graph are defined. The core nodes include transformer type nodes, component type nodes, fault type nodes, multimodal characteristic nodes, maintenance step nodes, and mechanism analysis nodes. Based on the core nodes of the multimodal knowledge graph, the relationship edges between the core nodes are defined to generate the multimodal knowledge graph. The relationship edges between the core nodes include the occurrence relationship between component type and fault type, the inclusion relationship between fault type and multimodal features, the correspondence between fault type and maintenance steps, and the association relationship between fault type and mechanism analysis.

[0010] Optionally, the step of using a preset collaborative training system to pre-test trainees, determine their initial proficiency, and initialize the training scenario includes: The trainees are pre-tested using pre-set pre-test questions to determine their pre-test scores. If the trainee's pretest score is less than a preset first score threshold, the trainee's initial proficiency is determined to be the first proficiency. If the trainee's pretest score is not less than the preset first score threshold and is less than the preset second score threshold, the trainee's initial proficiency is determined to be the second proficiency, where the preset first score threshold is less than the preset second score threshold. If the trainee's pretest score is greater than the preset second score threshold, the trainee's initial proficiency is determined to be the third proficiency, where the first proficiency is less than the second proficiency, and the second proficiency is less than the third proficiency.

[0011] Optionally, determining the eye-tracking trigger threshold of the trainee based on the trainee's initial proficiency using a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm includes: Based on the trainee's initial proficiency, a preset fault complexity optimization algorithm is used to determine the initial fault feature complexity corresponding to the trainee's initial proficiency. Based on the initial fault feature complexity corresponding to the trainee's initial proficiency, the preset eye-tracking trigger threshold algorithm is used to determine the trainee's eye-tracking trigger threshold.

[0012] Optionally, the method further includes: The eye-tracking module of the MR headset is used to collect the eye movement data of the trainee. The eye movement data includes the three-dimensional coordinates of the fixation point, the duration of fixation point dwell, and the eye movement trajectory. The eye movement data of the trainees were denoised and smoothed to determine the denoised and smoothed eye movement data. Feature extraction is performed on the denoised and smoothed eye-tracking data to determine the eye-tracking trajectory entropy value and the gaze region distribution characteristics, and the processed eye-tracking data is determined. The processed eye-tracking data includes the three-dimensional coordinates of the gaze point, the gaze point dwell time, the eye-tracking trajectory, the eye-tracking trajectory entropy value, and the gaze region distribution characteristics.

[0013] Optionally, the method further includes: The cross-modal fault feature injection module of the MR headset calls the fault association data in the multimodal knowledge graph and uses a rendering engine to generate visual features; The visualization features are spatially bound to the transformer components in the virtual transformer scene.

[0014] Optionally, the step of calling fault association data from the multimodal knowledge graph based on the trainee's eye-tracking data and the trainee's eye-tracking trigger threshold, and using a preset virtual-real feedback intensity algorithm to obtain fault interaction data between the trainee and the virtual transformer scenario in the preset collaborative training system, includes: If the duration of fixation in the trainee's eye movement data is greater than the trainee's eye movement trigger threshold, determine the virtual transformer component corresponding to the three-dimensional coordinates of the fixation point; Based on the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point, the visualization features corresponding to the virtual transformer component are displayed in the cross-modal fault feature injection module of the MR headset. Based on the trainee's initial proficiency and the visualization features corresponding to the virtual transformer component with the three-dimensional coordinates of the gaze point, the virtual-real feedback intensity is determined using the preset virtual-real feedback intensity algorithm. Based on the intensity of the virtual and real feedback, the vibration frequency and amplitude of the vibration simulator and the operating resistance of the force feedback handle are determined. Based on the vibration frequency and amplitude of the vibration simulator and the operating resistance of the force feedback handle, the virtual-real interaction feedback module controls the vibration simulator and the force feedback handle to output physical feedback response information. The virtual-real interaction feedback module controls the voice interaction unit to play the fault mechanism analysis of the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point; Collect fault operation data of the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point of the trainee; The step of generating a personalized training plan based on the trainee's eye-tracking data and the fault interaction data includes: The fault operation data of the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point of the trainee are evaluated in real time to obtain the fault operation evaluation results of the trainee. The eye-tracking data of the trainee, the physical feedback response information output by the vibration simulator and the force feedback handle, and the fault operation evaluation results of the trainee are recorded simultaneously to generate a personalized training plan for the trainee.

[0015] A second aspect of this application provides a virtual training system for transformer fault repair based on multimodal collaboration, comprising: The multimodal knowledge graph construction module is used to collect real transformer fault sample data and construct a multimodal knowledge graph with a semantic hierarchy of components, faults, multimodal features, and mechanisms. The scenario initialization module is used to pre-test the trainees using a preset collaborative training system, determine the trainees' initial proficiency, and initialize the training scenario. An eye-tracking trigger threshold determination module is used to determine the eye-tracking trigger threshold of the trainee based on the trainee's initial proficiency, using a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm. The virtual reality fault interaction module is used to call fault association data in the multimodal knowledge graph based on the trainee's eye movement data and the trainee's eye movement trigger threshold, and to obtain fault interaction data between the trainee and the virtual transformer scene in the preset collaborative training system using a preset virtual-real feedback intensity algorithm. The personalized training program generation module is used to generate a personalized training program based on the trainee's eye movement data and the fault interaction data.

[0016] A third aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0017] A fourth aspect of this application provides an electronic device comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods provided in the first aspect of this application.

[0018] This application presents a virtual training method for transformer fault repair based on multimodal collaboration. It employs a pre-set collaborative training system to pre-test trainees, obtaining their initial proficiency (initial level) and initializing the training scenario to effectively understand their current knowledge level. Furthermore, it uses a pre-set fault complexity optimization algorithm and a pre-set eye-tracking trigger threshold algorithm to set appropriate initial fault complexity and eye-tracking trigger thresholds for each trainee. This provides personalized training starting points for trainees with different backgrounds, breaking the limitations of traditional standardized training and generating personalized training plans that ensure both training difficulty and efficiency. Matching trainees' abilities to their abilities avoids training effectiveness being affected by inappropriate difficulty, improves the relevance and efficiency of training, and shortens the skill improvement cycle. By capturing trainees' gaze behavior through eye-tracking data, when the trainee's eye-tracking trigger threshold is reached, the fault-related data corresponding to the training fault is invoked. A preset virtual-real feedback intensity algorithm is used to realize virtual reality interaction between the trainee and the virtual transformer scene in the preset collaborative training system. Trainees can actively acquire fault-related knowledge in the immersive scene, avoiding the problems of knowledge fragmentation and lack of practical operation, and helping trainees quickly build a complete fault diagnosis and repair knowledge framework.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a virtual training method for transformer fault repair based on multimodal collaboration, according to an exemplary embodiment.

[0021] Figure 2 This is a schematic diagram illustrating the structure of a virtual training system for transformer fault repair based on multimodal collaboration, according to an exemplary embodiment. Detailed Implementation

[0022] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0023] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0025] Traditional transformer fault repair training suffers from insufficient realism in simulated scenarios, lack of multi-source data fusion, and fixed safety thresholds, resulting in poor training effectiveness. Using actual transformers for hands-on training is costly and poses safety risks. Furthermore, existing transformer fault repair training lacks consideration for individual trainees' differences, employing uniform content and pace, making it difficult for those with weak foundations to keep up and hindering the improvement of those with stronger foundations. The knowledge transfer process lacks systematic integration, leading to fragmented knowledge. Mastering repair skills relies on manual experience and judgment, and the training effectiveness lacks systematic quantitative evaluation, making it difficult for trainees to handle actual faults. Based on these problems, this application provides a virtual training method and system for transformer fault repair based on multimodal collaboration to address the aforementioned issues.

[0026] Figure 1 This is a flowchart illustrating a virtual training method for transformer fault repair based on multimodal collaboration, according to an exemplary embodiment.

[0027] Reference Figure 1 As shown in one embodiment of this application, a virtual training method for transformer fault repair based on multimodal collaboration is provided, including S11 to S15.

[0028] S11. Collect real transformer fault sample data and construct a multimodal knowledge graph with component-fault-multimodal features-mechanism semantic hierarchy.

[0029] S12, a pre-set collaborative training system is used to pre-test the trainees, determine their initial proficiency, and initialize the training scenario.

[0030] S13. Based on the trainee's initial proficiency, a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm are used to determine the trainee's eye-tracking trigger threshold.

[0031] S14. Based on the trainee's eye movement data and the trainee's eye movement trigger threshold, call the fault association data in the multimodal knowledge graph, and use the preset virtual-real feedback intensity algorithm to obtain the fault interaction data between the trainee and the virtual transformer scenario in the preset collaborative training system.

[0032] Specifically, the fault interaction data between the trainee and the virtual transformer scenario in the pre-set collaborative training system includes the visual characteristics of the transformer components in the displayed virtual transformer scenario, the physical feedback response information output by the vibration simulator and force feedback handle, and the fault mechanism analysis of the transformer components in the played virtual transformer scenario.

[0033] S15 generates a personalized training plan based on the trainee's eye movement data and fault interaction data.

[0034] Specifically, the trainees' eye movement data and fault interaction data are recorded simultaneously to generate personalized training programs.

[0035] The embodiments described above employ a pre-set collaborative training system to pre-test trainees, obtaining their initial proficiency (initial level) and initializing the training scenario. This effectively understands the current knowledge level of the trainees. Pre-set fault complexity optimization and eye-tracking trigger threshold algorithms are used to set appropriate initial fault complexity and eye-tracking trigger thresholds for each trainee, providing personalized training starting points for trainees with different skill levels. This breaks the limitations of traditional standardized training, generating personalized training plans for trainees. It ensures that the training difficulty matches the trainee's ability, avoiding inappropriate difficulty from affecting training effectiveness, improving the relevance and efficiency of training, and shortening the skill improvement cycle. By capturing the trainee's gaze behavior through eye-tracking data, when the trainee's eye-tracking trigger threshold is reached, fault-related data corresponding to the training fault is invoked. A pre-set virtual-real feedback intensity algorithm is used to achieve virtual reality interaction between the trainee and the virtual transformer scenario in the pre-set collaborative training system. Trainees can actively acquire fault-related knowledge in an immersive scenario, avoiding fragmented knowledge and lack of practical experience, and helping trainees quickly build a complete fault diagnosis and repair knowledge framework.

[0036] In some specific embodiments of this application, the preset collaborative training system includes an MR headset, a vibration simulator, a force feedback handle, and a voice interaction unit. The MR headset includes an eye-tracking module, a cross-modal fault feature injection module, and a virtual-real interaction feedback module. The preset collaborative training system is used to create a virtual transformer scene and to interact with the trainee in virtual reality.

[0037] Specifically, the eye-tracking module is used to collect and process the trainee's eye-tracking data; the cross-modal fault feature injection module is used to call the multimodal knowledge graph to generate visual features and spatially bind them to the transformer component faults in the virtual transformer scene, and also dynamically loads and displays the visual features corresponding to the faulty components when triggered by the processed trainee's eye-tracking data; the virtual-real interaction feedback module is used to control the vibration simulator and force feedback handle to output physical feedback response information, and control the voice interaction unit to play the fault mechanism analysis of the transformer components in the virtual transformer scene.

[0038] The eye-tracking module synchronously transmits the processed eye-tracking data to the cross-modal fault feature injection module and the virtual-real interaction feedback module.

[0039] The pre-defined collaborative training system in this application is a multimodal collaborative virtual reality training system.

[0040] The embodiments described above in this application construct a pre-defined collaborative training system using an MR head-mounted display, a vibration simulator, a force feedback handle, and a voice interaction unit. This system creates a virtual transformer scenario without requiring high-cost equipment purchases or maintenance, and eliminates the safety risks associated with high-voltage operation. It simulates various fault scenarios, allowing trainees to repeatedly practice fault handling procedures in an immersive environment, thus expanding the breadth and depth of their training.

[0041] To construct a multimodal knowledge graph, in some specific embodiments of this application, for S11, collecting real transformer fault sample data and constructing a multimodal knowledge graph at the component-fault-multimodal feature-mechanism semantic level can be adopted from S111 to S114.

[0042] S111, collect real fault sample data of transformer.

[0043] Specifically, the types of faulty components included in the actual transformer fault sample data include radiators, breather pipes, and bushings.

[0044] The collected sample data of real transformer faults include sample data of real transformer faults such as abnormal radiator oil temperature, abnormal breather color, and bushing oil leakage.

[0045] The methods for collecting actual transformer fault sample data may include, but are not limited to: First, historical maintenance records are retrieved from the power operation and maintenance database. For example, the historical maintenance records of 110kV transformers over the past three years are retrieved from the power operation and maintenance database, and 200 records of abnormal radiator oil temperature are extracted. These real records can make the fault cases in subsequent training scenarios more in line with actual operation and maintenance conditions, avoiding virtual simulations that are detached from the actual site.

[0046] Secondly, fault phenomena and handling data are recorded simultaneously during on-site maintenance. For example, during on-site maintenance of a 110kV transformer in a substation, the fault phenomenon of the breather changing from blue to pink and the handling process data of oil leakage from the bushing flange are recorded simultaneously. This ensures that the collected fault characteristic data is consistent with the actual performance of the equipment during operation, thereby making the fault characteristics encountered by trainees more realistic.

[0047] Thirdly, laboratory simulations were used to reproduce typical faults such as abnormal radiator oil temperature, abnormal breather color, and sleeve oil leakage, and corresponding parameters were collected. For example, the laboratory simulations reproduced three typical faults and collected corresponding test data: simulating abnormal oil temperature due to radiator fin blockage, an infrared thermal imager was used to capture the surface temperature distribution of the radiator, and oil temperature data was collected in real time using an oil temperature sensor; simulating silicone saturation in the breather, a high-definition industrial camera was used to capture color changes, and a humidity sensor was used to collect the internal humidity of the breather; simulating aging and leakage of the sleeve gasket, an oil concentration sensor was used to collect the oil mist concentration. Laboratory simulations can supplement detailed data that is difficult to capture on-site, improve multimodal feature parameters, and provide a comprehensive data foundation for subsequent knowledge graphs.

[0048] S112, clean and label the real fault sample data of transformers, and determine the transformer type, transformer component type, fault type, multimodal characteristic parameters, maintenance steps and mechanism analysis information corresponding to each real fault sample data of transformers.

[0049] Specifically, for each transformer's real fault sample data, information on three types of faults—abnormal radiator oil temperature, abnormal breather color, and bushing oil leakage—is extracted, including transformer type, transformer component type, fault type, multimodal characteristics, maintenance plan, and mechanism analysis.

[0050] S113. Based on the transformer type, transformer component type, fault type, multimodal characteristic parameters, maintenance steps, and mechanism analysis information corresponding to the actual fault sample data of each transformer, define the core nodes of the multimodal knowledge graph.

[0051] Specifically, the core nodes include transformer type nodes, component type nodes, fault type nodes, multimodal characteristic nodes, maintenance step nodes, and mechanism analysis nodes.

[0052] For example, this application takes a 110kV transformer as an example. The transformer type node is 110kV transformer, the component type nodes include: radiator, breather, bushing; the fault type nodes include: abnormal oil temperature of radiator, abnormal color of breather, oil leakage of bushing; the multimodal feature nodes include: temperature, humidity, oil mist concentration, color.

[0053] S114. Based on the core nodes of the multimodal knowledge graph, define the relationship edges between the core nodes to generate the multimodal knowledge graph.

[0054] Specifically, the relationships between core nodes include the relationship between component type and fault type, the relationship between fault type and multimodal features, the relationship between fault type and maintenance steps, and the relationship between fault type and mechanism analysis.

[0055] The embodiments described above in this application construct a knowledge graph at the component-fault-multimodal feature-mechanism semantic level. By cleaning and labeling real transformer fault sample data, invalid data interference can be eliminated, ensuring the accuracy of each sample information and providing clear and accurate information basis for the definition of knowledge graph nodes. By defining the core nodes of the multimodal knowledge graph and the relationship edges between core nodes, it can be ensured that in subsequent training, when it is necessary to retrieve the fault information of a certain component, the corresponding fault type, multimodal features, maintenance steps, and mechanism analysis information can be quickly matched, avoiding data retrieval chaos and ensuring the accuracy of information transmission during the training process.

[0056] To enable initial skill testing of trainees and initialization of the training scenario, trainees wear MR headsets and hold force feedback handles to enter a virtual transformer scenario, thus initializing the training scenario. The immersive display of the MR headset allows trainees to quickly immerse themselves in the maintenance scenario, while the grip of the force feedback handles allows trainees to get used to the operating tools in advance, reducing the unfamiliarity of subsequent operations.

[0057] To assess the trainees' initial abilities, in some specific embodiments of this application, for S12, a preset collaborative training system is used to pre-test the trainees, determine the trainees' initial proficiency, and initialize the training scenario. This can be achieved through S121 to S124.

[0058] S121, Use pre-set pre-test questions to conduct pre-tests on trainees and determine their pre-test scores.

[0059] Specifically, the pre-test includes a theoretical test and a basic operation test. The theoretical test includes 10 multiple-choice questions about the normal oil temperature range of the radiator, abnormal color of the breather, and common locations of bushing leakage. The basic operation test requires trainees to locate the radiator, breather, and bushing in a virtual transformer scenario and complete simple disassembly operations.

[0060] Pre-testing trainees to assess their initial abilities.

[0061] S122, If the trainee's pretest score is less than the preset first score threshold, the trainee's initial proficiency is determined as the first proficiency.

[0062] For example, the preset first score threshold is 60 points.

[0063] For example, when the pretest score is <60, the trainee's initial proficiency S is 0.2.

[0064] S123, if the trainee's pretest score is not less than a preset first score threshold and is less than a preset second score threshold, the trainee's initial proficiency is determined to be the second proficiency.

[0065] Specifically, the preset first score threshold is less than the preset second score threshold.

[0066] For example, the preset second score threshold is 80 points.

[0067] For example, when 60 ≤ pretest score < 80, the trainee’s initial proficiency S is 0.5.

[0068] S124, If the trainee's pretest score is greater than the preset second score threshold, the trainee's initial proficiency is determined to be the third proficiency.

[0069] Specifically, the first level of proficiency is less than the second level of proficiency, and the second level of proficiency is less than the third level of proficiency.

[0070] For example, when the pretest score is >80, the trainee’s initial proficiency S is 0.8.

[0071] For example, based on the above method, if the trainee's theoretical test score is 52 points and the basic operation only involves positioning two parts, the pre-test score is 52 points. According to the rule, when the pre-test score is <60 points, S=0.2.

[0072] The embodiments described above in this application use pre-test questions to pre-test trainees, obtain their pre-test scores, and convert the pre-test scores into the trainees' initial proficiency based on preset first and second score thresholds. This initial proficiency parameter allows the subsequent training difficulty to be tailored to the trainees' actual abilities, providing personalized training starting points for trainees with different levels of experience, breaking the limitations of traditional standardized training, and ensuring the effectiveness of the training.

[0073] To determine the eye-tracking trigger threshold of the trainee, in some specific embodiments of this application, for S13, based on the trainee's initial proficiency, a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm are used to determine the trainee's eye-tracking trigger threshold, which can be done using S131 to S132.

[0074] S131, Based on the trainee's initial proficiency, a preset fault complexity optimization algorithm is used to determine the initial fault feature complexity corresponding to the trainee's initial proficiency.

[0075] Specifically, the preset fault complexity optimization algorithm is as follows: in, This represents the final set complexity of the initial fault characteristics. Indicates the maximum complexity. Indicates the number of current fault types. Indicates the number of cross-modal features. Indicates the first The coupling coefficient of this feature Indicates the first The normalized value of the physical intensity of this feature. This indicates the trainee's initial proficiency level.

[0076] Among them, the maximum complexity Given constant quantities, The pre-defined known feature coupling weight coefficients are known quantities. The number of current fault types n and the number of cross-modal features m are obtained based on the fault types and associated features corresponding to the initial fault feature complexity corresponding to the trainee's initial proficiency. Specifically, the associated features may include, but are not limited to, temperature features, vibration features, and acoustic features. The value is calculated in real time based on the initial fault feature complexity corresponding to the trainee's initial proficiency and the fault type and associated features.

[0077] Specifically, cross-modal features refer to the visual features generated by the cross-modal fault feature injection module, which may include, but are not limited to, temperature cloud maps and fault waveform diagrams.

[0078] Fault complexity is used to define the degree of feature coupling of the initial fault.

[0079] Following the example above, with trainee S=0.2, the initial fault can be set as a single abnormal radiator oil temperature, resulting in a low degree of feature coupling.

[0080] S132, based on the initial fault feature complexity corresponding to the trainee's initial proficiency, a preset eye-tracking trigger threshold algorithm is used to determine the trainee's eye-tracking trigger threshold.

[0081] Specifically, the preset eye-tracking trigger threshold algorithm is as follows: in, Indicates the eye-tracking trigger threshold. Indicates the basic trigger threshold. This represents the adjustment coefficient. Indicates the complexity of the initial fault characteristics. This indicates the trainee's initial proficiency level.

[0082] The preset eye-tracking trigger threshold represents the minimum fixation duration required to activate fault training.

[0083] The embodiments described above in this application determine the initial fault feature complexity based on the trainee's initial proficiency using a preset fault complexity optimization algorithm. This sets faults corresponding to the trainee's initial level, gradually building maintenance awareness, enhancing learning confidence, and avoiding initial training setbacks due to excessively high fault complexity or reduced training efficiency due to excessively low fault complexity. This helps trainees gradually master transformer fault maintenance skills. Furthermore, based on the trainee's initial proficiency and initial fault feature complexity, an eye-tracking trigger threshold is determined. This ensures that when the trainee's fixation time reaches the eye-tracking trigger threshold, a fault corresponding to the trainee's initial level is triggered, providing personalized training starting points for different trainees. This breaks the limitations of traditional standardized training and significantly improves the relevance and efficiency of training.

[0084] In some specific embodiments of this application, a virtual training method for transformer fault repair based on multimodal collaboration may further include S16 to S18.

[0085] S16, The eye-tracking module of the MR headset collects the eye movement data of the trainee.

[0086] Specifically, eye-tracking data includes the three-dimensional coordinates of the fixation point, the duration of fixation, and the eye-tracking trajectory.

[0087] S17, Denoise and smooth the trainee's eye movement data to determine the denoised and smoothed eye movement data.

[0088] For example, the denoising process can employ wavelet thresholding denoising technology, which decomposes the eye-tracking signal into multiple scales using the db4 wavelet basis, and reconstructs the noisy wavelet coefficients after setting a soft threshold to remove random noise.

[0089] Smoothing can be achieved by using moving average filtering technology. By setting a sliding window with 5 sampling points, the mean within the window is calculated point by point for time series data such as the three-dimensional coordinates of the gaze point and the duration of gaze, thus achieving data smoothing.

[0090] Specifically, noise reduction and smoothing processes are used to remove blink data and noise generated by rapid head movements from the trainees' eye movement data.

[0091] S18. Feature extraction is performed on the denoised and smoothed eye-tracking data to determine the entropy value of the eye-tracking trajectory and the distribution characteristics of the fixation area, and to determine the processed eye-tracking data. Specifically, the sample entropy algorithm is used to extract the eye-track entropy value, and the specific steps are as follows: Three-dimensional coordinate time series of eye movement trajectory Set embedding dimensions Similarity tolerance , Representing the standard deviation of a time series; constructing dimensional vector Calculate the maximum absolute difference between the vectors: Statistical satisfaction number of vector pairs Calculate the probability ; Increase embedding dimension to Repeat the above steps to obtain Sample Entropy This characterizes the complexity of the trajectory; The K-means clustering algorithm was used to extract the gaze region distribution features. The specific steps are as follows: 3D coordinates of the gaze point As a sample, set the number of clusters. ; Random initialization Cluster centers ; Calculate the Euclidean distance from each sample to each cluster center: in, Indicates the first i Three-dimensional coordinates of a gaze point Indicates the first j The cluster centers of each cluster. Indicates the first i Each gaze point x Axis coordinates Indicates the first i Each gaze point y Axis coordinates Indicates the first i Each gaze point z Axis coordinates Indicates the first j Cluster centers of each cluster x Axis coordinates Indicates the first j Cluster centers of each cluster y Axis coordinates Indicates the first j Cluster centers of each cluster z Axis coordinates.

[0092] Assign the sample to the nearest cluster; Update the cluster centers to the mean of all samples within the cluster. ,in, Cluster The number of samples is calculated; the distance calculation and center update steps are repeated until the cluster centers converge, and the clustering features such as the distribution density and number of regions of the gaze region are obtained, which are used as the distribution features of the gaze region.

[0093] Steps S16 to S18 of this application can be performed after step S13 and before step S14.

[0094] In the above embodiments of this application, noise reduction and smoothing processes can eliminate the interference of invalid data on subsequent triggering logic, ensuring that the extracted eye movement trajectory entropy value and gaze area distribution features can truly reflect the trainee's focus of attention. Based on the processed eye movement data, real-time linkage between eye movement behavior, feature display, and physical feedback can be achieved, avoiding training experience lag caused by data delays.

[0095] To achieve visualized display of fault characteristics, in some specific embodiments of this application, a virtual training method for transformer fault repair based on multimodal collaboration may further include S19 to S20.

[0096] S19 uses the cross-modal fault feature injection module of the MR headset to call fault association data in the multimodal knowledge graph and uses a rendering engine to generate visual features.

[0097] S20 spatially binds visual features to transformer components in a virtual transformer scene.

[0098] Specifically, steps S19 to S20 of this application can be performed after step S18 or simultaneously during step S14.

[0099] The embodiments described above in this application call fault-related data from a multimodal knowledge graph and spatially bind visualization features with transformer components in a virtual transformer scene to achieve precise binding of multimodal information with the virtual transformer scene. This facilitates the dynamic loading of fault visualization features when the trainee's eye-tracking trigger threshold is reached, enabling the trainee to actively acquire fault-related knowledge in an immersive scene and avoiding the problem of knowledge fragmentation in traditional training.

[0100] In some specific embodiments of this application, for S14, fault association data in the multimodal knowledge graph is called based on the trainee's eye movement data and the trainee's eye movement trigger threshold, and a preset virtual-real feedback intensity algorithm is used to obtain fault interaction data between the trainee and the virtual transformer scenario in the preset collaborative training system. This can be implemented as S141 to S147.

[0101] S141, if the fixation duration in the trainee's eye movement data is greater than the trainee's eye movement trigger threshold, determine the virtual transformer component corresponding to the three-dimensional coordinates of the fixation point.

[0102] S142, based on the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point, display the visualization features corresponding to the virtual transformer component in the cross-modal fault feature injection module of the MR headset.

[0103] Specifically, based on the above step S20, the visualization features are spatially bound to the transformer components in the virtual transformer scene, which enables the dynamic loading and display of visualization feature content based directly on eye-tracking trigger signals.

[0104] For example, different display visualization features are executed for different fault types. When the radiator oil temperature is abnormal, the surface temperature cloud map of the radiator, the schematic diagram of the heat sink blockage, and the oil temperature change curve are displayed. When the breather color is abnormal, the 3D model of the breather silicone canister, the humidity value label, and the schematic diagram of the silicone moisture absorption principle are displayed. When the sleeve oil leakage is triggered, the dynamic oil traces at the sleeve flange, the oil mist concentration value label, and the cross-sectional view of the sealing structure are displayed.

[0105] S143. Based on the trainee's initial proficiency and the visualization features corresponding to the virtual transformer component with the three-dimensional coordinates of the gaze point, a preset virtual-real feedback intensity algorithm is used to determine the virtual-real feedback intensity.

[0106] Specifically, the preset algorithm for virtual-real feedback intensity is as follows: in, Indicates the intensity of virtual-real feedback. Indicates the first The normalized value of the physical intensity of this feature. Indicates the basic feedback coefficient. Indicates the first Feedback weights for various features This indicates the trainee's initial proficiency level.

[0107] S144, based on the intensity of the virtual and real feedback, determine the vibration frequency and amplitude of the vibration simulator, as well as the operating resistance of the force feedback handle.

[0108] S145, based on the vibration frequency and amplitude of the vibration simulator and the operating resistance of the force feedback handle, controls the vibration simulator and the force feedback handle to output physical feedback response information through the virtual-real interaction feedback module; Specifically, the physical feedback response information includes the vibration frequency, vibration amplitude, and operating resistance output by the force feedback handle of the vibration simulator.

[0109] S146, through the virtual-real interaction feedback module, the voice interaction unit is controlled to play the fault mechanism analysis of the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point.

[0110] S147: Collect fault operation data of virtual transformer components corresponding to the three-dimensional coordinates of the gaze point from the trainee.

[0111] In the embodiments described above, the cross-modal fault feature injection module of the MR headset visually displays the three-dimensional coordinates of the trainee's eye gaze point corresponding to the visualized features of the virtual transformer component. It can provide targeted information based on the trainee's focus, avoiding irrelevant information interference, improving training concentration, and enabling the trainee to actively acquire fault-related knowledge in an immersive scenario, helping them to quickly understand the fault and avoiding misunderstandings caused by the separation of features and components. The virtual-real interaction feedback module, under the trainee's eye-tracking trigger signal, synchronously controls the vibration simulator to output the frequency and amplitude corresponding to the fault features, allowing the trainee to perceive the severity of the fault through touch, enhancing multi-dimensional understanding of the fault. The control force feedback handle adjusts the operating resistance according to the feature intensity, making virtual operation closer to the feel of real maintenance, helping the trainee become familiar with force control in actual operation. The virtual-real feedback module also controls the voice interaction unit to play a fault mechanism analysis of the virtual transformer component under the trainee's eye-tracking trigger signal, allowing the trainee to simultaneously acquire mechanistic knowledge while observing fault features, achieving collaborative learning of vision, touch, and hearing, and deepening understanding and memory of the fault.

[0112] In order to generate personalized training for trainees, in some specific embodiments of this application, for S15, a personalized training plan is generated based on the trainee's eye movement data and fault interaction data, which can be implemented using S151 to S152.

[0113] S151, perform real-time evaluation of the fault operation data of the virtual transformer component corresponding to the three-dimensional coordinates of the trainee's gaze point, and obtain the trainee's fault operation evaluation results.

[0114] Specifically, the correctness of the trainee's faulty operations is evaluated in real time.

[0115] S152 synchronously records the trainee's eye-tracking data, the physical feedback response information output by the vibration simulator and force feedback handle, as well as the trainee's fault operation evaluation results, and generates a personalized training plan for the trainee.

[0116] Specifically, the trainees' eye movement data includes the three-dimensional coordinates of the fixation point, the duration of fixation, and the eye movement trajectory.

[0117] The physical feedback response information output by the vibration simulator and the force feedback handle includes the vibration frequency and amplitude of the vibration simulator and the operating resistance output by the force feedback handle.

[0118] The evaluation results of trainees' fault operation include the correctness of the trainees' fault operation procedures.

[0119] The embodiments described above in this application evaluate and archive the fault operations of trainees, providing support for standardized training of maintenance skills and traceability of training effectiveness. Based on the archived data, personalized training programs are generated, ensuring that the training difficulty matches the trainees' abilities and avoiding the impact of inappropriate difficulty on learning effectiveness. At the same time, the training effectiveness can be continuously tracked through data archiving, dynamically optimizing the training content and pace, significantly improving the relevance and efficiency of training, helping trainees to gradually master transformer fault maintenance skills, and shortening the skills improvement cycle.

[0120] This application provides a virtual training method for transformer fault repair based on a multimodal system. This method enables trainees to actively acquire fault-related knowledge in an immersive environment, addressing the problems of fragmented knowledge and lack of practical application in traditional training. It strengthens the understanding of the correlation between fault types, characteristics, and mechanisms, helping trainees quickly build a complete fault repair knowledge framework. The method sets initial fault feature complexity, eye-tracking trigger threshold, and virtual-real feedback intensity based on the trainee's initial proficiency level, providing personalized training starting points for trainees with different skill levels. This breaks the limitations of traditional standardized training. Furthermore, the method records and evaluates fault operation data in real time during training, generating personalized training plans, improving training relevance and efficiency, shortening the skill improvement cycle, and effectively enhancing the training effect of transformer fault repair, thus achieving intelligent, safe, and efficient transformer repair training.

[0121] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0122] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0123] This paper uses a virtual training method for transformer fault repair based on multimodal collaboration provided in this application, taking the training on the repair of 110kV transformer winding deformation fault as an example, to illustrate the training of newly hired maintenance personnel on the repair of abnormal radiator oil temperature.

[0124] Taking a 110kV voltage level transformer as the core, a multimodal knowledge graph is constructed at the component-fault-multimodal feature = mechanism semantic level.

[0125] Newly hired maintenance personnel wear MR headsets and hold force feedback handles to enter a virtual 110kV transformer scene, thus initializing the training scenario. The immersive display of the MR headset allows trainees to quickly immerse themselves in the maintenance scenario, while the grip of the force feedback handles allows trainees to get used to the operating tools in advance, reducing the sense of unfamiliarity in subsequent operations.

[0126] Pre-testing trainees is conducted to obtain their initial proficiency.

[0127] For example, the trainee scored 52 points in the theoretical test and only completed the positioning of 2 parts in the basic operation, for a total score of 52 points. According to the rules, when the predicted score is <60 points, the initial proficiency S=0.2.

[0128] Based on the trainee's initial proficiency, the eye movement trigger threshold of the trainee is calculated using a preset fault complexity optimization algorithm and a preset eye movement trigger threshold algorithm.

[0129] Based on the trainee's initial proficiency S=0.2, the initial fault is set as a single abnormal radiator oil temperature, with low feature coupling, allowing the trainee to start with simple faults and gradually build up troubleshooting knowledge.

[0130] The eye-tracking module captures the trainee's gaze behavior in real time. Real-time capture ensures timely response to the trainee's focus and avoids missing key interaction opportunities. When the trainee gazes at the radiator component of the virtual transformer and the gaze duration reaches the preset eye-tracking trigger threshold, the system automatically retrieves the associated data for "abnormal radiator oil temperature" from the multimodal knowledge graph. This triggering logic ensures that the trainee obtains the corresponding fault information only when actively focusing on a component, avoiding the passive information intake that leads to poor memory retention.

[0131] The cross-modal fault feature injection module loads and displays multimodal fault features.

[0132] For example, the temperature cloud map on the radiator surface can intuitively show the oil temperature distribution, helping trainees quickly locate the core area of ​​the fault; the diagram of the radiator blockage can clearly show the cause of the fault, allowing trainees to understand the physical root cause of the fault; the oil temperature change curve can show the fault development trend, enhancing the understanding of the fault's hazards. The combination of these three can provide trainees with comprehensive fault characteristic information from three dimensions: location, cause, and trend.

[0133] Meanwhile, the virtual-real interaction feedback module controls the vibration simulator to output a vibration frequency of 2Hz and an amplitude of 0.4mm based on a preset virtual-real feedback intensity algorithm. This allows trainees to perceive the fault state through touch, compensating for the lack of tactile sensation in the virtual scene and making fault perception more three-dimensional. The control force feedback handle provides 5N of operating resistance when trainees attempt to virtually disassemble the heat sink. This allows trainees to practice operating force in a safe virtual environment, avoiding damage to the equipment due to improper force in actual operation. At the same time, it also allows trainees to become familiar with the resistance feedback of disassembly actions, improving their operational proficiency.

[0134] The virtual-real interactive feedback module controls the voice interaction unit to synchronously play a fault mechanism analysis: "The current fault is abnormal radiator oil temperature. The fault mechanism is that dust has accumulated inside the heat sink for a long time, causing blockage of the heat dissipation channels. The heat generated by the transformer cannot be transferred to the air through the heat sink, and the oil temperature continues to rise. If not dealt with in time, the high temperature will accelerate the aging of the transformer insulation material and may cause serious faults such as inter-turn short circuits. The maintenance requires stopping the machine first and then cleaning the blockage in the heat sink." The synchronous playback allows trainees to observe visual features and feel physical feedback while simultaneously acquiring mechanistic knowledge, realizing multi-sensory collaborative learning. This makes fault cognition more comprehensive and memory more solid, and also helps trainees understand "why maintenance is done this way," rather than simply memorizing operating procedures.

[0135] Real-time evaluation of trainees' troubleshooting operations: This involves determining whether trainees have correctly performed the preliminary step of "stopping the virtual transformer," accurately located the blockage in the heat sink, cleaned the blockage according to standards, and performed the verification step of "testing the oil temperature upon startup" after the repair is completed. Real-time evaluation can promptly identify oversights in trainees' operations, prevent the formation of incorrect operating habits, and guide trainees to follow standard repair procedures, cultivate a sense of standardized operation, and ensure the safety and efficiency of subsequent actual repair operations.

[0136] Three types of data are recorded simultaneously: eye-tracking data, physical feedback response data, and operational result data. Eye-tracking data can analyze whether the trainee's focus is reasonable. If the proportion of fixation on irrelevant areas is too high, subsequent training can add guidance prompts to help the trainee focus on key areas. Physical feedback response data can assess the matching degree between physical feedback and the trainee's operation. If the feedback intensity is not well adapted to the trainee's operation, the feedback parameters can be optimized in subsequent training. Operational result data can quantify the trainee's training effect and identify the areas where skills need to be improved.

[0137] Based on the above three types of data, a personalized training plan is generated: the focus is on strengthening the training on the completeness of the troubleshooting steps for "abnormal radiator oil temperature," which can specifically make up for operational omissions; the eye-tracking trigger threshold is appropriately lowered, which can provide more support for trainees in the initial training; subsequent training gradually incorporates the coupling of humidity characteristics, which can progressively improve trainees' ability to handle complex faults, ensure that the training effect gradually improves, and avoid the waste of resources or insufficient effect caused by a one-size-fits-all training model.

[0138] This application provides a virtual training method for transformer fault repair based on multimodal collaboration. First, in the multimodal knowledge graph construction and collaborative training system setup phase, historical maintenance records of 110kV transformers, field data, and laboratory simulation samples are integrated. Multimodal data such as temperature and humidity are collected to construct a component-fault-feature-mechanism graph. Simultaneously, hardware such as an MR headset is integrated to build a system with three modules, including eye tracking, achieving precise configuration. Next, through scene initialization and multimodal feedback linkage, eye-tracking data is captured in real time. Fault characteristics are displayed differentially after reaching a threshold, and a virtual-real feedback intensity algorithm achieves precise linkage between physical feedback and voice analysis. Then, the correctness of the operation is evaluated, and eye-tracking, feedback, and operation data are recorded to generate personalized solutions. Finally, the entire process data is archived, providing support for standardized training of maintenance skills and traceability of training effectiveness. This method solves the problems of insufficient realism in simulated scenarios, lack of multi-source data fusion and application, high training risks due to fixed safety thresholds, reliance on manual experience for mastering maintenance skills, and lack of systematic quantitative evaluation of training effectiveness in traditional transformer fault repair training.

[0139] Figure 2 This is a schematic diagram illustrating the structure of a virtual training system for transformer fault repair based on multimodal collaboration, according to an exemplary embodiment.

[0140] Reference Figure 2 As shown in one embodiment of this application, a virtual training system 100 for transformer fault repair based on multimodal collaboration is provided, including: a multimodal knowledge graph construction module 110, a scene initialization module 120, an eye-tracking trigger threshold determination module 130, a virtual reality fault interaction module 140, and a personalized training program generation module 150.

[0141] The multimodal knowledge graph construction module 110 is used to collect real transformer fault sample data and construct a multimodal knowledge graph at the component-fault-multimodal feature-mechanism semantic level. The scenario initialization module 120 is used to pre-test trainees using a preset collaborative training system, determine the trainees' initial proficiency, and initialize the training scenario. The eye-tracking trigger threshold determination module 130 is used to determine the eye-tracking trigger threshold of the trainee based on the trainee's initial proficiency, using a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm. The virtual reality fault interaction module 140 is used to call fault association data in the multimodal knowledge graph based on the trainee's eye movement data and the trainee's eye movement trigger threshold, and to obtain fault interaction data between the trainee and the virtual transformer scene in the preset collaborative training system using a preset virtual-real feedback intensity algorithm. The personalized training program generation module 150 is used to generate personalized training programs based on the trainee's eye movement data and fault interaction data.

[0142] The above embodiments of this application describe a virtual training method for transformer fault repair based on multimodal collaboration. This method employs a pre-set collaborative training system to pre-test trainees, obtaining their initial proficiency (initial level) and initializing the training scenario. This effectively understands the current knowledge level of the trainees. Furthermore, it uses a pre-set fault complexity optimization algorithm and a pre-set eye-tracking trigger threshold algorithm to set appropriate initial fault complexity and eye-tracking trigger thresholds for each trainee. This provides personalized training starting points for trainees with different backgrounds, breaking the limitations of traditional standardized training and generating personalized training plans for each trainee. This approach ensures that training is conducted effectively and efficiently. The training difficulty is matched with the trainee's ability to avoid affecting the training effect due to inappropriate difficulty, thereby improving the relevance and efficiency of training and shortening the skill improvement cycle. By capturing the trainee's gaze behavior through eye movement data, when the trainee's eye movement trigger threshold is reached, the fault-related data corresponding to the training fault is called up. A preset virtual-real feedback intensity algorithm is used to realize virtual reality interaction between the trainee and the virtual transformer scene in the preset collaborative training system. The trainee can actively acquire fault-related knowledge in the immersive scene, avoiding the problems of knowledge fragmentation and lack of practical operation, and helping the trainee quickly build a complete fault diagnosis and repair knowledge framework.

[0143] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0144] Based on the same technical concept, in some specific embodiments of this application, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a method that the processor can use to execute when executing the program.

[0145] Based on the same technical concept, in some specific embodiments of this application, a computer-readable storage medium is provided on which a computer program is stored, which can be used to execute a method when the program is executed by a processor.

[0146] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0147] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0148] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0149] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] The specific embodiments of this application have been described above. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A virtual training method for transformer fault diagnosis and maintenance based on multimodal collaboration, characterized in that, include: Collect real transformer fault sample data and construct a multimodal knowledge graph with component-fault-multimodal features-mechanism semantic hierarchy; A pre-set collaborative training system is used to conduct pre-tests on trainees to determine their initial proficiency and initialize the training scenario; Based on the trainee's initial proficiency, a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm are used to determine the trainee's eye-tracking trigger threshold. Based on the trainee's eye movement data and the trainee's eye movement trigger threshold, the fault association data in the multimodal knowledge graph is called, and a preset virtual-real feedback intensity algorithm is used to obtain the fault interaction data between the trainee and the virtual transformer scenario in the preset collaborative training system. A personalized training plan is generated based on the trainee's eye movement data and the fault interaction data.

2. The virtual training method for transformer fault repair based on multimodal collaboration according to claim 1, characterized in that, The preset collaborative training system includes an MR headset, a vibration simulator, a force feedback handle, and a voice interaction unit. The MR headset includes an eye-tracking module, a cross-modal fault feature injection module, and a virtual-real interaction feedback module. The preset collaborative training system is used to create the virtual transformer scene and to interact with the trainee in virtual reality.

3. The virtual training method for transformer fault repair based on multimodal collaboration according to claim 1, characterized in that, The collected real transformer fault sample data is used to construct a multimodal knowledge graph at the component-fault-multimodal feature-mechanism semantic level, including: Collect real transformer fault sample data, the types of faulty components included in the real transformer fault sample data include radiators, breather pipes and bushings; The real fault sample data of the transformer is cleaned and labeled to determine the transformer type, transformer component type, fault type, multimodal characteristic parameters, maintenance steps and mechanism analysis information corresponding to each real fault sample data of the transformer. Based on the transformer type, transformer component type, fault type, multimodal characteristic parameters, maintenance steps, and mechanism analysis information corresponding to each actual transformer fault sample data, the core nodes of the multimodal knowledge graph are defined. The core nodes include transformer type nodes, component type nodes, fault type nodes, multimodal characteristic nodes, maintenance step nodes, and mechanism analysis nodes. Based on the core nodes of the multimodal knowledge graph, the relationship edges between the core nodes are defined to generate the multimodal knowledge graph. The relationship edges between the core nodes include the occurrence relationship between component type and fault type, the inclusion relationship between fault type and multimodal features, the correspondence between fault type and maintenance steps, and the association relationship between fault type and mechanism analysis.

4. The virtual training method for transformer fault diagnosis and maintenance based on multimodal collaboration according to claim 1, characterized in that, The process of using a pre-set collaborative training system to pre-test trainees, determine their initial proficiency, and initialize the training scenario includes: The trainees are pre-tested using pre-set pre-test questions to determine their pre-test scores. If the trainee's pretest score is less than a preset first score threshold, the trainee's initial proficiency is determined to be the first proficiency. If the trainee's pretest score is not less than the preset first score threshold and is less than the preset second score threshold, the trainee's initial proficiency is determined to be the second proficiency, where the preset first score threshold is less than the preset second score threshold. If the trainee's pretest score is greater than the preset second score threshold, the trainee's initial proficiency is determined to be the third proficiency, where the first proficiency is less than the second proficiency, and the second proficiency is less than the third proficiency.

5. The virtual training method for transformer fault diagnosis and maintenance based on multimodal collaboration according to claim 1, characterized in that, The step of determining the trainee's eye-tracking trigger threshold based on the trainee's initial proficiency using a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm includes: Based on the trainee's initial proficiency, a preset fault complexity optimization algorithm is used to determine the initial fault feature complexity corresponding to the trainee's initial proficiency. Based on the initial fault feature complexity corresponding to the trainee's initial proficiency, the preset eye-tracking trigger threshold algorithm is used to determine the trainee's eye-tracking trigger threshold.

6. The virtual training method for transformer fault repair based on multimodal collaboration according to claim 2, characterized in that, The method further includes: The eye-tracking module of the MR headset is used to collect the eye movement data of the trainee. The eye movement data includes the three-dimensional coordinates of the fixation point, the duration of fixation point dwell, and the eye movement trajectory. The eye movement data of the trainees were denoised and smoothed to determine the denoised and smoothed eye movement data. Feature extraction is performed on the denoised and smoothed eye movement data to determine the eye movement trajectory entropy value and the gaze region distribution characteristics, and the processed eye movement data is determined. The processed eye movement data includes the three-dimensional coordinates of the gaze point, the gaze point dwell time, the eye movement trajectory, the eye movement trajectory entropy value, and the gaze region distribution characteristics. The method further includes: The cross-modal fault feature injection module of the MR headset calls the fault association data in the multimodal knowledge graph and uses a rendering engine to generate visual features; The visualization features are spatially bound to the transformer components in the virtual transformer scene.

7. The virtual training method for transformer fault diagnosis and maintenance based on multimodal collaboration according to claim 6, characterized in that, The step of retrieving fault-related data from the multimodal knowledge graph based on the trainee's eye-tracking data and eye-tracking trigger threshold, and using a preset virtual-real feedback intensity algorithm to obtain fault interaction data between the trainee and the virtual transformer scenario in the preset collaborative training system, includes: If the duration of fixation in the trainee's eye movement data is greater than the trainee's eye movement trigger threshold, determine the virtual transformer component corresponding to the three-dimensional coordinates of the fixation point; Based on the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point, the visualization features corresponding to the virtual transformer component are displayed in the cross-modal fault feature injection module of the MR headset. Based on the trainee's initial proficiency and the visualization features corresponding to the virtual transformer component with the three-dimensional coordinates of the gaze point, the virtual-real feedback intensity is determined using the preset virtual-real feedback intensity algorithm. Based on the intensity of the virtual and real feedback, the vibration frequency and amplitude of the vibration simulator and the operating resistance of the force feedback handle are determined. Based on the vibration frequency and amplitude of the vibration simulator and the operating resistance of the force feedback handle, the virtual-real interaction feedback module controls the vibration simulator and the force feedback handle to output physical feedback response information. The virtual-real interaction feedback module controls the voice interaction unit to play the fault mechanism analysis of the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point; Collect fault operation data of the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point of the trainee; The step of generating a personalized training plan based on the trainee's eye-tracking data and the fault interaction data includes: The fault operation data of the virtual transformer component corresponding to the three-dimensional coordinates of the gaze point of the trainee are evaluated in real time to obtain the fault operation evaluation results of the trainee. The eye-tracking data of the trainee, the physical feedback response information output by the vibration simulator and the force feedback handle, and the fault operation evaluation results of the trainee are recorded simultaneously to generate a personalized training plan for the trainee.

8. A virtual training system for transformer fault diagnosis and repair based on multimodal collaboration, characterized in that, include: The multimodal knowledge graph construction module is used to collect real transformer fault sample data and construct a multimodal knowledge graph with a semantic hierarchy of components, faults, multimodal features, and mechanisms. The scenario initialization module is used to pre-test the trainees using a preset collaborative training system, determine the trainees' initial proficiency, and initialize the training scenario. An eye-tracking trigger threshold determination module is used to determine the eye-tracking trigger threshold of the trainee based on the trainee's initial proficiency, using a preset fault complexity optimization algorithm and a preset eye-tracking trigger threshold algorithm. The virtual reality fault interaction module is used to call fault association data in the multimodal knowledge graph based on the trainee's eye movement data and the trainee's eye movement trigger threshold, and to obtain fault interaction data between the trainee and the virtual transformer scene in the preset collaborative training system using a preset virtual-real feedback intensity algorithm. The personalized training program generation module is used to generate a personalized training program based on the trainee's eye movement data and the fault interaction data.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images