Electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graph

Through a system combining artificial intelligence and knowledge graph, automatic perception and digital recording of electric train equipment maintenance training and evaluation are realized, the problems of lack of resources and strong subjectivity are solved, and the evaluation efficiency and learning experience are improved.

CN114139968BActive Publication Date: 2024-11-05SICHUAN QIENTHALPY TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111462135.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-03
Publication Date
2024-11-05
Estimated Expiration
2041-12-03

AI Technical Summary

Technical Problem

The existing electric train equipment maintenance training and evaluation system has problems such as lack of resources, strong subjectivity of teachers, and inability to effectively simulate and record digitally, and it is difficult to integrate offline and online teaching.

Method used

The system based on artificial intelligence, knowledge graph and three-dimensional simulation technology is adopted to collect video data through induction point cameras, the GPU server processes intelligent model identification results, the database stores knowledge graph and rule data, and the application server performs ratings and three-dimensional reconstruction displays.

Benefits of technology

It realizes automatic perception, objective scoring and digital recording of the maintenance and training process, improves the evaluation efficiency and students' learning experience, and solves the problem of insufficient resource allocation and subjectivity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114139968B_ABST
    Figure CN114139968B_ABST
Patent Text Reader

Abstract

The present invention discloses an electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graph, which is used to solve the problem of lack of examination resources in the current electric train equipment maintenance training and evaluation process. The system consists of: a sensing point camera; a GPU server cluster; a database server cluster, an application server cluster; and a large-screen display device. The core functional units of the present invention include: an intelligent model unit, an electric train maintenance knowledge graph unit, a maintenance rule engine unit, and a three-dimensional simulation unit. When the present invention is implemented, it also includes an electric train equipment training and maintenance warehouse. The present invention solves the problem that the scoring is not objective enough due to the subjectivity of the instructor in the evaluation process and the lack of evaluation experience, solves the problem that the offline maintenance training test and the online teaching process cannot be organically integrated, and solves the problem that the maintenance training test process cannot be effectively digitally simulated and recorded.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field related to information-based practical training and assessment, and specifically to an electric train equipment maintenance practical training and assessment system based on artificial intelligence and knowledge graph. Background Art

[0002] With the rapid development of my country's rail transit industry, the railway departments and rail transit operating units have an increasing demand for high-skilled maintenance personnel for related vehicle equipment. At present, most electric train equipment maintenance training and assessments are conducted by instructors on-site at the training teaching base to supervise and score the trainees' maintenance test process. However, due to the shortage of test equipment, test venues, senior technicians and other test resources, as well as the subjectivity of instructors in the assessment process and lack of assessment experience, the configuration of test resources and the assessment results are difficult to meet the test needs of trainees.

[0003] Although some online training teaching platforms based on Internet technology can meet some theoretical teaching and examination needs, they are unable to organically integrate offline maintenance training examinations with online teaching processes, nor are they able to effectively digitally simulate and record the maintenance training examination process. Summary of the invention

[0004] Therefore, in order to solve the above-mentioned deficiencies, the present invention provides a system for automatically sensing, evaluating, three-dimensionally reconstructing and recording the maintenance training evaluation process by utilizing technologies such as artificial intelligence, knowledge graphs, rule engines and three-dimensional simulation, so as to solve the problem of shortage of examination resources in the current electric train equipment maintenance training evaluation process, solve the problem of objective scoring due to the subjectivity of instructors in the evaluation process and lack of evaluation experience, solve the problem that offline maintenance training examinations and online teaching processes cannot be organically integrated, and solve the problem that the maintenance training examination process cannot be effectively digitally simulated and recorded.

[0005] The present invention is implemented by constructing an electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graph, which is characterized in that the system comprises:

[0006] Sensing point cameras: These cameras include head-mounted wireless cameras, sensing point cameras in the top area of ​​the training and maintenance warehouse, sensing point cameras in the operating table area of ​​the training and maintenance warehouse, and sensing point cameras on the tool table. These cameras are responsible for collecting videos of the trainees' maintenance and training operation process and sending the collected video data to the GPU server cluster;

[0007] GPU server cluster: The GPU server cluster includes multiple GPU servers, each of which is equipped with multiple GPU graphics cards. The GPU server cluster is equipped with an intelligent model unit. Each GPU server is responsible for processing the corresponding video data and sending the results of intelligent model recognition to the application server.

[0008] Database server cluster: The database server cluster includes multiple application servers, in which an electric train maintenance knowledge graph unit is deployed and installed; each application server is responsible for storing train maintenance knowledge graph data, train maintenance rule data and application data, respectively.

[0009] Application server cluster: The application server cluster deploys and installs the maintenance rule engine unit. The application server cluster contains multiple application servers. Each application server deploys the maintenance rule engine service, maintenance rule management service, basic parameter management service, sensing point management service, 3D model management service and 3D simulation visualization service. All application services are deployed, managed and scheduled in the form of microservices. In particular, the maintenance rule engine service is responsible for receiving the recognition results sent by the intelligent model service, and obtaining application data and graph data from the database server. Then, the maintenance training operations of the trainees are scored according to the maintenance rules. At the same time, the results recognized by the intelligent model are reconstructed in 3D and displayed on the visualization screen.

[0010] Large-screen display device: This large-screen display device is responsible for displaying the 3D model and operation animation of the application server after reconstruction, which is convenient for instructors and other students to watch and learn.

[0011] According to the electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graphs of the present invention, it is characterized in that the core functional units in the system include: an intelligent model unit, an electric train maintenance knowledge graph unit, a maintenance rule engine unit and a three-dimensional simulation unit.

[0012] The first functional unit is the intelligent model unit, which includes four intelligent models, namely, a tool detection and recognition model, a tool instance segmentation model, an instrument reading recognition model and a maintenance action recognition model. These intelligent models are responsible for intelligently identifying the video stream data collected by the camera, and after the recognition is completed, converting the recognition results of the tools and maintenance actions into corresponding tool instructions and maintenance action instructions respectively, and then sending the above-mentioned tool instructions, maintenance action instructions and instrument readings to the maintenance rule engine.

[0013] As mentioned above, the four intelligent models of the first functional unit include:

[0014] Intelligent model 1 is a tool detection and recognition model, which is used to detect and recognize train equipment, equipment parts, maintenance tools, instruments, human bodies and hands in images.

[0015] The tool detection and recognition model is iteratively trained on the labeled tool dataset through the target detection and recognition algorithm. The target detection and recognition algorithm uses a single-stage fully convolutional detection network based on pixel-level prediction. Its network structure includes a backbone network, a feature pyramid network and a detector. The detector contains three branches, namely, the classification prediction branch, the center prediction branch and the regression prediction branch. The working process of the algorithm is divided into the following steps:

[0016] Step 1.1: Perform preprocessing operations on the input image, such as converting RGB to grayscale, resizing, and converting image data to tensor.

[0017] Step 1.2: Send the tensor data obtained in step 1.1 to the backbone network to obtain feature maps of five different scales of the input data.

[0018] Step 1.3: The feature map obtained in step 1.2 is sent to the next step, the Feature Pyramid Network (FPN), and a pre-defined anchor box (AnchorBoxes) is used for regression prediction on each pixel point of the feature map.

[0019] Step 1.4: Calculate the center-ness of each predicted bounding box obtained in step 1.3 through binary cross entropy loss (BCE), and calculate the confidence of the predicted bounding box by multiplying the center-ness by the corresponding classification score.

[0020] Step 1.5: Finally, filter out the low-quality bounding box obtained in step 1.4 through non-maximum suppression (NMS), and sort by confidence to obtain the final bounding box and classification category of the target object.

[0021] Intelligent model two is the tool instance segmentation model, which is used to perform instance segmentation of train equipment, equipment parts, maintenance tools, instruments, human bodies and hands in the image. The results of tool instance segmentation can be used to calculate the intersection-and-union ratio between tools. The maintenance rule engine needs to combine the recognition results of the maintenance action and the intersection-and-union ratio between tools to determine whether the maintenance action is compliant.

[0022] The tool instance segmentation model is iteratively trained on the labeled tool dataset through the instance segmentation algorithm. The instance segmentation algorithm consists of a detection module and a hybrid mask module; the detection module is used to predict the bounding box coordinates and classification category of the instance object, and the hybrid mask module consists of three parts: a bottom module for generating a mask, a top module for generating top-level attention, and a hybrid module that combines the mask and attention. The specific functions of each module are as follows:

[0023] Detection module: The detection module has the same network structure and working process as the intelligent model 1. It is mainly responsible for predicting the bounding box and classification category of the instance object in the input image, and providing feature input for the bottom module and the top module.

[0024] Top module: The bounding box proposal of each instance object is obtained through the detection module, and then a convolution operation is performed to obtain the top-level attention.

[0025] Bottom module: The bottom module takes the feature maps of the backbone network and feature pyramid in the detection module as input, and obtains K masks of the entire image through operations such as up sampling, concatenation and convolution.

[0026] Hybrid module: The hybrid module takes the instance candidate bounding box, the top-level attention, and K masks of the entire image as input, crops, scales, and aligns the K masks respectively, interpolates the top-level attention, and obtains a set of score maps after Softmax normalization. Finally, the score map is multiplied by the processed K masks and added by channel to obtain the final mask.

[0027] Intelligent model three is the instrument reading recognition model. This model uses the results of the tool detection and recognition model as input, cuts out the area where the instrument is located from the image, and recognizes the measurement data displayed on the instrument.

[0028] The instrument reading recognition model is iteratively trained on the labeled instrument reading data set and optical character data set by the optical character recognition algorithm. The optical character recognition algorithm uses a multi-task parallel full convolution point aggregation network to support real-time detection and recognition of text of any shape. The working process of the algorithm is divided into the following steps:

[0029] Step 2.1: Rectify the image using corner detection and Hough transform.

[0030] Step 2.2: Send the image obtained in step 2.1 to the backbone network (Backbone Network) with Feature Pyramid Network (FPN) to obtain 1 / 4 down-sampled feature maps (Feature Maps).

[0031] Step 2.3: Use a fully convolutional network to perform multi-task parallel learning on the feature map obtained in step 2.2 to obtain feature information of four modes of the text area, including text centerline, text edge offset, text direction offset and text character classification map.

[0032] Step 2.4: Use the text centerline and text direction offset obtained in step 2.3 to extract the text center point sequence according to the reading order of the text, and then calculate the text boundary box through the text edge offset.

[0033] Step 2.5: Use the text centerline and text direction offset obtained in step 2.3 to upsample the text character classification map. After splicing the sampling points into a sequence of fixed length, use the Connectionist Temporal Classification (CTC) loss function for training to obtain the character classification probability sequence, and decode it into the final text recognition result.

[0034] Intelligent model 4 is a maintenance action recognition model, which is used to identify the maintenance actions of trainees during maintenance training operations.

[0035] The maintenance action recognition model is iteratively trained on the labeled maintenance action dataset through the human action recognition algorithm. The human action recognition algorithm uses a two-stream adaptive graph convolutional neural network, which includes an adaptive convolution layer, an adaptive graph convolution block, an adaptive graph convolutional network and a two-stream network. The working process of the algorithm is divided into the following steps:

[0036] Step 3.1: Use the pre-trained human pose estimation model to extract human skeleton key point data from the image.

[0037] Step 3.2: Extract joint data and skeleton data from the human skeleton key point data obtained in step 3.1. The joint data includes the coordinates of the source joint point and the coordinates of the target joint point. The skeleton data is represented by a vector pointing from the source joint point to the target joint point.

[0038] Step 3.3: Send the joint data and skeleton data obtained in step 3.2 to the joint data stream processing network (Joint Stream Network) and the skeleton data stream processing network (Bones Stream Network), respectively. The joint data stream processing network and the skeleton data stream processing network use distance-based sampling functions to sample the joint data and skeleton data respectively, and then use the spatial temporal graph convolutional network (Spatial Temporal Graph Convolutional Networks, ST-GCN) to adaptively construct the joint samples and skeleton samples into different topological structure graphs, and then send the topological structure graphs of the two types of samples to their respective Softmax classifiers to obtain the corresponding action prediction scores.

[0039] Step 3.4: Add the motion prediction scores of the joint samples and bone samples obtained in step 3.3 to obtain the final motion prediction label.

[0040] According to the electric train equipment maintenance training and assessment system based on artificial intelligence and knowledge graphs of the present invention, it is characterized by: the second functional unit is the train maintenance knowledge graph unit, which is based on the electric train maintenance operation instructions, electric train quality inspection manuals and electric train maintenance worker teaching materials, and extracts the unstructured knowledge content in the original bibliography into graphed data through document layout analysis technology, text detection and recognition technology, text parsing technology and knowledge graph construction technology, providing a knowledge basis for the maintenance rule engine.

[0041] As described below, the train maintenance knowledge graph will be further explained from two aspects:

[0042] First, the construction process of the train maintenance knowledge graph. The construction process is divided into the following steps:

[0043] Step 4.1: First, extract the knowledge content from the electric train maintenance operation instruction manual and the electric train quality inspection manual. Since these two books are text versions of PDF, the contents in the bibliography can be read directly, and the chapters, paragraphs and table data in the book are parsed through matching rules, and the text data is converted into a tree structure according to the original knowledge structure in the bibliography; then extract the knowledge content from the electric train maintenance textbook. Since this series of textbooks are scanned PDFs, the contents in the bibliography cannot be read directly, so the headers, footers, illustrations, tables and texts in the textbooks are identified through document layout analysis technology and text recognition technology, and then the text data is converted into tree structure data with the same knowledge structure as the original bibliography through matching rules.

[0044] Step 4.2: Extract relevant domain vocabulary and vocabulary classification from various publicly released rail transit professional dictionaries to obtain the rail transit domain vocabulary library, extract text data from various standardized documents, maintenance worker teaching materials, operation instructions, maintenance manuals and other materials in the rail transit field, and then obtain the pre-selected vocabulary library by calculating the left and right entropy and solidification degree based on the seed words, and then obtain the custom domain vocabulary library through custom filtering rules and some manual screening. Finally, the rail transit domain vocabulary library and the custom domain vocabulary library are merged and deduplicated to obtain the keyword vocabulary library required to construct the train maintenance knowledge graph, and on this basis, the categories of the vocabulary in the keyword vocabulary library are marked.

[0045] Step 4.3: Use the multi-modal string matching algorithm to load the keyword thesaurus obtained in step 4.2, then perform keyword matching on the tree structure data obtained in step 4.1, mark the matched keywords and add corresponding category labels to obtain the corpus of the named entity recognition model, and then train the corpus through a bidirectional long short-term memory neural network to finally obtain the entity recognition model in the rail transit field.

[0046] Step 4.4: Use the rail transit domain entity recognition model obtained in step 4.3 to perform named entity recognition on the tree structure data obtained in step 4.1, and then perform text parsing through domain entity extraction rules to extract rail transit domain entities.

[0047] Step 4.5: Use dependency syntax analysis to analyze the grammatical relationship between rail transit domain entities in the tree structure data obtained in step 4.1, and then extract the relationship through the predefined entity relationship template and domain entity relationship extraction rules to obtain the subordinate relationship between domain entities.

[0048] Step 4.6: After obtaining the subordinate relationship between domain entities through steps 4.4 and 4.5, extract the attributes of domain entities from the tree structure data through named entity recognition, part-of-speech analysis, dependency syntax analysis and attribute extraction rules.

[0049] Step 4.7: The domain entities, entity relationships, and entity attributes extracted in steps 4.4, 4.5, and 4.6 are respectively constructed into a triple structure, and the data of the triple structure is imported into the graph database to obtain a train maintenance knowledge graph.

[0050] Step 4.8: Load the train maintenance knowledge graph obtained in step 4.7 into the visualization interface, manually review the train maintenance knowledge graph, and manually correct the incorrectly extracted domain entities, entity relationships, and entity attributes.

[0051] Secondly, the application interface of the electric train maintenance knowledge graph. This application interface is based on graph computing algorithms such as depth-first traversal, breadth-first traversal, relationship retrieval, path extraction, strongly connected components, shortest path analysis, relevance analysis, in-and-out degree analysis, and Bayesian reasoning, and realizes various retrieval and computing functions of the maintenance rule engine and application layer services on the train maintenance knowledge graph, and provides functional support for the maintenance rule engine and other application layer services in the form of an application interface.

[0052] According to the electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graph of the present invention, it is characterized by: the third functional unit is the maintenance rule engine unit, which is based on the visual process engine and provides the instructor with a set of visual configuration interface of the training and examination rule template and the calling interface of the train maintenance knowledge graph. When configuring the training and examination rule template, the instructor can dynamically select the required train equipment, equipment parts, maintenance tasks, maintenance tools, maintenance methods and technical requirements and other knowledge nodes from the train maintenance knowledge graph. After the configuration of the training and examination rule template is completed, in the actual maintenance training test, the trainees will pass the interactive interface in the electric train equipment maintenance training and evaluation system. After selecting the train model, train equipment and maintenance level to be inspected, the maintenance rule engine will generate the practical training test content, maintenance task timing rules, tool use compliance judgment rules, maintenance action compliance judgment rules and scoring rules according to the practical training test rule template configured by the instructor and the built-in graph calculation algorithm. Students complete various maintenance tasks according to the specified test content. The maintenance rule engine uses the above rules to judge whether the timing relationship of the completion of maintenance tasks during the student's maintenance process is reasonable, whether the tools used are compliant, and whether the maintenance actions are compliant. Finally, the scoring rules are used to comprehensively score the student's entire maintenance process and give the test results. In addition, during the student's maintenance training operation, the maintenance rule engine also needs to forward the tool instructions, maintenance action instructions and instrument readings sent by the intelligent model unit to the three-dimensional simulation unit in real time.

[0053] According to the electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graph of the present invention, it is characterized by: the fourth functional unit is the three-dimensional simulation unit, which first constructs the maintenance site 3D model, train equipment 3D model, tool 3D model, human body 3D model and hand 3D model through the three-dimensional modeling tool. Then, after receiving the tool instructions, maintenance action instructions and instrument readings sent by the maintenance rule engine, the 3D animation engine is responsible for real-time loading and displaying the corresponding 3D model animation, thereby realizing the virtual simulation of the maintenance training test process.

[0054] like Figure 3As shown, the overall process of the electric train equipment maintenance training and evaluation system is divided into the following steps:

[0055] Step 5.1: The trainee selects and submits the train model, train equipment and maintenance level for this maintenance training test on the interactive terminal at the maintenance training site. The interactive terminal is responsible for sending this information to the maintenance rule engine, which then forwards it to the 3D animation engine. The 3D animation engine loads and displays the 3D models of the corresponding maintenance site and train equipment, and prompts the start of the test.

[0056] Step 5.2: After receiving the data sent by the interactive terminal, the maintenance rule engine is responsible for obtaining the maintenance task list of this practical training test from the train maintenance knowledge graph, and loading the maintenance task timing rules of the current train equipment. Then, according to the sequence of maintenance tasks, it obtains the tools, maintenance methods and technical requirements required for each maintenance task from the train maintenance knowledge graph in sequence, and loads the tool use compliance judgment rules, maintenance action compliance judgment rules and scoring rules.

[0057] Step 5.3: The streaming media service is responsible for collecting multiple video streams of the trainees' maintenance training test process, extracting key frames from the video streams, converting the key frames into images, preprocessing the images and converting them into Base64 encoding, and then sending the Base64 encoding to the intelligent model service.

[0058] Step 5.4: The intelligent model service is responsible for identifying and segmenting the tools in the image, identifying the readings of the instruments in the image, and identifying the maintenance actions in the continuous video frames. It then converts the identified tools and maintenance actions into corresponding tool instructions and maintenance action instructions, and finally sends the tool instructions, maintenance action instructions and instrument readings to the maintenance rule engine.

[0059] Step 5.5: The maintenance rule engine first forwards the tool instructions, maintenance action instructions and instrument readings to the 3D animation engine, which then loads and displays the maintenance scene, train equipment, equipment parts and the 3D models and operation animations of the received tools and maintenance actions in real time.

[0060] Step 5.6: The maintenance rule engine judges whether the tools used in the maintenance process are compliant, whether the maintenance actions are compliant, and whether the maintenance operation sequence is compliant based on the maintenance task timing rules, tool usage compliance judgment rules, and maintenance action compliance judgment rules, and then scores the completion of the maintenance task based on the scoring rules and the above judgment results.

[0061] Step 5.7: The maintenance rule engine is responsible for sending the scoring results of the maintenance tasks to the 3D animation engine, and the 3D animation engine displays the scores of the maintenance tasks in real time.

[0062] According to the electric train equipment maintenance training and assessment system based on artificial intelligence and knowledge graph of the present invention, it is characterized in that: the system also includes an electric train equipment training and maintenance warehouse, which mainly includes the following parts: the first part is a tool bench in the training and maintenance warehouse, the table top of which is a scale plate, and a tool bench induction point camera is installed directly above it. The tool bench is used to place the tools selected by the trainees and provide a scale reference for the identification of tool specifications; the video data collected by the tool bench induction point camera is mainly used for the detection, identification and specification judgment of tools.

[0063] The second part is the operating table of the training maintenance warehouse. The operating table adopts a grid layout, and provides an installation position for a sensing point camera at each grid intersection. Each installation position embeds the camera inside the operating table. This does not affect the trainees' operation, and it is convenient for the instructor to configure and switch the sensing point camera according to the type of equipment to be repaired, so that each sensing point camera that is turned on can shoot from the best angle. The video data collected by the sensing point camera of the operating table is mainly used for the detection and identification of tools and tools, as well as the identification of trainees' maintenance actions.

[0064] The third part is the top area of ​​the training maintenance warehouse. The top area also adopts a grid layout. A sensing point camera is installed at each grid intersection. The top sensing point camera also supports the instructor to configure and switch the top sensing point camera according to the specific requirements of the maintenance task. The video data collected by the top sensing point camera is mainly used for the recognition of trainees' maintenance actions.

[0065] The fourth part is the trainee's head-mounted wireless camera, which can take close-up photos of equipment parts, instruments, maintenance tools, and trainees' hand operations, solving the problem that the operating table sensor point camera and the top sensor point camera cannot take high-quality photos of equipment parts, instruments, maintenance tools, and trainees' hand operations. The video data collected by the head-mounted wireless camera is mainly used for the detection and identification of tools and the identification of trainees' maintenance actions.

[0066] The present invention has the following advantages:

[0067] (1) The present invention is based on an intelligent model unit, which realizes the detection, identification and segmentation of tools and instruments, the identification of instrument readings, and the identification of maintenance actions during the maintenance training process. The results are provided as input to the maintenance rule engine for compliance judgment of tool use, compliance judgment of maintenance actions, compliance judgment of maintenance operation sequence, and virtual simulation of maintenance operation process. As the core module of the electric train equipment maintenance training and evaluation system, the intelligent model unit provides a data basis for automated maintenance training evaluation and virtual simulation, so that the maintenance training operation process can be automatically perceived and digitally reconstructed in real time.

[0068] (2) The present invention is based on an electric train maintenance knowledge graph unit, which realizes the automatic extraction of electric train maintenance work instructions, electric train quality inspection manuals and electric train maintenance worker teaching materials, as well as the graphing of knowledge content. Based on the electric train maintenance knowledge graph, by providing external functional encapsulation based on graph computing algorithms, various retrieval and analysis requirements for electric train maintenance knowledge are realized, providing a knowledge basis for automated maintenance training and evaluation.

[0069] (3) The present invention is based on a maintenance rule engine unit, which realizes the dynamic extraction of knowledge nodes in the electric train maintenance map, the visual construction of tool use compliance judgment rules, the visual construction of maintenance action compliance judgment rules, and the visual construction of scoring rules. The instructor sets various fault points on the electric train equipment and constructs various judgment rules and scoring rules on each maintenance task, thereby forming a set of practical training test assignments. During the actual operation of the trainees, the maintenance rule engine unit is responsible for extracting the required knowledge content from the electric train maintenance knowledge map based on the results of the intelligent model unit recognition, and automatically scoring the trainees' operation process through the built-in judgment rules and scoring rules, and then sends the obtained tool instructions, maintenance action instructions and scoring results to the three-dimensional simulation unit.

[0070] (4) The present invention is based on a three-dimensional simulation unit, which realizes the 3D reconstruction of the training site, train equipment, equipment parts, maintenance tools, instruments and meters, and the human body, as well as the 3D animation simulation of the maintenance action. The three-dimensional simulation unit, as a visualization module of the electric train equipment maintenance teaching and evaluation system, provides students and teachers with an intuitive teaching scene and a good learning experience, and provides rich materials for the implementation of the teaching process.

[0071] (5) The present invention proposes an electric train equipment training and maintenance cabin, in which the top area and the operating table area of ​​the cabin are equipped with sensing point cameras in a grid layout, and the instructor can configure and switch these sensing point cameras according to the specific requirements of the maintenance task, which not only saves operating resources but also allows the trainees to shoot the operation process from the best viewing angle; in addition, the trainees' head-mounted wireless camera can further enhance the quality of video data acquisition by shooting the operation process at close range, thus providing a good data basis for the accurate recognition of the intelligent model. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The technical architecture diagram of the electric train equipment maintenance training and evaluation system provided for this application;

[0073] Figure 2 Functional unit diagram of the electric train equipment maintenance training and evaluation system provided for this application;

[0074] Figure 3 The overall flow chart of the electric train equipment maintenance training and evaluation system provided for this application;

[0075] Figure 4 A schematic diagram of the electric train maintenance knowledge graph construction process provided for this application;

[0076] Figure 5 A schematic diagram of the structure of the electric train maintenance knowledge graph provided for this application;

[0077] Figure 6 The hardware deployment architecture diagram of the electric train equipment maintenance training and evaluation system provided for this application;

[0078] Figure 7 A schematic diagram of the electric train equipment training and maintenance warehouse provided for this application;

[0079] Figure 8 A schematic diagram of the arrester maintenance rules engine provided for this application. DETAILED DESCRIPTION

[0080] The following will be combined with the attached Figure 1-Figure 8 The present invention is described in detail, and the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] The present invention provides, through improvement, a system for automatically perceiving, evaluating, three-dimensionally reconstructing and recording the maintenance training evaluation process by using technologies such as artificial intelligence, knowledge graph, rule engine and three-dimensional simulation. The technical solution of the present invention will be further explained below through specific implementation methods in combination with the accompanying drawings.

[0082] See also Figure 1 , the technical architecture of the electric train maintenance training and evaluation system of the present invention will be described. The overall technical architecture of the system can be divided into three layers, including a basic layer, a model layer and an application layer.

[0083] (1) Basic layer: The basic layer mainly includes the data environment and the operating environment. The data environment mainly includes the train maintenance knowledge graph, the train maintenance rule base and the application database. The operating environment mainly includes the software operating environment, the deep learning framework operating environment and the graphics and image processing library operating environment. The basic layer provides software and data support for the deployment and operation of the entire electric train equipment maintenance training and evaluation system.

[0084] (2) Model layer: The model layer provides algorithm and model support for the electric train equipment maintenance training and evaluation system. The core algorithms used include target detection and recognition algorithm, image instance segmentation algorithm, human posture estimation algorithm, human behavior recognition algorithm and optical character recognition algorithm. On the basis of these algorithms, the tool detection and recognition model, tool instance segmentation model, maintenance action recognition model and instrument reading recognition model are obtained by training the labeled tool data set, maintenance action data set and instrument reading data set respectively. These intelligent models realize the automatic perception and intelligent recognition of the trainees' maintenance training operation process, and provide a data basis for various evaluation and scoring rules of the maintenance rule engine.

[0085] (3) Application layer: The application layer provides the electric train equipment maintenance training and evaluation system with the configuration and management of various maintenance rules and scoring rules, the management of 3D models, the visualization of 3D models, the management of video acquisition equipment, the configuration and management of sensing points, the setting and management of fault points, the parameter management of tools and maintenance actions, as well as the deployment, operation and management of streaming media services, intelligent model services and maintenance rule engine services.

[0086] See also Figure 2 , the functional units of the electric train maintenance training and evaluation system of the present invention will be described, and its core functional units include: an intelligent model unit, an electric train maintenance knowledge graph unit, a maintenance rule engine unit and a three-dimensional simulation unit.

[0087] The first functional unit is the intelligent model unit, which includes four intelligent models, namely, a tool detection and recognition model, a tool instance segmentation model, an instrument reading recognition model and a maintenance action recognition model. These intelligent models are responsible for intelligently identifying the video stream data collected by the camera, and after the recognition is completed, converting the recognition results of the tools and maintenance actions into corresponding tool instructions and maintenance action instructions respectively, and then sending the above-mentioned tool instructions, maintenance action instructions and instrument readings to the maintenance rule engine.

[0088] As mentioned above, the four intelligent models of the first functional unit include:

[0089] Intelligent model 1 is a tool detection and recognition model, which is used to detect and recognize train equipment, equipment parts, maintenance tools, instruments, human bodies and hands in images.

[0090] The tool list and its corresponding instruction codes and classification categories are as described in the following table:

[0091] Tool name Instruction encoding category Tool name Instruction encoding category lightning arrester EQ00100 1 Paint Pen TL00500 15 Lightning arrester base EQ00101 2 Alcohol barrel TL00600 16 Connection line PT00100 3 White cloth TL00700 17 fastener PT00200 4 screwdriver TL00800 18 Arrester DC parameter tester IT00100 5 Vernier caliper TL00900 19 Lead clip IT00101 6 pliers TL01000 20 Power switch IT00102 7 Sleeve TL01100 21 Detection button IT00103 8 Conductive paste TL01200 22 Reset button IT00104 9 Digital Multimeter IT00200 23 Reading display IT00105 10 Multimeter Probes IT00201 24 Open-end wrench TL00100 11 Feeler gauge TL01300 25 Socket wrench TL00200 12 Nuts PT00300 26 Torque wrench TL00300 13 Dynamometer TL01500 27 Inside Micrometer TL00400 14 hand BT00100 28 ……… ……… ……… ……… ……… ………

[0092] Coding rules for tool list: (1) The first to second digits represent the first-level classification of tools. The first-level classification is as follows: EQ: train equipment, PT: maintenance parts, IT: testing instruments, TL: maintenance tools, BT: human body parts. (2) The third to fifth digits represent the unique code of a tool under the first-level classification. (3) The sixth to seventh digits represent the unique code of the accessory parts of a tool under the first-level classification. The above coding rules can be used to uniquely represent train equipment, equipment parts, maintenance parts, testing instruments, instrument accessories, maintenance tools and human body parts.

[0093] The tool detection and recognition model is iteratively trained on the labeled tool dataset through the target detection and recognition algorithm. The target detection and recognition algorithm uses a single-stage fully convolutional detection network based on pixel-level prediction. Its network structure includes a backbone network, a feature pyramid network, and a detector. The detector contains three branches, namely, the classification prediction branch, the center prediction branch, and the regression prediction branch. The execution process is divided into the following steps:

[0094] Step 1.1: Perform preprocessing operations on the input image, such as converting RGB to grayscale, resizing, and converting image data to tensor.

[0095] Step 1.2: Send the tensor data obtained in step 1.1 to the backbone network to obtain feature maps of five different scales of the input data.

[0096] Step 1.3: The feature map obtained in step 1.2 is sent to the next step, the Feature Pyramid Network (FPN), and a pre-defined anchor box (AnchorBoxes) is used for regression prediction on each pixel point of the feature map.

[0097] Step 1.4: Calculate the center-ness of each predicted bounding box obtained in step 1.3 through binary cross entropy loss (BCE), and calculate the confidence of the predicted bounding box by multiplying the center-ness by the corresponding classification score.

[0098] Step 1.5: Finally, filter out the low-quality bounding box obtained in step 1.4 through non-maximum suppression (NMS), and sort by confidence to obtain the final bounding box and classification category of the target object.

[0099] Intelligent model two is the tool instance segmentation model, which is used to perform instance segmentation of train equipment, equipment parts, maintenance tools, instruments, human bodies and hands in the image. The results of tool instance segmentation can be used to calculate the intersection-and-union ratio between tools. The maintenance rule engine needs to combine the recognition results of the maintenance action and the intersection-and-union ratio between tools to determine whether the maintenance action is compliant.

[0100] The tool instance segmentation model is iteratively trained on the labeled tool dataset through the instance segmentation algorithm. The instance segmentation algorithm consists of a detection module and a hybrid mask module; the detection module is used to predict the bounding box coordinates and classification category of the instance object, and the hybrid mask module consists of three parts: a bottom module for generating a mask, a top module for generating top-level attention, and a hybrid module that combines the mask and attention. The specific functions of each module are as follows:

[0101] Detection module: The detection module has the same network structure and working process as the intelligent model 1. It is mainly responsible for predicting the bounding box and classification category of the instance object in the input image, and providing feature input for the bottom module and the top module.

[0102] Top module: The bounding box proposal of each instance object is obtained through the detection module, and then a convolution operation is performed to obtain the top-level attention.

[0103] Bottom module: The bottom module takes the feature maps of the backbone network and feature pyramid in the detection module as input, and obtains K masks of the entire image through operations such as up sampling, concatenation and convolution.

[0104] Hybrid module: The hybrid module takes the instance candidate bounding box, the top-level attention, and K masks of the entire image as input, crops, scales, and aligns the K masks respectively, interpolates the top-level attention, and obtains a set of score maps after Softmax normalization. Finally, the score map is multiplied by the processed K masks and added by channel to obtain the final mask.

[0105] Intelligent model three is the instrument reading recognition model. This intelligent model uses the results of the tool detection and recognition model as input, cuts out the area where the instrument is located from the image, and recognizes the measurement data displayed on the instrument.

[0106] The instrument reading recognition model is iteratively trained on the labeled instrument reading data set and optical character data set by the optical character recognition algorithm. The optical character recognition algorithm uses a multi-task parallel full convolution point aggregation network to support real-time detection and recognition of text of any shape. The corresponding working process is divided into the following steps:

[0107] Step 2.1: Rectify the image using corner detection and Hough transform.

[0108] Step 2.2: Send the image obtained in step 2.1 to the backbone network (Backbone Network) with Feature Pyramid Network (FPN) to obtain 1 / 4 down-sampled feature maps (Feature Maps).

[0109] Step 2.3: Use a fully convolutional network to perform multi-task parallel learning on the feature map obtained in step 2.2 to obtain feature information of four modes of the text area, including text centerline, text edge offset, text direction offset and text character classification map.

[0110] Step 2.4: Use the text centerline and text direction offset obtained in step 2.3 to extract the text center point sequence according to the reading order of the text, and then calculate the text boundary box through the text edge offset.

[0111] Step 2.5: Use the text centerline and text direction offset obtained in step 2.3 to upsample the text character classification map. After splicing the sampling points into a sequence of fixed length, use the Connectionist Temporal Classification (CTC) loss function for training to obtain the character classification probability sequence, and decode it into the final text recognition result.

[0112] Intelligent model 4 is a maintenance action recognition model, which is used to identify the maintenance actions of trainees during maintenance training operations.

[0113] The list of maintenance actions is as described in the following table:

[0114] Maintenance action Instruction encoding category Maintenance action description Disassembly Operation AC001 1 Refers to the action that occurs when a wrench is used to remove a fastener. Fastening operation AC002 2 Refers to the action that occurs when a fastener is tightened using a wrench. Smear operation AC003 3 Refers to the action that occurs when conductive paste is applied to the conductive parts of equipment components. Dumping operation AC004 4 Refers to the action that occurs when a barrel of alcohol or other liquid is dumped. Wipe operation AC005 5 Refers to the action that occurs when cleaning the surface of equipment components with a white cloth. Rotary knob operation AC006 6 Refers to the action that occurs when adjusting the knob on the detection instrument panel. Visual operation AC007 7 Refers to the actions that occur when observing train equipment or components. Marking Operations AC008 8 Refers to the action that occurs when a fastener is marked for anti-loosening using a paint pen or other marking pen. Push button operation AC009 9 Refers to the action that occurs when a button on the test instrument panel is pressed. Pulling operation AC010 10 Refers to the action that occurs when a dynamometer is used to measure the tension of equipment components. Lifting operation AC011 11 Refers to the action that occurs when lifting a piece of equipment. ……… ……… ……… ………

[0115] It is impossible to completely and accurately judge whether a maintenance operation is correct based on the recognition result of the maintenance action alone. It is also necessary to combine the recognition result of the tool, the intersection-over-union ratio after the tool instance segmentation, and the judgment rules to determine whether a maintenance operation is correct.

[0116] The maintenance action recognition model is iteratively trained on the labeled maintenance action dataset through the human action recognition algorithm. The human action recognition algorithm uses a two-stream adaptive graph convolutional neural network, which includes an adaptive convolution layer, an adaptive graph convolution block, an adaptive graph convolutional network and a two-stream network. The working process of the algorithm is divided into the following steps:

[0117] Step 3.1: Use the pre-trained human pose estimation model to extract human skeleton key point data from the image.

[0118] Step 3.2: Extract joint data and skeleton data from the human skeleton key point data obtained in step 3.1. The joint data includes the coordinates of the source joint point and the coordinates of the target joint point. The skeleton data is represented by a vector pointing from the source joint point to the target joint point.

[0119] Step 3.3: Send the joint data and skeleton data obtained in step 3.2 to the joint data stream processing network (Joint Stream Network) and the skeleton data stream processing network (Bones Stream Network), respectively. The joint data stream processing network and the skeleton data stream processing network use distance-based sampling functions to sample the joint data and skeleton data respectively, and then use the spatial temporal graph convolutional network (Spatial Temporal Graph Convolutional Networks, ST-GCN) to adaptively construct the joint samples and skeleton samples into different topological structure graphs, and then send the topological structure graphs of the two types of samples to their respective Softmax classifiers to obtain the corresponding action prediction scores.

[0120] Step 3.4: Add the motion prediction scores of the joint samples and bone samples obtained in step 3.3 to obtain the final motion prediction label.

[0121] The second functional unit is the train maintenance knowledge graph unit, which is based on the electric train maintenance operation instructions, electric train quality inspection manual and electric train maintenance worker textbooks. Through document layout analysis technology, text detection and recognition technology, text parsing technology and knowledge graph construction technology, the unstructured knowledge content in the original bibliography is extracted into graph data, providing a knowledge basis for the maintenance rule engine.

[0122] The third functional unit is the maintenance rule engine unit. This functional unit is based on the visual process engine and provides instructors with a set of visual configuration interfaces for practical training and examination rule templates and a calling interface for the train maintenance knowledge graph. When configuring the practical training and examination rule templates, instructors can dynamically select the required train equipment, equipment components, maintenance tasks, maintenance tools, maintenance methods and technical requirements from the train maintenance knowledge graph. After the practical training and examination rule templates are configured, in the actual maintenance training test, trainees will select the train model, train equipment and Maintenance level, the maintenance rule engine will generate the practical training test content, maintenance task timing rules, tool use compliance judgment rules, maintenance action compliance judgment rules and scoring rules based on the practical training test rule template configured by the instructor and the built-in graph calculation algorithm. Students complete various maintenance tasks according to the specified test content. The maintenance rule engine uses the above rules to judge whether the timing relationship of the maintenance task completion during the student's maintenance process is reasonable, whether the tools used are compliant, and whether the maintenance actions are compliant. Finally, the scoring rules are used to comprehensively score the student's entire maintenance process and give the test results. In addition, during the student's maintenance training operation, the maintenance rule engine also needs to forward the tool instructions, maintenance action instructions and instrument readings sent by the intelligent model unit to the three-dimensional simulation unit in real time.

[0123] The fourth functional unit is the three-dimensional simulation unit, which first constructs the 3D model of the maintenance site, the 3D model of the train equipment, the 3D model of the tool, the 3D model of the human body and the 3D model of the hand through the three-dimensional modeling tool. Then, after receiving the tool instructions, maintenance action instructions and instrument readings sent by the maintenance rule engine, the 3D animation engine is responsible for loading and displaying the corresponding 3D model animation in real time, thereby realizing the virtual simulation of the maintenance training test process.

[0124] See also Figure 4 , the construction process of the train maintenance knowledge graph of the present invention will be described. The construction process is divided into the following steps:

[0125] Step 4.1: First, extract the knowledge content from the electric train maintenance operation instruction manual and the electric train quality inspection manual. Since these two books are text versions of PDF, the contents in the bibliography can be read directly, and the chapters, paragraphs and table data in the book are parsed through matching rules, and the text data is converted into a tree structure according to the original knowledge structure in the bibliography; then extract the knowledge content from the electric train maintenance textbook. Since this series of textbooks are scanned PDFs, the contents in the bibliography cannot be read directly, so the headers, footers, illustrations, tables and texts in the textbooks are identified through document layout analysis technology and text recognition technology, and then the text data is converted into tree structure data with the same knowledge structure as the original bibliography through matching rules.

[0126] Step 4.2: Extract relevant domain vocabulary and vocabulary classification from various publicly released rail transit professional dictionaries to obtain the rail transit domain vocabulary library, extract text data from various standardized documents, maintenance worker teaching materials, operation instructions, maintenance manuals and other materials in the rail transit field, and then obtain the pre-selected vocabulary library by calculating the left and right entropy and solidification degree based on the seed words, and then obtain the custom domain vocabulary library through custom filtering rules and some manual screening. Finally, the rail transit domain vocabulary library and the custom domain vocabulary library are merged and deduplicated to obtain the keyword vocabulary library required to construct the train maintenance knowledge graph, and on this basis, the categories of the vocabulary in the keyword vocabulary library are marked.

[0127] Step 4.3: Use the multi-modal string matching algorithm to load the keyword thesaurus obtained in step 4.2, then perform keyword matching on the tree structure data obtained in step 4.1, mark the matched keywords and add corresponding category labels to obtain the corpus of the named entity recognition model, and then train the corpus through a bidirectional long short-term memory neural network to finally obtain the entity recognition model in the rail transit field.

[0128] Step 4.4: Use the rail transit domain entity recognition model obtained in step 4.3 to perform named entity recognition on the tree structure data obtained in step 4.1, and then perform text parsing through domain entity extraction rules to extract rail transit domain entities.

[0129] Step 4.5: Use dependency syntax analysis to analyze the grammatical relationship between rail transit domain entities in the tree structure data obtained in step 4.1, and then extract the relationship through the predefined entity relationship template and domain entity relationship extraction rules to obtain the subordinate relationship between domain entities.

[0130] Step 4.6: After obtaining the subordinate relationship between domain entities through steps 4.4 and 4.5, extract the attributes of domain entities from the tree structure data through named entity recognition, part-of-speech analysis, dependency syntax analysis and attribute extraction rules.

[0131] Step 4.7: The domain entities, entity relationships, and entity attributes extracted in steps 4.4, 4.5, and 4.6 are respectively constructed into a triple structure, and the data of the triple structure is imported into the graph database to obtain a train maintenance knowledge graph.

[0132] Step 4.8: Load the train maintenance knowledge graph obtained in step 4.7 into the visualization interface, manually review the train maintenance knowledge graph, and manually correct the incorrectly extracted domain entities, entity relationships, and entity attributes.

[0133] See also Figure 3 The overall process of the electric train maintenance training and evaluation system of the present invention is described. The overall processing flow of the system is divided into the following steps:

[0134] Step 5.1: The trainee selects and submits the train model, train equipment and maintenance level for this maintenance training test on the interactive terminal at the maintenance training site. The interactive terminal is responsible for sending this information to the maintenance rule engine, which then forwards it to the 3D animation engine. The 3D animation engine loads and displays the 3D models of the corresponding maintenance site and train equipment, and prompts the start of the test.

[0135] Step 5.2: After receiving the data sent by the interactive terminal, the maintenance rule engine is responsible for obtaining the maintenance task list of this practical training test from the train maintenance knowledge graph, and loading the maintenance task timing rules of the current train equipment. Then, according to the sequence of maintenance tasks, it obtains the knowledge content such as tools, maintenance methods and technical requirements required for each maintenance task from the train maintenance knowledge graph in sequence, and loads the tool use compliance judgment rules, maintenance action compliance judgment rules and scoring rules.

[0136] Step 5.3: The streaming media service is responsible for collecting multiple video streams of the trainees' maintenance training test process, extracting key frames from the video streams, converting the key frames into images, preprocessing the images and converting them into Base64 encoding, and then sending the Base64 encoding to the intelligent model service.

[0137] Step 5.4: The intelligent model service is responsible for detecting, identifying and segmenting the tools in the image, identifying the readings of the instruments in the image, identifying the maintenance actions in the continuous video frames, and then converting the identified tools and maintenance actions into corresponding tool instructions and maintenance action instructions respectively, and finally sending the tool instructions, maintenance action instructions and instrument readings to the maintenance rule engine.

[0138] Step 5.5: The maintenance rule engine first forwards the tool instructions, maintenance action instructions and instrument readings to the 3D animation engine, which then loads and displays the maintenance scene, train equipment, equipment parts and the 3D models and operation animations of the received tools and maintenance actions in real time.

[0139] Step 5.6: The maintenance rule engine judges whether the tools used in the maintenance process are compliant, whether the maintenance actions are compliant, and whether the maintenance operation sequence is compliant based on the maintenance task timing rules, tool usage compliance judgment rules, and maintenance action compliance judgment rules, and then scores the completion of the maintenance task based on the scoring rules and the above judgment results.

[0140] Step 5.7: The maintenance rule engine is responsible for sending the scoring results of the maintenance tasks to the 3D animation engine, and the 3D animation engine displays the scores of the maintenance tasks in real time.

[0141] See also Figure 5 , which is the pantograph maintenance related subgraph in the train maintenance knowledge graph of the present invention. Among them, the entity nodes include equipment name, component name, maintenance task, inspection item, maintenance method, tool, tooling, related equipment, auxiliary materials, technical requirements, maintenance cycle and status, and the entity relationships are all subordinate relationships.

[0142] See also Figure 6 , is a hardware deployment architecture diagram of the electric train maintenance training and evaluation system of the present invention, wherein the hardware equipment includes:

[0143] (1) Sensing point cameras: The sensing point cameras include head-mounted wireless cameras, sensing point cameras in the top area of ​​the training and maintenance warehouse, sensing point cameras in the operating table area of ​​the training and maintenance warehouse, and sensing point cameras on the tool table. These cameras are responsible for capturing videos of the trainees' maintenance training operation process and sending the captured video data to the GPU server.

[0144] (2) GPU server cluster: The GPU server cluster includes multiple GPU servers, each of which is equipped with multiple GPU graphics cards. The intelligent model unit is deployed and installed in the GPU server cluster, and the software operating environment, deep learning framework, trained intelligent model, and related streaming media services are deployed. Each GPU server is responsible for processing the corresponding video data and sending the results of intelligent model recognition to the application server.

[0145] (3) Database server cluster: The database server cluster includes multiple application servers, each of which is responsible for storing train maintenance knowledge graph data, train maintenance rule data and application data. The electric train maintenance knowledge graph unit is deployed and installed in the database server cluster, and structured database management software and graph database management software are deployed.

[0146] (4) Application server cluster: The maintenance rule engine unit is deployed and installed in the application server cluster, which contains multiple application servers. Each application server is respectively deployed with maintenance rule engine service, maintenance rule management service, basic parameter management service, sensing point management service, 3D model management service and 3D simulation visualization service. All application services are deployed, managed and scheduled in the form of microservices. In particular, the maintenance rule engine service is responsible for receiving the recognition results sent by the intelligent model service and obtaining application data and graph data from the database server. Then, the maintenance training operations of the trainees are scored according to the maintenance rules. At the same time, the results recognized by the intelligent model are reconstructed in three dimensions and displayed on the visualization large screen.

[0147] (5) Large-screen display device: This large-screen display device is responsible for displaying the 3D model and operation animation reconstructed by the application server, making it convenient for instructors and other students to watch and learn.

[0148] See also Figure 7 , is an electric train equipment training and maintenance warehouse proposed by the present invention, and the training and maintenance warehouse mainly includes the following parts:

[0149] The first part is the tool bench in the training and maintenance warehouse. The top of the tool bench is a scale plate with a tool bench sensing point camera installed directly above it. The tool bench is used to place the tools selected by the trainees and provide a scale reference for the identification of tool specifications. The video data collected by the tool bench sensing point camera is mainly used for the detection, identification and specification judgment of tools.

[0150] The second part is the operating table of the training maintenance warehouse. The operating table adopts a grid layout, and provides an installation position for a sensing point camera at each grid intersection. Each installation position embeds the camera inside the operating table. This does not affect the trainees' operation, and it is convenient for the instructor to configure and switch the sensing point camera according to the type of equipment to be repaired, so that each sensing point camera that is turned on can shoot from the best angle. The video data collected by the sensing point camera of the operating table is mainly used for the detection and identification of tools and tools, as well as the identification of trainees' maintenance actions.

[0151] The third part is the top area of ​​the training maintenance warehouse. The top area also adopts a grid layout. A sensing point camera is installed at each grid intersection. The top sensing point camera also supports the instructor to configure and switch the top sensing point camera according to the specific requirements of the maintenance task. The video data collected by the top sensing point camera is mainly used for the recognition of trainees' maintenance actions.

[0152] The fourth part is the trainee's head-mounted wireless camera, which can take close-up photos of equipment parts, instruments, maintenance tools, and trainees' hand operations, solving the problem that the operating table sensor point camera and the top sensor point camera cannot take high-quality photos of equipment parts, instruments, maintenance tools, and trainees' hand operations. The video data collected by the head-mounted wireless camera is mainly used for the detection and identification of tools and the identification of trainees' maintenance actions.

[0153] See also Figure 8 , is a maintenance rule engine for the arrester balanced maintenance operation, which is one example of the train maintenance rule engine described in the present invention. Its working process is divided into the following steps:

[0154] Step 6.1: The trainee selects tools and puts them on the tool bench. The camera at the tool bench sensing point will capture video of the tools selected by the trainee. The tool detection and recognition model is responsible for detecting and identifying the tools selected by the trainee. The tool selection correctness judgment rule determines whether the type, specification and number of tools are correct based on the recognition results; if the type and specification of the tools selected by the trainee are correct, without multiple selections or omissions, 1 point will be awarded, otherwise no points will be awarded.

[0155] Step 6.2: The trainee checks whether the connecting wires on the arrester are damaged, whether the fasteners are intact, and whether the anti-loosening marks are clear and not offset. The head-mounted wireless camera is responsible for collecting the appearance images of the connecting wires and fasteners, the top sensing point camera is responsible for collecting the trainee's visual actions, the tool detection and recognition model is responsible for identifying the connecting wires and fasteners, and the maintenance action recognition model is responsible for identifying the visual actions; if the visual action, connecting wires and fasteners are recognized at the same time, 1 point is awarded, otherwise no points are awarded.

[0156] Step 6.3: The trainee disconnects the connecting wires on the arrester. The head-mounted wireless camera is responsible for collecting the maintenance tools held by the trainee, the top sensing point camera is responsible for collecting the trainee's disassembly operation of the continuous line, the tool detection and recognition model is responsible for identifying tools, the maintenance action recognition model is responsible for identifying disassembly operations, the tool instance segmentation model is responsible for segmenting tools of various categories from the image, and the socket wrench compliance judgment rule determines whether the trainee is using a socket wrench, whether the disassembly operation is correct, and whether the intersection and union ratio of the nut and the socket reaches the set threshold based on the recognition results; if all of the above judgments are yes, 1 point is awarded, otherwise no points are awarded.

[0157] Step 6.4: The trainee inspects the jacket on the arrester to see if it is intact, cracked, or has any signs of ablation. The head-mounted wireless camera is responsible for collecting images of the arrester's appearance, the top sensing point camera is responsible for collecting the trainee's visual actions, the tool detection and recognition model is responsible for identifying the arrester from the image, and the maintenance action recognition model is responsible for identifying the visual action; if the visual action and the arrester are recognized at the same time, 1 point is awarded, otherwise no points are awarded.

[0158] Step 6.5: The trainee cleans the arrester. The tool detection and recognition model is responsible for identifying the arrester, white cloth and alcohol barrel. The maintenance action recognition model is used to identify the pouring operation of the alcohol barrel and the wiping operation of the white cloth. The tool instance segmentation model is used to segment the arrester and white cloth in the image. The cleaning operation compliance judgment rule judges whether the tool used by the trainee is correct, whether the pouring operation and the wiping operation exist, and whether the timing relationship is correct, and whether the intersection and union ratio of the arrester and the white cloth reaches the set threshold according to the recognition results. If all of the above judgments are yes, 1 point is awarded, otherwise no points are awarded.

[0159] Step 6.6: The trainee uses the arrester DC parameter tester to test the voltage withstand performance of the arrester. Connect the positive pole of the tester lead clip to the top of the arrester, and connect the negative pole of the tester lead clip to the bottom. Press the power switch and the test button in sequence. The tester starts testing. The current display and voltage display show the voltage reading and current reading respectively. After the reading is completed, press the reset button to end the test. The tool detection and identification model is responsible for identifying the tester, lead clip, lightning arrester, power switch, detection button and reset button. The maintenance action recognition model is responsible for identifying the wiring operation of the lead clip and the button pressing operation. The instrument reading recognition model is responsible for voltage readings and current readings. The tool instance segmentation model segments the tester, lead clip, lightning arrester, power switch, detection button and reset button from the image. The tester use correctness judgment rule judges whether the tool used by the trainee is correct, whether the wiring operation of the lead clip and the button pressing operation are correct, whether the button pressing sequence is correct, whether the intersection and union ratio of the lead clip and the top and bottom of the lightning arrester respectively reaches the set threshold, and whether the intersection and union ratio of the hand and the power switch, detection button and reset button respectively reaches the set threshold; if all of the above judgments are yes, then 3 points will be awarded, otherwise no points will be awarded.

[0160] Step 6.7: Trainees use a socket wrench to install the arrester. The tool detection and recognition model is responsible for identifying the socket wrench, socket, arrester, connecting wire, fasteners and arrester base from the image, the maintenance action recognition model is responsible for identifying the tightening operation of the connecting wire, and the tool instance segmentation model is responsible for segmenting the socket wrench, socket, arrester, connecting wire, fasteners and arrester base from the image. The socket wrench use compliance judgment rule determines whether the tool is correct, whether the tightening operation of the connecting wire is correct, and whether the intersection and union ratio of the connecting wire to the top and bottom of the arrester reaches the set threshold based on the recognition results; if all of the above judgments are yes, 1 point is awarded, otherwise no points are awarded.

[0161] Step 6.8: Trainees use paint pens to mark fasteners for prevention of loosening. The tool detection and recognition model is responsible for identifying the paint pen, lightning arrester and fasteners from the image, the maintenance action recognition model is responsible for identifying the operation of using the paint pen to mark for prevention of loosening, the tool instance segmentation model is responsible for segmenting the paint pen and fasteners from the image, and the compliance judgment rule for marking for prevention of loosening is responsible for judging whether the tool is correct, whether the operation of using the paint pen to mark for prevention of loosening is correct, and whether the intersection of the paint pen and the fastener reaches the set threshold based on the recognition results; if all of the above judgments are yes, 1 point is awarded, otherwise no points are awarded.

[0162] Step 6.9: The trainee returns the used tools to the tool bench after maintenance. The camera at the tool bench sensing point will capture the video of the tools returned by the trainee. The tool detection and recognition model is responsible for identifying the tools on the tool bench. The tool return correctness judgment rule determines whether the type and number of tools returned are correct based on the recognition results; if the type of tools returned by the trainee is correct and there is no omission, 1 point will be awarded, otherwise no points will be awarded.

[0163] Step 6.10: The trainee fills in and submits the balanced maintenance record sheet for this maintenance training task on the interactive terminal.

[0164] Step 6.11: After submitting the balance repair record sheet, the trainee will compare the results of the visual inspection actions in the above steps with the set fault points according to the contents of the record sheet. If all comparisons are successful, the scores of the visual inspection actions will all take effect. Otherwise, the scores of the visual inspection actions that failed the comparison will be deducted.

[0165] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graph, characterized by: The system consists of: Sensing point cameras: These cameras include head-mounted wireless cameras, sensing point cameras in the top area of ​​the training and maintenance warehouse, sensing point cameras in the operating table area of ​​the training and maintenance warehouse, and sensing point cameras on the tool table. These cameras are responsible for collecting videos of the trainees' maintenance and training operation process and sending the collected video data to the GPU server cluster; GPU server cluster: The GPU server cluster includes multiple GPU servers, each of which is equipped with multiple GPU graphics cards. The GPU server cluster is equipped with an intelligent model unit. Each GPU server is responsible for processing the corresponding video data and sending the results of intelligent model recognition to the application server. Database server cluster: The database server cluster includes multiple application servers, in which an electric train maintenance knowledge graph unit is deployed and installed; each application server is responsible for storing train maintenance knowledge graph data, train maintenance rule data and application data, respectively. Application server cluster: The application server cluster deploys and installs the maintenance rule engine unit. The application server cluster contains multiple application servers. Each application server deploys the maintenance rule engine service, maintenance rule management service, basic parameter management service, sensing point management service, 3D model management service and 3D simulation visualization service. All application services are deployed, managed and scheduled in the form of microservices. In particular, the maintenance rule engine service is responsible for receiving the recognition results sent by the intelligent model service, and obtaining application data and graph data from the database server. Then, the maintenance training operations of the trainees are scored according to the maintenance rules. At the same time, the results recognized by the intelligent model are reconstructed in 3D and displayed on the visualization screen. Large-screen display device: This large-screen display device is responsible for displaying the 3D model and operation animation of the application server after reconstruction, which is convenient for teachers and other students to watch and learn; The intelligent model unit includes four intelligent models, namely, a tool detection and recognition model, a tool instance segmentation model, an instrument reading recognition model, and a maintenance action recognition model. These intelligent models are responsible for intelligently recognizing the video stream data collected by the camera, and after the recognition is completed, the recognition results of the tool and the maintenance action are converted into corresponding tool instructions and maintenance action instructions, and then the tool instructions, maintenance action instructions and instrument readings are sent to the maintenance rule engine; The tool detection and recognition model is used to detect and recognize train equipment, equipment parts, maintenance tools, instruments, human bodies and hands in the image; The tool detection and recognition model is iteratively trained on the labeled tool dataset through the target detection and recognition algorithm. The target detection and recognition algorithm uses a single-stage fully convolutional detection network based on pixel-level prediction. Its network structure includes a backbone network, a feature pyramid network, and a detector. The detector contains three branches, namely, the classification prediction branch, the center prediction branch, and the regression prediction branch. The corresponding working process is divided into the following steps: Step 1.1: Perform RGB to grayscale conversion, resizing, and image data to tensor preprocessing on the input image; Step 1.2: Send the tensor data obtained in step 1.1 into the backbone network to obtain feature maps of five different scales of the input data; Step 1.3: Send the feature map obtained in step 1.2 to the feature pyramid network in the next step, and use the pre-defined anchor box to perform regression prediction on each pixel point in the feature map; Step 1.4: Calculate the center of each predicted bounding box obtained in step 1.3 through binary cross entropy loss, and calculate the confidence of the predicted bounding box by multiplying the center by the corresponding classification score; Step 1.5: Finally, filter out the low-quality bounding box obtained in step 1.4 through non-maximum suppression, and sort by confidence to obtain the final bounding box and classification category of the target object; The tool instance segmentation model is used to perform instance segmentation on train equipment, equipment parts, maintenance tools, instruments, human bodies and hands in the image. The tool instance segmentation result is used to calculate the intersection-and-union ratio between tools. The maintenance rule engine needs to combine the recognition result of the maintenance action and the intersection-and-union ratio between tools to determine whether the maintenance action is compliant. The tool instance segmentation model is iteratively trained on annotated tool datasets using an instance segmentation algorithm, which consists of a detection module and a hybrid mask module. The detection module is used to predict the bounding box coordinates and classification category of instance objects. The hybrid mask module consists of three parts: a bottom module for generating masks, a top module for generating top-level attention, and a hybrid module that fuses masks and attention. The specific functions of each module are as follows: Detection module: The detection module has the same network structure and working process as the intelligent model 1. It is responsible for predicting the bounding box and classification category of the instance object in the input image and providing feature input for the bottom module and the top module. Top module: The bounding box proposal of each instance object is obtained through the detection module, and then a convolution operation is performed to obtain the top-level attention; Bottom module: The bottom module takes the feature map of the backbone network and feature pyramid in the detection module as input, and obtains K masks of the entire image through upsampling, concatenation and convolution operations respectively; Hybrid module: The hybrid module takes the instance candidate bounding box, the top-level attention, and K masks of the entire image as input, crops, scales, and aligns the K masks respectively, interpolates the top-level attention, and obtains a set of score maps after Softmax normalization. Finally, the score map is multiplied by the processed K masks and added by channel to obtain the final mask. The instrument reading recognition model uses the results of the tool detection and recognition model as input, cuts out the area where the instrument is located from the image, and recognizes the measurement data displayed on the instrument; The instrument reading recognition model is iteratively trained on the labeled instrument reading dataset and optical character dataset through the optical character recognition algorithm. The optical character recognition algorithm uses a multi-task parallel full convolution point aggregation network to support real-time detection and recognition of text of any shape; the corresponding working process is divided into the following steps: Step 2.1: Correct the image using corner detection and Hough transform; Step 2.2: Send the image obtained in step 2.1 to the backbone network with the feature pyramid network to obtain a 1 / 4 downsampled feature map; Step 2.3: Use a fully convolutional network to perform multi-task parallel learning on the feature map obtained in step 2.2, and obtain feature information of four modes of the text area, including text centerline, text edge offset, text direction offset, and text character classification map; Step 2.4: Use the text centerline and text direction offset obtained in step 2.3 to extract the text center point sequence according to the reading order of the text, and then calculate the text boundary box through the text edge offset; Step 2.5: Use the text centerline and text direction offset obtained in step 2.3 to upsample the text character classification map, concatenate the sampling points into a fixed-length sequence, and then use the connectionist temporal classification loss function for training to obtain the character classification probability sequence, and decode it into the final text recognition result; The maintenance action recognition model is used to recognize the maintenance actions of trainees during maintenance training operations; the maintenance action recognition model is obtained by iteratively training the labeled maintenance action data set through a human behavior recognition algorithm, and the human behavior recognition algorithm uses a two-stream adaptive graph convolutional neural network, whose components include an adaptive convolution layer, an adaptive graph convolution block, an adaptive graph convolutional network and a two-stream network; the corresponding working process is divided into the following steps: Step 3.1: Use the pre-trained human pose estimation model to extract human skeleton key point data from the image; Step 3.2: Extract joint data and bone data from the human skeleton key point data obtained in step 3.1, the joint data includes the coordinates of the source joint point and the coordinates of the target joint point, and the bone data is represented by a vector pointing from the source joint point to the target joint point; Step 3.3: Send the joint data and skeleton data obtained in step 3.2 to the joint data stream processing network and the skeleton data stream processing network respectively, wherein the joint data stream processing network and the skeleton data stream processing network respectively use distance-based sampling functions to sample the joint data and the skeleton data, and then adaptively construct the joint samples and the skeleton samples into different topological structure graphs through the spatiotemporal graph convolution network, and then send the topological structure graphs of the two types of samples to their respective Softmax classifiers to obtain the corresponding action prediction scores; Step 3.4: Add the motion prediction scores of the joint samples and the bone samples obtained in step 3.3 to obtain the final motion prediction label; The train maintenance knowledge graph unit is based on the electric train maintenance operation instruction book, the electric train quality inspection manual and the electric train maintenance worker textbook. Through document layout analysis technology, text detection and recognition technology, text parsing technology and knowledge graph construction technology, the unstructured knowledge content in the original bibliography is extracted into graph data, providing a knowledge basis for the maintenance rule engine. The construction process of the train maintenance knowledge graph is divided into the following steps: Step 4.1: First, extract the knowledge content from the electric train maintenance operation instruction book and the electric train quality inspection manual. Since these two books are text-based PDFs, directly read the content in the bibliography, and parse the chapters, paragraphs, and table data in the book through matching rules, and convert the text data into a tree structure according to the original knowledge structure in the bibliography; then extract the knowledge content from the electric train maintenance worker textbook. Since this series of textbooks are scanned PDFs, it is impossible to directly read the content in the bibliography, so the headers, footers, illustrations, tables, and text in the textbooks are recognized through document layout analysis technology and text recognition technology, and then the text data is converted into tree structure data with the same knowledge structure as the original bibliography through matching rules; Step 4.2: Extract relevant domain vocabulary and vocabulary classification from various publicly released rail transit professional dictionaries to obtain the rail transit domain vocabulary library, extract text data from various standardized documents, maintenance worker teaching materials, operation instructions, and maintenance manuals in the rail transit field, and then obtain the pre-selected vocabulary library by calculating the left and right entropy and solidification degree based on the seed words, and then obtain the custom domain vocabulary library through custom filtering rules and some manual screening. Finally, the rail transit domain vocabulary library and the custom domain vocabulary library are merged and deduplicated to obtain the keyword vocabulary library required for constructing the train maintenance knowledge graph, and on this basis, the categories of the vocabulary in the keyword vocabulary library are marked; Step 4.3: Use the multi-mode string matching algorithm to load the keyword thesaurus obtained in step 4.2, then perform keyword matching on the tree structure data obtained in step 4.1, and mark the matched keywords and add corresponding category labels to obtain the corpus of the named entity recognition model. Then, the corpus is trained through a bidirectional long short-term memory neural network, and finally a rail transit field entity recognition model is obtained; Step 4.4: Use the rail transit domain entity recognition model obtained in step 4.3 to perform named entity recognition on the tree structure data obtained in step 4.1, and then perform text parsing using domain entity extraction rules to extract rail transit domain entities; Step 4.5: Use dependency syntax analysis to analyze the grammatical relationship between rail transit domain entities in the tree structure data obtained in step 4.1, and then extract the relationship through the predefined entity relationship template and domain entity relationship extraction rules to obtain the subordinate relationship between domain entities; Step 4.6: After obtaining the subordinate relationship between domain entities through steps 4.4 and 4.5, extract the attributes of domain entities from the tree structure data through named entity recognition, part-of-speech analysis, dependency syntax analysis and attribute extraction rules; Step 4.7: construct the domain entities, entity relationships and entity attributes extracted in steps 4.4, 4.5 and 4.6 into a triple structure respectively, and import the data of the triple structure into the graph database, so as to obtain the train maintenance knowledge graph; Step 4.8: Load the train maintenance knowledge graph obtained in step 4.7 into the visualization interface, manually review the train maintenance knowledge graph, and manually correct the incorrectly extracted domain entities, entity relationships, and entity attributes; The maintenance rule engine unit, based on the visual process engine, provides instructors with a set of visual configuration interfaces for practical training and examination rule templates and a calling interface for the train maintenance knowledge graph. When configuring the practical training and examination rule templates, instructors dynamically select the required train equipment, equipment components, maintenance tasks, maintenance tools, maintenance methods and technical requirements knowledge nodes from the train maintenance knowledge graph. After the practical training and examination rule templates are configured, in the actual maintenance training and examination, trainees will select the train model, train equipment and maintenance level to be inspected through the interactive interface in the electric train equipment maintenance training and evaluation system. The maintenance rule engine will generate the practical training and examination content, maintenance task timing rules, tool usage and other information according to the practical training and examination rule templates configured by the instructor and the built-in graph calculation algorithm. Compliance judgment rules, maintenance action compliance judgment rules and scoring rules. Students complete various maintenance tasks according to the specified test content. The maintenance rule engine uses the algorithm to generate maintenance task timing rules, tool use compliance judgment rules, maintenance action compliance judgment rules and scoring rules to judge whether the timing relationship of the maintenance task completion during the student's maintenance process is reasonable, whether the tools used are compliant, and whether the maintenance actions are compliant. Finally, the scoring rules are used to comprehensively score the student's entire maintenance process and give the test results. In addition, during the student's maintenance training operation, the maintenance rule engine also needs to forward the tool instructions, maintenance action instructions and instrument readings sent by the intelligent model unit to the three-dimensional simulation unit in real time. The three-dimensional simulation unit first constructs a 3D model of the maintenance site, a 3D model of train equipment, a 3D model of tools, a 3D model of the human body and a 3D model of the hand through a three-dimensional modeling tool; After receiving the tool instructions, maintenance action instructions and instrument readings sent by the maintenance rule engine, the 3D animation engine is responsible for loading and displaying the corresponding 3D model animation in real time, thereby realizing the virtual simulation of the maintenance training test process; The overall process of the electric train equipment maintenance training and evaluation system is divided into the following steps: Step 5.1: The trainee selects and submits the train model, train equipment and maintenance level for this maintenance training test on the interactive terminal at the maintenance training site. The interactive terminal is responsible for sending this information to the maintenance rule engine, which then forwards it to the 3D animation engine. The 3D animation engine loads and displays the 3D models of the corresponding maintenance site and train equipment, and prompts the start of the test; Step 5.2: After receiving the data sent by the interactive terminal, the maintenance rule engine is responsible for obtaining the maintenance task list of this practical training test from the train maintenance knowledge graph, and loading the maintenance task timing rules of the current train equipment. Then, according to the order of the maintenance tasks, the tools, maintenance methods and technical requirements required for each maintenance task are obtained from the train maintenance knowledge graph, and the tool use compliance judgment rules, maintenance action compliance judgment rules and scoring rules are loaded; Step 5.3: The streaming media service is responsible for collecting multiple video streams of the trainee maintenance training test process, extracting key frames from the video streams, converting the key frames into images, preprocessing the images and converting them into Base64 codes, and then sending the Base64 codes to the intelligent model service; Step 5.4: The intelligent model service is responsible for identifying and segmenting the tools in the image, identifying the readings of the instruments in the image, and identifying the maintenance actions in the continuous video frames. It then converts the identified tools and maintenance actions into corresponding tool instructions and maintenance action instructions, and finally sends the tool instructions, maintenance action instructions and instrument readings to the maintenance rule engine. Step 5.5: The maintenance rule engine first forwards the tool instructions, maintenance action instructions and instrument readings to the 3D animation engine, which then loads and displays the maintenance scene, train equipment, equipment parts and the received 3D models and operation animations of the tools and maintenance actions in real time; Step 5.6: The maintenance rule engine judges whether the tools used in the maintenance process are compliant, whether the maintenance actions are compliant, and whether the maintenance operation sequence is compliant according to the maintenance task timing rules, tool use compliance judgment rules, and maintenance action compliance judgment rules, and then scores the completion of the maintenance task according to the scoring rules and judgment results; Step 5.7: The maintenance rule engine is responsible for sending the scoring results of the maintenance tasks to the 3D animation engine, and the 3D animation engine displays the scores of the maintenance tasks in real time.

2. The electric train equipment maintenance training and evaluation system based on artificial intelligence and knowledge graph according to claim 1 is characterized in that; The system also includes an electric train equipment training and maintenance warehouse, which includes the following parts: The first part is the tool bench in the training and maintenance warehouse. The table top of the tool bench is a scale plate with a tool bench induction point camera installed directly above it. The tool bench is used to place the tools selected by the trainees and provides a scale reference for the identification of tool specifications. The video data collected by the tool bench sensing point camera is used for tool detection, identification and specification judgment; The second part is the operating table of the training maintenance warehouse. The operating table adopts a grid layout, and provides an installation position for a sensing point camera at each grid intersection. Each installation position embeds the camera into the operating table. This does not affect the trainees' operation, and is convenient for the instructor to configure and switch the sensing point camera according to the type of equipment to be repaired, so that each turned-on sensing point camera can shoot from the best angle; the video data collected by the sensing point camera of the operating table is used for the detection and identification of tools, as well as the identification of trainees' maintenance actions; The third part is the top area of ​​the training maintenance warehouse. The top area also adopts a grid layout. A sensing point camera is installed at each grid intersection. The top sensing point camera also supports the instructor to configure and switch the top sensing point camera according to the specific requirements of the maintenance task. The video data collected by the top sensing point camera is used to identify the trainees’ maintenance actions; The fourth part is the trainee's head-mounted wireless camera, which can take close-up photos of equipment parts, instruments, maintenance tools and trainees' hand operations, solving the problem that the operating table sensing point camera and the top sensing point camera cannot take high-quality photos of equipment parts, instruments, maintenance tools and trainees' hand operations; The video data collected by the head-mounted wireless camera is used to detect and identify tools, as well as to identify the trainees’ maintenance actions.

Citation Information

Patent Citations

  • Freight train axle state monitoring and early warning system and method

    CN111521421A

  • HS code matching method and system based on intelligent analysis and recognition, HS code display method and system based on intelligent analysis and recognition and storage medium

    CN112434691A

  • Rail transit emergency method and device based on knowledge graph

    CN112612904A

  • IT0010510