Transformer substation safety training method and system based on gesture recognition and voice interaction

Abstract

The invention provides a simulation ability training method and system based on gesture recognition and voice interaction, and the method comprises the steps: building the engineering task data of corresponding training in a VR system, and training a technician based on a VR virtual simulation technology; a technician wears VR equipment within the range of the VR system, carries out operation interaction on the VR system based on a voice interaction mode, and carries out manual simulation operation on a given engineering task; the operation process of a technician is recorded in real time based on a camera, action postures corresponding to operation are recorded and analyzed, and the action postures comprise gesture actions and limb actions. According to the invention, based on the VR virtual simulation technology, a mode of simulating on-site operation interaction and increasing training is realized, a mode of adopting VR equipment is adopted, the training cost is saved, various complex environment conditions can be effectively simulated, and a better training effect is brought.

Description

Technical Field

[0001] The present invention relates to the technical field of substation safety training, and in particular to a substation safety training method and system based on gesture recognition and voice interaction, and also to a corresponding computer terminal and a computer-readable storage medium. Background Art

[0002] With the rapid development of the power industry and the widespread application of smart grids, substations are an indispensable key link in the power system. Their operating efficiency and safety are directly related to the stability and reliability of the entire power grid. Substation safety training is a key measure to ensure the safe operation of substations and improve the professional skills and emergency response capabilities of technical personnel. The main goal of substation safety training is to enhance the safety awareness, professional skills and emergency response capabilities of technical personnel, ensure that all substation operations comply with safety regulations, reduce accident risks, and ensure safe and stable operation of the power grid. Therefore, improving the professional skills and emergency response capabilities of substation technicians is of great significance to ensuring the safety of the power grid and improving operation and maintenance efficiency. However, traditional substation technical training methods have many limitations. The training methods are single. Current substation technical training mostly adopts a combination of theoretical explanation and on-site practice. However, on-site practice is limited by time, space and safety factors, and is difficult to carry out frequently. There are also great safety risks. The training cost is high. On-site practice requires a lot of manpower, material and financial resources, including equipment maintenance, site rental, and security costs, which increases the training cost and makes it difficult to simulate complex working conditions. Various complex working conditions and emergencies may be encountered during actual substation operation, but traditional training methods are difficult to fully and accurately simulate these conditions, resulting in technical personnel lacking the ability to deal with complex situations. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a substation safety training method and system based on gesture recognition and voice interaction, and also provides a corresponding computer terminal and computer-readable storage medium.

[0004] According to one aspect of the present invention, a substation safety training method based on gesture recognition and voice interaction is provided, comprising:

[0005] Construct engineering task data corresponding to training in the VR system;

[0006] Based on VR equipment, operate and interact with the VR system, and perform hands-on simulation operations on the engineering task data;

[0007] Record the action posture of the simulated operation process in real time;

[0008] Matching the recorded action posture with a preset standard action posture, and scoring the completed action posture of each task;

[0009] Evaluate the current task operation capability based on the score.

[0010] Preferably, the construction of engineering task data corresponding to the training in the VR system includes:

[0011] Configure the parameters of the training project in the VR system, build the scene of the engineering task resources, construct a virtual VR engineering scene, and obtain the engineering task data corresponding to the training as visual information for training based on the VR device;

[0012] Preferably, the parameters of the training project are configured in the VR system, the scene of the engineering task resources is built, a virtual VR engineering scene is constructed, and the engineering task data corresponding to the training is obtained, including:

[0013] Determine the goals to be achieved in this VR training, conduct a comprehensive analysis of related engineering tasks, and sort out the required engineering task resources, including: engineering drawings, equipment parameters, operating manuals and safety standards;

[0014] According to the training objectives and engineering task resources, set various parameters of the virtual scene in the VR system, including: lighting conditions, weather conditions and time changes;

[0015] Using 3D modeling technology, based on the acquired engineering task resources, the engineering equipment in the virtual scene is constructed, and the engineering equipment is placed in the virtual scene according to the actual layout to construct a virtual VR engineering scene;

[0016] The actual engineering task process is simulated in a virtual scene, including equipment operation, construction steps and safety inspection, to obtain the engineering task data for the corresponding training.

[0017] Preferably, placing the engineering equipment in the virtual scene according to the actual layout includes:

[0018] The engineering equipment has a coordinate (x', y', z') in three-dimensional space and is referenced and positioned based on the following formula: A = (x'-x1', y'-y1', z'-z1'), where (x1', y1', z1') is the reference coordinate point of the engineering equipment in the three-dimensional coordinate space. It is used as the reference coordinate for positioning and substituted into the three-dimensional coordinate in the virtual scene.

[0019] Preferably, the operation interaction with the VR system based on the VR device includes:

[0020] Based on the VR device, voice interaction is performed through an intercom microphone and a loudspeaker; wherein the intercom microphone is used to obtain the voice of the trainee and form operation instructions to be transmitted to the VR system to perform operations on the interactive page, and the loudspeaker is used to play the interactive sound in the VR system.

[0021] Preferably, the acquiring of the trainer's voice and forming an operation instruction to transmit to the VR system for the operation of the interactive page includes:

[0022] Acquire the voice instructions of the trainer and pre-process the audio information of the voice instructions;

[0023] Perform feature extraction on the preprocessed audio information, transform the voice instructions into text, and convert the words in the text into vectors to capture the semantic relationship between the word vectors to obtain the text instructions of the operation;

[0024] The text instructions of the operation are transmitted to the VR system, and the operation plan required by the training personnel is executed through the VR device.

[0025] Preferably, the preprocessing of the audio information of the voice command includes: noise reduction, frame division and normalization processing.

[0026] Preferably, the feature extraction of the preprocessed audio information, converting the voice instruction into text, converting the words in the text into vectors, capturing the semantic relationship between the word vectors, and obtaining the text instruction of the operation includes:

[0027] A pre-trained deep learning model is used to extract features from the pre-processed audio information, translate the audio information into text, and intelligently divide the text into simple vocabulary sentences. The divided text is segmented based on the features learned by the pre-trained deep learning model to obtain the meaning of the vocabulary. Based on the understood vocabulary meaning, the VR system performs corresponding related operations.

[0028] Preferably, the feature extraction of the pre-processed audio information based on the deep learning algorithm includes:

[0029] The formula for extracting features based on the deep learning algorithm is:

[0030]

[0031] Among them, x m (t) is the signal of the mth frame windowed, B is the length of the frame, k is the frequency index, e is the imaginary unit, X m (k) is the spectrum obtained for each frame.

[0032] Preferably, matching the recorded action posture with a preset standard action posture and scoring the completed action posture of each task includes:

[0033] Preprocessing the recorded image data of the action posture;

[0034] Perform feature extraction on the preprocessed image data to obtain the hand gestures and body postures of the trainees;

[0035] Based on gesture recognition technology, the gestures of the trainees are analyzed to determine whether their gestures are standard. The gestures are compared with the standard gestures in the database to obtain the gesture standardization degree P. s ;

[0036] Based on posture analysis technology, the body posture of the trainees is analyzed and compared with the standard body posture in the database to obtain the standard degree P of the body posture. z ;

[0037] According to the obtained standard degree P s and P z The data is scored.

[0038] Preferably, the gesture recognition technology includes:

[0039] For the obtained gesture posture, the shape of the gesture is extracted by using the edge contour extraction method, and then the extracted gesture is recognized by using the template matching method to obtain the gesture posture data;

[0040] The gesture posture data and the standard gesture action data are output as gesture feature vectors respectively, and the two are matched to obtain the matching similarity, which is the gesture standardization.

[0041] Preferably, the posture analysis technology includes:

[0042] Based on the obtained limb posture, a human posture model is established, and the specific posture of the trainee is obtained between key points through machine learning algorithms;

[0043] Match the specific posture obtained with the standard posture action to obtain the standard degree P of the limb posture z ;

[0044] Preferably, the standard degree P obtained according to s and P z The data is scored and the following is obtained:

[0045] F q =(P s +P z )*100%

[0046] In the formula, F q Provides a global posture matching score for trainees.

[0047] Preferably, the edge contour extraction method obtains the gesture shape based on vertical edge detection and horizontal edge detection, and the detection form is:

[0048]

[0049] In the formula, G x is the horizontal edge detection form, G y A form of vertical edge detection.

[0050] Preferably, the gesture posture data and the standard gesture action data are output as gesture feature vectors, respectively, represented as A(c1, c2, ... c n ) and B(d1, d2, ... d n ), and the two are matched using the following matching formula:

[0051]

[0052] Where P s is the standard degree of gesture, and n is the number of gesture features.

[0053] Preferably, the evaluating the current task operation capability according to the score includes:

[0054] According to the operational safety requirements of the substation, a score threshold for operation training is established; when the score corresponding to the trainee is greater than or equal to the score threshold, the current trainee's operation is considered qualified; when the score corresponding to the trainee is less than the score threshold, the current trainee's operation is considered unqualified and needs to be retrained;

[0055] An engineering task evaluation formula is established: R=α*Nv+β*Td, where R is the assessed safety risk value, α and β are weight coefficients, Nv is the number of illegal operations, and Td is the time spent in the danger zone; a safety assessment of user operations is performed based on the engineering task evaluation formula.

[0056] According to another aspect of the present invention, a substation safety training system based on gesture recognition and voice interaction is provided, comprising:

[0057] A data construction module, which is used to construct engineering task data corresponding to training in the VR system;

[0058] A human-computer interaction module, which is based on VR equipment and interacts with the VR system to perform hands-on simulation operations on the engineering task data;

[0059] A data recording module, which is used to record the action postures of the simulated operation process in real time;

[0060] An operation scoring module, which is used to match the recorded action posture with a preset standard action posture and score the completed action posture of each task;

[0061] Training result analysis module, which is used to evaluate the current task operation capability based on the scores.

[0062] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute any one of the methods described above in the present invention, or to run the system described above in the present invention.

[0063] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above in the present invention, or to run the system described above in the present invention.

[0064] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0065] The substation safety training method and system based on gesture recognition and voice interaction provided by the present invention realizes simulated on-site operation interaction based on VR virtual simulation technology, thereby increasing the training methods.

[0066] The substation safety training method and system based on gesture recognition and voice interaction provided by the present invention saves the cost of training by adopting a training method based on VR equipment, and can effectively simulate various complex environmental conditions, thereby bringing better training effects.

[0067] The substation safety training method and system based on gesture recognition and voice interaction provided by the present invention perform standardized scoring of technicians' gestures and postures based on gesture recognition, thereby ensuring the objectivity and consistency of the evaluation.

[0068] The substation safety training method and system based on gesture recognition and voice interaction provided in this application utilize 3D modeling technology to organize resources and build scenarios around key training methods, construct virtual scenarios, and simulate actual engineering task processes to conduct safety assessments on user operations. This enables timely feedback on trainees' performance during training, reduces training costs, and promotes innovation and collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0070] Figure 1 The present invention is a flowchart of a substation safety training method based on gesture recognition and voice interaction in one embodiment of the present invention.

[0071] Figure 2 The present invention is a schematic diagram of the working contents of a substation safety training method based on gesture recognition and voice interaction in a preferred embodiment of the present invention.

[0072] Figure 3 The figure is a schematic diagram of component modules of a substation safety training system based on gesture recognition and voice interaction in one embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0074] Existing substation safety training usually has the problem of being unable to fully and accurately simulate the operation of the substation and the lack of technical personnel's ability to deal with complex situations. To address this problem, an embodiment of the present invention provides a substation safety training method based on gesture recognition and voice interaction. The method is based on VR virtual simulation technology, generates corresponding engineering tasks for technical personnel, records the corresponding technical personnel's engineering task completion posture, matches the technical personnel's posture movements with standard posture movements, and intelligently scores each technician's task completion posture, realizing simulated on-site operation interaction, increasing training methods, and using VR equipment to save training costs. It can effectively simulate various complex environmental conditions and bring better training results.

[0075] Specifically, Figure 1 As shown, the substation safety training method based on gesture recognition and voice interaction provided in this embodiment may include the following operations:

[0076] S1, construct engineering task data corresponding to training in the VR system;

[0077] S2, based on VR equipment, interacts with the VR system and performs hands-on simulation operations on engineering task data;

[0078] S3, real-time recording of the action postures during the simulation operation process;

[0079] S4, matching the recorded action posture with the preset standard action posture, and scoring the completed action posture of each task;

[0080] S5, evaluate the current task operation capability based on the score.

[0081] The technical solutions provided by the above embodiments of the present invention are further described in detail below in conjunction with preferred embodiments.

[0082] like Figure 2 As shown, in this preferred embodiment, the substation safety training method based on gesture recognition and voice interaction provided by the above embodiment of the present invention includes the following steps:

[0083] S1, constructing engineering task data corresponding to the training in the VR system, which is used to train technicians based on VR virtual simulation technology;

[0084] S2, the technician wears the VR device within the VR system, operates the VR system based on voice interaction, and performs hands-on simulation operations on the given engineering tasks;

[0085] S3, during the simulation operation, the technician records the operation process in real time based on the camera, and records the corresponding operation posture, where the posture includes hand gestures and body movements;

[0086] S4, matching the recorded action posture with the preset standard action posture, and scoring the completed action posture of each task;

[0087] S5, based on the action posture scoring results, judge the current technician's operating ability and evaluate the current task operating ability.

[0088] In order to obtain a virtual scene that is highly similar to the real scene, in some preferred implementations, the above S1, constructing engineering task data corresponding to the training in the VR system, may further include the following operations:

[0089] Install necessary software and plug-ins in the VR system, configure the parameters of the training project, use the tools of the development platform to build the scenes of engineering task resources, construct a virtual VR engineering scene, and present the visual information of technicians based on VR equipment.

[0090] Furthermore, in some preferred implementations, the above S1, installing necessary software and plug-ins in the VR system, configuring parameters of the training project, using the tools of the development platform to build scenes of engineering task resources, and constructing a virtual VR engineering scene, may further include the following operations:

[0091] Determine the goals to be achieved in this VR training, conduct a comprehensive analysis of related engineering tasks, and sort out the required engineering task resources, including: engineering drawings, equipment parameters, operating manuals and safety standards;

[0092] According to the training objectives and engineering task resources, set various parameters of the scene in the VR system, including: lighting conditions, weather conditions and time changes;

[0093] Using 3D modeling technology, based on the acquired engineering task resources, engineering equipment in the virtual scene is constructed, and the constructed engineering equipment is placed in the virtual scene according to the actual layout to construct a virtual VR engineering scene; further, the constructed engineering equipment is placed in the virtual scene according to the actual layout, including: the engineering equipment has a coordinate (x', y', z') in the three-dimensional space and is referenced and positioned based on the following formula, A = (x'-x1', y'-y1', z'-z1') where (x1', y1', z1') is the reference coordinate point of the engineering equipment in the three-dimensional coordinate, which is used as the reference coordinate for positioning and substituted into the three-dimensional coordinate in the virtual scene;

[0094] The actual engineering task process is simulated in a virtual scene, including equipment operation, construction steps and safety inspection, to obtain the engineering task data for the corresponding training.

[0095] In some preferred implementations, the above S2, operating and interacting with the VR system through voice interaction, may further include the following operations:

[0096] An intercom microphone and a loudspeaker are used to realize voice interaction. The intercom microphone is used to obtain the voice of the technician, and the loudspeaker is used to play the interactive sound in the VR system. The technician operates the interactive page of the VR system through intercom.

[0097] In order to perform voice interaction with the VR system, in this preferred embodiment, it is further preferred that the technician operates the interactive page of the VR system by means of intercom, and the following operations may be further included:

[0098] S21, uploading the voice command spoken by the technician through the intercom microphone, and pre-processing the uploaded voice command audio;

[0099] S22, extracting features from the preprocessed voice command audio, transforming the voice into text, converting words in the text into vectors, capturing semantic relationships between words, and obtaining text instructions for the operation;

[0100] S23, transmitting the operation instructions to the VR system, so that the VR device executes the operation plan desired by the technician.

[0101] In some preferred implementations, further preferably, the above S21, pre-processing the uploaded voice audio, may further include: noise reduction, frame division and normalization processing.

[0102] In order to provide accurate interactive instruction information, in some preferred embodiments, it is further preferred that the above S22, extracting features from the preprocessed speech audio, converting the speech into text, converting the words in the text into vectors, capturing the semantic relationship between the words, and obtaining the text instructions for the operation, may further include:

[0103] Through natural language processing, based on a pre-trained deep learning model, the pre-processed speech audio is subjected to feature extraction, the speech is translated into text, the text is intelligently divided, and the text is divided into simple vocabulary sentences. Based on the features learned by the pre-trained deep learning model, the divided files are segmented to obtain the meaning of the vocabulary. Based on the understood vocabulary meaning, the VR system performs corresponding related operations.

[0104] In some preferred implementations, feature extraction of the preprocessed speech audio based on a deep learning algorithm may further include:

[0105] The formula for extracting features based on the deep learning algorithm is:

[0106]

[0107] Among them, x m (t) is the signal of the mth frame windowed, B is the length of the frame, k is the frequency index, e is the imaginary unit, X m (k) is the spectrum obtained for each frame.

[0108] In some preferred implementations, the above S4, matching the recorded action posture with the preset standard action posture and scoring the completed action posture of each task, may further include the following operations:

[0109] S41, uploading the recorded action posture image data, and preprocessing the image data, the preprocessing may preferably include: filtering, normalization and image enhancement processing;

[0110] S42, extracting features from the processed image data, where the extracted features are the technician's hand gestures and body postures;

[0111] S43, based on gesture recognition technology, analyze the technician's gestures, determine whether the technician's gestures are standard, and compare them with the standard gestures in the database to obtain the gesture standardization degree P s ,

[0112] S44, based on the posture analysis technology, analyze and compare the technician's body posture with the standard body posture in the database to obtain the standard degree P of the body posture z , score according to the obtained standard data.

[0113] In some preferred implementations, the above S43, gesture recognition technology, may further include the following operations:

[0114] The gesture part in the image is segmented in advance, the shape of the gesture is extracted by edge contour extraction method, and then the extracted gesture is recognized by template matching method to obtain gesture data, where the template is the standard gesture action in the database, and the extracted features are compared and matched based on the similarity, and the gesture feature image is output as a gesture feature vector, which is expressed as A(c1, c2, ... c n ) and B(d1, d2, ... d n ), match it using the following formula;

[0115]

[0116] In the formula, p s is the matching similarity, n is the number of gesture features, p s The smaller the result value is, the more similar the gesture made by the technician is to the standard gesture in the database. s The larger the result, the less similar the gesture made by the technician is to the standard gesture in the database;

[0117] Among them, the edge contour extraction method obtains the gesture shape based on vertical edge detection and horizontal edge detection. The detection form is:

[0118]

[0119]

[0120] In the formula, G x is the horizontal edge detection form, G y A form of vertical edge detection.

[0121] In some preferred implementations, the above S44, posture analysis technology, may further include the following operations:

[0122] The body posture features of the technicians are extracted to establish a human posture model. Through the machine learning algorithm, the specific posture of the technicians is obtained between the key points. The posture obtained is matched with the posture standard in the database. The matching formula is equal to the hand gesture matching formula to obtain the posture standard P.z .

[0123] Scoring is performed based on the matching standard. The scoring formula is:

[0124] F q =(P s +P z )*100%

[0125] where F q is the technician’s global posture matching score, P s is the gesture standard, P z The standard degree of posture is used to obtain the score of current technician training.

[0126] In some preferred implementations, the above S5, judging the operation ability of the current technician according to the action posture scoring result, may further include the following operations:

[0127] According to the basic operational safety requirements of the substation, a score threshold for operation training is established. If the score corresponding to the technician is greater than the score threshold, the current technician's operation is judged to be qualified. If the score corresponding to the technician is less than the score threshold, the current technician's operation is judged to be unqualified and re-training is required.

[0128] In some preferred implementations, the above S5, evaluating the current task operation capability, may further include the following operations:

[0129] Establish an engineering task evaluation formula: R=α*Nv+β*Td, where R is the assessed safety risk value, α and β are weight coefficients, Nv is the number of illegal operations, and Td is the time spent in the dangerous area; perform a safety assessment of user operations based on the engineering task evaluation formula.

[0130] Based on the same inventive concept, an embodiment of the present invention further provides a substation safety training system based on gesture recognition and voice interaction.

[0131] Specifically, Figure 3 As shown, the substation safety training system based on gesture recognition and voice interaction provided by this embodiment may include the following modules:

[0132] A data construction module, which is used to construct engineering task data corresponding to training in the VR system;

[0133] Human-computer interaction module: This module is based on VR equipment and interacts with the VR system through voice interaction, and performs hands-on simulation operations on engineering task data;

[0134] Data recording module, which is based on a camera and records the action postures of the simulated operation process in real time;

[0135] An operation scoring module, which is used to match the recorded action posture with the preset standard action posture and score the completed action posture of each task;

[0136] The training result analysis module is used to judge the operational capability of the current technicians based on the action posture scoring results.

[0137] The following further describes the working contents of each functional module in the substation safety training system based on gesture recognition and voice interaction provided by the above embodiment of the present invention.

[0138] The data construction module is used to construct training engineering task data in the VR system. By installing the necessary software and plug-ins in the VR system in advance and configuring the project settings, the tools of the development platform are used to build the scenes of engineering task resources, construct a virtual VR engineering scene, and present the visual information of technicians based on VR equipment.

[0139] Human-computer interaction module: This module uses voice interaction technology to perform human-computer interaction through an intercom microphone and a speaker. The intercom microphone is used to obtain the technician's voice, and the speaker is used to play the interactive sound in the VR system. The technician operates the interactive page of the VR system through intercom. The technician performs simulated operations according to the engineering task scenario simulated by VR. Among them:

[0140] The voice interaction method is based on natural speech processing. The voice spoken by the technician is uploaded through the intercom microphone, the uploaded voice audio is pre-processed, the voice is converted into text, the words in the text are converted into vectors, the semantic relationship between the words is captured, the text instructions of the operation are obtained, the operation instructions are transmitted, and the VR device executes the operation plan desired by the technician.

[0141] Audio preprocessing includes noise reduction, frame processing, normalization and feature extraction. After the processing is completed, the extracted features are processed based on the deep learning algorithm, the audio is translated into text, the text is intelligently divided, and the file is segmented based on the features learned by the deep learning algorithm to obtain the meaning of the vocabulary. Based on the understood vocabulary meaning, the VR system performs corresponding related operations;

[0142] The feature extraction formula of the deep learning algorithm is:

[0143]

[0144] where x m(t) is the signal of the mth frame windowed, B is the length of the frame, k is the frequency index, e is the imaginary unit, X m (k) is the spectrum obtained for each frame.

[0145] In the operation scoring module, when the technician performs hands-on simulation operation, the camera acquires the technician's posture image data in real time, uploads the acquired posture image data, pre-processes the image data, including filtering, normalization and image enhancement, and then extracts features from the processed image. The extracted features are the technician's gestures and body postures. The gestures of the technician are analyzed based on gesture recognition technology to determine whether the technician's action posture is standard, and compared with the standard gesture posture in the database to obtain the gesture standardization degree P. s Similarly, based on the posture analysis technology, the posture of the technicians is analyzed and compared to obtain the standard degree of posture P z , score according to the obtained standard data. Among them:

[0146] Gesture recognition pre-segments the gesture part in the image, uses the edge contour extraction method to extract the shape of the gesture, and finally uses the template matching method to recognize the extracted gesture to obtain gesture data, where the template is the standard gesture action in the database. Based on the extracted features, the gesture feature image is compared and matched for similarity, and the gesture feature image is output as a gesture feature vector, which is expressed as A(c1, c2, ... c n ) and B(d1, d2, ... d n ), match it using the following formula;

[0147]

[0148] Among them, p s is the matching similarity, n is the number of gesture features, p s The smaller the result value is, the more similar the gesture made by the technician is to the standard gesture in the database. s The larger the result, the less similar the gesture performed by the technician is to the standard gesture in the database.

[0149] The edge contour extraction method obtains the gesture shape based on vertical edge detection and horizontal edge detection. The detection form is:

[0150]

[0151] Among them, G x is the horizontal edge detection form, G y A form of vertical edge detection.

[0152] Posture analysis technology extracts the posture features of technicians and establishes a human posture model. Through machine learning algorithms, the specific posture of technicians is obtained between key points. The obtained posture is matched with the posture standard in the database. The matching formula is equal to the hand gesture matching formula to obtain the posture standard P. z .

[0153] Scoring is performed based on the matching standard. The scoring formula is:

[0154] F q =(P s +P z )*100%

[0155] Among them, F q is the technician’s global posture matching score, P s is the gesture standard, P z The standard degree of posture is used to obtain the score of current technician training.

[0156] Training result analysis module, which is used to establish a score threshold for operation training according to the basic operation safety requirements of the substation. If the score corresponding to the technician is greater than the score threshold, the current technician's operation is considered qualified. If the score corresponding to the technician is less than the score threshold, the current technician's operation is considered unqualified and needs to be retrained. Establish an engineering task evaluation formula: R = α*Nv+β*Td, where R is the assessed safety risk value, α and β are weight coefficients, Nv is the number of illegal operations, and Td is the time spent in the dangerous area; perform a safety assessment of user operations based on the engineering task evaluation formula.

[0157] It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding components in the system, and those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, which will not be elaborated here.

[0158] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0159] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0160] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0161] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0162] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0163] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0164] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0165] The substation safety training method and system based on gesture recognition and voice interaction provided by the above-mentioned embodiments of the present invention realizes simulated on-site operation interaction based on VR virtual simulation technology, and increases the training methods; by adopting a training method based on VR equipment, the cost of training is saved, and various complex environmental conditions can be effectively simulated, bringing better training results; based on the gesture recognition method, the gestures and postures of the technicians are standardized and scored, ensuring the objectivity and consistency of the evaluation.

[0166] All matters not covered in the above embodiments of the present invention are well known in the art.

[0167] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A substation safety training method based on gesture recognition and voice interaction, characterized in that: include: Construct engineering task data corresponding to training in the VR system; Based on VR equipment, operate and interact with the VR system, and perform hands-on simulation operations on the engineering task data; Record the action posture of the simulated operation process in real time; Matching the recorded action posture with a preset standard action posture, and scoring the completed action posture of each task; Evaluate the current task operation capability based on the score.

2. The substation safety training method based on gesture recognition and voice interaction according to claim 1 is characterized in that: The construction of engineering task data corresponding to the training in the VR system includes: Configure the parameters of the training project in the VR system, build the scene of the engineering task resources, construct a virtual VR engineering scene, and obtain the engineering task data corresponding to the training as the visual information for training presented based on the VR device; wherein: Determine the goals to be achieved in this VR training, conduct a comprehensive analysis of related engineering tasks, and sort out the required engineering task resources, including: engineering drawings, equipment parameters, operating manuals and safety standards; According to the training objectives and engineering task resources, set various parameters of the virtual scene in the VR system, including: lighting conditions, weather conditions and time changes; Using 3D modeling technology, based on the acquired engineering task resources, the engineering equipment in the virtual scene is constructed, and the engineering equipment is placed in the virtual scene according to the actual layout; wherein the engineering equipment has a coordinate (x', y', z') in the three-dimensional space, which is referenced and positioned based on A = (x'-x1', y'-y1', z'-z1'), wherein (x1', y1', z1') is the reference coordinate point of the engineering equipment in the three-dimensional coordinate, and the coordinate point is used as the reference coordinate for positioning, and is substituted into the three-dimensional coordinate in the virtual scene to construct a virtual VR engineering scene; The actual engineering task process is simulated in a virtual scene, including equipment operation, construction steps and safety inspections, to obtain the engineering task data for the corresponding training.

3. The substation safety training method based on gesture recognition and voice interaction according to claim 1 is characterized in that: The operation interaction with the VR system based on the VR device includes: Based on the VR device, voice interaction is performed through an intercom microphone and a loudspeaker; wherein the intercom microphone is used to obtain the voice of the trainee and form operation instructions to be transmitted to the VR system to perform operations on the interactive page, and the loudspeaker is used to play the interactive sound in the VR system.

4. The substation safety training method based on gesture recognition and voice interaction according to claim 3 is characterized in that: The acquisition of the trainer's voice and the formation of operation instructions to be transmitted to the VR system for the operation of the interactive page include: Acquire the voice instructions of the trainer and pre-process the audio information of the voice instructions; Perform feature extraction on the preprocessed audio information, transform the voice instructions into text, and convert the words in the text into vectors to capture the semantic relationship between the word vectors to obtain the text instructions of the operation; The text instructions of the operation are transmitted to the VR system, and the operation plan required by the training personnel is executed through the VR device.

5. The substation safety training method based on gesture recognition and voice interaction according to claim 4 is characterized in that: Also includes any one or more of the following: - the preprocessing of the audio information of the voice command includes: noise reduction, frame division and normalization processing; - The feature extraction of the pre-processed audio information, the conversion of the voice instruction into text form, the conversion of the words in the text into vectors, the capture of the semantic relationship between the word vectors, and the acquisition of the text instruction of the operation include: A pre-trained deep learning model is used to extract features from the pre-processed audio information, the audio information is translated into text, the text is divided into vocabulary short sentences, and the segmentation of the divided text is performed based on the features learned by the pre-trained deep learning model to obtain the meaning of the vocabulary, and based on the understood vocabulary meaning, the VR system performs corresponding related operations; -The feature extraction of the pre-processed audio information based on the deep learning algorithm includes: The formula for extracting features based on the deep learning algorithm is: Among them, x m (t) is the signal of the mth frame windowed, B is the length of the frame, k is the frequency index, e is the imaginary unit, X m (k) is the spectrum obtained for each frame.

6. The substation safety training method based on gesture recognition and voice interaction according to claim 1 is characterized in that: The recorded action posture is matched with a preset standard action posture, and scoring is performed for each task completion action posture, including: Preprocessing the recorded image data of the action posture; Perform feature extraction on the preprocessed image data to obtain the hand gestures and body postures of the trainees; Based on gesture recognition technology, the gestures of the trainees are analyzed to determine whether their gestures are standard. The gestures are compared with the standard gestures in the database to obtain the gesture standardization degree P. s ; Based on posture analysis technology, the body posture of the trainees is analyzed and compared with the standard body posture in the database to obtain the standard degree P of the body posture. z ; According to the obtained standard degree P s and P z The data is scored.

7. The substation safety training method based on gesture recognition and voice interaction according to claim 6 is characterized in that: Also includes any one or more of the following: -The gesture recognition technology includes: For the obtained gesture posture, the shape of the gesture is extracted by using the edge contour extraction method, and then the extracted gesture is recognized by using the template matching method to obtain the gesture posture data; The gesture posture data and the standard gesture action data are output as gesture feature vectors respectively, and the two are matched to obtain the matching similarity, which is the gesture standardization; -The posture analysis technology includes: Based on the obtained limb posture, a human posture model is established, and the specific posture of the trainee is obtained between key points through machine learning algorithms; Match the specific posture obtained with the standard posture action to obtain the standard degree P of the limb posture z ; - The standard degree P obtained according to s and P z The data is scored and we get: F q =(P s +P z )*100% In the formula, F q Provides a global posture matching score for trainees.

8. The substation safety training method based on gesture recognition and voice interaction according to claim 7 is characterized in that: Also includes any one or more of the following: -The edge contour extraction method obtains the gesture shape based on vertical edge detection and horizontal edge detection. The detection form is: In the formula, G x is the horizontal edge detection form, G y It is a vertical edge detection form; - The gesture posture data and the standard gesture action data are output as gesture feature vectors, expressed as A(c1, c2, ...c n ) and B(d1, d2, ... d n ), and the two are matched using the following matching formula: Where P s is the standard degree of gesture, and n is the number of gesture features.

9. The substation safety training method based on gesture recognition and voice interaction according to claim 1 is characterized in that: The evaluation of the current task operation capability according to the score includes: According to the operational safety requirements of the substation, a score threshold for operation training is established; when the score corresponding to the trainee is greater than or equal to the score threshold, the current trainee's operation is considered qualified; when the score corresponding to the trainee is less than the score threshold, the current trainee's operation is considered unqualified and needs to be retrained; An engineering task evaluation formula is established: R=α*Nv+β*Td, where R is the assessed safety risk value, α and β are weight coefficients, Nv is the number of illegal operations, and Td is the time spent in the danger zone; a safety assessment of user operations is performed based on the engineering task evaluation formula.

10. A substation safety training system based on gesture recognition and voice interaction, characterized in that: include: A data construction module, which is used to construct engineering task data corresponding to training in the VR system; A human-computer interaction module, which is based on VR equipment and interacts with the VR system to perform hands-on simulation operations on the engineering task data; A data recording module, which is used to record the action postures of the simulated operation process in real time; An operation scoring module, which is used to match the recorded action posture with a preset standard action posture and score the completed action posture of each task; Training result analysis module, which is used to evaluate the current task operation capability based on the scores.

11. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 9, or to run the system described in claim 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 9, or to run the system described in claim 10.

Images