Key video information extraction system of relay protection interactive guidance AR glasses

Abstract

The invention discloses a key video information extraction system for relay protection interactive guidance AR glasses, and relates to the technical field of AR glasses, and the key video information extraction system comprises an AR glasses system which is used for collecting on-site video data and transmitting the on-site video data to an operation modeling and optimization server; the target operation terminal is used for capturing guidance gesture data and transmitting the guidance gesture data to the operation modeling and optimizing server; and the operation modeling and optimizing server is used for extracting an operation object set based on the current work ticket, determining key video information based on the field video data, the operation object set and the guidance gesture data, and adding the key video information into a preset reference video library. According to the invention, the technical problem that follow-up operation cannot be effectively guided due to the fact that reference data cannot be rapidly and effectively extracted in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of AR glasses, and in particular to a key video information extraction system for relay protection interactive guidance AR glasses. Background Art

[0002] Currently, for the maintenance of relay protection in substations, maintenance personnel can use AR (Augmented Reality) glasses to perform maintenance. When maintenance personnel encounter problems during maintenance, they can refer to historical data for maintenance.

[0003] In the related technology, a large amount of analysis of historical inspection and operation data is required to build a model for the site, and then extract reference data based on the established model. These reference data use text and pictures as the basis for matching the inspection operation steps. Not only is the workload large, but it is also difficult to match the text and pictures to the work content and tasks in the current inspection operation, and it is impossible to effectively guide subsequent operations. In addition, before building the model, it is necessary to manually annotate the operation steps corresponding to the text and pictures, which is a large workload.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] An embodiment of the present invention provides a key video information extraction system for relay protection interactive guidance AR glasses, so as to at least solve the technical problem in the related art that reference data cannot be extracted quickly and effectively, resulting in the inability to effectively guide subsequent operations.

[0006] According to one aspect of an embodiment of the present invention, a key video information extraction system for relay protection interactive guidance AR glasses is provided, and the key video information extraction system includes: an AR glasses system, used to collect on-site video data, and transmit the on-site video data to a job modeling and optimization server, wherein the on-site video data is data collected from the relay protection maintenance site; a target operation terminal, used to capture guidance gesture data, and transmit the guidance gesture data to the job modeling and optimization server, wherein the guidance gesture data is data for maintenance guidance of maintenance equipment; the job modeling and optimization server, based on a current work ticket, extracts a set of job objects, and based on the on-site video data, the set of job objects and the guidance gesture data, determines key video information, and adds the key video information to a preset reference video library, wherein the key video information includes key videos associated with the job objects.

[0007] Furthermore, the AR glasses system includes: a camera module for collecting the on-site video data; a data transmission module for transmitting the on-site video data to the job modeling and optimization server and the target job terminal; a display module for displaying an interactive video or a historical reference video, wherein the interactive video is a video generated based on the on-site video data and the guiding gesture data, and the historical reference video is a key video matched from the preset reference video library, and the historical reference video is used to provide a reference for the maintenance object wearing AR glasses to perform relay protection operations.

[0008] Furthermore, the target operation terminal includes: a touch screen for displaying the on-site video data; and a motion capture module for capturing the guiding gesture data, wherein the guiding gesture data is data generated based on the on-site video data.

[0009] Furthermore, the job modeling and optimization server includes: a job analysis module, which is used to extract the job object in the work ticket and the job content corresponding to the job object, wherein the work ticket is a specification document of the maintenance steps of the pre-planned maintenance job, and each of the maintenance steps includes a maintenance operation on the maintenance equipment; a job interaction analysis module, which is used to determine the key video based on the on-site video data, the guidance gesture data and the job object; a storage module, which is used to use the job object and the job content corresponding to the job object as the retrieval index of the key video, and associate the retrieval index with the key video for storage; a model building and optimization module, which is used to count the number of times the retrieval index appears, and based on the number of times each of the retrieval index appears, assign a weight to the key video associated with the retrieval index.

[0010] Furthermore, the extraction of the job object in the work ticket and the job content corresponding to the job object includes: extracting the equipment noun in the work ticket and representing the equipment noun as the job object; extracting the operation verb in the work ticket and representing the operation verb as the job content; identifying the equipment noun corresponding to the operation verb, and associating the job object represented by the equipment noun with the job content represented by the operation verb.

[0011] Furthermore, determining the key video based on the on-site video data, the guiding gesture data and the work object includes: obtaining an interactive video generated by fusing the on-site video data and the guiding gesture data, wherein an interactive area of ​​the interactive video has a guiding gesture; determining a key area based on the interactive area; and when the work object is located in the key area, capturing a sub-video containing the guiding gesture from the interactive video, and using the sub-video as the key video of the work object.

[0012] According to another aspect of an embodiment of the present invention, a guidance method for a key video information extraction system using the above-mentioned relay protection interactive guidance AR glasses is also provided, including: when performing relay protection operations, extracting a set of operation objects of a current work ticket and the operation content corresponding to each operation object in the set of operation objects to obtain a retrieval index set; when receiving a guidance request, determining a target retrieval index from the retrieval index set based on the current operation step, wherein the current operation step includes equipment to be repaired and operations to be repaired; based on the target retrieval index, matching a target reference video from a preset reference video library; and completing the current operation step based on the target reference video.

[0013] Furthermore, the guidance method also includes: in a case where the current operation step is not completed based on the target reference video, issuing an interactive request to the target operation terminal, wherein the target operation terminal captures guidance gesture data based on the interactive request; receiving an interactive video generated by the guidance gesture data and the on-site video data indicated by the current operation step, and completing the current operation step based on the interactive video.

[0014] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned guiding methods.

[0015] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned guiding methods.

[0016] In the present invention, the key video information extraction system includes: an AR glasses system, which is used to collect on-site video data and transmit the on-site video data to the job modeling and optimization server; a target job terminal, which is used to capture guidance gesture data and transmit the guidance gesture data to the job modeling and optimization server; the job modeling and optimization server, based on the current work ticket, extracts the job object set, and based on the on-site video data, the job object set and the guidance gesture data, determines the key video information, and adds the key video information to the preset reference video library.

[0017] In the present invention, on-site video data can be collected through the AR glasses system in the key video information extraction system, and then the guidance gesture data generated according to the on-site video data can be obtained through the target operation terminal in the key video information extraction system. At the same time, the job modeling and optimization server in the key video information extraction system can extract the job object set from the current work ticket, so that the job modeling and optimization server can extract key video information from the on-site video data according to the on-site video data, the job object set and the guidance gesture data, so as to facilitate the subsequent guidance of relay protection operations. Not only can the key video information be extracted quickly and effectively, but also the subsequent guidance efficiency can be improved by using the job object as the matching basis for the key video, thereby solving the technical problem in the related technology that reference data cannot be extracted quickly and effectively, resulting in the inability to effectively guide subsequent operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a schematic diagram of an optional key video information extraction system for relay protection interactive guidance AR glasses according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of an optional interactive video screenshot according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of an optional key video information extraction system according to an embodiment of the present invention;

[0022] Figure 4 is a flow chart of an optional guidance method according to an embodiment of the present invention;

[0023] Figure 5 It is a hardware structure block diagram of an electronic device (or mobile device) for guiding the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] The present invention proposes a relay protection interactive guidance AR glasses key video information extraction system, which does not require modeling of field data in advance, and can avoid the work of sorting and analyzing historical data. In addition, the present invention uses the work ticket as the clustering basis for the work content, and can achieve the establishment of a key data model for relay protection operations with zero prior knowledge, avoiding manual calibration work during the model establishment process. In addition, based on the work ticket, the present invention can save the interactive process (i.e., key video information) between experts and on-site operators (i.e., maintenance objects) working in the core area, which not only avoids the workload of data input, but also saves the core important link data to facilitate subsequent guidance of on-site maintenance operations.

[0027] The present invention is described in detail below in conjunction with various embodiments.

[0028] Embodiment 1

[0029] According to an embodiment of the present invention, an embodiment of a key video information extraction system for relay protection interactive guidance AR glasses is provided.

[0030] Figure 1 is a schematic diagram of an optional key video information extraction system for relay protection interactive guidance AR glasses according to an embodiment of the present invention, such as Figure 1 As shown, the key video information extraction system of relay protection interactive guidance AR glasses includes: AR glasses system, target operation terminal and operation modeling and optimization server.

[0031] The AR glasses system is used to collect on-site video data and transmit the on-site video data to the operation modeling and optimization server, wherein the on-site video data is data collected from the relay protection maintenance site;

[0032] A target operation terminal is used to capture the guidance gesture data and transmit the guidance gesture data to the operation modeling and optimization server, wherein the guidance gesture data is data for providing maintenance guidance for the maintenance equipment;

[0033] The job modeling and optimization server extracts a set of job objects based on the current work ticket, and determines key video information based on on-site video data, the set of job objects, and guidance gesture data, and adds the key video information to a preset reference video library, wherein the key video information includes key videos associated with the job objects.

[0034] In an embodiment of the present invention, on-site video data can be collected through the AR glasses system in the key video information extraction system, and then the guidance gesture data generated according to the on-site video data can be obtained through the target operation terminal in the key video information extraction system. At the same time, the job modeling and optimization server in the key video information extraction system can extract the job object set from the current work ticket, so that the job modeling and optimization server can extract key video information from the on-site video data according to the on-site video data, the job object set and the guidance gesture data, so as to facilitate the subsequent guidance of relay protection operations. Not only can the key video information be extracted quickly and effectively, but also the subsequent guidance efficiency can be improved by using the job object as the matching basis for the key video, thereby solving the technical problem in the related technology that reference data cannot be extracted quickly and effectively, resulting in the inability to effectively guide subsequent operations.

[0035] In an embodiment of the present invention, the maintenance object can wear AR glasses to collect on-site video data (i.e., real-time video data collected from the relay protection maintenance site), and these on-site video data can be displayed in real time on the target operation terminal (i.e., the expert operation terminal), so that the background experts can provide real-time guidance based on these on-site video data through the target operation terminal to generate guidance gesture data (i.e., data for maintenance guidance of the maintenance equipment). At the same time, the on-site video data and the guidance gesture data can be transmitted to the operation modeling and optimization server, so that the operation modeling and optimization server can extract key video information (i.e., including key videos associated with the operation object) from the on-site video data. In this way, the construction of the preset reference video library can be completed through the key video information extracted from each maintenance operation, so that when problems are encountered in subsequent maintenance operations, the matching key videos can be called from the preset reference video library in time, thereby providing a reference for the maintenance operation.

[0036] In an embodiment of the present invention, the current work ticket of the current relay protection maintenance operation can be transmitted to the job modeling and optimization server in advance, so that the job modeling and optimization server can extract the job object set (i.e., the maintenance equipment identified from the work ticket) from the current work ticket, and determine the key video information based on the on-site video data, the job object set and the guidance gesture data, and then add the key video information to the preset reference video library, so that subsequent operations can match the optimal video from the preset reference video library as a reference video for job guidance.

[0037] In the embodiment of the present invention, the work ticket is a specification document of the maintenance steps of the pre-planned maintenance work, for example, the first step is to perform A operation on device A, the second step is to perform B operation on device B, and so on.

[0038] In some optional embodiments, the maintenance object may be an on-site worker, a robot, or mechanical equipment.

[0039] Optionally, the AR glasses system includes: a camera module for collecting on-site video data; a data transmission module for transmitting on-site video data to an operation modeling and optimization server and a target operation terminal; a display module for displaying interactive videos or historical reference videos, wherein the interactive video is a video generated based on on-site video data and guidance gesture data, and the historical reference video is a key video matched from a preset reference video library, and the historical reference video is used to provide a reference for the maintenance object wearing AR glasses to perform relay protection operations.

[0040] In an embodiment of the present invention, the AR glasses system includes: a camera module, a data transmission module and a display module.

[0041] In this embodiment, the camera module can obtain real-time video data (ie, on-site video data) of the relay protection maintenance site.

[0042] In this embodiment, the data transmission module can send the on-site video data to the job modeling and optimization server and the target job terminal through a 5G or wireless network, so that the target job terminal can display the on-site video data, generate guidance gesture data based on the on-site video data, and enable the job modeling and optimization server to extract key video information from the on-site video data based on the guidance gesture data.

[0043] In this embodiment, the display module can display interactive videos generated based on on-site video data and guidance gesture data (i.e., videos that integrate expert guidance gestures). It can also display historical reference videos (i.e., key videos matched from a preset reference video library) when the maintenance object needs it (i.e., when the maintenance object initiates a call request through AR glasses). This provides a reference for the maintenance object wearing AR glasses to perform relay protection operations, which can effectively reduce the real-time guidance services of background experts.

[0044] Optionally, the target operation terminal includes: a touch screen for displaying on-site video data; and a motion capture module for capturing guidance gesture data, wherein the guidance gesture data is data generated based on the on-site video data.

[0045] In the embodiment of the present invention, the target operation terminal includes: a touch screen and a motion capture module.

[0046] In this embodiment, the touch screen is used to display on-site video data so that back-end experts can provide interactive guidance based on the real-time on-site video displayed through the touch screen, thereby simultaneously displaying the background real-time fused interactive video on the touch screen and the display module of the AR glasses worn by the maintenance object.

[0047] Figure 2 is a schematic diagram of an optional interactive video screenshot according to an embodiment of the present invention, such as Figure 2 As shown, a screenshot of an interactive video is shown, which integrates the guidance gestures of the background expert into the on-site video of the relay protection operation. The on-site video of the relay protection operation includes the gestures of the on-site maintenance object. Figure 2 The two hands in the picture come from the expert and the on-site maintenance object respectively.

[0048] In this embodiment, the motion capture module can extract the expert's guiding gesture from the background to obtain guiding gesture data, which is data generated by the expert's maintenance guidance of the maintenance equipment displayed by the real-time on-site video data. Optionally, to ensure the extraction effect, the background can be a green table mat.

[0049] In some optional embodiments, the target operation terminal may also call the operation modeling and optimization server as needed to match the corresponding key video from the preset reference video library as a maintenance reference.

[0050] Optionally, the job modeling and optimization server includes: a job analysis module, which is used to extract the job objects in the work ticket and the job content corresponding to the job objects, wherein the work ticket is a specification document of the maintenance steps of the pre-planned maintenance job, and each maintenance step includes the maintenance operation of the maintenance equipment; a job interaction analysis module, which is used to determine the key video based on the on-site video data, the guidance gesture data and the job object; a storage module, which is used to use the job object and the job content corresponding to the job object as the retrieval index of the key video, and associate the retrieval index with the key video for storage; a model building and optimization module, which is used to count the number of times the retrieval index appears, and based on the number of times each retrieval index appears, assign weights to the key videos associated with the retrieval index.

[0051] In an embodiment of the present invention, the job modeling and optimization server includes: a job analysis module, a job interaction analysis module, a storage module, and a model building and optimization module.

[0052] In this embodiment, the job analysis module can extract the subject noun (i.e., maintenance equipment) of the work ticket (also known as the operation ticket, which is a standard document of the maintenance steps of the pre-planned maintenance work, and each maintenance step includes a maintenance operation on the maintenance equipment) as the job object, and extract the verb (i.e., the maintenance operation) as the job content (i.e., the job analysis module is used to extract the job object in the work ticket and the job content corresponding to the job object).

[0053] Optionally, the job object in the work ticket and the job content corresponding to the job object are extracted, including: extracting the equipment nouns in the work ticket and representing the equipment nouns as the job objects; extracting the operation verbs in the work ticket and representing the operation verbs as the job content; identifying the equipment nouns corresponding to the operation verbs, and associating the job objects represented by the equipment nouns with the job content represented by the operation verbs.

[0054] In the embodiment of the present invention, the job analysis module can extract the equipment noun (i.e., the equipment name, for example, a component on the edge device) in the work ticket by identifying the work ticket, so as to characterize the equipment noun as the job object, and can extract the operation verb (for example, how many degrees to rotate) in the work ticket, and characterize the operation verb as the job content. At the same time, the equipment noun corresponding to the operation verb can also be identified, and the job object represented by the equipment noun is associated with the job content represented by the operation verb, so as to use the associated job object and job content as a dual index, so as to more accurately match the key video.

[0055] In this embodiment, the work interaction analysis module can match the guidance area of ​​the interactive video with the object area where the work object is located. If the object area is within the guidance area, the key video information can be determined (that is, the work interaction analysis module is used to determine the key video based on the on-site video data, guidance gesture data and work object).

[0056] Optionally, based on the on-site video data, the guidance gesture data and the work object, a key video is determined, including: obtaining an interactive video generated by fusing the on-site video data and the guidance gesture data, wherein an interactive area of ​​the interactive video has a guidance gesture; based on the interactive area, a key area is determined; when the work object is located in the key area, a sub-video containing the guidance gesture is captured from the interactive video, and the sub-video is used as the key video of the work object.

[0057] In an embodiment of the present invention, the background can fuse the on-site video data and the guidance gesture data to generate an interactive video, in which the interactive area of ​​the interactive video has a guidance gesture (that is, the area containing the guidance gesture is the interactive area), and then the operation interaction analysis module can determine the key area according to the interactive area, for example, with the interactive area as the center, expand outward a preset range (for example, 1 cm) to obtain the key area. If the operation object is located in the key area, the sub-video containing the guidance gesture is captured from the interactive video, and the sub-video is used as the key video of the operation object. In this way, only the video of the expert guiding the maintenance object can be retained, which not only reduces the storage space, but also effectively improves the subsequent guidance efficiency.

[0058] In some optional embodiments, the object area where the operation object is located can also be determined, and an overlap condition can be set (i.e., the overlap area of ​​the guiding gesture in the object area is larger than a preset area threshold (for example, 30% of the area of ​​the object area)). When the overlap condition is met, the sub-video that meets the overlap condition is captured as the key video of the operation object.

[0059] In this embodiment, the storage module can use the job object and the job content as dual indexes to save the key video information (i.e., the storage module is used to use the job object and the job content corresponding to the job object as the retrieval index of the key video, and associate the retrieval index with the key video and then store it).

[0060] In this embodiment, the model building and optimization module assigns higher weights to high-frequency problems based on probability statistics on the basis of data storage, that is, the model building and optimization module can count the number of times the retrieval index appears, and assign weights to key videos associated with the retrieval index based on the number of times each retrieval index appears. For example, the more times the maintenance operation of the maintenance equipment requires guidance, the higher the weight. In this way, the maintenance operation of the maintenance equipment with the most problems can be determined based on the weight of the retrieval index, so that the maintenance object can be trained on the maintenance operation of the maintenance equipment, or the maintenance equipment can be directly replaced.

[0061] Figure 3 is a schematic diagram of an optional key video information extraction system according to an embodiment of the present invention. Figure 3 As shown, it includes: AR glasses, expert operation terminal, operation modeling and optimization server. Among them, AR glasses include: camera, data transmission and display module; expert operation terminal includes: touch screen and expert action module. When the relay protection operation begins, the operation modeling and optimization server can extract the operation object and operation content from the work ticket / operation ticket; the data transmission module of the AR glasses transmits the on-site video data collected by the camera of the AR glasses to the expert operation terminal and the operation modeling and optimization server, so that the expert operation terminal generates guidance gesture data based on these on-site video data, and enables the operation modeling and optimization server to identify the operation object based on these on-site video data; in addition, the background can generate interactive video results based on these on-site video data and guidance gesture data, and transmit the interactive video results to the display module of the AR glasses through the data transmission module for display, and transmit the interactive video results to the touch screen of the expert operation terminal for display, and can also transmit the interactive video results to the operation modeling and optimization server, so that the operation modeling and optimization server can judge whether the interaction is within the scope of the operation object based on the extracted operation object, the operation pair in the identified on-site video, and the interactive video results, and save the interactive video within the scope of the operation object as a key video, and use the corresponding operation object and operation content as an index, so as to achieve modeling and optimization, and obtain a preset reference video library. Afterwards, both the AR glasses and the expert operation terminal can call the matching key video from the preset reference video library for reference.

[0062] In the embodiment of the present invention, based on the work ticket as the clustering basis of the work content, the establishment of a key data model for relay protection operations with zero prior knowledge can be achieved, avoiding the manual calibration work in the model establishment process, and based on the work ticket, the interaction process between experts and maintenance objects working in the core area can be saved, which not only avoids the workload of data input, but also can save the core important link data to facilitate the subsequent guidance of on-site maintenance operations.

[0063] Embodiment 2

[0064] According to an embodiment of the present invention, an embodiment of a guiding method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] Figure 4 is a flow chart of an optional guidance method according to an embodiment of the present invention, such as Figure 4 As shown, the method comprises the following steps:

[0066] Step S401, when performing a relay protection operation, extract the operation object set of the current work ticket and the operation content corresponding to each operation object in the operation object set to obtain a retrieval index set.

[0067] Step S402: upon receiving a guidance request, a target retrieval index is determined from a retrieval index set based on a current operation step, wherein the current operation step includes equipment to be repaired and an operation to be repaired.

[0068] Step S403: matching a target reference video from a preset reference video library based on the target retrieval index.

[0069] Step S404, completing the current operation step based on the target reference video.

[0070] In an embodiment of the present invention, when performing relay protection operations, the operation modeling and optimization server can first extract the operation object set and the operation content corresponding to each operation object in the operation object set from the current work ticket to obtain a retrieval index set. If a guidance request is received from the maintenance object through AR glasses, the target retrieval index can be determined from the retrieval index set according to the current operation step, so as to match the target reference video from the preset reference video library according to the target retrieval index, so as to complete the current operation step according to the target reference video, which can improve the maintenance efficiency and reduce the real-time guidance service of the background experts, thereby solving the technical problem that the related technology cannot quickly and effectively extract reference data, resulting in the inability to effectively guide subsequent operations.

[0071] In the embodiment of the present invention, the preset reference video library can be continuously updated according to each relay protection operation to provide the latest maintenance reference for the maintenance object.

[0072] Optionally, the guidance method also includes: when the current work step is not completed based on the target reference video, issuing an interactive request to the target work terminal, wherein the target work terminal captures the guidance gesture data based on the interactive request; receiving an interactive video generated by the guidance gesture data and the on-site video data indicated by the current work step, and completing the current work step based on the interactive video.

[0073] In an embodiment of the present invention, if the current operation step is still not completed according to the matched target reference video, the real-time guidance service of the background expert can be requested, that is, an interactive request is sent to the target operation terminal through the AR glasses, so that the target operation terminal captures the guidance gesture data according to the interactive request, and then receives the interactive video generated by the guidance gesture data and the on-site video data indicated by the current operation step, and completes the current operation step according to the interactive video.

[0074] In some optional embodiments, a key video may be captured from the current expert-guided interactive video, and the current assignment object and the current assignment content may be associated with the key video as key indexes and saved to a preset reference video library.

[0075] Another optional specific implementation is described in detail below.

[0076] In an embodiment of the present invention, a relay protection operation process based on a relay protection interactive guidance AR glasses key video information extraction system is provided, including the following process:

[0077] (1) After the relay protection operation begins, the operation modeling and optimization server extracts the subject noun in the work ticket as the operation object and extracts the verb as the operation content to form keywords;

[0078] (2) Under the current operation object and operation content, the on-site staff of relay protection can call the expert through AR glasses when encountering problems. After the call is connected, the on-site staff and the expert can discuss the current operation steps through the fusion of two-way video;

[0079] (3) If the on-site operator needs it, the video that best matches the current operation object and the current operation content can be called for reference. For example, the matching rule can be: the area of ​​the operator's gesture and the expert's gesture within the physical range of the operation object is ≥ 30%;

[0080] (4) After the current job is completed, save the key information of the current job;

[0081] (5) If the interactive guidance ends, proceed to step (6), otherwise return to step (2);

[0082] (6) Save interactive instruction video data and create an index based on the assignment object and assignment content;

[0083] (7) calculating the statistical probability of the interactive guidance video data under the corresponding index according to the task object and task content;

[0084] (8) Execute the next operation content. If interactive guidance is required, proceed to step (2) until the relay protection operation is completed.

[0085] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: when performing a relay protection operation, extracting the operation object set of the current work ticket and the operation content corresponding to each operation object in the operation object set to obtain a retrieval index set; when receiving a guidance request, based on the current operation step, determining a target retrieval index from the retrieval index set, wherein the current operation step includes equipment to be repaired and operations to be repaired; based on the target retrieval index, matching a target reference video from a preset reference video library; and completing the current operation step based on the target reference video.

[0086] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned guiding method.

[0087] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned guidance method.

[0088] Figure 5 is a hardware structure block diagram of an electronic device (or mobile device) for guiding the method according to an embodiment of the present invention. Figure 5 As shown, the electronic device may include one or more processors (e.g., Figure 5 The processor 502a, processor 502b, ..., processor 502n, etc., which may include but are not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 504 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 5The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components as shown, or with Figure 5 Different configurations are shown.

[0089] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0090] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0091] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0092] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0093] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0094] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0095] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0096] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A key video information extraction system for relay protection interactive guidance AR glasses, characterized in that: The key video information extraction system comprises: The AR glasses system is used to collect on-site video data and transmit the on-site video data to the operation modeling and optimization server, wherein the on-site video data is data collected from the relay protection maintenance site; A target operation terminal is used to capture the guidance gesture data and transmit the guidance gesture data to the operation modeling and optimization server, wherein the guidance gesture data is data for providing maintenance guidance for maintenance equipment; The job modeling and optimization server extracts a set of job objects based on the current work ticket, and determines key video information based on the on-site video data, the set of job objects and the guidance gesture data, and adds the key video information to a preset reference video library, wherein the key video information includes key videos associated with the job objects.

2. The key video information extraction system according to claim 1, characterized in that: The AR glasses system comprises: A camera module, used for collecting the on-site video data; A data transmission module, used to transmit the on-site video data to the operation modeling and optimization server and the target operation terminal; A display module is used to display an interactive video or a historical reference video, wherein the interactive video is a video generated based on the on-site video data and the guiding gesture data, and the historical reference video is a key video matched from the preset reference video library, and the historical reference video is used to provide a reference for the maintenance object wearing AR glasses to perform relay protection operations.

3. The key video information extraction system according to claim 1, characterized in that: The target operation terminal includes: A touch screen, used for displaying the live video data; The motion capture module is used to capture the guiding gesture data, wherein the guiding gesture data is data generated based on the on-site video data.

4. The key video information extraction system according to claim 1, characterized in that: The job modeling and optimization server includes: A work analysis module, used to extract the work object in the work ticket and the work content corresponding to the work object, wherein the work ticket is a specification document of the maintenance steps of the pre-planned maintenance work, and each maintenance step includes a maintenance operation on the maintenance equipment; A work interaction analysis module, used for determining key videos based on the on-site video data, the guidance gesture data and the work object; A storage module, used to use the operation object and the operation content corresponding to the operation object as a retrieval index of the key video, and store the retrieval index after associating it with the key video; The model building and optimization module is used to count the number of times the search index appears, and based on the number of times each search index appears, assign weights to the key videos associated with the search index.

5. The key video information extraction system according to claim 4, characterized in that: The extracting of the operation object in the work ticket and the operation content corresponding to the operation object includes: Extracting the equipment noun in the work ticket and characterizing the equipment noun as the work object; Extracting the operation verbs in the work ticket, and representing the operation verbs as the work content; The device noun corresponding to the operation verb is identified, and the operation object represented by the device noun is associated with the operation content represented by the operation verb.

6. The key video information extraction system according to claim 4, characterized in that: The determining of the key video based on the on-site video data, the guiding gesture data and the operation object includes: Acquire an interactive video generated by fusing the on-site video data and the guiding gesture data, wherein a guiding gesture is provided on an interactive area of ​​the interactive video; Based on the interactive area, determining a key area; When the operation object is located in the key area, a sub-video including the guiding gesture is captured from the interactive video, and the sub-video is used as the key video of the operation object.

7. A method for guiding the key video information extraction system of the relay protection interactive guidance AR glasses described in claim 1, characterized in that: include: In the case of a relay protection operation, extracting the operation object set of the current work ticket and the operation content corresponding to each operation object in the operation object set to obtain a retrieval index set; In case of receiving the guidance request, determining a target retrieval index from the retrieval index set based on a current operation step, wherein the current operation step includes the equipment to be repaired and the operation to be repaired; Based on the target retrieval index, matching the target reference video from a preset reference video library; Based on the target reference video, the current operation step is completed.

8. The guidance method according to claim 7, characterized in that: The guiding method further comprises: In a case where the current operation step is not completed based on the target reference video, an interaction request is sent to a target operation terminal, wherein the target operation terminal captures the guidance gesture data based on the interaction request; An interactive video generated by the guiding gesture data and the on-site video data indicated by the current operation step is received, and the current operation step is completed based on the interactive video.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the guiding method according to any one of claims 7 to 8.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the guiding method described in any one of claims 7 to 8.

Images