Operation recorder data management method and device

Abstract

The invention provides an operation recorder data management method and device, and the method comprises the steps: obtaining on-site audio and video data collected by an operation recorder, carrying out the deep noise reduction of the on-site audio and video data through a target voice recognition system, converting the on-site audio and video data into on-site text data, and transmitting the on-site text data to a server; aligning and splicing the field audio and video data on a time axis, analyzing based on the field text data to obtain a currently executed field task, segmenting the field audio and video data based on a task type of the field task to obtain a field task video of each field task, and obtaining a target electric power professional semantic understanding model; and extracting operation key information in the field text data based on the target electric power professional semantic understanding model, archiving the field task video based on the operation key information, identifying a key voice event and a key visual scene of the field task video, and labeling the field task video based on the key voice event and the key visual scene.

Description

Technical Field

[0001] This application relates to the field of intelligent management technology for power operations, and more specifically, to a data management method and device for a work recorder. Background Technology

[0002] The development of power operation management technology is fundamentally about a profound shift from relying on human experience to data-driven and intelligent decision-making, aiming to address the increasing complexity of power systems and safeguard safety as the lifeline.

[0003] Currently, on-site operation management generally adopts a manual registration and decentralized charging mode. Data is transmitted back through manual copying and manual annotation, while equipment status and storage space are maintained manually.

[0004] However, existing on-site operation management methods rely on manual labor, resulting in low equipment management efficiency, easy loss and insufficient power, time-consuming data processing, and manual labeling prone to errors and omissions with inconsistent standards. Furthermore, data flow lacks effective supervision, and data storage suffers from resource waste. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a data management method and device for work recorders. By converting and naming the on-site audio and video data collected by the work recorder, splicing and segmenting the on-site audio and video data to obtain the on-site task video for each on-site task, and annotating the on-site task video based on key voice events and key visual scenes, this method eliminates reliance on manual labor, improves management efficiency and data processing efficiency, avoids the problems of errors and omissions and inconsistent standards in manual annotation, improves the accuracy of data processing, and allows for effective monitoring of data flow. In addition, it also avoids the problem of resource waste in data storage.

[0006] In a first aspect, embodiments of this application provide a data management method for a work recorder, the method comprising: The system acquires on-site audio and video data collected by the work recorder, performs deep noise reduction on the on-site audio and video data using a target speech recognition system, and converts the on-site audio and video data into on-site text data. The on-site audio and video data are aligned and stitched together on the timeline. The currently executed on-site task is obtained by parsing the on-site text data. The on-site audio and video data is segmented according to the task type of the on-site task to obtain the on-site task video for each on-site task. The on-site task video represents the complete workflow of the on-site task. A target power industry semantic understanding model is obtained, and key operational information is extracted from the field text data based on the target power industry semantic understanding model. The field task video is then archived based on the key operational information. The key operational information includes at least time, location, and work content. Identify key audio events and key visual scenes in the on-site task video, and annotate the on-site task video based on the key audio events and key visual scenes; wherein, the key audio events and key visual scenes are constructed with a time-related mapping.

[0007] In one possible implementation, the target speech recognition system includes at least an acoustic model, a language model, and a search engine; the conversion of the on-site audio and video data into on-site text data includes: Extract the audio and video features from the on-site audio and video data and input them into the search engine; In the search engine, the audio and video features are matched with a preset pronunciation dictionary, and processed in conjunction with the acoustic model and the language model to obtain the corresponding on-site text data.

[0008] In one possible implementation, segmenting the on-site audio and video data based on the task type of the on-site task to obtain the on-site task video for each task includes: The estimated task duration window for the on-site task is set based on a preset power operation knowledge base; If a new on-site task is identified within the estimated task duration window, the identification time point is used as the end boundary of the current on-site task. If no new on-site task is identified within the estimated task duration window, the blank time period of the current on-site task is analyzed, and the actual end point of the on-site task is determined based on the on-site characteristics of the current on-site task, so as to obtain the on-site task video of each on-site task.

[0009] In one possible implementation, identifying key audio events and key visual scenes in the on-site task video includes: Based on the target speech recognition system, the on-site task video is analyzed to obtain key speech events; The corresponding key visual scene is obtained based on the key voice event matching; wherein, the key visual scene represents the key visual scene corresponding to the key voice event.

[0010] In one possible implementation, the annotation of the on-site task video based on the key voice events and the key visual scenes includes: The speech stream and video stream of the on-site task video are determined, and the speech stream and video stream of the on-site task video are mapped into high-dimensional feature sequences based on the target speech recognition system to obtain the speech feature vector and visual feature vector of the on-site task video. The speech feature vector and the visual feature vector are fused to obtain the corresponding fused feature; Based on the fusion features, key audio events and time periods of the key audio events in the on-site mission video are determined, and the on-site mission video is annotated based on the key audio events and time periods of the key audio events.

[0011] In one possible implementation, fusing the speech feature vector with the visual feature vector to obtain the corresponding fused feature includes: An attention weight formula is constructed based on the speech feature vector and the visual feature vector, and a fusion feature formula is constructed based on the attention weight formula. The speech feature vector and the visual feature vector are fused based on the fusion feature formula to obtain the corresponding fusion feature.

[0012] In one possible implementation, the annotation of the on-site mission video includes: Insert corresponding structured annotation points into the on-site task video, and determine the event type and event information of the key voice events; Based on the event type and event information of the key voice events, standardized segment names are generated for the on-site task video, and the on-site task video is labeled based on the standardized segment names.

[0013] Secondly, embodiments of this application also provide a work recorder data management device, the device comprising: The processing module is used to acquire on-site audio and video data collected by the work recorder, and to perform deep noise reduction processing on the on-site audio and video data using a target speech recognition system, and to convert the on-site audio and video data into on-site text data. The acquisition module is used to align and stitch the on-site audio and video data on the timeline, parse the on-site text data to obtain the currently executed on-site task, and segment the on-site audio and video data according to the task type of the on-site task to obtain the on-site task video for each on-site task; wherein, the on-site task video represents the complete workflow of the on-site task; The archiving module is used to acquire a target power industry semantic understanding model, extract key operational information from the field text data based on the target power industry semantic understanding model, and archive the field task video based on the key operational information; wherein, the key operational information includes at least time, location, and work content; The annotation module is used to identify key audio events and key visual scenes in the on-site task video, and to annotate the on-site task video based on the key audio events and key visual scenes; wherein, the key audio events and key visual scenes are constructed with a time-related mapping.

[0014] In one possible implementation, the target speech recognition system includes at least an acoustic model, a language model, and a search engine; the processing module is specifically used for: Extract the audio and video features from the on-site audio and video data and input them into the search engine; In the search engine, the audio and video features are matched with a preset pronunciation dictionary, and processed in conjunction with the acoustic model and the language model to obtain the corresponding on-site text data.

[0015] In one possible implementation, the acquisition module is specifically used for: The estimated task duration window for the on-site task is set based on a preset power operation knowledge base; If a new on-site task is identified within the estimated task duration window, the identification time point is used as the end boundary of the current on-site task. If no new on-site task is identified within the estimated task duration window, the blank time period of the current on-site task is analyzed, and the actual end point of the on-site task is determined based on the on-site characteristics of the current on-site task, so as to obtain the on-site task video of each on-site task.

[0016] In one possible implementation, the annotation module is specifically used for: Based on the target speech recognition system, the on-site task video is analyzed to obtain key speech events; The corresponding key visual scene is obtained based on the key voice event matching; wherein, the key visual scene represents the key visual scene corresponding to the key voice event.

[0017] In one possible implementation, the annotation module is specifically used for: The speech stream and video stream of the on-site task video are determined, and the speech stream and video stream of the on-site task video are mapped into high-dimensional feature sequences based on the target speech recognition system to obtain the speech feature vector and visual feature vector of the on-site task video. The speech feature vector and the visual feature vector are fused to obtain the corresponding fused feature; Based on the fusion features, key audio events and time periods of the key audio events in the on-site mission video are determined, and the on-site mission video is annotated based on the key audio events and time periods of the key audio events.

[0018] In one possible implementation, the annotation module is specifically used for: An attention weight formula is constructed based on the speech feature vector and the visual feature vector, and a fusion feature formula is constructed based on the attention weight formula. The speech feature vector and the visual feature vector are fused based on the fusion feature formula to obtain the corresponding fusion feature.

[0019] In one possible implementation, the annotation module is specifically used for: Insert corresponding structured annotation points into the on-site task video, and determine the event type and event information of the key voice events. Based on the event type and event information of the key voice events, standardized segment names are generated for the on-site task video, and the on-site task video is labeled based on the standardized segment names.

[0020] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the job recorder data management method as described in any of the first aspects.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the job recorder data management method described in any one of the first aspects.

[0022] This application provides a data management method and apparatus for a work recorder. The method involves acquiring on-site audio and video data collected by the work recorder, performing deep noise reduction on the on-site audio and video data using a target speech recognition system, converting the on-site audio and video data into on-site text data, aligning and splicing the on-site audio and video data along the timeline, parsing the on-site text data to obtain the currently executed on-site task, segmenting the on-site audio and video data based on the task type to obtain on-site task videos for each task, acquiring a target power industry semantic understanding model, extracting key work information from the on-site text data based on the target power industry semantic understanding model, archiving the on-site task videos based on the key work information, identifying key voice events and key visual scenes in the on-site task videos, and annotating the on-site task videos based on the key voice events and key visual scenes. This application eliminates reliance on manual labor by converting and naming the on-site audio and video data collected by the work recorder, splicing and segmenting the on-site audio and video data to obtain the on-site task video for each task, and annotating the on-site task video based on key voice events and key visual scenes. This improves management and data processing efficiency, avoids the problems of errors and inconsistencies in manual annotation, enhances the accuracy of data processing, and allows for effective monitoring of data flow. In addition, it also avoids the problem of resource waste in data storage.

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a work recorder data management method provided according to an embodiment of this application; Figure 2 This is a schematic diagram of the overall process for managing work recorder data; Figure 3 This is a schematic diagram of a target speech recognition system and the text conversion performed by the target speech recognition system; Figure 4 This is a diagram illustrating the acquisition of on-site mission video; Figure 5 This is a schematic diagram of the visual recognition module; Figure 6This is a schematic diagram of the structure of the work recorder data management device provided according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0027] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0029] Considering the development of power operation management technology, the core is a profound shift from relying on human experience to data-driven and intelligent decision-making, aiming to cope with the increasing complexity of the power system and safeguard the lifeline of safety.

[0030] Currently, on-site operation management generally adopts a manual registration and decentralized charging mode. Data is transmitted back through manual copying and manual annotation, while equipment status and storage space are maintained manually.

[0031] However, existing on-site operation management methods rely on manual labor, resulting in low equipment management efficiency, easy loss and insufficient power, time-consuming data processing, and manual labeling prone to errors and omissions with inconsistent standards. Furthermore, data flow lacks effective supervision, and data storage suffers from resource waste.

[0032] To address this issue, this application provides a data management method and apparatus for work recorders. By converting and naming the on-site audio and video data collected by the work recorder, splicing and segmenting the on-site audio and video data to obtain the on-site task video for each task, and annotating the on-site task video based on key voice events and key visual scenes, this method eliminates reliance on manual labor, improves management and data processing efficiency, avoids the problems of errors and inconsistent standards in manual annotation, enhances the accuracy of data processing, and allows for effective monitoring of data flow. Furthermore, it avoids the problem of resource waste in data storage.

[0033] Figure 1 This is a flowchart of a work recorder data management method provided according to an embodiment of this application. For example... Figure 1 As shown in the embodiments of this application, the data management method for the work recorder may specifically include: S101. Acquire the on-site audio and video data collected by the work recorder, and use the target speech recognition system to perform deep noise reduction processing on the on-site audio and video data, and convert the on-site audio and video data into on-site text data.

[0034] S102. Align and stitch the on-site audio and video data on the timeline, parse the on-site text data to obtain the currently executed on-site task, and segment the on-site audio and video data according to the task type of the on-site task to obtain the on-site task video for each on-site task.

[0035] S103. Obtain the target power industry semantic understanding model, extract key operational information from the field text data based on the target power industry semantic understanding model, and archive the field task video based on the key operational information.

[0036] S104. Identify key audio events and key visual scenes in the on-site task video, and annotate the on-site task video based on the key audio events and key visual scenes.

[0037] The above-mentioned data management method for work recorders involves text conversion and naming archiving of the on-site audio and video data collected by the work recorder, splicing and segmenting the on-site audio and video data to obtain the on-site task video for each on-site task, and annotating the on-site task video based on key voice events and key visual scenes. This eliminates reliance on manual labor, improves management and data processing efficiency, avoids the problems of errors and omissions and inconsistent standards in manual annotation, improves the accuracy of data processing, and allows for effective monitoring of data flow. In addition, it also avoids the problem of resource waste in data storage.

[0038] The exemplary steps described above in the embodiments of this application are illustrated below with specific examples: S101 acquires the on-site audio and video data collected by the work recorder, and uses a target speech recognition system to perform deep noise reduction on the on-site audio and video data, and converts the on-site audio and video data into on-site text data.

[0039] In this embodiment, the on-site audio and video data refers to the audio and video data of the work site collected by the work recorder. The target speech recognition system is a pre-set improved speech recognition system that integrates the required speech enhancement technologies, such as FlowSE speech enhancement technology. The target speech recognition system is specifically optimized for professional terminology in engineering scenarios. The target speech recognition system using FlowSE speech enhancement technology performs deep noise reduction processing on the on-site ambient sound of the on-site audio and video data, and converts the noise-reduced on-site audio and video data into on-site text data in real time for subsequent processing. For example, such as... Figure 2 As shown.

[0040] Optionally, in response to successful identification and mounting of the work recorder, the on-site audio and video data collected by the work recorder is acquired, and the on-site audio and video data of the work recorder is distinguished based on the device serial number of different work recorders.

[0041] Specifically, for example, such as Figure 2 As shown, the work recorder is connected to the USB port. After successful recognition and mounting, the work recorder collects on-site audio and video data and distinguishes and associates the on-site audio and video data of the work recorder according to the device serial number of different work recorders.

[0042] The target speech recognition system includes at least an acoustic model, a language model, and a search engine.

[0043] Optionally, when converting on-site audio and video data into on-site text data, audio and video features are extracted from the data and input into a search engine. In the search engine, these features are matched against a pre-defined pronunciation dictionary and processed using acoustic and language models to obtain the corresponding on-site text data. For example, ... Figure 3 As shown, text conversion is performed using this target speech recognition system.

[0044] S102, align and stitch the on-site audio and video data on the timeline, parse the on-site text data to obtain the currently executing on-site task, and segment the on-site audio and video data according to the task type of the on-site task to obtain the on-site task video for each on-site task.

[0045] In this embodiment, the on-site task refers to the specific task performed at the work site, such as "110kV main transformer inspection and oiling test." The task type refers to the different types of on-site tasks. The file metadata of the video segment contains a corresponding timestamp. The on-site task video represents the complete workflow of the on-site task; that is, the on-site task video is a complete workflow of an on-site task. After aligning and splicing the on-site audio and video data on the timeline, the on-site text data in step S101 is parsed to obtain the currently executed on-site task. When the start of the on-site task is identified, the on-site audio and video data is segmented according to the task type to obtain the on-site task video for each on-site task. This achieves time synchronization and video splicing for subsequent processing. For example, as shown... Figure 2 As shown.

[0046] Optionally, when segmenting the on-site audio and video data based on the task type to obtain the on-site task video for each task, an estimated task duration window is set based on a preset power operation knowledge base. Within the estimated task duration window, if a new on-site task is identified, the identification time point is used as the end boundary of the current on-site task. If no new on-site task is identified within the estimated task duration window, the blank time period of the current on-site task is analyzed, and the actual end point of the on-site task is determined based on the on-site characteristics of the current on-site task to obtain the on-site task video for each task. The on-site characteristics include at least on-site ambient sound and personnel activities; the identification time point is the time point when the next new on-site task is identified.

[0047] Specifically, for example, for the on-site task "110kV main transformer inspection and oiling test," the typical duration preset in the power operation knowledge base is approximately 60 minutes. Therefore, an estimated task duration window is set based on this. Within this estimated task duration window, if a new task is detected, the detection time is used as the end boundary of the current task. If no new task is detected, the blank period when the task should have ended but did not is analyzed. By analyzing characteristics such as ambient sound and personnel activities, the actual end point of the task is determined. Finally, all video clips of the same task (regardless of whether they are consecutive or from multiple 15-minute recording segments) are automatically spliced ​​into a complete logical unit, specifically by first splicing along the timeline and then segmenting. For example, as... Figure 4 As shown, the complete workflow of aligning and splicing video clips to obtain each action task is achieved by segmenting the video into segments.

[0048] S103, Obtain the target power industry semantic understanding model, extract key operational information from the field text data based on the target power industry semantic understanding model, and archive the field task video based on the key operational information.

[0049] Optionally, considering the characteristics of the power industry, a set of power industry terminology (professional dictionary and parsing rules) (power field voice data) is established; a power industry semantic understanding model to be trained is obtained; and the model is trained based on the set of power industry terminology to obtain the target power industry semantic understanding model. The set of power industry terminology, i.e., the professional dictionary and parsing rules, should at least include equipment types, operational items, and safety regulations in the power industry.

[0050] In this embodiment of the application, the key information of the operation includes at least time, location and work content. That is, the key information of the operation represents the information of task-time-personnel. The key information of the operation representing task-time-personnel in the on-site text data in step S101 is extracted based on the target power professional semantic understanding model, and the on-site task video is archived based on the key information of the operation.

[0051] Optionally, key operational information can be standardized to generate corresponding standardized filenames (standardized naming), and the on-site audio and video data can be archived based on these standardized filenames. The archived on-site audio and video data represents structured data representing time, task, and personnel.

[0052] Specifically, after standardizing the key information of the operation, corresponding standardized file names are generated, that is, standardized naming. The on-site audio and video data are archived through standardized naming, thereby establishing an engineering data archiving and traceability system.

[0053] S104 identifies key audio events and key visual scenes in the on-site task video, and annotates the on-site task video based on the key audio events and key visual scenes.

[0054] In this embodiment, key visual scenes refer to key video segments, and key audio events are matched with key visual scenes; a time-related mapping is constructed between key audio events and key visual scenes; in step S102, key audio events and corresponding key visual scenes of the on-site task video are identified, and standardized annotations are performed on the timeline of the on-site task video based on the identified key audio events and key visual scenes, ultimately forming an annotated on-site task video for each on-site task, with annotations of the key events of that on-site task on the video. For example, as... Figure 2 As shown.

[0055] In some implementations, when identifying key audio events and key visual scenes in a live video of a task, the live video is analyzed using a target speech recognition system to obtain key audio events; the corresponding key visual scenes are then obtained by matching the key audio events. Key audio events include at least operational instructions and anomaly reports; key visual scenes represent the key visual scenes corresponding to the key audio events.

[0056] Specifically, the target speech recognition system uses deep learning-based acoustic and language models to analyze the operation segments of on-site task videos, identify key speech events such as operation instructions and abnormal reports, and match the key visual scenes corresponding to the key speech events.

[0057] Optionally, when annotating the on-site task video based on key speech events and key visual scenes, the speech stream and video stream of the on-site task video are determined, and the speech stream and video stream of the on-site task video are mapped into high-dimensional feature sequences respectively based on the target speech recognition system to obtain the speech feature vector and visual feature vector of the on-site task video; the speech feature vector and visual feature vector are fused to obtain the corresponding fused features; the key speech events and the time periods of the key speech events in the on-site task video are determined based on the fused features, and the on-site task video is annotated based on the key speech events and the time periods of the key speech events.

[0058] Optionally, when fusing speech feature vectors and visual feature vectors to obtain corresponding fused features, an attention weight formula is constructed based on the speech feature vectors and visual feature vectors, and a fused feature formula is constructed based on the attention weight formula; the speech feature vectors and visual feature vectors are then fused based on the fused feature formula to obtain the corresponding fused features. Here, the attention weight formula represents the weight of the visual context on the audio features.

[0059] Specifically, the audio and video streams of the video segments are first mapped to high-dimensional feature sequences. Let At∈ Let da represent the acoustic feature vector (i.e., speech feature vector) at time t, and determine the corresponding visual feature vector V. t To enable cross-modal interaction, this application can use a collaborative attention module to calculate the weights of visual context with respect to audio features, and the attention weight formula is as follows:

[0060] Cross-modal fusion features are constructed by using attention weights to integrate speech and visual feature vectors. Based on this fusion feature, the key audio events and their time periods (i.e., start and end time periods, or start and end boundaries) corresponding to the operation segments of the on-site task video can be determined. Then, standardized annotations corresponding to the key audio events can be added based on the key audio events and their time periods.

[0061] Optionally, when annotating the on-site task video, corresponding structured annotation points are inserted into the video, and the event types and information of key audio events are determined. Standardized segment names for the on-site task video are generated based on the event types and information of the key audio events, and standardized annotations are performed on the video based on these standardized segment names. For example, ... Figure 2 As shown.

[0062] The event information includes the event type and content; each labeled video clip is associated with a complete metadata description, including the event type, time of occurrence, duration, equipment involved, and operators.

[0063] Specifically, structured annotation points are inserted into the video timeline, and standardized segment names are generated based on event type and key information. Each annotated segment is associated with a complete metadata description, including event type, occurrence time, duration, equipment involved, and operators, forming a complete video segment index system. The final output annotated video data supports precise content-based retrieval, providing technical support for engineering quality management, abnormal event tracing, and operational standard evaluation.

[0064] In summary, this application utilizes speech recognition technology to automatically extract key information from voice recordings during power field operations, establishing a strong "task-time-person" correlation and enabling automatic archiving. It also forcibly binds data to predefined metadata. The core technology lies in replacing manual intervention with speech recognition, achieving a fully automated process from data generation to structured management, thus resolving errors, omissions, non-standardization, and disconnections caused by manual operation. Furthermore, the technology, through collaborative analysis of voice signals and video scene features, accurately identifies and locates the start and end points of individual inspection tasks, abnormal inspection reports, and line fault repairs in engineering videos, automatically labeling and standardizing these segments. The core of this technology lies in the synergistic mechanism of the two recognition technologies and the resulting ability to automatically and precisely segment and label unstructured video content, aiming to address the pain point of difficulty in quickly retrieving and managing key video segments.

[0065] The work recorder data management method provided in this application embodiment acquires on-site audio and video data collected by the work recorder, performs deep noise reduction processing on the on-site audio and video data using a target speech recognition system, converts the on-site audio and video data into on-site text data, aligns and splices the on-site audio and video data on the timeline, parses the on-site text data to obtain the currently executed on-site task, segments the on-site audio and video data according to the task type of the on-site task to obtain the on-site task video for each on-site task, acquires a target power professional semantic understanding model, extracts key work information from the on-site text data based on the target power professional semantic understanding model, archives the on-site task video based on the key work information, identifies key voice events and key visual scenes in the on-site task video, and annotates the on-site task video based on the key voice events and key visual scenes. The work recorder data management method of this application converts and names the on-site audio and video data collected by the work recorder into text, splices and segments the on-site audio and video data to obtain the on-site task video for each on-site task, and annotates the on-site task video based on key voice events and key visual scenes. This eliminates the reliance on manual labor, improves management efficiency and data processing efficiency, avoids the problems of errors and omissions and inconsistent standards in manual annotation, improves the accuracy of data processing, and allows for effective monitoring of data flow. In addition, it also avoids the problem of resource waste in data storage.

[0066] Furthermore, in response to the detection of target anomalies, the corresponding abnormal region is circled in the video stream, and the image of the abnormal region is extracted and labeled on the corresponding video timeline to obtain the visual detection result; based on the visual detection result and the voice event, cross-modal correlation analysis is performed to form the evidence chain for event determination.

[0067] Furthermore, when an abnormal operation is detected, the abnormal part will be circled in real time on the video screen, and the image will be extracted and marked on the video timeline.

[0068] The application process of this application is described below in specific scenarios: In routine inspection scenarios, through methods such as Figure 5The visual recognition module continuously analyzes equipment status and environmental elements to accurately detect potential anomalies. This module, trained on a YOLOv8 model, has the ability to recognize various equipment components and anomaly types, including switches and insulating tools. When the system detects an anomaly, it draws bounding boxes and labels in real-time in the video stream for visual annotation. For example, when a pressure indicator reading exceeds a safety threshold, the system automatically circles the indicator area and extracts the image, labeling it on the video timeline. If an oil leak is found on the equipment surface, the corresponding area is circled and the image is extracted and labeled on the video timeline. These visual detection results are then analyzed cross-modally with anomaly reports output by voice recognition, forming a chain of evidence for event determination. The multimodal analysis engine uses the visual behavior recognition module to detect whether the patrol equipment covers the entire preset area and no new monitoring targets appear. The voice event detection module captures voice reports such as "Patrol completed" from maintenance personnel. After the scene understanding analysis module confirms that there are no anomalies in the environmental elements, the three components undergo cross-modal verification in the "collaborative analysis and event localization" stage, ultimately determining that the routine patrol task has ended normally and automatically recording the task end time.

[0069] In switching operation scenarios, visual recognition focuses on the compliance of the operation process and real-time monitoring of equipment status. A specially trained YOLOv8 model accurately identifies key elements such as switch positions and operating tools, while also detecting anomalies such as incorrect operation sequences and improper tool use. A 3D-CNN motion recognition network is responsible for parsing the motion features of standard operations such as opening and closing switches. When the visual sequence deviates from the preset operation process, the system uses a dynamic time warping algorithm to align the sequence and locate the anomaly. When an abnormal operation is detected, the video feed will highlight the abnormal area in real time, such as highlighting incorrectly positioned switches or marking improperly used tools, and extracting the image and annotating it on the video timeline. These visual information and the repeated instructions from the voice channel corroborate each other, jointly determining the boundaries and integrity of the operation phase. The visual behavior recognition module detects that the operating tool has left the work area and there are no subsequent operating actions. The voice event detection module captures the repeated instruction of "operation completed". After the scene understanding and analysis module confirms that the final position of the switch is compliant, the three are correlated across modalities through the "collaborative analysis and event localization" stage to locate the normal completion of the switching operation task and record the final status of the equipment and the operation completion time.

[0070] Visual recognition in emergency repair scenarios primarily detects fault characteristics and the effectiveness of safety measures. A scene classifier trained with a YOLOv8 model can identify abnormal states such as emergency lighting malfunctions and missing safety barriers, while also locating fault characteristics such as equipment smoke and detached components. When the system detects an malfunction in emergency lighting equipment, it immediately circles the affected area and labels the fault type; similarly, when a missing safety barrier or visible smoke is detected, it is prominently marked in the video frame. These visual features, along with keywords captured by speech recognition, form the basis for event determination. When the overall confidence level exceeds a preset threshold, the system automatically segments the event and generates a structured record containing complete metadata. The visual behavior recognition module detects the disappearance of the original fault characteristics and the evacuation of personnel and tools from the site; the speech event detection module captures the keyword report "repair completed"; and the scene understanding and analysis module confirms that safety measures have been restored. Through collaborative analysis and event localization, these three modules determine the normal completion of the emergency repair task and record the fault handling result and task completion time.

[0071] Thus, through multi-model collaboration and specialized anomaly training, the entire visual recognition system achieves accurate detection and visual annotation of various anomalies in different engineering scenarios, significantly improving the retrieval efficiency and analytical value of video data.

[0072] Figure 6 This is a schematic diagram of the structure of the work recorder data management device provided according to an embodiment of this application; as shown below. Figure 6 As shown, the work recorder data management device 600 of this application embodiment may specifically include: The processing module 601 is used to acquire on-site audio and video data collected by the work recorder, and to perform deep noise reduction processing on the on-site audio and video data using a target speech recognition system, and to convert the on-site audio and video data into on-site text data.

[0073] The acquisition module 602 is used to align and stitch the on-site audio and video data on the timeline, parse the on-site text data to obtain the currently executed on-site task, and segment the on-site audio and video data according to the task type of the on-site task to obtain the on-site task video for each on-site task; wherein, the on-site task video represents the complete workflow of the on-site task.

[0074] The archiving module 603 is used to acquire the target power professional semantic understanding model, extract key operation information from the field text data based on the target power professional semantic understanding model, and archive the field task video based on the key operation information; wherein, the key operation information includes at least time, location and work content.

[0075] The annotation module 604 is used to identify key audio events and key visual scenes in the on-site task video, and to annotate the on-site task video based on the key audio events and key visual scenes; wherein, the key audio events and key visual scenes are constructed with time-related mappings.

[0076] In one possible implementation, the target speech recognition system includes at least an acoustic model, a language model, and a search engine; the processing module is specifically used for: Extract the audio and video features from the on-site audio and video data and input them into the search engine; In the search engine, audio and video features are matched with a pre-set pronunciation dictionary, and then processed using acoustic and language models to obtain the corresponding on-site text data.

[0077] In one possible implementation, the acquisition module is specifically used for: The estimated task duration window for on-site tasks is set based on a pre-defined power operation knowledge base; If a new on-site task is identified within the estimated task duration window, the identification time point is used as the end boundary of the current on-site task. If no new on-site task is identified within the estimated task duration window, the blank time period of the current on-site task is analyzed, and the actual end point of the on-site task is determined based on the on-site characteristics of the current on-site task, so as to obtain the on-site task video of each on-site task.

[0078] In one possible implementation, the annotation module is specifically used for: Key speech events are obtained by analyzing on-site task videos based on the target speech recognition system; The key visual scene is obtained based on the matching of key speech events; whereby the key visual scene represents the key visual scene corresponding to the key speech event.

[0079] In one possible implementation, the annotation module is specifically used for: The speech stream and video stream of the on-site task video are determined, and the speech stream and video stream of the on-site task video are mapped into high-dimensional feature sequences based on the target speech recognition system, so as to obtain the speech feature vector and visual feature vector of the on-site task video. The speech feature vector and the visual feature vector are fused to obtain the corresponding fused feature; Based on the fusion features, key audio events and their time periods in the on-site mission video are identified, and the on-site mission video is annotated based on the key audio events and their time periods.

[0080] In one possible implementation, the annotation module is specifically used for: An attention weight formula is constructed based on speech feature vectors and visual feature vectors, and a fusion feature formula is constructed based on the attention weight formula. Based on the fusion feature formula, the speech feature vector and the visual feature vector are fused to obtain the corresponding fused feature.

[0081] In one possible implementation, the annotation module is specifically used for: Insert corresponding structured annotation points into the on-site mission video and determine the event type and event information of key voice events. Standardized segment names are generated for on-site task videos based on the event type and event information of key voice events, and the on-site task videos are labeled based on the standardized segment names.

[0082] The work recorder data management device provided in this application embodiment acquires on-site audio and video data collected by the work recorder, performs deep noise reduction processing on the on-site audio and video data using a target speech recognition system, converts the on-site audio and video data into on-site text data, aligns and splices the on-site audio and video data on the timeline, parses the on-site text data to obtain the currently executed on-site task, segments the on-site audio and video data according to the task type of the on-site task to obtain the on-site task video for each on-site task, acquires a target power professional semantic understanding model, extracts key work information from the on-site text data based on the target power professional semantic understanding model, archives the on-site task video based on the key work information, identifies key voice events and key visual scenes in the on-site task video, and annotates the on-site task video based on the key voice events and key visual scenes. The work recorder data management device of this application converts and names the on-site audio and video data collected by the work recorder into text, splices and segments the on-site audio and video data to obtain the on-site task video for each on-site task, and annotates the on-site task video based on key voice events and key visual scenes. This eliminates the reliance on manual labor, improves management efficiency and data processing efficiency, avoids the problems of errors and omissions and inconsistent standards in manual annotation, improves the accuracy of data processing, and can effectively monitor the data flow. In addition, it also avoids the problem of resource waste in data storage.

[0083] like Figure 7 As shown in the embodiment of this application, an electronic device 700 includes a processor 701, a memory 702, and a bus. The memory 702 stores machine-readable instructions that can be executed by the processor 701. When the electronic device is running, the processor 701 communicates with the memory 702 via the bus, and the processor 701 executes the machine-readable instructions to perform the steps of the work recorder data management method described above.

[0084] Specifically, the memory 702 and processor 701 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 701 runs the computer program stored in the memory 702, it can execute the above-mentioned work recorder data management method.

[0085] Corresponding to the above-described work recorder data management method, this application embodiment also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the above-described work recorder data management method.

[0086] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0087] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0089] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the deployment methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0090] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data management method for a work recorder, characterized in that, The method includes: The system acquires on-site audio and video data collected by the work recorder, performs deep noise reduction on the on-site audio and video data using a target speech recognition system, and converts the on-site audio and video data into on-site text data. The on-site audio and video data are aligned and stitched together on the timeline. The currently executed on-site task is obtained by parsing the on-site text data. The on-site audio and video data is segmented according to the task type of the on-site task to obtain the on-site task video for each on-site task. The on-site task video represents the complete workflow of the on-site task. A target power industry semantic understanding model is obtained, and key operational information is extracted from the field text data based on the target power industry semantic understanding model. The field task video is then archived based on the key operational information. The key operational information includes at least time, location, and work content. Identify key audio events and key visual scenes in the on-site task video, and annotate the on-site task video based on the key audio events and key visual scenes; wherein, the key audio events and key visual scenes are constructed with a time-related mapping.

2. The method according to claim 1, characterized in that, The target speech recognition system includes at least an acoustic model, a language model, and a search engine; The process of converting the on-site audio and video data into on-site text data includes: Extract the audio and video features from the on-site audio and video data and input them into the search engine; In the search engine, the audio and video features are matched with a preset pronunciation dictionary, and processed in conjunction with the acoustic model and the language model to obtain the corresponding on-site text data.

3. The method according to claim 1, characterized in that, The on-site audio and video data is segmented based on the task type of the on-site task to obtain the on-site task video for each on-site task, including: The estimated task duration window for the on-site task is set based on a preset power operation knowledge base; If a new on-site task is identified within the estimated task duration window, the identification time point is used as the end boundary of the current on-site task. If no new on-site task is identified within the estimated task duration window, the blank time period of the current on-site task is analyzed, and the actual end point of the on-site task is determined based on the on-site characteristics of the current on-site task, so as to obtain the on-site task video of each on-site task.

4. The method according to claim 1, characterized in that, The identification of key audio events and key visual scenes in the on-site task video includes: Based on the target speech recognition system, the on-site task video is analyzed to obtain key speech events; The corresponding key visual scene is obtained based on the key voice event matching; wherein, the key visual scene represents the key visual scene corresponding to the key voice event.

5. The method according to claim 4, characterized in that, The annotation of the on-site task video based on the key voice events and the key visual scenes includes: The speech stream and video stream of the on-site task video are determined, and the speech stream and video stream of the on-site task video are mapped into high-dimensional feature sequences based on the target speech recognition system to obtain the speech feature vector and visual feature vector of the on-site task video. The speech feature vector and the visual feature vector are fused to obtain the corresponding fused feature; Based on the fusion features, key audio events and time periods of the key audio events in the on-site mission video are determined, and the on-site mission video is annotated based on the key audio events and time periods of the key audio events.

6. The method according to claim 5, characterized in that, The step of fusing the speech feature vector with the visual feature vector to obtain the corresponding fused feature includes: An attention weight formula is constructed based on the speech feature vector and the visual feature vector, and a fusion feature formula is constructed based on the attention weight formula. The speech feature vector and the visual feature vector are fused based on the fusion feature formula to obtain the corresponding fusion feature.

7. The method according to claim 5, characterized in that, The annotation of the on-site task video includes: Insert corresponding structured annotation points into the on-site task video, and determine the event type and event information of the key voice events; Based on the event type and event information of the key voice events, standardized segment names are generated for the on-site task video, and the on-site task video is labeled based on the standardized segment names.

8. A data management device for a work recorder, characterized in that, The device includes: The processing module is used to acquire on-site audio and video data collected by the work recorder, and to perform deep noise reduction processing on the on-site audio and video data using a target speech recognition system, and to convert the on-site audio and video data into on-site text data. The acquisition module is used to align and stitch the on-site audio and video data on the timeline, parse the on-site text data to obtain the currently executed on-site task, and segment the on-site audio and video data according to the task type of the on-site task to obtain the on-site task video for each on-site task; wherein, the on-site task video represents the complete workflow of the on-site task; The archiving module is used to acquire a target power industry semantic understanding model, extract key operational information from the field text data based on the target power industry semantic understanding model, and archive the field task video based on the key operational information; wherein, the key operational information includes at least time, location, and work content; The annotation module is used to identify key audio events and key visual scenes in the on-site task video, and to annotate the on-site task video based on the key audio events and key visual scenes; wherein, the key audio events and key visual scenes are constructed with a time-related mapping.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the job recorder data management method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the job recorder data management method as described in any one of claims 1 to 7.

Images