A field operation data acquisition and labeling method based on visual recognition

Abstract

The application discloses a kind of based on visual identification's field operation data acquisition and marking method, comprising the following steps: obtaining video image data and carrying out data preprocessing, form the image sequence after preprocessing;Target detection is carried out to the image sequence after preprocessing, and initial recognition result is obtained;Multiple source weak label set is constructed based on initial recognition result, fusion processing is carried out to form fusion label sequence, and input to improved automatic labeling inference model;Weak label fusion processing is carried out to the fusion label sequence, and generate reinforced label sequence;Reinforced label sequence is matched with the standard operation flow procedure sequence of preestablished, and generate process label sequence;Based on process label sequence, construct global constraint set, and generate optimization label sequence satisfying all constraint conditions;Optimization label sequence is converted into structured annotation data, and is output after formatting processing.This application realizes field operation data acquisition and marking.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a method for field operation data acquisition and annotation based on visual recognition. Background Technology

[0002] In the field of on-site operation monitoring and management, using video surveillance equipment to collect images of the operation process and automatically identifying the objects and actions of the operation through visual recognition technology is a common approach. Existing technologies typically rely on target detection, instance segmentation, or action recognition models to process video frames to obtain information on the categories, locations, and actions of personnel, tools, and equipment, providing foundational data for subsequent analysis. However, in real-world operating environments, factors such as changing lighting, frequent occlusion, and diverse movement amplitudes can easily lead to false positives, missed detections, and unstable categories in visual recognition results, making it difficult to meet the accuracy requirements of operation analysis.

[0003] Existing methods for generating field operation annotation data often directly apply recognition results or rely on fixed rules for annotation, lacking mechanisms to handle the differences in labels from multiple sources. Labels from multiple sources, such as visual recognition, rule bases, and industry knowledge bases, often exhibit varying reliability. Simply merging them can easily amplify erroneous labels, resulting in high noise and low reliability in the annotation data. Furthermore, existing weak label fusion methods mostly employ majority voting or simple weighting, failing to dynamically evaluate the credibility of labels from each source based on specific scenarios, and also failing to effectively infer implicit true labels, leading to inaccurate fusion results.

[0004] Therefore, how to provide a method for field operation data collection and annotation based on visual recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for collecting and labeling field operation data based on visual recognition. This invention constructs a multi-source weak label set, introduces a weak label fusion mechanism based on Dawid-Skene, uses dynamic time warping and hidden semi-Markov methods for operation sequence alignment, and utilizes Max-SAT to achieve global consistency optimization. This results in an overall improvement in label reliability, process continuity, and procedure compliance of the labeling results for field operation data. It significantly improves the accuracy and stability of automatic labeling in complex operation scenarios, reduces the need for manual intervention, improves data processing efficiency, and ensures the applicability and consistency of the labeling results in enterprise management and model training.

[0006] A method for collecting and labeling field operation data based on visual recognition according to an embodiment of the present invention includes the following steps:

[0007] Acquire video image data of on-site operations, and perform data preprocessing including noise reduction, brightness correction, distortion correction and multi-frame enhancement to form a preprocessed image sequence;

[0008] The preprocessed image sequence is subjected to object detection, instance segmentation and task action recognition to obtain initial recognition results containing object category, action category and time location;

[0009] Based on the initial identification results, a multi-source weak label set is constructed, which is then fused to form a fused label sequence and input into an improved automatic labeling inference model. The improved automatic labeling inference model includes a multi-source weak label fusion layer, a job sequence alignment layer, and a global consistency optimization layer.

[0010] In the multi-source weak label fusion layer, the fused label sequence is subjected to weak label fusion processing to construct a set of weak label functions. The reliability of each weak label source is estimated by the expectation-maximization-based truth inference algorithm to generate a strengthened label sequence.

[0011] In the work sequence alignment layer, the enhanced label sequence is matched with the preset standard work process sequence. The dynamic time warping method is used to align the action sequences with different rhythms, omissions, and disordered orders. The hidden semi-Markov determination method is used to determine the duration distribution of the steps and generate the process label sequence.

[0012] In the global consistency optimization layer, a global constraint set is constructed based on the process label sequence, according to the tool co-occurrence rules, equipment usage rules, and operation protection rules. The satisfiability solution method is used to solve the global problem and generate an optimized label sequence that satisfies all constraints.

[0013] The optimized label sequence is converted into structured labeled data and then formatted before being output, so that the structured labeled data can be directly called by the on-site operation management equipment for use by on-site operation management personnel.

[0014] Optionally, the formation of the preprocessed image sequence specifically includes:

[0015] Acquire video image data of on-site operations, continuously collect data from the work area, split the acquired video into frame-by-frame images in chronological order to form an original image frame sequence, and uniformly manage and store the original image frame sequence;

[0016] Denoising and brightness correction are performed on each frame of the original image frame sequence, noise suppression is performed on the image pixels, and the brightness range of the entire frame image is redistributed to generate an intermediate image frame sequence that has undergone denoising and brightness correction. The intermediate image frame sequence is then subjected to order preservation processing to ensure inter-frame consistency.

[0017] Distortion correction and multi-frame enhancement are performed on the intermediate image frame sequence. The geometric distortion of each frame is corrected, and inter-frame fusion and image enhancement operations are performed using temporally adjacent multiple frames. The enhanced image frames are then recombined in chronological order to form a preprocessed image sequence.

[0018] Optionally, obtaining the initial identification result specifically includes:

[0019] The preprocessed image sequence is arranged in chronological order of acquisition time. Object recognition processing is performed on each frame of the image to obtain the different objects appearing in the current frame. Category identifiers and location parameters are generated for each object, forming object detection results organized by time index.

[0020] Based on the object detection results, region extraction processing is performed on each object in each frame of the image to generate a pixel region mask corresponding to the object in the image, and associate it with the object's category identifier and position parameters to form an object-level spatial description sequence.

[0021] The object-level spatial description sequence is integrated temporally to extract object change features between consecutive frames. Action recognition processing is performed on hotspot areas across time periods to obtain the action category identifier of the operation action and the corresponding start and end time positions. These are then matched with the object-level spatial description to generate initial recognition results.

[0022] Optionally, the formation of the fusion tag sequence specifically includes:

[0023] The initial recognition results are organized by classifying the object category, action category, and corresponding time and location information obtained from each frame of the image according to time order and object identifier, and generating an initial label sequence organized by time index and object index.

[0024] Based on the initial label sequence, rules related to object category, action category and time location are called from the pre-set rule base to generate corresponding weak labels for each initial label record. At the same time, knowledge items related to on-site operation process, equipment use and safety protection are retrieved from the pre-set industry knowledge base to generate corresponding weak labels for each initial label record. The labels obtained by visual recognition, weak labels from the rule base and weak labels from the knowledge base are combined to construct a multi-source weak label set.

[0025] Weak tag fusion processing is performed on the multi-source weak tag set. Weak tags from different sources corresponding to the same object and the same time period are uniformly encoded and merged to generate a fusion tag sequence arranged in chronological order.

[0026] The fused label sequence is input into an improved automatic labeling inference model for processing. The improved automatic labeling inference model includes a multi-source weak label fusion layer, a job sequence alignment layer, and a global consistency optimization layer.

[0027] Optionally, the generation of the enhanced tag sequence specifically includes:

[0028] In the multi-source weak label fusion layer, different labeling functions are defined for each label record in the fused label sequence. Each labeling function outputs a weak label for the same record based on different information in the visual recognition label, rule base and industry knowledge base entries. The output results of all labeling functions are associated with the corresponding labeling function identifiers to form a multi-source weak label set based on data programming.

[0029] Each label record in the multi-source weak label set is regarded as a sample. A hidden true label variable to be determined is set for each sample. A reliability parameter is set for each labeling function to represent the output of different weak labels under each true label. The hidden true label variables of all samples and the reliability parameters of all labeling functions are initialized.

[0030] The method employs Dawid-Skene expectation-maximization truth inference to iteratively update the latent true label variables and reliability parameters. In the expectation step of each iteration, the probability distribution of each sample on each candidate label is calculated based on the current reliability parameters. In the maximization step of each iteration, the number of times each labeling function outputs each weak label under different candidate labels is counted based on the current probability distribution, and the corresponding reliability parameters are updated. The update stops when the iteration meets a preset convergence condition. The preset convergence condition is that the parameter update magnitude is lower than a set threshold or the number of iterations reaches a set upper limit in two adjacent iterations. Based on the converged probability distribution, a single latent true label is determined for each sample, and the latent true labels of all samples are arranged in chronological order according to the fused label sequence to generate a reinforced label sequence.

[0031] Optionally, the generation of the process label sequence specifically includes:

[0032] In the work sequence alignment layer, the reinforcement label sequence is organized in chronological order. The corresponding action category and time position information are extracted from each label record in the reinforcement label sequence to form an action observation sequence arranged by time index. The standard operation process sequence is read from the pre-configured work specifications, and each standard step in the standard operation process sequence is numbered sequentially to form a reference step sequence arranged in step order.

[0033] Dynamic time warping is performed on the action observation sequence and the reference step sequence. The action records in the action observation sequence and the step records in the reference step sequence are read one by one. The matching cost between any action record and any reference step record is calculated. A cumulative cost table is constructed on the two-dimensional time-step plane. Path accumulation calculation is performed on all possible combinations of time steps and step indices. After the cumulative cost table calculation is completed, the alignment path with the minimum cumulative cost is selected from the start position to the end position. A correspondence is established between each action record in the action observation sequence and the step record in the reference step sequence.

[0034] For each step in the reference step sequence, the number of consecutively matched time positions in the alignment path is counted, and this number is regarded as the duration observation value of the current step. The duration determination method of the hidden semi-Markov model is adopted, and each step in the reference step sequence is regarded as a hidden state. A corresponding set of duration parameters is pre-set for each hidden state. Based on the duration observation value of each step, the probability value of each step at each duration is calculated according to the duration calculation rule of the hidden semi-Markov model. The corresponding duration estimation result is selected according to the calculated probability value. The step number of each step, the action category corresponding to the current step, and the duration estimation result are combined and arranged in the order of the reference steps to generate a process label sequence.

[0035] Optionally, the generation of the optimized label sequence specifically includes:

[0036] Organize the process label sequence, extract the step number, action category and step duration of each record, set label decision variables for each record, and set corresponding state variables for the tools, equipment and protective elements involved in the on-site operation. Organize the label decision variables and state variables into a set of variables to describe the relationship between process and operation elements.

[0037] Based on the different specifications regarding tool use, equipment operation, and protection requirements in field operations, the variable set is constructed into Max-SAT global constraints, resulting in hard constraints and soft constraints clauses. Specifications that must be met are converted into hard constraint clauses, and specifications that are prioritized to be met under the premise of meeting hard constraints are converted into soft constraint clauses. Weight information is configured for each soft constraint clause, forming a Max-SAT constraint structure composed of hard constraints and weighted soft constraints.

[0038] The Max-SAT constraint structure is solved by systematically searching all possible combinations of label decision variables and various state variables. Combinations that do not satisfy the hard constraint clause set are eliminated. Among all combinations that satisfy the hard constraints, the total weight of the soft constraints is calculated based on the weight of each soft constraint. The combination with the largest total weight is selected as the Max-SAT solution. Based on the values ​​of the label decision variables, the process label records with true values ​​are selected and rearranged in chronological order to generate an optimized label sequence that meets the global consistency requirement.

[0039] Optionally, the transformation process of the structured labeled data specifically includes:

[0040] The optimized label sequence is organized, and fixed fields are set for each record according to the preset structured format. The fields are written into the corresponding fields to form structured label entries.

[0041] The structured annotation entries are arranged in chronological order according to the optimized label sequence, the order relationship between the entries is kept consistent, and the entries are merged to form a structured annotation dataset. At the same time, additional fields for representing time range, number of records and sequence integrity are added, and the structured annotation dataset is organized into a data structure that can be directly processed.

[0042] The structured labeled dataset is formatted and converted into a target format that can be read by the field operation management equipment. After formatting, the dataset is output so that the field operation management equipment can directly obtain and call the structured labeled data and display, store or schedule the operation process based on the data.

[0043] The beneficial effects of this invention are:

[0044] This invention introduces a multi-source weak label construction mechanism based on data programming and adopts an expectation-maximization truth inference method based on the Dawid-Skene model to jointly evaluate the reliability of visual recognition labels, rule base labels, and industry knowledge base labels, thereby obtaining a reinforced label sequence that is closer to the real situation. This effectively overcomes the problems of inconsistent weak labels and difficulty in evaluating the reliability of label sources in the prior art.

[0045] This invention uses a dynamic time warping method to match the enhanced label sequence with the standard operating procedure sequence, and combines a hidden semi-Markov model to calculate the duration of steps, so that the operation labels are consistent with the standard procedure in the time dimension, effectively solving the problem of operation process chaos caused by missed action detection, out-of-order and abnormal duration in the prior art.

[0046] This invention further introduces the Max-SAT global consistency solution strategy to unify various operational specifications such as tool use, equipment operation, and work protection into a global constraint model that includes hard and soft constraints. Under the premise of ensuring that the constraint rules must be met, the degree of satisfaction of soft constraints is maximized, thereby generating an optimized label sequence that conforms to industry standards, equipment requirements, and safety conditions, significantly improving the logical integrity and safety compliance of the labeled data.

[0047] This invention converts the optimized label sequence into structured labeled data and outputs it in a formatted manner, enabling on-site operation management equipment to directly call upon it. This achieves accurate recording, automatic analysis, and traceable management of the operation process, providing reliable and high-quality basic data for intelligent operation supervision, risk warning, and subsequent model training. Attached Figure Description

[0048] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0049] Figure 1 This is a flowchart of a field operation data acquisition and annotation method based on visual recognition proposed in this invention;

[0050] Figure 2 This is a schematic diagram of the algorithm structure of a field operation data acquisition and annotation method based on visual recognition proposed in this invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0052] refer to Figure 1-2 A method for field operation data collection and annotation based on visual recognition includes the following steps:

[0053] Acquire video image data of on-site operations, and perform data preprocessing including noise reduction, brightness correction, distortion correction and multi-frame enhancement to form a preprocessed image sequence;

[0054] The preprocessed image sequence is subjected to object detection, instance segmentation and task action recognition to obtain initial recognition results containing object category, action category and time location;

[0055] Based on the initial identification results, a multi-source weak label set is constructed, which is then fused to form a fused label sequence and input into an improved automatic labeling inference model. The improved automatic labeling inference model includes a multi-source weak label fusion layer, a job sequence alignment layer, and a global consistency optimization layer.

[0056] In the multi-source weak label fusion layer, the fused label sequence is subjected to weak label fusion processing to construct a set of weak label functions. The reliability of each weak label source is estimated by the expectation-maximization-based truth inference algorithm to generate a strengthened label sequence.

[0057] In the work sequence alignment layer, the enhanced label sequence is matched with the preset standard work process sequence. The dynamic time warping method is used to align the action sequences with different rhythms, omissions, and disordered orders. The hidden semi-Markov determination method is used to determine the duration distribution of the steps and generate the process label sequence.

[0058] In the global consistency optimization layer, a global constraint set is constructed based on the process label sequence, according to the tool co-occurrence rules, equipment usage rules, and operation protection rules. The satisfiability solution method is used to solve the global problem and generate an optimized label sequence that satisfies all constraints.

[0059] The optimized label sequence is converted into structured labeled data and then formatted before being output, so that the structured labeled data can be directly called by the on-site operation management equipment for use by on-site operation management personnel.

[0060] In this embodiment, the formation of the preprocessed image sequence specifically includes:

[0061] Acquire video image data of on-site operations, continuously collect data from the work area, split the acquired video into frame-by-frame images in chronological order to form an original image frame sequence, and uniformly manage and store the original image frame sequence;

[0062] Denoising and brightness correction are performed on each frame of the original image frame sequence, noise suppression is performed on the image pixels, and the brightness range of the entire frame image is redistributed to generate an intermediate image frame sequence that has undergone denoising and brightness correction. The intermediate image frame sequence is then subjected to order preservation processing to ensure inter-frame consistency.

[0063] Distortion correction and multi-frame enhancement are performed on the intermediate image frame sequence. The geometric distortion of each frame is corrected, and inter-frame fusion and image enhancement operations are performed using temporally adjacent multiple frames. The enhanced image frames are then recombined in chronological order to form a preprocessed image sequence.

[0064] In this embodiment, obtaining the initial identification result specifically includes:

[0065] The preprocessed image sequence is arranged in chronological order of acquisition time. Object recognition processing is performed on each frame of the image to obtain the different objects appearing in the current frame. Category identifiers and location parameters are generated for each object, forming object detection results organized by time index.

[0066] Based on the object detection results, region extraction processing is performed on each object in each frame of the image to generate a pixel region mask corresponding to the object in the image, and associate it with the object's category identifier and position parameters to form an object-level spatial description sequence.

[0067] The object-level spatial description sequence is integrated temporally to extract object change features between consecutive frames. Action recognition processing is performed on hotspot areas across time periods to obtain the action category identifier of the operation action and the corresponding start and end time positions. These are then matched with the object-level spatial description to generate initial recognition results.

[0068] In this embodiment, the formation of the fusion tag sequence specifically includes:

[0069] The initial recognition results are organized by classifying the object category, action category, and corresponding time and location information obtained from each frame of the image according to time order and object identifier, and generating an initial label sequence organized by time index and object index.

[0070] Based on the initial label sequence, rules related to object category, action category and time location are called from the pre-set rule base to generate corresponding weak labels for each initial label record. At the same time, knowledge items related to on-site operation process, equipment use and safety protection are retrieved from the pre-set industry knowledge base to generate corresponding weak labels for each initial label record. The labels obtained by visual recognition, weak labels from the rule base and weak labels from the knowledge base are combined to construct a multi-source weak label set.

[0071] Weak tag fusion processing is performed on the multi-source weak tag set. Weak tags from different sources corresponding to the same object and the same time period are uniformly encoded and merged to generate a fusion tag sequence arranged in chronological order.

[0072] The fused label sequence is input into an improved automatic labeling inference model for processing. The improved automatic labeling inference model includes a multi-source weak label fusion layer, a job sequence alignment layer, and a global consistency optimization layer.

[0073] In this embodiment, the generation of the enhancement tag sequence specifically includes:

[0074] In the multi-source weak label fusion layer, different labeling functions are defined for each label record in the fused label sequence. Each labeling function outputs a weak label for the same record based on different information in the visual recognition label, rule base and industry knowledge base entries. The output results of all labeling functions are associated with the corresponding labeling function identifiers to form a multi-source weak label set based on data programming.

[0075] Each label record in the multi-source weak label set is regarded as a sample. A hidden true label variable to be determined is set for each sample. A reliability parameter is set for each labeling function to represent the output of different weak labels under each true label. The hidden true label variables of all samples and the reliability parameters of all labeling functions are initialized.

[0076] The method employs Dawid-Skene expectation-maximization truth inference to iteratively update the latent true label variables and reliability parameters. In the expectation step of each iteration, the probability distribution of each sample on each candidate label is calculated based on the current reliability parameters. In the maximization step of each iteration, the number of times each labeling function outputs each weak label under different candidate labels is counted based on the current probability distribution, and the corresponding reliability parameters are updated. The update stops when the iteration meets a preset convergence condition. The preset convergence condition is that the parameter update magnitude is lower than a set threshold or the number of iterations reaches a set upper limit in two adjacent iterations. Based on the converged probability distribution, a single latent true label is determined for each sample, and the latent true labels of all samples are arranged in chronological order according to the fused label sequence to generate a reinforced label sequence.

[0077] This invention introduces a data-programming-based annotation function system into the multi-source weak label fusion layer and combines it with the Dawid-Skene expectation-maximization truth inference mechanism to achieve a comprehensive reliability assessment of visual recognition labels, rule-based labels, and knowledge labels. It can automatically infer the unified label that best reflects the real-world situation. This method significantly improves the accuracy and stability of weak label fusion, reduces interference from noisy labels, and effectively enhances the consistency, confidence, and usability of the generated reinforced label sequence, providing high-quality foundational data for subsequent alignment and global optimization processes.

[0078] In this embodiment, the generation of the process label sequence specifically includes:

[0079] In the work sequence alignment layer, the reinforcement label sequence is organized in chronological order. The corresponding action category and time position information are extracted from each label record in the reinforcement label sequence to form an action observation sequence arranged by time index. The standard operation process sequence is read from the pre-configured work specifications, and each standard step in the standard operation process sequence is numbered sequentially to form a reference step sequence arranged in step order.

[0080] Dynamic time warping is performed on the action observation sequence and the reference step sequence. The action records in the action observation sequence and the step records in the reference step sequence are read one by one. The matching cost between any action record and any reference step record is calculated. A cumulative cost table is constructed on the two-dimensional time-step plane. Path accumulation calculation is performed on all possible combinations of time steps and step indices. After the cumulative cost table calculation is completed, the alignment path with the minimum cumulative cost is selected from the start position to the end position. A correspondence is established between each action record in the action observation sequence and the step record in the reference step sequence.

[0081] For each step in the reference step sequence, the number of consecutively matched time positions in the alignment path is counted, and this number is regarded as the duration observation value of the current step. The duration determination method of the hidden semi-Markov model is adopted, and each step in the reference step sequence is regarded as a hidden state. A corresponding set of duration parameters is pre-set for each hidden state. Based on the duration observation value of each step, the probability value of each step at each duration is calculated according to the duration calculation rule of the hidden semi-Markov model. The corresponding duration estimation result is selected according to the calculated probability value. The step number of each step, the action category corresponding to the current step, and the duration estimation result are combined and arranged in the order of the reference steps to generate a process label sequence.

[0082] This invention achieves precise alignment between action sequences and standard operating procedures by introducing dynamic time warping and hidden semi-Markov duration calculation methods into the work sequence alignment layer. This effectively corrects problems such as out-of-order actions, missing steps, and abnormal durations. It strengthens the mapping of label sequences to the standard process in the time dimension, making the process label sequences more consistent with the logical order and duration of actual work steps. This improves the continuity, standardization, and stability of sequence labeling, providing a clear and time-reliable foundation for subsequent global consistency optimization.

[0083] In this embodiment, the generation of the optimized label sequence specifically includes:

[0084] Organize the process label sequence, extract the step number, action category and step duration of each record, set label decision variables for each record, and set corresponding state variables for the tools, equipment and protective elements involved in the on-site operation. Organize the label decision variables and state variables into a set of variables to describe the relationship between process and operation elements.

[0085] Based on the different specifications regarding tool use, equipment operation, and protection requirements in field operations, the variable set is constructed into Max-SAT global constraints, resulting in hard constraints and soft constraints clauses. Specifications that must be met are converted into hard constraint clauses, and specifications that are prioritized to be met under the premise of meeting hard constraints are converted into soft constraint clauses. Weight information is configured for each soft constraint clause, forming a Max-SAT constraint structure composed of hard constraints and weighted soft constraints.

[0086] The Max-SAT constraint structure is solved by systematically searching all possible combinations of label decision variables and various state variables. Combinations that do not satisfy the hard constraint clause set are eliminated. Among all combinations that satisfy the hard constraints, the total weight of the soft constraints is calculated based on the weight of each soft constraint. The combination with the largest total weight is selected as the Max-SAT solution. Based on the values ​​of the label decision variables, the process label records with true values ​​are selected and rearranged in chronological order to generate an optimized label sequence that meets the global consistency requirement.

[0087] This invention introduces a Max-SAT solution mechanism into the global consistency optimization layer, unifying multiple specifications between process labels and work elements into a logical structure with hard and soft constraints. It then utilizes a weighted soft constraint solution strategy to achieve the optimal combination of work specification compliance. This method maximizes overall specification matching while meeting mandatory work requirements, effectively avoiding problems such as tool usage conflicts, improper equipment operation, and inadequate protection. This results in a more accurate and reliable process label sequence in terms of global logic, safety requirements, and specification consistency, providing a highly reliable data foundation for subsequent work supervision and intelligent analysis.

[0088] In this embodiment, the conversion process of the structured annotation data specifically includes:

[0089] The optimized label sequence is organized, and fixed fields are set for each record according to the preset structured format. The fields are written into the corresponding fields to form structured label entries.

[0090] The structured annotation entries are arranged in chronological order according to the optimized label sequence, the order relationship between the entries is kept consistent, and the entries are merged to form a structured annotation dataset. At the same time, additional fields for representing time range, number of records and sequence integrity are added, and the structured annotation dataset is organized into a data structure that can be directly processed.

[0091] The structured labeled dataset is formatted and converted into a target format that can be read by the field operation management equipment. After formatting, the dataset is output so that the field operation management equipment can directly obtain and call the structured labeled data and display, store or schedule the operation process based on the data.

[0092] Example 1:

[0093] To verify the feasibility of this invention in practice, it was applied to the task of collecting and intelligently labeling on-site work behaviors in an industrial production scenario. In this scenario, on-site workers need to complete various complex operations, such as equipment start-up and shutdown, tool installation, parts replacement, and maintenance, within a limited work area. The work process involves multiple consecutive steps, each containing action details, equipment usage requirements, and protection restrictions. Due to significant changes in lighting conditions, complex background structures, and subtle and easily obscured movements of personnel, traditional work recording methods based on manual labeling or simple visual recognition are prone to problems such as inconsistent labeling, incorrect action sequence recognition, abnormal step durations, and tool co-occurrence conflicts, failing to meet the production management system's requirement for high-quality structured work data.

[0094] In practical applications, the first step is to use existing on-site video monitoring equipment to acquire a complete sequence of images of the work process. The data preprocessing method of this invention is then used to denoise, equalize brightness, correct distortion, and enhance multiple frames, making image details clearer and more stable, providing high-quality input for subsequent action recognition. Based on this, an initial recognition result is obtained through a visual recognition module, including personnel, tools, equipment, and action categories. Due to the complex on-site environment, relying solely on visual recognition is prone to false positives and false negatives. Therefore, this invention further constructs a multi-source weak label set, unifying the visual recognition results with rule base and industry knowledge base labels into a unified annotation function system. A Dawid-Skene-based truth inference method is then used to jointly estimate the reliability of the multi-source labels, ultimately obtaining a more reliable enhanced label sequence.

[0095] In the matching process between action sequences and work processes, this invention enhances the dynamic time warping of the tag sequence and the standard work process sequence, making the action sequence more closely resemble the actual process requirements in terms of time structure. Simultaneously, it uses a hidden semi-Markov model to infer the duration of each step, making the step time boundaries more accurate and avoiding problems often found in traditional methods, such as large deviations in step duration, missed steps, or unstable manual alignment. The process tag sequence generated in this way significantly improves temporal continuity, step completeness, and action-process correspondence.

[0096] To address the complex and multi-dimensional nature of operational requirements, this invention employs the Max-SAT solution method to construct global rule constraints. This method integrates tool usage relationships, equipment operation requirements, and protection conditions into a unified system of hard and soft constraints for solution. During the solution process, by maximizing the satisfaction of soft constraints, the final generated tag sequence not only meets the process specifications but also exhibits higher consistency in tool co-occurrence, equipment start-up and shutdown sequences, and protection action matching. After the global solution using Max-SAT, all non-compliant tags that cannot be accurately identified by the visual recognition module are effectively removed, resulting in an optimized tag sequence with higher stability and specification consistency.

[0097] To evaluate the practical effectiveness of this invention, the structured annotation data generated by this invention was compared with traditional visual recognition annotation methods. Specific experimental data are shown in Table 1.

[0098] Table 1 Comparison Data Table

[0099] Evaluation indicators Traditional visual recognition methods Method of the present invention Increase Labeling accuracy 82.4% 94.7% +12.3% Process timing consistency rate 76.1% 93.5% +17.4% Average deviation of step duration 3.6 seconds 1.2 seconds -2.4 seconds Tool co-occurrence conflict rate 8.3% 1.9% -6.4% Work specification compliance rate 84.5% 97.2% +12.7% Manual review time (per job) 9.5 minutes 3.1 minutes -6.4 minutes Labeled data availability 78.9% 96.4% +17.5%

[0100] As shown in Table 1, this invention significantly outperforms traditional visual recognition methods in on-site work annotation tasks: In terms of annotation accuracy, the traditional method achieves only 82.4%, while this invention, through multi-source weak label fusion and truth inference mechanisms, improves the accuracy to 94.7%. Regarding process sequence consistency, the traditional method achieves 76.1%, while this invention, through the combined effects of dynamic time warping and duration inference, improves it to 93.5%. In terms of step duration accuracy, the traditional method has an average deviation of 3.6 seconds, while this invention has an average deviation of only 1.2 seconds, significantly improving the time boundary offset problem. The tool co-occurrence conflict rate decreased from 8.3% in the traditional method to 1.9%, fully demonstrating the positive effect of Max-SAT global consistency optimization on work specification compliance; simultaneously, the overall work specification compliance rate of this invention reaches 97.2%, 12.7% higher than the traditional method. Regarding labor costs, this invention reduces the manual review time for a single task from 9.5 minutes to 3.1 minutes, and the availability of annotated data also increases from 78.9% to 96.4%.

[0101] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for on-site operation data collection and annotation based on visual recognition, characterized in that, Includes the following steps: Acquire video image data of the on-site operation, perform data preprocessing, and form a preprocessed image sequence; The preprocessed image sequence is subjected to object detection, instance segmentation, and task action recognition to obtain initial recognition results; Based on the initial identification results, a multi-source weak label set is constructed, which is then fused to form a fused label sequence and input into an improved automatic labeling inference model. The improved automatic labeling inference model includes a multi-source weak label fusion layer, a job sequence alignment layer, and a global consistency optimization layer. In the multi-source weak label fusion layer, the fused label sequence is subjected to weak label fusion processing to construct a set of weak label functions. The reliability of each weak label source is estimated by the expectation-maximization-based truth inference algorithm to generate a strengthened label sequence. In the work sequence alignment layer, the enhanced label sequence is matched with the preset standard work process sequence, and the duration distribution of the steps is determined by the hidden semi-Markov method to generate the process label sequence. In the global consistency optimization layer, a global constraint set is constructed based on the process label sequence, and a satisfiability solution method is used for global solution to generate an optimization label sequence that satisfies all constraints. The optimized label sequence is converted into structured labeled data and then formatted before being output.

2. The method for on-site operation data acquisition and annotation based on visual recognition according to claim 1, characterized in that, The formation of the preprocessed image sequence specifically includes: Acquire video image data of the on-site operation, continuously collect data of the operation area, and split the acquired video into frame-by-frame images in chronological order to form the original image frame sequence; Denoising and brightness correction are performed on each frame of the original image frame sequence, noise suppression is performed on the image pixels, and the brightness range of the entire frame image is redistributed to generate an intermediate image frame sequence. Distortion correction and multi-frame enhancement are performed on the intermediate image frame sequence. The geometric distortion of each frame is corrected, and inter-frame fusion and image enhancement operations are performed using temporally adjacent multiple frames. The enhanced image frames are then recombined in chronological order to form a preprocessed image sequence.

3. The method for on-site operation data acquisition and annotation based on visual recognition according to claim 1, characterized in that, The initial identification result is obtained specifically by: The preprocessed image sequence is arranged in chronological order of acquisition time. Object recognition processing is performed on each frame of the image to obtain the different objects appearing in the current frame. Category identifiers and location parameters are generated for each object, forming object detection results organized by time index. Based on the object detection results, region extraction processing is performed on each object in each frame of the image to generate a pixel region mask corresponding to the object in the image, and associate it with the object's category identifier and position parameters to form an object-level spatial description sequence. The object-level spatial description sequence is integrated temporally to extract object change features between consecutive frames. Action recognition processing is performed on hotspot areas across time periods to obtain the action category identifier of the operation action and the corresponding start and end time positions. These are then matched with the object-level spatial description to generate initial recognition results.

4. The method for on-site operation data acquisition and annotation based on visual recognition according to claim 1, characterized in that, The formation of the fusion tag sequence specifically includes: The initial recognition results are organized by classifying the object category, action category, and corresponding time and location information obtained from each frame of the image according to the time sequence and object identifier to generate an initial label sequence. Based on the initial label sequence, rules related to object category, action category and time location are called from the pre-set rule base. A corresponding weak label is generated for each initial label record. At the same time, knowledge entries are retrieved from the pre-set industry knowledge base to generate a corresponding weak label for each initial label record. The labels obtained by visual recognition, the weak labels from the rule base and the weak labels from the knowledge base are combined to construct a multi-source weak label set. Weak label fusion processing is performed on a multi-source weak label set. Weak labels from different sources corresponding to the same object and the same time period are uniformly encoded and merged to generate a fused label sequence. The fused label sequence is input into an improved automatic labeling inference model for processing. The improved automatic labeling inference model includes a multi-source weak label fusion layer, a job sequence alignment layer, and a global consistency optimization layer.

5. The method for on-site operation data acquisition and annotation based on visual recognition according to claim 1, characterized in that, The generation of the enhanced tag sequence specifically includes: In the multi-source weak label fusion layer, different labeling functions are defined for each label record in the fused label sequence. Each labeling function outputs a weak label for the same record based on different information in the visual recognition label, rule base and industry knowledge base entries. The output results of all labeling functions are associated with the corresponding labeling function identifiers to form a multi-source weak label set based on data programming. Each label record in the multi-source weak label set is regarded as a sample. A hidden true label variable to be determined is set for each sample. A reliability parameter is set for each labeling function to represent the output of different weak labels under each true label. The hidden true label variables of all samples and the reliability parameters of all labeling functions are initialized. The method employs Dawid-Skene expectation-maximization truth inference to iteratively update the latent true label variables and reliability parameters. In the expectation step of each iteration, the probability distribution of each sample on each candidate label is calculated based on the current reliability parameters. In the maximization step of each iteration, the number of times each labeling function outputs each weak label under different candidate labels is counted based on the current probability distribution, and the corresponding reliability parameters are updated. The update stops when the iteration meets the preset convergence condition. A single latent true label is determined for each sample based on the converged probability distribution, and the latent true labels of all samples are arranged in chronological order according to the fused label sequence to generate a reinforced label sequence.

6. The method for on-site operation data acquisition and annotation based on visual recognition according to claim 1, characterized in that, The generation of the process label sequence specifically includes: In the work sequence alignment layer, the reinforcement label sequence is organized in chronological order. The corresponding action category and time position information are extracted from each label record in the reinforcement label sequence to form an action observation sequence arranged by time index. The standard operation process sequence is read from the pre-configured work specifications, and each standard step in the standard operation process sequence is numbered sequentially to form a reference step sequence. Dynamic time warping is performed on the action observation sequence and the reference step sequence. The action records in the action observation sequence and the step records in the reference step sequence are read one by one. The matching cost between any action record and any reference step record is calculated. A cumulative cost table is constructed on the two-dimensional time-step plane. Path accumulation calculation is performed on all possible combinations of time steps and step indices. The alignment path with the minimum cumulative cost is selected from the start position to the end position. A correspondence is established between each action record in the action observation sequence and the step record in the reference step sequence. For each step in the reference step sequence, the number of consecutively matched time positions in the alignment path is counted. This number is considered as the duration observation value of the current step. The duration determination method of the hidden semi-Markov model is adopted. Each step in the reference step sequence is regarded as a hidden state. A corresponding set of duration parameters is pre-set for each hidden state. Based on the duration observation value of each step, the probability value of each step at each duration is calculated according to the duration calculation rule of the hidden semi-Markov model. The corresponding duration estimation result is selected according to the calculated probability value and combined to generate the process label sequence.

7. The method for on-site operation data acquisition and annotation based on visual recognition according to claim 1, characterized in that, The generation of the optimized label sequence specifically includes: Organize the process label sequence, extract the step number, action category and step duration of each record, set label decision variables for each record, and set corresponding state variables for the tools, equipment and protective elements involved in the on-site operation. Organize the label decision variables and state variables into a variable set. Based on the different specifications regarding tool use, equipment operation and protection requirements in on-site operations, the variable set is constructed into Max-SAT global constraints, resulting in hard constraint and soft constraint clauses. Weight information is then configured for each soft constraint clause to form a Max-SAT constraint structure. The Max-SAT constraint structure is solved by systematically searching the label decision variables and various state variables, filtering out value combinations that do not satisfy the hard constraint clause set. Among all value combinations that satisfy the hard constraints, the total weight of the soft constraints is calculated based on the weight of each soft constraint. The variable combination with the largest total weight is selected as the Max-SAT solution result. Based on the value of the label decision variables, the process label records with true values ​​are filtered and rearranged in chronological order to generate an optimized label sequence that meets the global consistency requirement.

8. The method for on-site operation data acquisition and annotation based on visual recognition according to claim 1, characterized in that, The transformation process of the structured labeled data specifically includes: The optimized label sequence is organized, and fixed fields are set for each record according to the preset structured format. The fields are written into the corresponding fields to form structured label entries. The structured labeled entries are arranged in chronological order according to the optimized label sequence, the order relationship between the entries is kept consistent, and the entries are merged to form a structured labeled dataset; Perform formatting on the structured labeled dataset, converting the dataset into a target format that can be read by field operation management equipment, and output the dataset after formatting is complete.

Images