A power operation scene recognition method based on multi-modal feature fusion

Abstract

The application discloses a power operation scene recognition method based on multi-modal feature fusion, comprising the following steps: inputting an operation image frame into a pre-trained target detection model to obtain a detection result of key scene elements and perform object-level feature coding to obtain an object-level detection feature matrix; performing character recognition and field analysis on a work ticket image, and obtaining a work ticket feature matrix through a pre-trained text coding model; inputting the operation image frame into a pre-trained visual coding model to obtain an image modal feature matrix; inputting the object-level detection feature matrix, the work ticket feature matrix and the image modal feature matrix into a multi-modal fusion multi-label classification model, adopting a multi-modal fusion mechanism based on double cross attention to obtain a multi-modal fusion feature matrix, and inputting the multi-modal fusion feature matrix into a multi-label classifier after global average pooling to output prediction probabilities of various power operation safety supervision scenes. The application has the advantages of strong multi-modal fusion, high recognition accuracy, good interpretability and good real-time performance.

Description

Technical Field

[0001] This invention belongs to the field of power safety monitoring and artificial intelligence image recognition technology, specifically a power operation scene recognition method based on multimodal feature fusion. Background Technology

[0002] Safety supervision of power field operations has become an important part of ensuring the stable operation of the power grid and preventing personal accidents. Traditional safety supervision mainly relies on manual inspections, on-site spot checks, or subjective judgment by video monitoring personnel. This method has problems such as high manpower consumption, poor real-time performance, and low identification accuracy, which makes it difficult to meet the current development needs.

[0003] In recent years, computer vision-based intelligent recognition technology has been widely applied in the analysis of power operation scenarios. By detecting elements such as workers, protective equipment, and construction machinery in surveillance videos, it is possible to initially achieve automatic identification of some violations. However, this type of method generally has the following shortcomings: (1) Single visual modality, lacking semantic prior. Relying solely on monitoring images or video streams for judgment, it is unable to understand the work background and type. For example, when the camera detects that a person is not wearing goggles, the system has difficulty distinguishing whether the scene belongs to welding work, thus generating false alarms.

[0004] (2) Insufficient understanding of target relationships. Simple target detection results cannot accurately express the spatial topological relationships between objects, such as whether a person is located under the crane arm or crosses the safety fence, which makes it difficult to identify violations.

[0005] (3) Work order information is not being used effectively. Power work orders usually record information such as work name, work category, work location, work equipment and time, which contain rich task semantics and are an important basis for judging the rationality of the work. However, existing methods generally do not integrate this text information and lack a unified model of "task semantics-visual evidence".

[0006] (4) It is difficult to identify multiple safety supervision scenarios at the same time. In actual operation, multiple safety supervision and inspection requirements may be involved at the same time. For example, the same site needs to carry out supervision of not entering under the crane boom during hoisting operations, and also needs to carry out supervision of wearing safety helmets. The traditional single-label classification model cannot identify multiple safety supervision scenarios at the same time.

[0007] Therefore, it can be seen that the existing power operation safety identification system is still in the stage of single-modality and shallow detection, lacking a unified identification framework that can simultaneously integrate image, detection and text information, making it difficult to achieve accurate and real-time identification of multiple safety supervision scenarios in complex operating environments.

[0008] Therefore, there is an urgent need for a multimodal feature fusion and recognition method that integrates monitoring images, key scene element detection results, and work order text to achieve a deep semantic understanding of power operation scenarios and improve the accuracy, interpretability, and real-time response capability of safety supervision scenario recognition. Summary of the Invention

[0009] To address the shortcomings of existing technologies in single-modal recognition and contextual understanding, this invention proposes a power operation scenario recognition method based on multimodal feature fusion. This method extracts and integrates semantic features from three types of information: video surveillance images, target detection results, and work order text. It constructs a unified multimodal semantic embedding space to achieve automatic recognition of power operation safety supervision scenarios.

[0010] This invention adopts the following technical solution: a method for identifying power operation scenarios based on multimodal feature fusion, comprising the following steps: S1: Real-time acquisition of operation image frames generated by monitoring at the power operation site, and simultaneous acquisition of work ticket images corresponding to the operation tasks; The acquired data is matched with timestamps and task numbers to realize the association between operation image frames and work ticket images; S2, input the operation image frame into the pre-trained target detection model to obtain the category, bounding box coordinates and detection confidence of key scene elements at the power operation site, and perform object-level feature encoding to obtain the object-level detection feature matrix; S3, perform text recognition and field parsing on the work order image, extract the structured fields of the text information, and obtain the work order feature matrix after each field is encoded by a pre-trained text encoding model; S4. Input the image frame of the task into the pre-trained visual coding model, perform global visual feature coding on the entire image, and obtain the image modal feature matrix; S5, the object-level detection feature matrix, work order feature matrix and image modality feature matrix are input into the multimodal fusion multi-label classification model. A multimodal fusion mechanism based on dual cross-attention is adopted. The fusion process takes the image modality feature matrix as the core query and fuses the object-level detection feature matrix and work order feature matrix in two stages. Furthermore, through a multi-head self-attention network and a feedforward network, the contextual dependencies between different image blocks are integrated to obtain the multimodal fusion feature matrix. The multimodal fusion feature matrix constitutes a unified semantic embedding representation. S6 inputs the multimodal fusion feature matrix into a multi-label classifier after global average pooling, and outputs the predicted probability of the power operation safety supervision scenario.

[0011] By integrating monitoring images, target detection results, and work order text information, this invention can achieve deep semantic understanding of power operation scenarios and multi-label safety supervision scenario recognition within a unified semantic embedding space, thereby effectively improving the accuracy and interpretability of scenario recognition.

[0012] This invention utilizes a pre-trained visual coding model to extract global visual features from monitoring images. It uses the detection results (category, bounding box coordinates, and detection confidence) of key scene elements at the power operation site as object-level feature input, preserving the category and geometric location information of objects such as personnel, machinery, and safety equipment. This provides a foundation for the subsequent fusion module to learn the spatial relationships between objects. Simultaneously, it extracts structured fields such as work content, equipment, and safety requirements from the work order text. After encoding by a pre-trained text coding model, the work order features, object-level detection features, and image modal features are input into a multimodal fusion multi-label classification model for fusion. This invention, through a dual cross-attention mechanism centered on image modal features, can fuse object-level detection features and work order features in stages, achieving unified modeling of global visual information, object-level detection information, and work order task semantics. Furthermore, a multi-label classifier is used to output the predicted probabilities of multiple safety supervision scenarios, thereby enabling the identification of "supervision scenario of wearing safety goggles during welding or cutting operations", "supervision scenario of prohibiting entry under the boom during hoisting operations", "supervision scenario of preventing crossing of safety fences", "supervision scenario of wearing safety helmets" and "supervision scenario of operating procedures during power outages".

[0013] Furthermore, the multimodal fusion multi-label classification model is obtained through training before executing steps S1 to S6; the training adopts a multi-label supervised learning framework, and combines contrastive constraints and prior constraints to construct training objectives and optimize the model parameters involved in training, so as to enhance the robustness and generalization ability of the multimodal fusion multi-label classification model.

[0014] Furthermore, the training loss of the multimodal fusion multi-label classification model is a weighted sum of the multi-label binary cross-entropy loss, the mismatch comparison loss, and the work ticket semantic prior consistency loss.

[0015] This invention not only improves the robustness and real-time performance of recognition in complex environments, but also effectively reduces the model's dependence on text shortcuts by introducing mismatch comparison loss and work ticket semantic prior consistency loss, thereby enhancing the credibility and interpretability of the recognition results. It provides a multi-source fusion, scalable and practically valuable technical solution for intelligent safety supervision of power systems.

[0016] Furthermore, the formula for calculating the multi-label binary cross-entropy loss is as follows:

[0017] in, For multi-label binary cross-entropy loss, To determine the number of categories of safety supervision scenarios, For the first Real-world labels for safety supervision scenarios; For multi-label classifiers to the first Predicted probabilities for security supervision scenarios; These are class weights used to balance imbalanced samples; The mismatch ratio loss The calculation formula is as follows:

[0018] in, This refers to the batch sample size. For the first The fusion feature embedding of a correctly matched sample Embedding of positive samples obtained from the same work scenario after image enhancement or submodal enhancement. To make the first The first task image frame and the first The mismatched sample embedding obtained by combining mismatched work ticket images Represents cosine similarity. Indicates the temperature coefficient; Based on the job type and task semantics extracted from the work order, a prior vector for the applicability of safety supervision scenarios is generated. Furthermore, a consistency constraint is imposed on the prediction results of the multimodal fusion multi-label classification model, resulting in the semantic prior consistency loss of the work ticket. :

[0019]

[0020] in, Indicates the first Prior values ​​for the applicability of similar safety supervision scenarios to the current operational tasks. ; It is the numerical stability constant; The joint optimization objective of the multimodal fusion multi-label classification model L for:

[0021] In the formula, , , They are respectively , , The weight.

[0022] Furthermore, in step S1, each frame of the work image includes the workers, construction machinery, and safety protection facilities; the work order image includes text information about the work content, work equipment, and work safety requirements.

[0023] Furthermore, in step S2, the detection result B obtained by the pre-trained object detection model is:

[0024] in, Indicates the number of targets detected; For the index of the detection results, that is... ; For the first The categories of the targets; For the first The bounding box coordinates of each target. , , , They represent the first The x-coordinate, y-coordinate, width, and height of each target bounding box; For the first Confidence level of each target; To preserve the category and geometric location information of each detected object, and to provide a foundation for the subsequent fusion module to learn the spatial relationships between objects, the detection results are converted into object-level token representations. :

[0025] in, Embed a lookup table for the categories of trainable parameters; It is a multilayer perceptron structure; for The normalization result; This indicates that the feature vectors are concatenated along their dimensions; this encoding process yields the object-level detection feature matrix.

[0026] in, For real numbers, This indicates the uniform feature dimension used for modal features before they enter the dual cross-attention module. This is the object-level detection feature matrix.

[0027] Furthermore, in step S3, the work ticket feature matrix for:

[0028] in, This indicates that the number of text parsing fields related to the job scenario obtained is 3; Indicate the content of the assignment; Indicates the operating equipment; This represents a field-level semantic vector obtained by encoding and pooling the job content, equipment, and safety requirements using a pre-trained text encoding model. This represents a pre-trained text Transformer encoding model used for semantic encoding of structured field text in work orders. The model encodes three fields: job content, job equipment, and job safety requirements, and pools the output token features corresponding to each field to obtain field-level semantic vectors.

[0029] Furthermore, in step S4, the image modal feature matrix for:

[0030] in, This represents a pre-trained visual encoding model; Represents the image frame of the task; Indicates the first Semantic features of each image patch; for The model represents the number of features in an image patch.

[0031] Furthermore, in step S5, the specific steps of multimodal feature fusion include: First, the object-level detection feature matrix, work order feature matrix, and image modality feature matrix are linearly mapped and aligned in the feature space using the feature mapping module, so that all three features form a dimension of [missing information]. A unified feature representation; Secondly, the first cross-modal attention module uses the image modal feature matrix as the query and the object-level detection feature matrix as the key and value to calculate the first cross-modal attention output feature matrix. Then, the first cross-modal attention output feature matrix and the image modal feature matrix are added together by residual addition, and the residual addition result is layer normalized to obtain the first fused feature matrix that integrates object-level information. Then, the second cross-modal attention module uses the first fusion feature matrix as the query and the work ticket feature matrix as the key and value to calculate the second cross-modal attention output feature matrix; next, the second cross-modal attention output feature matrix is ​​added to the first fusion feature matrix by residual addition, and the residual addition result is layer normalized to obtain the second fusion feature matrix; Finally, the second fusion feature matrix integrates the contextual dependencies between different image patches through a multi-head self-attention and feedforward network to obtain a multimodal fusion feature matrix.

[0032] Furthermore, in step S6, the probability prediction result... The calculation formula is as follows:

[0033]

[0034] in, This is a multimodal fusion feature matrix; This is a global average pooling operation; and All are learnable parameters; The function is a sigmoid function with an output dimension of 5, corresponding to five safety supervision scenarios: "supervision scenario of wearing safety goggles during welding or cutting operations", "supervision scenario of prohibiting entry under the boom during hoisting operations", "supervision scenario of preventing crossing of safety fences", "supervision scenario of wearing safety helmets", and "supervision scenario of operating procedures during power outages".

[0035] Compared with traditional single-modal detection methods, the present invention has the following advantages: (1) Cross-modal semantic fusion: By comprehensively utilizing video images and work order text, a consistent understanding of visual information and work semantics can be achieved; (2) Strong multi-label recognition capability: Supports the discrimination of multiple safety supervision scene categories coexisting in the same operation image; (3) Mismatch suppression and prior constraints: By comparing mismatches and the prior loss of the ticket, the misjudgment caused by text bias and mismodal matching is significantly reduced; (4) Good interpretability: The detection objects that contribute more to the classification results are determined based on the first cross-modal attention weight, and the corresponding weights are mapped to the target bounding box; the work ticket fields that contribute more to the classification results are determined based on the second cross-modal attention weight, thereby generating the classification basis and interpretation information for the safety supervision scenario.

[0036] (5) Highly deployable: Supports real-time detection and low-latency inference, and can be directly embedded into the power operation safety supervision and control platform.

[0037] This invention extracts semantic features from three types of information—video surveillance images, target detection results, and work order text—and integrates them to achieve multi-label recognition and intelligent judgment in power operation safety supervision scenarios. It has the advantages of strong multimodal fusion, high recognition accuracy, good interpretability, and excellent real-time performance. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of the power operation safety supervision scenario identification process of the present invention; Figure 2 This is a work site diagram of the application of this invention; Figure 3 This is a scanned image of the work order when the present invention is applied. Detailed Implementation

[0040] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0041] This embodiment presents a method for identifying power operation scenarios based on multimodal feature fusion, such as... Figure 1 As shown, the steps are as follows: S1, Data Acquisition and Preprocessing: Deploying surveillance cameras at the power operation site to acquire and monitor operation image frames in real time. (i.e., monitoring images), and simultaneously obtain work ticket images corresponding to the operation tasks from the safety supervision and control platform; the collected data is matched with timestamps and task numbers to realize the association between operation image frames and work ticket images.

[0042] Each frame of the work image contains various objects such as workers, construction machinery, and safety protection facilities; the work order image contains text information such as the work content, work equipment, and work safety requirements.

[0043] S2, Target Detection and Object-Level Feature Encoding: Input the operation image frame into the pre-trained target detection model to obtain the detection results of the category, bounding box coordinates and detection confidence of key scene elements at the power operation site; perform object-level feature encoding on the detection results to obtain the object-level detection feature matrix.

[0044] Specifically, the process involves inputting operation image frames into a pre-trained YOLOv8 object detection model to identify key scene elements at the power operation site. These key scene elements include categories such as "person," "welding torch," "goggles," "crane," "crane boom," "safety fence," "safety helmet," "power pole," and "voltage testing pole," resulting in a set of detection results.

[0045] in, Indicates the number of targets detected; For the index of the detection results, that is... ; For the first The categories of the targets; For the first The bounding box coordinates of each target; For the first The confidence level of each objective.

[0046] To improve robustness to images at different resolutions, the coordinates of each bounding box obtained above are adjusted according to the image size. Normalization is performed to obtain dimensionless geometric vectors:

[0047] To preserve the category and geometric location information of each detected object, and to provide a foundation for the subsequent fusion module to learn the spatial relationships between objects, the detection results are converted into object-level token representations:

[0048] in, Embed a lookup table for the categories of trainable parameters; It is a multilayer perceptron structure; for The normalization result; This indicates that the feature vectors are concatenated along their dimensions; this encoding process yields the object-level detection feature matrix.

[0049] in, For real numbers, This indicates the uniform feature dimension used for modal features before they enter the dual cross-attention module. This is an object-level detection feature matrix. The object-level detection feature matrix is ​​used to characterize the category information, geometric location information, and detection confidence of each key scene element in the image, and provides a foundation for the subsequent fusion module to learn the spatial relationships between objects.

[0050] S3, Work Ticket Text Structuring and Feature Extraction: Perform text recognition and field parsing on the work ticket image to extract structured fields containing text information such as work content, work equipment, and work safety requirements. Each field is encoded by a pre-trained text encoding model to obtain the work ticket feature matrix.

[0051] Specifically, this involves: collecting the work order image. The input table is processed by the OCR module for text recognition and field parsing, extracting information including the assignment content. Operating equipment Work safety requirements Three fields. The text results are normalized using a rule dictionary and synonym mapping to remove noise and irrelevant information. Each field is constructed as a text sequence in the format "field name: field value" and input into a pre-trained text Transformer encoding model. The output token features corresponding to each field are pooled to obtain field-level semantic vectors, which in turn form the work ticket feature matrix.

[0052] in, This indicates that the number of text parsing fields related to the job scenario obtained is 3.

[0053] The resulting work ticket feature matrix It can reflect the semantic background of the current task and provide semantic priors for subsequent scene recognition.

[0054] S4, Visual Feature Extraction: Input the image frame of the task into the pre-trained visual coding model, perform global visual feature coding on the entire image, and obtain the image modality feature matrix.

[0055] Specifically, this refers to the operational image frames obtained from the surveillance camera. Input the pre-trained visual ViT Transformer encoding model to perform global visual feature encoding on the entire image, and obtain the image modality feature matrix:

[0056] in, This represents a pre-trained visual Transformer encoding model; Represents the image frame of the task; Represents the semantic features of each image patch; for The algorithm represents the number of features in an image patch. This indicates the uniform feature dimension used by the modal features before entering the dual cross-attention module.

[0057] The output of this step is the image modality feature matrix. It represents the visual information of the current image and provides global visual semantic support for subsequent scene recognition.

[0058] S5, Multimodal Feature Fusion: The object-level detection feature matrix, work ticket feature matrix, and image modal feature matrix are input into the multimodal fusion multi-label classification model. A multimodal fusion mechanism based on dual cross-attention is adopted. The image modal feature matrix is ​​used as the core query. The object-level detection feature matrix and work ticket feature matrix are fused in two stages. Furthermore, the contextual dependencies between different image patches are integrated through a multi-head self-attention network and a feedforward network to obtain the multimodal fusion feature matrix.

[0059] To make full use of , , To leverage the complementary information of the three features, this embodiment employs a multimodal fusion mechanism based on dual cross-attention. The fusion process uses image modal features as the core query, fusing object-level detection features and work order features in two stages to achieve unified modeling of global visual information, object-level detection information, and work order task semantics.

[0060] (1) The object-level detection feature matrix, work order feature matrix, and image modality feature matrix are linearly mapped and aligned in the feature space through the feature mapping module, so that all three features form a dimension of The unified feature representation.

[0061] (2) Let the image modality feature matrix be the query and the object-level detection feature matrix be the key and value, calculate:

[0062] in, These are trainable weights.

[0063] The first cross-modal attention module is used to calculate the first cross-modal attention output feature matrix. :

[0064] in, Represents the image modal feature matrix With object-level detection feature matrix New features after fusion; This represents the soft maximization operator; express Transpose of; This represents the dimension of the key vector in each attention head.

[0065] The first cross-modal attention output feature matrix is ​​added to the image modality feature matrix by residual summation, and the result of residual summation is layer normalized to obtain the first fused feature matrix:

[0066] (3) Using the first fusion feature matrix For querying, the work ticket feature matrix For the key and value, calculate:

[0067] in, These are trainable weights.

[0068] The second cross-modal attention module is used to compute the second cross-modal attention output feature matrix. ,have:

[0069] In the formula, express Transpose of; This represents the dimension of the key vector in each attention head.

[0070] The residuals of the second cross-modal attention output feature matrix and the first fused feature matrix are added together, and the result of the residual addition is layer normalized to obtain the second fused feature matrix:

[0071] Second fusion feature matrix Fusion , , Complementary information from the three features.

[0072] (4) Further, through a multi-head self-attention network and a feedforward network, context modeling and nonlinear feature transformation are performed on different image patches in the second fusion feature matrix. The calculation process is as follows:

[0073]

[0074] in, This represents a multi-head attention structure where the query, key, and value are all derived from the second fusion feature matrix. They were obtained through different linear mappings; This represents a feedforward neural network; This represents the intermediate features obtained after multi-head attention, residual connections, and layer normalization. This is the result of multimodal fusion feature matrix calculation.

[0075] S6, Multi-label Classification and Scene Recognition: The multi-modal fusion feature matrix is ​​input into the multi-label classifier after global average pooling, and the probability prediction results of various power operation safety supervision scenarios are output.

[0076] Probability prediction results The calculation formula is as follows:

[0077]

[0078] in, This is a multimodal fusion feature matrix; This is a global average pooling operation; and All are learnable parameters; The output is a sigmoid function with 5 dimensions, corresponding to five safety supervision scenarios: "Supervision of wearing safety goggles during welding or cutting operations," "Supervision of prohibiting entry under crane booms during hoisting operations," "Supervision of preventing crossing of safety fences," "Supervision of wearing safety helmets," and "Supervision of operating procedures during power outages." This embodiment uses a multi-label output method, which can simultaneously identify the coexistence of multiple safety supervision scenario categories in the same operation image.

[0079] For any work scenario to be identified, the model input includes the current work image frame, the target detection result obtained from that image frame, and the structured fields of the work order matched according to the task number and time information. The work order features characterize the work content, work equipment, and work safety requirements; the object-level detection features provide the category, location, and confidence level of key scene elements; and the image modal features provide personnel posture, equipment status, and overall spatial relationships. The model outputs the predicted probability of each safety supervision scenario category by jointly analyzing task semantics, object-level evidence, and global visual evidence.

[0080] Specifically, the monitoring scenario for wearing safety goggles during welding or cutting operations is determined by the semantics of the welding or cutting task, as well as visual evidence such as personnel, welding torches, or cutting tools. Whether safety goggles are detected and their positional relationship with the personnel's heads can serve as the basis for subsequent wearing status checks, but are not necessary conditions for determining whether the monitoring scenario is applicable. The monitoring scenario for prohibiting entry under the boom during hoisting operations is determined by the semantics of the hoisting task and key scene elements such as the crane and boom. The monitoring scenario for preventing crossing of safety fences is determined by the isolation safety requirements in the work order and key scene elements such as safety fences and personnel. The monitoring scenario for wearing safety helmets is determined by the semantics of the on-site work task and key scene elements such as the workers. Whether safety helmets are detected and their positional relationship with the personnel's heads can serve as the basis for subsequent safety helmet wearing status checks, but are not necessary conditions for determining whether the monitoring scenario is applicable. The semantics of the work order are used to characterize the relevance of the scene inspection items to the current work task, and do not directly indicate that a violation has occurred. The final identification result is determined by the semantics of the work order task and visual evidence in the work image.

[0081] Step S7: Training the multimodal fusion multi-label classification model. This training process is completed before steps S1 to S6. The training adopts a multi-label supervised learning framework and combines contrastive constraints and prior constraints to construct training objectives and optimize the model parameters involved in the training. A training sample set is constructed, and each training sample includes a job image frame, structured text of the work order matched according to the task number and time information, and a five-dimensional multi-label ground truth label; the ground truth label's... Each component being set to 1 indicates that the current work image and its corresponding work order together constitute the first... In security monitoring scenarios, setting the value to 0 indicates that the current sample does not belong to the first category. In safety supervision scenarios, samples are grouped according to job task numbers, and samples corresponding to different job tasks are divided into training sets, validation sets, and test sets to avoid adjacent image frames of the same job task appearing in different data subsets at the same time.

[0082] In this invention, the pre-trained target detection model, pre-trained text encoding model, and pre-trained visual encoding model serve as pre-feature extraction models, the table OCR module serves as a work ticket structuring processing module, the object-level feature encoding module serves as a pre-feature encoding module for the target detection results, and the multimodal fusion multi-label classification model includes a feature mapping module, a first cross-modal attention module, a second cross-modal attention module, a multi-head self-attention and feedforward network, a global average pooling module, and a multi-label classifier.

[0083] The pre-trained object detection model is used to extract object category, bounding box coordinates, and detection confidence from the job image frame. The object-level feature encoding module is used to encode the object detection results to obtain the object-level detection feature matrix. The table OCR module is used to extract structured text fields from the work order image. The pre-trained text encoding model is used to encode the structured text fields to obtain the work order feature matrix. The pre-trained visual encoding model is used to perform global visual feature encoding on the job image frame to obtain the image modality feature matrix.

[0084] The pre-trained object detection model, pre-trained text encoding model, and pre-trained visual encoding model serve as pre-feature extraction models, and their parameters remain fixed during training. The object-level feature encoding module is randomly initialized and optimized, as well as the feature mapping module, the first cross-modal attention module, the second cross-modal attention module, the multi-head self-attention network, the feedforward network, and the multi-label classifier in the multi-modal fusion multi-label classification model.

[0085] For each batch of training samples, object-level detection feature matrix, work order features, and image modality features are obtained sequentially. Forward propagation is completed through dual cross-attention, multi-head self-attention and feedforward networks, and a multi-label classifier. Matched job images and work orders are considered positively matched samples. Mismatched work orders are selected from other work orders with corresponding task numbers but whose job type or safety requirement semantics are inconsistent with the current job task. The current job image is then combined with these mismatched work orders to form a mismatched sample. Work orders with different task numbers but the same task semantics are not considered strong mismatched samples. Multi-label binary cross-entropy loss, mismatch comparison loss, and work order semantic prior consistency loss are calculated based on the ground truth labels, positively matched samples, mismatched samples, and work order semantic prior, respectively.

[0086] The model training loss is a weighted sum of the multi-label binary cross-entropy loss, the mismatch comparison loss, and the work ticket semantic prior consistency loss.

[0087] (1) Multi-label binary cross-entropy loss The main loss is in the form of multi-label binary cross-entropy:

[0088] in, For multi-label binary cross-entropy loss, To determine the number of categories of safety supervision scenarios, For the first Real-world labels for safety supervision scenarios; For multi-label classifiers to the first Predicted probabilities for security supervision scenarios; These are class weights used to balance imbalanced samples; This loss is calculated by averaging the samples within the batch and the safety supervision scenario categories. During training, the multi-label classifier outputs the classification logit without Sigmoid processing, and the classification logit is directly input into BCEWithLogitsLoss for calculation; during inference, the classification logit is then converted into the predicted probability of each safety supervision scenario through the Sigmoid function to avoid repeatedly performing the Sigmoid operation.

[0089] (2) Mismatched ratio loss

[0090] To avoid the model relying too heavily on a single modality, a mispairing ratio loss is introduced. :

[0091] in, This refers to the batch sample size. For the first The fusion feature embedding of a correctly matched sample Embedding of positive samples obtained from the same work scenario after image enhancement or submodal enhancement. To make the first The first task image and the first Embedding of mismatched samples obtained by combining mismatched work tickets; Indicates cosine similarity; The value represents the temperature coefficient, which is 0.07. This loss encourages correctly matched job image-work order combinations and their semantically invariant augmented views to move closer to each other in the feature space, while keeping job image-work order combinations that do not match the task semantics further apart, thereby suppressing the model's reliance on single modalities or incorrect text matching.

[0092] (3) Work order semantic prior consistency loss Based on the job type and task semantics extracted from the work order, a prior vector for the applicability of safety supervision scenarios is generated. The applicability prior vector is used to represent the applicability or relevance of each safety supervision scenario to the current task, and does not directly represent the probability of violation, and imposes a consistency constraint on the model prediction results.

[0093]

[0094] in, Indicates the first The applicability or relevance of the safety supervision scenario to the current operational task. ; This represents the numerical stability constant. In this embodiment, the applicability prior value of the safety supervision scenario related to the current task is 0.7, and the applicability prior value of the safety supervision scenario unrelated to the current task is 0.3; when the work order field is missing or the relevance cannot be determined, the corresponding scenario does not participate in the calculation of the semantic prior consistency loss of the work order.

[0095] (4) Joint optimization objectives

[0096] in, , , These are the weighting coefficients for each loss term.

[0097] In one specific embodiment, the input image size for the object detection model is 640×640, and the input image size for the pre-trained visual encoding model is 224×224. The maximum text length for the work order field is 256 tokens, and the feature dimension is unified. The number of parameters is 256, with 8 cross-modal attention heads; training uses the AdamW optimizer, and the batch size is [missing value]. The initial learning rate is The weight decay coefficient is 0.05, and the maximum number of training rounds is 80; the numerical stability constant of the work ticket semantic prior consistency loss is... The temperature coefficient of the ratio loss due to mispairing is The weight coefficients for the multi-label binary cross-entropy loss, mispairing comparison loss, and work ticket semantic prior consistency loss are respectively... , , .

[0098] The model parameters used in training are updated via backpropagation; training stops when 80 training epochs are reached, or when the joint loss on the validation set fails to decrease for 20 consecutive epochs. The classification threshold for each security monitoring scenario category is preset to 0.5; when the... When the predicted probability of a security-related regulatory scenario is not lower than the corresponding threshold of 0.5, output the first... Results of identifying safety supervision scenarios.

[0099] The following application demonstrates the power operation scene recognition method based on multimodal feature fusion of this invention: a low-voltage power supply operation site is selected as the test scenario. First, a surveillance camera deployed at the operation site acquires current operation image frames, such as... Figure 2 As shown; simultaneously, based on the task number and time, the work order image corresponding to the current task is retrieved from the safety supervision and control platform, such as... Figure 3 As shown, the association between the job image frame and the work order image is established.

[0100] Will Figure 2 The image frame shown is input into a pre-trained object detection model to identify key scene elements such as people, safety helmets, safety fences, and other relevant on-site objects. Based on the category, bounding box coordinates, and detection confidence of each key scene element, object-level feature encoding is performed to obtain an object-level detection feature matrix. Figure 2 It is evident that the presence of safety fences and workers at the work site provides object-level visual evidence for identifying scenarios involving the prevention of crossing safety fences and the monitoring of helmet wearing. The detected helmets and their positional relationship with the workers' heads can be further used to determine the helmet wearing status.

[0101] Will Figure 3The work order image shown is input into a table OCR module for text recognition and field parsing, extracting structured fields such as work content, work equipment, and work safety requirements. These structured fields are then input into a pre-trained text encoding model to obtain the work order feature matrix. The work order features are used to characterize the task semantics of the current low-voltage power supply operation and the corresponding safety management requirements, providing a semantic basis for determining whether inspection items such as safety fences and safety helmets are applicable to the current work task.

[0102] At the same time, Figure 2 The image frames of the operation shown are input into a pre-trained visual encoding model to perform global visual feature encoding on the workers, the site environment, the fence layout, and the overall spatial relationships between objects, obtaining an image modality feature matrix. Subsequently, the object-level detection feature matrix, the work order feature matrix, and the image modality feature matrix are input into a multimodal fusion multi-label classification model. Through a dual cross-attention mechanism, based on the image modality feature matrix, the object-level detection feature matrix and the work order feature matrix are fused sequentially to obtain a multimodal fusion feature matrix. Finally, after global average pooling and a multi-label classifier, the predicted probabilities of each safety supervision scenario are output.

[0103] In this embodiment, the model outputs two safety supervision scenario categories: "Safety Fence Anti-Crossing Supervision Scenario" and "Safety Helmet Wearing Supervision Scenario." The "Safety Fence Anti-Crossing Supervision Scenario" indicates that a safety fence exists at the current work site, and nearby workers are present, requiring supervision and inspection of whether personnel have crossed the safety fence. The "Safety Helmet Wearing Supervision Scenario" indicates that the current work task and site environment require supervision and inspection of workers' helmet-wearing status. The above output indicates that the corresponding safety supervision scenario applies to the current work task and does not directly indicate that workers have committed violations such as crossing the fence or not wearing safety helmets.

[0104] according to Figure 2 The on-site images shown indicate that the workers are actually wearing safety helmets, and safety barriers have been set up on-site. Therefore, this embodiment is mainly used to verify that the present invention can accurately identify the safety supervision scenario involved in the current operation by combining the semantics of the work order and on-site visual evidence, providing a scenario basis for subsequent continuous monitoring, risk warning, and safety management regarding safety barrier crossing status and safety helmet wearing status.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0106] This specification and accompanying drawings are merely illustrative examples of the present invention and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the present invention. Clearly, those skilled in the art can make various alterations and modifications to the present invention without departing from its scope. Therefore, if such modifications and variations fall within the scope of the present invention and its equivalents, the present invention intends to include these modifications and variations.

Claims

1. A method for identifying power operation scenarios based on multimodal feature fusion, characterized in that, Including the following steps: S1: Real-time acquisition of operation image frames generated by monitoring at the power operation site, and simultaneous acquisition of work ticket images corresponding to the operation tasks; The acquired data is matched with timestamps and task numbers to realize the association between operation image frames and work ticket images; S2, input the operation image frame into the pre-trained target detection model to obtain the category, bounding box coordinates and detection confidence of key scene elements at the power operation site, and perform object-level feature encoding to obtain the object-level detection feature matrix; S3, perform text recognition and field parsing on the work order image, extract the structured fields of the text information, and obtain the work order feature matrix after each field is encoded by a pre-trained text encoding model; S4. Input the image frame of the task into the pre-trained visual coding model, perform global visual feature coding on the entire image, and obtain the image modal feature matrix; S5, the object-level detection feature matrix, work ticket feature matrix and image modality feature matrix are input into the multimodal fusion multi-label classification model. A multimodal fusion mechanism based on dual cross-attention is adopted. The image modality feature matrix is ​​used as the core query. The object-level detection feature matrix and work ticket feature matrix are fused in two stages. The contextual dependencies between different image patches are further integrated through a multi-head self-attention network and a feedforward network to obtain the multimodal fusion feature matrix. S6 inputs the multimodal fusion feature matrix into a multi-label classifier after global average pooling, and outputs the probability prediction results for various power operation safety supervision scenarios.

2. The power operation scene recognition method according to claim 1, characterized in that, The multimodal fusion multi-label classification model is obtained through training before executing steps S1 to S6; the training adopts a multi-label supervised learning framework, and combines contrastive constraints and prior constraints to construct training objectives and optimize the model parameters involved in the training.

3. The power operation scene recognition method according to claim 2, characterized in that, The training loss of the multimodal fusion multi-label classification model is a weighted sum of the multi-label binary cross-entropy loss, the mismatch comparison loss, and the work ticket semantic prior consistency loss.

4. The power operation scene recognition method according to claim 3, characterized in that, The formula for calculating the multi-label binary cross-entropy loss is as follows: in, For multi-label binary cross-entropy loss, To determine the number of categories of safety supervision scenarios, For the first Real-world labels for safety supervision scenarios; For multi-label classifiers to the first Predicted probabilities for security supervision scenarios; These are class weights used to balance imbalanced samples; The mismatch ratio loss The calculation formula is as follows: in, This refers to the batch sample size. For the first The fusion feature embedding of a correctly matched sample This refers to the embedding of positive samples obtained from the same power operation scenario after image enhancement or submodal enhancement. To make the first The first task image frame and the first The mismatched sample embedding obtained by combining mismatched work ticket images Represents cosine similarity. Indicates the temperature coefficient; Based on the job type and task semantics extracted from the work order image, a prior vector for the applicability of safety supervision scenarios is generated. Furthermore, a consistency constraint is imposed on the prediction results of the multimodal fusion multi-label classification model, resulting in the semantic prior consistency loss of the work ticket. : in, Indicates the first Prior values ​​for the applicability of similar safety supervision scenarios to the current operational tasks. ; It is the numerical stability constant; The joint optimization objective of the multimodal fusion multi-label classification model for: In the formula, , , They are respectively , , The weight.

5. The power operation scene recognition method according to any one of claims 1-3, characterized in that, In step S1, each frame of the work image contains the workers, construction machinery, and safety protection facilities; the work order image contains text information about the work content, work equipment, and work safety requirements.

6. The power operation scene recognition method according to any one of claims 1-3, characterized in that, In step S2, the detection result B obtained by the pre-trained object detection model is: in, Indicates the number of targets detected; For the index of the detection results, that is... ; For the first The categories of the targets; For the first The bounding box coordinates of each target. , , , They represent the first The x-coordinate, y-coordinate, width, and height of each target bounding box; For the first Confidence level of each target; To preserve the category and geometric location information of each detected object, and to provide a foundation for the subsequent fusion module to learn the spatial relationships between objects, the detection results are converted into object-level token representations. : in, Embed a lookup table for the categories of trainable parameters; It is a multilayer perceptron structure; for The normalization result; This indicates that the feature vectors are concatenated along their dimensions; this encoding process yields the object-level detection feature matrix. in, For real numbers, This indicates the uniform feature dimension used for modal features before they enter the dual cross-attention module. This is the object-level detection feature matrix.

7. The power operation scene recognition method according to claim 6, characterized in that, In step S3, the work ticket feature matrix for: in, This indicates that the number of text parsing fields related to the job scenario obtained is 3; Indicate the content of the assignment; Indicates the operating equipment; This represents a field-level semantic vector obtained by encoding and pooling the job content, equipment, and safety requirements using a pre-trained text encoding model. This represents a pre-trained text Transformer encoding model used for semantic encoding of structured field text in work orders. The model encodes three fields: job content, job equipment, and job safety requirements, and pools the output token features corresponding to each field to obtain field-level semantic vectors.

8. The power operation scene recognition method according to claim 7, characterized in that, In step S4, the image modal feature matrix for: in, This represents a pre-trained visual encoding model; Represents the image frame of the task; Indicates the first Semantic features of each image patch; for The model represents the number of features in an image patch.

9. The power operation scene recognition method according to claim 1, characterized in that, In step S5, the specific steps of multimodal feature fusion include: First, the object-level detection feature matrix, work order feature matrix, and image modality feature matrix are linearly mapped and aligned in the feature space using the feature mapping module, so that all three features form a dimension of [missing information]. A unified feature representation; Secondly, the first cross-modal attention module uses the image modal feature matrix as the query and the object-level detection feature matrix as the key and value to calculate the first cross-modal attention output feature matrix. Then, the first cross-modal attention output feature matrix and the image modal feature matrix are added together by residual addition, and the residual addition result is layer normalized to obtain the first fused feature matrix that integrates object-level information. Then, the second cross-modal attention module uses the first fusion feature matrix as the query and the work ticket feature matrix as the key and value to calculate the second cross-modal attention output feature matrix; next, the second cross-modal attention output feature matrix is ​​added to the first fusion feature matrix by residual addition, and the residual addition result is layer normalized to obtain the second fusion feature matrix; Finally, the second fusion feature matrix integrates the contextual dependencies between different image patches through a multi-head self-attention and feedforward network to obtain a multimodal fusion feature matrix.

10. The power operation scene recognition method according to any one of claims 1-3, characterized in that, In step S6, the probability prediction result The calculation formula is as follows: in, This is a multimodal fusion feature matrix; This is a global average pooling operation; and All are learnable parameters; The function is a sigmoid function with an output dimension of 5, corresponding to five safety supervision scenarios: "supervision scenario of wearing safety goggles during welding or cutting operations", "supervision scenario of prohibiting entry under crane boom during hoisting operations", "supervision scenario of preventing crossing of safety fences", "supervision scenario of wearing safety helmets", and "supervision scenario of operating procedures during power outages".

Images