Substation operation safety verification method and system based on multiple modes

Abstract

The invention discloses a substation operation safety verification method and device based on multiple modes, and the method comprises the steps: extracting coordinates of human body key points in a video stream, and converting the coordinates into a space-time skeleton diagram; obtaining a human body contour by using morphological filtering of image expansion and mask information, and generating a scene semantic feature map of a background region through a neural network; converting the space-time skeleton graph and the scene semantic feature graph into a scene description text through a Transform architecture; and comparing the similarity between the scene description text and the work ticket text through syntactic dependency analysis, and verifying and checking the operation process. The model is simple in structural design, large in optimization space and convenient to adjust and deploy in various application scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power automation, and in particular, relates to a multi-modal-based substation operation safety verification method and system. Background Art

[0002] Substations are a vital part of the power system, and their stable operation is directly related to the safety and stability of the entire system. Substations undertake key responsibilities such as voltage conversion and load switching. There are many devices inside substations, including transformers, mutual inductors, switchgear, and lightning protection equipment. Switchgear includes circuit breakers, disconnectors, load switches, high-voltage fuses and other components.

[0003] If the substation staff make mistakes in operation, it may cause casualties or equipment damage, and may cause power outages in a local area, which will seriously threaten the safe and stable operation of the power system and have a negative impact on social and economic activities and people's daily lives. Therefore, standardizing the operation process of substations and avoiding the operation of unauthorized equipment are necessary measures to improve the safety of substation operations.

[0004] The current traditional methods usually rely on on-site observations by staff or post-review of records, which is inefficient and susceptible to human factors. For example, long hours of work may lead to inattention and misjudgment. The original means of operating process monitoring often cannot achieve instant feedback and early warning, especially for the operation of unauthorized equipment and the detection of violations is not timely and accurate, which increases the risk of accidents. It may only focus on the status monitoring of specific objects (such as staff or key equipment) and ignore the influencing factors of the scene environment. For example, only monitoring the actions of operators without analyzing the semantic characteristics of their environment may lead to misjudgment of potential danger signals. In an environment such as a substation with dense equipment and complex operating processes, human supervision methods are prone to missing details or information overload, and the analysis of massive video surveillance data is inefficient. In addition, safety monitoring systems are often highly customized and costly to adjust, making it difficult to quickly adapt to new technologies and operational requirements, limiting their application scope and effect optimization potential in different scenarios. In short, traditional methods without the support of AI technology are difficult to achieve in-depth intelligent analysis and prediction, lack the ability to predict potential risks, and cannot provide advanced functions such as automatic verification of operational compliance.

[0005] Prior art document 1 (CN118485973A) provides a substation safety operation monitoring method and system based on OpenCV. The document performs image processing based on traditional opencv technology, and has limitations when dealing with complex scenes and diversified image data. It lacks the processing and verification functions of text information (such as work tickets, operating procedures, etc.), which limits the application of the system in comprehensive safety monitoring and verification. It lacks the ability to process multiple types of data (such as text, sensor data, etc.), which limits the application of multi-modal fusion and comprehensive analysis of multiple data.

[0006] Prior art document 2 (CN118470612A) provides a method and system for identifying the behavior of production workers based on multimodal information fusion. The document uses traditional convolutional neural networks and multi-layer perceptrons for feature extraction and behavior recognition. It mainly relies on human skeleton recognition and behavior recognition in videos. Although it involves multimodal information (video and skeleton data), the depth and breadth of fusion are relatively limited. The focus is on the generation of behavior recognition and assessment reports, but there is a lack of in-depth application of natural language processing. Although the assessment of behavioral standardization and accuracy is mentioned, there is no clear description of the processing mechanism when problems are found. Summary of the invention

[0007] In order to address the deficiencies in the prior art, the present invention applies multimodal processing of video streams, generates a spatiotemporal skeleton graph by monitoring key points of the human body, and combines the background semantic feature graph to use the attention mechanism to compare the behavior text description with the substation work ticket steps, automatically detect and warn of operational deviations, and ensure process compliance.

[0008] The first aspect of the present invention discloses a multi-modal-based substation operation safety verification method, comprising:

[0009] Extract the coordinates of key points of the human body in the video stream and convert them into a spatiotemporal skeleton graph;

[0010] The human body contour is obtained by using morphological filtering and mask information of image dilation, and the scene semantic feature map of the background area is generated by a neural network;

[0011] The spatiotemporal skeleton graph and scene semantic feature graph are converted into scene description text through the Transformer architecture;

[0012] The similarity between the scene description text and the work ticket text is compared through syntactic dependency analysis to verify and check the work process.

[0013] Preferably, the conversion space-time skeleton graph is specifically:

[0014] Analyze the key frames in the video and extract the coordinate information of the key points of the human body;

[0015] Encode key point information into a time series, extract features through convolution, and obtain a vector representation of human posture;

[0016] The spatial and temporal information in the human posture vector is extracted through a neural network to construct a spatiotemporal skeleton graph;

[0017] After the spatiotemporal skeleton graph is constructed, it is optimized through affinity calculation.

[0018] Preferably, the affinity calculation can be expressed by the following formula:

[0019]

[0020] Among them, v ij (s k ) is the kth vector between joint points i and j, u(s k ) is the kth unit vector, and N is the number of time points.

[0021] Preferably, the generation process of the scene semantic feature map is specifically as follows:

[0022] Use ResNet50 as the backbone network to extract the features of each frame of image;

[0023] Apply a global average pooling layer to compress the feature map into a one-dimensional vector;

[0024] The candidate regions are generated through the region proposal network, and the object category, bounding box and segmentation mask are simultaneously predicted through the branch network to obtain the scene semantic feature map after removing the human body.

[0025] Preferably, the global average pooling can be expressed by the following formula:

[0026]

[0027] Among them, c = 1, 2, ..., 2048, represents the index of the feature channel, H ′ and W ′ are the height and width of the feature map after ResNet50, F t is the feature map at each time point.

[0028] Preferably, the segmentation mask can be expressed by the following formula:

[0029]

[0030] Among them, M i Binary segmentation mask, is the set of real numbers.

[0031] Preferably, the similarity between the scene description text and the work ticket text can be expressed by the following formula:

[0032]

[0033] in, Describe the dependencies of text sentences for the scene, is the dependency relationship of the work ticket text sentences, It is the similarity between the scene description text sentence and the work ticket text sentence.

[0034] Preferably, the similarity can be expressed by the following formula:

[0035] Sim((gov1,dep1,rel1),(gov ′ 1,dep ′ 1,rel ′ 1))(26)

[0036] =α·Sim rel (rel1,rel ′ 1) +β·Sim gov (gov1,gov ′ 1)

[0037] +γ·Sim dep (dep1,dep ′ 1)+δ

[0038] Dist(gov1,dep1,gov ′ 1,dep ′ 1)

[0039] Among them, Sim rel 、Sim gov 、Sim dep They represent the similarity of dependency, main word and dependent word respectively, α, β, γ and δ are weight parameters, and Dist represents the distance similarity between word and dependent word.

[0040] Preferably, the operation process verification and checking are specifically as follows:

[0041] If the similarity Sim(S1, S2) between the scene description text and the work ticket text reaches the set threshold, it is considered that the scene description text and the work ticket text are matched successfully, and the system continues to monitor the status of the staff;

[0042] If the similarity does not reach the threshold, the system attempts to switch the work ticket to the next item and rematch;

[0043] If no match is found in any of the items on the work ticket, the system will issue an alarm to indicate that the current work is not being carried out according to the work ticket process.

[0044] The second aspect of the present invention discloses a multi-modal-based substation operation safety verification device, comprising:

[0045] Key point monitoring module, used to generate spatiotemporal skeleton graph based on key points of human body;

[0046] A scene extraction module is used to extract image features and generate a scene semantic feature map;

[0047] A text generation module is used to generate scene description text based on the spatiotemporal skeleton graph and the semantic feature graph;

[0048] Work monitoring module, used to monitor and check the work process.

[0049] The beneficial effect of the present invention is that, compared with the prior art, in the multimodal substation operation safety verification method and device, in the video text generation model based on the spatiotemporal skeleton graph and the scene semantic feature graph, the extraction module of the scene semantic feature graph can simultaneously capture the deep semantic information of the object and the scene. Compared with the traditional technology that only focuses on the object, the present invention understands the video content more comprehensively and accurately. The video text generation model adopts a seq2seq module based on the transformer attention mechanism to achieve high-precision video-to-text conversion, and its performance advantage is significant. The model has excellent scalability and can be trained on a larger data set to further improve the model performance and adapt to the needs of more application scenarios. The semantic segmentation method based on the scene feature extraction module shows high-precision performance that exceeds traditional technology in image processing tasks, which is attributed to the automatic feature learning ability of the deep learning framework and the effective use of large-scale data. The model structure design of the present invention is simple and has a large optimization space, which is convenient for adjustment and deployment in a variety of application scenarios, ensuring the standardization, safety and efficiency of the substation operation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a diagram of the architecture of the substation operation safety verification system based on multi-mode;

[0051] Figure 2 It is a flow chart of the safety verification method in the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0053] The first embodiment of the present invention is as follows Figure 2 As shown, a multi-modal substation operation safety verification method is provided, including:

[0054] Step 1: Human spatiotemporal skeleton graph and contour extraction: Capture the video stream through the front-end camera or video camera, and use the pre-trained deep learning model to detect the human posture and convert it into a spatiotemporal skeleton graph.

[0055] In a preferred but non-limiting embodiment of the present invention, step 1 specifically comprises:

[0056] Step 1.1, analyze the key frames in the video and extract the coordinate information of the key points of the human body. The coordinate information of the key points is expressed in a two-dimensional coordinate system. For each frame, the human posture feature extraction algorithm detects the key points of the human body and expresses them as two-dimensional coordinates, which can be expressed by the following formula:

[0057] P t ={(x1,y1),(x2,y2),…,(x k ,y k )} (1)

[0058] Among them, P t represents the set of key points of the human body at time point t, (x i ,y i ) represents the two-dimensional coordinates of the i-th key point, and k is the number of key points of the human body.

[0059] Step 1.2: Encode the key point information obtained in step 1.1 into a time series P1, P2, ..., P T , where T is the total number of time steps. Then, the human posture feature extraction algorithm is used as a key point detector to extract the spatiotemporal skeleton of the human body. The coordinates of these key points are extracted through convolution operations and finally embedded as vectors, which can be expressed by the following formula:

[0060] v t =f(W*P t +b) (2)

[0061] Among them, v t is the vector representation of the human body posture at time point t, W is the convolution kernel, * represents the convolution operation, b is the bias term, and f is the activation function.

[0062] The final human posture vector v at each time point t They will be combined into a spatiotemporal skeleton graph for subsequent processing.

[0063] Step 1.3, the module structure is two multi-stage convolutional neural networks, one of which is used to extract spatial information from human posture. The first multi-stage convolutional neural network is used for confidence calculation of the scoring graph, and the second multi-stage convolutional neural network is used for affinity calculation. Each multi-stage convolutional neural network uses a three-layer convolutional neural network, with 64 convolution kernels in each layer. Given the human posture at each time point as input, the spatial information is extracted through the convolutional neural network to obtain a matrix of shape (N, 64), where N represents the number of time points. The convolution operation can be expressed by the following formula:

[0064] h l+1 =f(W l *h l +b l ) (3)

[0065] Among them, h l is the input feature map of the lth layer, W l is the convolution kernel of the lth layer, * indicates the convolution operation, b l is the bias and f is the activation function.

[0066] Next, a bidirectional long short-term memory network (BiLSTM) is used to model the time dimension, with the input being the human posture at each time point, capturing the timing information, and obtaining a matrix of shape (N, 256). Finally, the human posture vectors are concatenated together to form a vector of shape (256,) as the representation of the spatiotemporal skeleton graph.

[0067] After the spatiotemporal skeleton graph is constructed, especially for scenes with multiple people and complex actions, affinity calculation is used to optimize the spatial relationship and matching relationship in order to quantify the correlation between the joints of the human body in the spatiotemporal skeleton graph. In this step, for any two joint positions i and j, the linear integral of affinity is calculated to characterize the confidence of the skeleton point pair, which can be expressed by the following formula:

[0068]

[0069] Among them, A ij represents the affinity between joint points i and j, v ij (s) is the vector between joint points i and j, u(s) is the unit vector, and L is the distance between the two points.

[0070] In order to quickly calculate the integral, the uniform sampling method is used to approximate the affinity calculation, which can be expressed by the following formula:

[0071]

[0072] Among them, v ij (s k) is the kth vector between joint points i and j, u(s k ) is the kth unit vector.

[0073] In the multi-person detection scenario, by taking the key points as the vertices of the graph and the affinity as the edge weights of the graph, a bipartite graph matching problem is constructed. The optimal matching can be expressed by the following formula:

[0074]

[0075] Among them, w ij is the affinity between joint points i and j, x ij is the indicator variable of the matching result.

[0076] Step 2: Based on the spatiotemporal skeleton graph established in step 1, the image is subjected to dilated morphological filtering, and the contour information of the human body is obtained by edge detection within the width of the skeleton line, and a human body mask is generated. Combining the mask and the original image, a convolutional neural network-based target detection and segmentation model is used to perform semantic segmentation on the objects in the image or video, and the background area after the human body is removed is obtained for subsequent scene semantic analysis.

[0077] In a preferred but non-limiting embodiment of the present invention, step 2 specifically comprises:

[0078] Step 2.1, first use ResNet50 as the backbone network to extract the features of each frame. Assume that the input is the image X at each time point t , the input dimension is C×H×W, where C is the number of channels, and H and W are the height and width of the image, respectively.

[0079] After extracting features through ResNet50, the feature map obtained is in, is a set of real numbers, N is the number of video frames, 2048 is the number of feature channels, H ′ and W ′ They are the height and width of the feature map after ResNet50.

[0080] Step 2.2: Apply the Global Average Pooling (GAP) layer to compress the feature map into a one-dimensional vector. The specific operation is to perform the feature map F at each time point. t Perform global average pooling. It can be expressed by the following formula:

[0081]

[0082] The operation of GAP is to perform average pooling on each feature channel in the global spatial dimension, namely:

[0083]

[0084] Where c = 1, 2, ..., 2048, represents the index of the feature channel. In this way, the feature vector corresponding to each time point t is obtained

[0085] Step 2.3, segmentation and object detection. Generate candidate regions through the region proposal network (RPN), and predict the object category, bounding box and segmentation mask through the branch network. For each candidate region R i , the network outputs an object category C i 、Location information B i and a binary segmentation mask M i .

[0086] Mask prediction predicts the pixel-level boundaries of the target area, that is, generates a segmentation mask for each candidate area, which can be expressed as follows:

[0087]

[0088] This mask represents the pixel-level distribution of the target object in the area and is used to accurately mark the boundaries of the target.

[0089] Scene semantic feature sequence generation. By applying feature extraction to each frame in the video stream, the extracted feature vector sequence can be expressed as G = {G1, G2, …, G N}, where each It is the scene semantic feature at the corresponding time point.

[0090] This set of feature vector sequences can describe the changes of objects and scenes in the video stream, and can be further used as input for various downstream tasks such as action recognition, scene understanding, etc.

[0091] As a prominent substantive feature of the present invention and one of the significant advances it brings to the prior art, the present invention utilizes morphological filtering and mask information of image dilation to obtain the human body contour, and generates a scene semantic feature map of the background area through a neural network, thereby ensuring a comprehensive understanding of the working environment.

[0092] Step 3: Convert the spatiotemporal skeleton graph obtained in step 1 and the scene semantic feature graph obtained in step 2 into scene description text using the Transformer architecture in the Seq2Seq model. The input is the spatiotemporal skeleton graph obtained in step 1 and the scene semantic feature graph obtained in step 2, which are processed by the encoder and decoder respectively.

[0093] Assume that the input sequence of the spatiotemporal skeleton graph is represented as:

[0094] X=[x1,x2,…,x T ] (10)

[0095] The input sequence of scene semantic feature map is represented as:

[0096] Y=[y1,y2,…,y T′ ] (11)

[0097] Among them, x i and i are the feature representations of the spatiotemporal skeleton graph and the scene semantic feature graph at each time step, T and T ′ is the length of the input sequence.

[0098] The encoder first processes the spatiotemporal skeleton graph. The encoder consists of L layers of Transformers. For the lth layer, the input It is processed through a multi-head self-attention mechanism and a feed-forward neural network.

[0099] In the self-attention mechanism, for each attention head i, the input vector H (l) is linearly transformed into query Q (i) , key K (i) Sum value V (i) , which can be expressed by the following formula:

[0100]

[0101] Then calculate the attention weight, which can be expressed as follows:

[0102]

[0103] Among them, A (i) is the weight of the i-th attention head, d k for…….

[0104] The attention output can be expressed as follows:

[0105] Z (i) =A (i) V (i) (14)

[0106] Then, the outputs of all heads are concatenated and transformed linearly, which can be expressed as follows:

[0107] MultiHead(H (l) )=Concat(Z (1) ,Z (2) ,…,Z (h) )W O (15)

[0108] Each layer output of the encoder needs to be further processed by the forward neural network, which can be expressed as follows:

[0109] FFN(x)=ReLU(xW1+b1)W2+b2 (16)

[0110] This step is applied to the output at each time step.

[0111] The decoder receives the scene semantic feature map as input and gradually generates the scene description text in combination with the intermediate vector representation generated by the encoder. The decoder input Y = [y1, y2, ..., y T′ ] is also processed through the self-attention mechanism.

[0112] The self-attention mechanism of the decoder is the same as that of the encoder. First, the input vector To calculate the query, key, and value, the following formula can be used:

[0113]

[0114] in, is the query vector of the decoder, is the key vector of the decoder, is the value vector of the decoder.

[0115] Then, the self-attention weights are calculated and the attention output is generated.

[0116] The decoder not only relies on the self-attention mechanism, but also needs to interact with the encoder’s output. The encoder’s output H and the decoder’s representation It is input into the attention mechanism of the encoder-decoder and can be expressed as follows:

[0117]

[0118] in, is the encoder-decoder query vector, is the key vector of the encoder-decoder, is the encoder-decoder value vector.

[0119] The encoder-decoder attention weights can be expressed as follows:

[0120]

[0121] The output can be expressed as follows:

[0122]

[0123] in, Output for multi-head attention.

[0124] The multi-head attention output is processed by splicing, which can be expressed as follows:

[0125]

[0126] The decoder will eventually generate the probability distribution of words in the vocabulary through the softmax layer at each time step, which can be expressed as follows:

[0127]

[0128] in, is the output of the last layer of the decoder, and W and b are the parameters used to generate the probability distribution.

[0129] The current semantics can be obtained by combining the data (X, Y) of the entire video sequence with the encoded feature vector H before the current time step. The overall image information of the video can be combined to analyze the time dimension of the operation behavior, so that the semantic information of the operation behavior can be output in sequence to facilitate operation verification.

[0130] As one of the outstanding substantive features of the present invention and the significant progress brought to the prior art, the present invention can realize multimodal data fusion and comprehensive analysis. Through the Transformer architecture and text generation based on the attention mechanism, these technologies have higher performance and flexibility in processing complex multimodal information and generating natural language descriptions. Combining the spatiotemporal skeleton graph and the scene semantic feature graph, these image information is converted into scene description text through the Transformer architecture, which integrates more semantic information and spatiotemporal features and provides a more comprehensive scene understanding. Through the text generation module based on the attention mechanism, natural language descriptions are generated from the spatiotemporal skeleton graph and the scene semantic feature graph, which improves the accuracy of semantic understanding and text generation, and efficiently verifies the operation process through syntactic dependency analysis to ensure the accuracy and safety of the operation. Its streamlined model structure design and large optimization space are convenient for flexible adjustment and deployment in a variety of application scenarios, which improves the versatility and practicality of the system. In addition, the method can also significantly improve the accuracy and efficiency of security monitoring, while reducing system resource consumption and usage costs, making the system more economical and efficient when deployed on a large scale.

[0131] Step 4: Use syntactic dependency analysis to compare the similarity between the scene description text and the work ticket text. If the match is successful, continue to monitor the worker status; otherwise, switch the work ticket to the next item or issue an alarm. The specific process is as follows:

[0132] Perform syntactic dependency analysis on the scene description text and the work ticket text respectively to obtain the dependency structure tree of each sentence. Let the sentence of the scene description text be S1 and the sentence of the work ticket text be S2. We can obtain the dependency tree of the two sentences through syntactic analysis and

[0133] Dependencies are extracted from the dependency tree of each sentence, and the dependency of each sentence is represented as a set of triples, which can be expressed by the following formula:

[0134] (gov,dep,rel)(23)

[0135] in:

[0136] gov indicates the main word in a sentence, such as a noun or a verb.

[0137] dep means dependent word, usually a modifier.

[0138] rel indicates the type of dependency relationship, such as subject-predicate relationship, attributive relationship, etc.

[0139] For sentences S1 and S2, the extracted dependencies are represented as sets of triples, which can be expressed by the following formula:

[0140]

[0141] in, is the dependency relation of sentence S1, is the dependency relationship of sentence S2, gov1 and gov ′ 1 represents the first main word in sentence S1 and sentence S2, dep1 and dep ′ 1 represents the first dependent word in sentence S1 and sentence S2, rel1 and rel ′ 1 represents the type of the first dependency relation in sentence S1 and sentence S2 respectively.

[0142] Similarity calculation. For each triple, calculate the sentence dependency similarity. The similarity calculation is based on the following features:

[0143] Dependency type similarity: compare whether rel1 and rel2 are the same.

[0144] Parts of speech of principal and dependent words: Comparing gov and gov ′ The parts of speech (such as noun, verb) and dep and dep ′ Part of speech.

[0145] The distance between the main word and the dependent word: Calculate the distance between gov and dep and compare it with gov ′ and dep′ Distance comparison.

[0146] ″′

[0147] Calculate the similarity of two triples Sim((gov1,dep1,rel1),(gov1,dep1,rel1)), which can be expressed by the following formula:

[0148] Sim((gov1,dep1,rel1),(gov ′ 1,dep ′ 1,rel ′ 1))(25)

[0149] =α·Sim rel (rel1,rel ′ 1) +β·Sim gov (gov1,gov ′ 1)

[0150] +γ·Sim dep (dep1,dep ′ 1)+δ

[0151] Dist(gov1,dep1,gov ′ 1,dep ′ 1)

[0152] Among them, Sim rel 、Sim gov 、Sim dep They represent the similarity of dependency, main word and dependent word respectively, α, β, γ and δ are weight parameters, and Dist represents the distance similarity between word and dependent word.

[0153] Finally, the similarity between two sentences S1 and S2 is calculated by weighting the similarity of all triples, which can be expressed as follows:

[0154]

[0155] In work ticket matching and process monitoring, if the similarity Sim(S1, S2) reaches the set threshold, it is considered that the scene description text matches the work ticket text successfully, and the system continues to monitor the status of the staff.

[0156] If the similarity does not reach the threshold, the system attempts to switch the work ticket to the next item and rematch.

[0157] If no match is found in any of the items on the work ticket, the system will issue an alarm to indicate that the current work is not being carried out according to the work ticket process.

[0158] As one of the outstanding substantive features of the present invention and the significant progress brought to the prior art, the present invention adopts multimodality and Transformer architecture, which can process multiple data types (such as images and text). The spatiotemporal skeleton graph and the scene semantic feature graph are converted into scene description text through the Transformer architecture, and the similarity between the scene description text and the work ticket text is compared using syntactic dependency analysis. This method not only enhances scene understanding, but also improves the interpretability and operability of the system. It is clearly stated that when the scene description text does not match the current work ticket step, the system will alarm, and promptly remind operators and managers to prevent potential safety hazards, ensure the standardization, safety and efficiency of the substation operation process, reduce the possibility of human errors, and improve the reliability of the operation process.

[0159] Moreover, through multimodal data processing and advanced semantic analysis technology, the system can more accurately identify and analyze the safety status of the working area, improve the accuracy and efficiency of safety monitoring, and meet the safety verification requirements of different environments and needs.

[0160] Therefore, the method proposed in the present invention can intelligently match the work ticket text with the actual scene and dynamically monitor the workflow.

[0161] The second embodiment of the present invention provides a substation operation safety verification system based on multi-mode, such as Figure 1 As shown, including:

[0162] Key point monitoring module, used to generate spatiotemporal skeleton graph based on key points of human body;

[0163] A scene extraction module is used to extract image features and generate a scene semantic feature map;

[0164] A text generation module is used to generate scene description text based on the spatiotemporal skeleton graph and the scene semantic feature graph;

[0165] Work monitoring module, used to monitor and check the work process.

[0166] As one of the outstanding substantive features of the present invention and a significant improvement over the prior art, the system of the present invention achieves high intelligence and automation through deep learning models and natural language processing technology, and is adaptable to complex practical application scenarios. The multimodal fusion of spatiotemporal skeleton graphs and scene semantic feature graphs enhances the recognition and analysis capabilities of the system. The generated natural language description text and alarm mechanism improve the user-friendliness and operability of the system, enabling operators and managers to more intuitively understand and handle problems in the work process.

[0167] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-modal substation operation safety verification method, comprising the following steps: Extract the coordinates of key points of the human body in the video stream and convert them into a spatiotemporal skeleton graph; The human body contour is obtained by using morphological filtering and mask information of image dilation, and the scene semantic feature map of the background area is generated by a neural network; The spatiotemporal skeleton graph and scene semantic feature graph are converted into scene description text through the Transformer architecture; By comparing the similarity between the scene description text and the work ticket text, the work process is verified and checked.

2. A multi-modal substation operation safety verification method according to claim 1, characterized in that: The transformation of the spatiotemporal skeleton graph includes: Analyze the key frames in the video and extract the coordinate information of the key points of the human body; Encode key point information into a time series, extract features through convolution, and obtain a vector representation of human posture; The spatial and temporal information in the human posture vector is extracted through a neural network to construct a spatiotemporal skeleton graph; After the space-time skeleton graph is constructed, the spatial relationship and matching relationship in the space-time skeleton graph are optimized through a neural network.

3. A multi-modal substation operation safety verification method according to claim 2, characterized in that: The spatiotemporal skeleton graph is optimized by affinity calculation, which can be expressed by the following formula: Among them, v ij (s k ) is the kth vector between joint points i and j, u(s k ) is the kth unit vector, and N is the number of time points.

4. The multi-modal substation operation safety verification method according to claim 1 is characterized in that: The generation process of scene semantic feature map includes: Use ResNet50 as the backbone network to extract the features of each frame of image; Apply a global average pooling layer to compress the feature map into a one-dimensional vector; The candidate regions are generated through the region proposal network, and the object category, bounding box and segmentation mask are simultaneously predicted through the branch network to obtain the scene semantic feature map after removing the human body.

5. A multi-modal substation operation safety verification method according to claim 4, characterized in that: Global average pooling can be expressed as follows: Among them, c = 1, 2, ..., 2048, represents the index of the feature channel, H ′ and W ′ are the height and width of the feature map after ResNet50, F t is the feature map at each time point.

6. A multi-modal substation operation safety verification method according to claim 4, characterized in that: The segmentation mask can be expressed as follows: Among them, M i Binary segmentation mask, is the set of real numbers.

7. The multi-modal substation operation safety verification method according to claim 1 is characterized in that: The similarity between the scene description text and the work ticket text can be expressed by the following formula: in, Describe the dependencies of text sentences for the scene, is the dependency relationship of the work ticket text sentences, It is the similarity between the scene description text sentence and the work ticket text sentence.

8. A multi-modal substation operation safety verification method according to claim 7, characterized in that: Dependencies between sentences in scene description text Dependencies with the work ticket text sentences It can be expressed by the following formula: in, is the dependency relation of sentence S1, is the dependency relationship of sentence S2, gov1 and gov ′ 1 represents the first main word in sentence S1 and sentence S2, dep1 and dep ′ 1 represents the first dependent word in sentence S1 and sentence S2, rel1 and rel ′ 1 represents the type of the first dependency relation in sentence S1 and sentence S2 respectively.

9. The multi-modal substation operation safety verification method according to claim 1, characterized in that: The operation process verification and checking includes: If the similarity between the scene description text and the work ticket text reaches the set threshold, it is considered that the scene description text and the work ticket text are matched successfully, and the status of the worker continues to be monitored; If the similarity does not reach the set threshold, try to switch the work ticket to the next item and rematch; If no match is successful in all items of the work ticket, an alarm will be issued to indicate that the current work is not being carried out according to the work ticket process.

10. A multi-modal substation operation safety verification system, running a multi-modal substation operation safety verification method according to any one of claims 1 to 9, characterized in that: include: Key point monitoring module, used to generate spatiotemporal skeleton graph based on key points of human body; A scene extraction module is used to extract image features and generate a scene semantic feature map; A text generation module is used to generate scene description text based on the spatiotemporal skeleton graph and the semantic feature graph; Work monitoring module, used to monitor and check the work process.

Images