A method for abnormal behavior recognition based on spatiotemporal graph convolutional network

Abstract

The present invention discloses an abnormal behavior recognition method based on a spatiotemporal graph convolutional network, which belongs to the field of target detection technology, and comprises S1, collecting monitoring videos of operation and maintenance personnel at work site, inputting them into a target detection algorithm to identify and mark the detection targets to obtain detection videos; S2, inputting the detection video into a posture estimation algorithm to calculate and obtain a skeleton graph of the detection targets; S3, constructing a behavior spatiotemporal graph based on the skeleton graph, and dividing the behavior spatiotemporal graph into a training atlas and a test atlas; S4, training a behavior recognition model improved according to an attention mechanism based on the training atlas; S5, inputting the test atlas into the trained behavior recognition model to identify whether there are abnormalities in the behavior of operation and maintenance personnel at the work site, and giving an alarm when the behavior of the operation and maintenance personnel is abnormal; the present invention recognizes and detects the behavior of the operation and maintenance personnel by analyzing the spatiotemporal characteristics of the monitoring videos at the work site, and significantly improves the accuracy of identifying abnormal behaviors of the operation and maintenance personnel at the work site.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to an abnormal behavior recognition method based on a spatiotemporal graph convolutional network. Background Art

[0002] In the traditional operation and maintenance process, safety operation hazards are discovered, eliminated and avoided through standardized operation training for operation and maintenance personnel, self-supervision of operation and maintenance personnel, and manual supervision and observation through video surveillance systems. However, in the actual operation and maintenance process, operation and maintenance personnel often forget about standardized operation standards due to fluke mentality, which brings safety hazards to the operation and maintenance operations. Although there is manual video supervision to remind, due to the limited energy of the staff and the long operation and maintenance time, the staff responsible for video supervision cannot discover and eliminate safety hazards in time.

[0003] Chinese Patent, Publication Number: CN113076825A, Publication Date: July 6, 2021, discloses a method for safely monitoring the height climbing of substation workers, including Step 1, extracting human joint points from the video screen, and then connecting each pair of associated body parts through a posture line, and the posture line will be combined into the full-body posture of the human body detected in the image; wherein, the Openpose algorithm is used to extract the joint points of the personnel in the video screen, and human body detection and human body area fusion steps are added thereto; Step 2, detection and judgment of the staff's climbing action; if it is detected that the staff has a climbing behavior, the next step is to be carried out; Step 3, identification of protective measures for staff during climbing: determine whether the safety belt and safety helmet are correctly worn as required. Specifically, the YOLO algorithm is used to detect whether there is a corresponding object in the human image area; however, the invention does not take into account the spatiotemporal characteristics involved in the behavior recognition process, resulting in low accuracy in the identification of staff protective measures. Summary of the invention

[0004] The purpose of the present invention is to address the problem that the prior art does not take into account the spatiotemporal characteristics in the behavior recognition process, resulting in low accuracy in identifying abnormal behaviors of operation and maintenance personnel at the job site; an abnormal behavior recognition method based on a spatiotemporal graph convolutional network is proposed, which uses a target detection algorithm to identify and mark detection targets in the operation and maintenance monitoring video to obtain a detection video, thereby reducing the influence of non-essential factors in the video on the recognition result, and then inputs the detection video into a posture estimation algorithm to calculate a skeleton graph, and constructs a behavior spatiotemporal graph based on the skeleton graph to extract the spatiotemporal characteristics in the detection video, and at the same time improves the behavior recognition model according to the attention mechanism to enhance the model's processing capability for the behavior spatiotemporal graph with spatiotemporal characteristics, and finally inputs the behavior spatiotemporal graph into the behavior recognition model to identify the abnormal behavior of the operation and maintenance personnel and issue an alarm, thereby significantly improving the accuracy of abnormal behavior recognition of the operation and maintenance personnel at the job site.

[0005] In a first aspect, a technical solution provided in an embodiment of the present invention is a method for identifying abnormal behavior based on a spatiotemporal graph convolutional network, comprising the following steps: S1. Collect the monitoring video of the operation and maintenance personnel's work site and input it into the target detection algorithm to identify and mark the detection target to obtain the detection video; S2, inputting the detection video into a posture estimation algorithm to calculate and obtain a skeleton graph of the detection target; S3, constructing a behavior spatiotemporal graph based on the skeleton graph, and dividing the behavior spatiotemporal graph into a training graph set and a test graph set; S4, training a behavior recognition model improved according to the attention mechanism based on the training atlas; S5. Input the test atlas into the trained behavior recognition model to identify whether there is any abnormality in the behavior of the operation and maintenance personnel at the work site, and issue an alarm when the operation and maintenance personnel behave abnormally.

[0006] In this scheme, the monitoring video of the operation and maintenance personnel's work site is input into the target detection algorithm, and the area with the operation and maintenance personnel as the main body in the monitoring screen is identified as the detection target area to obtain the detection video, and the skeleton map of the operation and maintenance personnel in the detection video is calculated by the posture estimation algorithm, that is, the skeleton map of the detection target, to reduce the influence of non-essential factors in the video on the recognition result, and then the behavior spatiotemporal map is constructed and divided based on the skeleton map to obtain the training atlas and the test atlas, and the spatiotemporal characteristics in the detection video are extracted. Based on the training atlas, the behavior recognition model that has been improved according to the attention mechanism is trained, so that the performance of the behavior recognition model reaches the optimal performance while meeting the actual recognition work needs. Finally, the test atlas is input into the trained behavior recognition model, and the behavior recognition model analyzes the test atlas on the basis of combining the spatiotemporal characteristics of the detection video to determine whether there is any abnormality in the behavior of the operation and maintenance personnel at the operation site, and issues an alarm when the behavior of the operation and maintenance personnel is abnormal, thereby significantly improving the accuracy of abnormal behavior recognition of the operation and maintenance personnel at the operation site.

[0007] Preferably, in S1, the specific process of collecting the monitoring video of the operation and maintenance personnel's work site and inputting it into the target detection algorithm to identify and mark the detection target to obtain the detection video is: S11, decomposing the surveillance video according to time series to obtain a video image set; S12, normalizing the images in the video image set using an image adjustment technology to obtain a normalized image set; S13, extracting features based on the images in the normalized image set to determine the detection target and marking it; S14, fusing the images of the marked detection targets to obtain the detection video.

[0008] In this solution, the surveillance video is decomposed into a video image set according to the time series, so that the video data that does not meet the input data type requirements of the target detection algorithm is converted into image data that meets the input data type requirements of the target detection algorithm, and the image adjustment technology is used to normalize the images in the video image set, so that the target detection algorithm does not have to repeatedly calculate the anchor frame used for marking due to different image sizes, effectively avoiding the waste of computing resources.

[0009] Preferably, in S2, the posture estimation algorithm is specifically an OpenPose algorithm that can obtain spatial information and position information of each joint point of the human skeleton.

[0010] Preferably, in S3, the specific process of constructing the behavior spatiotemporal graph based on the skeleton graph is: S31, extracting skeleton sequence information of the detection video based on the skeleton graph, and constructing a skeleton graph corresponding graph structure according to the skeleton sequence information; S32, performing spatial graph convolution according to the graph structure of the skeleton graph to determine the aggregated neighborhood information corresponding to the graph structure, and performing temporal graph convolution according to the aggregated neighborhood information to obtain local time series features; S33, constructing a behavior spatiotemporal graph based on the local temporal features and combining the graph structure.

[0011] In this scheme, spatial graph convolution is performed according to the graph structure of the skeleton graph to determine the aggregated neighborhood information corresponding to the graph structure. In fact, the spatial information in the skeleton sequence information is made visible through spatial graph convolution. Secondly, local temporal features are obtained by performing temporal graph convolution according to the aggregated neighborhood information. In fact, the temporal information in the skeleton sequence information is made visible through temporal graph convolution, and the temporal graph convolution is performed on the basis of the aggregated neighborhood information. In this way, the connection between spatial information and temporal information can be established in the process of convolution, so that the local temporal features can simultaneously make spatial information and temporal information visible.

[0012] Preferably, in S32, the specific process of performing spatial graph convolution according to the graph structure of the skeleton graph to determine the aggregated neighborhood information corresponding to the graph structure is: S3211, establishing a feature matrix according to the graph structure of the skeleton graph, and establishing an adjacency matrix according to the connection relationship between the joint points in the graph structure of the skeleton graph; S3212. Perform multiplication calculation based on the feature matrix and the adjacency matrix to obtain aggregated neighborhood information.

[0013] In this scheme, the spatial information corresponding to the skeleton graph is extracted by multiplication calculation between the feature matrix and the adjacency matrix to obtain the aggregated neighborhood information.

[0014] Preferably, in S32, the specific process of performing time graph convolution according to the aggregated neighborhood information to obtain local time series features is: S3221, determining the time domain of the time graph convolution according to the aggregated neighborhood information, and partitioning the joint points in the graph structure according to the aggregated neighborhood information to obtain a root node subgraph, a centripetal subgraph, and a centrifugal subgraph; S3222, convolve the root node subgraph based on the time domain to determine the stationary feature, convolve the centripetal subgraph based on the time domain to determine the centripetal feature, and convolve the centrifugal subgraph based on the time domain to determine the centrifugal feature; S3223. Arrange the stationary features, centripetal features, and centrifugal features to obtain local time series features.

[0015] In this scheme, time information is extracted based on the aggregation of neighborhood information, and the local time series features that contain both spatial features and time series features are divided into stationary features, centripetal features, and centrifugal features according to the spatial information, which provides strong support for judging the behavior of the detection target.

[0016] Preferably, in S3221, the formula for partitioning the joint points in the graph structure according to the aggregated neighborhood information is specifically: In the formula, r j is the distance from the neighborhood node j to the center of gravity of the skeleton in the aggregated neighborhood information, r i is the distance from the root node i to the center of gravity of the skeleton in the aggregated neighborhood information, l ti is the neighborhood subset in the aggregated neighborhood information, v tj It is the neighbor node in the aggregated neighborhood information.

[0017] Preferably, in S4, improving the behavior recognition model according to the attention mechanism is specifically as follows: The attention mechanism is integrated into an attention module, and the attention module is installed at the input end of the behavior recognition model. The attention module determines the weight of the feature channel of the behavior recognition model according to the training atlas of the input behavior recognition model.

[0018] In this solution, the attention mechanism is used to improve the behavior recognition model, and the weights of the feature channels of the behavior recognition model are adjusted. The above-mentioned target detection algorithm is combined to mark the detection target, and the behavior spatiotemporal graph is constructed based on the skeleton graph to extract the spatiotemporal characteristics in the skeleton graph. By adjusting the weights of the feature channels of the behavior recognition model, more computing resources can be used to calculate important areas and features such as detection targets and spatiotemporal characteristics in the surveillance video, thereby maximizing the utilization of computing resources.

[0019] Preferably, the specific process of the attention module determining the feature channel weights of the behavior recognition model according to the training atlas of the input behavior recognition model is as follows: Use global average pooling to compress the spatiotemporal graphs in the training atlas to obtain feature vectors; Based on the feature vector combined with a nonlinear activation function, the relationship between the feature channels of the behavior recognition model is established to obtain a weight vector; the weight of the feature channel of the behavior recognition model is determined by multiplying the channel dimension according to the weight vector and the spatiotemporal graph in the training graph set.

[0020] Preferably, in the process of training the behavior recognition model in S4, an evaluation index of the behavior recognition model is also calculated to determine the optimal behavior recognition model. The calculation formula of the evaluation index is specifically: Where P is the precision of the behavior recognition model, R is the recall of the behavior recognition model, TP is the number of operations and maintenance personnel behaviors correctly identified by the behavior recognition model as normal, FP is the number of operations and maintenance personnel behaviors incorrectly identified by the behavior recognition model as normal, and FN is the number of operations and maintenance personnel behaviors incorrectly identified by the behavior recognition model as abnormal.

[0021] In this solution, the evaluation index is calculated during the training process to judge the performance of the behavior recognition model, which helps to continuously improve the performance of the behavior recognition model during the training process and allows the behavior recognition model to develop in a preset direction during the training process.

[0022] The present invention has at least the following substantial effects: This solution inputs the monitoring video of the operation and maintenance personnel's work site into the target detection algorithm, identifies the area with the operation and maintenance personnel as the main body in the monitoring screen as the detection target area to obtain the detection video, and calculates the skeleton map of the operation and maintenance personnel in the detection video through the posture estimation algorithm, that is, the skeleton map of the detection target, to reduce the influence of non-essential factors in the video on the recognition result, and then constructs and divides the behavior spatiotemporal map based on the skeleton map to obtain the training atlas and the test atlas, extracts the spatiotemporal characteristics in the detection video, and trains the behavior recognition model that has been improved according to the attention mechanism based on the training atlas, so that the performance of the behavior recognition model reaches the optimal performance while meeting the needs of actual recognition work. Finally, the test atlas is input into the trained behavior recognition model, and the behavior recognition model analyzes the test atlas on the basis of combining the spatiotemporal characteristics of the detection video to determine whether there is any abnormality in the behavior of the operation and maintenance personnel at the operation site, and issues an alarm when the behavior of the operation and maintenance personnel is abnormal, thereby significantly improving the accuracy of identifying abnormal behavior of the operation and maintenance personnel at the operation site. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the drawings to represent the same parts.

[0024] Figure 1 This is a flow chart of an abnormal behavior recognition method based on a spatiotemporal graph convolutional network of the present invention; Figure 2 This is a simplified diagram of the skeleton topology; Figure 3 Schematic diagram of the process of temporal graph convolution. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the drawings; the process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0027] Embodiment 1: like Figure 1 As shown, this embodiment provides an abnormal behavior recognition method based on a spatiotemporal graph convolutional network, comprising the following steps: S1, collecting on-site monitoring video of operation and maintenance personnel, inputting the target detection algorithm to identify and mark the detection target to obtain a detection video.

[0028] In one embodiment, the specific process of the target detection algorithm identifying and marking the detection target is as follows: S11, decomposing the surveillance video according to time series to obtain a video image set; S12, normalizing the images in the video image set using an image adjustment technology to obtain a normalized image set; S13, extracting features based on the images in the normalized image set to determine the detection target and marking it; S14, fusing the images of the marked detection targets to obtain the detection video.

[0029] In this embodiment, the surveillance video is first decomposed into a large number of corresponding pictures according to its own video frames, and these pictures are organized into a video image set. Then, the letterbox function is used to normalize the pictures in the video image set, and the pictures are converted into a standard size that meets the input picture size of the target detection algorithm. In addition, the target detection algorithm specifically adopts the YOLOv5 algorithm that can automatically calculate the size of the anchor box for annotation according to the input picture. Finally, after the YOLOv5 algorithm recognizes and marks the detection target, the pictures with the marked detection target are fused into the detection video according to the video frame of the surveillance video.

[0030] S2. Input the detection video into a posture estimation algorithm to calculate and obtain a skeleton graph of the detection target.

[0031] In one embodiment, the posture estimation algorithm is specifically an OpenPose algorithm that can obtain spatial information and position information of each joint point of a human skeleton.

[0032] S3. Construct a behavior spatiotemporal graph based on the skeleton graph, and divide the behavior spatiotemporal graph into a training graph set and a test graph set.

[0033] In one embodiment, the specific process of constructing the behavior spatiotemporal graph is as follows: S31, extracting skeleton sequence information of the detection video based on the skeleton graph, and constructing a skeleton graph corresponding graph structure according to the skeleton sequence information; S32. Perform spatial graph convolution according to the graph structure of the skeleton graph to determine aggregated neighborhood information corresponding to the graph structure, and perform temporal graph convolution according to the aggregated neighborhood information to obtain local time series features.

[0034] In one embodiment, the specific process of spatial graph convolution is: S3211, establishing a feature matrix according to the graph structure of the skeleton graph, and establishing an adjacency matrix according to the connection relationship between the joint points in the graph structure of the skeleton graph; S3212. Perform multiplication calculation based on the feature matrix and the adjacency matrix to obtain aggregated neighborhood information.

[0035] In one embodiment, the specific process of temporal graph convolution is: S3221. Determine the time domain of the time graph convolution according to the aggregated neighborhood information, and partition the joint points in the graph structure according to the aggregated neighborhood information to obtain a root node subgraph, a centripetal subgraph, and a centrifugal subgraph.

[0036] In one embodiment, the formula for partitioning the joint points in the graph structure according to the aggregated neighborhood information is specifically: In the formula, r j is the distance from the neighborhood node j to the center of gravity of the skeleton in the aggregated neighborhood information, r i is the distance from the root node i to the center of gravity of the skeleton in the aggregated neighborhood information, l ti is the neighborhood subset in the aggregated neighborhood information, v tj It is the neighbor node in the aggregated neighborhood information.

[0037] S3222, convolve the root node subgraph based on the time domain to determine the stationary feature, convolve the centripetal subgraph based on the time domain to determine the centripetal feature, and convolve the centrifugal subgraph based on the time domain to determine the centrifugal feature; S3223. Arrange the stationary features, centripetal features, and centrifugal features to obtain local time series features.

[0038] S33, constructing a behavior spatiotemporal graph based on the local temporal features and combining the graph structure.

[0039] In this embodiment, the operation object of the graph convolution is a graph topology structure in a non-Euclidean form, that is, the skeleton graph structure constructed in step S31, as shown in Figure 2 Taking the simplified bone topology diagram as an example, the diagram contains 5 joint points A, B, C, D, and E, and the feature matrix and adjacency matrix can be obtained according to the bone topology diagram to describe the connection relationship between the joint points. If expressed using a mathematical formula, the feature matrix is ​​as follows: A0~A4 in the formula corresponds to joint point A in the skeleton topology diagram, B0~B4 corresponds to joint point B in the skeleton topology diagram, C0~C4 corresponds to joint point C in the skeleton topology diagram, D0~D4 corresponds to joint point D in the skeleton topology diagram, and E0~E4 corresponds to joint point E in the skeleton topology diagram; The adjacency matrix is ​​as follows: In the formula, 1 indicates that the joint points at the corresponding positions are connected to each other, and 0 indicates that the joint points at the corresponding positions are not connected. For example, 1 in the first row and first column and 1 in the third row of the first column together indicate that joint point A is connected to joint point C; At this point, the convolution of the spatial graph can be completed by calculating the feature matrix and the adjacency matrix according to the matrix multiplication rule, and the spatial information in the skeleton graph is extracted to obtain the aggregated neighborhood information. At the same time, after the spatial graph convolution is completed, the obtained aggregated neighborhood information is used as the input of the temporal graph convolution to determine the time domain of the temporal graph convolution. The time domain can be expressed as G T, as shown below: T =(V T ,E T ); Where V T represents the set of consecutive frames of the same joint point, E T Indicates V T The connection relationship between Figure 3 Taking the temporal graph convolution process shown in the figure as an example, the temporal convolution process consists of 4 skeleton graphs T0~T3. Figure 3 The dotted line in the figure is the time edge, and the solid line is the space edge. If the size of the convolution kernel used in the convolution process is K×1, then after completing the convolution of one node and K frames, the stride step frame is moved (here K=3). When the convolution of one node is completed, the convolution operation of the next node is continued.

[0040] S4. Training a behavior recognition model improved according to the attention mechanism based on the training atlas.

[0041] In one embodiment, the improved behavior recognition model based on the attention mechanism is specifically: The attention mechanism is integrated into an attention module, and the attention module is installed at the input end of the behavior recognition model. The attention module determines the weight of the feature channel of the behavior recognition model according to the training atlas of the input behavior recognition model.

[0042] The specific process of the attention module determining the feature channel weights of the behavior recognition model according to the training atlas of the input behavior recognition model is as follows: Use global average pooling to compress the spatiotemporal graphs in the training atlas to obtain feature vectors; Based on the feature vector combined with a nonlinear activation function, the relationship between the feature channels of the behavior recognition model is established to obtain a weight vector; the weight of the feature channel of the behavior recognition model is determined by multiplying the channel dimension according to the weight vector and the spatiotemporal graph in the training graph set.

[0043] In this embodiment, the attention mechanism is an attention mechanism established based on human visual attention. Similar to human vision, the attention mechanism can quickly scan the global image and obtain the target area that needs to be focused on, thereby adjusting the computing resources to focus more on these important areas. Therefore, the attention mechanism is set at the input end of the behavior recognition model, and the allocation of computing resources is achieved by adjusting the weights of each input channel of the behavior recognition model; secondly, the behavior recognition model is specifically an ST-GCN model.

[0044] In one embodiment, during the process of training the behavior recognition model in S4, an evaluation index of the behavior recognition model is also calculated to determine the optimal behavior recognition model. The calculation formula of the evaluation index is specifically: Where P is the precision of the behavior recognition model, R is the recall of the behavior recognition model, TP is the number of operations and maintenance personnel behaviors correctly identified by the behavior recognition model as normal, FP is the number of operations and maintenance personnel behaviors incorrectly identified by the behavior recognition model as normal, and FN is the number of operations and maintenance personnel behaviors incorrectly identified by the behavior recognition model as abnormal.

[0045] S5. Input the test atlas into the trained behavior recognition model to identify whether there is any abnormality in the behavior of the operation and maintenance personnel at the work site, and issue an alarm when the operation and maintenance personnel behave abnormally.

[0046] In order to verify the actual effect of this embodiment, this embodiment uses video editing software to intercept relevant operation clips and perform data enhancement for verification. The relevant operation clips are videos with a duration of 6S, a resolution of 640×480, and a frame rate of 30fps. The training set obtained according to the video contains a total of 1297 video clips, including 288 insulation detection clips, 340 insulation shielding clips, 176 climbing over the working bucket clips, 240 phone call clips, and 253 current measurement clips. In addition, the test set contains 45 clips of each of the five types of target actions. After the detection is completed, the accuracy calculation formula and the intersection-combination ratio formula are used to calculate the accuracy of the corresponding detection results. The accuracy of the corresponding detection results is shown in Table 1: Table 1. Behavior recognition accuracy Behavior Category Video behavior recognition accuracy / % Insulation testing 96.1 Climb over the work bucket 98.9 Make phone calls 97.3 Measuring current 96.0 Insulation shielding 96.7 According to the data in the table, for different types of operation and maintenance personnel behaviors, the lowest accuracy of this embodiment reaches more than 96%, which has excellent recognition ability.

[0047] This embodiment has at least the following substantial effects: In this embodiment, the monitoring video of the operation and maintenance personnel's work site is input into the target detection algorithm, the area with the operation and maintenance personnel as the main body in the monitoring screen is identified as the detection target area to obtain the detection video, and the skeleton map of the operation and maintenance personnel in the detection video is calculated by the posture estimation algorithm, that is, the skeleton map of the detection target, so as to reduce the influence of non-essential factors in the video on the recognition result, and then the behavior spatiotemporal map is constructed and divided based on the skeleton map to obtain the training atlas and the test atlas, and the spatiotemporal characteristics in the detection video are extracted. Based on the training atlas, the behavior recognition model that has been improved according to the attention mechanism is trained, so that the performance of the behavior recognition model reaches the optimal performance while meeting the actual recognition work needs. Finally, the test atlas is input into the trained behavior recognition model, and the behavior recognition model analyzes the test atlas on the basis of combining the spatiotemporal characteristics of the detection video to determine whether there is any abnormality in the behavior of the operation and maintenance personnel at the operation site, and issues an alarm when the behavior of the operation and maintenance personnel is abnormal, thereby significantly improving the accuracy of abnormal behavior recognition of the operation and maintenance personnel at the operation site.

[0048] The above specific embodiments are preferred embodiments of the present invention, and are not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to the present specific embodiments. All equivalent changes made in accordance with the shape, structure, and method of the present invention are within the protection scope of the present invention.

Claims

1. A method for abnormal behavior recognition based on spatiotemporal graph convolutional network, characterized in that: The method comprises the following steps: S1. Collect the monitoring video of the operation and maintenance personnel's work site and input it into the target detection algorithm to identify and mark the detection target to obtain the detection video; S2, inputting the detection video into a posture estimation algorithm to calculate and obtain a skeleton graph of the detection target; S3, constructing a behavior spatiotemporal graph based on the skeleton graph, and dividing the behavior spatiotemporal graph into a training graph set and a test graph set; S4, training a behavior recognition model improved according to the attention mechanism based on the training atlas; S5. Input the test atlas into the trained behavior recognition model to identify whether there is any abnormality in the behavior of the operation and maintenance personnel at the work site, and issue an alarm when the operation and maintenance personnel behave abnormally.

2. According to claim 1, the abnormal behavior recognition method based on spatiotemporal graph convolutional network is characterized in that: In S1, the specific process of collecting the monitoring video of the operation and maintenance personnel's work site and inputting it into the target detection algorithm to identify and mark the detection target to obtain the detection video is as follows: S11, decomposing the surveillance video according to time series to obtain a video image set; S12, normalizing the images in the video image set using an image adjustment technology to obtain a normalized image set; S13, extracting features based on the images in the normalized image set to determine the detection target and marking it; S14, fusing the images of the marked detection targets to obtain the detection video.

3. According to claim 1, the abnormal behavior recognition method based on spatiotemporal graph convolutional network is characterized in that: In S2, the posture estimation algorithm is specifically an OpenPose algorithm that can obtain spatial information and position information of each joint point of the human skeleton.

4. The abnormal behavior recognition method based on spatiotemporal graph convolutional network according to claim 1 is characterized in that: In S3, the specific process of constructing the behavior spatiotemporal graph based on the skeleton graph is as follows: S31, extracting skeleton sequence information of the detection video based on the skeleton graph, and constructing a skeleton graph corresponding graph structure according to the skeleton sequence information; S32, performing spatial graph convolution according to the graph structure of the skeleton graph to determine the aggregated neighborhood information corresponding to the graph structure, and performing temporal graph convolution according to the aggregated neighborhood information to obtain local time series features; S33, constructing a behavior spatiotemporal graph based on the local temporal features and combining the graph structure.

5. According to claim 4, the abnormal behavior recognition method based on spatiotemporal graph convolutional network is characterized in that: In S32, the specific process of performing spatial graph convolution according to the graph structure of the skeleton graph to determine the aggregated neighborhood information corresponding to the graph structure is: S3211, establishing a feature matrix according to the graph structure of the skeleton graph, and establishing an adjacency matrix according to the connection relationship between the joint points in the graph structure of the skeleton graph; S3212. Perform multiplication calculation based on the feature matrix and the adjacency matrix to obtain aggregated neighborhood information.

6. According to claim 4, the abnormal behavior recognition method based on spatiotemporal graph convolutional network is characterized in that: In S32, the specific process of performing time graph convolution according to the aggregated neighborhood information to obtain local time series features is: S3221, determining the time domain of the time graph convolution according to the aggregated neighborhood information, and partitioning the joint points in the graph structure according to the aggregated neighborhood information to obtain a root node subgraph, a centripetal subgraph, and a centrifugal subgraph; S3222, convolve the root node subgraph based on the time domain to determine the stationary feature, convolve the centripetal subgraph based on the time domain to determine the centripetal feature, and convolve the centrifugal subgraph based on the time domain to determine the centrifugal feature; S3223. Arrange the stationary features, centripetal features, and centrifugal features to obtain local time series features.

7. The abnormal behavior recognition method based on spatiotemporal graph convolutional network according to claim 6 is characterized in that: In the S3221, the formula for partitioning the joint points in the graph structure according to the aggregated neighborhood information is specifically: In the formula, r j is the distance from the neighborhood node j to the center of gravity of the skeleton in the aggregated neighborhood information, r i is the distance from the root node i to the center of gravity of the skeleton in the aggregated neighborhood information, l ti is the neighborhood subset in the aggregated neighborhood information, v tj It is the neighbor node in the aggregated neighborhood information.

8. The abnormal behavior recognition method based on spatiotemporal graph convolutional network according to claim 1 is characterized in that: In S4, the behavior recognition model is improved according to the attention mechanism as follows: The attention mechanism is integrated into an attention module, and the attention module is installed at the input end of the behavior recognition model. The attention module determines the weight of the feature channel of the behavior recognition model according to the training atlas of the input behavior recognition model.

9. The abnormal behavior recognition method based on spatiotemporal graph convolutional network according to claim 8 is characterized in that: The specific process of the attention module determining the feature channel weights of the behavior recognition model according to the training atlas of the input behavior recognition model is as follows: Use global average pooling to compress the spatiotemporal graphs in the training atlas to obtain feature vectors; Based on the feature vector combined with a nonlinear activation function, the relationship between the feature channels of the behavior recognition model is established to obtain a weight vector; the weight of the feature channel of the behavior recognition model is determined by multiplying the channel dimension according to the weight vector and the spatiotemporal graph in the training graph set.

10. The abnormal behavior recognition method based on spatiotemporal graph convolutional network according to claim 1 is characterized in that: In the process of training the behavior recognition model in S4, an evaluation index of the behavior recognition model is also calculated to determine the optimal behavior recognition model. The calculation formula of the evaluation index is specifically: Where P is the precision of the behavior recognition model, R is the recall of the behavior recognition model, TP is the number of operations and maintenance personnel behaviors correctly identified by the behavior recognition model as normal, FP is the number of operations and maintenance personnel behaviors incorrectly identified by the behavior recognition model as normal, and FN is the number of operations and maintenance personnel behaviors incorrectly identified by the behavior recognition model as abnormal.

Images