Railway maintenance group behavior recognition method and system based on data fusion

Abstract

The present application discloses a method and system for railway maintenance group behavior recognition based on data fusion, which belongs to the technical field of group behavior recognition; the method comprises: performing feature processing on RGB data of railway maintenance monitoring video to obtain extraction features of multiple different levels, and processing through a multi-scale local feature fusion network and a spatiotemporal attention network to obtain RGB spatiotemporal attention features; obtaining coordinate branch output features of RGB data, and fusing the coordinate branch output features with the RGB spatiotemporal attention features to obtain RGB target features; in obtaining key point data, processing the key point data to obtain key point target features; finally, fusing the RGB target features with the key point target features to obtain recognition features; by extracting and fusing key features of different scales, the accuracy and robustness of group behavior recognition in complex scenes in railway maintenance monitoring videos are improved.

Description

Technical Field

[0001] The present application belongs to the technical field of group behavior recognition, and specifically relates to a method and system for railway maintenance group behavior recognition based on data fusion. Background Art

[0002] With the rapid development of the railway transportation industry, railway inspection and maintenance work has also increased. However, the railway maintenance environment is generally more complex and there are many safety risks. At present, there are many works devoted to the research of group behavior recognition methods. These methods usually use the backbone network of convolutional neural network (CNN) to extract individual features, and use recurrent neural network (RNN), graph convolutional network (GCN) or Transformer for further processing.

[0003] However, the recognition methods in the prior art will have low recognition efficiency when encountering problems such as complex maintenance environments, background changes caused by traffic and pedestrian flow, motion blur, and occlusion. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and system for identifying group behavior in railway maintenance based on data fusion, which can solve the problem of low recognition efficiency caused by complex maintenance environment in the recognition method in the prior art.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for identifying group behavior of railway maintenance personnel based on data fusion, the method comprising:

[0007] Acquire RGB data and key point data of the railway maintenance monitoring video, and perform feature processing on the RGB data to obtain extraction features at multiple different levels;

[0008] Inputting the extracted features at the multiple different levels into a multi-scale local feature fusion network for processing, and outputting multi-scale fusion features;

[0009] Constructing a similarity adjacency matrix of the multi-scale fusion features, and processing the similarity adjacency matrix and the multi-scale fusion features to obtain spatiotemporal interaction features;

[0010] Inputting the spatiotemporal interaction features into the spatiotemporal attention network for processing to output RGB spatiotemporal attention features;

[0011] Processing the individual coordinate frame features in the RGB data to obtain coordinate branch output features, and fusing the coordinate branch output features with the RGB spatiotemporal attention features to obtain RGB target features;

[0012] Acquire key point features in the key point data, establish a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and process the key point features and the key point adjacency matrix to obtain key point target features;

[0013] The RGB target features and the key point features are fused to obtain identification features, and the identification features are input into a classifier to obtain railway maintenance group behavior recognition and classification results.

[0014] As an optional implementation of the first aspect of the present application, the extraction features of the multiple different levels are input into a multi-scale local feature fusion network for processing, and a multi-scale fusion feature is output; specifically:

[0015] Processing the extracted features at multiple different levels through an efficient local attention module embedded in the multi-scale local feature fusion network to obtain local features corresponding to each of the extracted features;

[0016] Adaptively adjust the dimensions of the local features, and perform convolution processing on the local features after the dimension adjustment to obtain adaptive dimensional features corresponding to each of the local features;

[0017] The adaptive dimensional features are multiplied element by element to obtain the multi-scale fusion features.

[0018] As an optional implementation of the first aspect of the present application, the extracted features at multiple different levels are processed by the efficient local attention module embedded in the multi-scale local feature fusion network to obtain local features corresponding to each of the extracted features; specifically:

[0019] The efficient local attention module performs pooling on the extracted features in the horizontal and vertical directions, and each of the extracted features corresponds to obtaining a horizontal pooling feature and a vertical pooling feature;

[0020] Performing feature conversion, group normalization and nonlinear activation processing on the horizontal pooling features and the vertical pooling features respectively to obtain corresponding horizontal branch features and vertical branch features;

[0021] The horizontal branch feature, the vertical branch feature, and the corresponding extracted feature are multiplied to obtain the local feature corresponding to the extracted feature.

[0022] As an optional implementation of the first aspect of the present application, the similarity adjacency matrix of the multi-scale fusion feature is constructed, and the similarity adjacency matrix and the multi-scale fusion feature are processed to obtain the spatiotemporal interaction feature; specifically:

[0023] The multi-scale fusion feature is cut by the RoIAlign function to obtain a plurality of monomer features, and the plurality of monomer features are mapped to obtain a monomer mapping feature corresponding to each of the monomer features;

[0024] Calculating the appearance similarity between the monomer mapping features according to the cosine similarity, and constructing a similarity adjacency matrix of the multi-scale fusion features according to the appearance similarity;

[0025] Inputting the similarity adjacency matrix and the multi-scale fusion features into a spatial graph convolutional neural network, and outputting spatial interaction features corresponding to the multi-scale fusion features;

[0026] The spatial interaction features are input into a multi-scale temporal convolutional network for processing to obtain the spatiotemporal interaction features.

[0027] As an optional implementation of the first aspect of the present application, the spatiotemporal interaction feature is input into the spatiotemporal attention network for processing to output RGB spatiotemporal attention features; specifically:

[0028] Slicing the space-time interaction feature to obtain a spatial branch and a temporal branch of the space-time interaction feature;

[0029] Performing attention pooling processing on the spatial branch and the temporal branch respectively to obtain corresponding spatial pooling branches and temporal pooling branches;

[0030] The spatial pooling branch and the temporal pooling branch are spliced ​​together through a splicing function to obtain a spatiotemporal splicing feature;

[0031] The spatiotemporal splicing features are dimensionally adjusted through a fully connected layer, the spatiotemporal splicing features after the dimension adjustment are reshaped, and the spatiotemporal splicing features after the reshaping are residually connected with the spatiotemporal interaction features to obtain the RGB spatiotemporal attention features.

[0032] As an optional implementation of the first aspect of the present application, the individual coordinate frame features in the RGB data are processed to obtain coordinate branch output features, and the coordinate branch output features are fused with the RGB spatiotemporal attention features to obtain RGB target features; specifically:

[0033] Calculating the spatial difference between the individual coordinate frame features, and reshaping the individual coordinate frame features according to the spatial difference to obtain a reshaped coordinate frame feature;

[0034] The reshaped coordinate frame features are sequentially normalized and flattened to obtain flattened coordinate frame features;

[0035] Inputting the flattened coordinate frame features into a convolutional neural network composed of 3×3 convolutions for feature processing, and performing mapping processing on the flattened coordinate frame features after the feature processing to obtain mapped coordinate frame features;

[0036] Inputting the mapping coordinate frame features into a multi-scale temporal convolutional network and an efficient local attention module in sequence to obtain the coordinate branch output features;

[0037] A first learnable weight is set for the coordinate branch output feature, and the coordinate branch output feature is fused with the RGB spatiotemporal attention feature through the first learnable weight to obtain the RGB target feature.

[0038] As an optional implementation of the first aspect of the present application, the key point features in the key point data are obtained to establish a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and the key point features and the key point adjacency matrix are processed to obtain key point target features; specifically:

[0039] Inputting the key point data into a posture estimation algorithm for processing, and outputting the key point features;

[0040] Calculating the Euclidean distance between every two key points in the key point features according to the preset human joint graph, so as to establish the key point adjacency matrix according to the Euclidean distance;

[0041] Input the key point adjacency matrix and the key point features into the multi-scale spatiotemporal graph convolutional neural network composed of the spatial graph convolutional neural network, the multi-scale temporal convolutional network and the spatiotemporal attention network for processing, and output the key point spatiotemporal features;

[0042] The key point spatiotemporal features are sequentially reshaped and mapped to obtain the key point target features;

[0043] The RGB target feature and the key point feature are fused to obtain the recognition feature; specifically:

[0044] A second learnable weight is set for the key point target feature, and the key point target feature is fused with the RGB target feature through the second learnable weight to obtain the recognition feature.

[0045] In a second aspect, an embodiment of the present application provides a railway maintenance group behavior recognition system based on data fusion, the system comprising:

[0046] The acquisition module is used to obtain the RGB data and key point data of the railway maintenance monitoring video;

[0047] An extraction module, used to perform feature extraction on the RGB data and the key point data to obtain extraction features at multiple different levels;

[0048] A first fusion module processes the extracted features at different levels according to a preset multi-scale local feature fusion algorithm to obtain a multi-scale fusion feature;

[0049] A first processing module constructs a similarity adjacency matrix of the multi-scale fusion features, processes the similarity adjacency matrix and the multi-scale fusion features, and obtains spatiotemporal interaction features;

[0050] The second processing module processes the spatiotemporal interaction features according to a preset spatiotemporal attention algorithm to obtain RGB spatiotemporal attention features;

[0051] The second fusion module processes the individual coordinate frame features in the RGB data to obtain coordinate branch output features, and fuses the coordinate branch output features with the RGB spatiotemporal attention features to obtain RGB target features;

[0052] A third processing module is configured to obtain key point features in the key point data, establish a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and process the key point features and the key point adjacency matrix to obtain key point target features;

[0053] The third fusion module fuses the RGB target features and the key point features to obtain recognition features.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0056] In the embodiment of the present application, by extracting features of multiple different levels in RGB data, and optimizing the processing by multi-scale local feature fusion and spatiotemporal interaction features, it is possible to capture richer information, and at the same time, use the posture estimation algorithm to extract human key point data, fuse RGB data and key point data, and describe the behavioral characteristics of the railway maintenance group from multiple angles, thereby enhancing the robustness of the model; improving the problem of insufficient single-modal information in the group behavior recognition task caused by complex environment and occlusion in railway maintenance; by using a multi-scale spatiotemporal graph convolutional neural network composed of a spatial graph convolutional neural network, a multi-scale temporal convolutional network and a spatiotemporal attention network, the network During the training process, the network model can explore the interactive relationship between spatial and temporal participants in multiple video frames, realize effective modeling of spatiotemporal information, and focus on key individuals in crowded and occluded scenes, thereby helping the model to better perform group behavior recognition tasks; by extracting and fusing key features of different scales, exploring the interactive relationship between spatial and temporal participants in multiple video frames, it effectively improves the problems of low recognition rate caused by background changes, motion blur and occlusion caused by complex maintenance environment, vehicle flow and pedestrian flow in railway maintenance group behavior, and achieves a significant improvement in accuracy at a very low computational cost, thereby improving the accuracy and robustness of group behavior recognition in complex scenes in railway maintenance monitoring videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of a railway maintenance group behavior recognition method based on data fusion provided by some embodiments of the present application;

[0058] Figure 2 It is a framework structure diagram of a railway maintenance group behavior recognition method based on data fusion provided by some embodiments of the present application;

[0059] Figure 3 It is a multi-scale local feature fusion network structure diagram in a railway maintenance group behavior recognition method based on data fusion provided by some embodiments of the present application;

[0060] Figure 4 This is a spatiotemporal attention network structure diagram in a railway maintenance group behavior recognition method based on data fusion provided in some embodiments of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0063] In conjunction with the accompanying drawings, a method and system for identifying group behavior in railway maintenance based on data fusion provided by an embodiment of the present application will be described in detail through specific embodiments and their application scenarios.

[0064] Example

[0065] A railway maintenance group behavior recognition method based on data fusion includes the following steps:

[0066] S100: Obtain RGB data and key point data of the railway maintenance monitoring video, and perform feature processing on the RGB data to obtain extraction features at multiple different levels;

[0067] It is important to understand that RGB data and key point data are obtained from railway maintenance monitoring videos; RGB data includes the original color information and depth information of the video frame, while key point data is the location information of key parts of the human body; feature processing is performed on the RGB data to extract multiple different levels of extracted features, which contain information from low-level to high-level.

[0068] S200: Inputting the extracted features of multiple different levels into a multi-scale local feature fusion network for processing, and outputting multi-scale fusion features;

[0069] It should be understood that these extracted features at different levels are input into a multi-scale local feature fusion network, which can capture local features at different scales and fuse them to form more representative multi-scale fusion features.

[0070] Furthermore, in S200, the extracted features of multiple different levels are input into the multi-scale local feature fusion network for processing, and the multi-scale fusion features are output; specifically:

[0071] S210: Processing the extracted features at multiple different levels through the efficient local attention module embedded in the multi-scale local feature fusion network to obtain the local features corresponding to each extracted feature;

[0072] S220: Adaptively adjust the dimensions of the local features, and perform convolution processing on the local features after the dimension adjustment to obtain adaptive dimension features corresponding to each local feature;

[0073] S230: Perform element-by-element multiplication on the adaptive dimension features to obtain multi-scale fusion features.

[0074] It should be understood that the efficient local attention module (ELA) is used to process the extracted features of multiple different levels in order to enhance the saliency of local areas in the features of each level, so that the network can focus on more informative local features. ELA uses the attention mechanism to weight different parts of each feature map. This helps to highlight key information while suppressing irrelevant or noisy information. After the extracted features of each level are processed by ELA, an enhanced local feature is obtained, which is more focused on the key information area. Since the features of different levels may have different dimensions (such as the number of channels), they need to be adaptively dimensionalized for subsequent feature fusion. Upsampling (such as bilinear interpolation) or downsampling The convolutional layer can introduce nonlinear activation functions (such as ReLU) to increase the expressiveness of the model. After dimension adjustment and convolution, each local feature will obtain an adaptive dimension feature, which is consistent in dimension and convenient for subsequent fusion. Multiple adaptive dimension features are fused to obtain multi-scale fusion features. Element-wise multiplication is an effective fusion strategy, which allows features of different levels to interact at the same spatial position, thereby capturing cross-scale feature interactions. The result of element-wise multiplication is a multi-scale fusion feature, which combines feature information from different levels and has stronger expressiveness and robustness.

[0075] Furthermore, in S210, the extracted features at multiple different levels are processed by the efficient local attention module embedded in the multi-scale local feature fusion network to obtain the local features corresponding to each extracted feature; specifically:

[0076] S211: The efficient local attention module performs pooling of the extracted features in the horizontal and vertical directions, and each extracted feature corresponds to a horizontal pooling feature and a vertical pooling feature;

[0077] S212: performing feature conversion, group normalization, and nonlinear activation processing on the horizontal pooling features and the vertical pooling features, respectively, to obtain corresponding horizontal branch features and vertical branch features;

[0078] S213: Multiply the horizontal branch feature, the vertical branch feature, and the corresponding extracted feature to obtain a local feature corresponding to the extracted feature.

[0079] Specifically, in S212, feature conversion, group normalization and nonlinear activation function processing are performed on the horizontal pooling features and the vertical pooling features respectively. The specific formulas are as follows:

[0080] ,

[0081] in, Indicates the horizontal direction, Indicates the vertical direction, Indicates feature conversion, group normalization and nonlinear activation function processing of features. represents the horizontal pooling feature corresponding to the Kth extracted feature, represents the vertical pooling feature corresponding to the Kth extracted feature, represents the horizontal branch feature corresponding to the K-th extracted feature, Represents the vertical branch feature corresponding to the Kth extracted feature.

[0082] In S213, the horizontal branch features, the vertical branch features and the corresponding extracted features are multiplied. The specific formula is as follows:

[0083] ,

[0084] in, represents the Kth extracted feature, represents the Kth local feature.

[0085] In S220, the dimension of the local features is adjusted. The specific formula is as follows:

[0086] ,

[0087] in, represents adaptive average pooling, represents the identity mapping, represents bilinear interpolation, represents the Kth local feature after dimension adjustment, represents the horizontal dimension of the Kth local feature before adjustment, represents the vertical dimension of the Kth local feature before adjustment, represents the horizontal target dimension, Indicates the target vertical dimension.

[0088] In S230, the adaptive dimension features are multiplied element by element, and the specific formula is as follows:

[0089] ,

[0090] in, represents the first adaptive dimension feature, Represents the second adaptive dimension feature, represents the Kth adaptive dimension feature, Represents multi-scale fusion features.

[0091] It is important to understand that the global information of the extracted features in different directions is captured through pooling operations in the horizontal and vertical directions; this helps the network understand the spatial structure of the features and provides global context information for subsequent feature transformations; ELA performs horizontal pooling (such as global average pooling along the horizontal direction) and vertical pooling (such as global average pooling along the vertical direction) on each extracted feature; in this way, each extracted feature will correspond to two pooling features: horizontal pooling features and vertical pooling features; horizontal pooling features and vertical pooling features capture the global information of the extracted features in the horizontal and vertical directions respectively; the pooled features are further processed to enhance their feature expression capabilities and are prepared to be fused with the original extracted features; the pooled features are transformed through convolutional layers or fully connected layers to learn new feature representations; the transformed features are group normalized (Group Normalization) to alleviate the problem of internal covariate shift and improve the training stability and performance of the model; apply nonlinear activation functions (such as ReLU) to increase the nonlinear expression ability of the model; output horizontal branch features and vertical branch features after feature conversion, group normalization and nonlinear activation function processing; fuse the horizontal branch features, vertical branch features and the original extracted features to obtain enhanced local features; element-by-element multiplication can emphasize the commonly activated feature areas; the fused local features, which combine the original extracted features and the global information in the horizontal and vertical directions, have stronger expression ability and robustness.

[0092] S300: constructing a similarity adjacency matrix of multi-scale fusion features, processing the similarity adjacency matrix and the multi-scale fusion features, and obtaining spatiotemporal interaction features;

[0093] It is important to understand that a similarity adjacency matrix of multi-scale fusion features is constructed to capture the similarity relationship between features; spatiotemporal interaction features are extracted by processing the similarity adjacency matrix and multi-scale fusion features; these features contain not only spatial information (such as the relative position between objects), but also temporal information (such as the motion trajectory of objects).

[0094] Furthermore, in S300, a similarity adjacency matrix of multi-scale fusion features is constructed, and the similarity adjacency matrix and multi-scale fusion features are processed to obtain spatiotemporal interaction features; specifically:

[0095] S310: cutting the multi-scale fusion feature through the RoIAlign function to obtain multiple monomer features, performing mapping processing on the multiple monomer features to obtain the monomer mapping feature corresponding to each monomer feature;

[0096] S320: Calculating the appearance similarity between the monomer mapping features according to the cosine similarity, and constructing a similarity adjacency matrix of the multi-scale fusion features according to the appearance similarity;

[0097] S330: inputting the similarity adjacency matrix and the multi-scale fusion features into the spatial graph convolutional neural network, and outputting the spatial interaction features corresponding to the multi-scale fusion features;

[0098] S340: Input the spatial interaction features into a multi-scale temporal convolutional network for processing to obtain spatiotemporal interaction features.

[0099] It should be understood that the features representing a single target (or monomer) are extracted from the multi-scale fusion features, and these features are mapped for subsequent similarity calculation; the multi-scale fusion features are accurately cut using the RoIAlign (Region of InterestAlign) function to obtain multiple monomer features; RoIAlign can avoid quantization errors and improve the accuracy of feature extraction; the extracted monomer features are mapped, such as feature conversion through a fully connected layer or a convolutional layer to obtain the monomer mapping features corresponding to each monomer feature; these mapping features have the same dimension, which is convenient for subsequent similarity calculation; the appearance similarity between the monomer mapping features is calculated, and a similarity adjacency matrix is ​​constructed based on the similarity; using cosine similarity as a metric, the similarity between each pair of monomer mapping features is calculated. Cosine similarity can measure the similarity of two vectors in direction without being affected by the length of the vector; based on the calculated similarity, a similarity adjacency matrix of the multi-scale fusion features is constructed, which is a two-dimensional array in which each element represents the similarity between two monomer features; using a spatial graph convolutional neural network (Spatial Graph ConvolutionalNetwork, The S-GCN processes the similarity adjacency matrix and multi-scale fusion features to extract spatial interaction features; the similarity adjacency matrix is ​​used as the edge weight in the graph structure, and the monomer mapping features are used as the node features in the graph structure, which are input into the spatial graph convolutional neural network; S-GCN can learn the spatial relationship between nodes and output the spatial interaction features corresponding to the multi-scale fusion features; after being processed by S-GCN, the spatial interaction features are obtained, which contain the spatial relationship information between the monomer features; the spatial interaction features are input into the Multi-scale Temporal Convolutional Network (M-TCN) to extract spatiotemporal interaction features; M-TCN can process time series data and learn features at different time scales; the spatial interaction features are used as the input of M-TCN, and the spatiotemporal interaction features are extracted through structures such as convolutional layers and pooling layers; after being processed by M-TCN, the spatiotemporal interaction features are obtained, which contain both spatial information (such as the relative position relationship between monomers) and temporal information (such as the movement trajectory of monomers).

[0100] S400: Input the spatiotemporal interaction features into the spatiotemporal attention network for processing to output RGB spatiotemporal attention features;

[0101] It is important to understand that the spatiotemporal interaction features are input into the spatiotemporal attention network; the network is able to dynamically adjust the attention weights of different regions and time periods, thereby extracting more critical and useful RGB spatiotemporal attention features.

[0102] Furthermore, in S400, the spatiotemporal interaction features are input into the spatiotemporal attention network for processing to output RGB spatiotemporal attention features; specifically:

[0103] S410: Slicing the spatiotemporal interaction feature to obtain a spatial branch and a temporal branch of the spatiotemporal interaction feature;

[0104] S420: performing attention pooling processing, batch normalization processing, and nonlinear activation processing on the spatial branch and the temporal branch in sequence to obtain corresponding spatial pooling branches and temporal pooling branches;

[0105] S430: splicing the spatial pooling branch and the temporal pooling branch through a splicing function to obtain a spatiotemporal splicing feature;

[0106] S440: The spatiotemporal splicing features are dimensionally adjusted through a fully connected layer, the dimensionally adjusted spatiotemporal splicing features are reshaped, and the reshaped spatiotemporal splicing features are residually connected with the spatiotemporal interaction features to obtain RGB spatiotemporal attention features.

[0107] It is important to understand that the spatiotemporal interaction features are decomposed into spatial branches and temporal branches so that they can be processed separately for attention. The spatiotemporal interaction features are sliced ​​in a specific dimension using a slicing function (such as the slicing operation in NumPy or tensor slicing in PyTorch) to obtain a spatial branch (mainly containing spatial information) and a temporal branch (mainly containing temporal information). The spatial branch and the temporal branch are subjected to attention pooling respectively to weight different parts of the feature map, thereby highlighting key information. The features after attention pooling are batch normalized. Normalization) is used to alleviate the problem of internal covariate shift and improve the training stability and performance of the model; nonlinear activation functions (such as ReLU, Sigmoid or Tanh) are applied to increase the nonlinear expression ability of the model; after the above processing, the spatial pooling branch and the temporal pooling branch are obtained, which enhance the key information in space and time respectively; the spatial pooling branch and the temporal pooling branch are spliced ​​to fuse the spatial and temporal information; the splicing function (such as torch.cat in PyTorch) is used to splice the spatial pooling branch and the temporal pooling branch in a specific dimension to obtain the spatiotemporal splicing feature; the spatiotemporal splicing feature is dimensionalized and the dimensionalized spatiotemporal splicing feature is reshaped to ensure that its dimension matches the spatiotemporal interaction feature for subsequent operations; the reshaped spatiotemporal splicing feature is residually connected with the original spatiotemporal interaction feature. Residual connection helps alleviate the gradient vanishing problem in deep networks and improves the training efficiency and performance of the model. After reshaping and residual connection, RGB spatiotemporal attention features are obtained. These features contain the information of the original spatiotemporal interaction features and integrate the key information in space and time, with stronger expression ability and robustness.

[0108] S500: Processing individual coordinate frame features in the RGB data to obtain coordinate branch output features, and fusing the coordinate branch output features with the RGB spatiotemporal attention features to obtain RGB target features;

[0109] It is important to understand that the individual coordinate frame features in the RGB data are processed to obtain the coordinate branch output features. These features describe the position and size of the individual in the video frame; the coordinate branch output features are fused with the RGB spatiotemporal attention features to obtain richer RGB target features.

[0110] Furthermore, in S500, the individual coordinate frame features in the RGB data are processed to obtain the coordinate branch output features, and the coordinate branch output features are fused with the RGB spatiotemporal attention features to obtain the RGB target features; specifically:

[0111] S510: Calculating the spatial differences between the individual coordinate frame features, and reshaping the individual coordinate frame features according to the spatial differences to obtain reshaped coordinate frame features;

[0112] S520: performing normalization processing and flattening processing on the reshaped coordinate frame features in sequence to obtain flattened coordinate frame features;

[0113] S530: Inputting the flattened coordinate frame features into a convolutional neural network composed of 3×3 convolutions for feature processing, and performing mapping processing on the flattened coordinate frame features after feature processing to obtain mapped coordinate frame features;

[0114] S540: inputting the mapping coordinate frame features into the multi-scale temporal convolutional network and the efficient local attention module in sequence to obtain the coordinate branch output features;

[0115] S550: Setting a first learnable weight for the coordinate branch output feature, and fusing the coordinate branch output feature with the RGB spatiotemporal attention feature through the first learnable weight to obtain an RGB target feature.

[0116] Specifically, in S510, the spatial difference between the individual coordinate frame features is calculated, and the specific formula is as follows:

[0117] ,

[0118] in, represents the difference matrix, represents the individual coordinate frame feature, and a and c represent different individual coordinate frame feature numbers.

[0119] In S550, the coordinate branch output feature is fused with the RGB spatiotemporal attention feature through the first learnable weight. The specific formula is as follows:

[0120] ,

[0121] in, represents the RGB target feature, represents the RGB spatiotemporal attention feature, represents the first learnable weight, Represents the coordinate branch output feature.

[0122] It is important to understand that the spatial differences between individual coordinate frame features are calculated, and the feature dimensions are made consistent through reshaping to facilitate subsequent processing; the relative position relationship between individual coordinate frame features (such as the coordinates of the bounding box) is analyzed, and the spatial differences are calculated, which helps to capture the spatial layout information between targets; based on the calculation results of the spatial differences, the individual coordinate frame features are reshaped, such as adjusting the shape or dimension of the feature map to make it compatible with subsequent processing steps; the reshaped coordinate frame features are normalized to eliminate the dimensional differences between different features and improve the generalization ability of the model; the normalized coordinate frame features are flattened into a one-dimensional vector for easy input into the convolutional network for processing; the flattened coordinate frame features are input into a convolutional neural network composed of 3×3 convolutions for feature processing to extract local features and flatten them. Sliding processing; Mapping processing is performed on the convolutional features (such as through a fully connected layer or a 1×1 convolutional layer) to obtain mapping coordinate frame features; These features have stronger expressive power and robustness; Input the mapping coordinate frame features into a multi-scale temporal convolutional network to capture dynamic change information in time; Multi-scale processing helps capture features at different time scales; Input the output results of the multi-scale temporal convolutional network into an efficient local attention module to focus on key information areas and enhance the local expressive power of features; After the above processing, the coordinate branch output features are obtained; These features contain both spatial layout information and capture dynamic changes in time, and enhance local expressive power; Set the first learnable weight to adjust the fusion ratio between the coordinate branch output features and the RGB spatiotemporal attention features. This helps the model automatically learn the best fusion method between the two; The coordinate branch output features are fused with the RGB spatiotemporal attention features through the first learnable weight to obtain RGB target features; These features integrate spatial layout, temporal dynamic changes and local attention information, and have stronger expressive power and robustness.

[0123] S600: Acquire key point features in the key point data, establish a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and process the key point features and the key point adjacency matrix to obtain key point target features;

[0124] It is important to understand that a key point adjacency matrix is ​​established based on key point features and a preset human joint diagram; the matrix describes the connection relationship between human joints; the key point features and the key point adjacency matrix are processed to extract key point target features; these features reflect the human body's posture, movement patterns, and interactions between human bodies.

[0125] Further, in S600, key point features in the key point data are obtained to establish a key point adjacency matrix of the key point features according to the key point features and the preset human joint map, and the key point features and the key point adjacency matrix are processed to obtain key point target features; specifically:

[0126] S610: inputting the key point data into the posture estimation algorithm for processing, and outputting the key point features;

[0127] S620: Calculate the Euclidean distance between every two key points in the key point feature according to the preset human joint graph, so as to establish a key point adjacency matrix according to the Euclidean distance;

[0128] S630: inputting the key point adjacency matrix and the key point features into a multi-scale spatiotemporal graph convolutional neural network composed of a spatial graph convolutional neural network, a multi-scale temporal convolutional network and a spatiotemporal attention network for processing, and outputting the key point spatiotemporal features;

[0129] S640: performing reshaping and mapping processing on the key point spatiotemporal features in sequence to obtain key point target features;

[0130] S650: Fuse the RGB target features and key point features to obtain recognition features; specifically:

[0131] S660: Setting a second learnable weight for the key point target feature, and fusing the key point target feature with the RGB target feature through the second learnable weight to obtain a recognition feature.

[0132] Specifically, in S620, a key point adjacency matrix is ​​established according to the Euclidean distance, and the specific formula is as follows:

[0133]

[0134] ,

[0135] in, Represents a preset human joint diagram, and Numerical codes representing different key points, represents an undirected edge connecting two key points, represents the key point adjacency matrix, Represents the element in the mth column and nth row of the key point adjacency matrix, represents the total number of key points in the key point adjacency matrix, express The key points represented by the rows are The columns represent the Euclidean distances between keypoints, Indicates the preset parameters. represents the number of rows of elements in the key point adjacency matrix, Represents the number of columns of elements in the keypoint adjacency matrix.

[0136] In S660, the key point target features and the RGB target features are fused through the second learnable weight. The specific formula is as follows:

[0137] ,

[0138] in, Represents identification features, represents the key point target features, represents the second learnable weight.

[0139] It should be understood that key point features are extracted from key point data, which describe the position and shape information of human joints; the key point data is input into the posture estimation algorithm (such as OpenPose, DeepLabCut, etc.), which can identify and locate the key points of the human body (such as joint points); the posture estimation algorithm outputs key point features, which usually represent the position of the key points in the form of coordinates, and may also contain the relative position or direction information between the key points; the preset human joint graph is a graph structure that defines the connection relationship between the key points of the human body, usually based on the biological human joint connection; according to the coordinates of each two key points in the key point features, the Euclidean distance between them is calculated; based on the Euclidean distance and the preset human joint graph, a two-dimensional array is established as the key point adjacency matrix; each element in the matrix represents two key points. The connection strength between key points; the key point adjacency matrix and key point features are taken as input; a multi-scale spatiotemporal graph convolutional neural network (MST-GCN) composed of a spatial graph convolutional neural network (S-GCN), a multi-scale temporal convolutional network (M-TCN) and a spatiotemporal attention network (STAN) is used; the network first processes the graph structure data through S-GCN to capture the spatial relationship; then processes the time series data through M-TCN to capture the dynamic changes in time; finally, the expression of key features is enhanced through STAN; key point spatiotemporal features are output, which contain both temporal dynamic information and spatial structural information; the key point spatiotemporal features are reshaped to match the input requirements of subsequent processing steps; the reshaped features are mapped through a fully connected layer or a convolutional layer to obtain the key point target features. These features have stronger expressive power and robustness; a second learnable weight is set to adjust the fusion ratio between key point target features and RGB target features; the key point target features are fused with the RGB target features through the second learnable weight; this helps to combine the advantages of the two features and improve the accuracy and robustness of recognition; and the recognition features are output, which integrate the RGB image information and the key point information, providing strong support for subsequent recognition or classification tasks.

[0140] S700: fusing the RGB target features and the key point features to obtain identification features, and inputting the identification features into a classifier to obtain a railway maintenance group behavior identification and classification result;

[0141] It should be understood that the RGB target features and the key point target features are fused to obtain the final recognition features; the recognition features are input into the classifier for classification and recognition to obtain the classification results of railway maintenance group behavior recognition.

[0142] According to a railway maintenance group behavior recognition method based on data fusion in this embodiment, by extracting multiple different levels of extraction features in RGB data, and using multi-scale local feature fusion and spatiotemporal interaction features for optimization processing, it is possible to capture richer information, and at the same time use a posture estimation algorithm to extract human key point data, fuse RGB data and key point data, and describe the behavior characteristics of the railway maintenance group from multiple angles, thereby enhancing the robustness of the model; improve the problem of insufficient single-modal information in the group behavior recognition task caused by complex environment and occlusion in railway maintenance; by using a multi-scale spatiotemporal graph convolutional neural network composed of a spatial graph convolutional neural network, a multi-scale temporal convolutional network and a spatiotemporal attention network , so that the network model can explore the interactive relationship between spatial and temporal participants in multiple video frames during training, realize effective modeling of spatiotemporal information, and focus on key individuals in crowded and occluded scenes, thereby helping the model to better perform group behavior recognition tasks; by extracting and fusing key features of different scales, exploring the interactive relationship between spatial and temporal participants in multiple video frames, it effectively improves the problems of low recognition rate caused by background changes, motion blur and occlusion caused by complex maintenance environment, vehicle flow and pedestrian flow in railway maintenance group behavior, and by introducing coordinate branches, it can achieve a significant improvement in accuracy at a very low computational cost, thereby improving the accuracy and robustness of group behavior recognition in complex scenes in railway maintenance monitoring videos.

[0143] It should be noted that the railway maintenance group behavior recognition method based on data fusion provided in the embodiment of the present application can be executed by a railway maintenance group behavior recognition system based on data fusion, or a control module in the railway maintenance group behavior recognition system based on data fusion for executing and loading a railway maintenance group behavior recognition method based on data fusion. In the embodiment of the present application, a railway maintenance group behavior recognition system based on data fusion is used to execute and load a railway maintenance group behavior recognition method based on data fusion as an example to illustrate a railway maintenance group behavior recognition method based on data fusion provided in the embodiment of the present application.

[0144] A railway maintenance group behavior recognition system based on data fusion, comprising:

[0145] The acquisition module is used to obtain the RGB data and key point data of the railway maintenance monitoring video;

[0146] The extraction module is used to extract features from RGB data and key point data to obtain extracted features at multiple levels;

[0147] The first fusion module processes the extracted features at multiple different levels according to a preset multi-scale local feature fusion algorithm to obtain multi-scale fusion features;

[0148] The first processing module constructs a similarity adjacency matrix of multi-scale fusion features, processes the similarity adjacency matrix and multi-scale fusion features, and obtains spatiotemporal interaction features;

[0149] The second processing module processes the spatiotemporal interaction features according to a preset spatiotemporal attention algorithm to obtain RGB spatiotemporal attention features;

[0150] The second fusion module processes the individual coordinate frame features in the RGB data to obtain the coordinate branch output features, and fuses the coordinate branch output features with the RGB spatiotemporal attention features to obtain the RGB target features;

[0151] The third processing module obtains key point features in the key point data, establishes a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and processes the key point features and the key point adjacency matrix to obtain key point target features;

[0152] The third fusion module fuses the RGB target features and key point features to obtain recognition features.

[0153] A railway maintenance group behavior recognition system based on data fusion in the embodiment of the present application can be a device, or a component, integrated circuit or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), etc., which is not specifically limited in the embodiment of the present application.

[0154] A railway maintenance group behavior recognition system based on data fusion in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0155] The railway maintenance group behavior recognition system based on data fusion provided in the embodiment of the present application can realize Figures 1 to 4 In the method embodiment, each process of implementing a method for identifying group behavior of railway maintenance based on data fusion is not described here to avoid repetition.

[0156] According to a railway maintenance group behavior recognition system based on data fusion in this embodiment, RGB data and key point data are obtained through the acquisition module. These two data types can capture the behavior of the railway maintenance group from the two dimensions of vision and posture; this helps to improve the accuracy and comprehensiveness of behavior recognition; the extraction module extracts features from RGB data and key point data to obtain extracted features at multiple different levels; this multi-level and multi-scale feature extraction helps to capture more detailed and rich behavior information and improve the accuracy of behavior recognition; the first fusion module uses a multi-scale local feature fusion algorithm to fuse the extracted features at multiple different levels to obtain multi-scale fusion features. This helps to integrate information at different levels and improve the robustness and adaptability of the system; the first processing module constructs a similarity adjacency matrix and processes the multi-scale fusion features and the similarity adjacency matrix to obtain spatiotemporal interaction features. This helps to capture the dynamic changes of behavior in time and space and further improve the accuracy of behavior recognition; the second processing module uses a spatiotemporal attention algorithm to process the spatiotemporal interaction features to obtain RGB spatiotemporal attention features. This mechanism can automatically focus on important spatiotemporal areas and improve the system's ability to capture key behavioral features. The second fusion module processes the individual coordinate frame features in the RGB data and fuses them with the RGB spatiotemporal attention features to obtain RGB target features. At the same time, the third processing module uses key point features and human joint graphs to establish a key point adjacency matrix and obtain key point target features. This fusion method can integrate individual behavior and posture information and improve the overall performance of behavior recognition. The third fusion module fuses RGB target features and key point features to obtain recognition features. This comprehensive feature extraction and fusion can make full use of the information of RGB data and key point data to improve the accuracy and reliability of group behavior recognition.

[0157] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the above-mentioned railway maintenance group behavior recognition method embodiment based on data fusion is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0158] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned railway maintenance group behavior recognition method embodiment based on data fusion is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0159] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0160] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0162] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A railway maintenance group behavior recognition method based on data fusion, characterized in that: The method comprises: Obtaining RGB data and key point data of the railway maintenance monitoring video, and performing feature processing on the RGB data to obtain multiple levels of extracted features; wherein the key point data is to extract the position information of key parts of the human body; Inputting the extracted features at the multiple different levels into a multi-scale local feature fusion network for processing, and outputting multi-scale fusion features; Constructing a similarity adjacency matrix of the multi-scale fusion features, and processing the similarity adjacency matrix and the multi-scale fusion features to obtain spatiotemporal interaction features; Inputting the spatiotemporal interaction features into the spatiotemporal attention network for processing to output RGB spatiotemporal attention features; Processing the individual coordinate frame features in the RGB data to obtain coordinate branch output features, and fusing the coordinate branch output features with the RGB spatiotemporal attention features to obtain RGB target features; Acquire key point features in the key point data, establish a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and process the key point features and the key point adjacency matrix to obtain key point target features; The RGB target features and the key point features are fused to obtain identification features, and the identification features are input into a classifier to obtain railway maintenance group behavior recognition and classification results.

2. The method for identifying group behavior of railway maintenance based on data fusion according to claim 1 is characterized in that: The extraction features of the multiple different levels are input into a multi-scale local feature fusion network for processing, and a multi-scale fusion feature is output; specifically: Processing the extracted features at multiple different levels through an efficient local attention module embedded in the multi-scale local feature fusion network to obtain local features corresponding to each of the extracted features; Adaptively adjust the dimensions of the local features, and perform convolution processing on the local features after the dimension adjustment to obtain adaptive dimensional features corresponding to each of the local features; The adaptive dimensional features are multiplied element by element to obtain the multi-scale fusion features.

3. The method for identifying group behavior of railway maintenance based on data fusion according to claim 2 is characterized in that: The extracted features at multiple different levels are processed by the efficient local attention module embedded in the multi-scale local feature fusion network to obtain local features corresponding to each of the extracted features; specifically: The efficient local attention module performs pooling on the extracted features in the horizontal and vertical directions, and each of the extracted features corresponds to obtaining a horizontal pooling feature and a vertical pooling feature; Performing feature conversion, group normalization and nonlinear activation processing on the horizontal pooling features and the vertical pooling features respectively to obtain corresponding horizontal branch features and vertical branch features; The horizontal branch feature, the vertical branch feature, and the corresponding extracted feature are multiplied to obtain the local feature corresponding to the extracted feature.

4. The method for identifying group behavior of railway maintenance based on data fusion according to claim 3 is characterized in that: The similarity adjacency matrix of the multi-scale fusion features is constructed, and the similarity adjacency matrix and the multi-scale fusion features are processed to obtain spatiotemporal interaction features; Specifically: The multi-scale fusion feature is cut by the RoIAlign function to obtain a plurality of monomer features, and the plurality of monomer features are mapped to obtain a monomer mapping feature corresponding to each of the monomer features; Calculating the appearance similarity between the monomer mapping features according to the cosine similarity, and constructing a similarity adjacency matrix of the multi-scale fusion features according to the appearance similarity; Inputting the similarity adjacency matrix and the multi-scale fusion features into a spatial graph convolutional neural network, and outputting spatial interaction features corresponding to the multi-scale fusion features; The spatial interaction features are input into a multi-scale temporal convolutional network for processing to obtain the spatiotemporal interaction features.

5. The method for identifying group behavior of railway maintenance based on data fusion according to claim 4 is characterized in that: The spatiotemporal interaction features are input into a spatiotemporal attention network for processing to output RGB spatiotemporal attention features; Specifically: Slicing the space-time interaction feature to obtain a spatial branch and a temporal branch of the space-time interaction feature; Performing attention pooling processing, batch normalization processing, and nonlinear activation processing on the spatial branch and the temporal branch respectively, to obtain corresponding spatial pooling branches and temporal pooling branches; The spatial pooling branch and the temporal pooling branch are spliced ​​together through a splicing function to obtain a spatiotemporal splicing feature; The spatiotemporal splicing features are dimensionally adjusted through a fully connected layer, the spatiotemporal splicing features after the dimension adjustment are reshaped, and the spatiotemporal splicing features after the reshaping are residually connected with the spatiotemporal interaction features to obtain the RGB spatiotemporal attention features.

6. The method for identifying group behavior of railway maintenance based on data fusion according to claim 5 is characterized in that: The individual coordinate frame features in the RGB data are processed to obtain coordinate branch output features, and the coordinate branch output features are fused with the RGB spatiotemporal attention features to obtain RGB target features; Specifically: Calculating the spatial difference between the individual coordinate frame features, and reshaping the individual coordinate frame features according to the spatial difference to obtain a reshaped coordinate frame feature; The reshaped coordinate frame features are sequentially normalized and flattened to obtain flattened coordinate frame features; Inputting the flattened coordinate frame features into a convolutional neural network composed of 3×3 convolutions for feature processing, and performing mapping processing on the flattened coordinate frame features after the feature processing to obtain mapped coordinate frame features; Inputting the mapping coordinate frame features into a multi-scale temporal convolutional network and an efficient local attention module in sequence to obtain the coordinate branch output features; A first learnable weight is set for the coordinate branch output feature, and the coordinate branch output feature is fused with the RGB spatiotemporal attention feature through the first learnable weight to obtain the RGB target feature.

7. The method for identifying group behavior of railway maintenance based on data fusion according to claim 6 is characterized in that: The key point features in the key point data are obtained to establish a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and the key point features and the key point adjacency matrix are processed to obtain key point target features; Specifically: Inputting the key point data into a posture estimation algorithm for processing, and outputting the key point features; Calculating the Euclidean distance between every two key points in the key point features according to the preset human joint graph, so as to establish the key point adjacency matrix according to the Euclidean distance; Inputting the key point adjacency matrix and the key point features into a multi-scale spatiotemporal graph convolutional neural network composed of the spatial graph convolutional neural network, the multi-scale temporal convolutional network and the spatiotemporal attention network for processing, and outputting key point spatiotemporal features; The key point spatiotemporal features are sequentially reshaped and mapped to obtain the key point target features; The RGB target feature and the key point feature are fused to obtain a recognition feature; Specifically: A second learnable weight is set for the key point target feature, and the key point target feature is fused with the RGB target feature through the second learnable weight to obtain the recognition feature.

8. A railway maintenance group behavior recognition system based on data fusion, realizing a railway maintenance group behavior recognition method based on data fusion as described in any one of claims 1 to 7, characterized in that: The system comprises: The acquisition module is used to obtain the RGB data and key point data of the railway maintenance monitoring video; An extraction module, used to perform feature extraction on the RGB data and the key point data to obtain extraction features at multiple different levels; A first fusion module processes the extracted features at different levels according to a preset multi-scale local feature fusion algorithm to obtain a multi-scale fusion feature; A first processing module constructs a similarity adjacency matrix of the multi-scale fusion features, processes the similarity adjacency matrix and the multi-scale fusion features, and obtains spatiotemporal interaction features; The second processing module processes the spatiotemporal interaction features according to a preset spatiotemporal attention algorithm to obtain RGB spatiotemporal attention features; The second fusion module processes the individual coordinate frame features in the RGB data to obtain coordinate branch output features, and fuses the coordinate branch output features with the RGB spatiotemporal attention features to obtain RGB target features; A third processing module is configured to obtain key point features in the key point data, establish a key point adjacency matrix of the key point features according to the key point features and a preset human joint map, and process the key point features and the key point adjacency matrix to obtain key point target features; The third fusion module fuses the RGB target features and the key point features to obtain recognition features.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of a method for identifying group behavior of railway maintenance based on data fusion as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of a railway maintenance group behavior recognition method based on data fusion as described in any one of claims 1 to 7 are implemented.

Images