An industrial behavior recognition detection method and system

Abstract

This invention discloses an industrial behavior recognition and detection method and system. The method collects video, audio, and sensor data from industrial scenarios, performs preprocessing and time synchronization, and then generates a pseudo-label dataset using a pre-trained feature extraction model combined with self-supervised and weakly supervised learning. It extracts spatiotemporal features from the video using a ResNet network, extracts audio spectral features using MFCC combined with a VGGish network, and extracts temporal features from the sensors using an LSTM network. A Transformer-based attention mechanism is employed to dynamically weight and fuse the multimodal features, and finally, a classifier is used to achieve behavior recognition and anomaly alarm. This invention effectively solves the problem of low recognition accuracy caused by changes in lighting, occlusion, and noise interference in industrial scenarios, significantly reduces data annotation costs, and improves model generalization ability and system real-time performance.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an industrial behavior recognition and detection method and system. Background Technology

[0002] In modern industrial production environments, the standardization and safety of worker operations directly impact production efficiency and accident prevention. Traditional behavior recognition methods primarily rely on single-modal data, especially video surveillance data. However, industrial environments are complex and variable, presenting numerous challenges: First, unstable lighting conditions, such as uneven illumination, excessive darkness, or overexposure, severely affect video image quality, leading to a significant decline in the performance of video-based behavior recognition models. Studies show that even advanced video baseline models like ResNet can experience a 10-20% drop in recognition accuracy under drastic lighting conditions. Second, occlusion is a common problem. In densely populated work environments such as assembly lines, workers' bodies are frequently partially obscured by machinery, tools, or other personnel, resulting in the loss of crucial action information and increasing the difficulty of behavior recognition. For example, when a worker's arm is obscured by assembly equipment, video-based models struggle to accurately determine their specific operational behavior.

[0003] With the development of deep learning technology, video-based action recognition methods have become mainstream. Among them, 3D convolutional neural networks (CNNs), by extending 2D convolution to the temporal dimension, can capture both spatial features and temporal dynamics simultaneously, but they face challenges in terms of parameter count and computational complexity. To improve efficiency, some studies have reduced the number of parameters by decreasing kernel size and optimizing feature reuse mechanisms. Another typical approach adopts a "space-first, time-later" processing strategy, that is, first using 2D CNNs to extract spatial features, and then aggregating temporal information through long short-term memory networks or convolutional long short-term memory networks. However, this method may cause information loss when processing 2D spatial features.

[0004] While multimodal fusion technology has made some progress in fields such as healthcare and transportation, dedicated behavior recognition systems for industrial scenarios remain scarce. Existing systems often fail to fully consider the multi-source sensor data unique to industrial environments, such as information provided by industrial-specific sensors like vibration and temperature sensors. Furthermore, in terms of modal fusion strategies, existing methods often employ simple early or late fusion approaches, lacking sufficient consideration of the complexity of industrial scenarios, resulting in low fusion efficiency and difficulty in achieving real-time processing of large-scale industrial data.

[0005] In addition, existing technologies have two key bottlenecks: First, deep learning model training relies on large-scale, high-quality labeled data, while the labeling of behavioral data in industrial scenarios requires the participation of professionals, which is costly and difficult to guarantee consistency. Second, the dynamic changes in the industrial environment make the trained model unable to generalize when facing new work scenarios, and the recognition accuracy is difficult to meet the actual needs of industrial applications. Summary of the Invention

[0006] To address the technical problems existing in the background art, this invention proposes an industrial behavior recognition and detection method and system.

[0007] The present invention proposes an industrial behavior recognition and detection method, comprising the following steps: S1. Acquire multimodal data in industrial scenarios, and preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data; S2. Based on the preprocessed multimodal data, a pseudo-label dataset containing behavioral category labels is generated using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. S3. Input the video data with pseudo-label dataset into the trained video feature extraction model to extract the video spatiotemporal feature vector; input the audio data with pseudo-label dataset into the trained audio feature extraction model to extract the audio spectrum feature vector; input the sensor data with pseudo-label dataset into the trained sensor feature extraction model to extract the sensor temporal feature vector. S4. Input the video spatiotemporal feature vector, audio spectrum feature vector, and sensor temporal feature vector into the trained cross-modal fusion model, perform dynamic weighted fusion through the attention mechanism, and output a unified fusion feature vector. S5. Input the fused feature vector into the preset behavior recognition classifier to obtain the corresponding behavior recognition result.

[0008] Preferably, step S2 specifically includes: Obtain preprocessed multimodal data and a small seed dataset containing manually labeled behavioral categories; Multimodal data is input into a pre-trained feature extraction model to extract initial features for each modality. A self-supervised contrastive learning algorithm is used to generate preliminary behavioral labels for video and sensor data. At the same time, a weakly supervised multi-instance learning algorithm is used to infer fine-grained behavioral labels for audio data based on a seed dataset. The preliminary behavior labels and fine-grained behavior labels are jointly verified and noise filtered through a cross-modal consistency verification mechanism to obtain the final pseudo-label dataset.

[0009] Preferably, the pre-trained feature extraction model includes a ResNet model, an MFCC feature extractor, and an LSTM network.

[0010] Preferably, in step S3, the video feature extraction model is a ResNet-50 or ResNet-101 network, and the training process of the video feature extraction model includes: Acquire video data with a pseudo-label dataset, the video data containing behavioral categories of normal and illegal operations; The video frame sequence is input into the initial ResNet-50 or ResNet-101 network, and spatiotemporal features are extracted through 3D convolutional layers; The extracted spatiotemporal features are reduced in dimensionality using a max pooling layer to generate a 2048-dimensional feature vector. The model parameters are optimized using the cross-entropy loss function and the Adam optimizer until the model converges to the preset value, thus obtaining a trained video feature extraction model.

[0011] Preferably, the audio feature extraction model is a model constructed based on MFCC coefficients and a VGGish network; the training process of the audio feature extraction model includes: Obtain audio data with pseudo-labels; The audio data is pre-emphasized and framed, and then subjected to Fast Fourier Transform, Mel filter bank processing and Discrete Cosine Transform in sequence to extract MFCC features. The MFCC features are input into a VGGish-based neural network model, and the model is fine-tuned using a binary cross-entropy loss function and an Adam optimizer. A well-trained audio feature extraction model is obtained that can extract spectral feature vectors from audio signals.

[0012] Preferably, the sensor feature extraction model is a two-layer LSTM network; the training process of the sensor feature extraction model includes: Obtain sensor time-series data with pseudo-labeled datasets; The time series data from the sensor are subjected to Z-score normalization to eliminate dimensional differences; The normalized time series data is input into the initial two-layer LSTM network, and the network parameters are optimized using the cross-entropy loss function and the RMSprop optimizer. A well-trained sensor feature extraction model is obtained that can extract temporal feature vectors from sensor data.

[0013] Preferably, the cross-modal fusion model is a Transformer-based model that uses a multi-head attention mechanism, where the video spatiotemporal feature vector serves as the query vector, and the audio spectral feature vector and sensor temporal feature vector together serve as the key and value vectors; the training process of the cross-modal fusion model includes: Obtain video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors; The feature vectors of each modality are projected onto a feature space of a unified dimension and then input into a neural network based on the Transformer architecture. A multi-head attention mechanism is used to dynamically weight and fuse the projected features, and a combined loss function and AdamW optimizer are used to optimize the model parameters, resulting in a well-trained cross-modal fusion model that can output a unified fused feature vector.

[0014] Preferably, the behavior recognition result includes a specific behavior category and a confidence probability of that behavior category, a timestamp corresponding to the specific behavior category, and the original data index information associated with the specific behavior category. The behavior categories include normal operating behavior and abnormal / violation operating behavior. The method further includes: S6. When the behavior recognition result belongs to a preset abnormal / violation behavior category, an alarm signal is triggered.

[0015] Preferably, in step S1, the sensor data includes accelerometer data and gyroscope data from the inertial measurement unit, as well as vibration sensor data from the device; the preprocessing and time synchronization of the multimodal data to obtain preprocessed multimodal data specifically includes: Acquire video, audio, and sensor data in industrial settings; The video data is denoised using Gaussian filtering, the audio data is denoised using high-pass filtering to remove low-frequency noise, and the sensor data is normalized using the Z-score method. Based on the timestamps of each modal data, the processed video data, audio data, and sensor data are aligned to obtain time-synchronized preprocessed multimodal data.

[0016] The present invention proposes an industrial behavior recognition and detection system, comprising: The data acquisition and preprocessing module is used to acquire multimodal data in industrial scenarios, and to preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data. The pseudo-label generation module is used to generate a pseudo-label dataset containing behavioral category labels based on the preprocessed multimodal data, using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. The feature extraction module is used to input video data with pseudo-label datasets into a trained video feature extraction model to extract video spatiotemporal feature vectors, input audio data with pseudo-label datasets into a trained audio feature extraction model to extract audio spectral feature vectors, and input sensor data with pseudo-label datasets into a trained sensor feature extraction model to extract sensor temporal feature vectors. The cross-modal fusion module is used to input video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors into the trained cross-modal fusion model, and perform dynamic weighted fusion through an attention mechanism to output a unified fused feature vector; The behavior recognition module is used to input the fused feature vector into a preset behavior recognition classifier to obtain the corresponding behavior recognition result.

[0017] The proposed industrial behavior recognition and detection method and system in this invention effectively overcomes the problem of decreased recognition accuracy caused by changes in lighting, partial occlusion, and environmental noise in industrial scenarios by combining a multimodal dynamic fusion mechanism with automated annotation technology, significantly improving the accuracy and environmental adaptability of the behavior recognition model. The automated annotation method greatly reduces the reliance on manual annotation, reducing data preparation costs and time, while cross-modal collaborative verification ensures the quality of pseudo-label data. In addition, the model structure has been specifically optimized for the characteristics of industrial data, introducing an attention mechanism and lightweight network design in the feature extraction and fusion stages, which improves recognition efficiency while ensuring the real-time operation capability of the system on edge devices. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the workflow of an industrial behavior recognition and detection method proposed in this invention; Figure 2 This is a schematic diagram of one embodiment of the industrial behavior recognition and detection method proposed in this invention; Figure 3 This is a schematic diagram of one embodiment of the industrial behavior recognition and detection method proposed in this invention; Figure 4 This is a schematic diagram of the system architecture of an industrial behavior recognition and detection system proposed in this invention. Detailed Implementation

[0019] Reference Figures 1-4 The present invention proposes an industrial behavior recognition and detection method, comprising the following steps: S1. Acquire multimodal data in industrial scenarios, and preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data.

[0020] In this embodiment, in step S1, the sensor data includes accelerometer data and gyroscope data from the inertial measurement unit, as well as vibration sensor data from the device; preprocessing and time synchronization of the multimodal data to obtain preprocessed multimodal data specifically includes: Acquire video, audio, and sensor data in industrial settings; Gaussian filtering is used to denoise the video data, high-pass filtering is used to remove low-frequency noise from the audio data, and Z-score is used to normalize the sensor data. Based on the timestamps of each modal data, the processed video data, audio data, and sensor data are aligned to obtain time-synchronized preprocessed multimodal data.

[0021] Specifically, video data: video was captured using an RGB camera at a frame rate of 30fps. Audio data: captured via a microphone at a sampling frequency of 44.1kHz. Sensor data: readings from sensors such as an IMU were collected at a sampling rate of 100Hz.

[0022] S2. Based on the preprocessed multimodal data, a pseudo-label dataset containing behavioral category labels is generated using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning.

[0023] In this embodiment, step S2 specifically includes: Obtain preprocessed multimodal data and a small seed dataset containing manually labeled behavioral categories; Multimodal data is input into a pre-trained feature extraction model to extract initial features for each modality. A self-supervised contrastive learning algorithm is used to generate preliminary behavioral labels for video and sensor data. At the same time, a weakly supervised multi-instance learning algorithm is used to infer fine-grained behavioral labels for audio data based on a seed dataset. By using a cross-modal consistency verification mechanism, the initial behavioral labels and fine-grained behavioral labels are jointly verified and noise filtered to obtain the final pseudo-label dataset.

[0024] Specifically, the pre-trained feature extraction models include the ResNet model, the MFCC feature extractor, and the LSTM network.

[0025] It should be noted that automated labeling uses algorithms to generate pseudo-labels, reducing manual intervention and supporting the training of multimodal behavior recognition models. Its principle is based on self-supervised / weakly supervised learning to generate high-quality labels.

[0026] Self-supervised learning principle: Labels are generated through the inherent structure of the data, without the need for external annotation. For example, it utilizes the temporal consistency of video frames (to predict the next frame) or the periodicity of sensor sequences to learn behavioral patterns. The core is contrastive learning (such as SimCLR): positive samples (similar transformed data) are compared with negative samples to generate pseudo-labels.

[0027] The principle of weakly supervised learning is to infer fine-grained labels using a small number of annotations, such as in multi-instance learning (MIL): videos are treated as "bags", requiring only video-level labels, and frame-level labels are aggregated through attention.

[0028] S3. Input the video data with pseudo-label dataset into the trained video feature extraction model to extract the video spatiotemporal feature vector. Input the audio data with pseudo-label dataset into the trained audio feature extraction model to extract the audio spectrum feature vector. Input the sensor data with pseudo-label dataset into the trained sensor feature extraction model to extract the sensor temporal feature vector.

[0029] In this embodiment, in step S3, the video feature extraction model is a ResNet-50 or ResNet-101 network, and the training process of the video feature extraction model includes: Obtain video data with pseudo-labels, which includes behavioral categories of normal and illegal operations. The video frame sequence is input into the initial ResNet-50 or ResNet-101 network, and spatiotemporal features are extracted through 3D convolutional layers; The extracted spatiotemporal features are reduced in dimensionality using a max pooling layer to generate a 2048-dimensional feature vector. The model parameters are optimized using the cross-entropy loss function and the Adam optimizer until the model converges to the preset value, thus obtaining a trained video feature extraction model.

[0030] Specifically, the implementation process of the video feature extraction step is as follows: First, video data was collected and a dataset was constructed. On-site video data was collected using fixed or mobile cameras deployed on the assembly line, with a frame rate of 30fps and a resolution of 1920×1080, ensuring coverage of the workers' main operating areas (such as screw tightening and parts assembly). Combining automated annotation-generated pseudo-labels with on-site data collection, a video dataset of approximately 1000 hours was constructed, encompassing various behavioral categories, including normal operations (such as screw tightening and parts assembly) and violations. To ensure data quality, 10% (approximately 100 hours) of the total data was allocated as a seed dataset for precise manual annotation. Finally, the complete dataset was divided into training, validation, and test sets in a 7:2:1 ratio.

[0031] Subsequently, the acquired raw video frames were preprocessed. Gaussian filtering was used to denoise the video frames, and the images were cropped according to the actual application scenario requirements to remove irrelevant backgrounds (such as the conveyor belt area), thereby highlighting the main area of ​​worker operation.

[0032] Finally, a video spatiotemporal feature extraction model was constructed and trained. This model employs a ResNet-50 network structure, utilizing its internal residual blocks to mitigate the gradient vanishing problem in deep networks. Specifically, continuous video frames are processed using 3D convolutional kernels (3×3×3 in size) to simultaneously capture the spatial contours and temporal dynamics of worker actions (e.g., the hand movement trajectory and rhythm in the "tightening screws" action). The extracted spatiotemporal features are then dimensionality-reduced using max-pooling layers, ultimately generating a 2048-dimensional feature vector to represent the deep semantic information of the action. During model training, the cross-entropy loss function, combined with the Adam optimizer, is used to optimize the network parameters. After training, the model can output a 2048-dimensional spatiotemporal feature vector for each frame or group of video frames. This vector serves as an abstract representation of the video modality and is output for use by the subsequent multimodal fusion module.

[0033] In this embodiment, the audio feature extraction model is a model built based on MFCC coefficients and a VGGish network; the training process of the audio feature extraction model includes: Obtain audio data with pseudo-labels; The audio data is pre-emphasized and framed, and then subjected to Fast Fourier Transform, Mel filter bank processing and Discrete Cosine Transform in sequence to extract MFCC features. The MFCC features were input into a VGGish-based neural network model, and the model was fine-tuned using a binary cross-entropy loss function and an Adam optimizer. A well-trained audio feature extraction model is obtained that can extract spectral feature vectors from audio signals.

[0034] Specifically, in the audio feature extraction stage of the industrial behavior recognition and detection method, the specific implementation process is as follows: First, audio data was collected and prepared. Audio data was collected using a microphone array deployed on the assembly line, with a sampling frequency of 44.1 kHz. The collected audio covered worker voice commands, tool sounds (such as drills and hammering sounds), and equipment operation sounds. Pseudo-labels were generated based on self-supervised contrastive learning, and combined with an automated annotation mechanism, to construct an industrial audio dataset of approximately 1000 hours. This dataset was then proportionally divided into a training set (70%), a validation set (20%), and a test set (10%), used for model training, generalization ability evaluation, and final performance validation, respectively.

[0035] Subsequently, the original audio signal undergoes preprocessing and feature extraction. The preprocessing stage includes pre-emphasis, framing (25ms frame length, 10ms frame shift), and high-pass filtering to remove low-frequency noise. The feature extraction stage employs the Mel Frequency Cepstral Coefficients (MFCC) algorithm: first, the time-domain signal is converted to a spectrum using a Fast Fourier Transform (FFT); then, a Mel filter bank with 40 filters maps the linear spectrum to the Mel scale to simulate human auditory perception; finally, a Discrete Cosine Transform (DCT) is performed on the Mel spectrum, and the first 13 to 40 coefficients are used as key vectors characterizing the audio's time-frequency features.

[0036] Finally, an audio classification model was constructed and trained. The extracted MFCC features were used as input, and the network weights were initialized using a VGGish model pre-trained on a large-scale audio dataset. The model was fine-tuned for specific audio events in industrial scenarios (such as drilling sounds and knocking sounds). During training, a 1-second 44.1kHz audio clip was input, and the model output a 40-dimensional feature vector. The binary cross-entropy loss function (BCE Loss) combined with the Adam optimizer was used to optimize the model parameters. Ultimately, the trained audio feature extraction model effectively captures key sound features related to behavior (such as "drilling sounds" and "workers shouting in violation of regulations") and outputs high-quality audio spectral feature vectors for subsequent multimodal fusion.

[0037] In this embodiment, the sensor feature extraction model is a two-layer LSTM network; the training process of the sensor feature extraction model includes: Obtain sensor time-series data with pseudo-labeled datasets; Z-score normalization was performed on the sensor time series data to eliminate dimensional differences; The normalized time series data is input into the initial two-layer LSTM network, and the network parameters are optimized using the cross-entropy loss function and the RMSprop optimizer. A well-trained sensor feature extraction model is obtained that can extract temporal feature vectors from sensor data.

[0038] Specifically, the implementation process of the sensor feature extraction step is as follows: First, sensor data was collected and a dataset was constructed. Raw data on worker movements (e.g., arm movements) and equipment states (e.g., conveyor belt vibration) were collected and recorded using inertial measurement units (IMUs) worn by workers (containing accelerometers and gyroscopes, with a sampling rate of 100Hz) and vibration sensors installed on equipment. Pseudo-labels were generated based on a weakly supervised multi-instance learning algorithm, and combined with an automated annotation process, to construct a sensor time-series dataset with a total duration of approximately 1000 hours. This dataset was divided into training, validation, and test sets in a 7:2:1 ratio for model training, hyperparameter tuning, and final performance evaluation, respectively.

[0039] Subsequently, the acquired raw sensor data was preprocessed. Z-score normalization was used to process the IMU and vibration sensor readings to eliminate differences caused by different physical dimensions, ensuring the data were on the same scale and laying a stable foundation for subsequent model training.

[0040] Finally, a sensor temporal feature extraction model was constructed and trained. This model employs a two-layer Long Short-Term Memory (LSTM) network with 128 hidden layer units. During training, preprocessed time-series data (sampling rate 100Hz, with 1-second time windows) was input into the network. The model captures long-term dependencies in the data through its internal gating mechanism, thereby learning the temporal patterns of worker action patterns (e.g., "rapidly tightening screws") or equipment states (e.g., "abnormal vibration of the conveyor belt"). Cross-entropy loss was used during training, and the network parameters were optimized using the RMSprop optimizer, suitable for training with time-series data. After training, the model can convert the input sensor time-series data into a 128-dimensional temporal feature vector. This vector serves as an abstract representation of the sensor modalities and is output for use by the subsequent multimodal fusion module.

[0041] S4. Input the video spatiotemporal feature vector, audio spectrum feature vector, and sensor temporal feature vector into the trained cross-modal fusion model, perform dynamic weighted fusion through the attention mechanism, and output a unified fusion feature vector.

[0042] In this embodiment, the cross-modal fusion model is a Transformer-based model that uses a multi-head attention mechanism. The video spatiotemporal feature vector serves as the query vector, while the audio spectral feature vector and sensor temporal feature vector together serve as the key and value vectors. The training process of the cross-modal fusion model includes: Obtain video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors; The feature vectors of each modality are projected onto a feature space of a unified dimension and then input into a neural network based on the Transformer architecture. A multi-head attention mechanism is used to dynamically weight and fuse the projected features, and a combined loss function and AdamW optimizer are used to optimize the model parameters, resulting in a well-trained cross-modal fusion model that can output a unified fused feature vector.

[0043] Specifically, the Transformer-based attention mechanism is a deep learning framework based on self-attention and multi-head attention for processing sequential data. It is particularly suitable for dynamically integrating video, audio, and sensor features in multimodal fusion. The core of this mechanism lies in calculating intermodal correlations through a query-key-value (QKV) model, achieving dynamic weighted fusion and avoiding the static nature and information loss problems of traditional fusion methods (such as simple concatenation). The core of Transformer is the attention mechanism, which simulates the human attention allocation process, calculating the interdependencies of elements in the input sequence, rather than relying on the sequential processing of RNNs. The calculation formula for the attention mechanism is: Attention(Q,K,V)=softmax(QK T / (d k ) 1 / 2 V; Where Q (Query) is the query vector, representing the focus of the current modality; K (Key) is the key vector, used for matching relevance; V (Value) is the value vector, carrying actual information; d k It is the dimension of the key vector, (d k ) 1 / 2 Used for scaling to prevent gradient explosion; the softmax function converts attention scores into a probability distribution, ensuring that the weights sum to 1.

[0044] The algorithm computes in parallel using multiple attention heads, each learning a representation of a different subspace. The representations are then concatenated and linearly transformed to output the result. MultiHead(Q,K,V)=Concat(head1,head2,…,head h W o ; Among them, head i =Attention(QW i Q KW i K VW i V ), W i Q W i K Wi V It is the projection matrix, h is the number of heads, Concat means concatenating the outputs of all heads into a single matrix, W o It is the output projection matrix.

[0045] It should be noted that, in order to adapt to multimodal scenarios, the original Scaled Dot-Product Attention has been modified, and the modified formula is as follows: Attention = Softmax(Q) v (Ka⊕Ks) T ) / (d k ) 1 / 2 ); Q v The query vector comes from the video modality; Ka and Ks come from the key vectors from the audio and sensor modalities, respectively; ⊕ indicates concatenation or weighted combination of the key vectors from the audio and sensor modalities (e.g., weighted summation).

[0046] S5. Input the fused feature vector into the preset behavior recognition classifier to obtain the corresponding behavior recognition result.

[0047] In this embodiment, the behavior recognition result includes a specific behavior category and a confidence probability of the specific behavior category, a timestamp corresponding to the specific behavior category, and the original data index information associated with the specific behavior category. The behavior categories include normal operation behavior and abnormal / violation operation behavior.

[0048] Specifically, normal operating behaviors include actions such as "tightening screws," "assembling parts," and "walking normally." Abnormal / unauthorized operating behaviors include actions such as "unauthorized climbing," "unauthorized use of tools," "running," and "falling down."

[0049] Specifically, the action recognition classifier includes convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a fully connected layer + Softmax classifier. The fused feature vector is input into the action recognition classifier. After being locally refined by the CNN to eliminate noise and enhance short-term patterns, the processed fused feature vector is then input into the RNN to model temporal relationships. Finally, a fully connected layer outputs the action class probability to complete the action recognition.

[0050] S6. When the behavior recognition result belongs to the preset abnormal / violation behavior category, an alarm signal is triggered.

[0051] like Figure 2As shown, through dynamic fusion and complementary feature extraction, the system's recognition accuracy in complex industrial environments (such as noise SNR < 10dB and illumination variation > 50%) is improved from 85% to over 95% compared to traditional methods. The false positive rate for anomaly detection (such as violations of regulations or fatigue) is reduced by 20%, effectively reducing potential safety hazards.

[0052] like Figure 3 As shown, automated labeling generates diverse pseudo-labels, supporting cross-scenario migration (taking manufacturing assembly lines as an example, scene migration between different production lines), reducing generalization error by 10-15%. This enables the system to adapt to unseen behavioral patterns, improving reliability on dynamic production lines.

[0053] Reference Figures 1-4 The present invention proposes an industrial behavior recognition and detection system, comprising: The data acquisition and preprocessing module is used to acquire multimodal data in industrial scenarios, and to preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data. The pseudo-label generation module is used to generate a pseudo-label dataset containing behavioral category labels based on preprocessed multimodal data, using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. The feature extraction module is used to input video data with pseudo-label datasets into a trained video feature extraction model to extract video spatiotemporal feature vectors, input audio data with pseudo-label datasets into a trained audio feature extraction model to extract audio spectral feature vectors, and input sensor data with pseudo-label datasets into a trained sensor feature extraction model to extract sensor temporal feature vectors. The cross-modal fusion module is used to input video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors into the trained cross-modal fusion model, and perform dynamic weighted fusion through an attention mechanism to output a unified fused feature vector; The behavior recognition module is used to input the fused feature vector into a preset behavior recognition classifier to obtain the corresponding behavior recognition result.

[0054] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An industrial behavior recognition and detection method, characterized in that, Includes the following steps: S1. Acquire multimodal data in industrial scenarios, and preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data; S2. Based on the preprocessed multimodal data, a pseudo-label dataset containing behavioral category labels is generated using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. S3. Input the video data with pseudo-label dataset into the trained video feature extraction model to extract the video spatiotemporal feature vector; input the audio data with pseudo-label dataset into the trained audio feature extraction model to extract the audio spectrum feature vector; input the sensor data with pseudo-label dataset into the trained sensor feature extraction model to extract the sensor temporal feature vector. S4. Input the video spatiotemporal feature vector, audio spectrum feature vector, and sensor temporal feature vector into the trained cross-modal fusion model, perform dynamic weighted fusion through the attention mechanism, and output a unified fusion feature vector. S5. Input the fused feature vector into the preset behavior recognition classifier to obtain the corresponding behavior recognition result; Specifically, step S2 includes: Obtain preprocessed multimodal data and a small seed dataset containing manually labeled behavioral categories; Multimodal data is input into a pre-trained feature extraction model to extract initial features for each modality. A self-supervised contrastive learning algorithm is used to generate preliminary behavioral labels for video and sensor data. At the same time, a weakly supervised multi-instance learning algorithm is used to infer fine-grained behavioral labels for audio data based on a seed dataset. The preliminary behavior labels and fine-grained behavior labels are jointly verified and noise filtered through a cross-modal consistency verification mechanism to obtain the final pseudo-label dataset.

2. The industrial behavior recognition and detection method according to claim 1, characterized in that, The pre-trained feature extraction model includes the ResNet model, the MFCC feature extractor, and the LSTM network.

3. The industrial behavior recognition and detection method according to claim 1, characterized in that, In step S3, the video feature extraction model is a ResNet-50 or ResNet-101 network, and the training process of the video feature extraction model includes: Acquire video data with a pseudo-label dataset, the video data containing behavioral categories of normal and illegal operations; The video frame sequence is input into the initial ResNet-50 or ResNet-101 network, and spatiotemporal features are extracted through 3D convolutional layers; The extracted spatiotemporal features are reduced in dimensionality using a max pooling layer to generate a 2048-dimensional feature vector. The model parameters are optimized using the cross-entropy loss function and the Adam optimizer until the model converges to the preset value, thus obtaining a trained video feature extraction model.

4. The industrial behavior recognition and detection method according to claim 1, characterized in that, The audio feature extraction model is a model built based on MFCC coefficients and a VGGish network; the training process of the audio feature extraction model includes: Obtain audio data with pseudo-labels; The audio data is pre-emphasized and framed, and then subjected to Fast Fourier Transform, Mel filter bank processing and Discrete Cosine Transform in sequence to extract MFCC features. The MFCC features are input into a VGGish-based neural network model, and the model is fine-tuned using a binary cross-entropy loss function and an Adam optimizer. A well-trained audio feature extraction model is obtained that can extract spectral feature vectors from audio signals.

5. The industrial behavior recognition and detection method according to claim 1, characterized in that, The sensor feature extraction model is a two-layer LSTM network; the training process of the sensor feature extraction model includes: Obtain sensor time-series data with pseudo-labeled datasets; The time series data from the sensor are subjected to Z-score normalization to eliminate dimensional differences; The normalized time series data is input into the initial two-layer LSTM network, and the network parameters are optimized using the cross-entropy loss function and the RMSprop optimizer. A well-trained sensor feature extraction model is obtained that can extract temporal feature vectors from sensor data.

6. The industrial behavior recognition and detection method according to claim 1, characterized in that, The cross-modal fusion model is based on the Transformer architecture and uses a multi-head attention mechanism. The video spatiotemporal feature vector serves as the query vector, while the audio spectral feature vector and sensor temporal feature vector together serve as the key and value vectors. The training process of the cross-modal fusion model includes: Obtain video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors; The feature vectors of each modality are projected onto a feature space of a unified dimension and then input into a neural network based on the Transformer architecture. A multi-head attention mechanism is used to dynamically weight and fuse the projected features, and a combined loss function and AdamW optimizer are used to optimize the model parameters, resulting in a well-trained cross-modal fusion model that can output a unified fused feature vector.

7. The industrial behavior recognition and detection method according to claim 1, characterized in that, The behavior recognition result includes a specific behavior category and its confidence probability, a timestamp corresponding to the specific behavior category, and the original data index information associated with the specific behavior category. Behavior categories include normal operation behavior and abnormal / violation operation behavior. The method further includes: S6. When the behavior recognition result belongs to a preset abnormal / violation behavior category, an alarm signal is triggered.

8. The industrial behavior recognition and detection method according to claim 1, characterized in that, In step S1, the sensor data includes accelerometer data and gyroscope data from the inertial measurement unit, as well as vibration sensor data from the device; the preprocessing and time synchronization of the multimodal data to obtain preprocessed multimodal data specifically includes: Acquire video, audio, and sensor data in industrial settings; The video data is denoised using Gaussian filtering, the audio data is denoised using high-pass filtering to remove low-frequency noise, and the sensor data is normalized using the Z-score method. Based on the timestamps of each modal data, the processed video data, audio data, and sensor data are aligned to obtain time-synchronized preprocessed multimodal data.

9. An industrial behavior recognition and detection system, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data in industrial scenarios, and to preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data. The pseudo-label generation module is used to generate a pseudo-label dataset containing behavioral category labels based on the preprocessed multimodal data, using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. The feature extraction module is used to input video data with pseudo-label datasets into a trained video feature extraction model to extract video spatiotemporal feature vectors, input audio data with pseudo-label datasets into a trained audio feature extraction model to extract audio spectral feature vectors, and input sensor data with pseudo-label datasets into a trained sensor feature extraction model to extract sensor temporal feature vectors. The cross-modal fusion module is used to input video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors into the trained cross-modal fusion model, and perform dynamic weighted fusion through an attention mechanism to output a unified fused feature vector; The behavior recognition module is used to input the fused feature vector into a preset behavior recognition classifier to obtain the corresponding behavior recognition result; The pseudo-tag generation module specifically includes: Obtain preprocessed multimodal data and a small seed dataset containing manually labeled behavioral categories; Multimodal data is input into a pre-trained feature extraction model to extract initial features for each modality. A self-supervised contrastive learning algorithm is used to generate preliminary behavioral labels for video and sensor data. At the same time, a weakly supervised multi-instance learning algorithm is used to infer fine-grained behavioral labels for audio data based on a seed dataset. The preliminary behavior labels and fine-grained behavior labels are jointly verified and noise filtered through a cross-modal consistency verification mechanism to obtain the final pseudo-label dataset.

Images