Industrial behavior identification detection method and system

Abstract

The invention discloses an industrial behavior identification detection method and system, and the method comprises the steps: collecting video, audio and sensor data in an industrial scene, carrying out the preprocessing and time synchronization, and generating a pseudo-label data set through a pre-training feature extraction model in combination with self-supervision and weak supervision learning; video time-space features are extracted through a ResNet network, audio frequency spectrum features are extracted through combination of MFCC and a VGGish network, and sensor time sequence features are extracted through an LSTM network; and carrying out dynamic weighted fusion on the multi-modal features by adopting an attention mechanism based on Transform, and finally realizing behavior recognition and abnormity alarm through a classifier. According to the method, the problem of low recognition accuracy caused by illumination variation, shielding and noise interference in an industrial scene is effectively solved, the data labeling cost is remarkably reduced, and the model generalization ability and the system real-time performance are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to an industrial behavior recognition detection method and system. BACKGROUND

[0002] In modern industrial production environments, the standardization and safety of worker operations are directly related to production efficiency and accident prevention. Traditional behavior recognition methods mainly rely on single modal data, especially video surveillance data. However, the industrial site environment is complex and variable, and there are many challenges: first, unstable lighting conditions, such as uneven lighting, too dark or overexposed, can seriously affect the quality of video images, resulting in a significant decrease in the performance of video-based behavior recognition models. Studies have shown that under severe lighting conditions, even advanced video baselines such as ResNet can have a 10-20% decrease in recognition accuracy. Second, the problem of occlusion is widespread. In dense work scenes such as assembly lines, workers' body parts are often partially occluded by mechanical equipment, tools, or other personnel, resulting in the loss of key action information and increasing the difficulty of behavior recognition. For example, when a worker's arm is occluded by assembly equipment, video-based models have difficulty accurately determining their specific operation behavior.

[0003] With the development of deep learning technology, video-based motion behavior recognition methods have become mainstream. Among them, three-dimensional convolutional neural networks can capture spatial features and temporal dynamics by extending two-dimensional convolution to the time dimension, but they face challenges in terms of parameter quantity and computational complexity. To improve efficiency, some studies have reduced kernel size and optimized feature reuse mechanisms to reduce parameter quantity. Another typical method uses a "space first, then time" processing strategy, which first uses 2DCNN to extract spatial features, and then aggregates temporal information through long short-term memory networks or convolutional long short-term memory networks. However, this method can cause information loss when processing two-dimensional spatial features.

[0004] Although multi-modal fusion technology has made some progress in medical, transportation and other fields, there are still few specialized behavior recognition systems for industrial scenarios. Existing systems often fail to fully consider the information provided by industrial-specific sensors such as vibration sensors and temperature sensors. At the same time, in terms of modal fusion strategy, existing methods mostly use simple early fusion or late fusion, lacking sufficient consideration of the complexity of industrial scenarios, resulting in low fusion efficiency and difficulty in achieving real-time processing of large-scale industrial data.

[0005] In addition, the prior art also has two key bottlenecks: on the one hand, deep learning model training relies on large-scale high-quality labeled data, and the labeling of behavior data in industrial scenarios requires the participation of professionals, which is costly and difficult to ensure consistency; on the other hand, the dynamic changes of industrial environments make the trained model lack generalization ability when facing new work scenarios, and the recognition accuracy is difficult to meet the actual needs of industrial applications. SUMMARY

[0006] To solve the technical problems in the background art, the present application proposes an industrial behavior recognition detection method and system.

[0007] The industrial behavior recognition detection method proposed by the present application comprises the following steps: S1, acquiring multi-modal data in an industrial scenario, and pre-processing and time synchronizing the multi-modal data to obtain pre-processed multi-modal data, the multi-modal data including video data, audio data and sensor data; S2, based on the pre-processed multi-modal data, using a pre-trained feature extraction model, generating a pseudo-label data set containing behavior class labels in a combination of self-supervised and weakly supervised learning; S3, inputting the video data with the pseudo-label data set into a trained video feature extraction model to extract a video spatio-temporal feature vector, inputting the audio data with the pseudo-label data set into a trained audio feature extraction model to extract an audio spectrum feature vector, and inputting the sensor data with the pseudo-label data set into a trained sensor feature extraction model to extract a sensor time series feature vector; S4, inputting the video spatio-temporal feature vector, the audio spectrum feature vector and the sensor time series feature vector into a trained cross-modal fusion model, dynamically weighting and fusing through an attention mechanism to output a unified fusion feature vector; S5, inputting the fusion feature vector into a pre-set behavior recognition classifier to obtain a corresponding behavior recognition result.

[0008] Preferably, step S2 specifically comprises: acquiring the pre-processed multi-modal data and a small-scale seed data set containing artificially labeled behavior classes; inputting the multi-modal data into a pre-trained feature extraction model to extract initial features of each modality, and using a self-supervised contrast learning algorithm to generate preliminary behavior labels for the video data and the sensor data, while using a weakly supervised multi-instance learning algorithm to infer fine-grained behavior labels for the audio data based on the seed data set; cooperatively verifying and filtering noise of the preliminary behavior labels and the fine-grained behavior labels through a cross-modal consistency verification mechanism to obtain a final pseudo-label data set.

[0009] Preferably, the pre-trained feature extraction model comprises a ResNet model, an MFCC feature extractor, and an LSTM network.

[0010] Preferably, in step S3, the video feature extraction model is a ResNet-50 or ResNet-101 network, and the training process of the video feature extraction model comprises: obtaining video data with a pseudo-label dataset, the video data containing behavior categories of normal operation and violation operation; inputting a video frame sequence into an initial ResNet-50 or ResNet-101 network, and extracting spatio-temporal features through a 3D convolution layer; using a max-pooling layer to reduce the dimension of the extracted spatio-temporal features, and generating a 2048-dimensional feature vector; using a cross-entropy loss function and an Adam optimizer to optimize the model parameters until the model converges to a preset, to obtain a trained video feature extraction model.

[0011] Preferably, the audio feature extraction model is a model constructed based on MFCC coefficients and a VGGish network; and the training process of the audio feature extraction model comprises: obtaining audio data with a pseudo-label dataset; performing pre-emphasis and frame processing on the audio data, and then sequentially performing fast Fourier transform, Mel filter bank processing, and discrete cosine transform to extract MFCC features; inputting the MFCC features into a neural network model based on VGGish, and using a binary cross-entropy loss function and an Adam optimizer to fine-tune the model; obtaining a trained audio feature extraction model capable of extracting a spectral feature vector from audio signals.

[0012] Preferably, the sensor feature extraction model is a double-layer LSTM network; and the training process of the sensor feature extraction model comprises: obtaining sensor time series data with a pseudo-label dataset; performing Z-score normalization processing on the sensor time series data to eliminate dimensional differences; inputting the normalized time series data into an initial double-layer LSTM network, and using a cross-entropy loss function and an RMSprop optimizer to optimize network parameters; obtaining a trained sensor feature extraction model capable of extracting a time series feature vector from sensor data.

[0013] Preferably, the cross-modal fusion model is a model based on a Transformer architecture, which uses a multi-head attention mechanism, wherein the video spatio-temporal feature vector is used as a query vector, and the audio spectrum feature vector and the sensor time series feature vector are used as key vectors and value vectors; the training process of the cross-modal fusion model comprises: obtaining a video spatio-temporal feature vector, an audio spectrum feature vector and a sensor time series feature vector; projecting each modal feature vector into a unified-dimensional feature space and inputting a neural network based on a Transformer architecture; using a multi-head attention mechanism to dynamically weight and fuse the projected features, and using a combination loss function and an AdamW optimizer to optimize the model parameters to obtain a trained cross-modal fusion model capable of outputting unified fusion feature vectors.

[0014] Preferably, the behavior recognition result includes a specific behavior category, a confidence probability of the specific behavior category, a timestamp corresponding to the specific behavior category and original data index information associated with the specific behavior category, and the behavior category includes normal operation behavior and abnormal / violation operation behavior; the method further comprises: S6, when the behavior recognition result belongs to a preset abnormal / violation operation behavior category, triggering an alarm signal.

[0015] Preferably, in step S1, the sensor data includes accelerometer data from an inertial measurement unit, gyroscope data and vibration sensor data from a device; the pre-processing and time synchronization of the multi-modal data to obtain the pre-processed multi-modal data specifically comprises: obtaining video data, audio data and sensor data in an industrial scene; performing denoising processing on the video data using Gaussian filtering, removing low-frequency noise from the audio data using high-pass filtering, and performing normalization processing on the sensor data using a Z-score method; aligning the processed video data, audio data and sensor data based on the timestamps of the modal data to obtain time-synchronized pre-processed multi-modal data.

[0016] The present application provides an industrial behavior recognition detection system, comprising: a data acquisition and preprocessing module for obtaining multi-modal data in an industrial scene and pre-processing and time synchronizing the multi-modal data to obtain pre-processed multi-modal data, wherein the multi-modal data includes video data, audio data and sensor data; The pseudo label generation module is configured to generate a pseudo label data set containing behavior category labels based on the preprocessed multi-modal data by using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. The feature extraction module is configured to input the video data with the pseudo label data set into a trained video feature extraction model to extract a video space-time feature vector, input the audio data with the pseudo label data set into a trained audio feature extraction model to extract an audio spectrum feature vector, and input the sensor data with the pseudo label data set into a trained sensor feature extraction model to extract a sensor time sequence feature vector. The cross-modal fusion module is configured to input the video space-time feature vector, the audio spectrum feature vector, and the sensor time sequence feature vector into a trained cross-modal fusion model to perform dynamic weighted fusion through an attention mechanism and output a unified fusion feature vector. The behavior recognition module is configured to input the fusion feature vector into a preset behavior recognition classifier to obtain a corresponding behavior recognition result.

[0017] In the present application, the industrial behavior recognition detection method and system effectively overcome the problem of reduced recognition accuracy caused by changes in light, local occlusion, and environmental noise in an industrial scene by combining a multi-modal dynamic fusion mechanism with an automatic labeling technology, significantly improving the accuracy and environmental adaptability of the behavior recognition model. The use of an automatic labeling method significantly reduces the dependence on manual labeling, reduces data preparation costs and cycles, and ensures the quality of pseudo label data through cross-modal collaborative verification. In addition, the model structure is optimized for industrial data characteristics, and attention mechanisms and lightweight network designs are introduced in the feature extraction and fusion stages to improve recognition efficiency while ensuring real-time operation capabilities on edge devices. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A workflow schematic diagram of an industrial behavior recognition detection method proposed in the present application; Figure 2 A structure schematic diagram of an embodiment of an industrial behavior recognition detection method proposed in the present application; Figure 3 A structure schematic diagram of an embodiment of an industrial behavior recognition detection method proposed in the present application; Figure 4 A system architecture schematic diagram of an industrial behavior recognition detection system proposed in the present application. DETAILED DESCRIPTION

[0019] REFERENCE Figures 1-4 The industrial behavior recognition detection method proposed in the present application includes the following steps: S1, acquire multi-modal data in an industrial scene, and pre-process and time synchronize the multi-modal data to obtain pre-processed multi-modal data, the multi-modal data including video data, audio data and sensor data.

[0020] In this embodiment, in step S1, the sensor data includes accelerometer data from an inertial measurement unit, gyroscope data, and vibration sensor data from a device; the multi-modal data is pre-processed and time synchronized to obtain pre-processed multi-modal data, specifically including: acquiring video data, audio data and sensor data in an industrial scene; using Gaussian filtering to denoise the video data, using high-pass filtering to remove low-frequency noise from the audio data, and using the Z-score method to normalize the sensor data; aligning the processed video data, audio data and sensor data based on the timestamps of the respective modal data to obtain time-synchronized pre-processed multi-modal data.

[0021] Specifically, the video data: the video is collected using an RGB camera, and the video frame rate is 30fps. The audio data: is collected by a microphone, and the sampling audio frequency is 44.1kHz. The sensor data: IMU and other sensor readings are collected, and the sampling rate is 100Hz.

[0022] S2, based on the pre-processed multi-modal data, using a pre-trained feature extraction model, a pseudo-label data set containing behavior class labels is generated by combining self-supervised and weakly supervised learning.

[0023] In this embodiment, step S2 specifically includes: acquiring pre-processed multi-modal data and a small-scale seed data set containing manually labeled behavior classes; inputting the multi-modal data into the pre-trained feature extraction model to extract initial features of each modality, and using a self-supervised contrast learning algorithm to generate preliminary behavior labels for video data and sensor data, while using a weakly supervised multi-instance learning algorithm to infer fine-grained behavior labels for audio data based on the seed data set; cooperatively verifying and filtering noise of the preliminary behavior labels and the fine-grained behavior labels through a cross-modal consistency verification mechanism to obtain a final pseudo-label data set.

[0024] Specifically, the pre-trained feature extraction model includes a ResNet model, an MFCC feature extractor and an LSTM network.

[0025] It should be noted that automatic labeling uses algorithms to generate pseudo-labels, reduces human intervention, and supports multi-modal behavior recognition model training. Its principle is based on self-supervised / weakly supervised learning to generate high-quality labels.

[0026] Self-supervised learning principle: generate labels from the structure of data itself, without external annotation. For example, learn behavior patterns by using the temporal consistency of video frames (predict the next frame) or the periodicity of sensor sequences. The core is contrastive learning (such as SimCLR): compare positive samples (similarly transformed data) with negative samples to generate pseudo-labels.

[0027] Weakly supervised learning principle: use a small amount of labels to infer fine-grained labels, such as multi-instance learning (MIL): treat a video as a "bag", only need video-level labels, aggregate frame-level labels through attention.

[0028] S3, input the video data with pseudo-label dataset into the trained video feature extraction model to extract video spatio-temporal feature vector, input the audio data with pseudo-label dataset into the trained audio feature extraction model to extract audio spectrum feature vector, input the sensor data with pseudo-label dataset into the trained sensor feature extraction model to extract sensor time series feature vector.

[0029] In this embodiment, in step S3, the video feature extraction model is ResNet-50 or ResNet-101 network, and the training process of the video feature extraction model includes: Obtain video data with pseudo-label dataset, which contains normal operation and violation operation behavior categories; Input video frame sequence into initial ResNet-50 or ResNet-101 network, extract spatio-temporal features through 3D convolution layer; Use max-pooling layer to reduce dimension of extracted spatio-temporal features, generate 2048-dimensional feature vector; Use cross-entropy loss function and Adam optimizer to optimize model parameters until model converges to preset, to obtain trained video feature extraction model.

[0030] Specifically, the specific implementation process of the video feature extraction link is as follows: First, collect video data and build dataset. Deploy fixed or mobile cameras on the assembly line to collect on-site video data, set video frame rate to 30fps, resolution to 1920x1080, and ensure that the main operation area of workers (such as screwing, part assembly, etc.) is covered. Combine pseudo-labels generated by automatic labeling with on-site collection to build a video dataset with a total duration of about 1000 hours, which contains normal operation (such as screwing, part assembly) and violation operation and other behavior categories. Among them, to ensure data quality, 10% (about 100 hours) of the total data is divided into a seed dataset annotated by humans. Finally, divide the complete dataset into training set, validation set and test set in the ratio of 7:2:1.

[0031] Subsequently, the acquired raw video frames were preprocessed. Gaussian filtering was used to denoise the video frames, and the images were cropped according to the actual application scenario requirements to remove irrelevant backgrounds (such as the conveyor belt area), thereby highlighting the main area of ​​worker operation.

[0032] Finally, a video spatiotemporal feature extraction model was constructed and trained. This model employs a ResNet-50 network structure, utilizing its internal residual blocks to mitigate the gradient vanishing problem in deep networks. Specifically, continuous video frames are processed using 3D convolutional kernels (3×3×3 in size) to simultaneously capture the spatial contours and temporal dynamics of worker actions (e.g., the hand movement trajectory and rhythm in the "tightening screws" action). The extracted spatiotemporal features are then dimensionality-reduced using max-pooling layers, ultimately generating a 2048-dimensional feature vector to represent the deep semantic information of the action. During model training, the cross-entropy loss function, combined with the Adam optimizer, is used to optimize the network parameters. After training, the model can output a 2048-dimensional spatiotemporal feature vector for each frame or group of video frames. This vector serves as an abstract representation of the video modality and is output for use by the subsequent multimodal fusion module.

[0033] In this embodiment, the audio feature extraction model is a model built based on MFCC coefficients and a VGGish network; the training process of the audio feature extraction model includes: Obtain audio data with pseudo-labels; The audio data is pre-emphasized and framed, and then subjected to Fast Fourier Transform, Mel filter bank processing and Discrete Cosine Transform in sequence to extract MFCC features. The MFCC features were input into a VGGish-based neural network model, and the model was fine-tuned using a binary cross-entropy loss function and an Adam optimizer. A well-trained audio feature extraction model is obtained that can extract spectral feature vectors from audio signals.

[0034] Specifically, in the audio feature extraction stage of the industrial behavior recognition and detection method, the specific implementation process is as follows: First, audio data was collected and prepared. Audio data was collected using a microphone array deployed on the assembly line, with a sampling frequency of 44.1 kHz. The collected audio covered worker voice commands, tool sounds (such as drills and hammering sounds), and equipment operation sounds. Pseudo-labels were generated based on self-supervised contrastive learning, and combined with an automated annotation mechanism, to construct an industrial audio dataset of approximately 1000 hours. This dataset was then proportionally divided into a training set (70%), a validation set (20%), and a test set (10%), used for model training, generalization ability evaluation, and final performance validation, respectively.

[0035] Subsequently, the original audio signal undergoes preprocessing and feature extraction. The preprocessing stage includes pre-emphasis, framing (25ms frame length, 10ms frame shift), and high-pass filtering to remove low-frequency noise. The feature extraction stage employs the Mel Frequency Cepstral Coefficients (MFCC) algorithm: first, the time-domain signal is converted to a spectrum using a Fast Fourier Transform (FFT); then, a Mel filter bank with 40 filters maps the linear spectrum to the Mel scale to simulate human auditory perception; finally, a Discrete Cosine Transform (DCT) is performed on the Mel spectrum, and the first 13 to 40 coefficients are used as key vectors characterizing the audio's time-frequency features.

[0036] Finally, an audio classification model was constructed and trained. The extracted MFCC features were used as input, and the network weights were initialized using a VGGish model pre-trained on a large-scale audio dataset. The model was fine-tuned for specific audio events in industrial scenarios (such as drilling sounds and knocking sounds). During training, a 1-second 44.1kHz audio clip was input, and the model output a 40-dimensional feature vector. The binary cross-entropy loss function (BCE Loss) combined with the Adam optimizer was used to optimize the model parameters. Ultimately, the trained audio feature extraction model effectively captures key sound features related to behavior (such as "drilling sounds" and "workers shouting in violation of regulations") and outputs high-quality audio spectral feature vectors for subsequent multimodal fusion.

[0037] In this embodiment, the sensor feature extraction model is a two-layer LSTM network; the training process of the sensor feature extraction model includes: Obtain sensor time-series data with pseudo-labeled datasets; Z-score normalization was performed on the sensor time series data to eliminate dimensional differences; The normalized time series data is input into the initial two-layer LSTM network, and the network parameters are optimized using the cross-entropy loss function and the RMSprop optimizer. A well-trained sensor feature extraction model is obtained that can extract temporal feature vectors from sensor data.

[0038] Specifically, the implementation process of the sensor feature extraction step is as follows: First, sensor data was collected and a dataset was constructed. Raw data on worker movements (e.g., arm movements) and equipment states (e.g., conveyor belt vibration) were collected and recorded using inertial measurement units (IMUs) worn by workers (containing accelerometers and gyroscopes, with a sampling rate of 100Hz) and vibration sensors installed on equipment. Pseudo-labels were generated based on a weakly supervised multi-instance learning algorithm, and combined with an automated annotation process, to construct a sensor time-series dataset with a total duration of approximately 1000 hours. This dataset was divided into training, validation, and test sets in a 7:2:1 ratio for model training, hyperparameter tuning, and final performance evaluation, respectively.

[0039] Subsequently, the acquired raw sensor data was preprocessed. Z-score normalization was used to process the IMU and vibration sensor readings to eliminate differences caused by different physical dimensions, ensuring the data were on the same scale and laying a stable foundation for subsequent model training.

[0040] Finally, a sensor temporal feature extraction model was constructed and trained. This model employs a two-layer Long Short-Term Memory (LSTM) network with 128 hidden layer units. During training, preprocessed time-series data (sampling rate 100Hz, with 1-second time windows) was input into the network. The model captures long-term dependencies in the data through its internal gating mechanism, thereby learning the temporal patterns of worker action patterns (e.g., "rapidly tightening screws") or equipment states (e.g., "abnormal vibration of the conveyor belt"). Cross-entropy loss was used during training, and the network parameters were optimized using the RMSprop optimizer, suitable for training with time-series data. After training, the model can convert the input sensor time-series data into a 128-dimensional temporal feature vector. This vector serves as an abstract representation of the sensor modalities and is output for use by the subsequent multimodal fusion module.

[0041] S4. Input the video spatiotemporal feature vector, audio spectrum feature vector, and sensor temporal feature vector into the trained cross-modal fusion model, perform dynamic weighted fusion through the attention mechanism, and output a unified fusion feature vector.

[0042] In this embodiment, the cross-modal fusion model is a Transformer-based model that uses a multi-head attention mechanism. The video spatiotemporal feature vector serves as the query vector, while the audio spectral feature vector and sensor temporal feature vector together serve as the key and value vectors. The training process of the cross-modal fusion model includes: Obtain video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors; The feature vectors of each modality are projected onto a feature space of a unified dimension and then input into a neural network based on the Transformer architecture. A multi-head attention mechanism is used to dynamically weight and fuse the projected features, and a combined loss function and AdamW optimizer are used to optimize the model parameters, resulting in a well-trained cross-modal fusion model that can output a unified fused feature vector.

[0043] Specifically, the Transformer-based attention mechanism is a deep learning framework based on self-attention and multi-head attention for processing sequential data. It is particularly suitable for dynamically integrating video, audio, and sensor features in multimodal fusion. The core of this mechanism lies in calculating intermodal correlations through a query-key-value (QKV) model, achieving dynamic weighted fusion and avoiding the static nature and information loss problems of traditional fusion methods (such as simple concatenation). The core of Transformer is the attention mechanism, which simulates the human attention allocation process, calculating the interdependencies of elements in the input sequence, rather than relying on the sequential processing of RNNs. The calculation formula for the attention mechanism is: Attention(Q,K,V)=softmax(QK T / (d k ) 1 / 2 V; Where Q (Query) is the query vector, representing the focus of the current modality; K (Key) is the key vector, used for matching relevance; V (Value) is the value vector, carrying actual information; d k It is the dimension of the key vector, (d k ) 1 / 2 Used for scaling to prevent gradient explosion; the softmax function converts attention scores into a probability distribution, ensuring that the weights sum to 1.

[0044] The algorithm computes in parallel using multiple attention heads, each learning a representation of a different subspace. The representations are then concatenated and linearly transformed to output the result. MultiHead(Q,K,V)=Concat(head1,head2,…,head h W o ; Among them, head i =Attention(QW i Q KW i K VW i V ), W i Q W i K Wi V It is the projection matrix, h is the number of heads, Concat means concatenating the outputs of all heads into a single matrix, W o It is the output projection matrix.

[0045] It should be noted that, in order to adapt to multimodal scenarios, the original Scaled Dot-Product Attention has been modified, and the modified formula is as follows: Attention = Softmax(Q) v (Ka⊕Ks) T ) / (d k ) 1 / 2 ); Q v The query vector comes from the video modality; Ka and Ks come from the key vectors from the audio and sensor modalities, respectively; ⊕ indicates concatenation or weighted combination of the key vectors from the audio and sensor modalities (e.g., weighted summation).

[0046] S5. Input the fused feature vector into the preset behavior recognition classifier to obtain the corresponding behavior recognition result.

[0047] In this embodiment, the behavior recognition result includes a specific behavior category and a confidence probability of the specific behavior category, a timestamp corresponding to the specific behavior category, and the original data index information associated with the specific behavior category. The behavior categories include normal operation behavior and abnormal / violation operation behavior.

[0048] Specifically, normal operating behaviors include actions such as "tightening screws," "assembling parts," and "walking normally." Abnormal / unauthorized operating behaviors include actions such as "unauthorized climbing," "unauthorized use of tools," "running," and "falling down."

[0049] Specifically, the action recognition classifier includes convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a fully connected layer + Softmax classifier. The fused feature vector is input into the action recognition classifier. After being locally refined by the CNN to eliminate noise and enhance short-term patterns, the processed fused feature vector is then input into the RNN to model temporal relationships. Finally, a fully connected layer outputs the action class probability to complete the action recognition.

[0050] S6. When the behavior recognition result belongs to the preset abnormal / violation behavior category, an alarm signal is triggered.

[0051] like Figure 2As shown, through dynamic fusion and complementary feature extraction, the system's recognition accuracy in complex industrial environments (such as noise SNR < 10dB and illumination variation > 50%) is improved from 85% to over 95% compared to traditional methods. The false positive rate for anomaly detection (such as violations of regulations or fatigue) is reduced by 20%, effectively reducing potential safety hazards.

[0052] like Figure 3 As shown, automated labeling generates diverse pseudo-labels, supporting cross-scenario migration (taking manufacturing assembly lines as an example, scene migration between different production lines), reducing generalization error by 10-15%. This enables the system to adapt to unseen behavioral patterns, improving reliability on dynamic production lines.

[0053] Reference Figures 1-4 The present invention proposes an industrial behavior recognition and detection system, comprising: The data acquisition and preprocessing module is used to acquire multimodal data in industrial scenarios, and to preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data. The pseudo-label generation module is used to generate a pseudo-label dataset containing behavioral category labels based on preprocessed multimodal data, using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. The feature extraction module is used to input video data with pseudo-label datasets into a trained video feature extraction model to extract video spatiotemporal feature vectors, input audio data with pseudo-label datasets into a trained audio feature extraction model to extract audio spectral feature vectors, and input sensor data with pseudo-label datasets into a trained sensor feature extraction model to extract sensor temporal feature vectors. The cross-modal fusion module is used to input video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors into the trained cross-modal fusion model, and perform dynamic weighted fusion through an attention mechanism to output a unified fused feature vector. The behavior recognition module is used to input the fused feature vector into a preset behavior recognition classifier to obtain the corresponding behavior recognition result.

[0054] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An industrial behavior recognition and detection method, characterized in that, Includes the following steps: S1. Acquire multimodal data in industrial scenarios, and preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data; S2. Based on the preprocessed multimodal data, a pseudo-label dataset containing behavioral category labels is generated using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. S3. Input the video data with pseudo-label dataset into the trained video feature extraction model to extract the video spatiotemporal feature vector; input the audio data with pseudo-label dataset into the trained audio feature extraction model to extract the audio spectrum feature vector; input the sensor data with pseudo-label dataset into the trained sensor feature extraction model to extract the sensor temporal feature vector. S4. Input the video spatiotemporal feature vector, audio spectrum feature vector, and sensor temporal feature vector into the trained cross-modal fusion model, perform dynamic weighted fusion through the attention mechanism, and output a unified fusion feature vector. S5. Input the fused feature vector into the preset behavior recognition classifier to obtain the corresponding behavior recognition result.

2. The industrial behavior recognition and detection method according to claim 1, characterized in that, Step S2 specifically includes: Obtain preprocessed multimodal data and a small seed dataset containing manually labeled behavioral categories; Multimodal data is input into a pre-trained feature extraction model to extract initial features for each modality. A self-supervised contrastive learning algorithm is used to generate preliminary behavioral labels for video and sensor data. At the same time, a weakly supervised multi-instance learning algorithm is used to infer fine-grained behavioral labels for audio data based on a seed dataset. The preliminary behavior labels and fine-grained behavior labels are jointly verified and noise filtered through a cross-modal consistency verification mechanism to obtain the final pseudo-label dataset.

3. The industrial behavior recognition and detection method according to claim 2, characterized in that, The pre-trained feature extraction model includes the ResNet model, the MFCC feature extractor, and the LSTM network.

4. The industrial behavior recognition and detection method according to claim 1, characterized in that, In step S3, the video feature extraction model is a ResNet-50 or ResNet-101 network, and the training process of the video feature extraction model includes: Acquire video data with a pseudo-label dataset, the video data containing behavioral categories of normal and illegal operations; The video frame sequence is input into the initial ResNet-50 or ResNet-101 network, and spatiotemporal features are extracted through 3D convolutional layers; The extracted spatiotemporal features are reduced in dimensionality using a max pooling layer to generate a 2048-dimensional feature vector. The model parameters are optimized using the cross-entropy loss function and the Adam optimizer until the model converges to the preset value, thus obtaining a trained video feature extraction model.

5. The industrial behavior recognition and detection method according to claim 1, characterized in that, The audio feature extraction model is a model built based on MFCC coefficients and a VGGish network; the training process of the audio feature extraction model includes: Obtain audio data with pseudo-labels; The audio data is pre-emphasized and framed, and then subjected to Fast Fourier Transform, Mel filter bank processing and Discrete Cosine Transform in sequence to extract MFCC features. The MFCC features are input into a VGGish-based neural network model, and the model is fine-tuned using a binary cross-entropy loss function and an Adam optimizer. A well-trained audio feature extraction model is obtained that can extract spectral feature vectors from audio signals.

6. The industrial behavior recognition and detection method according to claim 1, characterized in that, The sensor feature extraction model is a two-layer LSTM network; the training process of the sensor feature extraction model includes: Obtain sensor time-series data with pseudo-labeled datasets; The time series data from the sensor are subjected to Z-score normalization to eliminate dimensional differences; The normalized time series data is input into the initial two-layer LSTM network, and the network parameters are optimized using the cross-entropy loss function and the RMSprop optimizer. A well-trained sensor feature extraction model is obtained that can extract temporal feature vectors from sensor data.

7. The industrial behavior recognition and detection method according to claim 1, characterized in that, The cross-modal fusion model is based on the Transformer architecture and uses a multi-head attention mechanism. The video spatiotemporal feature vector serves as the query vector, while the audio spectral feature vector and sensor temporal feature vector together serve as the key and value vectors. The training process of the cross-modal fusion model includes: Obtain video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors; The feature vectors of each modality are projected onto a feature space of a unified dimension and then input into a neural network based on the Transformer architecture. A multi-head attention mechanism is used to dynamically weight and fuse the projected features, and a combined loss function and AdamW optimizer are used to optimize the model parameters, resulting in a well-trained cross-modal fusion model that can output a unified fused feature vector.

8. The industrial behavior recognition and detection method according to claim 1, characterized in that, The behavior recognition result includes a specific behavior category and its confidence probability, a timestamp corresponding to the specific behavior category, and the original data index information associated with the specific behavior category. Behavior categories include normal operation behavior and abnormal / violation operation behavior. The method further includes: S6. When the behavior recognition result belongs to a preset abnormal / violation behavior category, an alarm signal is triggered.

9. The industrial behavior recognition and detection method according to claim 1, characterized in that, In step S1, the sensor data includes accelerometer data and gyroscope data from the inertial measurement unit, as well as vibration sensor data from the device; the preprocessing and time synchronization of the multimodal data to obtain preprocessed multimodal data specifically includes: Acquire video, audio, and sensor data in industrial settings; The video data is denoised using Gaussian filtering, the audio data is denoised using high-pass filtering to remove low-frequency noise, and the sensor data is normalized using the Z-score method. Based on the timestamps of each modal data, the processed video data, audio data, and sensor data are aligned to obtain time-synchronized preprocessed multimodal data.

10. An industrial behavior recognition and detection system, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data in industrial scenarios, and to preprocess and synchronize the multimodal data in time to obtain preprocessed multimodal data, which includes video data, audio data and sensor data. The pseudo-label generation module is used to generate a pseudo-label dataset containing behavioral category labels based on the preprocessed multimodal data, using a pre-trained feature extraction model and a combination of self-supervised and weakly supervised learning. The feature extraction module is used to input video data with pseudo-label datasets into a trained video feature extraction model to extract video spatiotemporal feature vectors, input audio data with pseudo-label datasets into a trained audio feature extraction model to extract audio spectral feature vectors, and input sensor data with pseudo-label datasets into a trained sensor feature extraction model to extract sensor temporal feature vectors. The cross-modal fusion module is used to input video spatiotemporal feature vectors, audio spectral feature vectors, and sensor temporal feature vectors into the trained cross-modal fusion model, and perform dynamic weighted fusion through an attention mechanism to output a unified fused feature vector. The behavior recognition module is used to input the fused feature vector into a preset behavior recognition classifier to obtain the corresponding behavior recognition result.

Images