Method and system for video behavior recognition based on neural network

Abstract

The application discloses a method and system for video behavior recognition based on a neural network, and relates to the technical field of computer vision.The application efficiently integrates video, audio and acceleration data through a multi-modal fusion network, dynamically adjusts the fusion weight, solves the problem of multi-modal data heterogeneity and interference, improves the robustness and accuracy of the behavior recognition system, adopts a light-weight convolution network and a dynamic time warping algorithm to optimize the calculation efficiency and real-time performance, meets the industrial real-time monitoring demand, further uses time series analysis and a conditional random field model to accurately locate the start and end time points of abnormal behavior, dynamically adjusts the model parameters, realizes adaptive optimization, and guarantees the safety of worker operation and production efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for video behavior recognition based on neural networks. Background Technology

[0002] In business scenarios where video behavior recognition is applied to real-time monitoring of worker operations in large industrial production workshops, numerous severe challenges arise. It requires integrating video images captured by cameras, ambient audio recorded by microphones, and accelerometer data provided by wearable devices worn by workers. However, multimodal inputs are highly heterogeneous and subject to significant interference; in video images, worker movements and postures are often blurred due to occlusion or changes in lighting; in audio streams, machine noises and worker conversations are mixed, masking subtle auditory cues related to behavior; and sensor data is offset by different device placement positions. When workers operate multiple devices simultaneously, the video frame sequence contains multiple actors, making it difficult to distinguish the relationship between foreground workers and background equipment during spatial feature extraction. The time series of audio signals is affected by noise superposition, masking subtle sound cues related to behavior. Furthermore, the sensor data sampling frequency is inconsistent with the video frame rate, and time alignment easily introduces spurious correlation interference. Existing models typically tend to focus on the dominant features of a single modality, such as focusing only on significant actions in the image while ignoring supplementary contextual information provided by audio or sensors. Moreover, behavior recognition needs to be completed within milliseconds to meet real-time requirements, but in complex scenarios, redundant features generated by multimodal fusion lead to an imbalance in the allocation of computational resources. How to design a mechanism that adaptively filters key spatiotemporal regions and efficiently integrates three types of data under limited computing power has become a pressing technical challenge. Summary of the Invention

[0003] The objective of this invention can be achieved through the following technical solutions:

[0004] This application provides a method for video behavior recognition based on neural networks, including the following steps:

[0005] S1. Obtain raw image data from the video frame sequence, extract spatial features of each frame through a convolutional neural network, input the extracted features into a preset multi-target detection algorithm, and identify the preliminary segmentation region between the foreground worker and the background equipment.

[0006] S2. Based on the segmentation results, an attention mechanism is used to weight the spatial features. When the overlap ratio of targets within the segmentation area exceeds a preset threshold, the depth map estimation module is called to separate the overlapping subjects and determine the spatial location features of the main operators.

[0007] S3. Obtain time-series signals from ambient audio recorded by the microphone, decompose the audio signals into a spectrum through short-time Fourier transform, use an adaptive filter to suppress machine noise based on spectrum characteristics, and extract subtle sound cues related to behavior.

[0008] S4. Acquire acceleration sensor data provided by the equipment worn by the worker, analyze the time series trend through a sliding window, input the analysis results into the dynamic time warping algorithm, align with the video frame sequence, correct acceleration data offset, and extract acceleration change features;

[0009] S5. Based on spatial location features, subtle sound cues, and acceleration change features, a multimodal fusion network is used to integrate the three types of data at the feature level to obtain a joint feature vector of the multimodal data. When the signal-to-noise ratio of a certain modality is lower than a preset threshold, its fusion weight is dynamically reduced according to the signal-to-noise ratio quantization result to generate a preliminary behavioral feature vector.

[0010] S6. By using preset key area filtering rules, extract the spatiotemporal regions that are highly related to the operation behavior from the behavior feature vector, input the filtering results into the long short-term memory network, analyze the contextual supplementary information, and determine the dynamic sequence characteristics of the worker's operation behavior.

[0011] S7. After obtaining dynamic sequence features, a lightweight convolutional network is used to classify the spatiotemporal region. In combination with real-time monitoring requirements, a pre-trained binary classification model is called to determine whether the current behavior complies with safety regulations and output the detection results of abnormal behavior.

[0012] Furthermore, raw image data is obtained from the video frame sequence, and spatial features of each frame are extracted using a convolutional neural network. The extracted features are then input into a pre-defined multi-object detection algorithm to identify the preliminary segmentation regions of the foreground worker and the background equipment. Specifically, this includes:

[0013] The video stream data is obtained from the surveillance camera, and the video stream is split into frames in chronological order. Then the video file is read and extracted frame by frame in sequence to obtain the original image data.

[0014] The acquired raw image data is preprocessed to adjust the image size to fit the model input size, and the pixel value range is normalized.

[0015] The preprocessed image is input into a convolutional neural network, and convolution operations are performed by sliding different convolutional kernels on the image to extract local features.

[0016] The feature image output by the convolutional layer is sampled by the pooling layer, and then processed by multiple convolutional and pooling layers to obtain the spatial feature representation of each frame of the image;

[0017] The spatial features extracted by the convolutional neural network are input into a preset multi-object detection algorithm. The input image is divided into a grid, and each grid predicts nearby objects. After convolution operation and prediction head, the object category and bounding box are output. Based on the algorithm output, the preliminary segmentation regions of the foreground worker and background equipment in each frame image are obtained.

[0018] Furthermore, based on the segmentation results, an attention mechanism is used to weight the spatial features. When the overlap ratio of targets within the segmented region exceeds a preset threshold, the depth map estimation module is invoked to separate the overlapping subjects and determine the spatial location features of the main operators, specifically including:

[0019] The spatial features of each frame of image extracted by the convolutional neural network are integrated into a set of feature vectors. The similarity between feature vectors at different locations is calculated through a self-attention mechanism to generate an attention weight matrix.

[0020] Attention weights are obtained by normalization using the Softmax function, which represent the degree of correlation between features at different locations. The numerical vectors are then weighted and summed to obtain the weighted segmentation results.

[0021] Based on the weighted segmentation results, the overlap ratio of targets within the region is calculated. The degree of overlap is measured by comparing the ratio of the intersection to the union of the bounding boxes of different targets. A preset threshold is then set. When the ratio of multiple targets within a segmentation region exceeds the threshold, it indicates that there is target overlap.

[0022] Using a pre-trained depth estimation model, an image region containing overlapping targets is input into the model. Through feature learning and inference of the image, a depth map of the region is output, and the spatial location of the main operator is determined based on the depth map.

[0023] Based on the results of separating overlapping subjects using the depth map estimation module, and combined with the bounding box information of the segmented regions, the spatial location features of the main operator are extracted. The pixel coordinate range of the area where the main operator is located is determined from the depth map and then mapped back to the coordinate system of the original image. At the same time, features related to the worker's body posture are extracted. The location coordinate information, depth information, and posture-related information are integrated to form a complete spatial location feature description of the main operator.

[0024] Furthermore, by analyzing the time series trend through a sliding window, the analysis results are input into a dynamic time warping algorithm and aligned with the video frame sequence, specifically including:

[0025] The sliding window technique is used to analyze the trend of acceleration data over time. A fixed-length window is defined and slides sequentially over the time series data. Within each window, statistical analysis of the acceleration data is performed.

[0026] By continuously moving the sliding window, the entire time series data is analyzed segment by segment to obtain a series of curves showing how window statistics change over time, which can be used to observe the trend of the average acceleration of workers moving along the X-axis at different times.

[0027] The acceleration data time series obtained from sliding window analysis is used as the sequence to be aligned, and the timestamps of the video frame sequence are used as the reference sequence. By calculating the similarity matrix between the two sequences, an optimal time warping path is found so that the two sequences match in time.

[0028] Specifically, the distance between each data point in the two sequences is calculated to construct a distance matrix. Then, a dynamic programming algorithm is used to search for a path with the minimum cumulative distance on the distance matrix. This path represents the optimal alignment between the two sequences. The time axis of the acceleration data is stretched or compressed to align it with the video frame sequence in time.

[0029] Furthermore, by using preset key region filtering rules, spatiotemporal regions highly related to operational behaviors are extracted from the behavioral feature vectors. Specifically, this includes: inputting the generated preliminary behavioral feature vectors into the preset key region filtering rules, and using a vector dimension-by-dimensional checking algorithm to check and judge the behavioral feature vectors dimension by dimension. For each behavioral feature vector, in the spatial location feature part, a coordinate matching algorithm is used to check whether the elements representing the location in the vector are within the specified coordinate range; in the subtle sound cue feature part, a frequency component comparison algorithm is used to confirm whether the corresponding frequency component elements meet the set range; in the acceleration change feature part, an acceleration value judgment algorithm is used to determine whether the acceleration value-related elements are within the specified change range. When the behavioral feature vectors satisfy the filtering rules in the feature dimensions corresponding to each modality, the spatiotemporal region extraction algorithm is used to extract the corresponding spatiotemporal region.

[0030] S8. Based on the detection results of abnormal behavior, when a violation of operation or fatigue error is detected, the trend of changes in previous features is traced through time series analysis, and the behavior boundary is optimized by using a conditional random field model to determine the start and end time points of the abnormal behavior.

[0031] Furthermore, time series analysis methods are used to trace the changing trends of preceding characteristics, specifically including:

[0032] Once abnormal behavior is detected, time series analysis is used to trace the trend of preceding feature changes. From the stored multimodal data, the worker's spatial location feature vector sequence, audio spectrum feature sequence, and acceleration change value sequence corresponding to the video frame sequence within a specific time period before the time when the abnormal behavior was detected are extracted.

[0033] Using a time series analysis model, the stationarity of time series data for each modality is tested, the order is determined, and parameters are estimated. The preceding time series is fitted and predicted, and the trends of spatial location, audio frequency components, and acceleration changes before the occurrence of abnormal behavior are analyzed.

[0034] Furthermore, a Conditional Random Field (CRF) model is used to optimize the behavioral boundary and determine the start and end times of abnormal behavior. Specifically, this includes: constructing input features for the CRF model based on the analysis results; combining multimodal features from different time points into a multidimensional feature vector sequence; defining a feature function that can capture the relationship between features at adjacent time points and the association between features and labels; training the CRF model using labeled training data; learning parameters through an iterative optimization algorithm; and then inputting the data optimized by time series analysis and the CRF model into the abnormal behavior start and end time point determination module. Based on the label sequence output by the CRF model and combined with the timestamp information of the time series data, the start time point is determined when the CRF model first predicts the label as abnormal behavior, and the end time point is determined when the label is predicted as normal behavior at multiple consecutive time steps, thus determining the start and end times of abnormal behavior on the time axis.

[0035] S9. By analyzing the start and end times of abnormal behavior, combined with spatial location features and auditory cues, a behavior recognition report is generated using preset mapping rules. Based on the key area screening results, the parameters of the fusion network are dynamically adjusted to adaptively optimize the monitoring model.

[0036] This invention provides a video behavior recognition system based on neural networks, and a method for implementing video behavior recognition based on neural networks, comprising:

[0037] The video data processing module acquires video stream data from the surveillance camera, splits it into single-frame images in chronological order, reads the video file and extracts the original image data frame by frame, performs preprocessing, extracts spatial features through a convolutional neural network, and inputs them into a multi-object detection algorithm to identify the preliminary segmentation areas of the foreground worker and the background equipment.

[0038] The spatial feature optimization module uses an attention mechanism to weight spatial features based on the segmentation results, highlighting important feature areas. When targets overlap, the depth map estimation module is called to separate the overlapping subjects, and the complete spatial position of the main operator is determined by combining the worker's posture features.

[0039] The audio data processing module collects ambient audio through a microphone, converts sound waves into electrical signals and samples them, decomposes them into a spectrum using short-time Fourier transform, uses an adaptive filter to suppress machine noise, and extracts behavior-related sound cues.

[0040] The acceleration data processing module acquires acceleration sensor data from the equipment worn by the worker, analyzes the time series trend through sliding window technology, and uses a dynamic time warping algorithm to align with the video frame sequence, corrects data offset, and extracts acceleration change features.

[0041] The multimodal fusion module integrates spatial location features, subtle acoustic cue features, and acceleration change features at the feature level to obtain a joint feature vector, and dynamically adjusts the fusion weights according to the signal-to-noise ratio to generate a preliminary behavioral feature vector.

[0042] The key region screening and analysis module presets key region screening rules, performs a dimension-by-dimensional check on the behavioral feature vector, extracts the spatiotemporal regions that are highly related to the operational behavior, and inputs the long short-term memory network analysis context information to determine the dynamic sequence features.

[0043] The behavior classification and detection module uses depthwise separable convolution technology to extract low-dimensional feature representation tensors, which are then input into a pre-trained binary classification model. This model combines real-time monitoring requirements to determine whether the behavior complies with safety regulations and outputs abnormal behavior detection results.

[0044] The abnormal behavior tracing module, when abnormal behavior is detected, extracts the feature sequence before the abnormal behavior occurs from the stored multimodal data, optimizes the behavior boundary using time series analysis model and conditional random field model, and determines the start and end time points;

[0045] The report generation and model optimization module generates behavior recognition reports based on the start and end times of abnormal behavior, combined with spatial location features and sound cues. It also dynamically adjusts the fusion network parameters based on the key area screening results to adaptively optimize the monitoring model.

[0046] The beneficial effects of this invention are as follows:

[0047] By integrating video images, audio signals, and accelerometer sensor data at the feature level through a multimodal fusion network and dynamically adjusting the fusion weights according to the signal-to-noise ratio, the system effectively solves the problems of high heterogeneity of multimodal data and interference such as blurred video images, masked audio signals, and offset sensor data. This significantly improves the robustness and accuracy of the behavior recognition system, enabling it to operate stably in complex industrial production workshop environments.

[0048] By employing lightweight convolutional networks (such as MobileNet) and dynamic time warping (DTW) algorithms, the model parameters and computational load are reduced, while the temporal alignment accuracy of multimodal data is optimized. This addresses the issue that in complex scenarios, low computational efficiency often fails to meet real-time requirements, enabling efficient processing of multimodal data with limited computing power. This allows behavior recognition tasks to be completed in milliseconds, meeting the stringent requirements of real-time monitoring in industrial production, while also reducing dependence on hardware resources.

[0049] By utilizing time series analysis and conditional random field models, this invention traces the changing trends of preceding features, optimizes behavioral boundaries, and dynamically adjusts the parameters of the fusion network. This achieves precise localization and dynamic optimization of abnormal behaviors, improving the system's adaptability and accuracy, ensuring worker safety, and increasing production efficiency. By accurately locating the start and end times of abnormal behaviors, this invention overcomes the limitations of traditional methods, which often suffer from misjudgments and omissions in abnormal behavior detection and are difficult to adapt to complex and ever-changing industrial environments. It provides a more reliable solution for safety monitoring in industrial production. Attached Figure Description

[0050] To better understand and implement this application, the technical solution is described in detail below with reference to the accompanying drawings.

[0051] Figure 1 This is a flowchart illustrating the video behavior recognition method based on neural networks provided in Embodiment 1 of this application;

[0052] Figure 2 This is a schematic diagram of the structure of a video behavior recognition system based on a neural network provided in Embodiment 2 of this application. Detailed Implementation

[0053] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, exemplary embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and systems consistent with some aspects of this application as detailed in the appended claims.

[0054] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0055] The following detailed description of the specific implementation methods, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided in detail.

[0056] Example 1

[0057] Please see Figure 1 This embodiment provides a method for video behavior recognition based on neural networks, including the following steps:

[0058] S1. Obtain raw image data from the video frame sequence, extract spatial features of each frame through a convolutional neural network, input the extracted features into a preset multi-target detection algorithm, and identify the preliminary segmentation region between the foreground worker and the background equipment.

[0059] Furthermore, raw image data is obtained from the video frame sequence, and spatial features of each frame are extracted using a convolutional neural network. The extracted features are then input into a pre-defined multi-object detection algorithm to identify the preliminary segmentation regions of the foreground worker and the background equipment. Specifically, this includes:

[0060] The video stream data is obtained from the surveillance camera, and the video stream is split into frames in chronological order. Then the video file is read and extracted frame by frame in sequence to obtain the original image data.

[0061] The acquired raw image data is preprocessed to adjust the image size to fit the model input size, and the pixel value range is normalized.

[0062] The preprocessed image is input into a convolutional neural network, and convolution operations are performed by sliding different convolutional kernels on the image to extract local features.

[0063] The feature image output by the convolutional layer is sampled by the pooling layer, and then processed by multiple convolutional and pooling layers to obtain the spatial feature representation of each frame of the image;

[0064] The spatial features extracted by the convolutional neural network are input into a preset multi-object detection algorithm. Taking YOLO as an example, the input image is divided into a grid, and each grid predicts nearby objects. After convolution operation and prediction head, the object category and bounding box are output. Based on the algorithm output, the preliminary segmentation regions of the foreground worker and background equipment in each frame image are obtained.

[0065] S2. Based on the segmentation results, an attention mechanism is used to weight the spatial features. When the overlap ratio of targets within the segmentation area exceeds a preset threshold, the depth map estimation module is called to separate the overlapping subjects and determine the spatial location features of the main operators.

[0066] Furthermore, based on the segmentation results, an attention mechanism is used to weight the spatial features. When the overlap ratio of targets within the segmented region exceeds a preset threshold, the depth map estimation module is invoked to separate the overlapping subjects and determine the spatial location features of the main operators, specifically including:

[0067] Spatial features extracted from each frame of an image by a convolutional neural network are integrated into a feature vector set. A self-attention mechanism is used to calculate the similarity between feature vectors at different locations, generating an attention weight matrix. Specifically, the spatial feature vectors are linearly transformed with the query, key, and value matrices respectively to obtain the corresponding Q, K, and V vectors. Then, the dot product of the transposes of Q and K is calculated and divided by a scale factor. After normalization using the Softmax function, attention weights are obtained. These weights represent the degree of correlation between features at different locations. The value vectors are then weighted and summed to obtain the weighted segmentation region result. In this way, the attention mechanism can highlight important feature regions related to the target and suppress irrelevant or secondary information, thereby performing more effective weighted processing of spatial features and providing more targeted feature representations for subsequent analysis.

[0068] Based on the weighted segmentation results, the overlap ratio of targets within the region is calculated. The degree of overlap is measured by comparing the ratio of the intersection to the union of the bounding boxes of different targets. A preset threshold (e.g., 0.5, adjustable according to the actual scenario and requirements) is set. When the ratio of multiple targets within a segmented region exceeds this threshold, it indicates target overlap. At this point, the depth map estimation module is invoked. Depth map estimation aims to obtain depth information of objects in a scene and determine the front-to-back relationship between objects. Taking monocular depth estimation based on deep learning as an example, a pre-trained depth estimation model (such as the Monodepth series models based on convolutional neural networks) is used. The image region containing overlapping targets is input into the model. Through feature learning and inference of the image, a depth map of the region is output. Each pixel value in the map represents the depth information of the corresponding object. Based on the depth map, objects located in front and behind in the overlapping subjects can be clearly distinguished, thereby determining the spatial location of the main operator.

[0069] Based on the results of separating overlapping subjects using the depth map estimation module, and combined with the bounding box information of the segmented regions, the spatial location features of the main operator are extracted. The pixel coordinate range of the area where the main operator is located is determined from the depth map and then mapped back to the coordinate system of the original image. Simultaneously, considering the influence of the worker's posture and movements on the operation, features related to the worker's body posture, such as joint position information (obtainable through human posture estimation techniques), are further extracted. These location coordinates, depth information, and posture-related information are integrated to form a complete spatial location feature description of the main operator. These features not only clarify the worker's two-dimensional position in the image but also include depth information and posture state, providing comprehensive and accurate basic data for subsequent analysis of the worker's operational behavior. This helps to more accurately identify the worker's operational actions and determine whether they comply with safety regulations.

[0070] S3. Obtain time-series signals from ambient audio recorded by the microphone, decompose the audio signals into a spectrum through short-time Fourier transform, use an adaptive filter to suppress machine noise based on spectrum characteristics, and extract subtle sound cues related to behavior.

[0071] Furthermore, time-series signals are obtained from ambient audio recorded by a microphone. These signals are then decomposed into a spectrum using a short-time Fourier transform. An adaptive filter is employed to suppress machine noise, and subtle behavioral-related sound cues are extracted, including:

[0072] By collecting ambient audio from industrial production workshops and converting the sound waves into electrical signals, audio acquisition equipment samples the electrical signals at a specific sampling frequency (e.g., common 44.1kHz or 48kHz), converting continuous analog audio signals into discrete digital signal sequences—the time-series signals. In practical applications, professional audio acquisition libraries such as PyAudio (in a Python environment) are used to record audio and acquire time-series signals. Sampling parameters are set using relevant functions, the recording process is initiated, and the acquired audio data is stored in a suitable format (e.g., WAV) for subsequent processing.

[0073] After obtaining the time-series signal, a Short-Time Fourier Transform (STFT) is used to analyze the characteristics of the audio signal at different frequencies and times. The core idea of ​​STFT is to divide the audio signal into many short time intervals, assuming that the signal is approximately stationary within each interval. Within each short time interval, a conventional Fourier transform is performed on the signal. Specifically, a suitable window function (such as a Hanning window or Hamming window) is selected. The window function's role is to window the audio signal, making the transitions at the edges within the window smoother and reducing spectral leakage. The window function is then slid sequentially across the audio time series, performing a Fourier transform on the signal within each window, converting the time-domain signal into a frequency-domain signal. The result after STFT is a two-dimensional matrix, with the horizontal axis representing time and the vertical axis representing frequency. The values ​​in the matrix represent the signal amplitude at the corresponding time and frequency, i.e., the spectrum of the audio signal. This decomposes the audio signal into a spectrum showing the different frequency components changing over time, which helps to observe the frequency composition of the audio at different times.

[0074] In industrial production workshops, machine noise is the primary noise source, with relatively stable frequency and intensity exhibiting certain regularities. Adaptive filters can automatically adjust their parameters based on the characteristics of the input signal to achieve optimal filtering results. In this scenario, an adaptive filter based on the Least Mean Square (LMS) algorithm can be used. First, a reference signal containing only machine noise needs to be acquired (this can be collected when the equipment is running normally but there is no worker activity). The mixed audio signal containing machine noise and activity-related sounds is used as the input to the adaptive filter, and the reference signal is used as the filter's desired response. The LMS algorithm continuously adjusts the filter coefficients based on spectral characteristics to minimize the mean square error between the filter output and the desired response. During this process, the adaptive filter gradually learns the characteristics of the machine noise and generates a signal with the same amplitude but opposite phase to the machine noise, canceling it out of the mixed audio signal and thus suppressing the machine noise. After filtering, subtle activity-related sound cues become apparent in the remaining audio signal, such as the sounds of workers pressing buttons on equipment or tools colliding. These subtle auditory cues are of great significance for identifying workers' operational behaviors. These cues can be further analyzed and processed, for example, by using feature extraction algorithms to extract features such as the frequency, amplitude, and duration of the sound, which can then be used for training and judging subsequent behavior recognition models.

[0075] Specifically, acquiring time-series signals involves collecting raw audio data. The spectrum decomposed by short-time Fourier transform can clearly show the characteristics of audio at different frequencies and times, thereby understanding the distribution of machine noise and behavior-related sounds in the frequency domain. This provides a clear input signal for the adaptive filter. The adaptive filter uses the least mean square algorithm to adjust the filter coefficients based on the spectral characteristics, specifically suppressing machine noise with stable frequency and intensity patterns, thereby extracting those masked subtle behavior-related sound cues.

[0076] S4. Acquire acceleration sensor data provided by the equipment worn by the worker, analyze the time series trend through a sliding window, input the analysis results into the dynamic time warping algorithm, align with the video frame sequence, correct acceleration data offset, and extract acceleration change features;

[0077] Furthermore, acquiring accelerometer data includes:

[0078] The wearable device worn by the worker integrates an accelerometer to monitor the worker's motion in real time. The sensor detects changes in acceleration along three axes, converting the physical quantity into electrical signals. An internal microcontroller samples these signals at a specific sampling frequency and digitizes the data. In practical applications, this digitized acceleration data can be transmitted to a data processing terminal via wireless communication technologies such as Bluetooth and Wi-Fi. For example, when using Bluetooth Low Energy (BLE) technology, a connection can be established between the device and the terminal using relevant Bluetooth communication libraries (such as the bluepy library in Python) to receive and store the accelerometer data. This data is recorded as timestamps and triaxial acceleration values, forming a time-series dataset.

[0079] The time series trend is analyzed using a sliding window, and the analysis results are input into a dynamic time warping algorithm to align with the video frame sequence. Specifically, this includes:

[0080] A sliding window technique is used to analyze the trend of acceleration data over time. A fixed-length window (e.g., 100 data points, adjustable as needed) is defined and slid sequentially across the time series data. Within each window, statistical analysis is performed on the acceleration data, such as calculating the mean, variance, maximum, and minimum values. These statistics reflect the central tendency, dispersion, and range of variation of the acceleration data within the window. By continuously moving the sliding window, segment-by-segment analysis of the entire time series data is performed, resulting in a series of curves showing the changes of window statistics over time. For example, by calculating the mean of the X-axis acceleration within each window, the average acceleration trend of the worker moving along the X-axis at different times can be observed, allowing for a preliminary judgment of whether the worker's motion is accelerating, decelerating, or relatively stable. This sliding window analysis method effectively captures local features and trend changes in time series data, providing more valuable information for subsequent data alignment and feature extraction.

[0081] Because the sampling frequency of the accelerometer data is inconsistent with the video frame rate, time alignment is required to ensure accurate matching between the acceleration data and the corresponding video frames. Dynamic Time Warping (DTW) is a method for solving time series alignment problems. First, the acceleration data time series obtained from sliding window analysis is used as the sequence to be aligned, and the timestamps of the video frame sequence are used as the reference sequence. The DTW algorithm finds an optimal time warping path by calculating the similarity matrix between the two sequences, making them time-matched. Specifically, the distance between each data point in the two sequences (e.g., Euclidean distance) is calculated to construct a distance matrix. Then, a dynamic programming algorithm is used to search for the path with the minimum cumulative distance on the distance matrix; this path represents the optimal alignment relationship between the two sequences. The DTW algorithm can stretch or compress the time axis of the acceleration data to align it with the video frame sequence in time. For example, when the video frame rate is 30fps and the accelerometer sampling frequency is 100Hz, the DTW algorithm can determine how the corresponding acceleration data points should be selected or interpolated at each frame of the video, thus achieving accurate synchronization between the two.

[0082] During accelerometer data acquisition, data offsets may occur due to variations in device placement and sensor characteristics. To correct these offsets, a calibration-based method is employed. A calibration process is performed before the worker begins actual work, during which the worker remains stationary while acceleration data is collected. Theoretically, the acceleration value should be zero since the worker is stationary. Analysis of the calibration data calculates the average offset along each axis. This average offset is then subtracted from the subsequently acquired acceleration data to correct the data offset. For example, if the average offset along the X-axis is found to be 0.2 m / s² during calibration... 2 Therefore, in the actual collected X-axis acceleration data, 0.2 m / s² is subtracted from each data point. 2 This allows us to obtain more accurate data reflecting the actual acceleration of workers. After data offset correction, the acceleration data can more accurately reflect the actual motion state of workers, laying the foundation for subsequent extraction of accurate acceleration change features.

[0083] Acceleration data, after time alignment and data offset correction, can be used to extract acceleration variation features. These features include the first derivative of acceleration (i.e., the rate of change of acceleration, reflecting how fast acceleration changes), the second derivative (reflecting the change in the rate of change of acceleration, which can be used to identify sudden stops, starts, and other drastic changes in motion), and peak and trough values ​​of acceleration over different time periods. For example, by calculating the first derivative of the acceleration data, the rate of change of acceleration at each time point can be approximated using the finite difference method. If the rate of change of acceleration suddenly increases at a certain moment, it likely indicates that the worker has performed a rapid action, such as a quick turn or forceful operation of equipment. After extracting these acceleration variation features, they are combined into feature vectors. These feature vectors can comprehensively describe the dynamic changes in acceleration during the worker's movement, which is of great significance for identifying the worker's operational behavior. Subsequently, these feature vectors can be input into a behavior recognition model, along with features extracted from video images and audio data, to comprehensively determine whether the worker's operation complies with safety regulations and whether there is any abnormal behavior.

[0084] S5. Based on spatial location features, subtle sound cues, and acceleration change features, a multimodal fusion network is used to integrate the three types of data at the feature level to obtain a joint feature vector of the multimodal data. When the signal-to-noise ratio of a certain modality is lower than a preset threshold, its fusion weight is dynamically reduced according to the signal-to-noise ratio quantization result to generate a preliminary behavioral feature vector.

[0085] Furthermore, the spatial location feature vector, subtle sound cue feature vector, and acceleration change feature vector are concatenated sequentially to form a longer joint feature vector. This joint feature vector contains rich information from different modalities, but the weights of the features across each modality are the same. The weighted summation method assigns a weight to the feature vector of each modality, and then adds the weighted feature vectors together. The initial weights can be set based on experience or simple heuristics, for example, initially setting the weight of each modality to 1 / 3. Through this feature-level integration, the data from different modalities complement each other at the feature level, providing richer information for a comprehensive description of worker behavior.

[0086] To address the differences in data quality across different modalities, it is necessary to evaluate the signal-to-noise ratio (SNR) of each modality. SNR is a metric that measures the ratio of effective signal to noise in a signal. For the spatial location features of video images, noise may originate from factors such as lighting changes and viewpoint occlusion; subtle sound cues in audio may be interfered with by noise such as machine noise; noise in acceleration data may stem from sensor errors or external electromagnetic interference. Specific algorithms are used to calculate the SNR of each modality. For example, for audio data, the SNR can be calculated by comparing the power spectral density of the signal with the power spectral density of the noise. A preset threshold is set (e.g., SNR = 10dB, which can be adjusted according to the actual scene and data characteristics). When the SNR of a certain modality is lower than this threshold, it indicates that the modality is significantly affected by noise and has low data quality. In this case, its weight in the fusion process is dynamically adjusted based on the quantized SNR result. Specifically, a linear or non-linear mapping relationship can be used to map the SNR value to a weight adjustment coefficient. For example, when the signal-to-noise ratio of audio data is 5dB, its weight adjustment coefficient is calculated to be 0.2 using a predefined mapping function (assuming the original weight is 1). In the weighted summation fusion method, the weight of the audio modality feature vector will be reduced from 1 / 3 to 0.2 / 3. Correspondingly, the weights of other modal data will be redistributed according to the principle that the total weight is 1, so as to highlight the contribution of high-quality modal data and reduce the impact of low-quality modal data on the final result.

[0087] After feature-level integration and dynamic weight adjustment, the feature vectors from different modalities are fused according to the adjusted weights to obtain a preliminary behavioral feature vector. This vector integrates information related to worker behavior from video images, audio, and accelerometer sensor data, and considers the quality differences between the various modalities. For example, in a weighted summation fusion method, the adjusted spatial location feature vector is multiplied by its corresponding weight, the subtle sound cue feature vector is multiplied by its adjusted weight, and the acceleration change feature vector is multiplied by its weight. These three weighted vectors are then summed to obtain the preliminary behavioral feature vector. This vector serves as an important input for subsequent behavior recognition analysis, used to further determine whether the worker's operational behavior complies with safety regulations and to identify potential abnormal actions. By using this multimodal fusion method combined with dynamic weight adjustment based on signal-to-noise ratio, data from different sources can be more effectively integrated in complex industrial production workshop environments, improving the accuracy and reliability of the behavior recognition system.

[0088] S6. By using preset key area filtering rules, extract the spatiotemporal regions that are highly related to the operation behavior from the behavior feature vector, input the filtering results into the long short-term memory network, analyze the contextual supplementary information, and determine the dynamic sequence characteristics of the worker's operation behavior.

[0089] Furthermore, the pre-defined key area filtering rules specifically cover the limitation of the numerical range of different modalities in the behavioral feature vector. In the spatial location feature dimension, through the analysis of a large amount of industrial production workshop worker operation behavior data, the position of workers in the video during standard operating procedures on specific equipment is observed, thereby defining specific coordinate intervals. In the sound feature dimension, the occurrence patterns of behavior-related sounds in the audio are determined, identifying the unique sound frequency components during equipment operation, and thus specifying the existence range of certain frequency components. In the acceleration change feature dimension, based on the correspondence between acceleration data and actions, the range of acceleration values ​​is set for the acceleration changes of specific operational actions. These rules are obtained through in-depth analysis of massive operational behavior samples containing video, audio, and acceleration sensor data, mining closely related spatiotemporal feature patterns.

[0090] The generated preliminary behavioral feature vectors are input into the preset key region filtering rules. The vector dimension-by-dimensional checking algorithm is used to check and judge the behavioral feature vectors dimension by dimension. For each behavioral feature vector, in the spatial location feature part, the coordinate matching algorithm is used to check whether the elements representing the location in the vector are within the specified coordinate interval; in the subtle sound clue feature part, the frequency component comparison algorithm is used to confirm whether the corresponding frequency component elements meet the set range; in the acceleration change feature part, the acceleration value judgment algorithm is used to determine whether the acceleration value related elements are within the specified change interval. When the behavioral feature vector meets the filtering rules in the feature dimensions corresponding to each modality, the spatiotemporal region extraction algorithm is used to extract its corresponding spatiotemporal region.

[0091] The filtered results are input into the LSTM network. The LSTM network's input gate, forget gate, and output gate structures utilize gating mechanisms to control the input, retention, and output of information. At each time step, the network receives the spatiotemporal feature vector of the current moment as input, combined with the hidden state information from the previous moment. The input gate uses an input gate control algorithm to determine which information from the current input data will be retained in the current memory unit; the forget gate uses a forget gate control algorithm to control which information from the previous moment's memory unit will be forgotten; the output gate, based on the current memory unit state and input information, uses an output gate control algorithm to determine the hidden state to be output to the next moment. Through this mechanism, the LSTM network can effectively utilize time series analysis algorithms to capture the changing trends of spatiotemporal features over time, as well as the contextual relationships between features at different moments. For example, it can learn the influence of the spatiotemporal features of the previous action on the features of subsequent actions when a worker performs a series of operations, thereby uncovering supplementary contextual information in the operational behavior.

[0092] After processing and analyzing the spatiotemporal features using an LSTM network, the hidden state sequence output by the network contains rich dynamic information about the worker's operational behavior. Statistical analysis algorithms are then applied to process the hidden state sequence, extracting the time and magnitude of peaks and troughs, which may correspond to key action nodes in the operational behavior. Alternatively, a rate of change algorithm can be used to calculate the rate of change of the hidden state sequence to reflect changes in the speed and acceleration of the operational behavior. These extracted key information are then combined into a new feature vector using a feature vector combination algorithm. This vector represents the dynamic sequence feature of the worker's operational behavior. This dynamic sequence feature vector comprehensively describes the dynamic changes of the worker's operational behavior in time and space.

[0093] S7. After obtaining dynamic sequence features, the spatiotemporal region is classified through a lightweight convolutional network. In combination with real-time monitoring requirements, a pre-trained binary classification model is called to determine whether the current behavior complies with safety regulations and output the detection results of abnormal behavior.

[0094] Furthermore, lightweight convolutional networks aim to reduce computational cost and model parameters while maintaining certain classification performance, thus meeting the stringent computational resource requirements of real-time monitoring. For example, the MobileNet series of networks employs depthwise separable convolution. When processing spatiotemporal regions, the acquired dynamic sequence feature vectors are reshaped into tensors suitable for input to lightweight convolutional networks.

[0095] After being input into a lightweight convolutional network, spatiotemporal features are extracted and transformed using depthwise separable convolution operations. Depthwise separable convolution decomposes conventional convolution into depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operations on the feature map of each channel individually, considering only spatial features, while pointwise convolution linearly combines the output channels of depthwise convolution using 1x1 convolution kernels to adjust the channel dimensions. This approach significantly reduces the number of convolution kernel parameters and computational cost. Through alternating processing of multiple convolutional layers, normalization layers, and activation function layers, the network gradually extracts more representative spatiotemporal features, outputting a low-dimensional feature representation tensor.

[0096] In this scenario, the low-dimensional feature representation tensor output by the lightweight convolutional network is used as the input to the pre-trained binary classification model. For the logistic regression-based binary classification model, the input feature tensor first undergoes a linear transformation, i.e., matrix multiplication with the model's weight matrix. Then, the linear transformation result is mapped to the [0,1] interval through the logistic function (Sigmoid function) to obtain the probability value of the behavior belonging to a certain category (such as safe behavior or abnormal behavior). For the SVM-based binary classification model, the input feature tensor is mapped in the high-dimensional space through a kernel function (such as the radial basis function kernel function). Then, an optimal hyperplane is found to separate the two types of behavioral features, and the model outputs the category to which the current behavior belongs.

[0097] Real-time monitoring requires the model to process data quickly and provide timely judgment results. Based on the probability value or category label output by the binary classification model, and in conjunction with the actual safety specification definition, behavior judgment is made. For example, if the probability value of a safe behavior output by the logistic regression-based binary classification model is greater than 0.5 (the threshold can be adjusted according to the actual scenario), the current behavior is determined to comply with the safety specification; otherwise, it is determined to be abnormal behavior. Once the behavior type is determined, the detection result of the abnormal behavior is output.

[0098] S8. Based on the detection results of abnormal behavior, when a violation of operation or fatigue error is detected, the trend of changes in previous features is traced through time series analysis, and the behavior boundary is optimized by using a conditional random field model to determine the start and end time points of the abnormal behavior.

[0099] Furthermore, when abnormal behavior is detected, time series analysis is used to trace the trend of preceding feature changes. From the stored multimodal data, the worker's spatial location feature vector sequence, audio spectrum feature sequence, and acceleration change value sequence corresponding to the video frame sequence within a specific time period before the time when the abnormal behavior was detected are extracted.

[0100] Using time series analysis models such as ARIMA, stationarity tests, order determination and parameter estimation are performed on the time series of each modality data. The preceding time series are fitted and predicted to analyze the changing trends of spatial location, audio frequency components and acceleration before the occurrence of abnormal behavior.

[0101] Based on the analysis results, the input features of the Conditional Random Field (CRF) model are constructed. Multimodal features at different time points are combined into a multidimensional feature vector sequence. Feature functions that can capture the relationship between features at adjacent time points and the association between features and labels are defined. The CRF model is trained using labeled training data, and the parameters are learned through iterative optimization algorithms. Finally, the data processed by time series analysis and CRF model optimization is input into the abnormal behavior start and end time point determination module. Based on the label sequence output by the CRF model and combined with the timestamp information of the time series data, the start time point is determined when the CRF model first predicts that the label is abnormal behavior, and the end time point is determined when the model predicts that the label is normal behavior for multiple consecutive time steps. This accurately determines the start and end time points of abnormal behavior on the time axis, providing a key basis for subsequent in-depth analysis and processing of abnormal behavior.

[0102] S9. By analyzing the start and end times of abnormal behavior, combined with spatial location features and auditory cues, a behavior recognition report is generated using preset mapping rules. The parameters of the fusion network are dynamically adjusted based on the key area screening results to achieve adaptive optimization of the monitoring model.

[0103] Furthermore, after determining the start and end times of the abnormal behavior, spatial location features and acoustic cues for that time period are extracted from the multimodal data. Spatial location features include the worker's specific coordinates and movement trajectory within the workshop in the video, while acoustic cues include operational sounds and ambient background noise in the audio. These features and cues are integrated according to preset mapping rules, such as associating spatial locations with corresponding acoustic features based on chronological order, generating a detailed behavior recognition report. The report may include the location of the abnormal behavior, accompanying sound characteristics, and the general process of the behavior. Simultaneously, referring to the key area screening results, the report analyzes which spatiotemporal features play a crucial role in identifying the anomaly during the abnormal behavior period. Based on these analyses, the parameters of the multimodal fusion network are dynamically adjusted, such as adjusting the weights of different modal data inputs and changing the parameter settings of convolutional layers or other processing layers in the network. This allows the monitoring model to better adapt to abnormal behavior recognition in complex scenarios, achieving adaptive optimization of the model and improving the accuracy and reliability of subsequent abnormal behavior monitoring.

[0104] Example 2

[0105] Please see Figure 2 This embodiment provides a video behavior recognition system based on neural networks, and a method for implementing video behavior recognition based on neural networks, including:

[0106] The video data processing module is used to acquire video stream data from surveillance cameras, split it into single-frame images in chronological order, read video files and extract raw image data frame by frame, preprocess the raw image data, extract spatial features through a convolutional neural network, input the features into a preset multi-target detection algorithm, and identify the preliminary segmentation area between the foreground worker and the background equipment.

[0107] The spatial feature optimization module uses an attention mechanism to weight spatial features based on the segmentation results, highlighting important feature regions related to the target. When the overlap ratio of targets within the segmentation region exceeds a preset threshold, the depth map estimation module is called to separate the overlapping subjects. Combining the segmentation region boundary box information and worker posture-related features, the complete spatial location features of the main operator are determined.

[0108] The audio data processing module collects ambient audio from the industrial production workshop through a microphone, converts the sound waves into electrical signals and samples them to obtain a time-series signal. It then uses short-time Fourier transform to decompose the audio signal into a spectrum and employs an adaptive filter based on the least mean square algorithm to suppress machine noise based on the spectrum characteristics and extract subtle sound cues related to behavior.

[0109] The acceleration data processing module acquires acceleration sensor data from the equipment worn by the worker, analyzes time series trends using sliding window technology, aligns with video frame sequences using dynamic time warping algorithm, corrects acceleration data offset, and extracts acceleration change features.

[0110] The multimodal fusion module integrates spatial location feature vectors, subtle sound cue feature vectors, and acceleration change feature vectors at the feature level to obtain a joint feature vector. Then, it evaluates the signal-to-noise ratio of each modality data. When the signal-to-noise ratio of a certain modality data is lower than a preset threshold, its fusion weight is dynamically adjusted according to the quantization result to generate a preliminary behavioral feature vector.

[0111] The key area screening and analysis module presets key area screening rules, performs a dimension-by-dimensional check on the behavioral feature vector, extracts the spatiotemporal regions that are highly related to the operational behavior, inputs the screening results into the long short-term memory network, and uses the network gating mechanism to analyze the context and supplement information to determine the dynamic sequence characteristics of the worker's operational behavior.

[0112] The behavior classification and detection module utilizes lightweight convolutional networks, such as the MobileNet series, and employs depthwise separable convolutional techniques to process the spatiotemporal region, extracting low-dimensional feature representation tensors. These tensors are then input into pre-trained binary classification models, such as logistic regression and SVM, and combined with real-time monitoring requirements, to determine whether the current behavior complies with safety regulations and output abnormal behavior detection results.

[0113] The abnormal behavior tracing module, when abnormal behavior is detected, extracts relevant feature sequences within a specific time period before the occurrence of abnormal behavior from the stored multimodal data, uses time series analysis models such as ARIMA to analyze the changing trend, constructs a conditional random field model to optimize the behavior boundary, and determines the start and end time points of the abnormal behavior.

[0114] The report generation and model optimization module generates behavior recognition reports by combining the start and end times of abnormal behavior with spatial location features and sound cues, based on preset mapping rules. According to the key area screening results, it dynamically adjusts the parameters of the multimodal fusion network to achieve adaptive optimization of the monitoring model and improve the recognition accuracy and reliability.

[0115] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for video behavior recognition based on neural networks, characterized in that: The method comprises the following steps: Obtain the original image data from the video frame sequence, extract the spatial features of each frame through the convolutional neural network, input the extracted features into the preset multi-target detection algorithm, and identify the preliminary segmentation area of the foreground worker and the background equipment; According to the segmentation area result, the spatial features are weighted processed by using the attention mechanism, when the overlapping proportion of the targets in the segmentation area exceeds the preset threshold, the deep map estimation module is called to separate the overlapping main body, and the spatial position features of the main operating worker are determined; Obtain the time sequence signal from the environmental audio recorded by the microphone, decompose the audio signal into a frequency spectrum through short-time Fourier transform, suppress the machine roar sound according to the spectral characteristics by using an adaptive filter, and extract the behavior-related subtle sound clues; Obtain the acceleration sensor data provided by the worker's wearing equipment, analyze the time sequence trend through a sliding window, input the analysis result into a dynamic time warping algorithm, align the video frame sequence, correct the acceleration data offset, and extract the acceleration change features; According to the spatial position features, the subtle sound clues and the acceleration change features, a multi-modal fusion network is used to integrate the three types of data at the feature level, to obtain a multi-modal joint feature vector, when the signal-to-noise ratio of a certain modal data is lower than a preset threshold, the fusion weight of the certain modal data is dynamically reduced according to the signal-to-noise ratio quantization result, and a preliminary behavior feature vector is generated; Through a preset key area screening rule, the spatio-temporal area highly related to the operation behavior is extracted from the behavior feature vector, the screening result is input into a long short-term memory network, context supplementary information is analyzed, and dynamic sequence features of the worker's operation behavior are determined; After obtaining the dynamic sequence features, the spatio-temporal area is classified through a lightweight convolutional network, a pre-trained binary classification model is called in combination with real-time monitoring requirements to judge whether the current behavior conforms to the safety specification, and the detection result of the abnormal behavior is output; According to the detection result of the abnormal behavior, when a rule violation or a fatigue mistake is detected, the previous feature change trend is traced back through a time sequence analysis method, a conditional random field model is used to optimize the behavior boundary, and the start and end time points of the abnormal behavior are determined; Through the start and end time points of the abnormal behavior, in combination with the spatial position features and the sound clues, a behavior recognition report is generated by using a preset mapping rule, the fusion network parameters are dynamically adjusted according to the key area screening result, and the adaptive optimization of the monitoring model is performed.

2. The method of claim 1, wherein: The method comprises the following steps: Obtain the original image data from the video frame sequence, extract the spatial features of each frame through the convolutional neural network, input the extracted features into the preset multi-target detection algorithm, and identify the preliminary segmentation area of the foreground worker and the background equipment, specifically comprising: Obtain video stream data from a monitoring camera, split the video stream into images one by one in time sequence, read the video file, and extract the images one by one in sequence to obtain the original image data; Preprocess the obtained original image data, adjust the image size to make it conform to the model input size, and normalize the pixel value range; Input the preprocessed image into the convolutional neural network, perform convolution operation on the image according to different convolution kernels, and extract local features; The feature images output by the convolutional layer are sampled by a pooling layer, and spatial feature representations of each frame of image are obtained through processing of multiple convolutional layers and pooling layers; The spatial features extracted by the convolutional neural network are input into a preset multi-target detection algorithm, the input image is divided into grids, each grid predicts nearby targets, and the target categories and bounding boxes are output through convolution operation and a prediction head; according to the output results of the algorithm, the preliminary segmentation regions of the foreground workers and the background equipment in each frame of image are obtained.

3. The method of claim 1, wherein: According to the segmentation region results, the spatial features are weighted using an attention mechanism, and when the overlap ratio of targets in the segmentation region exceeds a preset threshold, a depth map estimation module is called to separate the overlapping subjects, and the spatial position features of the main operator are determined, specifically including: The spatial features of each frame of image extracted by the convolutional neural network are integrated into a feature vector set, the similarity between different position feature vectors is calculated through a self-attention mechanism, and an attention weight matrix is generated; The attention weights are normalized through a Softmax function to represent the correlation degree between different position features, and the weighted segmentation region results are obtained by weighted summation of the numerical value vectors; According to the weighted segmentation region results, the overlap ratio of targets in the region is calculated, the overlap degree is measured by comparing the intersection and union ratio of different target bounding boxes, and a preset threshold is set; when the ratio of multiple targets in a segmentation region exceeds the threshold, it indicates that there is target overlap; A pre-trained depth estimation model is used, the image region containing overlapping targets is input into the model, the feature of the image is learned and inferred, and the depth map of the region is output; according to the depth map, the spatial position of the main operator is determined; According to the results of separating overlapping subjects by the depth map estimation module, combined with the bounding box information of the segmentation region, the spatial position features of the main operator are extracted, the pixel coordinate range of the region where the main operator is located is determined from the depth map, and then it is mapped back to the coordinate system of the original image; meanwhile, the features related to the body posture of the worker are extracted, and the position coordinate information, depth information and posture related information are integrated to form a complete spatial position feature description of the main operator.

4. The method of claim 1, wherein: The time series trend is analyzed through a sliding window, and the analysis results are input into a dynamic time warping algorithm to align with the video frame sequence, specifically including: The sliding window technology is used to analyze the trend of acceleration data over time, a fixed length window is defined, and the window is slid on the time series data; in each window, the acceleration data is statistically analyzed; The entire time series data is analyzed segment by segment through the continuous movement of the sliding window, and a series of window statistics curves are obtained, which are used to observe the average acceleration trend of the worker along the X-axis direction at different times; The acceleration data time series obtained by sliding window analysis is taken as the sequence to be aligned, and the timestamp of the video frame sequence is taken as the reference sequence; by calculating the similarity matrix between the two sequences, an optimal time warping path is found to match the two sequences in time. Specific is to calculate the distance between each data point in two sequences, construct a distance matrix, and then use a dynamic programming algorithm to search for a path with the minimum accumulated distance on the distance matrix. This path represents the optimal alignment relationship between the two sequences. The time axis of the acceleration data is stretched or compressed to align it with the video frame sequence in time.

5. The method of claim 1, wherein: The spatiotemporal region highly related to the operation behavior is extracted from the behavior feature vector through a preset key region screening rule, specifically including: inputting the generated preliminary behavior feature vector into the preset key region screening rule, using the vector-by-dimension checking algorithm to check and judge the behavior feature vector by dimension, for each behavior feature vector, in the spatial position feature part, the coordinate matching algorithm is used to check whether the position elements in the vector are within the specified coordinate interval; in the subtle sound clue feature part, the frequency component ratio algorithm is used to confirm whether the corresponding frequency component elements meet the set range; in the acceleration change feature part, the acceleration value judgment algorithm is used to judge whether the acceleration value related elements are in the specified change interval, when the behavior feature vector meets the screening rule in each modal corresponding feature dimension, the spatiotemporal region extraction algorithm is used to extract the corresponding spatiotemporal region.

6. The method of claim 1, wherein: The trend of the previous sequence characteristic is traced back through the time series analysis method, specifically including: After detecting the abnormal behavior, the trend of the previous sequence characteristic is traced back through the time series analysis method, and the worker spatial position feature vector sequence, audio frequency spectrum feature sequence and acceleration change value sequence corresponding to the video frame sequence within a specific time length before the time when the abnormal behavior is detected are extracted from the stored multi-modal data; The time series of each modal data is tested for stationarity, the order is determined and the parameters are estimated through the time series analysis model, the previous time series is fitted and predicted, and the change trend of the spatial position, audio frequency component and acceleration before the abnormal behavior occurs is analyzed.

7. The method of claim 1, wherein: The conditional random field model is used to optimize the behavior boundary to determine the start and end time points of the abnormal behavior, specifically including: based on the analysis result, the input features of the conditional random field model are constructed, the multi-modal features at different time points are combined into a multi-dimensional feature vector sequence, the feature function that can capture the feature relationship between adjacent time points and the association between the feature and the label is defined, the conditional random field model is trained using the labeled training data, the parameters are learned through the iterative optimization algorithm, and then the data processed by the time series analysis and the conditional random field model are input into the abnormal behavior start and end time point determination module. According to the label sequence output by the conditional random field model, combined with the timestamp information of the time series data, the start time point is determined when the conditional random field model first predicts the label as an abnormal behavior, and the end time point is determined when the label is predicted as a normal behavior for continuous multiple time steps, the start and end time points of the abnormal behavior on the time axis are determined.

8. A neural network based video behavior recognition system, applied to the neural network based video behavior recognition method according to any one of claims 1-7, characterized in that: Including: The video data processing module obtains video stream data from a monitoring camera, splits it into single-frame images in chronological order, reads video files and extracts original image data frame by frame, extracts spatial features through a convolutional neural network after preprocessing, and inputs them into a multi-target detection algorithm to identify the preliminary segmentation region of the foreground worker and the background equipment; The spatial feature optimization module uses attention mechanism to weight the spatial features according to the segmentation region result, highlighting the important feature area; when the target overlaps, the depth map estimation module is called to separate the overlapping main body, and the worker posture feature is combined to determine the complete spatial position of the main worker; The audio data processing module collects environmental audio through a microphone, converts sound waves into electrical signals and samples them, decomposes them into a frequency spectrum using short-time Fourier transform, suppresses machine noise using an adaptive filter, and extracts behavior-related sound cues; The acceleration data processing module obtains acceleration sensor data from the worker's wearable device, analyzes time series trends using sliding window technology, and aligns video frame sequences using dynamic time warping algorithms to correct data drift and extract acceleration change features; The multi-modal fusion module integrates spatial position features, subtle sound cue features, and acceleration change features at the feature level to obtain a joint feature vector, dynamically adjusts the fusion weight based on the signal-to-noise ratio, and generates a preliminary behavior feature vector; The key area screening and analysis module predefines key area screening rules, checks the behavior feature vector dimension by dimension, extracts time and space areas highly related to operation behavior, and inputs context information into a long short-term memory network to determine dynamic sequence features; The behavior classification and detection module uses depth separable convolution technology to extract low-dimensional feature representation tensors, inputs them into a pre-trained binary classification model, and determines whether the behavior meets safety specifications based on real-time monitoring needs to output abnormal behavior detection results; The abnormal behavior tracing module extracts feature sequences before the occurrence of abnormal behavior from stored multi-modal data when abnormal behavior is detected, uses time series analysis models and conditional random field models to optimize behavior boundaries, and determines start and end time points; The report generation and model optimization module generates a behavior recognition report based on the start and end time points of the abnormal behavior, combined with spatial position features and sound cues, and dynamically adjusts fusion network parameters based on key area screening results to adaptively optimize the monitoring model.

Images