Video behavior recognition method and system based on neural network

Abstract

The invention discloses a video behavior recognition method and system based on a neural network, and relates to the technical field of computer vision. According to the method, video, audio and acceleration data are efficiently integrated through the multi-modal fusion network, and the fusion weight is dynamically adjusted, so that the problems of heterogeneity and interference of the multi-modal data are solved, and the robustness and accuracy of a behavior recognition system are improved; meanwhile, a lightweight convolutional network and a dynamic time warping algorithm are adopted to optimize calculation efficiency and real-time performance, and industrial real-time monitoring requirements are met; besides, starting and ending time points of abnormal behaviors are accurately positioned by using time sequence analysis and a conditional random field model, and model parameters are dynamically adjusted, so that self-adaptive optimization is realized, and the operation safety of workers and the production efficiency are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically to a method and system for video behavior recognition based on a neural network. Background Art

[0002] In the business scenario where video behavior recognition is applied to real-time monitoring of workers' operation behaviors in a large industrial production workshop, many severe challenges are faced. It is necessary to integrate video images captured by cameras, environmental audio recorded by microphones, and acceleration sensor data provided by wearable devices worn by workers. However, multi-modal inputs are highly heterogeneous and severely interfered; in video images, the action postures of workers are often blurred due to perspective occlusion or light changes; in the audio stream, the roar of machines is mixed with the voices of workers, masking the subtle sound clues related to behaviors; sensor data has offsets due to different wearing positions of the devices. When a worker operates multiple devices simultaneously, there are multiple action subjects in the video frame sequence, making it difficult to distinguish the relevance between the foreground worker and the background devices in spatial feature extraction; the time series of audio signals is affected by noise superposition, and the subtle sound clues related to behaviors are masked; at the same time, the sampling frequency of sensor data is inconsistent with the video frame rate, and time alignment is likely to introduce pseudo-correlation interference; existing models usually tend to focus on the dominant features of a single modality, such as only focusing on the significant actions in the image while ignoring the context supplementary information provided by audio or sensors; moreover, behavior recognition needs to be completed within milliseconds to meet the real-time requirement, but in complex scenarios, the redundant features generated by multi-modal fusion lead to an unbalanced allocation of computing resources; how to design a mechanism to adaptively screen key spatio-temporal regions and efficiently integrate the three types of data with limited computing power has become a technical problem to be solved urgently. Summary of the Invention

[0003] The object of the present invention can be achieved by the following technical solutions:

[0004] The present application provides a method for video behavior recognition based on a neural network, including the following steps:

[0005] S1. Obtain original image data from the video frame sequence, extract the spatial features of each frame through a convolutional neural network, and input the extracted features into a preset multi-object detection algorithm to identify the preliminary segmentation regions of the foreground worker and the background devices;

[0006] S2. According to the segmentation region results, use an attention mechanism to weight the spatial features. When the target overlap ratio within the segmentation region exceeds a preset threshold, call the depth map estimation module to separate the overlapping subjects and determine the spatial position features of the main operating worker;

[0007] S3. Obtain the time series signal from the ambient audio recorded by the microphone, decompose the audio signal into a spectrum through short-time Fourier transform, use an adaptive filter to suppress the machine rumble according to the spectrum characteristics, and extract the subtle sound clues related to the behavior;

[0008] S4. Obtain the acceleration sensor data provided by the worker-worn device, analyze the time series trend through a sliding window, input the analysis result into the dynamic time warping algorithm, align it with the video frame sequence, correct the acceleration data offset, and extract the acceleration change characteristics;

[0009] S5. According to the spatial position characteristics, subtle sound clues, and acceleration change characteristics, use a multi-modal fusion network to perform feature-level integration on the three types of data to obtain a multi-modal joint feature vector. When the signal-to-noise ratio of a certain modal data is lower than the preset threshold, dynamically reduce its fusion weight according to the signal-to-noise ratio quantization result to generate a preliminary behavior feature vector;

[0010] S6. Through the preset key area screening rules, extract the spatio-temporal areas highly related to the operation behavior from the behavior feature vector, input the screening result into the long short-term memory network, analyze the context supplementary information, and determine the dynamic sequence characteristics of the worker's operation behavior;

[0011] S7. After obtaining the dynamic sequence characteristics, classify the spatio-temporal areas through a lightweight convolutional network, and in combination with the real-time monitoring requirements, call the pre-trained binary classification model to determine whether the current behavior complies with the safety specifications and output the detection result of abnormal behavior.

[0012] Furthermore, obtain the original image data from the video frame sequence, extract the spatial features of each frame through a convolutional neural network, and input the extracted features into the preset multi-object detection algorithm to identify the preliminary segmentation areas of the foreground workers and the background equipment, specifically including:

[0013] Obtain the video stream data from the monitoring camera, split the video stream into frames one by one in chronological order, then read the video file and extract frame by frame in sequence to obtain the original image data;

[0014] Preprocess the obtained original image data, adjust the image size to conform to the model input size, and at the same time normalize the pixel value range;

[0015] Input the preprocessed image into the convolutional neural network, perform convolutional operations by sliding different convolutional kernels on the image to extract local features;

[0016] Sample the feature image output by the convolutional layer through the pooling layer, and after being processed by multiple convolutional layers and pooling layers, obtain the spatial feature representation of each frame of the image;

[0017] The spatial features extracted by the convolutional neural network are input into a preset multi-object detection algorithm. The input image is divided into grids, and each grid predicts nearby objects. Through convolutional operations and a prediction head, the object category and bounding box are output. According to the output results of the algorithm, the preliminary segmentation regions of foreground workers and background devices in each frame of the image are obtained.

[0018] Further, according to the segmentation region results, an attention mechanism is used to weight the spatial features. When the overlap ratio of objects within the segmentation region exceeds a preset threshold, the depth map estimation module is called to separate the overlapping objects and determine the spatial position features of the main operating worker, specifically including:

[0019] Integrate the spatial features of each frame of the image extracted by the convolutional neural network into a feature vector set, calculate the similarity between feature vectors at different positions through the self-attention mechanism, and generate an attention weight matrix;

[0020] Normalize through the Softmax function to obtain the attention weights, which are used to represent the degree of association between features at different positions, perform weighted summation on the numerical vectors, and obtain the weighted segmentation region results;

[0021] According to the weighted segmentation region results, calculate the overlap ratio of objects within the region, measure the overlap degree by comparing the ratio of the intersection to the union of the bounding boxes of different objects, and then set a preset threshold. When the ratio of multiple objects within a certain segmentation region exceeds this threshold, it indicates that there is an object overlap situation;

[0022] Use a pre-trained depth estimation model, input the image region containing overlapping objects into the model, and through feature learning and inference of the image, output the depth map of this region, and determine the spatial position where the main operating worker is located according to the depth map;

[0023] According to the result of separating the overlapping objects by the depth map estimation module, combined with the bounding box information of the segmentation region, extract the spatial position features of the main operating worker, determine the pixel coordinate range of the region where the main operating worker is located from the depth map, and then map it back to the coordinate system of the original image. At the same time, extract the features related to the worker's body posture, and integrate the position coordinate information, depth information, and posture-related information to form a complete description of the spatial position features of the main operating worker.

[0024] Further, analyze the time series trend through a sliding window, and input the analysis results into the dynamic time warping algorithm to align with the video frame sequence, specifically including:

[0025] Adopt the sliding window technique to analyze the change trend of acceleration data over time, define a window with a fixed length, slide it sequentially on the time series data, and perform statistical analysis on the acceleration data within each window;

[0026] By continuously moving the sliding window, the entire time series data is analyzed segment by segment to obtain a curve of a series of window statistics changing with time, which is used to observe the average acceleration trend of the worker's movement along the X-axis direction at different times;

[0027] Taking the time series of acceleration data obtained from the sliding window analysis as the sequence to be aligned, and taking the timestamps of the video frame sequence as the reference sequence, by calculating the similarity matrix between the two sequences, an optimal time warping path is found to make the two sequences match in time;

[0028] Specifically, calculate the distance between each data point in the two sequences to construct a distance matrix, and then use the dynamic programming algorithm to search for a path with the minimum cumulative distance on the distance matrix. This path represents the best alignment relationship between the two sequences, and stretch or compress the time axis of the acceleration data to align it with the video frame sequence in time.

[0029] Furthermore, through the preset key region screening rules, the spatio-temporal regions highly relevant to the operation behavior are extracted from the behavior feature vectors, specifically including: inputting the generated preliminary behavior feature vectors into the preset key region screening rules, using the vector-by-dimension inspection algorithm to inspect and judge the behavior feature vectors dimension by dimension. For each behavior feature vector, in the spatial position feature part, use the coordinate matching algorithm to check whether the elements representing the position in the vector are within the specified coordinate range; in the subtle sound clue feature part, confirm whether the corresponding frequency component elements meet the set range through the frequency component comparison algorithm; in the acceleration change feature part, use the acceleration value judgment algorithm to judge whether the acceleration value-related elements are in the specified change interval. When the behavior feature vectors meet the screening rules in the feature dimensions corresponding to each modality, use the spatio-temporal region extraction algorithm to extract the corresponding spatio-temporal regions.

[0030] S8. According to the detection results of abnormal behaviors, when a violation operation or fatigue mistake is detected, the previous feature change trend is traced through the time series analysis method, and the conditional random field model is used to optimize the behavior boundary to determine the start and end time points of the abnormal behavior;

[0031] Furthermore, tracing the previous feature change trend through the time series analysis method specifically includes:

[0032] After an abnormal behavior is detected, use the time series analysis method to trace the previous feature change trend, and extract the sequence of the worker's spatial position feature vectors, the audio spectrum feature sequence, and the acceleration change value sequence corresponding to the video frame sequence within a specific duration before the moment when the abnormal behavior is detected from the stored multi-modal data;

[0033] Through the time series analysis model, perform stationarity tests, order determination, and parameter estimation on the time series of each modality data, fit and predict the previous time series, and analyze the change trends of spatial positions, audio frequency components, and accelerations before the occurrence of abnormal behaviors.

[0034] Furthermore, adopt a conditional random field model to optimize the behavior boundary and determine the start and end time points of abnormal behaviors, specifically including: based on the analysis results, construct the input features of the conditional random field model, combine the multi-modal features at different time points into a multi-dimensional feature vector sequence, define a feature function that can capture the relationship between features of adjacent time points and the association between features and labels, use the labeled training data to train the conditional random field model, learn the parameters through an iterative optimization algorithm, and then input the data processed by the time series analysis and the conditional random field model into the abnormal behavior start and end time point determination module. According to the label sequence output by the conditional random field model and combined with the timestamp information of the time series data, determine the start time point when the label predicted by the conditional random field model is an abnormal behavior for the first time, and determine the end time point when the labels are predicted as normal behaviors for multiple consecutive time steps, so as to determine the start and end time points of abnormal behaviors on the time axis.

[0035] S9. Through the start and end time points of abnormal behaviors, combined with spatial position features and sound clues, generate a behavior recognition report using a preset mapping rule, dynamically adjust the fusion network parameters according to the key area screening results, and perform adaptive optimization of the monitoring model.

[0036] The present invention provides a video behavior recognition system based on a neural network for implementing a method for video behavior recognition based on a neural network, including:

[0037] A video data processing module, which obtains video stream data from a monitoring camera, splits it into single-frame images in chronological order, reads the video file and extracts the original image data frame by frame, performs preprocessing and then extracts spatial features through a convolutional neural network, and inputs them into a multi-object detection algorithm to identify the preliminary segmentation regions of foreground workers and background devices;

[0038] A spatial feature optimization module, according to the segmentation region results, uses an attention mechanism to weight the spatial features to highlight important feature regions; when the targets overlap, call the depth map estimation module to separate the overlapping main bodies, and combine the worker pose features to determine the complete spatial positions of the main operating workers;

[0039] An audio data processing module, which collects environmental audio through a microphone, converts the sound wave into an electrical signal and samples it, decomposes it into a frequency spectrum using a short-time Fourier transform, suppresses machine noise using an adaptive filter, and extracts sound clues related to behaviors;

[0040] The acceleration data processing module acquires the acceleration sensor data of the worker's wearable device, analyzes the time series trend through the sliding window technique, aligns with the video frame sequence using the dynamic time warping algorithm, corrects the data offset, and extracts the acceleration change features;

[0041] The multi-modal fusion module performs feature-level integration on the spatial position features, subtle sound clue features, and acceleration change features to obtain a joint feature vector, and dynamically adjusts the fusion weights according to the signal-to-noise ratio to generate a preliminary behavior feature vector;

[0042] The key area screening and analysis module presets the key area screening rules, checks the behavior feature vector dimension by dimension, extracts the spatio-temporal area highly relevant to the operation behavior, and inputs it into the long short-term memory network to analyze the context information and determine the dynamic sequence features;

[0043] The behavior classification and detection module uses the depthwise separable convolution technique to extract a low-dimensional feature representation tensor, inputs it into a pre-trained binary classification model, combines with the real-time monitoring requirements to judge whether the behavior complies with the safety specifications, and outputs the abnormal behavior detection result;

[0044] The abnormal behavior tracing module, when detecting an abnormal behavior, extracts the feature sequence before the occurrence of the abnormal behavior from the stored multi-modal data, optimizes the behavior boundary using the time series analysis model and the conditional random field model, and determines the start and end time points;

[0045] The report generation and model optimization module generates a behavior recognition report according to the start and end time points of the abnormal behavior, combines with the spatial position features and sound clues, and dynamically adjusts the fusion network parameters according to the key area screening results to perform adaptive optimization of the monitoring model.

[0046] The beneficial effects of the present invention are:

[0047] Through the multi-modal fusion network, feature-level integration of video images, audio signals, and acceleration sensor data is carried out, and the fusion weights are dynamically adjusted according to the signal-to-noise ratio, effectively solving the problems of high heterogeneity of multi-modal data and interference such as blurred video images, masked audio signals, and sensor data offset, significantly improving the robustness and accuracy of the behavior recognition system, enabling it to operate stably in the complex industrial production workshop environment;

[0048] Using lightweight convolutional networks (such as MobileNet) and dynamic time warping algorithms (DTW), the model parameters and computational complexity are reduced, while the time alignment accuracy of multi-modal data is optimized, solving the problem that it is often difficult to meet the real-time requirements due to low computational efficiency in complex scenarios, achieving efficient processing of multi-modal data with limited computing power, enabling the behavior recognition task to be completed in milliseconds, meeting the strict requirements of industrial production real-time monitoring, and at the same time reducing the dependence on hardware resources;

[0049] By using time series analysis and conditional random field models, tracing the changing trends of previous features, optimizing the behavior boundaries, and dynamically adjusting the parameters of the fusion network, the accurate positioning and dynamic optimization of abnormal behaviors are achieved, enhancing the adaptability and accuracy of the system, ensuring the safety of workers' operations, improving production efficiency. The present invention overcomes the limitations of traditional methods in abnormal behavior detection, such as frequent misjudgments and missed judgments, and difficulty in adapting to complex and changing industrial environments, by accurately positioning the start and end time points of abnormal behaviors, providing a more reliable solution for the safety monitoring of industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] For better understanding and implementation, the technical solutions of the present application will be described in detail below with reference to the drawings.

[0051] Figure 1 It is a schematic flowchart of a method for video behavior recognition based on a neural network provided in Embodiment 1 of the present application;

[0052] Figure 2 It is a schematic structural diagram of a video behavior recognition system based on a neural network provided in Embodiment 2 of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] To further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, exemplary embodiments will be described in detail herein, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of methods and systems consistent with some aspects of the present application as detailed in the appended claims.

[0054] The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "the", and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0055] The following will describe in detail the specific embodiments, features, and their effects of the present invention in conjunction with the drawings and preferred embodiments.

[0056] Embodiment 1

[0057] Please refer to Figure 1 , this embodiment provides a method for video behavior recognition based on a neural network, including the following steps:

[0058] S1. Obtain the original image data from the video frame sequence, extract the spatial features of each frame through a convolutional neural network, and input the extracted features into a preset multi-object detection algorithm to identify the preliminary segmentation regions of foreground workers and background equipment;

[0059] Further, obtain the original image data from the video frame sequence, extract the spatial features of each frame through a convolutional neural network, and input the extracted features into a preset multi-object detection algorithm to identify the preliminary segmentation regions of foreground workers and background equipment, specifically including:

[0060] Obtain the video stream data from the surveillance camera, split the video stream into frames one by one according to the time sequence, then read the video file and extract frame by frame in order to obtain the original image data;

[0061] Preprocess the obtained original image data, adjust the image size to conform to the model input size, and normalize the pixel value range at the same time;

[0062] Input the preprocessed image into a convolutional neural network, perform convolution operations by sliding different convolutional kernels on the image to extract local features;

[0063] Sample the feature map output by the convolutional layer through the pooling layer, and obtain the spatial feature representation of each frame image after being processed by multiple convolutional layers and pooling layers;

[0064] Input the spatial features extracted by the convolutional neural network into a preset multi-object detection algorithm. Taking YOLO as an example, divide the input image into grids, each grid predicts nearby targets, and after convolution operations and prediction heads, output the target category and bounding box; according to the algorithm output results, obtain the preliminary segmentation regions of foreground workers and background equipment in each frame image.

[0065] S2. According to the segmentation region results, use the attention mechanism to weight the spatial features. When the target overlap ratio within the segmentation region exceeds the preset threshold, call the depth map estimation module to separate the overlapping objects and determine the spatial position features of the main operating worker;

[0066] Further, according to the segmentation region results, use the attention mechanism to weight the spatial features. When the target overlap ratio within the segmentation region exceeds the preset threshold, call the depth map estimation module to separate the overlapping objects and determine the spatial position features of the main operating worker, specifically including:

[0067] Integrate the spatial features of each frame of the image extracted by the convolutional neural network into a set of feature vectors, calculate the similarity between feature vectors at different positions through the self-attention mechanism, and generate an attention weight matrix; specifically, linearly transform the spatial feature vectors with the query (Query), key (Key), and value (Value) matrices respectively to obtain the corresponding Q, K, and V vectors, then calculate the dot product of Q and the transpose of K, divide by the scale factor, and normalize through the Softmax function to obtain the attention weights. These weights represent the degree of association between features at different positions, and perform a weighted sum on the value vectors to obtain the weighted segmentation region result. In this way, the attention mechanism can highlight the important feature regions related to the target, suppress irrelevant or secondary information, thus perform more effective weighted processing on the spatial features, and provide a more targeted feature representation for subsequent analysis.

[0068] According to the weighted segmentation region result, calculate the overlap ratio of the targets within the region, and measure the overlap degree by comparing the ratio of the intersection to the union of different target bounding boxes. By setting a preset threshold (for example, 0.5, which can be adjusted according to the actual scenario and requirements), when the ratio of multiple targets in a certain segmentation region exceeds this threshold, it indicates that there is a target overlap situation; at this time, call the depth map estimation module. The depth map estimation aims to obtain the depth information of the objects in the scene and determine the front-back relationship between the objects. Taking monocular depth estimation based on deep learning as an example, use a pre-trained depth estimation model (such as the Monodepth series models based on convolutional neural networks), input the image region containing the overlapping targets into the model, and through feature learning and inference of the image, output the depth map of this region. Each pixel value in the map represents the depth information of the object at the corresponding position. According to the depth map, the objects in the front and back among the overlapping objects can be clearly distinguished, and then the spatial position where the main operator is located can be determined.

[0069] According to the result of separating the overlapping objects by the depth map estimation module, combined with the bounding box information of the segmentation region, extract the spatial position features of the main operator. Determine the pixel coordinate range of the region where the main operator is located from the depth map, and then map it back to the coordinate system of the original image. At the same time, considering that the posture and actions of the worker affect the operation, further extract the features related to the body posture of the worker, such as the joint point position information (which can be obtained through human pose estimation technology). Integrate these position coordinate information, depth information, and posture-related information to form a complete spatial position feature description of the main operator. These features not only clarify the two-dimensional position of the worker in the image, but also include the information in the depth direction and the posture state, providing comprehensive and accurate basic data for subsequent analysis of the worker's operation behavior, and helping to more accurately identify the worker's operation actions and judge whether they comply with safety regulations, etc.

[0070] S3. Obtain a time series signal from the ambient audio recorded by the microphone, decompose the audio signal into a spectrum through short-time Fourier transform, and use an adaptive filter to suppress the machine roar according to the spectrum characteristics, and extract subtle sound clues related to behavior;

[0071] Further, obtain a time series signal from the ambient audio recorded by the microphone, decompose the audio signal into a spectrum through short-time Fourier transform, use an adaptive filter to suppress the machine roar, and extract subtle sound clues related to behavior, specifically including:

[0072] Collect the ambient audio in the industrial production workshop and convert the sound wave into an electrical signal. The audio acquisition device samples the electrical signal at a certain sampling frequency (such as common 44.1 kHz or 48 kHz), converts the continuous analog audio signal into a discrete digital signal sequence, and this digital signal sequence is the time series signal. In practical applications, use a professional audio acquisition library such as PyAudio (in a Python environment) to implement audio recording and the acquisition of time series signals. Set the sampling parameters through relevant functions, start the recording process, and store the collected audio data in a suitable format (such as WAV format) for subsequent processing.

[0073] After obtaining the time series signal, in order to analyze the characteristics of the audio signal at different frequencies and times, short-time Fourier transform (Short-Time Fourier Transform, STFT) is adopted. The core idea of STFT is to divide the audio signal into many short time periods, assuming that the signal is approximately stationary within each time period. Within each short time period, a conventional Fourier transform is performed on the signal. Specifically, when operating, select a suitable window function (such as Hanning window, Hamming window, etc.). The role of the window function is to window the audio signal, making the edge transition of the signal within the window smoother and reducing spectral leakage. Slide the window function sequentially on the audio time series, perform Fourier transform on the signal within each window, and convert the time-domain signal into a frequency-domain signal. After STFT, the result obtained is a two-dimensional matrix, where the abscissa represents time, the ordinate represents frequency, and the value in the matrix represents the signal amplitude at the corresponding time and frequency, that is, the spectrum of the audio signal. In this way, the audio signal is decomposed into a spectrogram of different frequency components changing with time, which helps to observe the frequency composition of the audio at different moments.

[0074] The roar of machines in an industrial production workshop is the main noise, with relatively stable frequency and intensity and certain regularity. An adaptive filter can automatically adjust the filter parameters according to the characteristics of the input signal to achieve the best filtering effect. In this scenario, an adaptive filter based on the Least Mean Square (LMS) algorithm can be adopted. First, a reference signal containing only the roar of machines needs to be obtained (which can be collected when the workshop equipment is running normally but there is no worker operation). The mixed audio signal containing the roar of machines and behavior-related sounds is used as the input of the adaptive filter, and the reference signal is used as the desired response of the filter. The LMS algorithm continuously adjusts the filter coefficients through spectral characteristics to minimize the mean square error between the filter output and the desired response. In this process, the adaptive filter gradually learns the characteristics of the machine roar and generates a signal with the same amplitude but opposite phase to the machine roar, which is cancelled from the mixed audio signal to suppress the machine roar. After filtering, the remaining audio signal highlights the subtle sound clues related to behavior, such as the key-pressing sound and tool-collision sound when workers operate the equipment. These subtle sound clues are of great significance for identifying the operation behavior of workers. Subsequently, these clues can be further analyzed and processed, such as extracting features such as the frequency, amplitude, and duration of the sound through feature extraction algorithms for subsequent behavior recognition model training and judgment.

[0075] Specifically, obtaining the time series signal is to collect the original audio data, and the spectrum decomposed by the short-time Fourier transform can clearly show the characteristics of the audio at different frequencies and times, so as to understand the frequency-domain distribution of the machine roar and behavior-related sounds, providing a clear input signal for the adaptive filter. The adaptive filter uses the least mean square algorithm to adjust the filter coefficients according to the spectral characteristics, specifically suppressing the machine roar with stable frequency and intensity patterns, thereby extracting those subtle behavior-related sound clues that are masked.

[0076] S4. Obtain the acceleration sensor data provided by the worker's wearable device, analyze the time series trend through a sliding window, input the analysis result into the dynamic time warping algorithm, align it with the video frame sequence, correct the acceleration data offset, and extract the acceleration change characteristics;

[0077] Furthermore, obtaining the acceleration sensor data includes:

[0078] The wearable device worn by workers integrates an acceleration sensor for real-time monitoring of the workers' motion states. The sensor converts physical quantities into electrical signals by sensing the acceleration changes of the device in three axial directions and outputs them. The microcontroller inside the device samples these electrical signals at a specific sampling frequency and digitally processes the sampled data. In practical applications, these digitized acceleration data can be transmitted to a data processing terminal through wireless communication technologies such as Bluetooth and Wi-Fi. For example, when using Bluetooth Low Energy (BLE) technology, a connection between the device and the terminal can be established with the help of relevant Bluetooth communication libraries (such as the bluepy library in a Python environment), and the acceleration sensor data can be received and stored. These data are recorded in the form of timestamps and triaxial acceleration values, forming a time series dataset.

[0079] Analyze the time series trend through a sliding window and input the analysis results into the dynamic time warping algorithm to align with the video frame sequence, specifically including:

[0080] Adopt the sliding window technique to analyze the change trend of acceleration data over time. Define a window with a fixed length (for example, the window size is 100 data points, which can be adjusted according to the actual situation), and this window slides successively on the time series data. Within each window, statistical analysis is performed on the acceleration data, such as calculating statistical quantities such as the mean, variance, maximum value, and minimum value of the data within the window. These statistical quantities can reflect the central tendency, dispersion degree, and change range of the acceleration data within the window. By continuously moving the sliding window, the entire time series data is analyzed segment by segment, obtaining a curve of a series of window statistical quantities changing over time. For example, by calculating the mean of the X-axis acceleration within each window, the average acceleration trend of the worker moving along the X-axis direction at different times can be observed, thereby initially judging whether the worker's motion state is accelerating, decelerating, or relatively stable. This sliding window analysis method can effectively capture the local features and trend changes in the time series data and provide more valuable information for subsequent data alignment and feature extraction.

[0081] Since the sampling frequency of the acceleration sensor data is inconsistent with the video frame rate, time alignment is required to ensure that the acceleration data accurately matches the corresponding video frames. The Dynamic Time Warping (DTW) algorithm is a method for solving the time series alignment problem. First, the time series of the acceleration data obtained by sliding window analysis is used as the sequence to be aligned, and the timestamps of the video frame sequence are used as the reference sequence. The DTW algorithm calculates the similarity matrix between the two sequences and finds an optimal time warping path to make the two sequences match in time. In the specific calculation process, the distance (such as Euclidean distance) between each data point in the two sequences is calculated first to construct a distance matrix. Then, the dynamic programming algorithm is used to search for a path with the minimum cumulative distance on the distance matrix, and this path represents the best alignment relationship between the two sequences. Through the DTW algorithm, the time axis of the acceleration data can be stretched or compressed to align it with the video frame sequence in time. For example, when the video frame rate is 30fps and the acceleration sensor sampling frequency is 100Hz, the DTW algorithm can determine how to select or interpolate the corresponding acceleration data points at each frame moment of the video to achieve accurate synchronization between the two.

[0082] Among them, during the acquisition of acceleration sensor data, due to the different wearing positions of the device and the characteristics of the sensor itself, data offset may occur. To correct this offset, a calibration data-based method is adopted. Before the worker wears the device for actual work, a calibration process is carried out. During the calibration period, the worker remains stationary, and the acceleration data at this time is collected. Since the worker is stationary, the acceleration value should theoretically be zero. By analyzing the calibration data, the average offset amount in each axial direction is calculated. In the subsequent actually collected acceleration data, the average offset amount in the corresponding axial direction is subtracted to achieve the correction of the data offset. For example, if the average offset amount of the X-axis is found to be 0.2m / s 2 during the calibration process, then in the actually collected X-axis acceleration data, 0.2m / s 2 is subtracted from each data point to obtain data that more accurately reflects the true motion acceleration of the worker. After data offset correction, the acceleration data can more accurately reflect the actual motion state of the worker, laying a foundation for subsequent extraction of accurate acceleration change characteristics.

[0083] The acceleration data after time alignment and data offset correction can be used to extract acceleration change features. The acceleration change features include the first derivative of acceleration (i.e., the rate of change of acceleration, reflecting the speed of acceleration change), the second derivative (reflecting the change of the acceleration rate of change, which can be used to judge sharp changes such as sudden stops and starts of motion), and features such as the peak and valley values of acceleration within different time periods. For example, by calculating the first derivative of the acceleration data, the acceleration change rate at each time point can be approximately obtained through the difference method. If the acceleration change rate suddenly increases at a certain moment, it is very likely that the worker has made a rapid movement, such as quickly turning around or operating the equipment vigorously. After extracting these acceleration change features, they are combined into feature vectors, which can comprehensively describe the dynamic change of acceleration during the worker's movement process and are of great significance for identifying the worker's operation behavior; subsequently, these feature vectors can be input into the behavior recognition model and used together with the features extracted from video images and audio data to comprehensively judge whether the worker's operation complies with safety regulations and whether there are abnormal behaviors, etc.

[0084] S5. According to the spatial position features, subtle sound clues, and acceleration change features, use a multi-modal fusion network to perform feature-level integration on the three types of data to obtain a multi-modal joint feature vector. When the signal-to-noise ratio of a certain modal data is lower than the preset threshold, the fusion weight is dynamically reduced according to the signal-to-noise ratio quantization result to generate a preliminary behavior feature vector;

[0085] Furthermore, the spatial position feature vector, subtle sound clue feature vector, and acceleration change feature vector are connected end to end in sequence to form a longer joint feature vector. This joint feature vector contains rich information from different modalities, but the weights between the features of each modality are the same. The weighted summation method assigns a weight to the feature vector of each modality and then adds the weighted feature vectors. The initial weights can be set according to experience or simple heuristic methods. For example, the weight of each modality is first set to 1 / 3. Through this feature-level integration, data from different modalities complement each other at the feature level, providing richer information for comprehensively describing the worker's behavior.

[0086] To address the differences in the quality of data from different modalities, it is necessary to evaluate the signal-to-noise ratio (SNR) of each modality of data. The signal-to-noise ratio is an indicator that measures the ratio of the effective signal to the noise in a signal. For the spatial position features of video images, noise may come from factors such as light changes and perspective occlusion; the subtle sound cues in audio are interfered by noises such as machine roars; the noise in acceleration data stems from errors in the sensor itself or external electromagnetic interference, etc. The SNR of each modality of data is calculated through specific algorithms. For example, for audio data, the SNR can be calculated by comparing the power spectral density of the signal with the power spectral density of the noise. A preset threshold is set (e.g., SNR = 10 dB, which can be adjusted according to the actual scenario and data characteristics). When the SNR of a certain modality of data is lower than this threshold, it indicates that this modality of data is greatly affected by noise and the data quality is low. At this time, the weight of this modality of data in the fusion process is dynamically adjusted according to the quantization result of the SNR. Specifically, a linear or non-linear mapping relationship can be adopted to map the SNR value to a weight adjustment coefficient. For example, when the SNR of audio data is 5 dB, through a predefined mapping function, the calculated weight adjustment coefficient is 0.2 (assuming the original weight is 1). Then, in the weighted summation fusion method, the weight of the audio modality feature vector will be reduced from 1 / 3 to 0.2 / 3. Correspondingly, the weights of other modality data will be redistributed according to the principle that the total weight is 1, so as to highlight the contribution of high-quality modality data and reduce the impact of low-quality modality data on the final result.

[0087] After feature-level integration and dynamic weight adjustment, the feature vectors of different modalities are fused and calculated according to the adjusted weights to obtain a preliminary behavior feature vector. This vector synthesizes the information related to the worker's behavior in the video image, audio, and acceleration sensor data, and takes into account the quality differences of each modality of data. For example, in the weighted summation fusion method, the adjusted spatial position feature vector is multiplied by its corresponding weight, the subtle sound cue feature vector is multiplied by its adjusted weight, and the acceleration change feature vector is multiplied by its weight, and then these three weighted vectors are added together to obtain a preliminary behavior feature vector. This vector will serve as an important input for subsequent behavior recognition analysis, used to further determine whether the worker's operation behavior conforms to safety regulations and to identify potential abnormal actions, etc. Through this method of multi-modal fusion combined with dynamic weight adjustment based on SNR, it is possible to more effectively integrate data from different sources in a complex industrial production workshop environment, improving the accuracy and reliability of the behavior recognition system.

[0088] S6. Through preset key area screening rules, extract the spatio-temporal areas highly relevant to the operation behavior from the behavior feature vector, input the screening result into the long short-term memory network, analyze the context supplementary information, and determine the dynamic sequence features of the worker's operation behavior;

[0089] Furthermore, the preset key area screening rules specifically cover the limitation of the numerical value ranges corresponding to different modalities in the behavioral feature vector. In the spatial position feature dimension, through the analysis of a large amount of operation behavior data of workers in industrial production workshops, observe the positions where workers' standard operation procedures are located during specific equipment operations in the video, so as to specify a specific coordinate interval; in the sound feature dimension, judge the occurrence rules of behavior-related sounds in the audio, determine the unique sound frequency components during equipment operations, and then specify the existence ranges of certain frequency components; in the acceleration change feature dimension, according to the corresponding relationship between acceleration data and actions, for the acceleration change situation of specific operation actions, set the change range of acceleration values. These rules are obtained by deeply analyzing a large number of operation behavior samples containing video, audio, and acceleration sensor data and mining the closely related spatio-temporal feature patterns therein.

[0090] Input the generated preliminary behavioral feature vector into the preset key area screening rules, and use the vector-by-dimension inspection algorithm to perform per-dimension inspection and judgment on the behavioral feature vector. For each behavioral feature vector, in the spatial position feature part, use the coordinate matching algorithm to check whether the elements representing positions in the vector are within the specified coordinate interval; in the subtle sound clue feature part, confirm whether the corresponding frequency component elements meet the set range through the frequency component comparison algorithm; in the acceleration change feature part, use the acceleration value judgment algorithm to judge whether the elements related to the acceleration value are within the specified change interval. When the behavioral feature vector meets the screening rules in the feature dimensions corresponding to each modality, use the spatio-temporal region extraction algorithm to extract its corresponding spatio-temporal region.

[0091] Input the screening result into the LSTM network. The input gate, forget gate, and output gate structures of the LSTM network use the gating mechanism to control information input, retention, and output. At each time step, the network receives the spatio-temporal region feature vector at the current moment as input and combines the hidden state information at the previous moment. The input gate determines which information in the current input data will be retained in the current memory unit through the input gate control algorithm; the forget gate uses the forget gate control algorithm to control which information in the memory unit at the previous moment will be forgotten; the output gate determines the output to the hidden state at the next moment based on the current memory unit state and input information through the output gate control algorithm. Through this mechanism, the LSTM network can effectively use the time series analysis algorithm to capture the change trend of spatio-temporal region features in the time series and the context relationship between features at different moments. For example, it can learn the influence of the spatio-temporal features of the previous action on the performance of the subsequent action features when a worker performs a series of operation actions, so as to mine the context supplementary information in the operation behavior.

[0092] After the LSTM network processes and analyzes the spatio-temporal region features, the hidden state sequence output by the network contains rich dynamic information about the worker's operation behavior. Using statistical analysis algorithms to process the hidden state sequence, the time and amplitude of the peaks and valleys in the sequence are extracted. These information may correspond to the key action nodes in the operation behavior; or calculate the change rate of the hidden state sequence through the change rate calculation algorithm to reflect the speed and acceleration changes of the operation behavior. The key information extracted is combined into a new feature vector using the feature vector combination algorithm. This vector is the dynamic sequence feature representation of the worker's operation behavior. This dynamic sequence feature vector comprehensively describes the dynamic change process of the worker's operation behavior in time and space.

[0093] S7. After obtaining the dynamic sequence features, classify the spatio-temporal region through a lightweight convolutional network. Combining the real-time monitoring requirements, call the pre-trained binary classification model to judge whether the current behavior conforms to the safety specifications, and output the detection results of abnormal behaviors;

[0094] Furthermore, the lightweight convolutional network aims to reduce the computational amount and model parameters while ensuring a certain classification performance to meet the strict requirements of real-time monitoring for computing resources. For example, the MobileNet series of networks uses the Depthwise Separable Convolution technology. When processing the spatio-temporal region, reshape the obtained dynamic sequence feature vector into a tensor form suitable for the input of the lightweight convolutional network.

[0095] After inputting into the lightweight convolutional network, extract and transform the spatio-temporal region features through depthwise separable convolution operations. Depthwise separable convolution decomposes the conventional convolution into depth convolution and pointwise convolution. Depth convolution performs convolution operations on the feature maps of each channel separately, only considering the features in the spatial dimension, while pointwise convolution linearly combines the output channels of the depth convolution through 1x1 convolution kernels to adjust the channel dimension. This way greatly reduces the number of convolution kernel parameters and the computational amount. After alternating processing through multiple convolutional layers, normalization layers, and activation function layers, the network gradually extracts more representative spatio-temporal region features and outputs a low-dimensional feature representation tensor.

[0096] In this scenario, the low-dimensional feature representation tensor output by the lightweight convolutional network is used as the input to the pre-trained binary classification model. For a binary classification model based on logistic regression, the input feature tensor first undergoes a linear transformation, that is, a matrix multiplication operation with the weight matrix of the model, and then the result of the linear transformation is mapped to the interval [0, 1] through a logistic function (Sigmoid function) to obtain the probability value that the behavior belongs to a certain class (such as a safe behavior or an abnormal behavior). For a binary classification model based on SVM, the input feature tensor is mapped in a high-dimensional space through a kernel function (such as a radial basis kernel function), and then an optimal hyperplane is found to separate the two types of behavior features, and the model outputs the class to which the current behavior belongs;

[0097] The real-time monitoring requires the model to be able to process data quickly and give judgment results in a timely manner. According to the probability value or class label output by the binary classification model, combined with the actual safety specification definition, behavior judgment is carried out. For example, if the probability value of the safe behavior output by the binary classification model based on logistic regression is greater than 0.5 (the threshold can be adjusted according to the actual scenario), it is determined that the current behavior conforms to the safety specification; otherwise, it is determined as an abnormal behavior; once the behavior type is determined, the detection result of the abnormal behavior is output.

[0098] S8. According to the detection result of the abnormal behavior, when a violation operation or a fatigue error is detected, the time series analysis method is used to trace the change trend of the previous features, and the conditional random field model is used to optimize the behavior boundary to determine the start and end time points of the abnormal behavior;

[0099] Furthermore, after the abnormal behavior is detected, the time series analysis method is used to trace the change trend of the previous features, and from the stored multi-modal data, the sequence of the worker's spatial position feature vectors, the audio spectrum feature sequence, and the acceleration change value sequence corresponding to the video frame sequence within a specific duration before the detection moment of the abnormal behavior are extracted;

[0100] Time series analysis models such as ARIMA are used to perform stationarity tests, order determination, and parameter estimation on the time series of each modal data, fit and predict the previous time series, and analyze the change trends of the spatial position, audio frequency components, and acceleration before the occurrence of the abnormal behavior;

[0101] Based on the analysis results, construct the input features of the conditional random field (CRF) model, combine the multi-modal features at different time points into a multi-dimensional feature vector sequence, define a feature function that can capture the relationship between features at adjacent time points and the association between features and labels, train the CRF model using the labeled training data, learn the parameters through an iterative optimization algorithm. Finally, input the data processed by time series analysis and CRF model optimization into the abnormal behavior start and end time point determination module. Based on the label sequence output by the CRF model and combined with the timestamp information of the time series data, determine the start time point when the CRF model first predicts the label as an abnormal behavior, and determine the end time point when the model predicts the label as a normal behavior for multiple consecutive time steps, so as to accurately determine the start and end time points of the abnormal behavior on the time axis, providing a key basis for subsequent in-depth analysis and processing of abnormal behaviors.

[0102] S9. Through the start and end time points of the abnormal behavior, combined with the spatial position features and sound clues, generate a behavior recognition report using a preset mapping rule, and dynamically adjust the parameters of the fusion network according to the key area screening results to achieve the adaptive optimization of the monitoring model.

[0103] Furthermore, after determining the start and end time points of the abnormal behavior, extract the corresponding spatial position features and sound clues within this time period from the multi-modal data. The spatial position features cover information such as the specific coordinates and movement trajectories of workers in the workshop in the video, and the sound clues include relevant contents such as operation sounds and environmental background sounds in the audio. Integrate these features and clues according to the preset mapping rule, for example, associate the spatial positions at different moments with the corresponding sound features in chronological order to generate a detailed behavior recognition report. The report content may include the location where the abnormal behavior occurred, the accompanying sound characteristics, and the general process of the behavior, etc. At the same time, refer to the key area screening results to analyze which spatio-temporal region features play a key role in identifying the abnormality during the abnormal behavior period. Based on these analyses, dynamically adjust the parameters of the multi-modal fusion network, such as adjusting the weights of different modal data inputs, changing the parameter settings of convolutional layers or other processing layers in the network, etc., so that the monitoring model can better adapt to the identification of abnormal behaviors in complex scenarios, realize the adaptive optimization of the model, and improve the accuracy and reliability of subsequent abnormal behavior monitoring.

[0104] Embodiment 2

[0105] Please refer to Figure 2 , this embodiment provides a video behavior recognition system based on a neural network for implementing the method of video behavior recognition based on a neural network, including:

[0106] A video data processing module, which is used to obtain video stream data from a surveillance camera, split it into single-frame images in chronological order, read a video file and extract raw image data frame by frame, preprocess the raw image data, then extract spatial features through a convolutional neural network, input the features into a preset multi-object detection algorithm, and identify the preliminary segmentation regions of foreground workers and background devices;

[0107] A spatial feature optimization module, according to the segmentation region results, uses an attention mechanism to weight the spatial features, highlighting important feature regions related to the targets. When the overlap ratio of targets within the segmentation region exceeds a preset threshold, it calls a depth map estimation module to separate overlapping objects, and combines the segmentation region bounding box information and worker pose-related features to determine the complete spatial position features of the main operating worker;

[0108] An audio data processing module, which collects the ambient audio of an industrial production workshop through a microphone, converts sound waves into electrical signals and samples them to obtain a time series signal, uses the short-time Fourier transform to decompose the audio signal into a frequency spectrum, and adopts an adaptive filter based on the least mean square algorithm to suppress machine roars according to the frequency spectrum characteristics and extract subtle sound clues related to behaviors;

[0109] An acceleration data processing module, which obtains the acceleration sensor data of the device worn by the worker, uses the sliding window technique to analyze the time series trend, aligns with the video frame sequence through the dynamic time warping algorithm, corrects the acceleration data offset, and extracts acceleration change features;

[0110] A multi-modal fusion module, which performs feature-level integration on the spatial position feature vector, subtle sound clue feature vector, and acceleration change feature vector to obtain a joint feature vector, then evaluates the signal-to-noise ratio of each modal data. When the signal-to-noise ratio of a certain modal data is lower than a preset threshold, it dynamically adjusts its fusion weight according to the quantization result to generate a preliminary behavior feature vector;

[0111] A key region screening and analysis module, which presets key region screening rules, checks the behavior feature vector dimension by dimension, extracts spatio-temporal regions highly related to the operation behavior, inputs the screening results into a long short-term memory network, and uses the network gating mechanism to analyze context supplementary information to determine the dynamic sequence features of the worker's operation behavior;

[0112] A behavior classification and detection module, which uses a lightweight convolutional network, such as the MobileNet series network, adopts depthwise separable convolution technology to process the spatio-temporal region, extracts a low-dimensional feature representation tensor, inputs it into a pre-trained binary classification model, such as a logistic regression, SVM, etc. model, and combines the real-time monitoring requirements to judge whether the current behavior conforms to safety specifications and outputs the abnormal behavior detection result;

[0113] An abnormal behavior tracing module, when detecting an abnormal behavior, extracts relevant feature sequences within a specific duration before the occurrence of the abnormal behavior from the stored multi-modal data, analyzes the change trend using a time series analysis model such as ARIMA, constructs a conditional random field model to optimize the behavior boundary, and determines the start and end time points of the abnormal behavior;

[0114] A report generation and model optimization module, based on the start and end time points of the abnormal behavior, combines spatial location features and sound clues, generates a behavior recognition report according to the preset mapping rules, and dynamically adjusts the parameters of the multi-modal fusion network according to the key area screening results to achieve the adaptive optimization of the monitoring model, improving the recognition accuracy and reliability.

[0115] The above is only a preferred embodiment of the present invention, and does not impose any form of limitation on the present invention. Although the present invention has been disclosed above with a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or equivalent changes and modifications using the above-disclosed technical content within the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for video behavior recognition based on a neural network, characterized in that: The steps are as follows: Obtain the original image data from the video frame sequence, extract the spatial features of each frame through a convolutional neural network, and input the extracted features into a preset multi-object detection algorithm to identify the preliminary segmentation regions of foreground workers and background devices; According to the segmentation region results, use the attention mechanism to weight the spatial features. When the target overlap ratio within the segmentation region exceeds the preset threshold, call the depth map estimation module to separate the overlapping objects and determine the spatial position features of the main operating worker; Obtain the time series signal from the ambient audio recorded by the microphone, decompose the audio signal into spectra through short-time Fourier transform, and use an adaptive filter to suppress the machine roar according to the spectral characteristics and extract the subtle sound clues related to the behavior; Obtain the acceleration sensor data provided by the worker-worn device, analyze the time series trend through a sliding window, input the analysis results into the dynamic time warping algorithm, align with the video frame sequence, correct the acceleration data offset, and extract the acceleration change features; According to the spatial position features, subtle sound clues, and acceleration change features, use a multi-modal fusion network to perform feature-level integration on the three types of data to obtain a multi-modal joint feature vector. When the signal-to-noise ratio of a certain modal data is lower than the preset threshold, dynamically reduce its fusion weight according to the signal-to-noise ratio quantization result to generate a preliminary behavior feature vector; Through the preset key region screening rules, extract the spatio-temporal regions highly relevant to the operation behavior from the behavior feature vector, input the screening results into the long short-term memory network, analyze the context supplementary information, and determine the dynamic sequence features of the worker's operation behavior; After obtaining the dynamic sequence features, classify the spatio-temporal regions through a lightweight convolutional network, and in combination with the real-time monitoring requirements, call the pre-trained binary classification model to determine whether the current behavior complies with the safety specifications and output the detection results of abnormal behaviors.

2. The method for video behavior recognition based on neural network according to claim 1, characterized in that: Obtain the original image data from the video frame sequence, extract the spatial features of each frame through a convolutional neural network, and input the extracted features into a preset multi-object detection algorithm to identify the preliminary segmentation regions of foreground workers and background devices, specifically including: Obtain the video stream data from the monitoring camera, split the video stream into frames one by one according to the time sequence, then read the video file and extract frame by frame in order to obtain the original image data; Preprocess the obtained original image data, adjust the image size to conform to the model input size, and at the same time normalize the pixel value range; Input the preprocessed image into the convolutional neural network, perform convolutional operations by sliding different convolutional kernels on the image to extract local features; Sample the feature images output by the convolutional layer through the pooling layer, and after processing by multiple convolutional layers and pooling layers, obtain the spatial feature representation of each frame of image; Input the spatial features extracted by the convolutional neural network into the preset multi-object detection algorithm, divide the input image into grids, each grid predicts the nearby targets, and through convolutional operations and prediction heads, output the target category and bounding box; according to the algorithm output results, obtain the preliminary segmentation regions of foreground workers and background devices in each frame of image.

3. The method for video behavior recognition based on neural network according to claim 1, wherein: According to the segmentation region results, the attention mechanism is used to weight the spatial features. When the overlap ratio of the targets within the segmentation region exceeds the preset threshold, the depth map estimation module is called to separate the overlapping objects, and the spatial position features of the main operating worker are determined, specifically including: Integrate the spatial features of each frame of the image extracted by the convolutional neural network into a feature vector set, calculate the similarity between the feature vectors at different positions through the self-attention mechanism, and generate an attention weight matrix; Normalize through the Softmax function to obtain the attention weights, which are used to represent the correlation degree between the features at different positions, perform weighted summation on the numerical vectors, and obtain the weighted segmentation region results; According to the weighted segmentation region results, calculate the overlap ratio of the targets within the region, measure the overlap degree by comparing the ratio of the intersection to the union of the bounding boxes of different targets, and then set a preset threshold. When the ratio of multiple targets within a certain segmentation region exceeds this threshold, it indicates that there is an overlap of targets; Use a pre-trained depth estimation model, input the image region containing the overlapping targets into the model, through feature learning and inference of the image, output the depth map of this region, and determine the spatial position where the main operating worker is located according to the depth map; According to the result of separating the overlapping objects by the depth map estimation module, combined with the bounding box information of the segmentation region, extract the spatial position features of the main operating worker, determine the pixel coordinate range of the region where the main operating worker is located from the depth map, then map it back to the coordinate system of the original image, and at the same time extract the features related to the worker's body posture, and integrate the position coordinate information, depth information and posture-related information to form a complete description of the spatial position features of the main operating worker.

4. The method for video behavior recognition based on a neural network according to claim 1, characterized in that: Analyze the time series trend through a sliding window, and input the analysis results into the dynamic time warping algorithm to align with the video frame sequence, specifically including: Adopt the sliding window technique to analyze the change trend of the acceleration data over time, define a window with a fixed length, slide it sequentially on the time series data, and perform statistical analysis on the acceleration data within each window; By continuously moving the sliding window, perform piecewise analysis on the entire time series data, and obtain a curve of the change of a series of window statistics over time, which is used to observe the average acceleration trend of the worker moving along the X-axis direction at different times; Take the time series of the acceleration data obtained by the sliding window analysis as the sequence to be aligned, and take the time stamps of the video frame sequence as the reference sequence. By calculating the similarity matrix between the two sequences, find an optimal time warping path to make the two sequences match in time; Specifically, calculate the distance between each data point in the two sequences, construct a distance matrix, and then use the dynamic programming algorithm to search for a path with the minimum cumulative distance on the distance matrix. This path represents the best alignment relationship between the two sequences, and stretch or compress the time axis of the acceleration data to align it with the video frame sequence in time.

5. The method for video behavior recognition based on neural network according to claim 1, characterized in that: Through a preset key region screening rule, a spatio-temporal region highly relevant to the operation behavior is extracted from the behavior feature vector, specifically including: inputting the generated preliminary behavior feature vector into the preset key region screening rule, using a vector-by-dimension inspection algorithm to perform dimension-by-dimension inspection and judgment on the behavior feature vector. For each behavior feature vector, in the spatial position feature part, a coordinate matching algorithm is used to check whether the elements representing the position in the vector are within the specified coordinate interval; in the subtle sound clue feature part, a frequency component comparison algorithm is used to confirm whether the corresponding frequency component elements meet the set range; in the acceleration change feature part, an acceleration value judgment algorithm is used to judge whether the elements related to the acceleration value are within the specified change interval. When the behavior feature vector meets the screening rule in the feature dimensions corresponding to each modality, the spatio-temporal region corresponding to it is extracted using the spatio-temporal region extraction algorithm.

6. The method for video behavior recognition based on a neural network according to claim 1, characterized in that: It also includes: According to the detection result of the abnormal behavior, when a violation operation or fatigue mistake is detected, the previous feature change trend is traced through a time series analysis method, and a conditional random field model is used to optimize the behavior boundary to determine the start and end time points of the abnormal behavior.

7. The method for video behavior recognition based on a neural network according to claim 6, characterized in that: Tracing the previous feature change trend through a time series analysis method specifically includes: After an abnormal behavior is detected, the previous feature change trend is traced through a time series analysis method. From the stored multi-modal data, the sequence of worker spatial position feature vectors, the audio spectrum feature sequence, and the acceleration change value sequence corresponding to the video frame sequence within a specific duration before the detection moment of the abnormal behavior are extracted. Through a time series analysis model, the stationarity test, order determination, and parameter estimation of the time series of each modality data are carried out, the previous time series is fitted and predicted, and the change trends of the spatial position, audio frequency components, and acceleration before the occurrence of the abnormal behavior are analyzed.

8. The method for video behavior recognition based on neural network according to claim 6, characterized in that: Using a conditional random field model to optimize the behavior boundary and determine the start and end time points of the abnormal behavior specifically includes: based on the analysis results, constructing the input features of the conditional random field model, combining the multi-modal features at different time points into a multi-dimensional feature vector sequence, defining a feature function that can capture the relationship between the features of adjacent time points and the association between the features and the labels, training the conditional random field model with the labeled training data, learning the parameters through an iterative optimization algorithm, and then inputting the data processed by the time series analysis and the conditional random field model optimization into the abnormal behavior start and end time point determination module. According to the label sequence output by the conditional random field model, combined with the timestamp information of the time series data, the start time point is determined when the label predicted by the conditional random field model for the first time is an abnormal behavior, and the end time point is determined when the labels predicted for multiple consecutive time steps are normal behaviors, so as to determine the start and end time points of the abnormal behavior on the time axis.

9. The method for video behavior recognition based on neural network according to claim 1, characterized in that: It also includes: Through the start and end time points of the abnormal behavior, combined with the spatial position feature and the sound clue, a behavior recognition report is generated using a preset mapping rule, and the fusion network parameters are dynamically adjusted according to the key region screening result to perform adaptive optimization of the monitoring model.

10. A video behavior recognition system based on a neural network, which is applied to the method for video behavior recognition based on a neural network as described in any one of claims 1-9, and is characterized in that: It includes: The video data processing module obtains video stream data from a surveillance camera, splits it into single-frame images in chronological order, reads a video file and extracts raw image data frame by frame, performs preprocessing, extracts spatial features through a convolutional neural network, and inputs them into a multi-object detection algorithm to identify the preliminary segmentation regions of foreground workers and background devices; The spatial feature optimization module, based on the segmentation region results, uses the attention mechanism to weight the spatial features and highlight important feature regions; when objects overlap, it calls the depth map estimation module to separate the overlapping objects and determines the complete spatial position of the main operating worker in combination with the worker's pose features; The audio data processing module collects ambient audio through a microphone, converts sound waves into electrical signals and samples them, decomposes them into spectra using the short-time Fourier transform, suppresses machine noise using an adaptive filter, and extracts sound cues related to behavior; The acceleration data processing module obtains the acceleration sensor data of the worker-worn device, analyzes the time series trend through the sliding window technique, aligns it with the video frame sequence using the dynamic time warping algorithm, corrects data offsets, and extracts acceleration change features; The multi-modal fusion module performs feature-level integration of the spatial position features, subtle sound cue features, and acceleration change features to obtain a joint feature vector, and dynamically adjusts the fusion weights according to the signal-to-noise ratio to generate a preliminary behavior feature vector; The key region screening and analysis module presets key region screening rules, checks the behavior feature vector dimension by dimension, extracts spatio-temporal regions highly relevant to the operation behavior, and inputs them into a long short-term memory network to analyze context information and determine dynamic sequence features; The behavior classification and detection module uses the depthwise separable convolution technique to extract a low-dimensional feature representation tensor, inputs it into a pre-trained binary classification model, and determines whether the behavior complies with safety regulations in combination with real-time monitoring requirements, and outputs the abnormal behavior detection result; The abnormal behavior tracing module, when an abnormal behavior is detected, extracts the feature sequence before the occurrence of the abnormal behavior from the stored multi-modal data, optimizes the behavior boundary using the time series analysis model and the conditional random field model, and determines the start and end time points; The report generation and model optimization module generates a behavior recognition report based on the start and end time points of the abnormal behavior, combined with the spatial position features and sound cues, and dynamically adjusts the fusion network parameters according to the key region screening results to perform adaptive optimization of the monitoring model.

Images