Data processing method and system for realizing moving object detection and behavior recognition

Abstract

The invention discloses a data processing method and system for realizing moving object detection and behavior recognition, and relates to the technical field of computer data processing, and the method comprises the steps: collecting multi-source data, and carrying out the denoising, point cloud outlier elimination and normalization processing; performing initial detection on a moving object, optimizing an anchor frame by using improved YOLOv8, and performing space-time matching with an image candidate frame after radar clustering; multi-dimensional feature extraction: extracting space-time, attitude and attribute features and splicing the space-time, attitude and attribute features into feature vectors; training and reasoning a behavior recognition model, constructing a hybrid model for training, and reasoning an output behavior category and confidence; optimizing a result, dynamically adjusting a confidence coefficient threshold value, performing multi-source verification and matching an abnormal rule; and outputting and feeding back a result, visually outputting and alarming, and storing a log to support retrieval. According to the method, environmental interference is solved through multi-source fusion, the anchor frame is optimized, missing detection is reduced, the model gives consideration to precision and speed, abnormal misjudgment is reduced through multi-source verification, and the method is adaptive to multiple scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data processing, and particularly relates to a data processing method and system for realizing active object detection and behavior recognition. BACKGROUND

[0002] In actual application, the current active object detection and behavior recognition technology first faces the limitations of multi-source data acquisition and preprocessing. Most existing schemes only rely on a single RGB camera to collect data, which is significantly affected by light changes (such as strong light, backlight, night low illumination), occlusion, resulting in increased image noise and loss of details. For example, in night monitoring, the human body contour is blurred, and traditional Gaussian filtering can only remove part of the noise, and cannot restore the key features. Although the application of multi-source devices such as millimeter wave radar and infrared sensor can supplement depth and temperature information, the data formats of different devices are quite different, the timestamp synchronization error often exceeds 50 ms, the point cloud data outliers are not effectively removed, and the data normalization is not uniform, which ultimately causes the multi-source data to be unable to be effectively fused, affecting the subsequent detection accuracy. In addition, in the preprocessing process, the salt and pepper noise is often processed by fixed window median filtering, which does not dynamically adjust the window size according to the noise density. In a high-noise scene, the filtering effect is poor, and there are still a large number of noise disturbances to the object detection.

[0003] There are obvious shortcomings in the initial detection of active objects, which are difficult to adapt to complex scene requirements. The anchor frame size of the traditional YOLO series algorithm is fixed, and the detection miss rate for small size objects is high. For example, in an intelligent park, children are shorter, and the existing anchor frame cannot completely wrap the object, resulting in a detection confidence lower than 0.5, which is directly judged as a missed detection. At the same time, the anchor frame is not self-adapted according to the image resolution. When the monitoring device resolution is switched from 1080P to 720P, the original anchor frame is out of proportion with the actual size of the object, and the detection accuracy drops sharply. In addition, although some schemes introduce radar data assistance, they only match the image and radar data through simple coordinate comparison, and do not combine edge detection to extract object contour features for fitting correction, resulting in an overlap rate of less than 70% between the candidate frame and the actual object contour, and the repeated candidate frame cannot be effectively removed, increasing the calculation amount of subsequent feature extraction, and at the same time, prone to the error of "multiple annotations of the same object".

[0004] The defects of the behavior recognition model and the result optimization link further limit the technology landing effect. The existing behavior recognition model mostly uses single CNN or LSTM structure. Although CNN can extract spatial features, it is difficult to capture the time sequence correlation of behavior. Although LSTM can process time sequence data, it lacks attention to spatial key features, resulting in low accuracy in distinguishing similar behaviors. During model training, there is a lack of means to improve the generalization ability for scene changes. When the scene has a sudden change in light or the area of occlusion exceeds 30%, the behavior confidence fluctuates greatly and is prone to "misjudgment". For example, under strong outdoor light, the human body movement speed is misjudged to be accelerated, and the "walking" behavior is misclassified as "running". Moreover, the model size is generally large (more than 300 MB), and the inference speed is less than 10 fps, which cannot adapt to edge computing devices and can only rely on cloud processing, increasing data transmission delay. In addition, the abnormal behavior judgment mostly uses a fixed rule library, and users cannot customize rules according to scene requirements (such as adding a "violation of crossing guardrails" rule in an industrial scene), and it is not combined with multi-source data fusion verification. When a single data deviates, the false positive rate of abnormal behavior is high. For example, only through image judgment "falling down", without combining radar speed data (falling down speed drops feature), "squatting to pick up things" is easily misjudged as "falling down". SUMMARY

[0005] The present application provides a data processing method and system for detecting and recognizing moving objects, which solves the problems mentioned in the prior art.

[0006] To achieve the above purpose, the present application adopts the following technical scheme: A data processing method for detecting and recognizing moving objects, comprising the following steps: Step 1: Multi-source data acquisition and preprocessing, collecting moving object data in the monitoring area, data sources including industrial cameras, millimeter wave radars and infrared sensors, preprocessing the collected data, first removing image noise by Gaussian filtering, then eliminating salt and pepper noise by median filtering, removing outliers from radar point cloud data, and normalizing the data to the [0, 1] interval, synchronizing the time stamp; Step 2: Moving object preliminary detection and candidate box generation, using an improved YOLOv8 algorithm to detect RGB images and optimizing small size object anchor boxes; clustering the positions of radar point cloud data, matching the image and radar results, removing duplicate boxes, and retaining effective candidate boxes with a confidence of ≥0.7 to determine the preliminary position and number of moving objects; Step 3: Multi-dimensional feature extraction, extracting spatio-temporal, posture and attribute features from the object region corresponding to the effective candidate box, and concatenating the three types of features into a feature vector after standardization; Step four: behavior recognition model training and reasoning, a "CNN-LSTM-attention" hybrid model is constructed for behavior recognition, the CNN layer reduces the dimension of the feature vector, the LSTM layer captures the time correlation of the behavior, and the attention layer strengthens the weight of the key behavior features; the cross-entropy loss function is used for model training, and data enhancement is used to improve the generalization ability; during reasoning, the real-time extracted feature vector is input into the trained model, and the behavior category and corresponding confidence are output; Step five: recognition result optimization and abnormal behavior determination, set a dynamic confidence threshold, when the behavior confidence is lower than the threshold, call multi-source data fusion verification; define an abnormal rule library, match the reasoning result with the rule library, and mark it as an abnormal behavior if it meets the abnormal rule; Step six: result output and feedback, the position, number, behavior category and abnormal marking information of the active object are visualized and output to the monitoring terminal, the abnormal behavior triggers sound and light alarm, and the behavior recognition log is generated and stored in the local database.

[0007] Further, it further includes a feature weight dynamic adjustment step, which is executed after the feature standardization and splicing into a feature vector in the multi-dimensional feature extraction step and before the feature vector is input into the behavior recognition model training and reasoning step; the specific process is: first calculate the contribution weight of each type of feature, the formula is W=(F×S) / λ, where W is the feature weight value, F is the correlation coefficient of the feature and the behavior category, S is the discrimination degree of the feature, and λ is the feature dimension normalization coefficient; then, according to the calculated feature weight value W, the contribution of the three types of features in the feature vector is weighted and adjusted, and the support of high weight features to behavior recognition is strengthened; the adjusted weighted feature vector is directly used as the input data of the CNN layer in the behavior recognition model training and reasoning step, realizing the connection with the model training and reasoning process.

[0008] Further, the active object preliminary detection in step two also includes candidate box size optimization, the anchor box proportion is adjusted for different resolution images, when the image resolution is lower than 1280x720, the original anchor box size is reduced by 0.8 times, when the resolution is higher than 2560x1440, it is enlarged by 1.2 times; at the same time, edge detection is introduced to assist in identifying the outline features of the object and correcting the candidate box.

[0009] Further, the recognition result optimization in step five also includes behavior confidence environment correction, when the monitoring scene has light mutation and shielding, the initial confidence is corrected, the correction method is C=C0×(1-I×D), where C is the corrected confidence, C0 is the initial output confidence of the model, I is the light interference coefficient, and D is the shielding influence coefficient.

[0010] Further, the behavior recognition model training in step four also adopts a transfer learning strategy, the pre-trained ImageNet dataset initializes the CNN layer parameters, and then the behavior dataset of the application is fine-tuned, the progressive learning rate is adopted during fine-tuning, and the knowledge distillation is introduced, which improves the model inference speed to more than 20fps on the premise of ensuring the recognition accuracy.

[0011] Further, step two also includes multi-source data space-time alignment optimization, the time stamp interpolation method is used to process data acquisition delay, when the camera and radar data time stamp deviation is greater than 10ms, the lag data is linearly interpolated; in terms of space alignment, the mapping relationship between image pixel coordinates and radar three-dimensional coordinates is established through the calibration board, the radar point cloud is projected into the image coordinate system, the IOU of the projection point and the image candidate box is calculated, and the same object is determined when IOU is greater than or equal to 0.5.

[0012] Further, it also includes a model online updating step, the trigger period of which is synchronized with the log storage period in the result output and feedback step, and the specific process is as follows: the behavior recognition logs in the past 7 days are extracted from the local database regularly, the low confidence samples with confidence less than 0.6 are screened, the samples are derived from the inference results output by the behavior recognition model training and inference steps and the logs stored in the result output and feedback step, and after artificial labeling, the training set is added, the model is iteratively optimized by using incremental training, the accuracy is verified by the test set after each update, the new model is saved when the accuracy is improved by more than 3%, the original inference model is replaced, the new model is immediately used in the inference process of the subsequent behavior recognition model training and inference steps, the connection of model updating and inference process is realized; at the same time, the model updating log is recorded, and the log is stored in the local database of the result output and feedback step.

[0013] Further, a system for realizing a data processing method of activity object detection and behavior recognition includes the following modules: A multi-source data acquisition and preprocessing module is used for acquiring and processing activity object data, which includes a camera unit, a millimeter wave radar unit, an infrared sensor unit and a preprocessing unit; the preprocessing unit integrates Gaussian filtering, median filtering, point cloud outlier rejection and data normalization functions, and synchronizes the time stamp; An activity object detection module is used for generating activity object candidate boxes, which includes an image detection unit, a radar clustering unit and a candidate box matching unit; the candidate box matching unit calculates the IOU of the image detection box and the radar clustering result, eliminates the duplicate box and retains the effective box with confidence greater than or equal to 0.7, and outputs the preliminary position and number of objects; The multi-dimensional feature extraction module is used for extracting object features, and comprises a space-time feature extraction unit, a posture feature extraction unit, an attribute feature extraction unit and a feature splicing unit; the feature splicing unit splices the three types of features into a 512-dimensional feature vector after standardization, and outputs to the behavior recognition module; The behavior recognition module is used for outputting behavior categories and confidence, and comprises a model training unit, a model inference unit and a model updating unit; the model training unit supports transfer learning and knowledge distillation; The recognition result optimization module comprises a dynamic threshold unit, a multi-source fusion verification unit, an abnormal rule library unit and a confidence correction unit; the abnormal rule library unit supports user-defined rules; The result output and storage module comprises a visual terminal unit, an alarm unit and a log storage unit; the log storage unit automatically retains log data for 3 months, and automatically archives to the cloud beyond that.

[0014] Further, the multi-source data acquisition and preprocessing module further comprises a data quality detection unit, which monitors the original data collected by the camera unit, the millimeter wave radar unit and the infrared sensor unit in the multi-source data acquisition and preprocessing module and the output data processed by the preprocessing unit in real time, and the detection indexes include the integrity, the definition and the validity of the temperature data of the data; the self-checking of the corresponding collection equipment in the multi-source data acquisition and preprocessing module is triggered first; if the data quality is still not up to standard after self-checking, an equipment fault alarm is pushed to the monitoring terminal of the result output and storage module immediately, and the output of substandard data to the active object detection module is suspended at the same time until the data quality is recovered to be qualified, so that the data input into the active object detection module meets the accuracy requirement of the candidate box generation.

[0015] Further, the behavior recognition module further comprises a model lightening unit, which is integrated in the behavior recognition module, and performs lightening processing after the model training unit completes model training and before the model inference unit deploys the model; the model pruning, quantization and layer fusion technologies are adopted to compress the model volume from the initial 200MB to below 50MB, while keeping the inference accuracy loss less than or equal to 3%; the model after lightening can still meet the requirement that the inference speed of the model inference unit in the behavior recognition module is greater than or equal to 20fps when the active object detection module outputs high-concurrency candidate boxes; the model lightening unit generates corresponding lightening model versions according to the computing power requirement of the target deployment hardware platform, which are respectively used for the inference tasks of the model inference unit in the behavior recognition module on different hardware platforms, so as to realize the cooperation with the model inference unit and the hardware deployment scene; at the same time, the model parameters after lightening are fed back to the model updating unit, which are used as the basis parameters for subsequent incremental training of online model updating.

[0016] Compared with the existing technology, the application has the following advantages: In the multi-source data acquisition and preprocessing stage, the system integrates high-definition cameras, millimeter-wave radar, and infrared sensors, covering multiple types of data such as RGB images, depth, and temperature. This effectively compensates for the limitations of a single data source. For example, in nighttime scenes, infrared sensors capture human body temperature characteristics, while radar provides depth information. By combining Gaussian filtering and dynamic window midpoint filtering, noise is removed more thoroughly. Data normalization and timestamp synchronization ensure the spatiotemporal consistency of multi-source data, laying a high-quality data foundation for subsequent detection and avoiding detection deviations caused by poor data quality.

[0017] The accuracy of moving object detection has been significantly improved, adapting to different scenarios and object size requirements. The improved YOLOv8 algorithm optimizes anchor boxes for small objects, and combines radar point cloud clustering and edge detection to correct candidate boxes, greatly increasing the overlap rate between candidate boxes and the actual object contour and significantly reducing the false negative rate. For example, in smart parks, the detection of children and small devices is no longer missed due to size issues. Images of different resolutions are adjusted adaptively by the anchor box ratio, resulting in stable detection accuracy. Spatiotemporal alignment and IOU matching of multi-source data effectively eliminate duplicate candidate boxes, reduce redundant calculations, and ensure accurate matching of the same object across multiple sources. In densely populated scenarios, the object matching accuracy is greatly improved, avoiding the error of "multiple annotations of the same object".

[0018] The behavior recognition model balances generalization ability and inference efficiency, adapting to diverse deployment scenarios. The "CNN-LSTM-Attention" hybrid model extracts both spatial features and captures temporal correlations. The attention mechanism strengthens key behavioral features. Combined with data augmentation and transfer learning, the model can still stably recognize behaviors in complex scenarios such as sudden changes in lighting and occlusion, with more accurate differentiation of similar behaviors and a lower false positive rate. Knowledge distillation and lightweight processing significantly compress the model size and improve inference speed, enabling real-time detection on edge devices and deep analysis in the cloud, meeting the needs of "edge-cloud" layered processing. The online update mechanism optimizes the model through incremental training, avoiding accuracy decay caused by scene changes. The recognition accuracy remains stable during long-term operation without the need for frequent manual retraining.

[0019] The system offers more accurate anomaly detection, resulting in superior output and system reliability. Dynamic confidence thresholds and multi-source data fusion verification reduce false positives caused by environmental interference. A customizable anomaly rule base meets the needs of different scenarios. Anomaly behavior triggers audible and visual alarms and detailed log recordings, facilitating timely response and traceability by maintenance personnel. A data quality detection unit monitors the integrity and clarity of collected data in real time, and automatic self-checks and alarms for equipment failures ensure long-term stable system operation and prevent detection interruptions due to data acquisition equipment malfunctions. Overall, this invention achieves a highly efficient and reliable end-to-end process of "data acquisition-detection-identification-optimization-output," applicable to various scenarios such as security, industry, and industrial parks. It significantly improves the intelligence level of active object monitoring and anomaly early warning, while reducing manual maintenance costs. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 a schematic block diagram of a data processing method for detecting and recognizing moving objects according to the present application; Figure 2 a schematic block diagram of a data processing system for detecting and recognizing moving objects according to the present application; Figure 3 a grouped column chart for comparing the detection accuracy of moving objects in different scenarios; Figure 4 a grouped column chart for comparing the model performance in different hardware platforms; Figure 5 a grouped column chart for comparing the misjudgment rate of different abnormal behaviors. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0022] Reference Figures 1 to 5 A data processing method for detecting and recognizing moving objects, comprising the following steps: Step 1: Multi-source data acquisition and preprocessing, collecting moving object data in the monitoring area, the data sources including high-definition industrial cameras (resolution 1920x1080, frame rate 25fps), millimeter wave radars (detection distance 0-50m, ranging accuracy ±0.1m) and infrared sensors (temperature measurement range -20℃-80℃), the collected data types including RGB images, depth images, radar point clouds and temperature data; preprocessing the collected data, first removing image noise by Gaussian filtering (standard deviation σ=1.2), then eliminating salt and pepper noise by median filtering (window size 3x3), removing outliers of radar point cloud data (based on K-neighbor algorithm, K=5, distance threshold 0.3m), finally normalizing the multi-source data to the interval [0,1] and synchronizing the time stamp (error ≤10ms) to realize data space consistency; Step 2: Initial detection of active objects and generation of candidate boxes. An improved YOLOv8 algorithm is used to detect objects in the preprocessed RGB image. For small objects (such as children and pets), the anchor box size is optimized (3 sets of anchor boxes are set: 10×14, 20×28, and 30×42). The radar point cloud data is clustered (DBSCAN algorithm, ε=0.5m, minimum number of points 5) to obtain object location information. The image detection candidate boxes are spatiotemporally matched with the radar clustering results (IOU threshold ≥0.6). Duplicate candidate boxes are removed, and valid candidate boxes with confidence ≥0.7 are retained to determine the initial location and number of active objects. Step 3: Multi-dimensional feature extraction. For the object region corresponding to the valid candidate box, three types of core features are extracted: First, spatiotemporal features, which are extracted from 16 consecutive frames of images through 3D convolution (3×3×3 kernel, stride 1), and the object's motion trajectory is calculated by combining optical flow (Farneback algorithm); Second, pose features, which are extracted from the human object using the MediaPipe pose estimation model to extract the coordinates of 21 key joints and calculate the angle, distance, and motion speed between joints; Third, attribute features, which are extracted from the object's color histogram (bins=64), texture features (LBP operator, radius 3, neighborhood 8), and radar reflection intensity (used to distinguish between metallic and non-metallic objects). The three types of features are standardized and concatenated into a feature vector (dimension 512). Step 4: Behavior recognition model training and inference. A hybrid "CNN-LSTM-Attention" model is constructed for behavior recognition. The CNN layer (containing 4 convolutional blocks, each containing 2 convolutional layers + 1 pooling layer) reduces the dimensionality of the feature vectors. The LSTM layer (256 hidden units, dropout=0.3) captures the temporal correlation of behaviors. The attention layer (using spatial attention mechanism) strengthens the weights of key behavioral features. During model training, the cross-entropy loss function is used, combined with data augmentation (random flipping, scaling, brightness adjustment) to improve generalization ability. The basic training set contains 10 common behaviors (walking, running, falling, carrying, waving, etc., 5000 frames per category). In industrial scenarios, each category is expanded to 8000 frames and subdivided into 5 part types. By increasing the sample size and fine-grained features, the accuracy of industrial behavior recognition is improved, conforming to the training specifications of vertical domain models. During inference, the feature vectors extracted in real time are input into the trained model, and the behavior category and corresponding confidence score are output. Step five: identify the result optimization and abnormal behavior determination, set the dynamic confidence threshold, adjust according to the scene complexity (indoor scene threshold 0.65, outdoor complex scene threshold 0.75), when the behavior confidence is lower than the threshold, call multi-source data fusion verification (compare the radar motion speed with the image behavior characteristics, and the deviation is less than or equal to 10% to determine the validity); define the abnormal behavior rule library (such as the falling behavior accompanied by a speed drop of more than 80%, the human body posture angle anomaly is greater than 60 degrees, and the carrying behavior is accompanied by an object volume change of more than 30%), match the inference result with the rule library, and mark the abnormal behavior as an abnormal behavior if it meets the abnormal rule; Step six: result output and feedback, the position, number, behavior category and abnormal mark information of the active object are output to the monitoring terminal in a visual interface (real-time display of monitoring picture, object bounding box, behavior label), abnormal behavior triggers sound and light alarm (buzzer frequency 2kHz, red light flashing), and behavior recognition log (including timestamp, object ID, behavior details, confidence) is generated at the same time, and stored in the local database (MySQL, read / write speed is greater than or equal to 100MB / s), which supports time interval and behavior type retrieval query.

[0023] In the application, a feature weight dynamic adjustment step is further included, which is executed after the feature standardization and splicing into a feature vector in the multi-dimensional feature extraction step (step three) and before the feature vector is input into the behavior recognition model training and reasoning step (step four); the contribution weight of each type of feature is calculated, and the weight calculation method is W=(F×S) / λ, wherein W is the feature weight value, F is the correlation coefficient of the feature and the behavior category, which is calculated through Pearson correlation analysis and the value range is [-1, 1]; S is the distinguishability of the feature, which is calculated through the ratio of the intra-class distance to the inter-class distance, and the value range is [0, 1]; and λ is the feature dimension normalization coefficient, which is set according to the total dimension of the feature vector 512, and the value of λ is 512; then the contribution of the three types of features in the feature vector is weighted and adjusted according to the calculated feature weight value W, so as to strengthen the supporting role of high weight features to behavior recognition; and the adjusted weighted feature vector is directly used as the input data of the CNN layer in the behavior recognition model training and reasoning step (step four), so as to realize the connection with the model training and reasoning process.

[0024] Through the weight adjustment method, the influence of high correlation and high distinguishability features can be strengthened, for example, the W value of the human joint angle feature is usually not less than 0.8; and the interference of low contribution features can be weakened, for example, the W value of the background texture feature is usually not higher than 0.3. This adjustment makes the feature vector input into the behavior recognition model more targeted, and the behavior recognition accuracy can be improved by more than 8% in the outdoor light change scene.

[0025] In the present application, the initial detection of the moving object in step two also includes candidate box size optimization, which adaptively adjusts the anchor box ratio for different resolution images: when the image resolution is lower than 1280*720, the original anchor box size is reduced by 0.8 times; when the image resolution is higher than 2560*1440, the original anchor box size is enlarged by 1.2 times.

[0026] At the same time, the candidate box size optimization also introduces edge detection assistance, which is realized by using Canny algorithm, and the algorithm threshold is set to 100-200. The object contour features are identified through edge detection, and then the edge fitting correction of the candidate box is performed according to the contour features. This correction makes the overlap rate of the candidate box and the actual contour of the object increase to more than 90%, effectively reducing the missed detection caused by the mismatch of anchor box size, especially for non-standard size objects such as strollers and large packages, the missed detection rate can be reduced to less than 3%.

[0027] In the present application, the recognition result optimization in step five also includes behavior confidence environment correction. When there is a light mutation (brightness change ≥ 50%) or occlusion (occlusion area ≥ 40%) in the monitoring scene, the initial confidence is corrected. The correction method is C=C0×(1-I×D), where C is the corrected confidence, C0 is the initial output confidence of the model, I is the light interference coefficient (calculated according to the image brightness variance, if the brightness variance is greater than 100, then I=0.15, otherwise I=0), and D is the occlusion influence coefficient (calculated according to the occlusion area ratio, if the occlusion area is 30%-40%, then D=0.1, if the occlusion area is 40%-60%, then D=0.2, and if the occlusion area is greater than 60%, then D=0.3). Through this correction, false positives caused by environmental interference are avoided, such as the probability of misjudging "walking" behavior as "running" under strong outdoor light is reduced to less than 5%, and the missed detection rate of "falling" behavior in the occlusion scene is controlled within 4%.

[0028] In the present application, the behavior recognition model training in step four also uses a transfer learning strategy to initialize the CNN layer parameters with a pre-trained ImageNet dataset, and then fine-tunes the behavior dataset of the present application. During fine-tuning, an incremental learning rate (initial learning rate 1e-4, decaying to 0.8 times of the previous stage every 5 epochs) is used, and knowledge distillation (teacher model is ResNet50+BiLSTM, student model is lightweight MobileNet+LSTM) is introduced. Under the premise of ensuring recognition accuracy (loss ≤ 0.05), the model inference speed is improved to more than 20fps, which adapts to the real-time processing needs of edge computing devices (such as embedded ARM platforms with computing power ≤ 5TOPS).

[0029] In the application, step two also includes multi-source data space-time alignment optimization. For the scene of industrial workshops, factory sites and the like existing electromagnetic interference and cumulative synchronization error of long-term operation of equipment, scene adaptation processing is added on the basis of the time stamp interpolation method: time alignment adopts the time stamp interpolation method to process data collection delay. First, through the data quality detection unit of the multi-source data collection and preprocessing module, the time stamp signal stability of the continuous 5 frames of camera and radar data is monitored in real time (calculate the time stamp standard deviation, and the preset threshold is 3ms) - if the standard deviation is less than 3ms (no significant electromagnetic interference), when the time stamp deviation is greater than 10ms, linear interpolation is performed on the lagging data. The interpolation is based on the time interval of the adjacent 2 frames of data, and the interpolation step is 1ms; if the standard deviation is greater than 3ms (determined as time stamp jump caused by electromagnetic interference), linear interpolation is not performed, and the mean value of the effective time stamps of the previous 3 frames is used to replace the time stamp of the jump frame to avoid alignment deviation caused by interference. At the same time, in order to solve the problem of cumulative synchronization error of long-term operation of industrial equipment, the time stamp reference calibration is triggered once every 24 hours (matching the regular operation and maintenance period of industrial equipment): through industrial Ethernet (such as Profinet protocol) access to the unified clock source of the workshop, the time stamps of the camera and the radar are synchronized with the clock source, and the accumulated time deviation is reset to less than or equal to 5ms. After calibration, the time stamp deviation judgment and interpolation logic are re-executed.

[0030] Space alignment establishes the mapping relationship between image pixel coordinates and radar three-dimensional coordinates through a calibration board. The calibration board is a chessboard with a size of 10x10cm, and the mapping error is not more than 2 pixels. After projecting the radar point cloud to the image coordinate system, the IOU of the projection point and the image candidate box is calculated. If the IOU is not less than 0.5, it is determined that it is the same object. In view of the characteristics of industrial scenes, such as dense equipment and many metal reflection interferences, additional matching verification of the projection point and the equipment contour is added (the equipment contour is extracted through edge detection, and if the projection point falls within the equipment contour with a proportion of more than 80%, the object matching validity is confirmed), so as to ensure that the descriptions of the same object by multi-source data are consistent. In a personnel-intensive scene (the number of people is not less than 5) or a device-intensive scene (the number of devices in a single region is greater than or equal to 10), the object matching accuracy can reach more than 95%, meeting the alignment requirements in complex industrial scenes.

[0031] In the application, the model online updating step is also included. The triggering period of the step is synchronized with the log storage period in the result output and feedback step (step six), the behavior recognition log of the recent 7 days is extracted from the local database regularly, and then the causes of low confidence samples are judged in combination with the system association module data, and the low confidence samples are processed in a targeted manner: Low confidence sample cause judgment: through the data quality detection unit log of the multi-source data collection and preprocessing module and the multi-source fusion verification unit record of the recognition result optimization module, the samples with a confidence less than 0.6 are classified: If the data quality detection unit records "image sharpness (Laplacian variance) <100", "radar point cloud quantity <100 points / frame" or "infrared temperature data deviation >2℃", it is judged as a data quality defect sample (due to invalid data collection leading to low confidence). If the multi-source fusion verification unit records "radar and image feature deviation > 10%" and the synchronization log shows "sudden change in illumination (brightness change ≥ 50%)" and "occlusion area ≥ 40%", it is judged as an environmental interference sample (the model misjudged due to scene interference). If both the data quality detection unit and the multi-source fusion verification unit record "data valid, feature matching", but the model output confidence score is <0.6, it is judged as a sample with model defects (because the model has not fully learned the behavioral features of this type, resulting in low confidence score).

[0032] Sample selection and annotation: Extract behavior recognition logs from the past 7 days (sample size no less than 10,000 frames), remove samples with data quality defects and environmental interference, and retain only low-confidence samples with model defects (confidence <0.6); manually annotate the retained samples, supplement the behavioral sub-feature descriptions (such as the joint angle range and equipment reflection intensity features of "illegally dismantling equipment" in industrial scenarios), and add them to the training set after ensuring the completeness of the annotation information.

[0033] Incremental training optimization: The model is iteratively optimized by incremental training, that is, the parameters of the CNN layer are fixed and only the weights of the LSTM and attention layers are updated. For the labeled behavioral sub-features, the learning weights of the LSTM layer for temporally related features are strengthened (such as increasing the temporal weights of the frames related to "device disassembly" by 15%), and the attention layer pays attention to key attribute features, so as to ensure that the model is optimized in a targeted manner and avoid invalid samples from interfering with the training effect.

[0034] After each update, accuracy is verified using a test set containing 5000 frames of samples not used in training. When the accuracy improvement is at least 3%, the new model is saved, replacing the original inference model. The new model is immediately used in the inference process of the subsequent behavior recognition model training and inference steps (step four), achieving seamless integration between model updates and the inference flow. Simultaneously, a model update log is recorded and synchronously stored in the local database of the results output and feedback steps (step six). This log includes update time, the number of new samples, and accuracy changes, preventing performance degradation due to scene changes such as seasonal variations in clothing color, ensuring that behavior recognition accuracy remains above 92% over long-term operation.

[0035] This invention includes the following modules: The multi-source data acquisition and preprocessing module is used for collecting and processing active object data, and the module includes a high-definition camera unit, a millimeter wave radar unit, an infrared sensor unit, and a preprocessing unit. The high-definition camera unit has a resolution of 1920x1080 and a frame rate of 25 fps, and supports an automatic focusing function. The millimeter wave radar unit has a detection distance of 0-50 m, a ranging accuracy of ±0.1 m, and a point cloud output frequency of 10 Hz. The infrared sensor unit has a temperature measurement range of -20℃-80℃ and a sampling interval of 1 s. The preprocessing unit integrates Gaussian filtering, median filtering, point cloud outlier rejection, and data normalization functions. The standard deviation σ of the Gaussian filtering is set to 1.2, the median filtering uses a 3x3 window, and the point cloud outlier rejection is based on the K-neighbor algorithm with a K value of 5 and a distance threshold of 0.3 m. The preprocessing unit can unify the data format and synchronize the time stamp, and the time stamp error is not more than 10 ms. The active object detection module is used for generating active object candidate boxes, and includes an image detection unit, a radar clustering unit, and a candidate box matching unit. The image detection unit deploys an improved YOLOv8 algorithm, and optimizes the anchor box for small size objects. The optimized anchor box size is 10x14, 20x28, and 30x42. The radar clustering unit uses the DBSCAN algorithm, and the ε value in the algorithm is set to 0.5 m and the minimum point number is set to 5. The candidate box matching unit calculates the IOU of the image detection box and the radar clustering result, and the IOU threshold is set to not less than 0.6. By calculating, the repeated boxes are removed and the effective boxes with a confidence of not less than 0.7 are retained, and finally the object preliminary position and quantity are output. The multi-dimensional feature extraction module is used for extracting object features, and includes a spatio-temporal feature extraction unit, a pose feature extraction unit, an attribute feature extraction unit, and a feature concatenation unit. The spatio-temporal feature extraction unit uses 3D convolution and Farneback optical flow method, and the convolution kernel size of 3D convolution is 3x3x3. The pose feature extraction unit is based on the MediaPipe model, and can realize 21 joint point detection. The attribute feature extraction unit extracts color histogram, LBP texture, and radar reflection intensity, and the bins value of color histogram is set to 64 and the radius of LBP texture is set to 3. The feature concatenation unit concatenates the three types of features into a 512-dimensional feature vector after standardization, and outputs to the behavior recognition module after concatenation. The behavior recognition module is used for outputting behavior categories and confidence, and includes a model training unit, a model inference unit, and a model updating unit. The model training unit uses a CNN-LSTM-attention hybrid model, and uses a cross-entropy loss function and combines data enhancement technology in the training process, and supports transfer learning and knowledge distillation. The model inference unit receives feature vector input, outputs corresponding behavior categories and confidence, and the processing speed is not less than 20 fps. The model updating unit uses incremental training method to realize online iterative optimization of the model. The recognition result optimization module is used to improve the recognition accuracy, and comprises a dynamic threshold unit, a multi-source fusion verification unit, an abnormal rule library unit and a confidence correction unit. The dynamic threshold unit sets the threshold value for different scenes, and the threshold value for indoor scenes is set to 0.65, and the threshold value for outdoor scenes is set to 0.75; the multi-source fusion verification unit compares the radar and image features, and when the deviation between the two is not more than 10%, it is determined that the feature is effective; the abnormal rule library unit stores the rules of abnormal behaviors such as falling and abnormal handling, and supports user-defined rules, for example, adding an abnormal behavior such as 'climbing'; the confidence correction unit corrects the confidence by a light and shielding correction formula; The result output and storage module is used to display and save the recognition result, and comprises a visual terminal unit, an alarm unit and a log storage unit. The visual terminal unit displays the monitoring picture, the object bounding box and the behavior label in real time; the alarm unit comprises a 2kHz buzzer and a red light, and the buzzer sounds and the red light flashes when an abnormality occurs; the log storage unit adopts a MySQL database, the database reading and writing speed is not less than 100MB / s, and multi-dimensional retrieval is supported; the log storage unit automatically retains log data for 3 months, and data exceeding 3 months is automatically archived to the cloud.

[0036] In the present application, the multi-source data acquisition and preprocessing module further comprises a data quality detection unit for real-time monitoring of the raw data collected by the camera unit, the millimeter wave radar unit and the infrared sensor unit in the multi-source data acquisition and preprocessing module and the output data after processing by the preprocessing unit; the detection indexes include the integrity, the definition and the temperature data validity of the data. The integrity requires that the image frame loss rate is not more than 1% and the number of radar point clouds is not less than 100 points per frame; the definition is calculated by the Laplacian operator to calculate the image blurriness, and when the variance is not less than 100, it is determined to be clear; the temperature data validity requires that the deviation of the infrared sensor data is not more than 2℃. When the data quality is not up to standard, the corresponding acquisition device in the multi-source data acquisition and preprocessing module is triggered for self-checking, specifically including automatic focusing of the camera, restarting of the radar and calibration of the sensor; if the data quality is still not up to standard after self-checking, a device fault alarm is pushed to the monitoring terminal of the result output and storage module, and the unqualified data is temporarily suspended from being output to the active object detection module until the data quality is restored to be qualified, and the fault device name and the fault type are displayed in the alarm information, so as to ensure the reliability of the data input to the subsequent modules and reduce the detection and recognition errors caused by the data quality problems.

[0037] In the present application, the behavior recognition module further comprises a model lightening unit integrated in the behavior recognition module, which performs lightening processing after the model training unit completes model training and before the model inference unit deploys the model; the unit realizes model lightening by using model pruning, quantization and layer fusion technology. Model pruning is performed by removing redundant convolution kernels, and the pruning rate is 30%; quantization quantizes 32-bit floating-point parameters to 8-bit integers; layer fusion combines consecutive convolution and activation layers. Through these technologies, the model volume is compressed from the initial 200MB to less than 50MB, while the inference accuracy loss is not more than 3%, ensuring that the lightened model can still meet the requirement that the inference speed of the model inference unit in the behavior recognition module is ≥20fps when the active object detection module outputs high-concurrent candidate boxes; the model lightening unit generates corresponding lightened model versions according to the computing power requirements of the target deployment hardware platform (such as edge devices, cloud servers). The model lightening unit adapts to different hardware platforms such as x86 servers, ARM embedded devices and FPGAs, generates corresponding model versions for each platform, meets the deployment requirements of different scenarios, such as deploying lightened models on edge devices and deploying high-precision complete models on cloud servers, thereby realizing a layered processing architecture of "edge real-time detection + cloud deep analysis".

[0038] The specific embodiments of the system are further illustrated by two embodiments as follows: Embodiment 1: Intelligent park security scene (outdoor environment, daily personnel flow 200-300 people, including walking, running, falling, waving and staying) 1. Detailed implementation of technical solutions (1) Multi-source data acquisition and preprocessing A combination of "high-definition camera + millimeter wave radar + infrared sensor" is used for acquisition: the high-definition camera is selected from Hikvision DS-2CD3T46DWD-I5 (1920x1080 resolution, 25fps frame rate, supporting wide dynamic range), installed on the park main road stand (height 3.5m, coverage radius 50m); the millimeter wave radar is TIIWR6843 (detection distance 0-50m, ranging accuracy ±0.1m, point cloud output frequency 10Hz), deployed 1m to the side of the camera; the infrared sensor uses MLX90614 (temperature measurement range -20℃-80℃, sampling interval 1s) single-point temperature measurement sensor, suitable for single target temperature monitoring, embedded in the camera shell.

[0039] Preprocessing procedure: ① Image noise processing: When the light is sufficient (brightness > 500 lux) during the day, a Gaussian filter with σ = 1.2 is used to remove high-frequency noise, and a 3x3 window median filter is used to eliminate salt and pepper noise; At night, when the light is low (brightness < 100 lux), the median filter window is expanded to 5x5 to enhance the noise suppression effect. ② Radar point cloud processing: Based on the K-nearest neighbor algorithm (K = 5, distance threshold 0.3 m), outliers caused by metal guardrail reflection are removed, and personnel reflection point clouds (reflection intensity -20 dB to -10 dB) are retained. ③ Data synchronization: Through the NTP server, the time stamp is unified, and the error is controlled within ≤10 ms; The RGB image, radar point cloud, and infrared temperature data are normalized to the [0, 1] interval, and the temperature data is mapped as (measured temperature + 20) / 100 (e.g. 25℃ corresponds to 0.45).

[0040] (2) Initial detection of moving objects and candidate box generation Deployment of improved YOLOv8 algorithm: For children (height 1.0-1.4 m), adults (1.5-1.8 m), optimize anchor box, set 3 groups of anchor box: 10x14 (adapt to children's torso), 20x28 (adapt to adult upper body), 30x42 (adapt to adult full body); When the camera resolution is automatically reduced to 1280x720 due to backlight, the anchor box is reduced by 0.8 times (8x11.2, 16x22.4, 24x33.6). Radar clustering uses the DBSCAN algorithm (ε = 0.5 m, minimum point number 5) to obtain the 3D coordinates (x, y, z) of the personnel position, project it to the image coordinate system, calculate the IOU (threshold ≥ 0.6) with the image candidate box, and remove the duplicate box (such as the image box and radar projection box IOU = 0.85 of the same person, keep the image box with confidence 0.82). At the same time, Canny edge detection (threshold 100-200) is introduced to extract the personnel contour (such as arm swing, leg bending contour), and the candidate box position is corrected to make the candidate box and the actual contour overlap rate increase to 92% (traditional scheme only 75%), finally output the real-time number of personnel in the park (such as 15:00 detects 28 people) and the position (accuracy ± 0.3 m).

[0041] (3) Multi-dimensional feature extraction and behavior recognition Feature extraction: ① Spatio-temporal features: 3D convolution (3x3x3 kernel, stride 1) is used to extract dynamic features of 16 consecutive frames of images, combined with Farneback optical flow method to calculate the speed of personnel motion (such as walking speed 1.2 m / s, running speed 3.5 m / s); ② Posture features: MediaPipe model extracts 21 joint points (such as head, shoulder, knee joint), calculates the knee joint angle (60°-90° when walking, >120° when falling), and the shoulder-hip distance (shortened by 30% when falling); ③ Attribute features: color histogram (bins=64, extract clothing color distribution), LBP texture (radius 3, neighborhood 8, distinguish clothing material), radar reflection intensity (personnel clothing reflection intensity-18dB, distinguish from metal guardrail-5dB), concatenated into 512-dimensional feature vector.

[0042] Model training: a hybrid model of "CNN-LSTM-attention" is constructed, CNN contains 4 convolution blocks (2 3x3 convolution layers + 1 2x2 pooling layer for each block), LSTM hidden unit 256 (dropout=0.3), attention layer strengthens the feature weight of knee joint angle and motion speed. ImageNet pre-trained weights are used to initialize CNN, data augmentation (random horizontal flip, ±10% scaling, ±15% brightness adjustment), cross-entropy loss function (initial loss 0.8, reduced to 0.04 after 50 epochs of training), after knowledge distillation, the model size is compressed from 220MB to 45MB, and the inference speed reaches 22fps (edge device NVIDIA Jetson Nano). Feature weight adjustment: according to the formula W=(F×S) / λ, F=0.92 (Pearsor correlation coefficient) for the feature of knee joint angle and fall behavior, S=0.88 (intra-class / inter-class distance ratio), λ=512, W=(0.92×0.88) / 512≈0.0016, the feature weight is improved by 20%, which strengthens the fall behavior recognition.

[0043] (4) Recognition result optimization and abnormality determination Dynamic confidence threshold: indoor corridor scene 0.65, outdoor square (complex lighting) 0.75. When strong light is detected (brightness > 1200 lux, I = 0.15) and the person is blocked by trees (blockage area 35%, D = 0.1), the person behavior confidence C0 = 0.73, according to the formula C = C0 x (1 - I x D) = 0.73 x (1 - 0.15 x 0.1) = 0.73 x 0.985 = 0.72, outdoor threshold 0.75, C = 0.72 < 0.75, trigger multi-source verification - radar detects that the speed of the person drops from 1.2 m / s to 0.3 m / s (consistent with the characteristics of falling), infrared temperature is normal, and it is determined that it is "falling" (confidence correction is 0.81). Abnormal rule library: falling behavior marking conditions (speed drop > 80%, knee angle > 120°, 3 consecutive posture unchanged), illegal stay marking conditions (staying at the same location for more than 10 minutes, no moving track). When the "falling" anomaly is identified, trigger 2kHz buzzer + red light flashing, push the alarm to the security terminal (including person location coordinates, behavior screenshot), at the same time log storage to MySQL database (record timestamp, object ID = 2024051001, behavior "falling", confidence 0.81).

[0044] (5) Performance comparison data Table 1: Performance comparison of intelligent park security scene behavior recognition

[0045] Table 1 is based on one month of running data of park 1 (daily average 100,000 frames of image). The traditional scheme has a missed detection rate of 18.5% due to the mismatch of anchor frame and the size of children, which easily leads to the risk of children lost; the present application reduces the missed detection rate to 2.8% through 10x14 anchor frame and edge correction, ensuring that there is no blind area for child safety monitoring. In terms of falling behavior misjudgment rate, the traditional scheme only relies on image judgment, and misjudges "squatting to pick up things" as falling (misjudgment rate 12.3%); the present application combines radar speed drop characteristics and multi-source verification, and the misjudgment rate is only 1.5%, avoiding invalid alarm. In strong light scene, the accuracy of the traditional scheme is only 72.1% due to light interference, and the accuracy of the present application is improved to 93.6% by using confidence correction formula to offset the influence of light. The edge device reasoning speed is improved from 8fps to 22fps, meeting the real-time monitoring demand; the abnormal response time is shortened from 15 seconds to 3 seconds, which saves the golden time for rescue of falling person, fully embodies the adaptation advantage of the present application in outdoor complex scene.

[0046] Example 2: Industrial workshop inspection scene (indoor environment, containing 4 behaviors of equipment moving, part assembling, illegal crossing guardrails, not wearing safety helmet, equipment density 15 / 100 square meters) 1. Detailed implementation of technical scheme (1) Multi-source data acquisition and preprocessing Data acquisition equipment is adapted to industrial environments: The high-definition camera used is the Dahua DH-IPC-HFW5449E (1920×1080 resolution, 25fps, dustproof and waterproof IP66), installed on a workshop pillar (4m high, covering the equipment control panel); the millimeter-wave radar is the ZTE Micro ZWM6843 (detection distance 0-30m, anti-electromagnetic interference), deployed in the aisle between equipment; the infrared sensor uses the AMG8833 (8×8 pixel array, temperature measurement accuracy ±1℃). In complex industrial scenarios, the AMG8833 8×8 infrared array sensor is used to capture regional temperature distribution characteristics, supporting thermal behavior analysis of equipment and human interaction, and monitoring equipment surface temperature and personnel body temperature. Preprocessing optimization: ① Image noise processing: For workshop dust (concentration 0.8mg / m³)... 3 ① Gaussian filtering σ is adjusted to 1.5, and median filtering window is 4×4 to ensure clear outlines of equipment parts (such as M8 bolts); ② Radar point cloud processing: outlier removal K=5, distance threshold 0.3m, retaining point clouds of metal equipment (reflection intensity -5dB to 0dB) and personnel (-20dB to -10dB) to distinguish between equipment and personnel; ③ Data synchronization: timestamp synchronization is achieved through industrial Ethernet (Profinet protocol), with an error ≤8ms; equipment temperature data is mapped according to (measured temperature -20) / 80 (e.g., 60℃ corresponds to 0.5), and is uniformly normalized with other data.

[0047] (2) Initial detection of active objects and generation of candidate boxes Improved YOLOv8 adaptation to industrial objects: Anchor frames of 8×10 and 12×16 are set for small parts (e.g., bearings, 20×15mm), and anchor frames of 60×40 and 80×50 are set for large equipment (e.g., motors, 500×300mm). When the camera resolution drops to 1024×768 due to equipment obstruction, the anchor frames are scaled by 0.7 times. Radar clustering and candidate box correction: The DBSCAN algorithm with ε=0.4m (narrow workshop aisles) and a minimum of 4 points obtains the 3D coordinates of equipment and personnel. Canny edge detection (threshold 100-200) extracts detailed contours of equipment gears, bolts, etc., and corrects image candidate boxes (e.g., the original coverage area deviation of the motor candidate box was 15%, and the correction deviation is ≤5%), achieving a 94% overlap rate between candidate boxes and actual contours. The IOU matching threshold is 0.55 (dense workshop objects), eliminating duplicate boxes (e.g., candidate boxes in areas where personnel and equipment overlap), and outputting the real-time positions of equipment (15 units) and personnel (8 people).

[0048] (3) Multi-dimensional feature extraction and behavior recognition Feature extraction focuses on industrial attributes: ① spatio-temporal features: 3D convolution extracts dynamic features of continuous 12-frame equipment handling (such as the trajectory of a robotic arm picking up a part), and optical flow calculates the handling speed (normal 0.5 m / s, violation 1.2 m / s); ② posture features: MediaPipe extracts 21 joint points of personnel, calculates safety helmet wearing judgment (head key point distance deviation > 20 mm is judged as not wearing), and hip height when crossing the guardrail (more than 1.2 times the height of the guardrail is a violation); ③ attribute features: radar reflection intensity distinguishes between metal parts (-8 dB) and non-metal tools (-15 dB), and LBP texture extracts equipment surface wear features (wear area texture variance > 50), which are spliced into 512-dimensional feature vectors.

[0049] Model training and updating: Transfer learning (ImageNet pre-trained CNN) is used, the training set contains industrial behavior samples (8000 frames per class, such as equipment handling samples containing 5 types of parts), batchsize=32, epoch=40, cross-entropy loss is reduced to 0.035; incremental update is performed every 7 days: extract the logs of the past 7 days (1200 frames of low confidence samples, add to the training set after manual annotation), fix the CNN layer parameters, update the LSTM and attention layer weights, and save the new model when the accuracy improves by ≥3% (such as the recognition rate of not wearing a safety helmet increasing from 88% to 92%).

[0050] (4) Recognition result optimization and abnormality judgment Dynamic threshold and abnormality rules: the confidence threshold for indoor scenes in the workshop is 0.7, and the threshold for equipment-intensive areas (such as assembly tables) is 0.75. Violation behavior rules: ① handling behavior (part volume change ≥30% is judged as a violation of overload); ② crossing the guardrail (hip height exceeds 1.2 m and leg angle > 70°); ③ not wearing a safety helmet (head joint point distance deviation > 20 mm and continuous 5 frames unchanged). Confidence correction: when the equipment blocks the person (blocking area 45%, D=0.2), the confidence of a person not wearing a safety helmet C0=0.68, according to the formula C=C0×(1-I×D)=0.68×(1-0×0.2)=0.68<0.7, infrared verification is triggered (head temperature without safety helmet insulation layer features, temperature fluctuation ±2℃), the confidence is corrected to 0.76, and the violation is judged, and the alarm is pushed to the workshop management terminal (including the violation of personnel position, behavior video segment).

[0051] (5) Performance comparison data Table 2: Performance comparison of behavior recognition in industrial workshop inspection scenarios

[0052] Table 2 is based on two months of workshop operation data (500,000 frames of images per day, including 20 times of equipment moving, and 5 times of violation behavior). The traditional scheme has a detection accuracy of only 75.3% due to the mismatch of small part anchor frames, which easily leads to part omission assembly; the application distinguishes the 8x10 anchor frame from the radar reflection intensity, and the accuracy is increased to 95.8%, ensuring part monitoring throughout the whole process. In terms of recognition rate of workers without safety helmets, the traditional scheme only relies on image color judgment (high misjudgment rate), and the application combines the distance of the joint node and the infrared temperature characteristics, with a recognition rate of 94.2%, reducing the risk of accidents in the workshop. The detection rate of equipment moving violation is increased from 68.9% to 92.5%, and through the volume change rule and multi-source verification, the damage to the equipment caused by overloading moving is timely stopped. The precision of the model after incremental update is increased by 4.5%, which is much higher than the 1.2% of the traditional scheme, and it is suitable for the scene of equipment update and change of personnel operation habits. The rate of equipment failure correlation identification (such as gear wear accompanied by temperature rise) is increased from 55.7% to 88.3%, realizing the linkage of "behavior identification-fault warning", reducing the loss of workshop shutdown (single shutdown loss exceeds 50,000 yuan), and fully embodying the practicality and reliability of the application in industrial scenes.

[0053] Reference Figure 3 The figure directly shows the advantages of multi-source data fusion of the application. The traditional scheme only relies on the RGB camera, which is significantly affected by the environment: at night in low light, the lack of light causes the loss of image details, and the accuracy drops to 70%; in the industrial dust scene, dust shielding the lens causes noise interference, and the accuracy is only 65%, which cannot meet the monitoring needs of complex scenes. The application realizes multi-source cooperation through "camera + radar + infrared", the radar provides depth information to make up for the lack of light, the infrared captures temperature characteristics to distinguish objects and backgrounds, and the radar point cloud is not affected by shielding in the dust scene, and finally the accuracy of various scenes is stable above 90%, the accuracy in night low light and dust scene is increased by 24% and 25% compared with the traditional scheme, completely solving the environmental adaptability short board of single data source, and ensuring the reliability of detection in complex scenes.

[0054] Reference Figure 4The figure verifies the practicability of the model lightweight of the application. The volume of the traditional model reaches 220MB, and the performance is limited on the edge device: the inference speed of the ARM embedded device (JetsonNano) is only 6fps, which cannot meet the real-time monitoring requirement of 25fps; the speed of the edge computing box is 8fps, and there is obvious delay. The model volume is compressed to 45MB, which is only 20% of the traditional model, through knowledge distillation (teacher model guiding student model), model pruning (removing 30% redundant convolution kernels) and quantization (32-bit to 8-bit). At the same time, the inference speed is greatly improved: the ARM device reaches 22fps, the edge box reaches 20fps, which can meet the real-time processing requirement; the speed of the x86 server and the cloud server reaches 30fps and 50fps respectively, which supports multi-channel concurrent monitoring. The characteristics of "small volume + high speed" make the model adaptable to the whole scene deployment from the edge to the cloud, and solve the problem of "strong hardware dependence and limited deployment" of the traditional model.

[0055] Referring to Figure 5 The figure reflects the accuracy of the multi-source verification and dynamic rules of the application. The traditional scheme only relies on single data to determine the exception: the image color recognition is used to determine whether the safety helmet is worn, and the misjudgment rate is 20% (dark hair is easily misjudged as a safety helmet); the image size is used to determine whether the load is overloaded, and the misjudgment rate is 25% (ignoring the density difference of parts), and a large number of invalid alarms increase the operation and maintenance burden. The application adopts multi-source data cross verification: the falling behavior is combined with the radar speed drop (≤0.3m / s) and the posture joint angle (>120°), and the misjudgment rate is reduced to 1%; the head joint point distance (deviation >20mm) and the infrared temperature feature (temperature fluctuation without thermal insulation layer) are used to verify whether the safety helmet is worn, and the misjudgment rate is 2%; the radar reflection intensity (metal part density feature) and the volume change (>30%) are combined to determine whether the load is overloaded, and the misjudgment rate is 3%. The misjudgment rate of each type of exception is controlled within 5%, which significantly improves the reliability of abnormal identification and reduces the interference of invalid alarms on operation and maintenance.

[0056] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the application within the technical range disclosed by the application, which should be covered within the protection scope of the application.

Claims

1. A data processing method for realizing moving object detection and behavior recognition, characterized in that, The method comprises the following steps: Step one: multi-source data acquisition and preprocessing, collecting the data of the moving object in the monitoring area, the data sources including industrial cameras, millimeter wave radars and infrared sensors, preprocessing the collected data, removing image noise by using Gaussian filtering, eliminating salt and pepper noise by using median filtering, removing outliers from the radar point cloud data, and synchronizing the time stamp; Step two: initial detection of the moving object and generation of the candidate box, detecting the RGB image by using the improved YOLOv8 algorithm, and optimizing the anchor box of the small size object; Step three: multi-dimensional feature extraction, extracting the space-time, posture and attribute features of the object region corresponding to the effective candidate box, and splicing the three types of features into a feature vector after standardization; Step four: behavior recognition model training and reasoning, constructing a CNN-LSTM-attention hybrid model for behavior recognition, the CNN layer reduces the dimension of the feature vector, the LSTM layer captures the time sequence correlation of the behavior, and the attention layer strengthens the weight of the key behavior features; During model training, the cross-entropy loss function is used, and data enhancement is combined to improve the generalization ability; during reasoning, the real-time extracted feature vector is input into the trained model, and the behavior category and the corresponding confidence are output; Step five: identification result optimization and abnormal behavior determination, setting a dynamic confidence threshold, calling multi-source data fusion verification when the behavior confidence is lower than the threshold, defining an abnormal rule library, matching the reasoning result with the rule library, and marking the abnormal behavior if the abnormal rule is met; Step six: result output and feedback, visualizing the object information to the monitoring terminal, triggering sound and light alarm for abnormal behavior, and storing the behavior recognition log in the local database. The method further comprises a feature weight dynamic adjustment step, which is executed after the multi-dimensional feature extraction step and before the behavior recognition model training and reasoning step; the specific process is as follows: calculating the contribution weight of each type of feature, the formula is W=(F×S) / λ, wherein W is the feature weight value, F is the correlation coefficient of the feature and the behavior category, S is the discrimination degree of the feature, and λ is the feature dimension normalization coefficient; then, the contribution degree of the three types of features in the feature vector is weighted and adjusted according to the calculated feature weight value W; the adjusted feature vector with weight is directly used as the input data of the CNN layer in the behavior recognition model training and reasoning step.

2. The data processing method for realizing moving object detection and behavior recognition according to claim 1, characterized in that, The initial detection of the moving object in step two further comprises candidate box size optimization, adjusting the anchor box proportion for different resolution images, reducing the original anchor box size by 0.8 times when the image resolution is lower than 1280*720, and enlarging it by 1.2 times when the resolution is higher than 2560*1440; at the same time, edge detection is introduced to assist in identifying the contour features of the object and correcting the edge of the candidate box.

3. The data processing method for realizing moving object detection and behavior recognition according to claim 1, characterized in that, ​ 4. The data processing method for realizing moving object detection and behavior recognition according to claim 1, characterized in that, The result optimization in step five further includes behavior confidence environment correction, which corrects the initial confidence when the monitoring scene has light mutation or shielding, and the correction method is C=C0*(1-I*D), wherein C is the corrected confidence, C0 is the initial output confidence of the model, I is a light interference coefficient, and D is a shielding influence coefficient.

5. The data processing method for realizing moving object detection and behavior recognition according to claim 1, characterized in that, The behavior recognition model training in step four further adopts a transfer learning strategy, initializes the CNN layer parameters using the pre-trained ImageNet dataset, and then fine-tunes the behavior dataset of the application. During fine-tuning, a progressive learning rate is adopted, and knowledge distillation is introduced to improve the model inference speed to more than 20 fps while ensuring recognition accuracy.

6. The data processing method for realizing moving object detection and behavior recognition according to claim 1, characterized in that, Step two further includes multi-source data space-time alignment optimization. The time stamp interpolation method is used to process data collection delays. When the camera and radar data time stamp deviation is greater than 10 ms, the lagging data is linearly interpolated. In terms of spatial alignment, the mapping relationship between image pixel coordinates and radar three-dimensional coordinates is established through a calibration board, the radar point cloud is projected into the image coordinate system, the IOU of the projection point and the image candidate box is calculated, and IOU≥0.5 is determined as the same object.

7. The data processing method for realizing moving object detection and behavior recognition according to claim 1, characterized in that, The step of online model updating is triggered periodically, and the period is synchronized with the log storage period in the result output and feedback step. The specific process is as follows: the behavior recognition logs of the past 7 days are extracted from the local database at regular intervals, and low-confidence samples with a confidence of less than 0.6 are screened. The sample is derived from the inference results output by the behavior recognition model training and inference steps and the logs stored in the result output and feedback step. After manual annotation, the training set is added, and the model is iteratively optimized using incremental training. After each update, the accuracy is verified by the test set. When the accuracy improves by more than 3%, the new model replaces the original inference model, and the new model is immediately used in the inference process of the subsequent behavior recognition model training and inference steps. At the same time, the model update log is recorded and stored in the local database of the result output and feedback step.

8. A system for implementing the data processing method of claim 1-7 for detecting moving objects and recognizing behavior, characterized in that, The following modules are included: A multi-source data acquisition and preprocessing module for acquiring and processing active object data. The module includes a camera unit, a millimeter wave radar unit, an infrared sensor unit, and a preprocessing unit. The preprocessing unit integrates Gaussian filtering, median filtering, point cloud outlier rejection, and data normalization functions, and synchronizes the data format and time stamp. An active object detection module for generating active object candidate boxes, including an image detection unit, a radar clustering unit, and a candidate box matching unit. The candidate box matching unit calculates the IOU of the image detection box and the radar clustering result, removes duplicate boxes, and retains valid boxes with a confidence of greater than or equal to 0.7, and outputs the object preliminary position and quantity. A multi-dimensional feature extraction module for extracting object features, including a space-time feature extraction unit, a posture feature extraction unit, an attribute feature extraction unit, and a feature splicing unit. The feature splicing unit splices the three types of features into a 512-dimensional feature vector after standardization, and outputs to the behavior recognition module. A behavior recognition module for outputting behavior categories and confidence, including a model training unit, a model inference unit, and a model updating unit. The model training unit supports transfer learning and knowledge distillation. The recognition result optimization module comprises a dynamic threshold unit, a multi-source fusion verification unit, an abnormal rule library unit and a confidence correction unit; the abnormal rule library unit supports user-defined rules; The result output and storage module comprises a visual terminal unit, an alarm unit and a log storage unit; the log storage unit automatically retains log data for 3 months and automatically archives data exceeding the period to the cloud.

9. The system for implementing the data processing method of active object detection and behavior recognition according to claim 8, characterized in that, The multi-source data acquisition and preprocessing module further comprises a data quality detection unit, which monitors the raw data collected by the camera unit, the millimeter wave radar unit and the infrared sensor unit and the output data processed by the preprocessing unit in the multi-source data acquisition and preprocessing module in real time, and the detection indexes include the integrity, the definition and the validity of the temperature data of the data; when the data quality is substandard, the self-checking of the corresponding acquisition device in the multi-source data acquisition and preprocessing module is triggered first; if the self-checking is still substandard, the device fault alarm is pushed to the monitoring terminal of the result output and storage module, and the output of the substandard data to the active object detection module is suspended at the same time until the data quality is qualified.

10. The system for implementing the data processing method of active object detection and behavior recognition according to claim 8, characterized in that, The behavior recognition module further comprises a model lightening unit, which is integrated in the behavior recognition module, performs lightening processing after the model training unit completes the model training and before the model inference unit deploys the model; the model pruning, quantization and layer fusion technologies are adopted to compress the model volume while keeping the inference accuracy loss; the model lightening unit is adapted to different hardware platforms to generate corresponding lightened model versions.

Images