Illegal sand excavation behavior identification method and system

Abstract

The invention relates to the technical field of intelligent monitoring, and provides an illegal sand excavation behavior identification method and system, and the method comprises the steps: collecting image data of a sand excavation region; sand excavation equipment is selected from the image data, and position information of the sand excavation equipment is extracted; for the sand excavation equipment in the sand excavation state, generating a movement track; calculating the speed and the acceleration of the sand excavation equipment based on the motion trail, and constructing a space-time diagram based on the motion trail, the speed and the acceleration of the sand excavation equipment; extracting spatiotemporal features of the sand excavation equipment based on the spatiotemporal diagram; time sequence action features of the sand excavation equipment are extracted from the image data of the sand excavation areas at the multiple different visual angles, and the time sequence action features are fused to obtain fused action features; and on the basis of the fused action features and the spatial-temporal features, illegal sand excavation behaviors of the sand excavation equipment are judged. According to the method, comprehensive real-time monitoring of the sand excavation area can be realized, and illegal sand excavation behaviors can be accurately identified.

Description

Technical Field

[0001] This disclosure belongs to the field of intelligent monitoring technology, and in particular relates to a method and system for identifying illegal sand mining activities. Background Technology

[0002] Sand mining technology is needed in projects such as river management and infrastructure construction. However, illegal sand mining technology can damage the riverbed structure and affect construction safety. Therefore, it is necessary to use illegal sand mining monitoring technology to monitor sand mining activities. In the existing monitoring technology for illegal sand mining, the main reliance is on manual patrols or simple video surveillance. However, manual patrols require a lot of manpower and material resources, resulting in high costs. Furthermore, they are limited by the working time and energy of the patrol personnel, and there is a possibility of human error. Simple video surveillance, due to its hardware only recording images and its software lacking dynamic analysis algorithms, can only make judgments based on static images. It lacks comprehensive analysis of the spatiotemporal dimensions of sand mining equipment, making it difficult to capture the dynamic movement characteristics of the equipment, resulting in the difficulty in identifying illegal behavior in a short period of time. Summary of the Invention

[0003] To address the aforementioned issues, this disclosure provides a method and system for identifying illegal sand mining activities. Employing multi-view data acquisition and multi-algorithm fusion analysis techniques, it enables comprehensive real-time monitoring of sand mining areas, accurately identifies illegal sand mining activities, and provides timely warnings. This effectively overcomes the shortcomings of existing technologies that rely on manual patrols and simple video surveillance, improves the real-time performance, intelligence, and accuracy of illegal sand mining monitoring, and meets the needs of effective supervision in complex sand mining scenarios.

[0004] The following are the technical details of this disclosure: A method for identifying illegal sand mining activities includes: Collect image data of the sand mining area from multiple different perspectives; The target detection algorithm is used to select sand mining equipment from image data and extract the location information of the sand mining equipment; Extract the motion characteristics of the sand mining equipment, and locate the sand mining equipment in the sand mining state based on the motion characteristics; For sand mining equipment in the sand mining state, a motion trajectory is generated based on its position information; the velocity and acceleration of the sand mining equipment are calculated based on the motion trajectory; a spatiotemporal map is constructed based on the motion trajectory, velocity, and acceleration of the sand mining equipment; and the spatiotemporal features of the sand mining equipment are extracted based on the spatiotemporal map. Temporal motion features of sand mining equipment are extracted from image data of sand mining areas from multiple different perspectives, and the temporal motion features are fused to obtain fused motion features. Based on the fusion of action features and spatiotemporal features, illegal sand mining activities of sand mining equipment can be identified.

[0005] Furthermore, The acquisition of image data of the sand mining area from multiple different perspectives includes: Preliminary image and video data of sand mining areas from multiple different perspectives were collected, and timestamps and GPS location information were added to the data; a high-frequency acquisition mode was used for active sand mining areas, and a low-frequency acquisition mode was used for stable areas. Multi-source data fusion processing was performed on the preliminary image data and preliminary video data to obtain image data of the sand mining area from multiple different perspectives.

[0006] Furthermore, The step of selecting sand mining equipment from image data includes: Multi-scale feature extraction is performed on image data using a feature pyramid network to generate feature maps at different scales. Based on feature maps of different scales, target detection algorithms are used to extract the location information, category information, and detection confidence of sand mining equipment. The location and category information of sand mining equipment with a detection confidence level higher than a preset threshold are selected as the detection results.

[0007] Furthermore, The step of extracting the motion characteristics of the sand mining equipment and locating the sand mining equipment in the sand mining state based on the motion characteristics includes: Based on the detection results of the sand mining equipment, a 3D convolutional neural network is used to extract temporal features from the video frame sequence of the sand mining equipment to obtain a feature map; and based on the feature map, it is determined whether the construction vehicle is in the sand mining state.

[0008] Furthermore, The construction of a spatiotemporal map based on the motion trajectory, velocity, and acceleration of the sand mining equipment; and the extraction of spatiotemporal features of the sand mining equipment based on the spatiotemporal map; include: The location of the sand mining equipment at different time points is obtained based on its movement trajectory; A spatiotemporal graph is constructed using the position, velocity, and acceleration of the sand mining equipment at different time points as node features and the association between adjacent time steps as edges; the weight of the edges is calculated based on the time interval, position distance, and velocity consistency. By inputting the spatiotemporal graph into a spatiotemporal graph convolutional network, spatiotemporal features representing the device's motion trajectory, velocity changes, and acceleration patterns are obtained.

[0009] Furthermore, The process involves extracting temporal motion features of sand mining equipment from image data of sand mining areas from multiple different perspectives, and fusing these temporal motion features to obtain fused motion features; including: Long Short-Term Memory (LSTM) networks were used to extract temporal action features from video data from multiple different perspectives. Different weight coefficients are assigned to the action features from different perspectives. The action features from different perspectives are then weighted using these weight coefficients to obtain the fused features.

[0010] Furthermore, The method of determining illegal sand mining behavior of sand mining equipment based on fused action features and spatiotemporal features includes: A feature distribution model is constructed, which is trained based on the fused action features and spatiotemporal features of historical normal behavior to output the probability density of the current behavior. The fused action features and spatiotemporal features to be detected are input into the feature distribution model to obtain the probability density; If the probability density is lower than a preset threshold, the sand mining equipment is deemed to be behaving abnormally.

[0011] A system for identifying illegal sand mining activities, comprising: The data acquisition module is used to collect image data of the sand mining area from multiple different perspectives; The target detection module is used to select sand mining equipment from image data using target detection algorithms and extract the location information of the sand mining equipment. The behavior judgment module is used to extract the action characteristics of the sand mining equipment and find the sand mining equipment in the sand mining state based on the action characteristics; The spatiotemporal feature extraction module is used to generate a motion trajectory for sand mining equipment in the sand mining state based on its location information; calculate the velocity and acceleration of the sand mining equipment based on the motion trajectory; construct a spatiotemporal map based on the motion trajectory, velocity, and acceleration of the sand mining equipment; and extract the spatiotemporal features of the sand mining equipment based on the spatiotemporal map. The feature fusion module is used to extract the temporal action features of sand mining equipment from image data of sand mining areas from multiple different perspectives, and to fuse the temporal action features to obtain fused action features. The identification module is used to determine illegal sand mining behavior of sand mining equipment based on fused action features and spatiotemporal features.

[0012] Furthermore, The behavior determination module includes: The temporal feature extraction unit is used to receive the detection results of the sand mining equipment output by the target detection module, and use a 3D convolutional neural network to extract temporal features from the video frame sequence of the sand mining equipment to obtain a feature map; The sand mining behavior determination unit is used to determine whether the sand mining equipment is in a sand mining state based on the feature map. The behavior analysis result output unit is used to output the analysis results of whether the sand mining equipment is in a certain state.

[0013] Furthermore, The identification module includes: The feature distribution modeling unit is used to probabilistically model the action characteristics of the sand mining equipment using a Gaussian mixture model, so as to output the probability density of the current behavior; the probability density function of the Gaussian mixture model is:

[0014] in, The number of Gaussian distributions, For the first The weights are distributed in a Gaussian manner, satisfying... , For the first The probability density function of a Gaussian distribution has a mean of . The covariance matrix is ; An anomaly determination unit is used to determine that the sand mining equipment is behaving abnormally if the probability density of the current behavior is lower than a preset threshold.

[0015] Compared with the prior art, this disclosure has the following advantages: This disclosure comprehensively captures image and video information of the sand mining area through multi-view data acquisition, which can prevent the monitoring blind spot problem caused by single viewpoint obstruction and changes in lighting. This study utilizes target detection algorithms to extract the location information of sand mining equipment from multi-view image data, achieving precise spatial positioning of the equipment (spatial dimension foundation). Based on continuous frame location information, motion trajectories are generated, and velocity and acceleration are calculated, transforming the spatial position changes of the equipment at different time points into dynamic temporal features (temporal dimension modeling). A spatiotemporal graph is constructed using motion trajectories, velocities, and accelerations, encoding the spatial position and temporal motion states as nodes and edges in a graph structure, explicitly capturing the spatiotemporal dependencies of equipment motion (such as position change patterns at different times, periodicity of actions, etc., spatiotemporal correlation construction). Temporal motion features are extracted from multi-view images and fused, encompassing motion information from different spatial perspectives and processing temporal changes through a temporal model (spatiotemporal feature cross-fusion). Finally, based on the fused spatiotemporal features, illegal sand mining behavior is determined, achieving a comprehensive analysis of the equipment's spatial position, motion trajectory, and dynamic behavior evolution over time.

[0016] Other features and advantages of this disclosure will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the method of this disclosure is shown; Figure 2 A schematic diagram of the system disclosed herein is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] like Figure 1 This disclosure provides a method for identifying illegal sand mining activities based on image processing technology. Through a series of steps, it achieves real-time monitoring, behavior recognition, and anomaly detection and early warning of sand mining equipment. It is applicable to monitoring and evidence collection of illegal sand mining activities in various sand mining scenarios. The following are specific implementation methods of this disclosure: S1. Collect image data of the sand mining area from multiple different perspectives; In practical applications, the system first needs to collect real-time image data of the sand mining area using equipment such as drones, fixed cameras, and satellite imaging. Timestamps and GPS location information are embedded in the collected data to ensure accurate spatial and temporal correlation of each frame.

[0021] S2. Inspect the type and location of construction vehicles and sand mining equipment; Through the target detection module, the system performs real-time analysis of the collected image data, automatically identifies construction vehicles and sand mining equipment, and pinpoints their specific locations. The system can identify various types of sand mining equipment and classify and label their categories and locations. This step ensures that the system can comprehensively monitor all types of equipment in the sand mining area, not only locating the equipment accurately but also providing basic target information for subsequent behavior analysis.

[0022] S3. Analyze the time-series video data of construction vehicles, extract their dynamic features, and find the sand mining equipment in the sand mining state based on the action features. After determining the location and type of sand mining equipment, the system performs time-series analysis on the video sequence data of the construction vehicles. Through time-series feature extraction, the system can identify the dynamic motion characteristics of the equipment, such as changes in position and movement at different time points, repeated lifting of the bucket, and loading and unloading operations of the loader. For specific sand mining scenarios, this module can effectively extract the motion characteristics of the construction vehicles, distinguishing between sand mining operations and non-sand mining actions, providing dynamic feature basis for subsequent behavior judgment.

[0023] S4. For sand mining equipment in the sand mining state, generate a motion trajectory based on its position information; calculate the speed and acceleration of the sand mining equipment based on the motion trajectory; After extracting the dynamic characteristics of the construction vehicles, the system further analyzes their movement trajectories. Through time-series data, the system can track the movement of the sand mining equipment within the mining area and calculate its speed and acceleration. This analysis not only helps determine the vehicle's movement trend but also identifies whether the vehicle lingers in a specific area for an extended period or performs repeated sand mining operations. Trajectory analysis provides crucial dynamic information for the system to identify illegal sand mining activities, enabling the system to make more intelligent judgments about the actions of the sand mining equipment. S5. Weighted fusion of multi-view video data improves the accuracy of sand mining behavior judgment. The system fuses video data from different angles, assigning weights to the time-series features of each perspective to generate comprehensive feature data containing multi-view information. By weighted fusion of multi-view features, the system can confirm the behavior of sand mining equipment from multiple angles, avoiding recognition errors caused by occlusion or misjudgment from a single perspective. This multi-view data fusion process improves the accuracy of sand mining behavior judgment, enabling the system to effectively identify sand mining behavior even in complex scenarios. S6. Perform anomaly detection for sand mining activities. If anomalies are detected, trigger an alarm and store the video and image data of the sand mining event as evidence. After completing multi-view data fusion, the system performs anomaly detection on the characteristics of sand mining activities to determine whether illegal sand mining exists. The system compares the extracted features with normal sand mining behavior patterns. If the action features are found to deviate significantly from the normal range, an alarm is automatically triggered. The alarm information includes data such as the time, location, and type of the abnormal behavior, and the image data at the time of the alarm is stored as evidence. This step enables real-time early warning of illegal sand mining activities, providing the regulatory platform with automated monitoring and alarm functions, ensuring that the system can continuously monitor and accurately identify sand mining areas.

[0024] The illegal sand mining behavior identification method disclosed herein can achieve comprehensive monitoring of construction vehicles and sand mining equipment in sand mining scenarios. Through feature extraction in time and space dimensions, multi-view data fusion and anomaly detection and early warning, it can ultimately achieve real-time identification and rapid alarm of illegal sand mining behavior, meeting the actual needs of illegal sand mining behavior control.

[0025] like Figure 2 The diagram shows the illegal sand mining identification system provided in this disclosure. Through the collaborative work of multiple modules, it achieves real-time monitoring of sand mining areas and accurate identification of sand mining activities. It can effectively detect illegal sand mining activities and trigger alarms, and provides evidence storage functionality for image data. The system mainly includes a data acquisition module, a target detection module, a behavior analysis module, a spatiotemporal analysis module, a multi-view fusion module, and an anomaly detection and early warning module. The following details each module of this disclosed system.

[0026] The system disclosed herein includes: 1. Data Acquisition Module: The data acquisition module acquires real-time image data of the sand mining area and adds a timestamp and GPS location information to each frame to ensure accurate analysis in both time and space for subsequent modules. This module provides precise data input for target detection and behavior analysis through high-frequency data acquisition and achieves comprehensive data correlation by combining location markers.

[0027] In this embodiment, the specific implementation of the data acquisition module includes: 1. Data Acquisition: The data acquisition module uses drones, fixed cameras, and satellite remote sensing equipment as the main acquisition tools to cover real-time image data of the sand mining area. Drones have high mobility and can flexibly acquire images from specific angles, fixed cameras are used for real-time continuous monitoring, and satellite remote sensing equipment can provide high-definition images with wide-area coverage.

[0028] 2. Timestamp and GPS Positioning: During data acquisition, to ensure accurate temporal and spatial correlation of the acquired images and videos, each frame of data is embedded with timestamps and GPS positioning information. The timestamp records the specific time of data acquisition, ensuring that subsequent behavior analysis modules can perform dynamic analysis on a precise time series; the GPS positioning information marks the geographical location of the sand mining equipment, achieving spatial positioning. Timestamps and GPS positioning information are generated in real time at the acquisition device and appended to the metadata of each frame of data to ensure temporal and spatial accuracy.

[0029] Results: By using timestamps and GPS information, the system can accurately locate vehicles and equipment within the sand mining area and analyze the dynamic behavior of the equipment using multi-frame time-series data. This method not only intelligently identifies the type of construction vehicle but also combines before-and-after video analysis to achieve more accurate sand mining behavior recognition.

[0030] 3. Multi-frequency data acquisition: The data acquisition module can dynamically adjust the acquisition frequency according to the conditions of the sand mining area. Specifically, for active sand mining areas, the module can be set to a high-frequency acquisition mode to ensure sufficient data density to accurately capture the dynamic changes of the sand mining equipment; for stable areas, a lower-frequency acquisition mode can be used to reduce the data processing pressure. The frequency adjustment is automatically controlled by the system, combined with feedback information from image processing algorithms, to dynamically adapt to the actual situation of the sand mining area.

[0031] 4. Data Preprocessing and Noise Reduction: During data acquisition, environmental factors such as weather and lighting may affect image quality. Therefore, the data acquisition module integrates a preliminary preprocessing unit. This unit uses image processing algorithms to perform preliminary noise reduction and image enhancement, improving image contrast and clarity. This ensures that the target detection module can accurately identify devices on high-quality images. Specific processing methods include Gaussian filtering and histogram equalization to remove noise and improve image contrast.

[0032] 5. Coordinate Mapping and Area Calibration: While acquiring image data, the data acquisition module maps the sand mining area using GPS coordinates to form a geographic calibration. Based on the coordinate data, the module can dynamically generate a geographic framework of the sand mining area, ensuring that the system can continuously monitor specific areas. This function is particularly suitable for monitoring needs in wide-area sand mining areas, guaranteeing the geographic consistency of system data.

[0033] 6. Multi-source data fusion: The data acquisition module fuses data from drones, fixed cameras, and satellites to ensure multi-view coverage of the collected data. Data from different acquisition devices are aligned using timestamps and GPS information, and synthesized into a unified spatiotemporal data stream, ensuring the system's comprehensive monitoring capabilities in complex environments.

[0034] The data acquisition module ensures the temporal and spatial accuracy of the data while supporting multi-source, multi-frequency acquisition and automated preprocessing. The high-quality data acquired provides an accurate data foundation for target detection and behavior analysis, contributing to the stable operation of subsequent modules and the accurate identification of illegal sand mining activities.

[0035] Effect: This publicly available system collects real-time image data from multiple angles using drones, fixed cameras, and satellite equipment. Compared to existing simple video surveillance methods with a single perspective, it can comprehensively cover illegal sand mining areas. Furthermore, weighted fusion of multi-view data improves the accuracy of sand mining behavior detection. Multi-view fusion reduces identification bias caused by occlusion from a single perspective, achieving a more comprehensive real-time monitoring effect.

[0036] 2. Target Detection Module: The target detection module is used to identify and locate sand mining equipment from the acquired image data. This module analyzes the image data provided by the data acquisition module, using deep learning algorithms to identify the specific location and bounding box information of the sand mining equipment, and extracting information such as the equipment's category and location. This provides accurate input data for subsequent behavior analysis and spatiotemporal analysis modules. The target detection module employs a multi-scale feature extraction method to ensure high-precision identification even in complex environments.

[0037] In this embodiment, the specific implementation of the target detection module includes: 1. Multi-scale Feature Extraction Unit: This unit extracts feature maps at different scales based on a Feature Pyramid Network (FPN) to meet the detection requirements of construction vehicles at different distances and resolutions. Through the multi-layered structure of the FPN, the target detection module extracts feature maps of different sizes from the generated feature pyramid, ensuring high detection accuracy even when the sand mining equipment is far away or small in scale. Multi-scale feature extraction adapts to the complex environment and varying target scales in the sand mining area, ensuring the system's robustness under different sand mining conditions.

[0038] 2. Target Detection Algorithm: The target detection module uses the YOLOv5 algorithm to detect sand mining equipment. YOLOv5 is a single-stage detection algorithm based on a deep convolutional neural network, processing target detection tasks in images in an end-to-end manner. Its efficient detection structure can process image data in real time and quickly generate bounding boxes and category information for the equipment. The YOLOv5 network structure contains multiple convolutional layers used to extract features of the sand mining equipment, and classifies and locates the detection area using the feature maps output by the convolutional layers. Let the input image size be... The feature pyramid contains the following scales: Each scale The extracted feature map size is .

[0039] 3. Loss Function Optimization: To ensure detection accuracy, the target detection module is optimized using a comprehensive loss function. The loss function is designed as follows:

[0040] in: This represents the classification loss, used to classify detected vehicles and equipment to distinguish sand mining equipment from other equipment. It represents the bounding box loss, which measures the deviation of the center position and size of the predicted bounding box from the true bounding box, ensuring the accuracy of the device position; This represents the IoU (Intersection over Union) loss, used to calculate the overlap between the predicted bounding box and the ground truth bounding box, measuring the accuracy and completeness of the detection.

[0041] By adjusting the weight coefficients in the loss function , and This allows for optimization of the detection module's performance under different detection conditions.

[0042] 4. Detection Result Output Unit: After completing the target detection, this unit outputs the detection results of the sand mining equipment, including the location information, category information, and detection confidence level of each piece of equipment. The detection results include the bounding box coordinates of the equipment. ,in The coordinates of the bounding box center are For width, This information will be transmitted to the behavior analysis module for further dynamic behavior analysis and judgment.

[0043] 5. Low Confidence Filtering: To improve the accuracy of detection results, the target detection module sets a confidence threshold to filter out low-confidence detection results. Only when the detection confidence level is higher than the preset threshold will the device's detection result be passed to subsequent modules. This unit filters based on the confidence score output by YOLOv5, avoiding false alarms caused by low confidence and improving the reliability of the detection results.

[0044] Through the above design, the target detection module in this embodiment can effectively identify sand mining equipment in different scales and complex environments, and optimize the detection accuracy through a comprehensive loss function, providing accurate data support for subsequent behavior analysis and anomaly detection.

[0045] 3. Behavior Analysis Module: The behavior analysis module analyzes the time-series characteristics of the sand mining equipment provided by the target detection module to determine whether it is in a sand mining state. This module extracts the dynamic action characteristics of the sand mining equipment from time-series data and analyzes its continuous actions over time to distinguish between sand mining and non-sand mining activities. The behavior analysis module is a crucial behavior judgment component in the entire system, and its analysis results provide a reliable data foundation for subsequent spatiotemporal analysis and anomaly detection modules.

[0046] In this embodiment, the specific implementation of the behavior analysis module includes: Temporal Feature Extraction Unit: This unit receives the detection results of the sand mining equipment from the target detection module and extracts temporal features based on these results. To achieve accurate analysis of dynamic behavior, the system employs a 3D Convolutional Neural Network (3D CNN), extracting the temporal features of the sand mining equipment by sliding 3D convolutional kernels along the time axis. The video input is a sequence of frames. 3D convolution kernel Sliding along the time axis maps the input sequence to a high-dimensional feature space. This convolutional operation can capture the motion variations of sand mining equipment in the time dimension, facilitating the identification of dynamic patterns in sand mining behavior. The calculation of 3D convolution is as follows:

[0047] in, This represents the output feature map. Indicates the input video frame sequence, convolution kernel The system slides across time and space dimensions to generate a temporal feature map for each frame. Through 3D convolution operations, the temporal feature extraction unit can generate a set of time-series data containing equipment motion features, providing fundamental features for the identification of sand mining activities.

[0048] 2. Dynamic Behavior Recognition Unit: Based on the time-series data generated by the temporal feature extraction unit, the dynamic behavior recognition unit analyzes the continuous movements of the sand mining equipment to determine whether it is in a sand mining state. This unit focuses on specific movement patterns of the equipment (such as the repetitive movement of the bucket, the loading and unloading actions of the vehicle, etc.), and determines whether it is a sand mining activity by analyzing the repetitiveness and continuity of these actions. The core of dynamic behavior recognition lies in identifying the typical operating patterns of the sand mining equipment based on multiple frames of data over a period of time.

[0049] Effect: By extracting the time-series dynamic features of construction vehicles, the system can determine the continuity and integrity of sand mining activities. This multi-frame data-based analysis method enhances the system's intelligence, enabling it to go beyond single-frame recognition and capture continuous equipment behavior, thus greatly improving recognition accuracy.

[0050] 3. Action Pattern Determination Unit: Based on dynamic behavior recognition, the action pattern determination unit further analyzes the continuous movement characteristics of the sand mining equipment to determine whether its behavior pattern conforms to sand mining operations. This unit employs a threshold determination strategy to monitor the frequency and range of the equipment's actions over a period of time. For example, it identifies whether repetitive sand mining behavior exists by monitoring the up-and-down movement frequency of the bucket, and determines whether loading or unloading operations are performed by monitoring the frequency of changes in the vehicle's position. If an action pattern consistent with sand mining operations is detected, it is determined to be sand mining behavior.

[0051] 4. Behavior Analysis Result Output Unit: After completing the behavior determination, the behavior analysis module outputs the identified sand mining status and action characteristic information to the subsequent spatiotemporal analysis module. The output data includes the sand mining equipment's action mode, location, and confidence level. This unit ensures that subsequent modules receive accurate behavior information, enabling the system to perform anomaly detection and early warning based on more reliable input data in subsequent analyses.

[0052] Through the above design, the behavior analysis module in this embodiment can accurately analyze the dynamic behavior of the sand mining equipment based on time series characteristics, identify the sand mining status of the equipment, and provide support for subsequent spatiotemporal analysis and anomaly detection of the system. It provided accurate input data.

[0053] 4. Spatiotemporal Analysis Module: The spatiotemporal analysis module extracts spatiotemporal features from the movement trajectory and dynamic behavior of sand mining equipment based on the equipment location and motion data provided by the behavior analysis module, thereby optimizing the judgment of sand mining behavior. This module identifies abnormal behavior patterns of sand mining equipment by analyzing changes in the equipment's temporal and spatial dimensions, such as whether it stays in a specific area for an extended period or performs repeated sand mining operations. This achieves more accurate sand mining behavior identification. The analysis results from the spatiotemporal analysis module provide spatiotemporal feature information to the anomaly detection and early warning module, supporting the real-time identification of illegal sand mining activities.

[0054] In this embodiment, the specific implementation of the spatiotemporal analysis module includes: 1. Trajectory Extraction Unit: This unit receives the location information of the sand mining equipment from the behavior analysis module, records the equipment's location information in multiple time frames, and generates a motion trajectory. The equipment's trajectory reflects its movement characteristics within the monitored area. Based on the sand mining equipment's motion trajectory and the action characteristics of the sand mining robotic arm, the system can monitor the equipment's dwell patterns and movement behavior through continuous trajectory recording, providing a data foundation for further behavior determination.

[0055] 2. Velocity and Acceleration Calculation Unit: Based on the trajectory information from the trajectory extraction unit, the velocity and acceleration calculation unit calculates the velocity and acceleration of the sand mining equipment to analyze its motion characteristics. Assume the sand mining equipment moves at a certain time... The position is Then speed and acceleration The calculation formula is as follows:

[0056]

[0057] in, , , The time interval is defined as the time interval. Through this calculation, the system can capture the acceleration fluctuations and motion trends of the equipment, providing data support for identifying motion characteristics such as abnormal stillness or rapid movement.

[0058] 3. Spatiotemporal Feature Extraction Unit: Based on trajectory and velocity information, the spatiotemporal feature extraction unit further extracts the device's spatiotemporal features. This unit employs a Spatiotemporal Graph Convolutional Network (ST-GCN) to construct a graph structure that includes device location and temporal relationships. The device's position at different time points is used as nodes in the graph, and the relationships between adjacent time steps are used as edges. The video frame sequence constitutes the spatiotemporal graph. ,in This is a set of nodes, representing the device location at each time step; This is a set of edges that connect nodes in the preceding and following frames.

[0059] The graph convolution operation of ST-GCN is defined as follows:

[0060] in, Represents a node In time Features For nodes The set of neighboring nodes, The weight of the edge. For convolution kernel parameters, This is the activation function. Through this operation, the system can extract the spatiotemporal characteristics of the sand mining equipment, enabling joint analysis of the equipment's movement and positional changes in the spatiotemporal map. The system can identify the spatiotemporal change patterns of the equipment and effectively distinguish between sand mining and other irrelevant behaviors.

[0061] 4. Behavioral Pattern Analysis Unit: After the spatiotemporal features are extracted, the behavioral pattern analysis unit determines whether the equipment exhibits abnormal behavioral patterns based on its spatiotemporal characteristics. This unit focuses on whether the equipment exhibits prolonged periods of inactivity, frequent movement, or significant changes in speed and acceleration. Through behavioral pattern analysis, the system can identify whether the equipment's motion characteristics conform to typical sand mining operation patterns or whether abnormal motion characteristics exist.

[0062] In this embodiment, the spatiotemporal analysis module achieves more accurate identification of sand mining behavior through multi-dimensional analysis of the trajectory, speed, acceleration, and spatiotemporal characteristics of the sand mining equipment, providing high-quality spatiotemporal data support for the system's anomaly detection and early warning module.

[0063] 5. Multi-view fusion module: The multi-view fusion module integrates video data from sand mining areas at different angles to improve the accuracy of sand mining behavior assessment. By acquiring video data from multiple angles, the module can comprehensively monitor sand mining equipment in both time and space, avoiding behavior recognition biases caused by limitations of a single viewpoint. This module generates comprehensive features containing multi-angle information through methods such as time-series feature extraction and weighted feature fusion, providing a reliable data foundation for the system's anomaly detection and early warning.

[0064] In this embodiment, the specific implementation of the multi-view fusion module includes: 1. Multi-view data receiving unit: This unit receives video data from different angles to ensure that the motion information of the sand mining equipment can be effectively acquired at different locations in the sand mining area. Data from different angles provides multiple positions and status information of the sand mining equipment in space, and the data from each angle is processed separately to preserve the original features of each angle, laying the foundation for subsequent fusion processing.

[0065] 2. Time Series Feature Extraction Unit: After obtaining the video sequences from multiple perspectives, the time series feature extraction unit performs independent time series feature extraction on the data from each perspective. To extract dynamic action features, this unit uses a Long Short-Term Memory (LSTM) network to model the time series data of the sand mining equipment, capturing the action features of the equipment from each perspective. The recursive formula for LSTM is as follows:

[0066] in, Indicates time The hidden state, For input features, and This is the weight matrix. For bias terms, is the activation function. Through time-series modeling using LSTM, the time-series feature extraction unit can generate dynamic features of the sand mining equipment from each perspective, ensuring analysis of equipment movement changes from multiple viewpoints.

[0067] 3. Feature Weighting Unit: After time-series feature extraction, the feature weighting unit performs weighted processing on the feature data from different perspectives to fuse the feature information from each perspective. Let the video features from different perspectives be... This unit assigns a corresponding weight coefficient to each viewpoint. Features are weighted and fused based on the importance or confidence of the viewpoint, and the calculation formula is as follows:

[0068] in, For the first Feature vectors from each perspective These are the weighting coefficients for the corresponding viewpoints. The weighting coefficients are adjusted based on the confidence level of each viewpoint. For example, if the data confidence level of a certain viewpoint is high, it is given a larger weight to enhance the feature contribution of that viewpoint.

[0069] 4. Feature Fusion Output Unit: After feature weighting, the feature fusion output unit outputs the weighted and fused multi-view feature data to subsequent modules. The fused features output by this unit include the status information and dynamic features of the sand mining equipment from multiple perspectives, providing the anomaly detection and early warning module with comprehensive features containing multi-angle information, thus enhancing the system's behavior recognition capabilities in complex environments.

[0070] In this embodiment, the multi-view fusion module integrates sand mining equipment data from multiple angles through time series modeling and weighted fusion, enabling the system to obtain more comprehensive and accurate judgment criteria in sand mining behavior identification.

[0071] 6. Anomaly Detection and Early Warning Module: The anomaly detection and early warning module is used to detect and issue early warnings for anomalies in the movement characteristics of sand mining equipment. This module receives feature data from the spatiotemporal analysis module and the multi-view fusion module, uses a probabilistic model to analyze the distribution of the movement characteristics of the sand mining equipment, and determines whether abnormal behavior exists. Once an anomaly is detected, the system triggers an early warning and stores the relevant data. The anomaly detection and early warning module can effectively identify abnormal patterns in sand mining activities, providing regulatory personnel with real-time alarm information and detailed data support.

[0072] In this embodiment, the specific implementation of the anomaly detection and early warning module includes: 1. Feature Distribution Modeling Unit: This unit is used to model the motion characteristic data of the sand mining equipment. A Gaussian Mixture Model (GMM) is used to construct the probability distribution of the sand mining motion characteristics for subsequent anomaly detection and judgment. The probability density function of the GMM is defined as:

[0073] in, The number of Gaussian distributions, For the first The weights are distributed in a Gaussian manner, satisfying... , For the first The probability density function of a Gaussian distribution has a mean of . The covariance matrix is The feature distribution modeling unit constructs a Gaussian mixture model by training on the normal action feature data of the sand mining equipment, which is used to identify abnormal behaviors that differ significantly from the normal action feature distribution.

[0074] 2. Anomaly Detection Unit: Based on the probability distribution generated by the feature distribution modeling unit, the anomaly detection unit judges the anomalies of the sand mining equipment's action characteristics. This unit calculates the probability density of the equipment's current action characteristics in the model distribution. When the probability density of a certain action characteristic is detected to be lower than a preset threshold, the action characteristic is judged to be abnormal. The threshold can be determined based on historical data statistics to ensure the system's adaptability in different scenarios.

[0075] 3. Alarm Trigger Unit: When the anomaly detection unit detects abnormal behavior, the alarm trigger unit automatically generates an alarm message and notifies the supervisory personnel. The alarm message includes detailed information such as the time, location, type of behavior, and confidence level of the abnormal sand mining behavior, allowing supervisory personnel to promptly grasp the abnormal situation in the sand mining area. The design of the alarm trigger unit realizes the system's real-time early warning function, immediately taking alarm measures upon detecting abnormal sand mining behavior to ensure rapid response.

[0076] 4. Data Storage Unit: To support subsequent law enforcement and analysis, the data storage unit saves the images, videos, and feature data generated during the anomaly detection process, forming a complete chain of evidence for abnormal behavior. This unit uniformly stores the original image frames, video clips, feature data, and alarm information from the anomaly detection process, facilitating subsequent retrieval and analysis. Furthermore, the data storage unit categorizes and manages the stored data to ensure the integrity and traceability of relevant evidence.

[0077] Effect: This publicly available system automatically stores image data of abnormal sand mining activities, forming a complete chain of evidence to facilitate subsequent investigations and law enforcement. This design not only ensures the integrity and traceability of the data but also provides reliable data support for the regulatory platform.

[0078] 5. Multi-level Early Warning Unit: This unit categorizes abnormal behavior into different warning levels based on its severity and deviation from normal behavioral patterns, including low-risk, medium-risk, and high-risk warnings. The multi-level early warning unit design allows regulators to take different measures based on the risk level of the warning; for example, observing low-risk warnings and taking emergency intervention for high-risk warnings.

[0079] The anomaly detection and early warning module in this embodiment constructs an action feature distribution using a Gaussian mixture model to accurately detect abnormal behavior and triggers multi-level early warnings based on the degree of anomaly. Ultimately, it provides real-time alarms and detailed stored data, providing important technical support for the control and evidence collection of illegal sand mining activities.

[0080] Effect: Existing target detection and behavior recognition technologies lack intelligent anomaly detection methods, cannot provide real-time early warnings and effective evidence support, and are difficult to meet the regulatory needs of complex scenarios; This publicly available system utilizes anomaly detection technology to analyze the characteristics of sand mining activities in real time. If abnormal behavior is detected, an alarm is automatically triggered, and the image data is stored as a chain of evidence. This function enables the system to automatically monitor and respond quickly to illegal sand mining activities, providing timely early warning information to regulatory personnel.

[0081] In summary, this disclosure achieves efficient and accurate identification of illegal sand mining activities through the collaborative work of six modules: data acquisition, target detection, behavior analysis, spatiotemporal analysis, multi-view fusion, and anomaly detection and early warning. The system can analyze sand mining behavior based on real-time acquired image data, trigger anomaly detection, and automatically issue alarms, providing regulatory authorities with an intelligent monitoring and early warning tool. The system exhibits excellent real-time performance, accuracy, and robustness, maintaining high-precision identification in complex environments and is applicable to various types of sand mining scenarios, providing strong technical support for the monitoring and crackdown on illegal sand mining.

[0082] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for identifying illegal sand mining activities, characterized in that, include: Collect image data of the sand mining area from multiple different perspectives; The target detection algorithm is used to select sand mining equipment from image data and extract the location information of the sand mining equipment; Extract the motion characteristics of the sand mining equipment, and locate the sand mining equipment in the sand mining state based on the motion characteristics; For sand mining equipment in the sand mining state, a motion trajectory is generated based on its position information; the velocity and acceleration of the sand mining equipment are calculated based on the motion trajectory; a spatiotemporal map is constructed based on the motion trajectory, velocity, and acceleration of the sand mining equipment; and the spatiotemporal features of the sand mining equipment are extracted based on the spatiotemporal map. Temporal motion features of sand mining equipment are extracted from image data of sand mining areas from multiple different perspectives, and the temporal motion features are fused to obtain fused motion features. Based on the fusion of action features and spatiotemporal features, illegal sand mining activities of sand mining equipment can be identified.

2. The method for identifying illegal sand mining activities according to claim 1, characterized in that, The acquisition of image data of the sand mining area from multiple different perspectives includes: Preliminary image and video data of sand mining areas from multiple different perspectives were collected, and timestamps and GPS location information were added to the data; a high-frequency acquisition mode was used for active sand mining areas, and a low-frequency acquisition mode was used for stable areas. Multi-source data fusion processing was performed on the preliminary image data and preliminary video data to obtain image data of the sand mining area from multiple different perspectives.

3. The method for identifying illegal sand mining activities according to claim 1, characterized in that, The step of selecting sand mining equipment from image data includes: Multi-scale feature extraction is performed on image data using a feature pyramid network to generate feature maps at different scales. Based on feature maps of different scales, target detection algorithms are used to extract the location information, category information, and detection confidence of sand mining equipment. The location and category information of sand mining equipment with a detection confidence level higher than a preset threshold are selected as the detection results.

4. The method for identifying illegal sand mining activities according to claim 1, characterized in that, The step of extracting the motion characteristics of the sand mining equipment and locating the sand mining equipment in the sand mining state based on the motion characteristics includes: Based on the detection results of the sand mining equipment, a 3D convolutional neural network is used to extract temporal features from the video frame sequence of the sand mining equipment to obtain a feature map; and based on the feature map, it is determined whether the construction vehicle is in the sand mining state.

5. The method for identifying illegal sand mining activities according to claim 1, characterized in that, The construction of a spatiotemporal map based on the motion trajectory, velocity, and acceleration of the sand mining equipment; and the extraction of spatiotemporal features of the sand mining equipment based on the spatiotemporal map; include: The location of the sand mining equipment at different time points is obtained based on its movement trajectory; A spatiotemporal graph is constructed using the position, velocity, and acceleration of the sand mining equipment at different time points as node features and the association between adjacent time steps as edges; the weight of the edges is calculated based on the time interval, position distance, and velocity consistency. By inputting the spatiotemporal graph into a spatiotemporal graph convolutional network, spatiotemporal features representing the device's motion trajectory, velocity changes, and acceleration patterns are obtained.

6. The method for identifying illegal sand mining activities according to claim 1, characterized in that, The process involves extracting temporal motion features of sand mining equipment from image data of sand mining areas from multiple different perspectives, and fusing these temporal motion features to obtain fused motion features; including: Long Short-Term Memory (LSTM) networks were used to extract temporal action features from video data from multiple different perspectives. Different weight coefficients are assigned to the action features from different perspectives. The action features from different perspectives are then weighted using these weight coefficients to obtain the fused features.

7. The method for identifying illegal sand mining activities according to claim 1, characterized in that, The method of determining illegal sand mining behavior of sand mining equipment based on fused action features and spatiotemporal features includes: A feature distribution model is constructed, which is trained based on the fused action features and spatiotemporal features of historical normal behavior to output the probability density of the current behavior. The fused action features and spatiotemporal features to be detected are input into the feature distribution model to obtain the probability density; If the probability density is lower than a preset threshold, the sand mining equipment is deemed to be behaving abnormally.

8. A system for identifying illegal sand mining activities, characterized in that, include: The data acquisition module is used to collect image data of the sand mining area from multiple different perspectives; The target detection module is used to select sand mining equipment from image data using target detection algorithms and extract the location information of the sand mining equipment. The behavior judgment module is used to extract the action characteristics of the sand mining equipment and find the sand mining equipment in the sand mining state based on the action characteristics; The spatiotemporal feature extraction module is used to generate a motion trajectory for sand mining equipment in the sand mining state based on its location information; calculate the velocity and acceleration of the sand mining equipment based on the motion trajectory; construct a spatiotemporal map based on the motion trajectory, velocity, and acceleration of the sand mining equipment; and extract the spatiotemporal features of the sand mining equipment based on the spatiotemporal map. The feature fusion module is used to extract the temporal action features of sand mining equipment from image data of sand mining areas from multiple different perspectives, and to fuse the temporal action features to obtain fused action features. The identification module is used to determine illegal sand mining behavior of sand mining equipment based on fused action features and spatiotemporal features.

9. The system for identifying illegal sand mining activities according to claim 8, characterized in that, The behavior determination module includes: The temporal feature extraction unit is used to receive the detection results of the sand mining equipment output by the target detection module, and use a 3D convolutional neural network to extract temporal features from the video frame sequence of the sand mining equipment to obtain a feature map; The sand mining behavior determination unit is used to determine whether the sand mining equipment is in a sand mining state based on the feature map. The behavior analysis result output unit is used to output the analysis results of whether the sand mining equipment is in a certain state.

10. The system for identifying illegal sand mining activities according to claim 8, characterized in that, The identification module includes: The feature distribution modeling unit is used to probabilistically model the action characteristics of the sand mining equipment using a Gaussian mixture model, so as to output the probability density of the current behavior; the probability density function of the Gaussian mixture model is: in, The number of Gaussian distributions, For the first The weights are distributed in a Gaussian manner, satisfying... , For the first The probability density function of a Gaussian distribution has a mean of . The covariance matrix is ; An anomaly determination unit is used to determine that the sand mining equipment is behaving abnormally if the probability density of the current behavior is lower than a preset threshold.

Images