Dynamic dumping site sensing method based on space-time consistency constraint

By fusing radar point cloud and image data through a spatiotemporal graph convolutional neural network and constructing a dynamic graph structure, the problem of misjudgment of traditional perception models in dynamic open-pit mine spoil dumps is resolved, accurate prediction and safe management of equipment and materials are achieved, and the efficiency and safety of automated operations are improved.

CN120707957APending Publication Date: 2025-09-26CISDI RES & DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510841251.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-23
Publication Date
2025-09-26

AI Technical Summary

Technical Problem

Traditional perception models find it difficult to update the scene status of dynamic open-pit mine dumps in real time, and lack the ability to model dynamic changes in time series, resulting in misjudgments of material collapse warnings and insufficient accuracy in equipment trajectory predictions, as well as complex multi-sensor data management.

Method used

A spatiotemporal graph convolutional neural network (STGN) is used to fuse radar point cloud and image data to construct a dynamic graph structure. The node status is updated through time series data and the scene model is corrected in real time. Combined with spatiotemporal consistency constraints and physical laws, joint prediction of equipment and materials is achieved.

Benefits of technology

It improves the safety and efficiency of automated operations in spoil dumps, reduces accident risks, simplifies multi-sensor data management, and improves production and management efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120707957A_ABST
    Figure CN120707957A_ABST
Patent Text Reader

Abstract

A dynamic dumping site sensing method based on space-time consistency constraint belongs to the field of computer vision, and comprises the following steps: S1, acquiring an image and radar point cloud multi-modal sensor data; s2, preliminarily identifying a scene object through the image and the radar point cloud; s3, constructing a space-time diagram convolutional neural network of the scene based on the identified object; s4, predicting a node state based on a space-time diagram convolutional neural network; and S5, carrying out space-time consistency constraint on a prediction result. According to the invention, the spatial information precision of the dumping site is improved, the use experience of equipment and personnel is enhanced, and the property safety is guaranteed. According to the invention, the accident rate of unmanned operation is reduced, the safety of automatic operation is improved, the capability of handling emergencies is enhanced, and the management efficiency is improved at the same time. Refined safety management and process control of automatic production are realized, the production efficiency is further improved, the production order and safety are maintained, and meanwhile, the management and production efficiency of the refuse dump is also improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to a dynamic spoil dump perception method based on spatiotemporal consistency constraints. Background Art

[0002] In the automated operations of unmanned dumping sites in open-pit mines, the dynamic environment (such as robotic arm movement, material collapse, and changes in stacking morphology) makes it difficult for traditional perception models to update scene state perception in real time. Existing technologies rely on static sensor data (such as single-frame radar point clouds or images) and lack the ability to model dynamic changes in time series, resulting in accumulated errors in scene segmentation and target tracking. For example, material collapse warning requires capturing both spatial structural changes and temporal continuity, while traditional methods are prone to misjudgment due to ignoring spatiotemporal correlation. In addition, data noise in dynamic environments (such as dust interference and metal reflections) further exacerbates the robustness challenges of the perception system.

[0003] Traditional methods (such as YOLO+Kalman filtering) track equipment and materials independently, severing the physical interaction relationship between loaders, transport vehicles and materials. This limits the accuracy of trajectory prediction and risk warning, making it difficult to cope with the complexity and real-time requirements of dynamic interactive scenarios. Summary of the Invention

[0004] In light of this, the present invention aims to provide a dynamic spoil dump perception framework with spatiotemporal consistency constraints. By fusing time-series radar point clouds with image data, a spatiotemporal graph convolutional neural network (STGN) is constructed to model the spoil dump scene as a dynamic graph structure. Using material blocks as nodes and spatial associations as edges, the framework updates node states based on time-series data and integrates newly acquired sensor data in real time to dynamically modify the scene model.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A dynamic spoil dump perception method based on spatiotemporal consistency constraints includes the following steps:

[0007] S1: Acquire multimodal sensor data including images and radar point clouds;

[0008] S2: Preliminary recognition of scene objects through images and radar point clouds;

[0009] S3: Constructing a spatiotemporal graph convolutional neural network of the scene based on the identified objects;

[0010] S4: Predict node status based on spatiotemporal graph convolutional neural network;

[0011] S5: Apply spatiotemporal consistency constraints to the prediction results.

[0012] Furthermore, in step S1, data is collected using lidar and cameras to obtain multimodal heterogeneous data. The image data is used to classify objects based on their texture features and guide point cloud segmentation. The radar point cloud is used to accurately locate the object. The object's position and volume are then obtained based on the point cloud data.

[0013] Furthermore, in step S2, based on RGB or infrared images, a deep learning pre-trained model is used to detect the equipment and materials in the scene, and output 2D bounding boxes, category labels and pixel-level semantic segmentation masks; the radar point cloud is preprocessed, and a density clustering algorithm is used to segment the point cloud into independent object clusters. Based on the image classification and segmentation results, the category of each cluster is determined, and the 3D bounding box, center of mass coordinates, volume and pose information of each cluster are calculated.

[0014] Furthermore, in step S3, different categories of nodes are constructed according to the category of scene objects, including device nodes and material nodes; these nodes are determined based on the recognition results of S2, and the feature representations of these nodes are calculated; for device nodes, the coordinates, orientation, action category, speed and category code of the point cloud centroid are used as feature representations; for material nodes, the coordinates, volume, projected area, roughness and category code of the point cloud centroid are used as feature representations; for edge construction, edge connections are established considering the functional relationship of the nodes; device nodes and material nodes have an functional relationship, directed edges are established, and weights are set according to the distance.

[0015] Furthermore, in step S4, two encoders are used to project the features into a unified dimension. The node state prediction process of the spatiotemporal graph convolutional neural network ST-GCN includes:

[0016] First, spatial convolution is used to aggregate node neighborhood information. For node i, the spatial aggregation process is as follows:

[0017]

[0018] Where N(i) represents the neighborhood set of node i, h j Represents the corresponding neighborhood node features, W (i,j) Indicates different weight matrices selected based on edge type. That is, different types of edges consisting of equipment and materials, and materials and materials will use different types of weight matrices for calculation. After aggregating neighboring nodes, they are updated in combination with their own nodes:

[0019] x i =σ(U·h i +m i )

[0020] Where σ represents the activation function, and U represents the different weight matrices selected according to the node type. That is, the weight matrices used by equipment nodes and material nodes are different.

[0021] Then, the node features aggregated on the historical frames are combined to perform sliding window time series modeling on the time series information; for each node, the time series features of the past T frames are extracted by sliding the window along the time axis to obtain the node time series features Temporal feature extraction is based on the transformer structure; three linear layers W are used respectively Q ,W K ,W V As a weight matrix, transform the time series feature X, that is, Q = XW Q ,K=XW K ,V=XW V ; Use transformer’s self-attention mechanism for encoding:

[0022]

[0023] After obtaining the time series features of the node, different predictors are selected for prediction according to the node type. The predictor consists of a linear layer:

[0024] yi=σ(Wclass (i) hi+bclass (i) )

[0025] Where σ represents the activation function, W class(i) and b class(i) They represent different weight matrices and biases selected according to different node types; for device nodes, the prediction results represent their position, posture, and motion status; for material nodes, the prediction results represent the deformation, volume change, and projected area change.

[0026] Furthermore, in step S5, the spatiotemporal consistency constraint forces the model prediction results to conform to the dynamic evolution logic of the real scene through physical laws and time continuity, and the smoothness loss L smooth Expressed as:

[0027]

[0028] λ is the smoothing coefficient, which is dynamically adjusted according to the environmental sensor data. When the sensor noise increases in extreme environments, λ is automatically increased to strengthen the timing smoothness constraint and avoid prediction jumps caused by false detection.

[0029] The beneficial effects of the present invention are as follows: the present invention integrates equipment, materials, and environment into a dynamic graph structure through spatiotemporal graph convolutional networks and heterogeneous node modeling, uses the graph structure to encode the interactive relationships between them, and injects physical law priors based on temporal smoothness constraints to achieve joint reasoning of equipment behavior and material status, overcoming the core defects of traditional technologies such as data fragmentation and insufficient dynamic adaptability.

[0030] The present invention significantly improves the efficiency and safety of automated operations in spoil dumps through full-scene dynamic perception and collaborative prediction. Accurate prediction of equipment trajectories and material risks can optimize the robot arm's obstacle avoidance path and multi-machine collaborative scheduling, reducing operation interruptions and energy waste; real-time collapse warning and collision detection functions significantly reduce the risk of safety accidents and ensure the safety of personnel and equipment. At the same time, a unified spatiotemporal perception framework simplifies the management complexity of multi-sensor data, reduces the need for manual inspections and parameter tuning, and improves overall operational management efficiency. In terms of economic benefits, this solution provides cost-effective automation upgrade support for spoil dump scenarios by reducing downtime, extending equipment life, and reducing accident losses, thereby facilitating industrial intelligent transformation.

[0031] In summary, this invention improves the spatial accuracy of spoil dump information, enhances the user experience of equipment and personnel, and ensures property safety. It also reduces the accident rate of unmanned operations, improves the safety of automated operations, enhances the ability to handle emergencies, and improves management efficiency. It enables refined safety management and process control of automated production, further improving production efficiency, maintaining production order and safety, and also enhancing spoil dump management and production efficiency.

[0032] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0034] Figure 1 This is a flow chart of the dynamic dump site perception method based on spatiotemporal consistency constraints;

[0035] Figure 2 Flowchart for spatiotemporal graph convolution state prediction. DETAILED DESCRIPTION

[0036] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0037] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0038] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0039] Example 1:

[0040] like Figure 1-2 As shown, the present invention provides a dynamic spoil dump perception framework with spatiotemporal consistency constraints. By fusing time-series radar point clouds and image data, a spatiotemporal graph neural network (STGN) is constructed to model the spoil dump scene as a dynamic graph structure. Using material blocks as nodes and spatial correlations as edges, the node states are updated in combination with time-series data, and newly collected sensor data is integrated in real time to dynamically correct the scene model. This method includes the following steps:

[0041] S1: Environmental data is acquired through a sensor array. To address the complexity of the spoil dump environment and the limitations of single-modality data, LiDAR and cameras are used for data collection, generating multimodal and heterogeneous data. Image data can be classified based on object texture features and guide point cloud segmentation. LiDAR point clouds can accurately locate object positions. Furthermore, object pose, volume, and other physical characteristics can be derived from point cloud data.

[0042] S2: Based on RGB or infrared images, a pre-trained deep learning model (such as YOLOv8 or Mask R-CNN) is used to detect equipment (loaders, transport trucks) and materials in the scene, outputting 2D bounding boxes, category labels, and pixel-level semantic segmentation masks. The radar point cloud is pre-processed (denoising and ground segmentation) and segmented into independent object clusters using a density clustering algorithm (such as DBSCAN). Based on the image classification and segmentation results, the category of each cluster is determined, and the 3D bounding box, center of mass coordinates, volume, and pose information of each cluster are calculated.

[0043] S3: Construct different categories of nodes based on the object categories in the scene, which mainly include equipment nodes and material nodes in the spoil dump operation scene. These nodes are determined based on the recognition results of S2, and the feature representations of these nodes are calculated. For equipment nodes, the coordinates, orientation, action category, speed, and category code of the point cloud center of mass are used as feature representations. For material nodes, the coordinates, volume, projected area, roughness, and category code of the point cloud center of mass are used as feature representations. For edge construction, edge connections are established considering the interaction relationship between nodes. Equipment nodes and material nodes have an interaction relationship, and directed edges are established, and weights are set according to distance.

[0044] S4: Since the feature dimensions of nodes of different categories are different, in order to facilitate calculation using the spatiotemporal graph convolutional network, two encoders (MLP) are used to project the features into a unified dimension. The node state prediction process of the spatiotemporal graph convolutional neural network (ST-GCN) is as follows: First, spatial convolution is used to aggregate node neighborhood information. For node i, the spatial aggregation process is as follows:

[0045]

[0046] Where N(i) represents the neighborhood set of node i, h j Represents the corresponding neighborhood node features, W (i,j) This indicates the different weight matrices selected based on the edge type. For example, different types of edges consisting of equipment and materials, or materials and materials, will be calculated using different weight matrices. After aggregating neighboring nodes, the node itself is then combined for update.

[0047] x i =σ(U·h i +m i )

[0048] Where σ represents the activation function, and U represents the different weight matrices selected according to the node type, that is, the weight matrices used by equipment nodes and material nodes are different.

[0049] Then, the node features aggregated on the historical frames are combined to perform sliding window time series modeling on the time series information. For each node, the time series features of the past T frames are extracted by sliding the window along the time axis to obtain the node time series features Temporal feature extraction is based on the transformer structure. Three linear layers (weight matrices) W are used. Q ,W K ,W V , transform the time series feature X, that is, Q = XW Q ,K=XW K ,V=XW V . The transformer’s self-attention mechanism is used for encoding.

[0050]

[0051] After obtaining the time series features of the node, different predictors are selected for prediction according to the node type. The predictor consists of a linear layer.

[0052] yi=σ(Wclass (i) hi+bclass (i) )

[0053] Where σ represents the activation function, W class(i) and b class(i) These represent the weight matrices and biases selected for each node type. Therefore, different node types produce different prediction results. For device nodes, the prediction results primarily represent their position, posture, and motion state. For material nodes, the prediction results primarily represent deformation, volume change, and projected area change.

[0054] S5: The spatiotemporal consistency constraint aims to force the model prediction results to conform to the dynamic evolution logic of the real scene through physical laws and time continuity. smooth Expressed as:

[0055]

[0056] λ is the smoothing coefficient, which is dynamically adjusted according to the environmental sensor data. When the sensor noise increases in extreme environments, λ is automatically increased to strengthen the timing smoothness constraint and avoid prediction jumps caused by false detection.

[0057] Example 2:

[0058] An electronic device comprising a memory and a processor;

[0059] The memory is used to store computer programs;

[0060] The processor is configured to implement the method described in Example 1 when executing the computer program.

[0061] Example 3:

[0062] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in Example 1 is implemented.

[0063] Example 4:

[0064] A computer program product includes a computer program, which implements the method described in embodiment 1 when executed by a processor.

[0065] In the above embodiments, references to "this embodiment" in the specification indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily refer to the same embodiment.

[0066] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the present invention are intended to encompass all such alternatives, modifications, and variations that fall within the broad scope of the appended claims.

[0067] Regarding the computer-readable storage medium in this embodiment, those skilled in the art will appreciate that all or part of the steps in the aforementioned method embodiments can be implemented using hardware associated with the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0068] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used for communication, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes the various steps of the above method.

[0069] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0070] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0071] The present invention can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0072] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A dynamic spoil dump perception method based on spatiotemporal consistency constraints, characterized by: The following steps are involved: S1: Acquire multimodal sensor data including images and radar point clouds; S2: Preliminary recognition of scene objects through images and radar point clouds; S3: Constructing a spatiotemporal graph convolutional neural network of the scene based on the identified objects; S4: Predict node status based on spatiotemporal graph convolutional neural network; S5: Apply spatiotemporal consistency constraints to the prediction results.

2. The dynamic spoil dump perception method based on spatiotemporal consistency constraints according to claim 1 is characterized by: In step S1, laser radar and camera are used to collect data to obtain multi-modal heterogeneous data; the image data is used to classify the object based on its texture features and guide point cloud segmentation; The radar point cloud is used to accurately locate the position of an object; and to obtain the object's posture and volume based on the point cloud data.

3. The dynamic spoil dump perception method based on spatiotemporal consistency constraints according to claim 1 is characterized by: In step S2, based on RGB or infrared images, a deep learning pre-trained model is used to detect the equipment and materials in the scene, and 2D bounding boxes, category labels, and pixel-level semantic segmentation masks are output; the radar point cloud is preprocessed and a density clustering algorithm is used to segment the point cloud into independent object clusters. Based on the image classification and segmentation results, the category of each cluster is determined, and the 3D bounding box, center of mass coordinates, volume, and pose information of each cluster are calculated.

4. The dynamic spoil dump perception method based on spatiotemporal consistency constraints according to claim 1 is characterized by: In step S3, different categories of nodes are constructed according to the scene object categories, including device nodes and material nodes; these nodes are determined based on the recognition results of S2, and the feature representations of these nodes are calculated; For device nodes, the coordinates of the point cloud centroid, orientation, motion category, speed, and category code are used as feature representations; for material nodes, the coordinates of the point cloud centroid, volume, projected area, roughness, and category code are used as feature representations; For edge construction, edge connections are established by considering the functional relationship of nodes; the equipment node and the material node have a functional relationship, a directed edge is established, and the weight is set according to the distance.

5. The dynamic spoil dump perception method based on spatiotemporal consistency constraints according to claim 1 is characterized by: In step S4, two encoders are used to project the features into a unified dimension. The node state prediction process of the spatiotemporal graph convolutional neural network ST-GCN includes: First, spatial convolution is used to aggregate node neighborhood information. For node i, the spatial aggregation process is as follows: Where N(i) represents the neighborhood set of node i, h j Represents the corresponding neighborhood node features, W (i,j) Indicates different weight matrices selected based on edge type. That is, different types of edges consisting of equipment and materials, and materials and materials will use different types of weight matrices for calculation. After aggregating neighboring nodes, they are updated in combination with their own nodes: x i =σ(U·h i +m i ) Where σ represents the activation function, and U represents the different weight matrices selected according to the node type. That is, the weight matrices used by equipment nodes and material nodes are different. Then, the node features aggregated on the historical frames are combined to perform sliding window time series modeling on the time series information; for each node, the time series features of the past T frames are extracted by sliding the window along the time axis to obtain the node time series features Temporal feature extraction is based on the transformer structure; three linear layers W are used respectively Q ,W K ,W V As a weight matrix, transform the time series feature X, that is, Q = XW Q ,K=XW K ,V=XW V ; Use transformer’s self-attention mechanism for encoding: After obtaining the time series features of the node, different predictors are selected for prediction according to the node type. The predictor consists of a linear layer: yi=σ(Wclass (i) hi+bclass (i) ) Where σ represents the activation function, W class(i) and b class(i) They represent different weight matrices and biases selected according to different node types; for device nodes, the prediction results represent their position, posture, and motion status; for material nodes, the prediction results represent the deformation, volume change, and projected area change.

6. The dynamic spoil dump perception method based on spatiotemporal consistency constraints according to claim 1 is characterized by: In step S5, the spatiotemporal consistency constraint forces the model prediction results to conform to the dynamic evolution logic of the real scene through physical laws and time continuity, and the smoothness loss L smooth Expressed as: λ is the smoothing coefficient, which is dynamically adjusted according to the environmental sensor data. When the sensor noise increases in extreme environments, λ is automatically increased to strengthen the timing smoothness constraint and avoid prediction jumps caused by false detection.

7. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the dynamic spoil dump perception method based on spatiotemporal consistency constraints as described in any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the dynamic spoil dump perception method based on spatiotemporal consistency constraints as described in any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that: It includes a computer program, which, when executed by a processor, implements the dynamic spoil dump perception method based on spatiotemporal consistency constraints as described in any one of claims 1 to 6.