Strip mine surface real-time monitoring method based on digital twinning
By constructing a digital twin scenario and combining it with UAV oblique photography data and video stream data, real-time 3D monitoring of open-pit mines was achieved, solving the problem of unintuitive monitoring in existing technologies and improving monitoring efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-12
- Publication Date
- 2026-03-27
AI Technical Summary
Existing open-pit mine monitoring methods suffer from issues such as drone battery life, inability to monitor the entire mining area in real time, and two-dimensional video footage lacking location information, resulting in unintuitive monitoring.
By using drones to acquire oblique photography data of open-pit mines, a digital twin scene is constructed. Real-time video streams are obtained through video hardware data docking, spatial range and location information are set, and the scene is displayed in a three-dimensional environment.
It enables real-time and intuitive monitoring of the entire mine, solving the problem of unintuitive monitoring in existing technologies and improving monitoring efficiency and accuracy.
Smart Images
Figure CN121750830A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of mining technology, specifically relating to a method for real-time monitoring of the surface of open-pit mines based on digital twins. Background Technology
[0002] Currently, open-pit mines mainly include processes such as drilling, blasting, loading and transportation, rock dumping, and soil dumping. In the drilling process, real-time monitoring ensures that the drilling equipment operates according to the predetermined trajectory and depth, improving drilling accuracy and efficiency. In the blasting process, real-time monitoring provides timely feedback on the execution of the blasting design and the quality of the blasting effect, facilitating timely observation of vibrations and flyrock during blasting, and providing reference guidance for improving subsequent blasting plans. In the loading and transportation process, real-time monitoring allows for monitoring the overall vehicle distribution, providing real-time feedback on current transportation efficiency, and enabling timely intervention to adjust vehicle configuration, thereby improving loading and transportation efficiency.
[0003] In existing technologies, the main monitoring methods for open-pit mines include drone aerial surveying and video equipment monitoring. These monitoring methods are subject to various unavoidable influencing factors. For example, drone aerial surveying has limitations due to the hardware limitations of the drones themselves, and since drone aerial surveying is mostly a planned, periodic task, it cannot monitor the entire mining area in real time. In addition, equipment and technology are updated and iterated rapidly, resulting in high investment costs. Video monitoring methods typically involve building video walls in the mine, splitting the video feed and projecting it onto multiple screens for personnel to observe. However, the video content is two-dimensional and cannot intuitively show changes in the mine's topography, and the video itself does not contain location information. Summary of the Invention
[0004] To address the aforementioned problems in the existing technology, this invention provides a method for real-time monitoring of the surface of open-pit mines based on digital twins. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, embodiments of the present invention provide a method for real-time monitoring of the surface of an open-pit mine based on digital twins, comprising: Using drones to perform aerial surveys and obtain oblique photography data of open-pit mines; A digital twin scene was constructed based on the oblique photography data of the open-pit mine; Connect to camera hardware devices to acquire real-time video stream data and parse spatial coordinate parameters; Based on the spatial coordinate parameters, a spatial mapping relationship between the video footage and the digital twin scene is established and displayed.
[0005] In one embodiment of the present invention, the step of constructing a digital twin scene based on the open-pit mine oblique photogrammetry data includes: An oblique photography model of the open-pit mine is established based on the oblique photography data of the open-pit mine. The oblique photography model of the open-pit mine is loaded using Unity3D to construct a digital twin scene.
[0006] In one embodiment of the present invention, the spatial coordinate parameters include the latitude and longitude coordinates of the camera, installation height, shooting angle parameters, and geographical boundary information of the video coverage area.
[0007] In one embodiment of the present invention, the step of using a drone to perform aerial surveying tasks and obtain oblique photogrammetry data of an open-pit mine includes: According to the preset mission cycle, drones are used to carry out aerial survey missions to obtain oblique photography data of open-pit mines for the current mission cycle.
[0008] In one embodiment of the present invention, the steps of establishing an open-pit mine oblique photogrammetry model based on the open-pit mine oblique photogrammetry data, loading the open-pit mine oblique photogrammetry model using Unity3D, and constructing a digital twin scene include: Based on the oblique photography data of the open-pit mine during the current mission period, establish the oblique photography model of the open-pit mine during the current mission period. Acquire oblique photography data of the open-pit mine from the previous task cycle; The oblique photography data of the open-pit mine in the current task cycle and the previous task cycle are converted into point cloud data. After aligning the coordinate system through the point cloud registration algorithm, the changed areas are determined. Based on the digital twin scene of the previous task cycle, the open-pit mine oblique photography model of the changing area in the current task cycle is loaded using Unity3D to obtain the digital twin scene of the current task cycle.
[0009] Secondly, embodiments of the present invention also provide a real-time monitoring device for the surface of an open-pit mine based on digital twins, comprising: The data acquisition module is used to perform aerial surveying tasks using drones to obtain oblique photogrammetry data of open-pit mines; The scene construction module is used to construct a digital twin scene based on the open-pit mine oblique photography data; The parameter parsing module is used to interface with camera hardware devices, acquire real-time video stream data, and parse spatial coordinate parameters. The display module is used to establish a spatial mapping relationship between the video image and the digital twin scene based on the spatial coordinate parameters, and then display the image.
[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method for real-time monitoring of open-pit mine surfaces based on digital twins. First, it uses drones to obtain oblique photography data of the open-pit mine. Then, it uses the Unity3D engine to construct a digital twin scene. Real-time video images are obtained through data processing via video hardware. The spatial range and location information of the video images are set, and the spatial range information of the video images is converted and combined with the spatial information of the oblique photography model of the open-pit mine. Finally, the real-time video is played, which can be intuitively displayed in the three-dimensional digital twin scene, solving the problem of real-time monitoring of the overall appearance of the mine for mine management personnel.
[0011] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0012] Figure 1 This is a flowchart of a method for real-time monitoring of the surface of an open-pit mine based on digital twins, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a real-time surface monitoring method for open-pit mines based on digital twins provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a real-time monitoring device for the surface of an open-pit mine based on digital twins, provided in an embodiment of the present invention. Detailed Implementation
[0013] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0014] Figures 1-2 This is a flowchart of a real-time surface monitoring method for open-pit mines based on digital twins, provided in an embodiment of the present invention. Figures 1-2 As shown, this embodiment of the invention provides a method for real-time monitoring of the surface of an open-pit mine based on digital twins, including: S1. Use drones to perform aerial surveys and obtain oblique photography data of open-pit mines.
[0015] Specifically, the first step is to plan the mission area and inspect the flight environment. Then, checkpoints are set up, the aircraft is configured, and the flight path parameters are adjusted according to the actual situation. When setting the flight path and parameters, the "oblique photography" mission type is selected. The software automatically plans the flight path mission. Flight parameters include mission altitude, flight speed, overlap rate, and gimbal pitch angle. Camera parameters include image ratio, exposure mode, and distortion correction.
[0016] It should be noted that if the drone encounters special circumstances during flight and needs to adjust its flight path parameters, after completing the aerial survey mission, the open-pit mine oblique photogrammetry data should be copied to a storage medium. Open-pit mine oblique photogrammetry data can accurately reflect the terrain features, obtain high-precision texture information, and includes precise coordinate data.
[0017] S2. Construct a digital twin scene based on oblique photogrammetry data from open-pit mines.
[0018] In this embodiment, step S2 includes: An oblique photogrammetry model of an open-pit mine is established based on oblique photogrammetry data. The oblique photogrammetry model of the open-pit mine is loaded using Unity3D to construct a digital twin scene.
[0019] Specifically, the open-pit mine oblique photogrammetry data is exported from the storage medium. The first pass of aerial triangulation is performed to generate a sparse point cloud of the open-pit mine oblique photogrammetry data. The image control point file is imported, and it is checked whether the image control points are within the range of the aerial triangulation point cloud. Then, the second pass of aerial triangulation is performed, and the open-pit mine oblique photogrammetry model in OSGB format is output.
[0020] Furthermore, relevant support packages are introduced: Burst (high-performance computing), JobSystem (multi-threaded loading), Addressables (dynamic resource loading), and MeshSimplifierPro (LOD generation). Because common oblique photogrammetry models have a large number of triangles, direct loading can lead to memory overflow, and the high texture resolution can cause excessive GPU memory usage. Therefore, a Unity3D script (which inherits from the MonoBehaviour class by default) is created to implement the open-pit mine oblique photogrammetry model loading function through the following steps: (1) Use the Unity3D feature IEnumerator to declare the method LoadOsgbTiles() (2) First, read the directory structure of the open-pit mine oblique photogrammetry model. "tile_config.json" is the JSON file with the oblique photogrammetry structure: var tileConfig = JsonUtility.FromJson <tileconfig>( File.ReadAllText(Path.Combine(dataPath, "tile_config.json"))); (3) Use Addressable to asynchronously load the open-pit mine oblique photogrammetry model, instantiate and set the model coordinates, and assign material textures to the model: foreach (var tile in tileConfig.tiles) { var loadHandle = Addressables.LoadAssetAsync <gameobject>(tile.prefabPath); yield return loadHandle; / / Instantiate and set coordinates var tileObj = Instantiate(loadHandle.Result); tileObj.transform.position = ConvertToUnityCoord(tile.longitude,tile.latitude, tile.height); / / Merge material atlas CombineTextures(tileObj, atlasMaterial); } (4) The coordinate acquisition method used in the above method is as follows (using local coordinate system transformation): Vector3 ConvertToUnityCoord(double lon, double lat, double alt) { Var localCoord = GISUtiles.GeodeticToLcal(lon,lat,alt); Return new Vector3(localCoord.x,alt*0.2f,localCoord.y); } (5) Optimization method for memory management of oblique photogrammetry model loading: Add a pagination loading strategy to the built-in Update method of the script: Void Update(){ Vector3 cameraPos = mainCamera.transform.position; foreach (var tile in loadedTiles){ float distance = Vector3.Distance(cameraPos, tile.center); if (distance <loadThreshold&&!tile.isLoaded){ StartCoroutine(LoadTileAsync(tile)); } Else if(distance>unloadThreshold&&tile.isLoaded) { ReleaseTile(tile); } } } By following the steps above, the oblique photography model is loaded into the Unity3D project, thus achieving basic 3D scene construction.
[0021] S3. Connect to camera hardware devices to acquire real-time video stream data and parse spatial coordinate parameters.
[0022] For example, the spatial coordinate parameters include the camera's latitude and longitude coordinates, installation height, shooting angle parameters, and geographical boundary information of the area covered by the video image.
[0023] Taking a certain brand of camera as an example, the method of connecting to real-time video mainly relies on the client software, SDK (Software Development Kit), or RTSP (Real Time Streaming Protocol) video stream address provided by the brand. This embodiment mainly uses the API data provided by the client software for connection.
[0024] (1) Register an account on the platform developed by the brand and obtain the API documentation; (2) Enter the client settings and store the client IP, port, APPKey, and APPSecret information; (3) Create a Unity script plugin, implement the logic code flow according to the required API interfaces, and initialize and obtain basic camera information: First, use UnityWebRequest, Unity's built-in API request class, to construct an API request and retrieve data from the API interface. Since Unity does not have the concept of multithreading, it is recommended to use coroutines to call the API request function.
[0025] UnityWebRequest request = new UnityWebRequest(url, method); The URL is Hikvision's API request address, and the method is the request parameter data, including the request header or request body (in this method, the request header needs to include the client APPKey and APPSecret obtained in the previous step). Next, declare the list container for storing camera data as List. <camera>CameraList; Furthermore, an API request ({IP address} / api / resource / v2 / camera / search) is sent to obtain a list of client devices. The returned data is a list of camera parameters, which mainly includes the camera's unique code and camera installation location, and is stored in CameraList. Finally, declare and implement the method for obtaining the video playback stream address (wait for the right time to use, do not actively call it), reuse the method for obtaining the video stream based on the camera information in the API document of the other party in (1), select Camera as the parameter to pass, mainly use the camera's unique code and camera installation location data stored in Camera, and request the returned data as the camera's screen range and the camera's RTSP video stream address.
[0026] S4. Based on spatial coordinate parameters, establish a spatial mapping relationship between the video footage and the digital twin scene, and then display it.
[0027] First, iterate through the CameraList that stores camera information. Based on the stored camera information, use the ConvertToUnityCoord method to obtain the camera range information, set the number of camera range points (the number determines the accuracy of the generated model), and use the Rays property of Unity3D to obtain the coordinate information points Vector3[] Vertices that fit the surface of the oblique photography model. Generate the corresponding camera range model object through Vertices, and establish the GameObjectGenerateMeshFromPoints(Vector3[]vertices) method: (1) The GenerateMeshFromPoints method first creates the Mesh component: MeshFilter meshFilter = gameObject.AddComponent <meshfilter>(); MeshRenderer renderer = gameObject.AddComponent <meshrenderer>(); renderer.material = meshMaterial; (2) Initialize Mesh: Mesh mesh = new Mesh(); mesh.name = "DynamicMesh"; (3) Set the vertices: mesh.vertices = vertices; (4) Generate triangle index List <int>triangles = new List <int>(); for(int i=2; i <vertices.Length; i++) { triangles.Add(0); triangles.Add(i-1); triangles.Add(i); } (5) Closed polygon triangles.Add(0); triangles.Add(vertices.Length-1); triangles.Add(1); mesh.triangles = triangles.ToArray(); (6) Automatically calculate normals and UVs mesh.RecalculateNormals(); mesh.RecalculateTangents(); mesh.Optimize(); (7) Application Grid meshFilter.mesh = mesh; After completing the above steps, you will obtain a List. <game>CameraGameList; Next, iterate through CameraGameList, create materials that support video playback, and bind them to the model object using the method: SetCameraGameMaterial(Game targetObject){ Material videoMat = new Material(Shader.Find("Standard")); videoMat.EnableKeyword("_NORMALMAP"); videoMat.SetTexture("_MainTex", videoPlayer.targetTexture); targetObject.targetRenderer.material = videoMat; } Finally, the video is played. By using the one-to-one correspondence between CameraGameList and CameraList indexes, the RTSP video stream addresses of the cameras stored in CameraList are used to process the video images as materials through a network request. The materials of the game objects in CameraGameList are then used for playback, thereby achieving a real-time video playback effect.
[0028] It should be noted that, in one embodiment of the present invention, the step of using a drone to perform aerial surveying tasks and obtain oblique photography data of an open-pit mine includes: performing aerial surveying tasks using a drone according to a preset task cycle to obtain oblique photography data of the open-pit mine for the current task cycle. In other words, in practical applications, drones can be used for periodic aerial surveying tasks, such as once a week or once every two days, to obtain the latest topographical information in a timely manner.
[0029] The steps of establishing an oblique photogrammetry model of the open-pit mine based on the oblique photogrammetry data, loading the oblique photogrammetry model of the open-pit mine using Unity3D, and constructing a digital twin scene include: Based on the oblique photography data of the open-pit mine during the current mission period, establish the oblique photography model of the open-pit mine during the current mission period. Acquire oblique photography data of the open-pit mine from the previous task cycle; The oblique photography data of the open-pit mine in the current task cycle and the previous task cycle are converted into point cloud data. After aligning the coordinate system through the point cloud registration algorithm, the changed areas are determined. Based on the digital twin scene of the previous task cycle, the open-pit mine oblique photography model of the changing area in the current task cycle is loaded using Unity3D to obtain the digital twin scene of the current task cycle.
[0030] Figure 3 This is a schematic diagram of a real-time surface monitoring device for open-pit mines based on digital twins, provided in an embodiment of the present invention. Please refer to... Figure 3 This invention also provides a real-time monitoring device for the surface of an open-pit mine based on digital twins, comprising: The data acquisition module 310 is used to perform aerial surveying tasks using drones to obtain oblique photography data of open-pit mines; Scene building module 320 is used to build digital twin scenes based on open-pit mine oblique photogrammetry data; The parameter parsing module 330 is used to connect to the camera hardware device, acquire real-time video stream data, and parse spatial coordinate parameters. Display module 340 is used to establish a spatial mapping relationship between video footage and digital twin scene based on spatial coordinate parameters, and then display the video footage.
[0031] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows: This invention provides a method for real-time monitoring of open-pit mine surfaces based on digital twins. First, it uses drones to obtain oblique photography data of the open-pit mine. Then, it uses the Unity3D engine to construct a digital twin scene. Real-time video images are obtained through data processing via video hardware. The spatial range and location information of the video images are set, and the spatial range information of the video images is converted and combined with the spatial information of the oblique photography model of the open-pit mine. Finally, the real-time video is played, which can be intuitively displayed in the three-dimensional digital twin scene, solving the problem of real-time monitoring of the overall appearance of the mine for mine management personnel.
[0032] In the description of this invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0033] Although this application has been described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in carrying out the claimed application by reviewing the accompanying drawings, the disclosure, and the appended claims.
[0034] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.< / game> < / int> < / int> < / meshrenderer> < / meshfilter> < / camera> < / gameobject> < / tileconfig>
Claims
1. A method for real-time monitoring of the surface of an open-pit mine based on digital twins, characterized in that, include: Using drones to perform aerial surveys and obtain oblique photography data of open-pit mines; A digital twin scene was constructed based on the oblique photography data of the open-pit mine; Connect to camera hardware devices to acquire real-time video stream data and parse spatial coordinate parameters; Based on the spatial coordinate parameters, a spatial mapping relationship between the video footage and the digital twin scene is established and displayed.
2. The method for real-time surface monitoring of open-pit mines based on digital twins according to claim 1, characterized in that, The steps for constructing a digital twin scene based on the open-pit mine oblique photogrammetry data include: An oblique photography model of the open-pit mine is established based on the oblique photography data of the open-pit mine. The oblique photography model of the open-pit mine is loaded using Unity3D to construct a digital twin scene.
3. The method for real-time surface monitoring of open-pit mines based on digital twins according to claim 1, characterized in that, The spatial coordinate parameters include the camera's latitude and longitude coordinates, installation height, shooting angle parameters, and geographical boundary information of the area covered by the video image.
4. The method for real-time monitoring of open-pit mine surface based on digital twin as described in claim 2, characterized in that, The steps for using drones to perform aerial surveys and obtain oblique photogrammetry data of open-pit mines include: According to the preset mission cycle, drones are used to carry out aerial survey missions to obtain oblique photography data of open-pit mines for the current mission cycle.
5. The method for real-time monitoring of open-pit mine surface based on digital twin as described in claim 4, characterized in that, The steps of establishing an oblique photogrammetry model of the open-pit mine based on the oblique photogrammetry data, loading the oblique photogrammetry model of the open-pit mine using Unity3D, and constructing a digital twin scene include: Based on the oblique photography data of the open-pit mine during the current mission period, establish the oblique photography model of the open-pit mine during the current mission period. Acquire oblique photography data of the open-pit mine from the previous task cycle; The oblique photography data of the open-pit mine in the current task cycle and the previous task cycle are converted into point cloud data. After aligning the coordinate system through the point cloud registration algorithm, the changed areas are determined. Based on the digital twin scene of the previous task cycle, the open-pit mine oblique photography model of the changing area in the current task cycle is loaded using Unity3D to obtain the digital twin scene of the current task cycle.
6. A real-time monitoring device for the surface of an open-pit mine based on digital twins, characterized in that, include: The data acquisition module is used to perform aerial surveying tasks using drones to obtain oblique photogrammetry data of open-pit mines; The scene construction module is used to construct a digital twin scene based on the open-pit mine oblique photography data; The parameter parsing module is used to interface with camera hardware devices, acquire real-time video stream data, and parse spatial coordinate parameters. The display module is used to establish a spatial mapping relationship between the video image and the digital twin scene based on the spatial coordinate parameters, and then display the image.