A method, system and storage medium for sorting coal gangue
Through machine vision technology and deep learning algorithm combined with virtual world system, efficient and precise sorting of coal gangue is achieved, problems of low efficiency, environmental pollution and radiation risks in the existing technology are solved, and production safety and equipment efficiency are improved.
Patent Information
- Application Number
- CN202411091840.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing coal gangue sorting methods are inefficient, environmental pollution, expensive equipment and radiation risks. The labor intensity of manual sorting is high and easy to miss or miss selection.
Using machine vision technology, deep learning algorithms and 3D reconstruction technology combined with virtual world system, the coal gangue is detected and segmented through pre-trained object detection model, and the 3D model is obtained in real time, and the virtual world system controls the grabbing equipment to achieve efficient and accurate gangue sorting.
It realizes high efficiency and high-precision detection and sorting of coal gangue, optimizes the production process, avoids the environmental and health risks of manual sorting, reduces equipment costs, and improves operational safety and coordination.
Smart Images

Figure CN118594987B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of coal gangue sorting, and in particular relates to a coal gangue sorting method, system and storage medium. Background Art
[0002] In the process of coal mining, gangue is produced, also known as coal gangue, which is a solid waste with low carbon content, harder than coal and lower combustion utilization rate. Separating coal gangue from coal can improve the combustion efficiency of coal and reduce pollutant emissions during combustion.
[0003] Gangue sorting is a method of coal pre-combustion treatment, an indispensable part of coal mine production, and one of the effective methods to improve coal quality. At present, there are mainly wet sorting, dry sorting and manual sorting methods for gangue sorting. The wet sorting method mainly includes heavy medium method and jigging method. Its characteristics are to use the different densities of coal and gangue to place the raw coal in the solution, and then realize the stratified separation of coal and gangue. The equipment used in this method is large, the process is complicated, the sorting efficiency is low, and it causes pollution to the environment; the dry sorting method uses the different transmittance of coal and gangue to light and adopts dual-energy gamma-ray transmission sorting. The equipment using this method is expensive, the sorting efficiency is low, and there is radiation; in addition, manual screening of gangue has the disadvantages of harsh working environment, high labor intensity, and easy misselection or omission. Summary of the invention
[0004] The present invention provides a method, system and storage medium for sorting coal gangue. The coal gangue in the coal-producing area is detected and segmented through a trained target detection model, and the target coal gangue is captured through a virtual world system that maps the actual conditions of the coal-producing area, thereby achieving high-efficiency and high-precision detection of coal gangue, while optimizing the production process and enhancing operation safety.
[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions.
[0006] In a first aspect, the present invention provides a method for sorting coal gangue, comprising:
[0007] Acquire a visible light image and a depth image of the coal area to be detected;
[0008] The visible light image is input into the pre-trained target detection model to obtain the detection and segmentation results of the target gangue;
[0009] Obtain a real-time 3D fusion model of the target gangue based on the depth image and tracking algorithm;
[0010] The 3D fusion model of the target gangue and the information of the grabbing equipment are synchronously mapped to the virtual world system, and the grabbing equipment is controlled by the virtual world system to grab the target gangue in the actual coal output area.
[0011] Optionally, the target detection model is periodically trained, and the periodic training process includes:
[0012] Using the training sample set of the current cycle to train the target detection model of the current cycle, wherein the training sample set of the current cycle is the video frames in the acquired current visible light image and depth image;
[0013] The accuracy of the target detection model of the current cycle is compared with that of the target detection model of the previous cycle, and the target detection model with higher accuracy is used as the target detection model of the next cycle.
[0014] Optionally, the target detection model includes an improved lightweight convolutional neural network, a first single-stage detection network and a second single-stage detection network connected in sequence, wherein the improved lightweight convolutional neural network is a convolutional layer added to the double convolutional bottleneck layer in the existing lightweight convolutional neural network.
[0015] Optionally, the detection result of the target gangue is a rectangular area map of the gangue, and the segmentation result of the target gangue is a segmented image of the rectangular area map of the gangue;
[0016] The method of inputting the visible light image into the pre-trained target detection model to obtain the detection and segmentation results of the target gangue includes:
[0017] The image features of visible light images are extracted through an improved lightweight convolutional neural network, and the extracted image features are fused;
[0018] The fused image features are mapped into a rectangular area map of coal gangue through the first single-stage detection network, and the rectangular area map is instance segmented through the second single-stage detection network to distinguish different coal gangue instances and generate a segmented image.
[0019] Optionally, a real-time 3D fusion model of the target gangue is obtained according to the detection and segmentation results of the target gangue and the tracking algorithm, including:
[0020] According to the rectangular area map of the target coal gangue, the tracking algorithm is used to track the real-time position information of the target coal gangue, and the depth information and speed information of the target coal gangue are obtained according to the segmented image of the target coal gangue. The real-time position information, speed information and depth information are fused to obtain a 3D fusion model of the target coal gangue, wherein the 3D fusion model includes the speed information, depth information and position information of the target coal gangue.
[0021] Optionally, the controlling the grabbing device to grab the target gangue in the actual coal output area through the virtual world system includes:
[0022] Obtain information about each grabbing robot in the virtual world;
[0023] Obtain the working range of the current grasping robot according to the information of each grasping robot and broadcast it in the virtual world;
[0024] According to the position and size information of the current target gangue in the virtual world, it is judged whether the current grasping robot can grasp it. If it can grasp it, the grasping device in the real world is controlled to grasp it and broadcast the grasping information in the virtual world. If it cannot grasp it, the judgment cycle is continued according to the position and size information of the next target gangue in the virtual world.
[0025] In a second aspect, the present invention provides a coal gangue sorting system, comprising:
[0026] Image acquisition module: used to acquire visible light images and depth images of the coal area to be detected;
[0027] Target detection and segmentation module: used to input the visible light image into the pre-trained target detection model to obtain the detection and segmentation results of the target gangue;
[0028] 3D target acquisition module: used to obtain the real-time 3D fusion model of target gangue based on the depth image and tracking algorithm;
[0029] Gangue grabbing module: used to synchronously map the 3D fusion model of the target gangue and the information of the grabbing equipment to the virtual world system, and control the grabbing equipment to grab the target gangue in the actual coal output area through the virtual world system.
[0030] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the coal gangue sorting method described in any one of the first aspects is implemented.
[0031] In a fourth aspect, the present invention provides an electronic terminal, comprising a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the coal gangue sorting method described in any one of the first aspects is executed.
[0032] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the coal gangue sorting method described in any one of the first aspects.
[0033] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention realizes high-efficiency and high-precision identification and sorting of coal gangue by integrating machine vision technology, deep learning algorithm, 3D reconstruction technology and virtual world system. Compared with the traditional manual detection method, the present invention effectively avoids the problems of harsh working environment, high labor intensity, low efficiency, misselection or missed selection, and at the same time, compared with gamma ray detection and X-ray detection, the present invention avoids expensive equipment costs, potential radiation hazards and dependence on the particle size of materials; the visible light image and depth information obtained by the depth camera, combined with the improved deep learning algorithm and the integration of the self-training module, enable the system to have the ability of continuous learning and self-optimization, can continuously adjust and improve the recognition accuracy according to real-time data, can accurately identify and segment coal mines, gangue and debris, and realize rapid detection and positioning of targets; the accuracy of target recognition is further improved by 3D reconstruction technology, so that the system can more accurately handle coal gangue sorting tasks in complex scenes; through the introduction of the virtual world system, a high-performance platform is provided for information synchronization and mutual cooperation between devices, which greatly improves the real-time and collaborative nature of the operation. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Shown is a flow chart of a method for sorting coal gangue in one embodiment of the present invention;
[0035] Figure 2 Shown is a framework diagram of module interaction in a method for sorting coal gangue in an embodiment of the present invention;
[0036] Figure 3 The figure shows a schematic diagram of the overall architecture of a target detection model in an embodiment of the present invention;
[0037] Figure 4 The figure is a schematic diagram of the structure of an improved bottleneck convolution module in a deep learning algorithm in an embodiment of the present invention;
[0038] Figure 5 The figure is a schematic diagram of the structure of the bottleneck layer in the deep learning algorithm in one embodiment of the present invention;
[0039] Figure 6 Shown is a self-training flow chart of a target detection model in one embodiment of the present invention;
[0040] Figure 7 Shown is a flow chart of using a tracking algorithm to track target coal gangue in one embodiment of the present invention;
[0041] Figure 8 It is a flow chart of grabbing target gangue through a virtual world system in one embodiment of the present invention;
[0042] Fig. 9 Shown is a flow chart of registering a virtual world system in an embodiment of the present invention;
[0043] Fig.10 Shown is a diagram of the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0045] Example 1
[0046] like Figure 1 As shown, this embodiment provides a method for sorting coal gangue, comprising:
[0047] Acquire a visible light image and a depth image of the coal area to be detected;
[0048] The visible light image is input into the pre-trained target detection model to obtain the target gangue to be identified and segmented;
[0049] Obtain a real-time 3D fusion model of the target gangue based on the depth image and tracking algorithm;
[0050] The 3D fusion model of the target gangue and the information of the grabbing equipment are synchronously mapped to the virtual world system, and the grabbing equipment is controlled by the virtual world system to grab the target gangue in the actual coal output area.
[0051] Example 2
[0052] Based on Example 1, this example also makes the following design.
[0053] like Figure 2 As shown, the gangue sorting method in Example 1 is realized by building and interacting a hardware system and a software system. The hardware system includes an external data acquisition module and a capture module, and the software system includes an algorithm module, a streaming media module, a 3D simulation module, and a virtual world system, which are specifically described as follows.
[0054] The architecture of the target detection model in the present invention is as follows Figure 3As shown in the figure, a lightweight CNN is used as an encoder to extract image features. The standard convolutional bottleneck module C2f is replaced with an improved convolutional bottleneck module. More features of the previous and next layers are extracted by adding convolutional layers, and the feature map is fed to the detection decoder for detection tasks. The detection decoder is based on the current best-performing single-stage detection network design, which is faster than the two-stage detection network, and the grid-based prediction mechanism is more relevant to the other two semantic segmentation tasks. The segmentation decoder uses the detection image output by the detection decoder to complete pixel-by-pixel semantic prediction.
[0055] The target detection model is divided into a backbone network structure and a feature extraction and prediction structure. The backbone network structure is used to preprocess the input data, and the feature extraction and prediction structure is used to perform feature extraction and target detection on the preprocessed data. The specific description of the two structures is as follows.
[0056] 1. Backbone network structure
[0057] First layer: The input first passes through a convolutional layer Conv with 64 output channels, a convolution kernel size of 3, and a stride of 2, and is downsampled to halve the size of the feature map.
[0058] The second layer: convolution layer Conv, the number of output channels is 128, the convolution kernel size is 3, the step size is 2, downsampling is performed again, and the size of the feature map is halved again.
[0059] The third layer: an improved convolutional bottleneck module, repeated 3 times, with an output channel number of 128 and a residual connection.
[0060] The fourth layer: convolution layer Conv, the number of output channels is 256, the convolution kernel size is 3, the step size is 2, and downsampling is performed.
[0061] The fifth layer: an improved convolutional bottleneck module, repeated 6 times, with an output channel number of 256, and also using residual connections.
[0062] The sixth layer: convolution layer Conv, the number of output channels is 512, the convolution kernel size is 3, the step size is 2, and downsampling is performed.
[0063] Layer 7: Improved convolutional bottleneck module, repeated 6 times, with 512 output channels and residual connection.
[0064] The eighth layer: convolution layer Conv, the number of output channels is 1024, the convolution kernel size is 3, the step size is 2, and downsampling is performed.
[0065] Layer 9: Improved convolutional bottleneck module, repeated 3 times, with 1024 output channels and residual connection.
[0066] The tenth layer: spatial pyramid pooling SPPF module, the number of input channels is 1024, and the pyramid level is 5.
[0067] 2. Feature extraction prediction structure
[0068] Upsampling and feature fusion: The output of the tenth layer in the backbone network structure first passes through an upsampling layer Upsample to enlarge the feature map size by 2 times, uses the nearest neighbor interpolation method, and then concatenates the upsampled features with the seventh layer features in the backbone network.
[0069] Convolutional module: This is followed by an improved convolutional bottleneck module, repeated 3 times with an output channel number of 512.
[0070] Second upsampling and feature fusion: Another upsampling layer Upsample is used to enlarge the feature map size by 2 times, and the upsampled features are concatenated with the fifth layer features in the backbone network.
[0071] Detect1 convolutional module: followed by an improved convolutional bottleneck module, repeated 3 times with an output channel number of 256.
[0072] Segmentation module: Then it passes through the mask protection layer MaskProt, calculates the mask matrix, and the output channel is 256. This part corresponds to the processing of the fifth layer feature map and outputs the segmentation feature map.
[0073] Downsampling and feature fusion: Downsampling is performed through another convolution layer Conv with an output channel number of 256, a convolution kernel size of 3, and a step size of 2. The downsampled features are then concatenated with the seventh layer features in the backbone network.
[0074] Detect2 convolution module: Next is the improved bottleneck convolution module, which is repeated 3 times with an output channel number of 512. This part corresponds to the processing of the seventh layer feature map.
[0075] Second downsampling and feature fusion: Pass through a convolution layer again with an output channel number of 512, a convolution kernel size of 3, and a step size of 2 for downsampling, and concatenate the downsampled features with the feature output of the tenth layer in the backbone network.
[0076] Detect3 convolution module: Next is the improved bottleneck convolution module, which is repeated 3 times with an output channel number of 1024. This part corresponds to the processing of the tenth layer feature map.
[0077] Detection prediction: The feature maps output by the fifth, seventh, and tenth layers are concatenated and passed through the non-maximum suppression algorithm NMS to obtain the target detection data.
[0078] Segmentation prediction: The target detection data is multiplied by the segmentation feature map output by the mask protection layer MaskProt through the multiplication layer Mul to obtain the final segmentation result.
[0079] like Figure 4 As shown in the figure, the improved bottleneck convolution module adds a convolution layer based on the standard bottleneck convolution module, so that the model can extract more features of the image, which is described in detail as follows.
[0080] The input first passes through the first convolutional layer Conv, using 1x1 convolution to convert the input channel number c1 into the number of intermediate channels; three outputs are separated through the split layer Split, one enters the triple bottleneck layer bottleneck to obtain three outputs, another enters the convolutional layer Conv, and the last one is directly output; a total of 5 outputs are merged and passed through a convolutional layer Conv operation to unify the depth.
[0081] like Figure 5 As shown, the bottleneck layer is used to extract the features of the model, which is described in detail as follows.
[0082] The input first passes through a convolutional layer Conv to increase the depth of the feature with a step size of 1; then passes through another convolutional layer Conv to reduce the depth of the feature with a step size of 1; finally, the initial input is merged with the final input.
[0083] In this embodiment, the target detection model can continuously optimize and improve the system's image recognition capabilities through a continuous autonomous learning process, such as Figure 6 As shown, the process of model self-training is as follows.
[0084] Continuous video acquisition: After the system is deployed, it automatically starts to collect video data from the streaming module without interruption, ensuring real-time monitoring of the site conditions, and automatically triggers image frame extraction according to preset parameters or work shift rotation.
[0085] Automatic image extraction and preprocessing: The module extracts key frames from the video stream according to a predetermined strategy, and improves the image quality through a preprocessing algorithm to provide clear image data for the subsequent self-labeling process.
[0086] Self-labeling algorithm: Automatically label the extracted images using advanced machine learning models that can identify and mark objects, scenes or anomalies in the images, reducing reliance on manual labeling.
[0087] Manual review and correction: After the automatic labeling is completed, the system will submit the labeling results to the background manual reviewer for verification. The reviewer can correct or delete the labeling results to ensure the accuracy and reliability of the labeling data.
[0088] Training task scheduling: Data that has been manually reviewed is sent to the training task queue. The system intelligently schedules training tasks, giving priority to periods of low computing resource load to optimize resource utilization and training efficiency.
[0089] Model fine-tuning and verification: The system uses fine-tuning training methods to iteratively optimize the model, and ensures continuous improvement in model accuracy through regular verification and performance evaluation.
[0090] Automatic model update: When the newly trained model surpasses the original model in performance, the system automatically deploys the new model as the currently used model to achieve real-time model update and optimization.
[0091] Adaptive learning: The module has adaptive learning capabilities and can adjust learning strategies according to the on-site environment and task requirements to improve the generalization and adaptability of the model.
[0092] Model self-training not only significantly improves the system's automation level and intelligent recognition capabilities, but also ensures that the model continues to adapt to on-site changes, providing users with more accurate and reliable image recognition services.
[0093] In this embodiment, in order to lock and track the detected target in real time, a tracking algorithm is introduced. At the same time, combined with depth camera technology, the speed and shape information of the target coal gangue is accurately obtained, which greatly improves the dynamic analysis capability of the system and ensures the efficiency and accuracy of coal mine product processing, thereby optimizing the production process and enhancing operation safety.
[0094] like Figure 7 As shown, the process of using the tracking algorithm to locate the target gangue in real time is as follows.
[0095] Initialize the tracking process, set the tracking matching threshold T1 and the tracking life threshold D1;
[0096] Obtaining the rectangular coordinates of the gangue according to the rectangular area map of the gangue;
[0097] Sort the rectangular coordinates of the gangue from high to low according to the matching degree, and give priority to the most likely matches;
[0098] Predict the speed of the existing tracking ID, and use the current detection result to calculate the intersection and union ratio with the existing result. For the detection result whose matching degree exceeds the matching degree threshold T1, update the corresponding tracking ID to the current coordinate;
[0099] Use the unmatched detection results and the unmatched parts of the existing results to calculate the center point distance. For the detection results whose matching degree exceeds the matching degree threshold T1, update the corresponding tracking ID to the current coordinates;
[0100] Set the detection results that do not match any existing tracking ID twice as the new tracking ID and start a new tracking;
[0101] Check the tracking life of each tracking ID. If a tracking ID exceeds the tracking life threshold D1, delete the tracking ID, and combine the depth information and speed information to perform 3D reconstruction on the tracking ID to generate a 3D fusion model.
[0102] After tracking the target gangue location information, the speed and depth information of the target gangue at that location is obtained through the depth camera, which is fused with the location information into a 3D fusion model and then transmitted to the virtual world. The 3D fusion model includes the speed information, depth information, and location information of the target gangue.
[0103] In this embodiment, the external data acquisition module is a multifunctional sensor system consisting of a depth camera and surrounding surveillance cameras, and is designed specifically for on-site environment monitoring and object recognition.
[0104] The external data acquisition module mainly includes: integrated depth camera: the depth camera combines lidar and visible light camera technology to obtain accurate depth information of the monitored object in real time, providing important three-dimensional data for subsequent image processing and analysis; multi-angle monitoring: the surrounding surveillance cameras capture scene images from different angles, which are combined with the depth camera data to provide the system with comprehensive visual information and enhance the understanding of the on-site environment.
[0105] The following steps are performed through the external data acquisition module: Real-time data fusion: Data fusion technology is used to fuse depth information and visible light images in real time to improve the accuracy of object recognition and scene reconstruction; Height perception and measurement: By sensing the height information of objects such as gangue and performing precise measurements, key parameters are provided for the grasping module to ensure that the robotic arm can accurately judge the lower claw distance when performing grasping tasks.
[0106] In order to adapt to various environmental conditions, including changes in lighting, weather influences, etc., protection devices are set up for the integrated depth camera and multi-angle monitoring to ensure stable operation under various conditions. Taking data security and privacy protection into consideration, the collected data is ensured to be safe and reliable during transmission and storage to prevent data leakage.
[0107] In this embodiment, the grabbing module is the main equipment for grabbing coal gangue. It is composed of a precision mechanical arm and an advanced controller. After registering in the virtual world, the visible light image and depth image transmitted by the streaming module can be used to obtain the position of the target to be grabbed in real time, and the existence of other grabbing devices can be obtained in real time to avoid collision. The controller is synchronously linked with the virtual world system to realize real-time data exchange and intelligent operation. Figure 8The main steps of grabbing the target gangue through this module are: judging whether the current robot arm can grab it according to the position and size information of the current target gangue in the virtual world; if it can be grabbed, controlling the robot arm to grab and broadcast the grabbing information in the virtual world; if it cannot be grabbed, continuing the judgment cycle according to the position and size information of the next target gangue in the virtual world.
[0108] Real-time information acquisition: The controller obtains key information from the virtual world system in real time, including the position and motion status of other robotic arms, the position and shape of gangue and coal blocks, etc., to ensure the accuracy and safety of the robotic arms when performing tasks.
[0109] Collaborative work: The robotic arms can share information and work together through the virtual world system, avoiding repeated grasping and collisions, and improving work efficiency and safety.
[0110] Intelligent path planning: The controller has a built-in intelligent path planning algorithm to plan the optimal movement and grasping path for the robot arm, reducing operation time and improving operation efficiency.
[0111] Adaptive grasping: The robotic arm is equipped with advanced sensors and actuators, which can adaptively adjust the grasping force and angle according to the shape, hardness and other information of gangue and coal blocks to ensure the stability and success rate of the grasping process.
[0112] Remote monitoring and control: Supports remote monitoring and control. Operators can monitor the status of the robotic arm in real time at the control center and perform remote operations. It is suitable for complex or unsafe working environments.
[0113] Fault diagnosis and maintenance: The system has a self-diagnosis function, which can monitor the status of the robot arm and controller in real time, and promptly alarm when a fault occurs, while providing maintenance suggestions to ensure the stable operation of the system.
[0114] Task scheduling and management: The module supports task scheduling and management functions, can automatically assign tasks according to work requirements, and monitor and manage task execution in real time.
[0115] In this embodiment, the streaming media module is responsible for the distribution and management of the video stream. The common surveillance cameras of the acquisition module are multiplexed after passing through the streaming media module. Multiple users can view the surveillance in real time. The following contents are realized through the streaming media module.
[0116] Video acquisition and storage: Real-time acquisition of high-definition video streams, and efficient compression and encoding for easy storage and transmission. The storage system uses advanced redundancy technology to ensure data security and reliability.
[0117] Real-time streaming transmission: supports the transmission of real-time video streams, allowing remote users to watch on-site conditions in real time via the Internet, suitable for remote monitoring and command and dispatch.
[0118] On-demand playback function: Users can access historical video data at any time through on-demand services, supporting multiple resolutions and bit rates to meet the needs of different network environments and devices.
[0119] Multi-screen monitoring: Supports multi-screen display, users can view multiple video sources on the same screen at the same time, facilitating centralized monitoring and management.
[0120] The streaming media module in this embodiment takes into account the integration with other systems, supports multiple standards and protocols, and ensures compatibility with existing video surveillance systems.
[0121] In this embodiment, the virtual world system is a high-performance synchronization system, which can achieve mutual cooperation and communication between devices through module registration. The virtual world system has a high synchronization frequency per second, synchronizes information in the virtual world with high real-time performance, and reduces communication consumption between devices. All messages are sent through the virtual world, and it also has a multiple confirmation mechanism to ensure that each device can accurately receive information in the virtual world; and through multi-redundancy design, a main world system and an inner world system are synchronized in real time to ensure that a single system can continue to work normally even if it is accidentally interrupted.
[0122] The virtual world main system is a highly integrated platform that not only provides basic functions such as inter-module communication, virtual world synchronization, module registration and module login, but also includes the following functions and features:
[0123] Distributed computing framework: The system adopts a distributed computing framework that can handle large-scale data and complex computing tasks, ensuring the efficient operation and scalability of the system.
[0124] Real-time data synchronization: The virtual world main system supports real-time data synchronization, ensuring that all modules can access the latest data, improving system response speed and decision-making accuracy.
[0125] 3D model rendering and management: The system has powerful 3D model rendering capabilities and can display the 3D model of the site in real time, making it easier for operators to intuitively understand the site environment and equipment status.
[0126] Intelligent decision support: The virtual world main system integrates an intelligent decision support system that can provide decision suggestions based on real-time data and preset rules, assisting operators to make faster and more accurate decisions.
[0127] Multi-user collaboration platform: The system supports multiple users to log in and collaborate at the same time. Different users can share data and resources and work together to complete complex tasks.
[0128] Security and access control: The virtual world main system has strict security and access control mechanisms to ensure that only authorized users can access system resources and data, protecting the security and stability of the system.
[0129] Historical data management: The system provides historical data management functions that can store, retrieve and analyze historical data to provide a basis for subsequent decision-making and optimization.
[0130] Custom module development: The virtual world main system supports the development and integration of custom modules, allowing users to develop new modules according to specific needs and further expand the functions and application scope of the system.
[0131] Through the integration of the above functions, the virtual world main system not only ensures data consistency and system stability, but also provides users with a powerful, flexible and secure platform to support the execution of complex tasks and decision-making.
[0132] like Fig. 9 As shown, the device registration process of the virtual world system is as follows.
[0133] Receive a registration signal from a device in the real world;
[0134] Get the registration information of the device;
[0135] Map the device in the virtual world according to its registration information;
[0136] Broadcast device information in the virtual world and perform continuous heartbeat maintenance.
[0137] In this embodiment, the 3D simulation module is mainly responsible for image rendering. By acquiring the information of the equipment and the information of the coal gangue, a 3D picture is generated in real time, allowing people to intuitively see the working status of the equipment.
[0138] Example 3
[0139] This embodiment provides a coal gangue sorting system, comprising:
[0140] Image acquisition module: used to acquire visible light images and depth images of the coal area to be detected;
[0141] Target detection and segmentation module: used to input the visible light image into the pre-trained target detection model to obtain the detection and segmentation results of the target gangue;
[0142] Three-dimensional target acquisition module: used to obtain a real-time 3D fusion model of the target gangue based on the depth image and tracking algorithm, wherein the 3D fusion model includes the speed information, depth information, and position information of the target gangue;
[0143] Gangue grabbing module: used to synchronously map the 3D fusion model of the target gangue and the information of the grabbing equipment to the virtual world system, and control the grabbing of the target gangue in the actual coal output area through the virtual world system.
[0144] The gangue sorting system provided in the embodiment of the present invention can execute the gangue sorting method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.
[0145] Example 4
[0146] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the following method are implemented:
[0147] Acquire a visible light image and a depth image of the coal area to be detected;
[0148] The visible light image is input into the pre-trained target detection model to obtain the target gangue to be identified and segmented;
[0149] Obtain a real-time 3D fusion model of the target gangue based on the depth image and tracking algorithm;
[0150] The 3D fusion model of the target gangue and the information of the grabbing equipment are synchronously mapped to the virtual world system, and the grabbing equipment is controlled by the virtual world system to grab the target gangue in the actual coal output area.
[0151] Example 5
[0152] The embodiment of the present invention further provides a computer device, which may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data obtained and generated in the method of the robot autonomously entering the packaging container. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the method of the aforementioned embodiment 2 is implemented.
[0153] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0154] The computer device provided in the embodiment of the present invention can execute the coal gangue sorting method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0155] Example 6
[0156] This embodiment provides a computer program product, including a computer program / instruction, which implements the steps of the following method when executed by a processor:
[0157] Acquire a visible light image and a depth image of the coal area to be detected;
[0158] The visible light image is input into the pre-trained target detection model to obtain the target gangue to be identified and segmented;
[0159] Obtain a real-time 3D fusion model of the target gangue based on the depth image and tracking algorithm;
[0160] The 3D fusion model of the target gangue and the information of the grabbing equipment are synchronously mapped to the virtual world system, and the grabbing equipment is controlled by the virtual world system to grab the target gangue in the actual coal output area.
[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0165] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.
Claims
1. A method for sorting coal gangue, characterized in that: include: Acquire a visible light image and a depth image of the coal area to be detected; The visible light image is input into the pre-trained target detection model to obtain the detection and segmentation results of the target gangue; Obtain a real-time 3D fusion model of the target gangue based on the detection and segmentation results of the target gangue and the tracking algorithm; The 3D fusion model of the target gangue and the information of the grabbing equipment are synchronously mapped to the virtual world system, and the grabbing equipment is controlled by the virtual world system to grab the target gangue in the actual coal output area; The target detection model is periodically trained, and the periodic training process includes: Using the training sample set of the current cycle to train the target detection model of the current cycle, wherein the training sample set of the current cycle is the video frames in the acquired current visible light image and depth image; Compare the accuracy of the target detection model of the current cycle with that of the target detection model of the previous cycle, and use the target detection model with higher accuracy as the target detection model of the next cycle; The target detection model includes an improved lightweight convolutional neural network, a first single-stage detection network, and a second single-stage detection network connected in sequence; wherein the improved lightweight convolutional neural network is to add a convolutional layer to the double convolutional bottleneck layer in the existing lightweight convolutional neural network; The detection result of the target gangue is a rectangular area map of the gangue, and the segmentation result of the target gangue is a segmented image of the rectangular area map of the gangue; The step of inputting the visible light image into a pre-trained target detection model to obtain the detection and segmentation results of the target gangue includes: The image features of visible light images are extracted through an improved lightweight convolutional neural network, and the extracted image features are fused; The fused image features are mapped into a rectangular area map of gangue through the first single-stage detection network, and the rectangular area map is instance segmented through the second single-stage detection network to distinguish different gangue instances and generate a segmented image; The method of controlling the grabbing device through the virtual world system to grab the target gangue in the actual coal output area includes: Obtain information about each grabbing robot in the virtual world; Obtain the working range of the current grasping device according to the information of each grasping device and broadcast it in the virtual world; According to the position and size information of the current target coal gangue in the virtual world, it is judged whether the current grabbing device can grab it. If it can grab it, the grabbing device in the real world is controlled to perform the grabbing work and broadcast the grabbing information in the virtual world. If it cannot grab it, the judgment cycle is continued according to the position and size information of the next target coal gangue in the virtual world.
2. The method for sorting coal gangue according to claim 1, characterized in that: According to the detection and segmentation results of the target gangue and the tracking algorithm, a real-time 3D fusion model of the target gangue is obtained, including: According to the rectangular area map of the target coal gangue, the tracking algorithm is used to track the position information of the target coal gangue, and the depth information and speed information of the target coal gangue are obtained according to the segmented image of the target coal gangue. The position information, speed information and depth information are fused to obtain a 3D fusion model of the target coal gangue.
3. A coal gangue sorting system, characterized in that: include: Image acquisition module: used to acquire visible light images and depth images of the coal area to be detected; Target detection and segmentation module: used to input the visible light image into the pre-trained target detection model to obtain the detection and segmentation results of the target gangue; 3D target acquisition module: used to obtain the real-time 3D fusion model of target gangue based on the depth image and tracking algorithm; Gangue grabbing module: used to synchronously map the 3D fusion model of the target gangue and the information of the grabbing equipment to the virtual world system, and control the grabbing equipment to grab the target gangue in the actual coal output area through the virtual world system; The target detection model is periodically trained, and the periodic training process includes: Using the training sample set of the current cycle to train the target detection model of the current cycle, wherein the training sample set of the current cycle is the video frames in the acquired current visible light image and depth image; Compare the accuracy of the target detection model of the current cycle with that of the target detection model of the previous cycle, and use the target detection model with higher accuracy as the target detection model of the next cycle; The target detection model includes an improved lightweight convolutional neural network, a first single-stage detection network, and a second single-stage detection network connected in sequence; wherein the improved lightweight convolutional neural network is to add a convolutional layer to the double convolutional bottleneck layer in the existing lightweight convolutional neural network; The detection result of the target gangue is a rectangular area map of the gangue, and the segmentation result of the target gangue is a segmented image of the rectangular area map of the gangue; The step of inputting the visible light image into a pre-trained target detection model to obtain the detection and segmentation results of the target gangue includes: The image features of visible light images are extracted through an improved lightweight convolutional neural network, and the extracted image features are fused; The fused image features are mapped into a rectangular area map of gangue through the first single-stage detection network, and the rectangular area map is instance segmented through the second single-stage detection network to distinguish different gangue instances and generate a segmented image; The method of controlling the grabbing device through the virtual world system to grab the target gangue in the actual coal output area includes: Obtain information about each grabbing robot in the virtual world; Obtain the working range of the current grasping device according to the information of each grasping device and broadcast it in the virtual world; According to the position and size information of the current target coal gangue in the virtual world, it is judged whether the current grabbing device can grab it. If it can grab it, the grabbing device in the real world is controlled to perform the grabbing work and broadcast the grabbing information in the virtual world. If it cannot grab it, the judgment cycle is continued according to the position and size information of the next target coal gangue in the virtual world.
4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the coal gangue sorting method as claimed in claim 1 or 2 is implemented.
5. An electronic terminal, characterized in that: It comprises a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the coal gangue sorting method as claimed in claim 1 or 2 is executed.
6. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the coal gangue sorting method described in claim 1 or 2 is implemented.
Citation Information
Patent Citations
Coal gangue sorting method and device, equipment and storage medium
CN111346842A
Coal gangue sorting system based on multi-modal imaging analysis
CN114515705A
