Hidden dangerous target real-time identification method and system for unmanned bulldozer in mining area

Abstract

The invention belongs to the technical field related to surface mine unmanned driving, and discloses a hidden dangerous target real-time identification method and system for an unmanned bulldozer in a mining area. The method comprises the following steps: acquiring images of a surrounding working environment of the unmanned bulldozer in the mining area, and marking the type, the position and the pixel size of a hidden dangerous target in each image so as to form a data set about the hidden dangerous target; using the data set to train a hidden dangerous target detection model until the precision index of the target detection model converges, the input of the hidden dangerous target detection model being an image, and the output being the type, position and pixel size of a hidden dangerous target; and inputting a to-be-detected image into the trained hidden dangerous target detection model to obtain the type, position and pixel size of a hidden dangerous target in the to-be-detected image. According to the invention, the problem of identifying hidden dangerous targets in complex scenes and severe environments is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to unmanned driving in open-pit mines, and more specifically, relates to a method and system for real-time identification of hidden dangerous targets for unmanned bulldozers in mining areas. Background Art

[0002] In recent years, with the rapid development of intelligent mining technology, construction machinery in open-pit mines has gradually moved towards unmanned and automated operations. The application of unmanned bulldozers has not only significantly reduced labor costs, but also effectively improved production efficiency and safety, and promoted the upgrading of the mining industry. However, the complex geographical environment and operating conditions in open-pit mines have brought many challenges to the application of unmanned driving technology, especially during the driving and operation of bulldozers, which often face the risk of potential hidden dangerous targets, such as boulders, pits, and accumulated water. These dangerous targets are not easy to detect and may be mixed in the materials or hidden under the ground, which can easily lead to safety accidents such as overturning, getting stuck, and collisions of bulldozers.

[0003] Traditional hidden target detection technology in mining areas mainly relies on LiDAR or a single visual perception device. Although LiDAR can provide distance information, it is difficult to identify hidden dangerous targets in complex scenes. Technologies that rely solely on visual detection often have problems such as low target recall rate and slow recognition speed. Especially in harsh environments such as dust and sand in mining areas, the contrast between the target and the background is low, which increases the difficulty of detection. Therefore, a real-time, accurate and reliable hidden dangerous target recognition system is urgently needed to ensure the safe operation of unmanned bulldozers. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a method and system for real-time identification of hidden dangerous targets for unmanned bulldozers in mining areas, which solves the problem of identifying hidden dangerous targets in complex scenes and harsh environments.

[0005] To achieve the above object, according to one aspect of the present invention, a method for real-time identification of hidden dangerous targets for unmanned bulldozers in mining areas is provided, the method comprising the following steps:

[0006] Collect images of the working environment around the unmanned bulldozer in the mining area, and annotate the type, location and pixel size of hidden dangerous targets in each image to form a data set about hidden dangerous targets;

[0007] Using the data set to train a hidden dangerous target detection model, the training is performed until the accuracy index of the target detection model converges, the input of the hidden dangerous target detection model is an image, and the output is the type, position and pixel size of the hidden dangerous target;

[0008] The image to be detected is input into the trained hidden dangerous target detection model to obtain the type, position and pixel size of the hidden dangerous target in the image to be detected.

[0009] Further preferably, after acquiring images of the surrounding working environment of the unmanned bulldozer in the mining area and marking the type, position and pixel size of the hidden dangerous targets in each image, a database of pixel size and actual size mapping is constructed according to the correspondence between the pixel size and the actual size of the hidden dangerous targets in each image.

[0010] Further preferably, a fast estimation model is trained using the database of pixel size and actual size, the input of the fast estimation model is a feature vector of pixel size, and the output is the actual size. The formula of the fast estimation model is as follows:

[0011]

[0012] Among them, W is the weight matrix, b is the bias vector, x(i) is the eigenvector of the i-th sample, y(i) is the probability interval distribution of the actual size corresponding to the i-th sample, and the highest probability interval is taken as the actual size.

[0013] Further preferably, after calculating and obtaining the actual size of the hidden dangerous target in the image to be detected, an early warning is issued according to the actual size in a preset early warning manner, and the preset early warning manner is as follows:

[0014] a) When X≤H / 2, it is a safe state;

[0015] b) When H / 2<X≤H, it is a first-level warning;

[0016] c) When H<X, it is a level 2 alarm;

[0017] Among them, H is the track width of the unmanned bulldozer, and X is the actual size of the hidden dangerous target.

[0018] Further preferably, when the first-level warning is given, the unmanned bulldozer in the mining area brakes and avoids obstacles, and when the second-level warning is given, the unmanned bulldozer in the mining area is remotely taken over and controlled.

[0019] Further preferably, the hidden dangerous target detection model is a model combining GHostNet and a bio-inspired algorithm, wherein GHostNet is used as a backbone network and the bio-inspired algorithm is used as a method for searching for an optimal solution.

[0020] Further preferably, the hidden dangerous target is one or a combination of isolated rocks, deep pits and stagnant water.

[0021] According to another aspect of the present invention, a system for performing recognition using the above-mentioned recognition method is provided, the system comprising an image acquisition module, a training module and a testing module, wherein:

[0022] The image acquisition module is used to collect images of the surrounding working environment of the unmanned bulldozer in the mining area, and to mark the type, position and pixel size of hidden dangerous targets in each image;

[0023] The training module is used to train a hidden dangerous target detection model using the data set, and train until the accuracy index of the target detection model converges;

[0024] The testing module is used to input the image to be detected into the trained hidden dangerous target detection model to obtain the type, position and pixel size of the hidden dangerous target in the image to be detected.

[0025] According to another aspect of the present invention, a system for real-time identification of hidden and dangerous targets for unmanned bulldozers in mining areas is provided, wherein the system includes an actuator for executing the above-mentioned method for real-time identification of hidden and dangerous targets for unmanned bulldozers in mining areas.

[0026] According to another aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for real-time identification of hidden dangerous targets for an unmanned bulldozer in a mining area as described above is implemented.

[0027] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0028] 1. In open-pit mining scenarios, boulders are often mixed in with materials and obscured by them, while deep pits and puddles are located below the ground and are not easily found. These targets are similar in color to the background and have similar textures, resulting in a small difference between the target and the background, which increases the difficulty of identification. In addition, when bulldozers are working in open-pit mines, a large amount of dust will be generated, and the mining area is easily disturbed by wind and sand, which further aggravates the concealment of the target. The hidden dangerous target detection model proposed in the present invention can quickly and efficiently identify hidden dangerous targets in complex scenarios and harsh environments.

[0029] 2. The hidden dangerous target detection model of the present invention is a model that combines GHostNet and a bio-inspired algorithm. GHostNet is used as the backbone network, and the bio-inspired algorithm performs well in hidden target detection, and is very suitable for hidden target detection in mining areas. The hidden target detection model in mining areas built based on SiNetv2 can effectively detect hidden targets such as isolated rocks, deep pits and puddles in open-pit mining scenes.

[0030] 3. The present invention proposes a fast size estimation model based on Softmax regression, which can quickly calculate the true size of the target by combining its pixel size and actual size after detecting the hidden dangerous target in real time.

[0031] 4. According to the comparison of the width of the bulldozer track, the system divides the target size into multiple danger levels (safety, first-level alarm, and second-level alarm) and makes corresponding operation decisions: if the size is small, the bulldozer can pass safely; if the target size is large, the system will automatically issue a brake or obstacle avoidance command; if the size exceeds the safety range, it will trigger remote control to prevent the bulldozer from falling into a serious accident. This early warning mechanism ensures the safe operation of unmanned bulldozers in the complex environment of mining areas and can respond quickly to dangerous targets.

[0032] 5. The present invention utilizes 360° panoramic video acquisition to eliminate the blind spot problem caused by the large size and working posture of the bulldozer. This innovative design ensures all-round safety monitoring of the bulldozer in complex mining environments, and is suitable for unmanned operation scenarios in open-pit mines, greatly improving operation safety and work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a method for real-time identification and early warning of hidden dangerous targets for unmanned bulldozers in mining areas constructed according to a preferred embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a constructed GHostNet backbone network constructed in accordance with a preferred embodiment of the present invention;

[0035] Figure 3 It is a schematic diagram of the process of identifying hidden dangerous targets constructed according to the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] like Figure 1 As shown, a real-time identification method and alarm system for hidden dangerous targets of unmanned bulldozers in mining areas include a panoramic surround video acquisition system, a multi-task detection system for hidden dangerous targets, and a central alarm and remote control system.

[0038] Panoramic video acquisition system:

[0039] S1 deploys two forward-looking cameras on both sides of the front of the unmanned bulldozer body to obtain scene information in front of the bulldozer shovel; deploys four mining monocular cameras on the top of the unmanned bulldozer cabin to obtain environmental information at a longer distance around the unmanned bulldozer; deploys four mining explosion-proof wide-angle cameras around the unmanned bulldozer cabin, and deploys one camera at the bottom of the towing hook to obtain rich scene information of the close-range blind spot. These cameras are used to collect video data of the bulldozer in real time when it is driving and working. At the same time, based on the collected video data, the hidden and dangerous targets in the image are segmented and labeled by instance. In this embodiment, manual labeling is used, and then the pixel size is obtained to form a hidden and dangerous target data set in the mining scene.

[0040] In one embodiment of the present invention, a panoramic video acquisition system includes:

[0041] (1) Taking into account the complex environmental conditions in the mining area and the blind spots of large mining machinery, the camera was selected. The forward-looking camera has better long-distance detection capabilities, and the wide-angle camera has a better viewing angle and less distortion.

[0042] (2) Select a storage device of sufficient size to store video data and connect the storage device to the camera so that the data collected by each data source can be viewed and traced in real time; select an edge computing unit with sufficient performance to detect the target.

[0043] (3) Label the hidden dangerous targets such as isolated rocks, deep pits, and accumulated water in the collected images, and classify them into categories 1, 2, and 3 respectively; other background pixels in the image do not need to be labeled and are classified as 0.

[0044] After completing the deployment of the edge computing unit, S2 uses the biological heuristic search and recognition network SiNetv2 to build a detection model for hidden dangerous targets such as boulders, potholes, and waterlogged objects. In open-pit mining scenarios, these targets are easily blocked by materials or hidden below the ground, and have a texture similar to that of the background with low contrast. In addition, a large amount of dust and sand in the environment can also cause serious interference. Using the biological heuristic algorithm SiNetv2, the problem of hidden target detection in mining areas is solved in a targeted manner. The input of the CNN hidden dangerous target detection model is an image, and the output is three hidden dangers: boulders, potholes, or waterlogged objects, with a pixel size of

[0045] like Figure 2 As shown in the figure, the backbone network of the hidden target detection model is modified to GHostNet, and the advanced lightweight backbone network GHostNet is used to quickly extract the feature information of the hidden target in the image, thereby improving the detection speed of the model to achieve real-time detection effect.

[0046] like Figure 3 As shown in FIG. 1 , the high-quality image obtained in S1 and the corresponding labeled data are input into the hidden dangerous target detection model for training until the accuracy index converges, indicating that the training is completed; the video data collected in real time by the camera mounted on the unmanned bulldozer is transmitted to the hidden dangerous target detection model. After processing, the model can detect hidden dangerous targets such as boulders, potholes and accumulated water in the video and output the pixel size of these targets.

[0047] The detection model of hidden dangerous targets such as boulders, holes and waterlogged objects is built using the biologically inspired algorithm SiNetv2. The biologically inspired algorithm SiNetv2 with search module and recognition module has excellent hidden target detection capability and can effectively detect hidden dangerous targets such as boulders, holes and waterlogged objects in complex mining environments.

[0048] A lightweight network structure GHostNet is used as the backbone network of the detection model. After adopting this backbone network, the hidden dangerous target detection model has a faster detection speed.

[0049] In one embodiment of the present invention, the precision index is mAP, and the average precision mAP is used to evaluate the performance of the hidden dangerous target detection model. First, AP is calculated according to the PR curve (Precision-Recall curve), and then mAP is obtained by averaging the AP indicators between different categories. The calculation formula is as follows:

[0050] m AP ＝mean( ∫ p(r) d r)

[0051] Among them, p(R) is the precision value corresponding to a certain recall value in the PR curve after training is completed.

[0052] S3 collects a large amount of data about target objects, including the pixel size of the target objects and the corresponding known actual size, and builds a fast size estimation model based on softmax regression to build the actual size. The target pixel size output in S4 is input into the model to quickly and automatically output the actual size of the target.

[0053] The size estimation model is built based on softmax regression. After the detection results of hidden dangerous targets in S3 are input into the size estimation model, the actual size y of the hidden dangerous targets is quickly estimated through the softmax function. The calculation formula of this process is as follows:

[0054]

[0055] Among them, the vector of the input sample after feature extraction is x, W is the weight matrix, and b is the bias vector.

[0056] Central alarm and remote control system:

[0057] Based on the relationship between the size X of the hidden dangerous target and the track width H of the unmanned bulldozer, a hierarchical alarm rule is established. After the actual size information of the hidden dangerous target is uploaded to the cloud multi-level alarm center, the center will analyze the danger level of the hidden dangerous target and issue instructions for normal driving, braking / obstacle avoidance or remote takeover to ensure the safety control of the construction operation.

[0058] The hierarchical alarm rules are used to assess the danger according to the size of hidden dangerous targets, including:

[0059] a) Safe state: The size of the hidden dangerous target is small (X≤H / 2);

[0060] b) Level 1 alarm: The size of the hidden dangerous target is large (H / 2<X≤H);

[0061] c) Second level alarm: the size of the hidden dangerous target is too large (H<X);

[0062] Among them, H is the track width of the unmanned bulldozer, and X is the size of the hidden dangerous target. The danger level is assessed according to the size of the hidden dangerous target, and no operation is performed in the safe state; when it is in the first level alarm state, the center issues a brake-obstacle avoidance command to the unmanned bulldozer; when it is in the second level alarm state, the center issues a remote takeover command to the unmanned bulldozer.

[0063] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time identification method for hidden dangerous targets of unmanned bulldozers in mining areas, characterized in that: The method comprises the following steps: Collect images of the working environment around the unmanned bulldozer in the mining area, and annotate the type, location and pixel size of hidden dangerous targets in each image to form a data set about hidden dangerous targets; Using the data set to train a hidden dangerous target detection model, the training is performed until the accuracy index of the target detection model converges, the input of the hidden dangerous target detection model is an image, and the output is the type, position and pixel size of the hidden dangerous target; The image to be detected is input into the trained hidden dangerous target detection model to obtain the type, position and pixel size of the hidden dangerous target in the image to be detected.

2. A method for real-time identification of hidden dangerous targets for unmanned bulldozers in mining areas as claimed in claim 1, characterized in that: After acquiring images of the surrounding working environment of the unmanned bulldozer in the mining area and marking the type, position and pixel size of the hidden dangerous targets in each image, a database of pixel size and actual size mapping is constructed according to the correspondence between the pixel size and actual size of the hidden dangerous targets in each image.

3. A real-time identification method for hidden dangerous targets of unmanned bulldozers in mining areas as claimed in claim 2, characterized in that: The database of pixel size and actual size is used to train a fast estimation model, the input of which is the feature vector of pixel size, and the output is the actual size. The formula of the fast estimation model is as follows: Among them, W is the weight matrix, b is the bias vector, x(i) is the eigenvector of the i-th sample, y(i) is the probability interval distribution of the i-th sample corresponding to the actual size, and the highest probability interval is taken as the actual size.

4. A method for real-time identification of hidden dangerous targets for unmanned bulldozers in mining areas as claimed in claim 3, characterized in that: After calculating the actual size of the hidden dangerous target in the image to be detected, an early warning is issued according to the preset early warning method based on the actual size. The preset early warning method is as follows: a) When X≤H / 2, it is a safe state; b) When H / 2<X≤H, it is a first-level warning; c) When H<X, it is a level 2 alarm; Among them, H is the track width of the unmanned bulldozer, and X is the actual size of the hidden dangerous target.

5. A real-time identification method for hidden dangerous targets of unmanned bulldozers in mining areas as claimed in claim 4, characterized in that: During the first-level warning, the unmanned bulldozer in the mining area brakes and avoids obstacles. When it is a second-level warning, the unmanned bulldozer in the mining area is remotely taken over and controlled.

6. A method for real-time identification of hidden dangerous targets for unmanned bulldozers in mining areas as claimed in claim 1, characterized in that: The hidden dangerous target detection model is a model that combines GHostNet and a biological heuristic algorithm, wherein GHostNet is used as a backbone network and the biological heuristic algorithm is used as a search method for the optimal solution.

7. The method for real-time identification of hidden dangerous targets for unmanned bulldozers in mining areas as claimed in claim 1, characterized in that: The hidden dangerous target is one or a combination of isolated rocks, deep pits and stagnant water.

8. A system for identification using the identification method according to any one of claims 1 to 7, characterized in that: The system includes an image acquisition module, a training module and a testing module, wherein: The image acquisition module is used to collect images of the surrounding working environment of the unmanned bulldozer in the mining area, and to mark the type, position and pixel size of hidden dangerous targets in each image; The training module is used to train a hidden dangerous target detection model using the data set, and train until the accuracy index of the target detection model converges; The testing module is used to input the image to be detected into the trained hidden dangerous target detection model to obtain the type, position and pixel size of the hidden dangerous target in the image to be detected.

9. A real-time identification system for hidden dangerous targets of unmanned bulldozers in mining areas, characterized in that: The system includes an actuator, which is used to execute the real-time identification method for hidden dangerous targets used in an unmanned bulldozer in a mining area as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, a real-time identification method for hidden dangerous targets used in an unmanned bulldozer in a mining area is implemented as described in any one of claims 1 to 7.

Images