Intelligent security detection method and related device for single-bucket excavator in strip mine area

Abstract

The invention discloses an intelligent security detection method for a single-bucket excavator in a strip mine area and a related device, and relates to the technical field of mine area security, a trained target detection model is utilized to detect a detection target in a real-time image of the single-bucket excavator in the strip mine area to obtain a detection image, and whether a dangerous behavior exists or not is determined based on the detection image. The dangerous behaviors include that the distance from a vehicle outside the single-bucket excavator to the single-bucket excavator is smaller than a first preset value, the distance from a person outside the single-bucket excavator to the single-bucket excavator is smaller than a second preset value, bucket teeth of the single-bucket excavator are broken or fall off, and an operator drives illegally; the driving behavior of the bucket teeth, the driving behavior of operators and the surrounding environment of the single-bucket excavator can be monitored at the same time, an alarm is given when dangerous behaviors exist, and the safety management level of a mining area is improved.

Description

Technical Field

[0001] The present application relates to the field of mining security technology, and in particular to an intelligent security detection method and related devices for single-bucket excavators in open-pit mines based on deep learning. Background Art

[0002] At present, solid mineral resources are abundant, and the mining methods of solid mineral resources can be divided into two categories: open-pit mining mode and underground mining mode. Among them, the open-pit mining mode accounts for a large part. Small open-pit mines are basically mined by single-bucket excavators combined with mining trucks. Large and medium-sized open-pit mines use single-bucket excavators combined with mining trucks while choosing to use belt conveyors as auxiliary equipment to better optimize the mining mode, save mining costs, and improve mining efficiency. Obviously, the single-bucket excavator is the core equipment for mining and loading in open-pit mines, and can be applied to mining and loading operations in open-pit mines (such as open-pit coal mines and open-pit iron mines).

[0003] Generally, single-bucket excavators are used in conjunction with mining trucks (such as 220-363t) and are mainly used for mining operations of open-pit coal mines, iron ore and other ores. When the single-bucket excavator is working, the bucket body is subjected to the pushing force, and the bucket teeth cut the ore and fall into the bucket body. Due to the interaction between the ore and the bucket teeth during the mining operation, the bucket teeth are very easy to wear, cracks appear on the surface, and even break. Especially when mining harder minerals, the bucket teeth will be subjected to a large reaction force, causing the bucket teeth on the tooth seat to loosen, fall off or break. Monitoring the bucket teeth is conducive to improving the safety management level of the mining area.

[0004] The working environment in open-pit mines is monotonous, and the average working hours of operators are about 8 to 10 hours. High-intensity work can cause operators to lose concentration. During mining operations, operators may engage in illegal behaviors such as smoking, answering and making phone calls, looking around, dozing off and driving due to fatigue. According to incomplete statistics, safety accidents caused by irregular driving behavior account for more than 63% of the total number of safety accidents in open-pit mines. Monitoring the driving behavior of operators is conducive to improving the safety management level of mining areas.

[0005] Single bucket excavator is the main heavy equipment in the open-pit mining process. Taking the domestic excavator WK-55 as an example, its whole machine weight is 1265t, the crawler reaches more than 2m, the total installed power is about 4850kw, and the bucket capacity is 55m 3, transport vehicles often shuttle around it, but it is difficult to be discovered, and miners shuttle around it, which is even more difficult to be discovered. And due to the huge size of the single-bucket excavator, the operator's vision is easily blocked during work, and they cannot see the situation around the single-bucket excavator, which brings great safety hazards to the single-bucket excavator itself and the surrounding working environment. Monitoring the surrounding environment of the single-bucket excavator is conducive to improving the safety management level of the mining area.

[0006] The safety of single-bucket excavators has attracted more and more attention. At present, many researchers have conducted research on issues related to intelligent security in open-pit mines. Many results have been achieved, but there is still a lack of technology to simultaneously monitor the bucket teeth, the operator's driving behavior and the surrounding environment of the single-bucket excavator. Summary of the invention

[0007] The purpose of this application is to provide an intelligent security detection method and related devices for single-bucket excavators in open-pit mines, which can simultaneously monitor the bucket teeth, the operator's driving behavior and the surrounding environment of the single-bucket excavator, and alarm when there is dangerous behavior, thereby improving the safety management level of the mining area.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] In a first aspect, the present application provides an intelligent security detection method for a single-bucket excavator in an open-pit mine, the intelligent security detection method for a single-bucket excavator in an open-pit mine comprising:

[0010] Acquire a real-time image of a single-bucket excavator in an open-pit mine; the real-time image includes an external image and an internal image; the external image is an image obtained by photographing vehicles, personnel and bucket bodies outside the cab of the single-bucket excavator; the internal image is an image obtained by photographing an operator inside the cab of the single-bucket excavator;

[0011] The real-time image is used as input, and the detection targets in the real-time image are detected using the trained target detection model to obtain a detection image; the detection targets include vehicles outside the single-bucket excavator, personnel outside the single-bucket excavator, the bucket body of the single-bucket excavator, bucket teeth of the single-bucket excavator, mobile phones, cigarettes, and the face, yawning mouth shape, and closed eye shape of the operator; the detection image is an image obtained by box-selecting each detection target in the real-time image;

[0012] Based on the detection image, determine whether there is any dangerous behavior, and issue an alarm when there is any dangerous behavior; the dangerous behavior includes that the distance from a vehicle outside the single-bucket excavator to the single-bucket excavator is less than a first preset value, the distance from a person outside the single-bucket excavator to the single-bucket excavator is less than a second preset value, the bucket teeth of the single-bucket excavator are broken or fallen off, and the operator drives in violation of regulations.

[0013] Optionally, before taking the real-time image as input and detecting the detection target in the real-time image using a trained target detection model to obtain the detection image, the open-pit mine single-bucket excavator intelligent security detection method further includes:

[0014] Determining whether there is fog in the real-time image;

[0015] If so, perform image defogging processing on the real-time image to obtain a defogged image, and use the defogged image as a new real-time image.

[0016] Optionally, determining whether fog exists in the real-time image specifically includes:

[0017] Enhancing the contrast of the real-time image to obtain an enhanced image;

[0018] Calculating a difference between the contrast of the enhanced image and the contrast of the real-time image, and determining whether the difference is greater than a first preset threshold, to obtain a first determination result;

[0019] If the first judgment result is no, then there is no fog in the real-time image;

[0020] If the first judgment result is yes, then calculating the transmittance of each pixel in the real-time image, and judging whether the variance of all the transmittances is less than a second preset threshold, to obtain a second judgment result;

[0021] If the second judgment result is yes, then there is fog in the real-time image;

[0022] If the second judgment result is no, then there is no fog in the real-time image.

[0023] Optionally, the transmittance is calculated as:

[0024]

[0025] Where t(x) is the transmittance of pixel x in the real-time image; ω is a constant; Ω(x) is the window at pixel x; I c (y) is the pixel value of pixel y in channel c in the real-time image, and the channels include red channel, green channel and blue channel; A c is the global atmospheric light value of channel c.

[0026] Optionally, performing image defogging processing on the real-time image to obtain a defogged image specifically includes: performing image defogging processing on the real-time image using a defogging formula to obtain a defogged image;

[0027] The defogging formula is:

[0028]

[0029] Where J(x) is the pixel value of pixel x in the defogging image; I(x) is the pixel value of pixel x in the real-time image; A is the atmospheric light value; t(x) is the transmittance of pixel x in the real-time image; t 0 is the third preset threshold.

[0030] Optionally, before taking the real-time image as input and detecting the detection target in the real-time image using a trained target detection model to obtain the detection image, the open-pit mine single-bucket excavator intelligent security detection method further includes:

[0031] Acquire multiple sample real-time images of a single-bucket excavator in an open-pit mine;

[0032] For each of the sample real-time images, determine whether there is fog in the sample real-time image; if so, perform image defogging processing on the sample real-time image to obtain a sample defogged image; if not, record the sample real-time image as a sample defogged image;

[0033] Performing image enhancement processing on each of the sample defogging images to obtain a sample enhanced image; the image enhancement processing includes at least one of geometric transformation, color jittering, random brightness adjustment and image stitching;

[0034] All the sample defogging images and all the sample enhanced images are combined into a data set; the data set includes a plurality of samples, and the samples are the sample defogging images or the sample enhanced images;

[0035] Labeling each detection target in each of the samples respectively to obtain a label of the sample;

[0036] The initial target detection model is trained using all the samples and the labels of each sample to obtain a trained target detection model.

[0037] In a second aspect, the present application provides an intelligent security detection device for a single-bucket excavator in an open-pit mine, the intelligent security detection device for a single-bucket excavator in an open-pit mine comprising:

[0038] An image acquisition module is used to acquire a real-time image of a single-bucket excavator in an open-pit mine; the real-time image includes an external image and an internal image; the external image is an image obtained by photographing vehicles, personnel and bucket bodies outside the cab of the single-bucket excavator; the internal image is an image obtained by photographing an operator inside the cab of the single-bucket excavator;

[0039] The target detection module is used to use the real-time image as input and use the trained target detection model to detect the detection targets in the real-time image to obtain a detection image; the detection targets include vehicles outside the single-bucket excavator, personnel outside the single-bucket excavator, the bucket body of the single-bucket excavator, bucket teeth of the single-bucket excavator, mobile phones, cigarettes, and the face, yawning mouth shape and closed eyes shape of the operator; the detection image is an image obtained by box-selecting each detection target in the real-time image;

[0040] A dangerous behavior alarm module is used to determine whether there is a dangerous behavior based on the detection image and to alarm when there is a dangerous behavior; the dangerous behavior includes the distance from a vehicle outside the single-bucket excavator to the single-bucket excavator being less than a first preset value, the distance from a person outside the single-bucket excavator to the single-bucket excavator being less than a second preset value, the bucket teeth of the single-bucket excavator being broken or falling off, and the operator driving in violation of regulations.

[0041] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described intelligent security detection methods for single-bucket excavators in open-pit mines.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned intelligent security detection methods for single-bucket excavators in open-pit mines.

[0043] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned intelligent security detection methods for single-bucket excavators in open-pit mines.

[0044] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0045] The present application provides an intelligent security detection method and related devices for single-bucket excavators in open-pit mines. The method uses a trained target detection model to detect detection targets in real-time images of single-bucket excavators in open-pit mines to obtain detection images. Based on the detection images, it is determined whether there are dangerous behaviors, and an alarm is issued when there are dangerous behaviors. Dangerous behaviors include the distance from a vehicle outside the single-bucket excavator to the single-bucket excavator being less than a first preset value, the distance from a person outside the single-bucket excavator to the single-bucket excavator being less than a second preset value, the bucket teeth of the single-bucket excavator being broken or falling off, and the operator driving in violation of regulations. The bucket teeth, the driving behavior of the operator, and the surrounding environment of the single-bucket excavator can be monitored at the same time, and an alarm is issued when there are dangerous behaviors, thereby improving the safety management level of the mining area. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 This is an application environment diagram of an intelligent security detection method for a single-bucket excavator in an open-pit mine provided in Example 1 of the present application.

[0048] Figure 2 A flow chart of an intelligent security detection method for a single-bucket excavator in an open-pit mine provided in Example 1 of the present application.

[0049] Figure 3 A schematic diagram of the functional modules of an intelligent security detection device for a single-bucket excavator in an open-pit mine provided in Example 2 of the present application.

[0050] Figure 4 A schematic diagram of the structure of a computer device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] In computer vision, target detection and recognition technology mainly studies the classification and detection of objects in images. In the field of intelligent security in open-pit mines, target detection has great prospects. The purpose of this application is to monitor the bucket teeth, the driving behavior of the operator and the surrounding environment of the single-bucket excavator based on target detection, and to alarm when there is dangerous behavior, so as to improve the safety management level of the mine area.

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0054] Example 1

[0055] The intelligent security detection method for single-bucket excavators in open-pit mines provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the real-time image to be processed to the server. After the server receives the real-time image to be processed, for the real-time image to be processed, the server uses the real-time image as input, and uses the trained target detection model to detect the detection target in the real-time image to obtain a detection image. Based on the detection image, it is determined whether there is a dangerous behavior, and an alarm is issued when there is a dangerous behavior.

[0056] In addition, in some embodiments, the intelligent security detection method for single-bucket excavators in open-pit mines can also be implemented independently by a server or a terminal. For example, the terminal can directly process the real-time image to be processed, or the server can obtain the real-time image to be processed from the data storage system and process the real-time image to be processed.

[0057] The terminal may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The server may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0058] like Figure 2 As shown, an intelligent security detection method for single-bucket excavators in open-pit mines is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 Taking the server in the example as an example, the intelligent security detection method for single-bucket excavators in open-pit mines includes the following steps:

[0059] Step S1, obtaining a real-time image of a single-bucket excavator in an open-pit mine; the real-time image includes an external image and an internal image; the external image is an image obtained by photographing vehicles, personnel and bucket bodies outside the cab of the single-bucket excavator; the internal image is an image obtained by photographing an operator inside the cab of the single-bucket excavator.

[0060] Step S2, taking the real-time image as input, using the trained target detection model to detect the detection targets in the real-time image to obtain a detection image; the detection targets include vehicles outside the single-bucket excavator, personnel outside the single-bucket excavator, the bucket body of the single-bucket excavator, bucket teeth of the single-bucket excavator, mobile phones, cigarettes, and the operator's face, yawning mouth shape, and closed eyes shape; the detection image is an image obtained by box-selecting each detection target in the real-time image.

[0061] Step S3, determining whether there is any dangerous behavior based on the detection image, and issuing an alarm when there is any dangerous behavior; the dangerous behavior includes the distance from the vehicle outside the single-bucket excavator to the single-bucket excavator being less than a first preset value, the distance from the person outside the single-bucket excavator to the single-bucket excavator being less than a second preset value, the bucket teeth of the single-bucket excavator being broken or falling off, and the operator driving in violation of regulations.

[0062] The above-mentioned steps S1 to S3 are implemented. This embodiment uses a deep learning method to study the real-time detection of safety issues inside and outside the cab of a single-bucket excavator according to the actual scene requirements of the open-pit mine area, so as to detect the health of the single-bucket excavator itself. When it is detected that the bucket teeth are broken or detached, an alarm message can be sent to the operator in time to remind the operator of the danger. When the operator has illegal driving behavior, an alarm message can be sent to the operator in time to remind the operator of the danger. When the single-bucket excavator is performing mining operations, if a person or vehicle enters a dangerous area, an alarm message can be sent to the operator in time to remind the operator of the danger.

[0063] In this embodiment, two vehicle-mounted cameras are arranged. One vehicle-mounted camera is installed outside the cockpit of the single-bucket excavator and is located at the front side of the cockpit of the single-bucket excavator, and is used to obtain real-time picture information of the surrounding conditions of the single-bucket excavator (including surrounding vehicles and surrounding personnel) and the health status of the bucket body (including bucket teeth) outside the cockpit of the single-bucket excavator to obtain an external image. The other vehicle-mounted camera is installed inside the cockpit of the single-bucket excavator to obtain real-time picture information of the behavior of the operator inside the cockpit of the single-bucket excavator to obtain an internal image, thereby obtaining a real-time image of the single-bucket excavator in the open-pit mine, which is the real scene image data in the mining area. The real-time image includes an external image and an internal image. The external image is an image obtained by shooting the vehicles, personnel and bucket body outside the cockpit of the single-bucket excavator, which is the image data of the surrounding environment of the single-bucket excavator outside the cockpit, including vehicles, personnel, bucket body, bucket teeth, etc.; the internal image is an image obtained by shooting the operator inside the cockpit of the single-bucket excavator, which is the behavior image data of the operator in the cockpit, including head, face, closing eyes, answering and making phone calls, smoking, yawning, etc.

[0064] In S2, the real-time image is used as input, and the detection targets in the real-time image are detected using the trained target detection model, and each detection target in the real-time image is framed to obtain a detection image. The detection targets include vehicles outside the single-bucket excavator, personnel outside the single-bucket excavator, the single-bucket excavator bucket body, the single-bucket excavator bucket teeth, mobile phones, cigarettes, and the operator's face, yawning mouth shape, and closed eyes. This is to determine whether there are vehicles and personnel around the single-bucket excavator that are too close to the single-bucket excavator, whether the single-bucket excavator bucket teeth are broken or fallen off, and whether the operator has any illegal driving behavior.

[0065] Taking a real-time image as input, using a trained target detection model to detect the detection target in the real-time image to obtain a detection image, specifically including: taking an external image as input, using a trained target detection model to detect the detection target in the external image to obtain an external detection image, taking an internal image as input, using a trained target detection model to detect the detection target in the internal image to obtain an internal detection image, and the external detection image and the internal detection image constitute a detection image.

[0066] In S3, it is determined whether there is any dangerous behavior based on the detection image. The dangerous behaviors include the distance from the external vehicle of the single-bucket excavator to the single-bucket excavator being less than the first preset value, the distance from the external personnel of the single-bucket excavator to the single-bucket excavator being less than the second preset value, the broken or fallen teeth of the single-bucket excavator, and the illegal driving of the operator. Specifically, after each detection target is framed, the trained target detection model will output the category of each detection target. For the external vehicle of the single-bucket excavator, based on the position of the external vehicle of the single-bucket excavator in the external image, combined with the position of the on-board camera used to shoot the external image, the distance from the external vehicle of the single-bucket excavator to the single-bucket excavator is further determined. If the distance is less than the first preset value, there is a dangerous behavior; similarly, for the external personnel of the single-bucket excavator, based on the position of the external personnel of the single-bucket excavator in the external image, combined with the position of the on-board camera used to shoot the external image, the distance from the external personnel of the single-bucket excavator to the single-bucket excavator is further determined. If the distance is less than the second preset value, there is Dangerous behavior; for the bucket body and teeth of a single-bucket excavator, determine the number of teeth inside the bucket body. If the number does not match the normal number, the teeth are broken or fallen off, and there is a dangerous behavior; for a mobile phone, as long as a mobile phone is detected, the operator is playing with the mobile phone, and there is a dangerous behavior; for smoke, as long as smoke is detected, the operator is smoking, and there is a dangerous behavior; for the operator's face, yawning mouth shape, and closed eyes shape, if the operator's face has a yawning mouth shape and closed eyes shape, the operator is driving fatigued, and there is a dangerous behavior. When there is a dangerous behavior, an alarm is issued to remind the operator. The yawning mouth shape refers to the shape of the operator's mouth when yawning, and the closed eyes shape refers to the shape of the operator's eyes when closing his eyes.

[0067] Before taking the real-time image as input and detecting the detection target in the real-time image using the trained target detection model to obtain the detection image, the intelligent security detection method for single-bucket excavators in open-pit mines of this embodiment also includes: determining whether there is fog in the real-time image; if so, performing image defogging processing on the real-time image to obtain a defogged image, and using the defogged image as a new real-time image to execute S2.

[0068] When judging whether there is fog in the real-time image, the contrast test of the real-time image is first performed, because fog will reduce the contrast of the image and make the image details blurred. Specifically, a stretching algorithm is used to enhance the contrast of the real-time image, and the contrast difference of the image before and after the processing is compared to set a threshold. If the contrast difference is greater than the threshold, fog may exist. After the first round of screening, the image with fog is compared with the image without fog. It is found that the two have obvious differences in color, clarity, and transmittance. The image with fog has a single color, smooth edges, and low transmittance, while the image without fog has bright colors, strong edges, and high transmittance. Considering that the open-pit mining area is mainly mountainous, most of the images are black and gray, so it is difficult to judge by color. Therefore, this embodiment judges whether there is fog based on the transmittance, and the variance of the transmittance of the image with fog and other normal images without fog in the image data set is counted to set a suitable threshold. When the variance of the transmittance is lower than the suitable threshold, fog may exist and defogging operation is required.

[0069] In this embodiment, determining whether there is fog in the real-time image specifically includes: enhancing the contrast of the real-time image by using a stretching algorithm to obtain an enhanced image; calculating the difference between the contrast of the enhanced image and the contrast of the real-time image, and determining whether the difference is greater than a first preset threshold value, to obtain a first judgment result; if the first judgment result is no, there is no fog in the real-time image; if the first judgment result is yes, calculating the transmittance of each pixel in the real-time image, and determining whether the variance of all transmittances is less than a second preset threshold value, to obtain a second judgment result; if the second judgment result is yes, there is fog in the real-time image; if the second judgment result is no, there is no fog in the real-time image.

[0070] The transmittance is calculated as:

[0071]

[0072] Wherein, t(x) is the transmittance of pixel x in the real-time image; ω is a constant, which can be 0.95; Ω(x) is the window at pixel x, which is the area in the real-time image with pixel x as the center and a side length of a preset value; I c (y) is the pixel value of pixel y in channel c in the real-time image. Pixel y is a pixel in the window. The channels include red channel R, green channel G and blue channel B. c is the global atmospheric light value of channel c.

[0073] In the field of computer vision, defogging mainly uses the defogging model shown in the following formula:

[0074] I(x)=J(x)t(x)+A(1-t(x));

[0075] Among them, I(x) is the pixel value of pixel x in the foggy image, which is the input; J(x) is the pixel value of pixel x in the normal fog-free image, which is the output; t(x) is the transmittance of pixel x in the foggy image; A is the atmospheric light value.

[0076] However, the above-mentioned defogging model is not suitable for real-time images of mining areas. Therefore, this embodiment selects a defogging algorithm based on an atmospheric scattering model to compensate for image loss under foggy conditions to obtain the original image, and then performs image defogging on the real-time image to obtain a defogged image, specifically including: performing image defogging on the real-time image using a defogging formula to obtain a defogged image.

[0077] The dehazing formula is:

[0078]

[0079] Among them, J(x) is the pixel value of pixel x in the defogging image; I(x) is the pixel value of pixel x in the real-time image; A is the atmospheric light value, which is determined as follows: the principle of the dark channel prior defogging algorithm is that the dark channel value of the fog-free image approaches 0, so 0.1% of the brightest pixels are selected in the dark channel map corresponding to the real-time image, and the pixel with the largest brightness among these 0.1% of the brightest pixels is found in the real-time image, and the brightness of the pixel with the largest brightness is taken as the atmospheric light value A; t(x) is the transmittance of pixel x in the real-time image; t 0 is the third preset threshold. If the transmittance t(x) is too small, the J(x) value will be too large and the image will transition to the white field. Therefore, a threshold t is usually set. 0 , when t(x) is less than t 0 , let t(x)=t 0 , to prevent the transmittance from being too small, causing the defogging to lose the characteristic protection lower limit of the original pixel.

[0080] Preferably, after the real-time image is defogged using the defogging formula to obtain the defogged image, the image processed by the dark channel in this embodiment is further subjected to guided filtering. Guided filtering is a method for filtering a target image based on a reference image. The reference image is a grayscale image of the real-time image, and the target image is the defogged image. While retaining edge information, the defogged image is refined with the grayscale image of the real-time image as a guide to remove the unprocessed foggy part of the edge, so as to make the effect more natural and obtain a new defogged image.

[0081] Before taking the real-time image as input and detecting the detection target in the real-time image using the trained target detection model to obtain the detection image, the open-pit mine single-bucket excavator intelligent security detection method of this embodiment also includes:

[0082] (1) Obtain multiple sample real-time images of a single-bucket excavator in an open-pit mine.

[0083] (2) For each sample real-time image, determine whether there is fog in the sample real-time image; if so, perform image defogging on the sample real-time image to obtain a sample defogged image; if not, record the sample real-time image as the sample defogged image.

[0084] (3) performing image enhancement processing on each sample dehazed image to obtain a sample enhanced image, wherein the image enhancement processing includes at least one of geometric transformation, color jittering, random brightness adjustment, and image stitching.

[0085] After the image is dehazed, it is then enhanced using data enhancement methods such as geometric transformation, color jittering, random brightness adjustment, and image stitching.

[0086] 1) Geometric transformation: The geometric transformation method does not change the pixel values ​​in the image but changes its spatial position, mainly including: translation transformation, flip transformation, scaling transformation, etc.

[0087] 2) Color jittering: In actual engineering, the purpose is to eliminate the differences between dataset images in different backgrounds. The color jittering method mainly adjusts the brightness, contrast and saturation of the image.

[0088] 3) Random brightness adjustment: The brightness change rules are as follows:

[0089] τ'=kτ;

[0090] Among them, τ' is the brightness of the transformed image; k is the adjustment coefficient, which can be 0.57-1.46; τ is the brightness of the original image.

[0091] 4) Image stitching: Randomly extract two sample dehazed images, select one sample dehazed image, randomly generate a cropping box of a certain proportion in it, and mask it to obtain the mask area. Then, cover the area at the corresponding position of the other sample dehazed image into the mask area, retain the latter's detection target annotation information, and fuse them into a new image.

[0092] The formula for stitching two images and their corresponding detection target annotation information can be expressed as:

[0093]

[0094]

[0095] Among them, x is an image with C channels and W×H pixels, A For image A, x Bis image B; y is the detection target annotation information in image x, y A Annotation information for the detected target in image A, y B Label the detected target in image B; is the new image obtained by concatenating image A and image B; M is a mask matrix marked by 0 and 1 of the same size; ⊙ is pixel-by-pixel multiplication; For new images The detection target annotation information in the new image; λ is the source ratio of the detection target annotation information in the new image, which is a linear interpolation coefficient used to smoothly fuse the detection target annotation information of two different images in the data enhancement process. When λ=1, Completely by y A Decision, that is, the detection target annotation information of the new image is completely inherited from image A; when λ=0, Completely by y B Decision, that is, the detection target annotation information of the new image is completely inherited from image B; when λ takes the middle value, Yes A and B The weighted average of λ determines the proportion of the detection target annotation information of the two images in the new image.

[0096] The mask matrix M is used to control the region selection of the image, and λ is used to control the linear interpolation of the detection target annotation information. This allows the model to see more different forms of target combinations during training, enhances the generalization ability of the model, and reflects the scientific nature of the new image.

[0097] (4) All sample defogging images and all sample enhanced images are combined into a data set, where the data set includes multiple samples, and the samples are sample defogging images or sample enhanced images.

[0098] The data enhancement method uses preset transformation rules to transform the original image data while preserving the label, and expands the data based on the existing image data, which can be expressed as a mapping:

[0099] Ρ: O→S;

[0100] Among them, P is the conversion rule for data enhancement; O is the original data set, including all sample dehazed images; S is the augmented data set, including all sample enhanced images.

[0101] The dataset can be represented as:

[0102] O'=O∪S;

[0103] Among them, O' is the data set after data augmentation.

[0104] (5) Label each detection target in each sample separately to obtain the label of the sample.

[0105] After the real-life image is defogged and enhanced, the sample is labeled. Specifically, the detection target in the sample is labeled with a rectangular frame, such as the face, eyes, and mouth in the cockpit, the bucket body, bucket teeth, vehicles, and personnel outside the cockpit, etc., to create a data set that meets the requirements, thereby completing the establishment of a labeled data set. That is, this embodiment performs image defogging and image enhancement processing on the image data, and labels the processed image, thereby establishing a labeled data set. The labeled data set is subsequently used to train the initial target detection model.

[0106] (6) Using all samples and the labels of each sample, the initial target detection model is trained to obtain a trained target detection model.

[0107] This embodiment realizes intelligent and safe real-time detection of single-bucket excavators in open-pit mines based on the YOLOv5s model. The YOLOv5s model is used as the baseline model, and the improved YOLOv5s model is obtained by lightweighting and improving the loss function. The improved YOLOv5s model is used as the initial target detection model and trained.

[0108] The overall network structure of the YOLOv5s model includes: input end, Backbone part, Neck part, and Prediction part.

[0109] Input: YOLOv5s adds the Mosaic data enhancement method compared to YOLOv3, enriching the data set and improving the model robustness. At the same time, YOLOv5s uses the adaptive anchor box calculation function, which automatically calculates the best anchor box value for different data sets before each training and embeds it into the code. If you want to turn off this function, you can set it in the train.py file. At this time, the self-calibrated initial anchor box will be used for training during the training process.

[0110] BackBone part: First, the Focus structure is used. The image will be sliced ​​before entering the Backbone. The slicing operation will take values ​​for each pixel in each image, which is similar to the process of adjacent downsampling. In this way, four complementary images will be obtained. These four complementary images contain the R, G, and B three-channel information of each pixel in the original image, as well as the horizontal and vertical coordinate information of the pixel in the image. These width and height information are integrated into the channel space to obtain a new image containing 12 channels. After the new image is processed by the convolution operation, a feature map of double downsampling can be obtained without losing information. The role of Focus is to improve the reasoning speed of the network model, but it does not improve the accuracy of the model. Secondly, YOLOv5s also uses CSPDarkne as BackBone to extract rich feature information from the input image. The difference is that two CSP structures are designed, one is CSP1_X, which is applied to BackBone, and the other is CSP_2X, which is applied to Neck. The purpose is to ensure the reasoning speed and accuracy of the model while reducing the size of the network model.

[0111] Neck part: Its Neck structure adopts the FPN+PAN structure and the CSP2_X structure, which enhances the feature fusion capability of the network model.

[0112] In addition, the loss function of YOLOv5s consists of three parts: category prediction function, positioning loss function and boundary regression function. The specific formula is as follows:

[0113]

[0114] Among them, L BCE is the category prediction function; n is the total number of predicted bounding boxes; y n is the binary label value 0 or 1 of the nth predicted bounding box; x n is the probability of the binary label value of the nth predicted bounding box.

[0115]

[0116] Among them, L CIoU is the positioning loss function; IoU is the intersection over union ratio, which is an indicator used in target detection to measure the degree of overlap between the predicted bounding box and the true bounding box. It is calculated by dividing the intersection area of ​​the predicted bounding box and the true bounding box by their union area; ρ is the distance between the center points of the bounding boxes; b is the predicted bounding box; b gt is the real bounding box; c is the diagonal length of the predicted bounding box; is the weight function; is the width and height similarity, w gt is the width of the true bounding box, hgt is the height of the true bounding box, w is the width of the predicted bounding box, and h is the height of the predicted bounding box.

[0117]

[0118] Among them, s i is the confidence of the i-th predicted bounding box; M is the predicted bounding box with the highest current confidence; b i is the i-th predicted bounding box; N is the IoU threshold.

[0119] YOLOv5s is superior to other target detection models in terms of accuracy, inference speed, compatibility and stability. Therefore, this embodiment selects YOLOv5s as the benchmark model and makes improvements based on YOLOv5s to achieve intelligent and safe real-time detection of excavators in open-pit mines.

[0120] First, for the YOLO series models, a comparative analysis was conducted using the same environment and experimental parameters under the data set constructed in this embodiment, and YOLOv5s was finally selected as the benchmark model based on the experimental results. Secondly, the demand for intelligent security detection of single-bucket excavators in open-pit mines in actual scenarios was analyzed, and the following improvement plans were designed based on actual needs:

[0121] 1) Lightweight model design. Considering the real-time requirements of intelligent security detection of single-bucket excavators in open-pit mines, this embodiment optimizes the inference speed of the model based on the benchmark model, modifies the benchmark model, introduces GSConv convolution to reduce the number of model parameters, and combines the SlimNeck network to improve the inference speed of the model.

[0122] 2) Improvement of loss function, the design concept of lightweight model, the number of model parameters is greatly reduced during training, but the accuracy of the model will be reduced. This embodiment optimizes the loss function, introduces the SioU loss function, improves the convergence of the model, and improves the accuracy of the model.

[0123] For the above improvements, the model is trained in the same data set, server and experimental parameters, tested in the same CPU server, and the optimization of the loss function is compared with the experimental results of the lightweight model design, and analyzed one by one, and finally the integrated results are analyzed. The experimental results show that the model inference speed of the improved model in the real scene is 65.78, and the mAP@0.5 accuracy is 80.7%, which meets the real-time detection requirements of this embodiment, so it is used as the final open-pit mining excavator intelligent security detection model.

[0124] At this time, the trained target detection model adopts the improved YOLOv5s model. The improved YOLOv5s model is obtained by improving the YOLOv5s model. The improvements include: introducing GSConv and SlimNeck in the YOLOv5s model; introducing the SioU loss function in the loss function of the YOLOv5s model.

[0125] The embodiment of the present application also provides an application scenario, which applies the above-mentioned intelligent security detection method for single-bucket excavators in open-pit mines. Specifically, the intelligent security detection method for single-bucket excavators in open-pit mines provided by this embodiment can be applied in the working scenario of single-bucket excavators. The working scenario of single-bucket excavators includes an image acquisition link and an image processing link. The image acquisition link is used to obtain real-time images of single-bucket excavators in open-pit mines. The image processing link is used to process real-time images to determine whether there are dangerous behaviors and to issue an alarm when there are dangerous behaviors. The intelligent security detection method for single-bucket excavators in open-pit mines provided by this embodiment belongs to the image processing link.

[0126] Example 2

[0127] Based on the same inventive concept, the embodiment of the present application also provides an open-pit mine single-bucket excavator intelligent security detection device for implementing the above-mentioned open-pit mine single-bucket excavator intelligent security detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiment of the open-pit mine single-bucket excavator intelligent security detection device provided below can refer to the limitations of the open-pit mine single-bucket excavator intelligent security detection method above, and will not be repeated here.

[0128] like Figure 3 As shown, an intelligent security detection device for a single-bucket excavator in an open-pit mine is provided, and the intelligent security detection device for a single-bucket excavator in an open-pit mine comprises:

[0129] The image acquisition module M1 is used to acquire the real-time image of the single-bucket excavator in the open-pit mine; the real-time image includes an external image and an internal image; the external image is an image obtained by photographing the vehicle, personnel and bucket body outside the cab of the single-bucket excavator; the internal image is an image obtained by photographing the operator inside the cab of the single-bucket excavator.

[0130] The target detection module M2 is used to use the real-time image as input, and use the trained target detection model to detect the detection targets in the real-time image to obtain a detection image; the detection targets include vehicles outside the single-bucket excavator, personnel outside the single-bucket excavator, the bucket body of the single-bucket excavator, bucket teeth of the single-bucket excavator, mobile phones, cigarettes, and the operator's face, yawning mouth shape, and closed eyes shape; the detection image is an image obtained after box-selecting each detection target in the real-time image.

[0131] The dangerous behavior alarm module M3 is used to determine whether there is a dangerous behavior based on the detection image and to alarm when there is a dangerous behavior; the dangerous behavior includes the distance from the vehicle outside the single-bucket excavator to the single-bucket excavator being less than a first preset value, the distance from the person outside the single-bucket excavator to the single-bucket excavator being less than a second preset value, the bucket teeth of the single-bucket excavator being broken or falling off, and the operator driving in violation of regulations.

[0132] The dangerous behavior alarm module M3 includes an abnormality detection unit and an alarm unit. The abnormality detection unit is used to detect the detection image, and the alarm unit is used to alarm when a dangerous behavior is detected.

[0133] Example 3

[0134] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store real-time image data of a single-bucket excavator in an open-pit mine. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an intelligent security detection method for a single-bucket excavator in an open-pit mine is implemented.

[0135] Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In an exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intelligent security detection method for a single-bucket excavator in an open-pit mine as described in Example 1.

[0137] Example 4

[0138] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the intelligent security detection method for a single-bucket excavator in an open-pit mine described in Example 1 is implemented.

[0139] Example 5

[0140] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the intelligent security detection method for single-bucket excavators in open-pit mines described in Example 1.

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An intelligent security detection method for single-bucket excavators in open-pit mines, characterized in that: The intelligent security detection method for single-bucket excavators in open-pit mines includes: Acquire a real-time image of a single-bucket excavator in an open-pit mine; the real-time image includes an external image and an internal image; the external image is an image obtained by photographing vehicles, personnel and bucket bodies outside the cab of the single-bucket excavator; the internal image is an image obtained by photographing an operator inside the cab of the single-bucket excavator; The real-time image is used as input, and the detection targets in the real-time image are detected using the trained target detection model to obtain a detection image; the detection targets include vehicles outside the single-bucket excavator, personnel outside the single-bucket excavator, the bucket body of the single-bucket excavator, bucket teeth of the single-bucket excavator, mobile phones, cigarettes, and the face, yawning mouth shape, and closed eye shape of the operator; the detection image is an image obtained by box-selecting each detection target in the real-time image; Based on the detection image, determine whether there is any dangerous behavior, and issue an alarm when there is any dangerous behavior; the dangerous behavior includes that the distance from a vehicle outside the single-bucket excavator to the single-bucket excavator is less than a first preset value, the distance from a person outside the single-bucket excavator to the single-bucket excavator is less than a second preset value, the bucket teeth of the single-bucket excavator are broken or fallen off, and the operator drives in violation of regulations.

2. The intelligent security detection method for single-bucket excavators in open-pit mines according to claim 1 is characterized in that: Before the real-time image is used as input and the detection target in the real-time image is detected using the trained target detection model to obtain the detection image, the open-pit mine single-bucket excavator intelligent security detection method further includes: Determining whether there is fog in the real-time image; If so, perform image defogging processing on the real-time image to obtain a defogged image, and use the defogged image as a new real-time image.

3. The intelligent security detection method for single-bucket excavators in open-pit mines according to claim 2 is characterized in that: Determining whether there is fog in the real-time image specifically includes: Enhancing the contrast of the real-time image to obtain an enhanced image; Calculating a difference between the contrast of the enhanced image and the contrast of the real-time image, and determining whether the difference is greater than a first preset threshold, to obtain a first determination result; If the first judgment result is no, then there is no fog in the real-time image; If the first judgment result is yes, then calculating the transmittance of each pixel in the real-time image, and judging whether the variance of all the transmittances is less than a second preset threshold, to obtain a second judgment result; If the second judgment result is yes, then there is fog in the real-time image; If the second judgment result is no, then there is no fog in the real-time image.

4. The intelligent security detection method for single-bucket excavators in open-pit mines according to claim 3 is characterized in that: The transmittance is calculated as: Where t(x) is the transmittance of pixel x in the real-time image; ω is a constant; Ω(x) is the window at pixel x; I c (y) is the pixel value of pixel y in channel c in the real-time image, and the channels include red channel, green channel and blue channel; A c is the global atmospheric light value of channel c.

5. The intelligent security detection method for single-bucket excavators in open-pit mines according to claim 2 is characterized in that: Performing image defogging processing on the real-time image to obtain a defogged image, specifically comprising: performing image defogging processing on the real-time image using a defogging formula to obtain a defogged image; The defogging formula is: Among them, J(x) is the pixel value of pixel x in the dehazed image; I(x) is the pixel value of pixel x in the real-time image; A is the atmospheric light value; t(x) is the transmittance of pixel x in the real-time image; t0 is the third preset threshold.

6. The intelligent security detection method for single-bucket excavators in open-pit mines according to claim 1 is characterized in that: Before the real-time image is used as input and the detection target in the real-time image is detected using the trained target detection model to obtain the detection image, the open-pit mine single-bucket excavator intelligent security detection method further includes: Acquire multiple sample real-time images of a single-bucket excavator in an open-pit mine; For each of the sample real-time images, determine whether there is fog in the sample real-time image; if so, perform image defogging processing on the sample real-time image to obtain a sample defogged image; if not, record the sample real-time image as a sample defogged image; Performing image enhancement processing on each of the sample defogging images to obtain a sample enhanced image; the image enhancement processing includes at least one of geometric transformation, color jittering, random brightness adjustment and image stitching; All the sample defogging images and all the sample enhanced images are combined into a data set; the data set includes a plurality of samples, and the samples are the sample defogging images or the sample enhanced images; Labeling each detection target in each of the samples respectively to obtain a label of the sample; The initial target detection model is trained using all the samples and the labels of each sample to obtain a trained target detection model.

7. An intelligent security detection device for single-bucket excavators in open-pit mines, characterized in that: The open-pit mine single-bucket excavator intelligent security detection device comprises: An image acquisition module is used to acquire a real-time image of a single-bucket excavator in an open-pit mine; the real-time image includes an external image and an internal image; the external image is an image obtained by photographing vehicles, personnel and bucket bodies outside the cab of the single-bucket excavator; the internal image is an image obtained by photographing an operator inside the cab of the single-bucket excavator; The target detection module is used to use the real-time image as input and use the trained target detection model to detect the detection targets in the real-time image to obtain a detection image; the detection targets include vehicles outside the single-bucket excavator, personnel outside the single-bucket excavator, the bucket body of the single-bucket excavator, bucket teeth of the single-bucket excavator, mobile phones, cigarettes, and the face, yawning mouth shape and closed eyes shape of the operator; the detection image is an image obtained by box-selecting each detection target in the real-time image; A dangerous behavior alarm module is used to determine whether there is a dangerous behavior based on the detection image and to alarm when there is a dangerous behavior; the dangerous behavior includes the distance from a vehicle outside the single-bucket excavator to the single-bucket excavator being less than a first preset value, the distance from a person outside the single-bucket excavator to the single-bucket excavator being less than a second preset value, the bucket teeth of the single-bucket excavator being broken or falling off, and the operator driving in violation of regulations.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intelligent security detection method for single-bucket excavators in open-pit mines as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent security detection method for single-bucket excavators in open-pit mines according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the intelligent security detection method for single-bucket excavators in open-pit mines according to any one of claims 1 to 6 is implemented.

Images