A method and system for detecting dirty lens of camera of unmanned mining truck

Abstract

The present invention discloses a method and system for detecting dirty lenses of unmanned mining truck cameras, which relates to the field of image processing and computer vision technology, including: reading the positioning information of the combined inertial navigation to obtain the vehicle's motion state and speed; calculating the correlation and gradient grayscale of adjacent image frames when the vehicle is moving; performing multi-scale analysis on the fusion features based on the analysis features of the cumulative fusion of multiple frames of motion images; and detecting dirty areas of different sizes based on the analysis of the fused correlation gradient grayscale features. By extracting the three features of image correlation, gradient, and brightness, the lens dirtiness of the ground background area and the lens dirtiness of the sky background area can be effectively detected. The present invention uses a statistical method of image sub-region features of different scales to detect lens dirty areas of different sizes. The present invention uses a sliding window for binarization during the feature detection process, which can more accurately extract dirty areas and reduce the instability of using a global binarization method.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a method and system for detecting dirt on a camera lens of an unmanned mining truck. Background Art

[0002] Intelligent driverless cars are equipped with a large number of sensors, which enable the vehicles to perceive the environment and automatically analyze the road conditions and target obstacles around the vehicle, ensuring the safety of the vehicle and providing a basis for path planning. Most of the on-board sensors such as cameras and radars are exposed to the outside of the vehicle. During driving, these sensors will be contaminated to varying degrees, especially in harsh environmental conditions. For example, when the vehicle drives on a muddy road, mud may splash onto the surface of the camera, causing the lens to become dirty, reducing the clarity of the captured image, and causing the image recognition perception to decline, which will have a great impact on the safety of autonomous driving. Therefore, it is very important to accurately detect camera dirt and handle it in a timely manner after detecting camera dirt.

[0003] There are two methods for detecting camera dirtiness: the traditional method based on image features and the model detection method based on deep learning. The traditional detection method based on image features is more common to calculate the quality of the image captured by the camera through information entropy, and determine whether the camera is dirty based on whether the image quality is greater than the preset judgment threshold. This method has a relatively accurate detection effect when the image quality is good, but in scenes such as night or foggy days, dust, etc., the camera image quality itself is very poor, and it is not really that the camera is dirty, so it is easy to have false detection. In addition, the size of the dirty area also affects the accuracy of the detection result; the detection method based on deep learning includes direct judgment based on the classification model, and also based on the detection / segmentation model + post-processing analysis. The detection effect of the method based on the deep learning model requires the support of a large amount of sample collection and calibration work. At the same time, model detection will take up a lot of computing resources, which is a great burden on the vehicle end with limited computing resources. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing camera dirt detection method based on image features has the technical problems of inaccurate detection, difficulty in adapting to different environments, and poor robustness. The model-based method has the problem of large sample requirements and high computing load, which is a challenge to the scarce computing resources on the vehicle side.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for detecting dirt on the camera lens of an unmanned mining truck, which comprises the following steps:

[0007] The positioning information of the combined inertial navigation is read to obtain the vehicle's motion state and speed; the correlation and gradient grayscale of adjacent image frames when the vehicle is moving are calculated; based on the analysis features of the accumulated fusion of multiple frames of motion images, the fusion features are analyzed at multiple scales; based on the analysis of the fused correlation gradient grayscale features, dirty areas of different sizes are detected.

[0008] As a preferred solution of the method for detecting dirt on the camera lens of an unmanned mining truck described in the present invention, the obtaining of the vehicle's motion state and speed includes real-time motion detection of the vehicle to detect whether the vehicle is in motion. The motion detection includes using camera image data and combined inertial navigation positioning data. If the vehicle is equipped with a positioning device, the vehicle's motion state is directly determined by the positioning data. If the vehicle is not equipped with a positioning device, there is no positioning data, that is, motion detection is performed using the camera image.

[0009] After obtaining the positioning data, the spatial coordinate position of the positioning information and the speed information of the X-axis, Y-axis and Z-axis are read. When any of the speed vector modulus values ​​of the X-axis, Y-axis and Z-axis is greater than the threshold, the vehicle is judged to be in motion. If it is less than or equal to the threshold, it is judged to be non-moving.

[0010] The use of camera images for motion detection includes reading image data, obtaining image data collected by the camera, and performing motion judgment based on image frame differences.

[0011] As a preferred solution of the method for detecting dirty camera lenses of unmanned mining trucks described in the present invention, the operation judgment through image frame difference includes calculating the difference between two adjacent image frames using an inter-frame interpolation method. When the difference between two adjacent image frames is greater than a threshold, it is judged that the vehicle is moving. If the difference between two adjacent image frames is less than or equal to the threshold, it is judged that the vehicle is stationary.

[0012] Determine whether the image needs to be processed based on the vehicle's motion status.

[0013] As a preferred solution of the method for detecting dirt on the camera lens of an unmanned mining truck described in the present invention, the method comprises: judging whether the image needs to be processed based on the motion state of the vehicle includes: if the vehicle is moving, performing subsequent processing; if the vehicle is stationary, no operation is performed.

[0014] The subsequent operations include image downsampling, image sub-region division, and image feature calculation.

[0015] The calculation of the image features includes performing image correlation mean calculation on each sub-region, performing image gradient mean calculation on each sub-region, and performing grayscale mean calculation on each sub-region.

[0016] The image correlation mean value calculation includes calculating the correlation value of each pixel point in the local WINDOW and taking the average value of the correlation values ​​in the sub-region.

[0017] The image gradient mean calculation includes using two 3*3 matrix operators in the horizontal and vertical directions of SOBEL to convolve with the original image, respectively, to obtain the horizontal G(x) and vertical G(y) gradient values, taking the maximum value of the horizontal G(x) and vertical G(y) gradient values ​​as the gradient value, and taking the average of the gradient values ​​in the sub-area.

[0018] The grayscale mean value calculation includes converting the color image into a grayscale image, and calculating the grayscale mean value of each sub-region according to the grayscale image.

[0019] As a preferred solution of the method for detecting dirty lens of camera of unmanned mining truck described in the present invention, the multi-scale analysis of fusion features includes: accumulating the sub-region correlation mean, gradient mean and grayscale mean obtained in each frame according to the region for multiple frames, obtaining the sum of the sub-region correlation mean, the sum of the gradient mean and the sum of the grayscale mean of the sub-region for multiple frames, accumulating them, and accumulating them to the frame number threshold, calculating the frame average value of the sum of the multi-frame cumulative correlation mean, the multi-frame cumulative gradient mean and the multi-frame cumulative grayscale mean for each sub-region, and obtaining the frame mean of the original M*N sub-region scales. After obtaining the frame mean, upsampling is performed to obtain the mean, and the number of upsampled sub-regions is The correlation frame mean, gradient frame mean, and grayscale frame mean are calculated in the upsampled sub-region.

[0020] As a preferred solution of the method for detecting dirty lens of unmanned mining truck camera described in the present invention, the analysis based on the fused correlation gradient grayscale features includes selecting the gradient threshold T at the scale of the original sub-region and the up-sampled sub-region. grad The sub-region gradient frame mean is binarized, and the connected domain is extracted and the holes are filled in the binarized image to obtain two types of connected regions: high gradient regions and low gradient regions.

[0021] On the scale of the original sub-region and the upsampled sub-region, select a sliding window of G*H, and use the window mean V in the sliding window area arv +T corr The correlation frame mean of the sub-region at the center of the sliding window is binarized, and the area above the threshold is taken to obtain the high gradient area.

[0022] On the scale of the original sub-region and the upsampled sub-region, select a sliding window of G*H, and use the window mean V in the sliding window area arv —T grayThe correlation frame mean of the sub-region at the center of the sliding window is binarized, and the area below the threshold is taken to obtain the low gradient area.

[0023] The correlation extraction region and the high gradient region are intersected to obtain the dirty region of the high gradient region at the scale of the original sub-region and the upsampled sub-region.

[0024] On the scales of the original sub-region and the up-sampled sub-region, the low grayscale region and the low gradient region are intersected to obtain the dirty region of the low gradient region on the scales of the original sub-region and the up-sampled sub-region.

[0025] As a preferred solution of the method for detecting dirt on the lens of an unmanned mining truck camera described in the present invention, the detection of dirty areas of different sizes includes merging the dirty area of ​​the high gradient area extracted at the original sub-area scale and the dirty area of ​​the low gradient area extracted at the original sub-area scale to obtain the dirt detection result of the original sub-area.

[0026] The high-gradient dirty region extracted from the up-sampling sub-region and the low-gradient dirty region extracted from the up-sampling sub-region are combined to obtain the dirt detection result of the up-sampling sub-region.

[0027] The dirt detection results of the original sub-region and the dirt detection results of the upsampled sub-region are scale-aligned and then fused. The fusion adopts a conservative fusion method, that is, the detection results of the original sub-region and the upsampled sub-region are retained to obtain the fused sub-region detection results.

[0028] The fused sub-region detection results are analyzed and the number of dirty areas is calculated. When the number is greater than the set threshold, the dirt information and dirt location are output. When the number is less than the set threshold, the normal state without dirt is output.

[0029] Another object of the present invention is to provide a system for detecting dirt on the camera lens of an unmanned mining truck, which can read the positioning information of the combined inertial navigation in real time and perform in-depth analysis on adjacent image frames when the vehicle is moving, thereby solving the problems that the existing traditional dirt detection methods rely on fixed environmental conditions, have poor adaptability to dynamically changing scenes, and have low detection accuracy.

[0030] To solve the above technical problems, the present invention provides the following technical solutions: a system for detecting dirty camera lenses of unmanned mining trucks, comprising: a combined inertial navigation positioning module, an image frame processing module, a multi-frame motion image fusion analysis module, a dirty area detection module, a vehicle motion state determination module, and an image feature calculation module.

[0031] The combined inertial navigation positioning module is responsible for reading the combined inertial navigation positioning information to obtain the motion state and speed of the vehicle.

[0032] The image frame processing module calculates the correlation and gradient grayscale of adjacent image frames when the vehicle is moving.

[0033] The multi-frame motion image fusion analysis module performs multi-scale analysis on the fusion features according to the analysis features of the accumulated fused multi-frame motion images.

[0034] The dirty area detection module performs analysis based on the fused correlation gradient grayscale features to detect dirty areas of different sizes.

[0035] The vehicle motion state determination module performs motion detection on the vehicle in real time to detect whether the vehicle is in motion. It uses camera image data and combined inertial navigation positioning data to make a judgment.

[0036] The image feature calculation module includes image correlation mean calculation, image gradient mean calculation and grayscale mean calculation.

[0037] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the above-mentioned method for detecting dirt on the camera lens of an unmanned mining truck are implemented.

[0038] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned method for detecting dirt on a camera lens of an unmanned mining truck are implemented.

[0039] The beneficial effects of the present invention are as follows: the present invention proposes a lens dirt detection method combining multiple image features. In order to effectively detect dirt in the sky area and the ground area, the present invention extracts three features of image correlation, gradient, and brightness. The combined detection can effectively detect lens dirt in the ground background area and lens dirt in the sky background area. The present invention uses a statistical method of image sub-area features of different scales. The combination of detections of different scales can effectively detect lens dirt areas of different sizes. The present invention uses a sliding window for binarization during the feature detection process, which can more accurately extract the dirty area and reduce the instability of using a global binarization method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0041] Figure 1An overall flow chart of a method for detecting dirt on a camera lens of an unmanned mining truck provided in the first embodiment of the present invention.

[0042] Figure 2 An overall framework diagram of a system for detecting dirty camera lenses of unmanned mining trucks provided in accordance with a second embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0046] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0047] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0048] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] Example 1

[0050] Reference Figure 1 , is an embodiment of the present invention, and provides a method for detecting dirt on the camera lens of an unmanned mining truck, which is characterized by:

[0051] S1: Read the positioning information of the combined inertial navigation to obtain the vehicle's motion state and speed.

[0052] Furthermore, obtaining the vehicle's motion state and speed includes performing real-time motion detection on the vehicle to detect whether the vehicle is in motion. The motion detection includes using camera image data and combined inertial navigation positioning data. If the vehicle is equipped with a positioning device, the vehicle's motion state is directly determined by the positioning data. If the vehicle is not equipped with a positioning device, there is no positioning data, that is, motion detection is performed using camera images.

[0053] After acquiring the positioning data, read the spatial coordinate position of the positioning information and the speed information of the X-axis, Y-axis and Z-axis. When any of the speed vector modulus values ​​of the X-axis, Y-axis and Z-axis is greater than the threshold, it is judged to be moving. If it is less than or equal to the threshold, it is judged to be not moving.

[0054] The use of camera images for motion detection includes reading image data, obtaining image data collected by the camera, and performing motion judgment based on image frame differences.

[0055] Furthermore, operation judgment through image frame difference includes calculating the difference between two adjacent image frames using the inter-frame interpolation method. When the difference between two adjacent image frames is greater than a threshold, it is judged that the vehicle is moving. If the difference between two adjacent image frames is less than or equal to the threshold, it is judged that the vehicle is stationary.

[0056] Determine whether the image needs to be processed based on the vehicle's motion status.

[0057] It is further explained that the inter-frame interpolation method is used to calculate the difference between two adjacent image frames. The inter-frame difference method is simple to implement and has low program complexity. It is not very sensitive to scene changes such as light, can adapt to various dynamic environments, and has good stability.

[0058] S2: Calculate the correlation and gradient grayscale of adjacent image frames when the vehicle is moving.

[0059] Further, judging whether the image needs to be processed according to the motion determination state of the vehicle includes performing subsequent processing if the vehicle is moving, and not performing any operation if the vehicle is stationary.

[0060] The subsequent operations include image downsampling, image sub-region division, and image feature calculation.

[0061] In image frequency reduction, the camera frame rate is generally high, the time interval between two adjacent frames is short, and the spatial transformation is small. The present invention frequency-reduces the captured image frames at a certain frequency, taking 1 frame for every N frames. Frame reduction can reduce the calculation frequency and increase the change of the image content of adjacent frames. The previous frame image data is cached during the frequency reduction post-processing process.

[0062] In the image sub-region division, the image is evenly divided into M*N sub-regions, each sub-region has the same width and height, and the subsequent processing is statistically analyzed based on the sub-region. In order to detect dirt at different scales, the present invention performs pyramid upsampling on the original sub-region and performs the same analysis on the up-sampled region.

[0063] The calculation of the image features includes performing image correlation mean calculation on each sub-region, performing image gradient mean calculation on each sub-region, and performing grayscale mean calculation on each sub-region.

[0064] The image correlation mean value calculation includes calculating the correlation value of each pixel point in the local WINDOW and taking the average value of the correlation values ​​in the sub-region.

[0065] The image gradient mean calculation includes using two 3*3 matrix operators in the horizontal and vertical directions of SOBEL to convolve with the original image, respectively, to obtain the horizontal G(x) and vertical G(y) gradient values, taking the maximum value of the horizontal G(x) and vertical G(y) gradient values ​​as the gradient value, and taking the average of the gradient values ​​in the sub-area.

[0066] The grayscale mean value calculation includes converting the color image into a grayscale image, and calculating the grayscale mean value of each sub-region according to the grayscale image.

[0067] It is further explained that the present invention processes the frame-sampling image in motion state. Due to the persistence characteristics of the dirty area, the present invention uses the multi-frame accumulated image features to more stably distinguish the dirty area from the normal area. The sub-region correlation mean, gradient mean, and grayscale mean obtained in each frame are numerically accumulated by region to obtain the multi-frame accumulated sub-region correlation mean sum, accumulated sub-region gradient mean sum, and accumulated sub-region grayscale mean sum.

[0068] S3: Based on the analysis features of the accumulated fused multi-frame motion images, a multi-scale analysis is performed on the fused features.

[0069] Furthermore, the multi-scale analysis of the fusion features includes accumulating the sub-region correlation mean, gradient mean, and grayscale mean obtained in each frame according to the region, and obtaining the multi-frame accumulated sub-region correlation mean sum, the accumulated sub-region gradient mean sum, and the accumulated sub-region grayscale mean sum. After the accumulation reaches the frame number threshold, the multi-frame accumulated correlation mean sum, the multi-frame accumulated gradient mean sum, and the multi-frame accumulated grayscale mean sum of each sub-region are calculated to obtain the frame mean of the original M*N sub-region scale. After obtaining the frame mean, the upsampling mean is calculated, and the number of upsampled sub-regions is The correlation frame mean, gradient frame mean, and grayscale frame mean are calculated in the upsampled sub-region.

[0070] S4: Analyze the fused correlation gradient grayscale features and detect dirty areas of different sizes.

[0071] Further, the analysis based on the fused correlation gradient grayscale features includes selecting the gradient threshold T at the scale of the original sub-region and the upsampled sub-region. grad The sub-region gradient frame mean is binarized, and the connected domain is extracted and the holes are filled in the binarized image to obtain two types of connected areas: high gradient areas and low gradient areas. The gradient calculation is used to distinguish the sky area from the ground area. The high gradient area corresponds to the ground area, and the low gradient area corresponds to the sky area.

[0072] On the scale of the original sub-region and the upsampled sub-region, select a sliding window of G*H, and use the window mean V in the sliding window area arv +T corr The correlation frame mean of the sub-region at the center of the sliding window is binarized, and the area above the threshold is taken to obtain the high gradient area.

[0073] On the scale of the original sub-region and the upsampled sub-region, select a sliding window of G*H, and use the window mean V in the sliding window area arv —T gray The correlation frame mean of the sub-region at the center of the sliding window is binarized, and the area below the threshold is taken to obtain the low gradient area.

[0074] The correlation extraction region and the high gradient region are intersected to obtain the dirty region of the high gradient region at the scale of the original sub-region and the upsampled sub-region.

[0075] Grayscale features are used to detect dirt in the sky area. The grayscale value of the dirty area is lower than that of the normal sky area. At the scale of the original sub-region and the upsampled sub-region, the low grayscale area and the low gradient area are intersected to obtain the dirty area of ​​the low gradient area at the scale of the original sub-region and the upsampled sub-region.

[0076] Furthermore, detecting dirty regions of different sizes includes merging the dirty region of the high gradient region extracted at the original sub-region scale and the dirty region of the low gradient region extracted at the original sub-region scale to obtain the dirt detection result of the original sub-region.

[0077] The high-gradient dirty region extracted from the up-sampling sub-region and the low-gradient dirty region extracted from the up-sampling sub-region are combined to obtain the dirt detection result of the up-sampling sub-region.

[0078] The dirt detection results of the original sub-region and the dirt detection results of the upsampled sub-region are scale-aligned and then fused. The fusion adopts a conservative fusion method, that is, the detection results of the original sub-region and the upsampled sub-region are retained to obtain the fused sub-region detection results.

[0079] The fused sub-region detection results are analyzed and the number of dirty areas is calculated. When the number is greater than the set threshold, the dirt information and dirt location are output. When the number is less than the set threshold, the normal state without dirt is output.

[0080] Example 2

[0081] Reference Figure 2 , which is an embodiment of the present invention, provides a system for detecting a dirty camera lens of an unmanned mining truck. The system comprises a combined inertial navigation positioning module, an image frame processing module, a multi-frame motion image fusion analysis module, a dirty area detection module, a vehicle motion state determination module, and an image feature calculation module.

[0082] The combined inertial navigation positioning module is responsible for reading the combined inertial navigation positioning information to obtain the vehicle's motion state and speed; the image frame processing module calculates the correlation and gradient grayscale of adjacent image frames when the vehicle is moving; the multi-frame motion image fusion analysis module performs multi-scale analysis on the fusion features based on the analysis features of the accumulated fused multi-frame motion images; the dirty area detection module analyzes the fused correlation gradient grayscale features and detects dirty areas of different sizes; the vehicle motion state judgment module performs real-time vehicle motion detection to detect whether the vehicle is in a moving state, and it uses camera image data and combined inertial navigation positioning data to make judgments; the image feature calculation module includes image correlation mean calculation, image gradient mean calculation and grayscale mean calculation.

[0083] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0084] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0085] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0086] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0087] Example 3

[0088] In this embodiment, in order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments. This embodiment experiments the existing traditional method and the method of this embodiment respectively.

[0089] In a large open-pit mine, a team of unmanned mining trucks are transporting ore. Due to the dusty environment of the mine, the camera lenses are often dirty, which poses a great challenge to the autonomous driving and navigation of the unmanned mining trucks.

[0090] In order to solve this problem, the mine decided to adopt the unmanned mining truck camera lens dirt detection system provided by the present invention. The main components of the system include a combined inertial navigation positioning module, an image frame processing module, a multi-frame motion image fusion analysis module and a dirty area detection module.

[0091] When the unmanned mining truck starts working, the combined inertial navigation positioning module first reads the vehicle's motion state and speed information. At the same time, the image frame processing module starts to process the continuous image frames captured by the camera and calculates their correlation and gradient grayscale.

[0092] Then, the multi-frame motion image fusion analysis module starts working, which performs multi-scale analysis on the fusion features based on the analysis features of the accumulated fused multi-frame motion images. This step ensures that the system can accurately detect dirt at different scales.

[0093] Finally, the dirty area detection module analyzes the fused correlation gradient grayscale features and detects dirty areas of different sizes. Once dirt is detected, the system will immediately issue a warning and instruct the unmanned mining truck to perform necessary cleaning or other countermeasures.

[0094] By using the system of the present invention, the mine successfully ensures that the unmanned mining truck can maintain the clarity of the camera lens in a dusty environment, thereby greatly improving the working efficiency and safety of the mining truck. This embodiment experiments the existing traditional method and the method of this embodiment respectively. As described in Table 1.

[0095] Table 1 Experimental effect comparison chart

[0096]

[0097]

[0098] From the above comparison results, it can be seen that the accuracy of dirt detection of our invention is 95%, while that of traditional invention is 80%, an improvement of 15%. The response time of our invention is 30ms, while that of traditional invention is 50ms, which is 40% lower than that of traditional invention. The system stability of our invention is 500 hours, while that of traditional invention is 100 hours, which is an improvement of 400%. The adaptability of our invention to complex environment is 90%, while that of traditional invention is 70%, which is an improvement of 20%.

[0099] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for detecting dirt on the camera lens of an unmanned mining truck, characterized in that: include: Read the positioning information of the combined inertial navigation to obtain the vehicle's motion state and speed; Calculate the correlation and gradient grayscale of adjacent image frames when the vehicle is moving; According to the analysis features of the accumulated fusion of multiple frames of motion images, the fusion features are analyzed at multiple scales; Analyze the fused correlation gradient grayscale features and detect dirty areas of different sizes; The obtaining of the vehicle motion state and speed includes performing motion detection on the vehicle in real time to detect whether the vehicle is in motion. The motion detection includes using camera image data and combined inertial navigation positioning data. If the vehicle is equipped with a positioning device, the vehicle motion state is directly determined by the positioning data. If the vehicle is not equipped with a positioning device, there is no positioning data, that is, the camera image is used for motion detection. After obtaining the positioning data, read the spatial coordinate position of the positioning information and the speed information of the X-axis, Y-axis and Z-axis. When any of the speed vector modulus values ​​of the X-axis, Y-axis and Z-axis is greater than the threshold, the vehicle is judged to be in motion. If it is less than or equal to the threshold, it is judged to be non-moving. The use of camera images for motion detection includes reading image data, obtaining image data collected by the camera, and performing motion judgment through image frame difference; The multi-scale analysis of the fusion features includes: accumulating the sub-region correlation mean, gradient mean, and grayscale mean obtained in each frame according to the region for multiple frames, obtaining the multi-frame accumulated sub-region correlation mean sum, the accumulated sub-region gradient mean sum, and the accumulated sub-region grayscale mean sum; after accumulating to the frame number threshold, the multi-frame accumulated correlation mean sum, the multi-frame accumulated gradient mean sum, and the multi-frame accumulated grayscale mean sum of each sub-region are used to obtain the frame average value to obtain the frame mean of the original M*N sub-region scale; after obtaining the frame mean, the up-sampling mean is obtained, and the number of up-sampled sub-regions is Calculate the correlation frame mean, gradient frame mean, and grayscale frame mean in the upsampled sub-region; The analysis based on the fused correlation gradient grayscale features includes selecting a gradient threshold T at the scale of the original sub-region and the up-sampled sub-region. grad Binarize the sub-region gradient frame mean, extract the connected domain and fill the holes in the binarized image, and obtain two types of connected regions: high gradient region and low gradient region. On the scale of the original sub-region and the upsampled sub-region, select a sliding window of G*H, and use the window mean V in the sliding window area arv +T corr Binarize the correlation frame mean of the sub-region at the center of the sliding window, and take the area above the threshold to obtain the high gradient area; On the scale of the original sub-region and the upsampled sub-region, select a sliding window of G*H, and use the window mean V in the sliding window area arv —T gray Binarize the correlation frame mean of the sub-region at the center of the sliding window, and take the area below the threshold to obtain the low gradient area; Perform intersection operation on the correlation extraction region and the high gradient region to obtain the dirty region of the high gradient region on the scale of the original sub-region and the up-sampled sub-region; On the scales of the original sub-region and the up-sampled sub-region, the low grayscale region and the low gradient region are intersected to obtain the dirty region of the low gradient region on the scales of the original sub-region and the up-sampled sub-region; The detecting of dirty regions of different sizes includes merging the dirty region of the high gradient region extracted at the original sub-region scale and the dirty region of the low gradient region extracted at the original sub-region scale to obtain the dirt detection result of the original sub-region; The high-gradient dirty region extracted from the up-sampling sub-region and the low-gradient dirty region extracted from the up-sampling sub-region are combined to obtain the dirt detection result of the up-sampling sub-region; The dirt detection results of the original sub-region and the dirt detection results of the upsampled sub-region are scale-aligned and then fused. The fusion adopts a conservative fusion method, that is, the detection results of the original sub-region and the upsampled sub-region are retained to obtain the fused sub-region detection results; The fused sub-region detection results are analyzed and the number of dirty areas is calculated. When the number is greater than the set threshold, the dirt information and dirt location are output. When the number is less than the set threshold, the normal state without dirt is output.

2. A method for detecting dirt on the camera lens of an unmanned mining truck as claimed in claim 1, characterized in that: The operation judgment by image frame difference includes calculating the difference between two adjacent image frames by using an inter-frame interpolation method, and when the difference between the two adjacent image frames is greater than a threshold, it is determined that the vehicle is moving, and when the difference between the two adjacent image frames is less than or equal to the threshold, it is determined that the vehicle is stationary; Determine whether the image needs to be processed based on the vehicle's motion status.

3. A method for detecting dirt on the camera lens of an unmanned mining truck as claimed in claim 2, characterized in that: The determining whether the image needs to be processed according to the motion determination state of the vehicle includes: if the vehicle is moving, then performing subsequent processing, and if the vehicle is stationary, then not performing any operation; Subsequent operations include image downsampling, image sub-region division, and image feature calculation; The calculation of the image features includes performing image correlation mean calculation on each sub-region, performing image gradient mean calculation and grayscale mean calculation on each sub-region; The image correlation mean value calculation includes calculating the correlation value of each pixel in the local WINDOW and taking the average value of the correlation value in the sub-region; The image gradient mean calculation includes: using two 3*3 matrix operators in the horizontal and vertical directions of SOBEL to perform convolution with the original image, respectively, to obtain the horizontal G(x) and vertical G(y) gradient values, taking the maximum value of the horizontal G(x) and vertical G(y) gradient values ​​as the gradient value, and taking the average value of the gradient values ​​in the sub-region; The grayscale mean value calculation includes converting the color image into a grayscale image, and calculating the grayscale mean value of each sub-region according to the grayscale image.

4. A system using the method for detecting dirty lens of an unmanned mining truck camera as claimed in any one of claims 1 to 3, characterized in that: It includes a combined inertial navigation positioning module, an image frame processing module, a multi-frame motion image fusion analysis module, a dirty area detection module, a vehicle motion state determination module, and an image feature calculation module; The combined inertial navigation positioning module is responsible for reading the combined inertial navigation positioning information to obtain the motion state and speed of the vehicle; The image frame processing module calculates the correlation and gradient grayscale of adjacent image frames when the vehicle is moving; The multi-frame motion image fusion analysis module performs multi-scale analysis on the fusion features according to the analysis features of the accumulated fused multi-frame motion images; The dirty area detection module performs analysis based on the fused correlation gradient grayscale features to detect dirty areas of different sizes; The vehicle motion state determination module performs motion detection on the vehicle in real time to detect whether the vehicle is in motion, and it uses the camera image data and the combined inertial navigation positioning data to make the judgment; The image feature calculation module includes image correlation mean calculation, image gradient mean calculation and grayscale mean calculation.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a method for detecting dirt on a camera lens of an unmanned mining truck as described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for detecting dirt on a camera lens of an unmanned mining truck as described in any one of claims 1 to 3 are implemented.

Images