Shovel tooth missing state detection method and system based on morphological features and arrangement consistency

Abstract

The present application relates to the technical field of image recognition, and discloses a bucket tooth missing state detection method and system based on morphological features and arrangement consistency, which comprises the following steps: designing and installing a bucket image acquisition device to keep the arrangement of bucket teeth in the image always horizontal; extracting a bucket frame image, and performing image enhancement on the frame image by using the Retinex theory; setting a sliding window that moves horizontally, calculating the image moments of each window to represent morphological features, calculating the correlation coefficients of multiple image moments between each window and a bucket tooth template image by using the cross-correlation method, and forming a correlation coefficient graph; determining the spatial position of the bucket teeth in the image based on the correlation coefficient graph by using the characteristic that the bucket teeth are arranged uniformly in the horizontal direction, and setting a threshold to judge the missing state of the bucket teeth. The present application effectively improves the recognition accuracy of the missing state of the bucket teeth by extracting the morphological features of the multiple-order image moments of the bucket teeth and combining the horizontal arrangement consistency detection.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically, to a method and system for detecting missing bucket teeth based on morphological features and arrangement consistency. Background Technology

[0002] Excavators are an indispensable piece of equipment in modern construction machinery, widely used in various fields such as civil engineering, building construction, and mining. The main working component of an excavator is the bucket, and the bucket teeth are a crucial part of it. Bucket teeth are used for excavating and loading earth, and their strength and wear resistance directly affect the efficiency of excavation operations and the safety of the equipment. Because bucket teeth are subjected to impact, friction, and soil and rock forces for extended periods in harsh working environments, they are prone to wear and damage, sometimes even leading to tooth loss. This damage not only affects work efficiency but can also cause material to get into the bucket teeth, resulting in damage to subsequent equipment, additional maintenance costs, and safety hazards. Therefore, monitoring bucket teeth and timely detection of their missing state has become a critical issue in ensuring the safe operation of construction machinery. Currently, common bucket tooth monitoring methods are mainly based on visual inspection technology, using cameras to capture images of the bucket teeth and then using target detection and semantic segmentation algorithms to identify and segment the teeth in the images. While these methods can perform real-time monitoring through image processing technology, they still have many shortcomings: Limited image quality: During excavation, the bucket teeth are easily obscured by materials such as mud and rocks, affecting image quality and clarity, thus reducing detection accuracy. Inaccurate semantic segmentation: Existing semantic segmentation methods have certain limitations in segmenting bucket teeth, especially in complex backgrounds or under partial occlusion conditions, where the segmentation results are difficult to accurately match the actual contours of the bucket teeth.

[0003] Poor accuracy in determining missing teeth: Current methods typically determine missing teeth by comparing their height differences. However, due to the randomness in the shape, arrangement, and height of the teeth, this method is inaccurate and prone to misjudgment or omission.

[0004] Lack of utilization of morphological features: Existing methods mainly rely on visual information and geometric features, but have not conducted in-depth research on the morphological features and arrangement patterns of the teeth, neglecting the consistency and structural characteristics of the teeth's arrangement. Teeth are usually arranged according to certain rules; missing teeth may lead to irregular arrangements, becoming an important clue for identification.

[0005] In summary, existing tooth monitoring methods cannot fully meet the engineering requirements for high precision, low error, and real-time monitoring. There is an urgent need for a new detection method that can combine morphological characteristics and alignment consistency to achieve more accurate and reliable tooth loss detection. Summary of the Invention

[0006] In view of this, the present invention proposes a method and system for detecting missing bucket teeth based on morphological features and arrangement consistency. By extracting the morphological features of the bucket teeth and combining them with the arrangement pattern of the bucket teeth, the method can detect missing teeth, thereby improving the accuracy of detecting missing bucket teeth and avoiding the problems of misjudgment and missed judgment caused by the inaccuracy of image processing and geometric feature extraction in existing methods.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for detecting missing bucket teeth based on morphological features and arrangement consistency includes the following steps: Design and install a bucket image acquisition device to ensure that the bucket teeth are always horizontally aligned in the image; Extract bucket frame images and perform image enhancement on the frame images using Retinex theory; A horizontally moving sliding window is set up, and the image moments of each window are calculated to characterize the morphological features. The correlation coefficient between multiple image moments of each window and the bucket tooth template image is calculated using the cross-correlation method to form a correlation coefficient graph. By utilizing the characteristic that the bucket teeth are evenly arranged in the horizontal direction, the spatial position of the bucket teeth in the image is determined based on the correlation coefficient map, and a threshold is set to determine the missing state of the bucket teeth.

[0008] Furthermore, the process of designing and installing the bucket image acquisition device includes: Select an industrial camera with sufficient resolution and install a light source as needed; equip the industrial camera with protective devices to isolate it from external moisture and dust, and use an aviation interface; install the bucket image acquisition device on the excavator's boom, ensuring that the acquisition device and the bucket maintain a relatively fixed distance.

[0009] Furthermore, the process of image enhancement using Retinex theory for frame images includes: The pixel values ​​of the input image are normalized to the range of 0 to 1, and the illumination information of the image is obtained from the maximum value of the three color channels of the input image. Gaussian filtering is applied to the illumination information to obtain the illumination map; Enhanced images are obtained based on Retinex theory.

[0010] Furthermore, before using Retinex theory for image enhancement, the region of interest of the bucket is extracted based on prior knowledge to reduce the amount of image data.

[0011] Furthermore, the image moments of each window include the calculation of first-order, second-order, third-order, fourth-order, and fifth-order image moments; the morphological feature vector of the bucket tooth template image is obtained by collecting multiple complete bucket tooth image regions and calculating the average value of their first-order to fifth-order image moments.

[0012] Furthermore, the process of calculating the correlation coefficients of multiple image moments between each window and the bucket tooth template image using the cross-correlation method includes: The normalized correlation coefficient between the feature vector of the moving window morphology and the feature vector of the complete bucket tooth morphology is calculated, and the calculation process is applied to the frequency domain. Fast Fourier Transform is used to improve the processing speed.

[0013] Furthermore, the normalized correlation coefficient is calculated using the following formula:

[0014] in, Let be the covariance of the feature vector of the moving window shape and the feature vector of the complete tusk shape. and These are the variances of the feature vectors for the moving window morphology and the feature vectors for the complete tusk morphology, respectively. Furthermore, the process of determining the spatial position of the bucket teeth in the image and judging the missing state includes: Summing each row of the correlation coefficient graph forms a horizontal summation curve, with the peak position of the curve representing the vertical position of the bucket teeth in the image. Extract the horizontal curve of the vertical position and find the peak value and peak position of all peaks; Based on the number N of bucket teeth installed on the bucket, extract the top N peaks with larger peak values ​​and their corresponding peak positions; Calculate the average distance between all adjacent peaks, and then calculate the ratio between the distance of each adjacent peak and the average distance. A threshold is set, and when a ratio is greater than the threshold, the bucket teeth are determined to be in a missing state.

[0015] Based on the above method, this invention proposes a bucket tooth missing state detection system based on morphological features and arrangement consistency, comprising: The image acquisition module includes an industrial camera and a light source mounted on the excavator boom. The camera has a protective device and an aviation interface to keep the bucket teeth horizontally aligned in the image. The image preprocessing module is used to extract bucket frame images captured by the camera and to enhance the frame images using Retinex theory. The morphological feature extraction module is used to set a horizontally moving sliding window and calculate the first to fifth order image moments of each window to characterize morphological features. The matching calculation module is used to calculate the correlation coefficient of the image moments between each window and the bucket tooth template image using the cross-correlation method, and to form a correlation coefficient graph. The missing tooth detection module is used to determine the spatial position of the bucket teeth based on the correlation coefficient map by utilizing the characteristic that the bucket teeth are evenly arranged in the horizontal direction, and to determine the missing tooth status by analyzing the ratio of the distance between adjacent peaks and comparing it with a threshold.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention effectively improves the accuracy of identifying missing bucket teeth by extracting multi-order rectangular features from the images of the bucket teeth and combining them with horizontal alignment consistency detection. It overcomes the problems of misjudgment and missed judgment caused by image occlusion, lighting changes and reliance on geometric height difference in existing methods. At the same time, it uses frequency domain fast Fourier transform to achieve real-time and efficient processing, thereby realizing accurate and rapid detection of missing bucket teeth. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the bucket tooth missing state detection method based on morphological features and arrangement consistency of the present invention. Figure 2 This is an image of the bucket and related sparse schematic diagram in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] This embodiment proposes a method for detecting missing bucket teeth based on morphological features and arrangement consistency. Figure 1 As shown, it includes the following steps: Design a bucket image acquisition device and install it at a suitable location on site to ensure that the bucket teeth remain horizontal in the image arrangement. Extract bucket frame images captured by the camera, and use Retinex theory to enhance the frame images to reduce the impact of lighting changes on image details; A horizontally moving sliding window is set up, and the image moments of each window are calculated to represent morphological features. The correlation coefficients of multiple image moments between each window and the bucket tooth template image are calculated using the cross-correlation method to form a correlation coefficient graph.

[0020] By utilizing the characteristic that the bucket teeth are evenly arranged in the horizontal direction, the spatial position of the bucket teeth in the image is determined based on the correlation coefficient map, and a threshold is set to determine the missing state of the bucket teeth.

[0021] Detailed operation steps: (1) Design a bucket image acquisition device and install a fixed bucket image acquisition device on site to ensure that the bucket teeth remain horizontal in the image arrangement: 1) When selecting a camera, it should have sufficient resolution to provide more information about the shape and details of the bucket in the image, and a light source should be installed to ensure long-term stable operation in low-light environments. Therefore, the bucket image acquisition device should include an industrial camera and a light source. 2) In open mining environments, there will be rain, dust and strong vibrations. To reduce the impact of the harsh detection environment on industrial cameras, protective devices are installed on industrial cameras to isolate external moisture and dust, and aviation interfaces are used to ensure normal operation of the cameras.

[0022] 3) Install the bucket image acquisition device on the excavator boom, ensuring that the acquisition device maintains a relatively fixed distance from the bucket and that the bucket teeth remain horizontal in the image alignment.

[0023] (2) Extract the bucket frame images captured by the camera, and use Retinex theory to enhance the frame images to reduce the impact of lighting changes on image details. 4) The image acquisition device collects a large number of frame image data every second. Real-time processing of image data will inevitably lead to a large CPU consumption and cause long delays in online analysis. In reality, the bucket usually occupies a fixed area in the image. Therefore, based on prior knowledge, the region of interest of the bucket can be extracted to reduce image data and improve the system's processing speed.

[0024] 5) Furthermore, to reduce the impact of different lighting environments on image details, Retinex theory is used to enhance the frame images. The specific steps are as follows: Step 1: To estimate the illumination map I Normalizing the pixel values ​​of the input image to the range of 0 to 1, and obtaining the image's illumination information from the maximum values ​​of the three color channels of the input image, can be written as:

[0025] in, S This represents the input low-light image. R This is a reflection image. I It is a lighting diagram. These represent the three color channels of an RGB image. Represents the pixel coordinates in the image.

[0026] Step 2: Perform Gaussian filtering on the lighting information to obtain the lighting map. I :

[0027] in, Represents convolution. Indicated in pixels Size at position The standard deviation is Gaussian kernel.

[0028] Step 3: Based on Retinex theory, an enhanced image with good visual effects can be obtained, as shown in the following expression:

[0029] in, It is a very small integer to prevent division by zero. This is a parameter used to control the overall brightness of the prior image, set to 0.7.

[0030] (3) Set a horizontally moving sliding window, calculate the image moments of each window to characterize the morphological features, and use the cross-correlation method to calculate the correlation coefficients of multiple image moments between each window and the bucket tooth template image to form a correlation coefficient graph.

[0031] 6) Acquire relevant images of bucket teeth, manually extract individual image regions containing complete bucket teeth, and calculate the first, second, third, fourth, and fifth order image moments of these image regions to characterize the morphological features of the complete bucket teeth. Calculate the average values ​​of the first, second, third, fourth, and fifth order image moments respectively, and use these as morphological feature vectors that can characterize the complete bucket teeth. R ; 7) Set a horizontally moving sliding window, scan line by line on the enhanced image, and calculate the first, second, third, fourth, and fifth order image moments for each window to form the morphological feature vector of the moving window. O .

[0032] 8) Calculate the moving window shape feature vector using the following formula. O and complete bucket tooth morphology feature vector R(Normalized correlation coefficient) A correlation coefficient is generated for each position traversed by the center point of the moving window, forming a correlation coefficient graph.

[0033]

[0034] here Let be the covariance of the feature vector of the moving window shape and the feature vector of the complete tusk shape. and These are the variances of the feature vectors for the moving window shape and the complete canard shape, respectively. Applying the formula to the frequency domain using Fast Fourier Transform (FFT) can significantly improve processing speed.

[0035] (4) Taking advantage of the characteristic that the bucket teeth are evenly arranged in the horizontal direction, the spatial position of the bucket teeth in the image is determined based on the correlation coefficient diagram, and a threshold is set to judge the missing state of the bucket teeth.

[0036] 9) In the correlation coefficient graph, the higher the correlation coefficient, the higher the probability that the area is a beak tooth. Figure 2 As shown. Therefore, taking advantage of the characteristic that the bucket teeth are evenly arranged in the horizontal direction, the correlation coefficient graph is summed for each row to form a horizontal summation curve. The peak position of the curve is the vertical position of the bucket teeth in the image.

[0037] 10) Extract the horizontal curve of the longitudinal position of the bucket teeth in the correlation coefficient graph, and find the peak value and peak position of all peaks in the curve; 11) Based on the number of bucket teeth installed on the bucket N Extract the peak value of the front peak. N Each peak and its corresponding peak position; 12) Calculate the average distance between all adjacent peaks and the ratio between the distance between each adjacent peak and the average distance; Set a reasonable threshold. When there is a ratio greater than the threshold, it indicates that the bucket teeth are in a missing state. Conversely, when there is a ratio less than the threshold, it indicates that the bucket teeth are in a complete state. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting missing bucket teeth based on morphological features and arrangement consistency, characterized in that, Includes the following steps: Design and install a bucket image acquisition device to ensure that the bucket teeth are always horizontally aligned in the image; Extract bucket frame images and perform image enhancement on the frame images using Retinex theory; A horizontally moving sliding window is set up, and the image moments of each window are calculated to characterize the morphological features. The correlation coefficient between multiple image moments of each window and the bucket tooth template image is calculated using the cross-correlation method to form a correlation coefficient graph. By utilizing the characteristic that the bucket teeth are evenly arranged in the horizontal direction, the spatial position of the bucket teeth in the image is determined based on the correlation coefficient map, and a threshold is set to determine the missing state of the bucket teeth.

2. The method according to claim 1, characterized in that, The process of designing and installing a bucket image acquisition device includes: Select an industrial camera with sufficient resolution and install a light source as needed; equip the industrial camera with protective devices to isolate it from external moisture and dust, and use an aviation interface; install the bucket image acquisition device on the excavator's boom, ensuring that the acquisition device and the bucket maintain a relatively fixed distance.

3. The method according to claim 1, characterized in that, The process of image enhancement using Retinex theory includes: The pixel values ​​of the input image are normalized to the range of 0 to 1, and the illumination information of the image is obtained from the maximum value of the three color channels of the input image. Gaussian filtering is applied to the illumination information to obtain the illumination map; Enhanced images are obtained based on Retinex theory.

4. The method according to claim 1, characterized in that, Before using Retinex theory for image enhancement, the region of interest of the bucket is extracted based on prior knowledge to reduce the amount of image data.

5. The method according to claim 1, characterized in that, The image moments of each window include the calculation of first-order, second-order, third-order, fourth-order, and fifth-order image moments; the morphological feature vector of the bucket tooth template image is obtained by collecting multiple complete bucket tooth image regions and calculating the average value of their first-order to fifth-order image moments.

6. The method according to claim 1, characterized in that, The process of calculating the correlation coefficients of multiple image moments between each window and the bucket tooth template image using the cross-correlation method includes: The normalized correlation coefficient between the feature vector of the moving window morphology and the feature vector of the complete bucket tooth morphology is calculated, and the calculation process is applied to the frequency domain. Fast Fourier Transform is used to improve the processing speed.

7. The method according to claim 6, characterized in that, The normalized correlation coefficient is calculated using the following formula: ， in, Let be the covariance of the feature vector of the moving window shape and the feature vector of the complete tusk shape. and These are the variances of the feature vectors for the moving window shape and the feature vectors for the complete beak shape, respectively.

8. The method according to claim 1, characterized in that, The process of determining the spatial position of the bucket teeth in the image and judging the missing state includes: Summing each row of the correlation coefficient graph forms a horizontal summation curve, with the peak position of the curve representing the vertical position of the bucket teeth in the image. Extract the horizontal curve of the vertical position and find the peak value and peak position of all peaks; Based on the number N of bucket teeth installed on the bucket, extract the top N peaks with larger peak values ​​and their corresponding peak positions; Calculate the average distance between all adjacent peaks, and then calculate the ratio between the distance of each adjacent peak and the average distance. A threshold is set, and when a ratio is greater than the threshold, the bucket teeth are determined to be in a missing state.

9. A bucket tooth missing state detection system based on morphological features and arrangement consistency, characterized in that, include: The image acquisition module includes an industrial camera and a light source mounted on the excavator boom. The camera has a protective device and an aviation interface to keep the bucket teeth horizontally aligned in the image. The image preprocessing module is used to extract bucket frame images captured by the camera and to enhance the frame images using Retinex theory. The morphological feature extraction module is used to set a horizontally moving sliding window and calculate the first to fifth order image moments of each window to characterize morphological features. The matching calculation module is used to calculate the correlation coefficient of the image moments between each window and the bucket tooth template image using the cross-correlation method, and to form a correlation coefficient graph. The missing tooth detection module is used to determine the spatial position of the bucket teeth based on the correlation coefficient map by utilizing the characteristic that the bucket teeth are evenly arranged in the horizontal direction, and to determine the missing tooth status by analyzing the ratio of the distance between adjacent peaks and comparing it with a threshold.

Images