Excavator tooth state monitoring method, device, equipment and system

Abstract

This application relates to the field of intelligent monitoring of construction machinery, specifically a method, device, equipment, and system for monitoring the status of excavator bucket teeth. The solution acquires the joint angle information of the robotic arm using angle sensors installed on the boom, stick, and bucket. Based on the joint angle information and kinematic model, it calculates the position information of the excavator bucket tooth tips in the vehicle coordinate system, which is constructed based on the angle sensor data from the boom, stick, and bucket. When the position information meets preset position requirements, the bucket is scanned using an image acquisition unit to identify the excavator's bucket tooth status. This solves the problems of existing bucket tooth recognition schemes based on fixed perspectives or overall images, which are difficult to adapt to changes in bucket posture, are susceptible to occlusion and interference from complex working conditions, and lack real-time performance and accuracy. It improves the accuracy, real-time performance, and stability of bucket tooth monitoring under complex working conditions, and reduces false alarms and missed alarms.

Description

Technical Field

[0001] This application relates to the field of intelligent detection technology for construction machinery, and in particular to a method, device, equipment and system for monitoring the condition of excavator bucket teeth. Background Technology

[0002] During engineering operations, excavator buckets use teeth mounted on their edges to break, cut, and grab hard materials such as soil, rock, and minerals. However, as vulnerable components that directly bear impact loads, the bucket teeth operate in a high-vibration, high-wear environment for extended periods, making them highly susceptible to detachment or abnormal wear. Therefore, real-time monitoring of the bucket teeth's condition and rapid early warning of detachment events during excavator operations have become urgent requirements for ensuring both construction efficiency and equipment safety.

[0003] Currently, the industry primarily relies on visual recognition technology to monitor excavator bucket tooth detachment. This involves using cameras installed in the cab or on the robotic arm to capture images of the bucket area, or employing techniques that set fixed shooting angles or robotic arm movement trajectories to determine the condition of the bucket teeth.

[0004] However, the above-mentioned technical solutions cannot adapt to the dynamic changes in bucket posture under complex working conditions, resulting in positioning errors in the key areas of the bucket teeth, which in turn leads to technical problems such as misjudgment and decreased accuracy. Summary of the Invention

[0005] This application provides a method, device, equipment, and system for monitoring the condition of bucket teeth in an excavator. By integrating the motion status information of the excavator's robotic arm with the image recognition process of the target area of ​​the bucket, the condition of the bucket teeth is monitored in a targeted manner. This allows the bucket tooth condition detection to be carried out in conjunction with the actual spatial position of the bucket, thereby improving the accuracy, real-time performance, and environmental adaptability of bucket tooth condition monitoring under complex working conditions.

[0006] In a first aspect, this application provides a method for monitoring the condition of bucket teeth of an excavator, wherein an image acquisition unit is installed on the top of the excavator's cab, and the method includes:

[0007] Angle sensors installed on the boom, stick, and bucket of the excavator are used to obtain the joint angle information of the robotic arm.

[0008] Based on the joint angle information and kinematic model of the robotic arm, the position information of the bucket tooth tip of the excavator in the vehicle coordinate system is calculated. The vehicle coordinate system is constructed based on the angle sensor data on the boom, stick, and bucket.

[0009] When the location information meets the preset location requirements, the excavator's bucket teeth are identified by scanning the bucket based on the image acquisition unit.

[0010] In one possible implementation, before acquiring the robotic arm joint angle information using angle sensors mounted on the boom, stick, and bucket of the excavator, the method is as follows:

[0011] With the excavator's robotic arm base as the origin of a three-dimensional coordinate system, the forward direction of the robotic arm is set as the x-axis, the leftward direction of the robotic arm is set as the y-axis, and the upward vertical direction of the robotic arm is set as the z-axis, thus constructing a vehicle body coordinate system;

[0012] The angle sensors on the boom, stick, and bucket are respectively installed at the rotary joints of the robotic arm.

[0013] In one possible implementation, when the location information meets a preset location requirement, the excavator's bucket teeth are identified by scanning the bucket using the image acquisition unit, including:

[0014] When the location information meets the preset location requirements, the bucket is scanned based on the image acquisition unit to obtain bucket point cloud data and bucket image;

[0015] Based on the bucket point cloud data, the contact state of the working surface of the bucket is determined;

[0016] Based on the bucket image, determine the state of the bucket teeth detachment;

[0017] The state of the excavator's bucket teeth is identified based on the working surface contact state and the bucket tooth detachment state.

[0018] In one possible implementation, when the location information meets a preset location requirement, the bucket is scanned based on the image acquisition unit to obtain bucket point cloud data and a bucket image, including:

[0019] When the location information meets the preset location requirements, the lidar and camera in the image acquisition unit are determined;

[0020] Based on the laser radar scanning of the bucket, point cloud data of the bucket is obtained;

[0021] Based on the camera, an image of the bucket is acquired.

[0022] In one possible implementation, determining the working face contact state of the bucket based on the bucket point cloud data includes:

[0023] A preset clustering algorithm is used to cluster the bucket point cloud data to obtain clustering results;

[0024] Based on the clustering results, the working surface contact state of the bucket is determined;

[0025] Wherein, when the clustering result is a single point cluster, the working surface contact state of the bucket is determined to be a contact working surface;

[0026] When the clustering result is multiple point clusters, the working surface contact state of the bucket is determined to be non-contact working surface.

[0027] In one possible implementation, determining the detachment state of the bucket teeth based on the bucket image includes:

[0028] Based on the bucket image, determine the pixel spacing between the bucket teeth;

[0029] Based on the pixel spacing of the slits in the bucket teeth, calculate the actual size corresponding to each pixel;

[0030] Based on the actual size corresponding to each pixel, the bucket image is dimensioned to obtain the dimensioned bucket image;

[0031] Based on the annotated bucket image, the state of the bucket teeth falling off is determined.

[0032] In one possible implementation, determining the detachment state of the bucket teeth based on the annotated bucket image includes:

[0033] Based on a preset target detection algorithm, the labeled bucket image is detected to obtain the tooth gap coordinates and tooth tip coordinates;

[0034] Based on the least squares method, the tooth gap coordinates are fitted to a straight line of the tooth gap;

[0035] Calculate the length of the bucket teeth based on the tooth tip coordinates and the tooth gap line;

[0036] The length of the bucket teeth is compared with a preset length threshold to determine the state of the bucket teeth falling off.

[0037] Secondly, this application provides a device for monitoring the condition of bucket teeth of an excavator, comprising:

[0038] An angle acquisition module is used to acquire the joint angle information of the robotic arm through angle sensors installed on the boom, stick, and bucket of the excavator;

[0039] The position calculation module is used to calculate the position information of the excavator's bucket tooth tip in the vehicle coordinate system based on the joint angle information and kinematic model of the robotic arm. The vehicle coordinate system is constructed based on the angle sensor data on the boom, stick, and bucket.

[0040] The status recognition module scans the bucket based on the image acquisition unit to identify the status of the excavator's bucket teeth when the location information meets the preset location requirements.

[0041] Thirdly, this application provides a bucket tooth condition monitoring device for an excavator, including: a memory and a processor;

[0042] The memory stores computer-executed instructions;

[0043] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0044] Fourthly, this application provides a bucket tooth condition monitoring system for an excavator, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0045] This application relates to the field of intelligent monitoring of construction machinery, specifically a method, device, equipment, and system for monitoring the state of excavator bucket teeth. The solution acquires the joint angle information of the robotic arm by using angle sensors installed on the boom, stick, and bucket of the excavator. Based on the joint angle information and kinematic model, it calculates the position information of the excavator's bucket tooth tips in the vehicle coordinate system, which is constructed based on the angle sensor data from the boom, stick, and bucket. When the position information meets preset position requirements, the bucket is scanned using an image acquisition unit to identify the state of the excavator's bucket teeth. This solution solves the problems of existing bucket tooth recognition schemes based on fixed perspectives or overall images, which are difficult to adapt to changes in bucket posture, are easily affected by obstruction and complex working conditions, and lack real-time performance and accuracy. It improves the accuracy, real-time performance, and stability of bucket tooth monitoring under complex working conditions, and reduces false alarms and missed alarms. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating an embodiment of the excavator bucket tooth condition monitoring method provided in this application;

[0048] Figure 2 A flowchart illustrating Embodiment 2 of the excavator bucket tooth condition monitoring method provided in this application;

[0049] Figure 3 A flowchart illustrating Embodiment 3 of the excavator bucket tooth condition monitoring method provided in this application;

[0050] Figure 4This is a schematic diagram of the excavator structure proposed in the embodiments of this application;

[0051] Figure 5 A schematic diagram of the structure of the excavator bucket tooth condition monitoring device provided in this application;

[0052] Figure 6 A schematic diagram of the structure of the excavator bucket tooth condition monitoring device provided in this application.

[0053] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0055] This application applies to real-time monitoring of bucket tooth status in excavators operating in complex construction environments such as mines, construction sites, and ports. The system is deployed on the top of the excavator's cab (equipped with LiDAR and cameras) and on the robotic arm (equipped with angle sensors), running deep learning models and kinematic algorithms through a domain controller (onboard computing unit). When the excavator is performing digging operations, the system continuously monitors the bucket's posture and the contact status with the working surface, triggering image acquisition and bucket tooth recognition only when the bucket is in the optimal shooting position and not in contact with the working surface, thus avoiding misjudgments and resource waste.

[0056] In existing technologies, bucket tooth recognition relies on fixed-angle shooting or robotic arm trajectory planning, leading to frequent misjudgments in complex construction environments (such as bucket tilting, obstruction, and dynamic vibration). For example, when the bucket vibrates and the tooth tip deviates from the camera's field of view, traditional methods may misjudge it as a tooth falling off, triggering unnecessary alarms. Furthermore, the recognition process using full-image convolutional neural networks requires processing a large amount of pixel data, resulting in long single-recognition times and difficulty meeting real-time requirements. Simultaneously, recognition methods based on the overall shape of the bucket teeth have low sensitivity to localized wear or partial detachment, easily leading to missed detections. These problems are particularly prominent in high-intensity work scenarios such as mining, resulting in increased equipment downtime and decreased construction efficiency.

[0057] To address the aforementioned technical challenges, this application constructs a multimodal recognition system that dynamically locates key areas of the bucket and accurately identifies the state of the bucket teeth by integrating robotic arm kinematics calculations, environmental perception data, and deep learning key point detection algorithms. The system first uses angle sensors and LiDAR point cloud data to dynamically determine whether the bucket is in a suitable physical position for tooth identification. Secondly, it extracts key points of the tooth gaps and tips using a deep learning model and calculates abnormal tooth lengths based on geometric analysis, thereby achieving high-precision, low-resource-consumption identification of detached teeth. This technological concept breaks through the reliance of traditional visual recognition on fixed-angle shooting and full image processing. Through a layered processing logic (physical positioning → key point detection → geometric calculation), it solves the problems of positioning misjudgment, low computational efficiency, and insufficient recognition accuracy in existing technologies.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0059] Figure 1 This is a flowchart illustrating an embodiment of the excavator bucket tooth condition monitoring method provided in this application. Figure 1 As shown, this embodiment of the application uses an excavator as an engineering machinery platform for monitoring the condition of the bucket teeth. An image acquisition unit is fixedly installed on the top of its cab, and the robotic arm consists of a boom, stick, and bucket forming a continuous motion chain. The method includes:

[0060] S101. Obtain the joint angle information of the robotic arm through angle sensors installed on the boom, stick, and bucket of the excavator.

[0061] In this step, the robotic arm joint angle information specifically includes the rotation angles of the boom, stick, and bucket. This type of data forms the basis for the input of the subsequent kinematic model.

[0062] In practical applications, the angle sensor proposed in this application can be a magnetically encoded angle sensor or a rotary encoder. Its output signal is connected to the excavator's onboard processing unit via a protective wiring harness. The onboard processing unit completes the conversion between angle data and posture data based on a pre-set coordinate transformation relationship. In addition, the coordinate origin at the robotic arm base can be determined in conjunction with the overall machine assembly reference. The directions of the x-axis, y-axis, and z-axis are defined in accordance with the machine's travel direction, lateral direction, and vertical direction to reduce directional ambiguity during coordinate conversion in subsequent steps.

[0063] In this embodiment, the excavator's onboard control unit collects data from various angle sensors and forms a set of joint angle data at the corresponding sampling time. For example, the robotic arm joint angle information obtained at a certain sampling time can be represented as θ1, θ2, and θ3, where θ1 represents the boom angle, θ2 represents the stick angle, and θ3 represents the bucket angle, all three reflecting the current bucket posture.

[0064] After this step is completed, angle data corresponding to the sampling time is generated and used as input for the next step. Based on the above analysis, it can be seen that by obtaining the joint angle information of the robotic arm from the boom, stick, and bucket, the subsequent calculations no longer rely on the attitude inference from the image alone, but use the actual attitude of the mechanical structure as the basic data source, so that the determination of whether the bucket is in a identifiable position is based on continuously updated sensor data.

[0065] S102. Based on the joint angle information and kinematic model of the robotic arm, calculate the position information of the bucket tooth tip of the excavator in the vehicle coordinate system.

[0066] In this step, the kinematic model is used to describe the linkage and rotational relationships between the boom, stick, and bucket, with the bucket tooth tip serving as the target point for position determination.

[0067] The vehicle coordinate system is constructed based on angle sensor data from the boom, stick, and bucket, serving as a unified reference coordinate system for expressing the spatial position of the target point. Specifically, the vehicle coordinate system sets the excavator's boom base as the origin of the three-dimensional coordinate system, designates the forward direction of the boom as the x-axis, the leftward direction as the y-axis, and the upward vertical direction as the z-axis. This allows the postures of each boom joint to be expressed under the same coordinate reference, facilitating the subsequent conversion of boom joint angle information into the spatial pose of the boom's end effector.

[0068] During operation, angle sensors at the boom, stick, and bucket joints synchronously output the joint angle information of the robotic arm. After establishing a vehicle coordinate system with the robotic arm base, the system uniformly expresses the angles of each joint and provides basic data for subsequent calculation of the bucket tooth tip position. Since the coordinate system is established with the robotic arm base as the origin and along the overall machine direction, the angles of each joint can be directly mapped to their positional relationships in the vehicle space, thus providing a stable reference for subsequent identification of the bucket attitude.

[0069] By adopting the above method, the joint angle information of the robotic arm can be collected and expressed in a unified vehicle coordinate system. The relationship between the sensor installation position and the joint movement is clear, which facilitates the subsequent calculation of the bucket pose and provides a reliable angle input basis for monitoring the bucket teeth status.

[0070] In this embodiment, the excavator's onboard processing unit pre-stores relevant geometric parameters of the excavator's robotic arm. Based on the angle data generated in S101, the excavator's position calculation module calculates the tooth tip position according to kinematic relationships, obtaining the tooth tip position vector in the vehicle coordinate system. The components corresponding to the position vector are the position information of the bucket tooth tip in the vehicle coordinate system.

[0071] In one possible embodiment, the vehicle coordinate system can be updated to reflect the relative postures of the boom, stick, and bucket based on current angle sensor data, ensuring that the tooth tip position corresponds to the actual working posture. For example, when changes in angle data cause changes in the robotic arm's posture, the position calculation module synchronously updates and outputs new tooth tip spatial coordinates.

[0072] This step utilizes the joint angle information of the robotic arm and the kinematic model to solve the position information of the bucket tooth tip in the vehicle coordinate system. This allows for a quantitative description of the current spatial position of the bucket before image acquisition, providing a calculable basis for determining whether it has entered a suitable scanning area. This separates the attitude change from the bucket tooth status recognition process.

[0073] S103. When the location information meets the preset location requirements, the excavator's bucket teeth status is identified by scanning the bucket based on the image acquisition unit.

[0074] In this step, the preset position requirement is used to limit the scanning and recognition to the space range that the bucket tooth tip can enter before the image acquisition unit can observe.

[0075] The image acquisition unit is installed on the top of the cab to acquire scan data of the bucket area, and the status of the bucket teeth is the final identification object.

[0076] In this embodiment, the excavator's onboard processing unit pre-establishes an effective recognition space window corresponding to the field of view installed on the top of the cab. This space window can be defined using a positional range in the vehicle's coordinate system. For example, the spatial range of the tooth tip position can be used to define the recognizable area.

[0077] When the tooth tip position information output by S102 meets the above-mentioned spatial window conditions, the excavator's recognition trigger module sends a collection command to the image acquisition unit, and the image acquisition unit scans the bucket. For example, when the distance between the tooth tip and the bottom plane of the excavator's tracks is greater than 0.5m and less than the top plane of the cab, and the angle between the line connecting the bucket and the tooth tip and the vehicle body plane is less than -30° and greater than -100°, the bucket tooth tip faces downwards, resulting in better image recognition.

[0078] In practice, the image acquisition unit can limit the scanning range to the corresponding area of ​​the bucket based on known installation parameters, and identify the status of the bucket teeth based on the acquired scanning data. The identification result is output in the form of a status label and stored in association with the current time information and the acquired information. When the position information does not meet the preset position requirements, the image acquisition unit remains in standby mode without triggering until the next position determination meets the conditions before starting the scan.

[0079] Based on the above analysis, this step establishes the scanning trigger on the condition that the tip of the bucket tooth has entered the effective recognition space, so that image acquisition and state recognition are performed for the predetermined observable posture, the processing range is concentrated in the target area of ​​the bucket, and the recognition result directly corresponds to the actual state of the bucket tooth that meets the field of view constraint.

[0080] This application provides a method for monitoring the state of bucket teeth in an excavator. First, an angle sensor acquires the actual posture data of the robotic arm. Then, a kinematic model converts the posture data into the position of the bucket teeth tip in the vehicle's coordinate system. Finally, an image acquisition unit is triggered to perform scanning and recognition only when this position falls within a preset recognition space, ensuring that the bucket tooth state judgment is based on prior position constraints. Thus, during continuous operation involving the boom, stick, and bucket, the recognition process does not continuously process large-scale images at any given moment, but rather focuses on the target area of ​​the bucket that meets the position conditions, enabling bucket tooth state monitoring to adapt to posture changes under complex working conditions.

[0081] Figure 2 This is a flowchart illustrating Embodiment Two of the excavator bucket tooth condition monitoring method provided in this application. Figure 2 As shown, in Figure 1 Based on the embodiments, when the location information meets the preset location requirements, the excavator's bucket teeth status is identified by scanning the bucket using the image acquisition unit, including:

[0082] S201. When the location information meets the preset location requirements, the bucket is scanned based on the image acquisition unit to obtain bucket point cloud data and bucket image.

[0083] In this application, the image acquisition unit consists of a lidar and a camera, both mounted on the same bracket on the top of the cab and kept in a fixed relative position to maintain a consistent field of view of the bucket.

[0084] In actual implementation, once the bucket tooth tip position information obtained from the joint angle calculation of the robotic arm meets the preset position requirements, the image acquisition unit is controlled to enter the trigger state, and the lidar and camera are simultaneously activated to collect data on the bucket area.

[0085] In this embodiment, the lidar can be a rotating mirror-type 3D lidar, whose scanning lines can cover the working surface of the bucket and the mounting area of ​​the bucket teeth. The collected point cloud data is registered with coordinates to form a local point cloud model of the bucket. For example, the lidar emits a laser beam and receives the echo signal from the bucket surface to form point cloud data. The point cloud data is used to characterize the outline, edges, and local surface undulations of the bucket in three-dimensional space.

[0086] The camera can be an industrial camera with a resolution of at least 1920×1080 to ensure that the edges and missing areas of the bucket teeth have identifiable image quality. For example, the camera acquires visible light images of the bucket and outputs image data to characterize the appearance features of the bucket and its teeth.

[0087] Specifically, once the bucket tooth tip position information calculated based on the robotic arm joint angle information meets the preset position requirements, the control unit outputs a trigger signal to determine that the LiDAR and camera have entered the working state. The LiDAR scans the front and side areas of the bucket within a preset time window and converts the collected distance and reflection intensity information into bucket point cloud data. At the same time, the camera captures images of the bucket according to synchronous exposure parameters, outputting a bucket image containing the outline of the bucket teeth and the edge of the bucket. The point cloud data and image data can be sent to subsequent recognition modules for determining the contact state of the working surface and the state of bucket tooth detachment.

[0088] This step initiates point cloud data and image acquisition only when the bucket is in a suitable position, enabling the LiDAR and camera to simultaneously generate effective observations of the bucket, thus providing a consistent data foundation for subsequent status identification. Since point cloud data and image data respectively represent the spatial morphology and appearance details of the bucket, they can simultaneously meet the needs of spatial and visual judgment under complex working conditions.

[0089] S202. Based on the bucket point cloud data, determine the contact state of the bucket's working face.

[0090] The working face contact state is determined based on the spatial relationship between the bottom of the bucket and the points on the external surface in the point cloud data.

[0091] In this embodiment, a preset clustering algorithm is used to perform clustering analysis on the bucket point cloud data. Based on the clustering results, the contact state of the bucket's working surface is determined to be either a contact working surface or a non-contact working surface. The preset clustering algorithm is used to group spatially adjacent points with high connectivity and outputs the number of point clusters and their corresponding clustering results. The algorithm can employ a distance-threshold-based clustering method, a density-based clustering method, or other algorithms suitable for point cloud grouping. In practical applications, other models or implementation methods of the algorithm can also be selected, and this embodiment does not limit this.

[0092] In practical implementation, the bucket point cloud data can first be filtered, denoised, and have its coordinates unified to reduce the interference of discrete noise points on the clustering results. Then, the processed point cloud data is input into a preset clustering algorithm to form several point clusters according to the set spatial distance threshold or density connectivity rules. If the bucket is in contact with the working surface, the bucket surface usually appears as a highly connected overall structure in the point cloud data, forming a single point cluster after clustering. If the bucket is not in contact with the working surface, the leading edge, sidewall, or locally restricted area of ​​the bucket will appear as multiple separate point cloud data regions due to occlusion, compression, or segmentation by the working surface, forming multiple point clusters after clustering. The algorithm outputs the corresponding contact state based on this.

[0093] For example, the DBSCAN clustering algorithm is used to group the point cloud data, with a cluster radius of 0.25m and a minimum number of neighborhood points of 10. If the clustering result is a single point cluster, it is determined that the bucket is in contact with the working surface; if there are multiple point clusters, it is determined that the bucket is not in contact with the working surface.

[0094] The contact status at the working face indicates whether the bucket is in contact with the soil, rock surface, or other working medium. A single point cluster corresponds to continuous point cloud data of the entire bucket, while multiple point clusters correspond to discontinuous areas segmented into the bucket point cloud data, thus forming a determination result of whether the bucket is in contact with the working face or not. This determination result can be further used as a prerequisite for identifying the bucket teeth status and input into subsequent identification processes.

[0095] This step directly determines the bucket contact state based on point cloud data clustering results. It can complete state identification without relying on complex image features and maps the difference between continuous and split point cloud data into clear contact markers, thereby improving the consistency and stability of contact state judgment. Since this determination process only focuses on the number of point clusters, the calculation process is relatively simple and can provide a reliable status basis for subsequent bucket tooth status monitoring.

[0096] S203. Based on the bucket image, determine the state of the bucket teeth falling off.

[0097] Among them, the bucket image is an image of the bucket tooth area captured by the camera and input into the image processing module. It retains the pixel information of the leading edge of the bucket tooth, the tooth gap and the adjacent boundary, which is used for subsequent size conversion and detachment determination.

[0098] The status of detached teeth is obtained by comparing the outline, length, and continuity features of the teeth in the image with a preset threshold.

[0099] In this application, the pixel spacing of the bucket teeth gaps is first determined based on the bucket image. The pixel spacing of the bucket teeth gaps refers to the distance between the gaps between adjacent bucket teeth in the pixel coordinate system. It is usually obtained through edge extraction and contour localization and is a basic parameter for establishing the correspondence between pixel scale and actual scale.

[0100] Then, based on the pixel spacing of the slits in the slits, the actual size corresponding to each pixel is calculated; the actual size corresponding to each pixel is the length value represented by a unit pixel obtained by converting the pixel spacing of the slits in the slits, reflecting the size mapping relationship of each pixel in the image in the actual space.

[0101] Finally, based on the actual dimensions corresponding to each pixel, the bucket image is dimensioned to obtain a dimensioned bucket image, which is then used to determine the state of the bucket teeth detachment. Dimensioning involves mapping pixel coordinates to actual dimensions, and the dimensioned bucket image can be directly used for length comparison and target missingness detection.

[0102] In its implementation, after the camera acquires images of the bucket from the front or side, the image processing module first performs noise reduction, grayscale conversion, and edge enhancement. Then, based on the contour of the bucket teeth, it identifies the pixel positions of adjacent tooth gaps and calculates the pixel spacing between the two tooth gaps. Combining the known standard tooth spacing or pre-calibrated reference dimensions, the system converts the pixel spacing of the tooth gaps into the actual length corresponding to each unit pixel, and generates a size scale or coordinate annotation information on the image accordingly, ensuring that key boundaries in the bucket image are accompanied by actual size data. Subsequently, the system performs gap identification and integrity comparison on the tooth area based on the annotated image. When the expected tooth tip contour is not detected in the area corresponding to a certain tooth position, or the relationship between adjacent annotated dimensions is inconsistent with the standard dimension relationship of a complete bucket tooth, it is determined that a tooth has fallen off.

[0103] In this step, the image dimension annotation allows the determination of the detached teeth status to no longer rely solely on pixel differences, but rather on comparison based on actual dimensions. This converts pixel information into measurable size information, thereby ensuring that the determination of missing teeth corresponds consistently with the actual structural state and improving the stability and interpretability of the determination results.

[0104] S204. Identify the excavator's bucket tooth status based on the working face contact status and the bucket tooth detachment status.

[0105] In this step, the working face contact state and the bucket tooth detachment state are jointly determined. When both satisfy the preset logical relationship, the excavator's bucket tooth state result is output.

[0106] This embodiment of the application triggers point cloud data and image acquisition only when position requirements are met, ensuring the bucket is always in a recognizable observation posture. The point cloud data is then used to determine contact relationships, and the images are used to determine the integrity of the bucket teeth, and these are combined for identification, thus matching the bucket tooth status output with the bucket's operating posture. Because point cloud data and images perform different discrimination tasks, the identification results are more specific, and data acquisition and processing only when position requirements are met reduces unnecessary calculations.

[0107] Figure 3 This is a flowchart illustrating Embodiment 3 of the excavator bucket tooth condition monitoring method provided in this application. Figure 3 As shown, in Figure 2 Based on the embodiments, the state of the bucket teeth detachment is determined based on the annotated bucket image, including:

[0108] S301. Based on the preset target detection algorithm, the labeled bucket image is detected to obtain the tooth gap coordinates and tooth tip coordinates.

[0109] Among them, the preset target detection algorithm is used to identify the key parts of the bucket teeth in the image and output coordinate information. The output tooth gap coordinates represent the set of tooth gap positions between adjacent bucket teeth, and the tooth tip coordinates represent the set of bucket tooth tip positions.

[0110] The labeled bucket image is image data after dimension annotation. Dimension annotation is used to establish the correspondence between pixel size and actual size, so that the subsequent detection results of tooth gap coordinates and tooth tip coordinates can be processed under a unified scale.

[0111] In practice, after the labeled bucket image is input into the preset target detection algorithm, the algorithm extracts features from the bucket tooth region and outputs the tooth gap coordinates and tooth tip coordinates corresponding to each bucket tooth.

[0112] In this embodiment, the preset target detection algorithm can adopt a key point detection model based on a convolutional neural network, such as the YOLO target detection network training model, or other detection models that can output the location of key points. In practical applications, other models of the algorithm can also be selected, and this embodiment does not limit this.

[0113] S302. Based on the least squares method, the tooth gap coordinates are fitted to a straight line.

[0114] Among them, the tooth gap straight line is a reference benchmark obtained by linearly fitting the tooth gap coordinates, which is used to characterize the geometric trend of the bucket tooth installation edge.

[0115] In practice, after obtaining the tooth gap coordinates, the least squares method is used to fit a straight line to multiple tooth gap points in order to reduce the deviation caused by single-point detection error and make the fitted tooth gap straight line closer to the real baseline of the bucket tooth arrangement.

[0116] S303. Calculate the length of the bucket teeth based on the tooth tip coordinates and the tooth gap line.

[0117] The length of the tooth is the actual length calculated based on the geometric distance between the tooth tip coordinates and the straight line of the tooth gap, combined with the dimensioning results.

[0118] In specific implementation, the bucket tooth length is calculated based on the vertical distance from the tooth tip coordinate to the tooth gap straight line, combined with the conversion relationship between pixels and actual dimensions established by image dimension marking.

[0119] S304, comparing the bucket tooth length with a preset length threshold, and determining the bucket tooth shedding state of the bucket.

[0120] Wherein, the preset length threshold is a comparison reference for determining whether the bucket tooth is shed, and its value can be preset according to the standard length of the intact bucket tooth, the allowable wear allowance and the identification error range.

[0121] In specific implementation, when the calculated bucket tooth length is lower than the preset length threshold, it is determined that the corresponding bucket has a bucket tooth shedding state; when the bucket tooth length is not lower than the preset length threshold, it is determined that the bucket tooth is in an unshed state.

[0122] The embodiment of the present application combines key point detection of bucket teeth, tooth gap straight line fitting and length threshold comparison, so that the judgment of bucket tooth shedding does not depend on the overall contour of the bucket teeth, and the identification is completed based on key geometric features. Since the tooth gap straight line provides a stable reference, the calculation result of the bucket tooth length has good scale consistency, so that the bucket tooth missing state can be clearly determined.

[0123] Figure 4 is a schematic structural diagram of an excavator according to an embodiment of the present application. As Figure 4 shown, the excavator includes a crawler 401, a cab 402, a boom 403, a dipper arm 404 and a bucket 405. Angle sensors are respectively installed on the boom 403, the dipper arm 404 and the bucket 405, and an image acquisition unit is deployed on the cab.

[0124] Wherein, the angle sensors on the boom 403, the dipper arm 404 and the bucket 405 are respectively installed at the rotary joints of the mechanical arm to output mechanical arm joint angle information in real time.

[0125] Secondly, the angle sensor will be calibrated after installation, and it is qualified if the angle deviation is within 0.5°, and the converted tooth tip coordinate deviation is less than 5cm.

[0126] The sensor and the joint can be fixedly connected through a clamping seat or a bolt bracket, and a sealed protective housing is provided to adapt to dust and vibration environments. In practical applications, other models can also be selected for this component, which is not limited in the embodiments of the present application.

[0127] In actual operation, the angle sensors can collect the boom angle θ1, the dipper arm angle θ2 and the bucket angle θ3 respectively. Wherein, the coordinate system where the boom angle θ1 is located is formed by Y1 and Z1. The coordinate system where the dipper arm angle θ2 is located is formed by Y2 and Z2. The coordinate system where the bucket angle θ3 is located is formed by Y3 and Z3.

[0128] For example, in Figures 1 to 3 Based on the example, the coordinates of the tooth tip in the vehicle body coordinate system are calculated using the following formula:

[0129] x=L 动臂 cosθ1+L 斗杆 cos(θ1+θ2)+L 铲斗 cos(θ1+θ2+θ3)

[0130] y=0

[0131] z=L 动臂 sinθ1+L 斗杆 sin(θ1+θ2)+L 铲斗 sin(θ1+θ2+θ3)

[0132] Among them, L 动臂 L represents the length of the boom. 斗杆 L represents the length of the boom. 铲斗 This indicates the length of the bucket.

[0133] In addition, Figure 3 Based on the previous example, when performing gap fitting using the least squares method, four points can be set as: (x1, y1), (x2, y2), (x3, y3), and (x4, y4). The equation L is: y = mx + b, and the formulas for calculating m and b are as follows:

[0134]

[0135] Where: n=4 (number of points);

[0136] ∑x i =x1+x2+x3+x4;

[0137] ∑y i =y1+y2+y3+y4;

[0138] ∑(x i y i ) = x1y1 + x2y2 + x3y3 + x4y4;

[0139] ∑(x i 2 )=x1 2 +x2 2 +x3 2 +x4 2 .

[0140] Then, the formula for calculating the straight-line distance d from each tooth tip (x0, y0) to the tooth gap is as follows:

[0141]

[0142] Wherein, d is converted into an actual length d', which is compared with a set judgment threshold D. If d' < D, it is determined that the bucket tooth is fallen off.

[0143] Wherein, the judgment threshold D is generally set as half of the length of the original bucket tooth, which can be manually set according to working condition requirements, and the original size of different buckets shall be calibrated in advance. The quantity and size of bucket teeth are calibrated in advance, numbered sequentially from left to right, and the number of the fallen-off bucket tooth is output.

[0144] Figure 5 is a structural schematic diagram of the bucket tooth state monitoring device for an excavator provided in the present application. As Figure 5 shown, the bucket tooth state monitoring device 50 for an excavator provided in this embodiment includes:

[0145] an angle acquisition module 501, configured to acquire manipulator joint angle information through angle sensors installed on the boom, dipper arm and bucket of the excavator;

[0146] a position calculation module 502, configured to calculate position information of tooth tips of the excavator bucket in a body coordinate system based on the manipulator joint angle information and a kinematic model, wherein the body coordinate system is constructed according to data of the angle sensors on the boom, dipper arm and bucket;

[0147] a state identification module 503, configured to, when the position information meets a preset position requirement, scan the bucket based on an image acquisition unit, and identify the state of the bucket teeth of the excavator.

[0148] In a possible implementation manner, the manipulator base of the excavator is set as the origin of the three-dimensional coordinate system, the forward direction of the manipulator is set as the x-axis, the left direction of the manipulator is set as the y-axis, and the vertical upward direction of the manipulator is set as the z-axis to construct the body coordinate system;

[0149] the angle sensors on the boom, dipper arm and bucket are respectively installed at the rotary joints of the manipulator.

[0150] In a possible implementation manner, the state identification module 503 is further specifically configured to:

[0151] when the position information meets the preset position requirement, scan the bucket based on the image acquisition unit to acquire bucket point cloud data and a bucket image;

[0152] determine the working surface contact state of the bucket based on the bucket point cloud data;

[0153] determine the falling-off state of the bucket teeth based on the bucket image;

[0154] identify the state of the bucket teeth of the excavator based on the working surface contact state and the falling-off state of the bucket teeth.

[0155] In one possible implementation, the state recognition module 503 is further specifically used for:

[0156] When the location information meets the preset location requirements, the lidar and camera in the image acquisition unit are determined;

[0157] The bucket is scanned using LiDAR to obtain bucket point cloud data;

[0158] The camera captures images of the bucket.

[0159] In one possible implementation, the state recognition module 503 is further specifically used for:

[0160] A preset clustering algorithm is used to cluster the bucket point cloud data to obtain the clustering results;

[0161] Based on the clustering results, determine the contact state of the bucket's working surface;

[0162] When the clustering result is a single point cluster, the working surface contact state of the bucket is determined to be a contact working surface.

[0163] When the clustering result is multiple point clusters, the working surface contact state of the bucket is determined to be non-contact working surface.

[0164] In one possible implementation, the state recognition module 503 is further specifically used for:

[0165] Based on the bucket image, determine the pixel spacing between the bucket teeth;

[0166] Calculate the actual size of each pixel based on the pixel spacing of the slits in the bucket teeth.

[0167] Based on the actual size corresponding to each pixel, the bucket image is dimensioned to obtain the dimensioned bucket image;

[0168] Based on the labeled bucket image, the state of the bucket teeth falling off is determined.

[0169] In one possible implementation, the state recognition module 503 is further specifically used for:

[0170] Based on a preset target detection algorithm, the labeled bucket image is detected to obtain the tooth gap coordinates and tooth tip coordinates;

[0171] Based on the least squares method, the tooth gap coordinates are fitted to a straight line of the tooth gap;

[0172] Calculate the length of the bucket teeth based on the tooth tip coordinates and the tooth gap line;

[0173] The length of the bucket teeth is compared with a preset length threshold to determine the state of the bucket teeth falling off.

[0174] The excavator bucket tooth status monitoring device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0175] Figure 6 This is a structural schematic diagram of the excavator bucket tooth condition monitoring device provided in this application. Figure 6 As shown, the excavator bucket tooth condition monitoring device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the excavator bucket tooth condition monitoring device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0176] In the specific implementation process, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the above-described excavator bucket tooth status monitoring method.

[0177] By pre-storing computer execution instructions in memory 602 and having them executed by processor 601 in a vehicle-mounted environment, the device can implement all steps in the bucket tooth condition monitoring method. Processor 601 first calculates the position of the bucket tooth tip in the vehicle coordinate system based on the angle information of the boom, stick, and bucket joints collected by the angle sensor, combined with a kinematic model, thus determining whether the bucket is in a suitable position for identification. If the preset position requirements are met, the image acquisition unit is then controlled to scan and identify the bucket area, focusing image processing on effective timing and areas, thereby reducing unnecessary calculations and interference from complex working conditions. Therefore, this device can better adapt to changes in posture, vibration, dust, and strong light environments during excavator operation, improving the accuracy, real-time performance, and stability of identifying missing or worn bucket teeth.

[0178] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0179] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0180] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0181] This application also provides a bucket tooth condition monitoring system for an excavator, including a computer program that, when executed by a processor, implements the aforementioned excavator bucket tooth condition monitoring method.

[0182] This computer program can be deployed in the excavator's onboard processing unit and works in conjunction with the image acquisition unit on top of the cab and the angle sensors mounted on the boom, stick, and bucket. When the processor executes the computer program, it first calculates the position information of the bucket teeth tip in the vehicle coordinate system based on the angle information of each joint and the kinematic model. Then, when the position information meets the preset position requirements, it triggers image acquisition and bucket tooth status recognition, thereby avoiding blind recognition when the bucket is deflected, raised, or obstructed. This allows image processing to focus on suitable timing and areas for recognition, reducing unnecessary calculations and improving the accuracy and real-time performance of judging the status of missing or worn bucket teeth under complex working conditions. Therefore, it helps reduce false alarms and missed alarms and ensures continuous operation of the excavator.

[0183] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described excavator bucket tooth status monitoring method.

[0184] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0185] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0186] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0189] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0191] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for monitoring the condition of bucket teeth in an excavator, characterized in that, The excavator has an image acquisition unit installed on the top of its cab, and the method includes: Angle sensors installed on the boom, stick, and bucket of the excavator are used to obtain the joint angle information of the robotic arm. Based on the joint angle information and kinematic model of the robotic arm, the position information of the bucket tooth tip of the excavator in the vehicle coordinate system is calculated. The vehicle coordinate system is constructed based on the angle sensor data on the boom, stick, and bucket. When the location information meets the preset location requirements, the excavator's bucket teeth are identified by scanning the bucket based on the image acquisition unit.

2. The method for monitoring the condition of bucket teeth according to claim 1, characterized in that, Before acquiring the joint angle information of the robotic arm using angle sensors installed on the boom, stick, and bucket of the excavator, the method is as follows: With the excavator's robotic arm base as the origin of a three-dimensional coordinate system, the forward direction of the robotic arm is set as the x-axis, the leftward direction of the robotic arm is set as the y-axis, and the upward vertical direction of the robotic arm is set as the z-axis, thus constructing a vehicle body coordinate system; The angle sensors on the boom, stick, and bucket are respectively installed at the rotary joints of the robotic arm.

3. The method for monitoring the condition of bucket teeth according to claim 1 or 2, characterized in that, When the location information meets the preset location requirements, the excavator's bucket teeth are identified by scanning the bucket using the image acquisition unit, including: When the location information meets the preset location requirements, the bucket is scanned based on the image acquisition unit to obtain bucket point cloud data and bucket image; Based on the bucket point cloud data, the contact state of the working surface of the bucket is determined; Based on the bucket image, determine the state of the bucket teeth detachment; The state of the excavator's bucket teeth is identified based on the working surface contact state and the bucket tooth detachment state.

4. The method for monitoring the condition of bucket teeth according to claim 3, characterized in that, When the location information meets the preset location requirements, the bucket is scanned based on the image acquisition unit to obtain bucket point cloud data and bucket image, including: When the location information meets the preset location requirements, the lidar and camera in the image acquisition unit are determined; Based on the laser radar scanning of the bucket, point cloud data of the bucket is obtained; Based on the camera, an image of the bucket is acquired.

5. The method for monitoring the condition of bucket teeth according to claim 3, characterized in that, Based on the bucket point cloud data, the working face contact state of the bucket is determined, including: A preset clustering algorithm is used to cluster the bucket point cloud data to obtain clustering results; Based on the clustering results, the working surface contact state of the bucket is determined; Wherein, when the clustering result is a single point cluster, the working surface contact state of the bucket is determined to be a contact working surface; When the clustering result is multiple point clusters, the working surface contact state of the bucket is determined to be non-contact working surface.

6. The method for monitoring the condition of bucket teeth according to claim 3, characterized in that, Based on the bucket image, determining the state of the bucket teeth detachment includes: Based on the bucket image, determine the pixel spacing between the bucket teeth; Based on the pixel spacing of the slits in the bucket teeth, calculate the actual size corresponding to each pixel; Based on the actual size corresponding to each pixel, the bucket image is dimensioned to obtain the dimensioned bucket image; Based on the annotated bucket image, the state of the bucket teeth falling off is determined.

7. The method for monitoring the condition of bucket teeth according to claim 6, characterized in that, Based on the annotated bucket image, the state of the bucket teeth detachment is determined, including: Based on a preset target detection algorithm, the labeled bucket image is detected to obtain the tooth gap coordinates and tooth tip coordinates; Based on the least squares method, the tooth gap coordinates are fitted to a straight line of the tooth gap; Calculate the length of the bucket teeth based on the tooth tip coordinates and the tooth gap line; The length of the bucket teeth is compared with a preset length threshold to determine the state of the bucket teeth falling off.

8. A device for monitoring the condition of bucket teeth of an excavator, characterized in that, include: An angle acquisition module is used to acquire the joint angle information of the robotic arm through angle sensors installed on the boom, stick, and bucket of the excavator; The position calculation module is used to calculate the position information of the excavator's bucket tooth tip in the vehicle coordinate system based on the joint angle information and kinematic model of the robotic arm. The vehicle coordinate system is constructed based on the angle sensor data on the boom, stick, and bucket. The status recognition module scans the bucket based on the image acquisition unit to identify the status of the excavator's bucket teeth when the location information meets the preset location requirements.

9. A device for monitoring the condition of bucket teeth of an excavator, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A bucket tooth condition monitoring system for an excavator, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images