Method, device, work machine and medium for detecting missing work component condition

Abstract

The application discloses a method and device for detecting missing of a working component, a working machine and a medium. The method comprises the following steps: acquiring a visible light image, an infrared image of the working component, and attitude data of an executing mechanism carrying the working component; determining a visible light target region from the visible light image and an infrared target region from the infrared image based on the attitude data; determining a first recognition result according to the visible light target region; determining a second recognition result according to the infrared target region; and fusing the first recognition result and the second recognition result to obtain a target detection result of the missing of the working component. The application realizes real-time, accurate and anti-interference detection of the missing of the working component under complex working conditions.

Description

Technical Field

[0001] This application relates to the field of operating machinery technology, specifically to a method, apparatus, operating machinery, and medium for detecting missing working parts. Background Technology

[0002] The working parts (such as bucket teeth and shovel teeth) of construction machinery such as excavators and loaders directly contact the materials being worked on, undertaking important functions such as digging and crushing. Under complex working conditions such as mining and construction, these working parts are subjected to the impact and friction of materials for extended periods, making them prone to wear, loosening, or even detachment. If not detected in time, detached working parts may enter the subsequent crushing system with the material, causing equipment blockage or even damage, resulting in serious economic losses.

[0003] Currently, the detection of missing working parts mainly relies on manual visual inspection, where maintenance personnel visually inspect during work breaks. This method has the following drawbacks: first, it requires stopping the machine for inspection, affecting operational efficiency; second, in environments such as mines with high dust levels and complex lighting conditions, it is difficult for personnel to clearly observe the status of all working parts, resulting in a high rate of missed or false detections. Existing technologies have also proposed solutions for automatic detection using sensors, but due to the complex operating conditions, a single sensor cannot simultaneously guarantee both detection accuracy and reliability. For example, detection schemes based on visible light images are prone to failure in dusty or low-light environments.

[0004] Therefore, the existing methods for detecting missing working parts have the problem of low accuracy in the detection results. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, machine tool, and machine-readable storage medium for detecting missing working parts, in order to solve the problem that the accuracy of detection results is low in the existing working part missing detection methods.

[0006] To achieve the above objectives, the first aspect of this application provides a method for detecting missing working parts, applied to operating machinery, the method comprising: Acquire visible light and infrared images of the working components, as well as attitude data of the actuators that support the working components; Based on attitude data, the visible light target region is determined from the visible light image, and the infrared target region is determined from the infrared image; The first identification result is determined based on the visible light target area; The second identification result is determined based on the infrared target area; The first and second identification results are fused together to obtain the target detection results for missing working parts.

[0007] In this embodiment of the application, determining the first identification result based on the visible light target region includes: Input the visible light target region into the pre-built visible light target detection model; The visual features of the working parts in the visible light target area are extracted layer by layer by the visible light target detection model, and the missing state of each part of the working parts is identified according to the visual features to obtain the missing state of each part of the working parts and the corresponding confidence level. The missing status of each part of the working component and the corresponding confidence level are used as the first identification result.

[0008] In this embodiment of the application, determining the second identification result based on the infrared target region includes: Input the infrared target region into the pre-built infrared target detection model; The temperature distribution features of the working parts in the infrared target area are extracted layer by layer by the infrared target detection model, and the missing state of each part of the working parts is identified according to the temperature distribution features to obtain the missing state of each part of the working parts and the corresponding confidence level. The missing status of each part of the working component and the corresponding confidence level are used as the second identification result.

[0009] In this embodiment of the application, determining a visible light target region from a visible light image and an infrared target region from an infrared image based on attitude data includes: The spatial pose information of the actuator is calculated based on the attitude data; Based on spatial pose information and imaging parameters of the acquisition device, the visible light target area of ​​the working component is determined in the visible light image, and the infrared target area of ​​the working component is determined in the infrared image.

[0010] In this embodiment of the application, based on spatial pose information and imaging parameters of the acquisition device, the visible light target region of the working component is determined in the visible light image, and the infrared target region of the working component is determined in the infrared image, including: The spatial pose information is transformed from the world coordinate system to the visible light acquisition device coordinate system and the infrared acquisition device coordinate system, respectively. Based on the camera intrinsic parameters of the visible light acquisition device, the coordinates in the visible light acquisition device coordinate system are projected to the corresponding pixel coordinate system to obtain the first pixel coordinate mapping of the working part in the visible light image; and based on the camera intrinsic parameters of the infrared acquisition device, the coordinates in the infrared acquisition device coordinate system are projected to the pixel coordinate system to obtain the second pixel coordinate mapping of the working part in the infrared image. Based on the geometric dimensions of the working component and the first and second pixel coordinate mappings, the coverage area of ​​the working component in the visible light image and the infrared image are determined respectively, thus obtaining the visible light target area and the infrared target area.

[0011] In this embodiment of the application, before determining the visible light target region from the visible light image and the infrared target region from the infrared image based on attitude data, the method further includes: Based on the attitude data, determine whether the current attitude of the actuator meets the preset extraction conditions; If the preset extraction conditions are met, the steps of determining the visible light target region from the visible light image and the infrared target region from the infrared image based on the pose data are performed. The preset extraction conditions are that the working part is located within the field of view of the acquisition device, and the angle between the base plate of the actuator and the vertical direction is less than the preset angle threshold.

[0012] In this embodiment of the application, the first identification result and the second identification result are fused to obtain the target detection result of the missing working part, including: Obtain the current operating status information of the machinery; The fusion weights of the first and second identification results are determined based on the work condition information. The first and second identification results are weighted and fused according to the fusion weights to obtain the comprehensive confidence score. The target detection results are output based on the overall confidence level.

[0013] In this embodiment of the application, the target detection result is output based on the comprehensive confidence level, including: If the overall confidence level is greater than or equal to the first preset threshold, the target detection result is determined to be a missing working part; If the overall confidence level is less than the first preset threshold and greater than or equal to the second preset threshold, the target detection result is determined to be a suspected missing working part; If the overall confidence level is less than the second preset threshold, the target detection result is determined to be that the working component is normal. The first preset threshold is greater than the second preset threshold.

[0014] A second aspect of this application provides an apparatus for detecting the absence of a working component, comprising: The data acquisition module is used to acquire visible light images and infrared images of the working parts, as well as the attitude data of the actuators that support the working parts; The first determining module is used to determine the visible light target region from the visible light image and the infrared target region from the infrared image based on the attitude data. The second determining module is used to determine the first identification result based on the visible light target area; The third determining module is used to determine the second identification result based on the infrared target area; The fusion processing module is used to fuse the first recognition result with the second recognition result to obtain the target detection result for the missing working part.

[0015] A third aspect of this application provides a work machine, comprising: Working parts; The aforementioned device for detecting missing working components.

[0016] A fourth aspect of this application provides a machine-readable storage medium storing instructions for causing a machine to perform the aforementioned method for detecting missing working components.

[0017] The above technical solution first acquires visible light and infrared images of the working component, as well as attitude data of the actuator carrying the working component. Then, based on the attitude data, the visible light target region is determined from the visible light image, and the infrared target region is determined from the infrared image. Simultaneously, a first recognition result is determined based on the visible light target region, followed by a second recognition result based on the infrared target region. Finally, the first and second recognition results are fused to obtain the target detection result for the missing working component. This application achieves real-time, accurate, and interference-resistant detection of missing working components under complex working conditions by simultaneously acquiring visible light, infrared, and attitude data, utilizing attitude data to assist in quickly and accurately locating the target region in both types of images, and independently identifying the target region based on visible light and infrared target detection models before fusion processing.

[0018] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 A flowchart illustrating a method for detecting missing working components provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the principle of bucket posture during ROI acquisition, provided for a specific embodiment of this application; Figure 3 An example diagram of the ROI of a bucket tooth is provided for a specific embodiment of this application; Figure 4This is a structural block diagram of a device for detecting missing working parts, provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0022] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0023] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0024] Figure 1 This is a flowchart illustrating a method for detecting missing working components according to an embodiment of this application. Figure 1 As shown in the figure, this application provides a method for detecting missing working parts, which may include the following steps.

[0025] Step S101: Acquire visible light image and infrared image of the working component, as well as attitude data of the actuator carrying the working component.

[0026] In this embodiment, "operating machinery" refers to engineering machinery equipment that directly contacts the work object through its working parts during operation, performing operations such as digging, crushing, loading, or milling. Operating machinery in this application includes, but is not limited to, excavators, loaders, crushers, milling machines, drilling machines, and grab buckets. Working parts refer to easily worn components that are detachably installed on the working device of the operating machinery and directly contact the work material. Examples include the bucket teeth of an excavator, the shovel teeth of a loader, and the breaker hammer of a crusher. The actuator refers to the mechanical structure on the operating machinery that carries the working parts and drives their movement, such as the bucket of an excavator or a loader. In this embodiment, the bucket teeth of an excavator are used as an example. Visible light images are two-dimensional images reflecting the outline, position, edges, and other appearance information of the working parts, and can be acquired using a visible light camera. Infrared images are temperature distribution images reflecting the temperature information of the surface of the working parts, and can be acquired using an infrared thermal imager. It is understood that during normal operation, the working parts generate heat through friction with the material, resulting in a higher temperature; if a working part is missing, its installation location does not generate heat through friction, and its temperature is significantly lower than that of surrounding normal parts. Attitude data is data that characterizes the position and orientation of an actuator in space. For example, for an excavator bucket, attitude data includes boom angle, stick angle, and bucket angle, which can be collected by tilt sensors installed on each component.

[0027] Understandably, in complex working conditions such as mines and construction sites, a single sensor cannot simultaneously guarantee both detection accuracy and reliability. For example, the image quality of visible light cameras deteriorates significantly in dusty or low-light environments; while infrared thermal imagers are unaffected by light, thermal interference generated during operation can lead to misjudgments. Therefore, this application simultaneously acquires visible light and infrared images, leveraging the complementary advantages of different sensors to provide a data foundation for subsequent fusion detection.

[0028] Simultaneously, this application embodiment also acquires the attitude data of the actuator. It is understood that during excavator operation, the bucket's attitude (such as tilt angle and lifting height) changes in real time, and the position of the bucket teeth in the image changes accordingly. By acquiring the angles of the boom, stick, and bucket in real time using tilt sensors, the spatial pose of the bucket can be calculated, and the position of the bucket teeth in three-dimensional space can be mapped onto a two-dimensional image, achieving precise positioning of the target area of ​​the bucket teeth. The introduction of attitude data allows the system to quickly lock the position of the bucket teeth in the image even when the bucket's attitude changes, avoiding a complete image search and significantly improving detection efficiency.

[0029] In one example, a high-definition industrial camera and an infrared thermal imager can be deployed on top of the excavator cab, both with their viewing angles covering the bucket teeth area. The acquisition units are triggered synchronously to ensure that the visible light and infrared images acquired at the same time are time-aligned. The high-definition industrial camera uses a 1080P resolution and a 30fps frame rate, equipped with an electric wiper automatic cleaning device that cleans the lens every 15 minutes to prevent dust obstruction. The infrared thermal imager is deployed in conjunction with the visible light camera, with overlapping views covering the bucket teeth.

[0030] In another example, attitude data can also be acquired by angle encoders located at each articulation point, or by inertial measurement units located on the actuators, or by a combination of GPS / real-time dynamic positioning and inertial navigation systems.

[0031] In this way, by simultaneously acquiring visible light images, infrared images, and attitude data, the complementary characteristics of visible light and infrared are fully utilized. At the same time, attitude data is introduced to assist in the rapid localization of the target area, providing a data foundation for subsequent accurate detection and fusion recognition, and improving the system's adaptability and detection efficiency under complex working conditions.

[0032] Step S102: Based on the attitude data, determine the visible light target region from the visible light image and the infrared target region from the infrared image.

[0033] In this embodiment, the target region refers to a local region in an image that contains the working component. The visible light target region is a local region containing the working component determined from a visible light image, and the infrared target region is a local region containing the working component determined from an infrared image.

[0034] Understandably, during the operation of machinery, the posture of the actuator carrying the working parts changes in real time, and the position of the working parts in the image also changes dynamically. Performing a full-image search to locate the working parts would be computationally intensive and time-consuming, making it difficult to meet real-time detection requirements. Therefore, it is necessary to quickly locate the working parts in the image based on posture data and extract the local region containing the working parts as the target region to reduce background interference and improve detection efficiency.

[0035] In one example, a "pose-image region mapping table" can be used to determine the target region: The positions of the actuator's working parts in visible light and infrared images under multiple typical poses are pre-calibrated, and a mapping relationship between pose data and image region positions is established and stored in a lookup table. During actual detection, the mapping table is queried based on the current pose data, and the corresponding image region is directly read as the target region. If no exact match is found, interpolation is used to obtain the target region. This method eliminates the need for real-time kinematic calculations and perspective projection calculations, making it particularly suitable for edge computing scenarios where actuator pose changes are relatively regular and computational resources are limited.

[0036] Step S103: Determine the first recognition result based on the visible light target area.

[0037] In this embodiment, the first identification result refers to the result obtained independently based on the visible light target area, including the missing status of each part of the working component and the corresponding confidence level. The missing status can include missing and normal, etc. The confidence level is used to indicate the reliability of the identification result.

[0038] In one example, an image processing method can be used to determine the first recognition result based on the visible light target region. Specifically, edge detection processing is first performed on the visible light target region to extract the contour information of the working part; then, the extracted contour is matched with a pre-stored standard template of the working part to calculate the contour similarity; the missing state and the corresponding confidence level are determined based on the contour similarity. For example, when the similarity is lower than a preset threshold, it is judged as missing, and the larger the difference between the similarity and the preset threshold, the higher the confidence level; when the similarity is higher than the preset threshold, it is judged as normal, and the larger the difference between the similarity and the preset threshold, the higher the confidence level. This method has low computational resource consumption and is suitable for edge scenarios with limited computing power.

[0039] Step S104: Determine the second identification result based on the infrared target area.

[0040] In this embodiment, the second identification result refers to the result obtained independently based on the infrared target region, including the missing status of each part of the working component and the corresponding confidence level. Similar to the first identification result, the second identification result also includes the missing status and confidence level.

[0041] Understandably, normally functioning bucket teeth generate heat through friction with the material, causing a significant temperature increase. If a bucket tooth is missing, the original tooth mounting location (i.e., the tooth holder) generates no frictional heat, resulting in a significantly lower temperature than the surrounding normal bucket teeth. This temperature difference manifests as a clear temperature distribution anomaly in infrared images. Therefore, by analyzing the temperature distribution characteristics within the infrared target area, the abnormal temperature region caused by the missing component can be identified, thus determining the missing status of the working part.

[0042] In one example, a second recognition result can be determined based on the infrared target region using a temperature threshold segmentation method. Specifically, the infrared target region is first subjected to temperature calibration and median filtering for noise reduction to extract the temperature distribution information of each part of the working component. Then, the average temperature of each working component region is compared with a preset temperature threshold, and the missing status and corresponding confidence level are determined based on the temperature difference between that part and its adjacent normal parts. For example, when the average temperature of a part is lower than the preset temperature threshold and the temperature difference with its adjacent normal parts exceeds the preset temperature difference threshold, it is determined to be missing, and the larger the temperature difference, the higher the confidence level; when the temperature difference does not exceed the preset temperature difference threshold, it is determined to be normal, and the smaller the temperature difference, the higher the confidence level. This method has low computational complexity and high speed, making it suitable for edge computing scenarios with limited computing power.

[0043] Step S105: The first identification result and the second identification result are fused to obtain the target detection result for the missing working parts.

[0044] In this embodiment, fusion processing refers to the level of fusing multiple independent recognition results. That is, after each modality completes its recognition, these results are combined to output the final detection conclusion. Compared with feature-level fusion, fusion processing is more flexible and can fuse heterogeneous recognition results. The target detection result is the final output detection conclusion, including states such as normal, missing, or suspected missing working parts.

[0045] Understandably, the first identification result is based on visible light images, which have high reliability under good operating conditions but decrease in reliability under harsh conditions; the second identification result is based on infrared images, which are unaffected by illumination, but may be subject to false positives due to thermal interference. Both identification results have their advantages and limitations, and a single result cannot maintain high reliability under all operating conditions. Therefore, this application adopts an adaptive weighted fusion strategy to fuse the first and second identification results, achieving complementary advantages and effectively eliminating false alarms.

[0046] The above technical solution first acquires visible light and infrared images of the working component, as well as attitude data of the actuator carrying the working component. Then, based on the attitude data, the visible light target region is determined from the visible light image, and the infrared target region is determined from the infrared image. Simultaneously, a first recognition result is determined based on the visible light target region, followed by a second recognition result based on the infrared target region. Finally, the first and second recognition results are fused to obtain the target detection result for the missing working component. This application achieves real-time, accurate, and interference-resistant detection of missing working components under complex working conditions by simultaneously acquiring visible light, infrared, and attitude data, utilizing attitude data to assist in quickly and accurately locating the target region in both types of images, and independently identifying the target region based on visible light and infrared target detection models before fusion processing.

[0047] In this embodiment of the application, after acquiring the visible light image and infrared image of the working component, the method may further include: The visible light image is preprocessed, including denoising using a preset image filtering algorithm and enhancing image contrast using an enhancement algorithm. The infrared image is preprocessed, including removing temperature noise using a preset denoising algorithm, obtaining the ambient temperature of the current working environment of the machinery, and calibrating the infrared image based on the ambient temperature.

[0048] In this embodiment, the preset image filtering algorithm is a filtering method used to remove noise from visible light images, such as Gaussian filtering, median filtering, or bilateral filtering. The enhancement algorithm is a method used to improve image contrast, such as histogram equalization or adaptive histogram equalization. Temperature noise refers to interference signals present in the temperature data acquired by the infrared thermal imager, which may originate from ambient temperature fluctuations, sensor noise, or electromagnetic interference.

[0049] It is understandable that construction machinery typically operates in harsh environments such as mines and construction sites, resulting in varying degrees of noise and interference in the raw multimodal data collected. Visible light images are affected by dust and lighting conditions, resulting in blurred images and low contrast; infrared images are affected by ambient temperature and sensor noise, leading to inaccurate temperature measurements. If these noises and interferences are not preprocessed before subsequent processing, they will severely impact the accuracy of feature extraction and the reliability of recognition results. Therefore, it is necessary to perform targeted preprocessing on the raw image data.

[0050] Specifically, for visible light images, the controller can preprocess the original image using a Gaussian filtering algorithm to perform a convolution operation, removing Gaussian noise generated during the acquisition process and making the image smoother. Subsequently, a histogram equalization algorithm is used to stretch the pixel grayscale distribution of the image to the entire grayscale range, significantly improving image contrast and making the edges and contours of the teeth more clearly distinguishable. In one example, for a 1080P resolution image, a Gaussian kernel size of 3×3 resulted in an approximately 40% increase in image contrast after equalization.

[0051] For infrared images, on one hand, a median filtering algorithm is used to remove salt-and-pepper noise from the temperature data. This algorithm effectively suppresses isolated noise points by replacing the temperature value of each pixel with the median of the temperature values ​​of all pixels in its neighborhood. On the other hand, the controller acquires the ambient temperature of the current operating environment of the machinery, such as through an ambient temperature sensor installed on the top of the cab, and uses this ambient temperature to perform temperature calibration on the infrared image. Specifically, based on the difference between the ambient temperature and the calibration temperature, the temperature values ​​of all pixels in the infrared image are linearly corrected to eliminate the impact of ambient temperature fluctuations on measurement accuracy.

[0052] Thus, through the above preprocessing, noise in the visible light image is effectively suppressed and contrast is significantly improved, and the temperature measurement accuracy of the infrared image is calibrated, significantly improving the accuracy and robustness of the entire detection system.

[0053] In this embodiment of the application, determining a visible light target region from a visible light image and an infrared target region from an infrared image based on attitude data may include: The spatial pose information of the actuator is calculated based on the attitude data; Based on spatial pose information and imaging parameters of the acquisition device, the visible light target area of ​​the working component is determined in the visible light image, and the infrared target area of ​​the working component is determined in the infrared image.

[0054] In this embodiment, spatial pose information refers to the position and orientation of the actuator in three-dimensional space, including three-dimensional coordinate information and attitude angle information. The mapped region is the image area that the working part should theoretically occupy in the image, and is an intermediate calculation result.

[0055] Understandably, the actuator's posture changes in real time during operation, and the position of the working part in the image also changes accordingly. If a full-image search is performed to locate the working part, the computational load would be large and the time consumption would be long, making it difficult to meet the requirements of real-time detection. Therefore, it is necessary to quickly locate its position in the image based on the posture data. Since the actuator moves in three-dimensional space, the position of the working part in the two-dimensional image cannot be directly obtained from the posture data alone. It is necessary to first solve the posture data into the spatial pose of the actuator, and then combine it with the imaging parameters of the acquisition device to project the three-dimensional spatial position onto the two-dimensional image plane to determine the theoretical imaging area of ​​the working part in the image. Then, based on the mapped area, the target area of ​​the final input model is determined.

[0056] Specifically, the spatial pose information of the actuator is first calculated based on the attitude data. Taking the excavator bucket and its teeth as an example, the boom angle, stick angle, and bucket angle are collected by tilt sensors on the boom, stick, and bucket, respectively. Combined with the mechanical structural parameters of the excavator, such as the length of each component and the position of the hinge point, forward kinematics calculation is performed to obtain the position and orientation of the bucket in three-dimensional space, that is, the spatial pose information of the bucket.

[0057] Then, based on the spatial pose information and the imaging parameters of the acquisition device, the visible light target region of the working component is determined in the visible light image, and the infrared target region of the working component is determined in the infrared image. The imaging parameters include the intrinsic and extrinsic parameters of the acquisition device; the extrinsic parameters are used to describe the installation position and orientation of the acquisition device in the world coordinate system, realizing the transformation from world coordinates to acquisition device coordinates; the intrinsic parameters are used to describe the imaging characteristics of the acquisition device itself, realizing the projection from acquisition device coordinates to pixel coordinates.

[0058] In this embodiment of the application, determining the visible light target region of the working component in the visible light image and the infrared target region of the working component in the infrared image, based on spatial pose information and imaging parameters of the acquisition device, may include: The spatial pose information is transformed from the world coordinate system to the visible light acquisition device coordinate system and the infrared acquisition device coordinate system, respectively. Based on the camera intrinsic parameters of the visible light acquisition device, the coordinates in the visible light acquisition device coordinate system are projected to the corresponding pixel coordinate system to obtain the first pixel coordinate mapping of the working part in the visible light image; and based on the camera intrinsic parameters of the infrared acquisition device, the coordinates in the infrared acquisition device coordinate system are projected to the pixel coordinate system to obtain the second pixel coordinate mapping of the working part in the infrared image. Based on the geometric dimensions of the working component and the first and second pixel coordinate mappings, the coverage area of ​​the working component in the visible light image and the infrared image are determined respectively, thus obtaining the visible light target area and the infrared target area.

[0059] In this embodiment, the world coordinate system is a reference coordinate system describing the absolute position of the actuator in three-dimensional space, such as a world coordinate system established with the excavator's swing center as the origin. The acquisition device coordinate system is a coordinate system established with the optical center of the acquisition device as the origin, and its Z-axis is along the optical axis. The pixel coordinate system is a coordinate system on a two-dimensional image, with the upper left corner of the image as the origin, and pixels as the unit, used to describe the specific position of an object in the image.

[0060] Understandably, spatial pose information is described based on the world coordinate system, while the position of the working part in the image needs to be described based on the pixel coordinate system. Therefore, it is necessary to first transform the spatial pose information from the world coordinate system to the acquisition device coordinate system using extrinsic parameters, and then project the coordinates in the acquisition device coordinate system to the pixel coordinate system using intrinsic parameters, finally obtaining the pixel coordinate mapping of the working part in the image, that is, the pixel position of each key point on the working part (such as the center point of the tooth tip, the end point of the tooth seat, etc.) in the image.

[0061] Specifically, the spatial pose information is first transformed from the world coordinate system to the visible light acquisition device coordinate system and the infrared acquisition device coordinate system, respectively. Since the visible light acquisition device and the infrared acquisition device are installed at different locations in the world coordinate system, their extrinsic parameters are different, thus requiring separate transformations. Using the extrinsic parameters of the visible light acquisition device, the spatial pose information of the actuator is transformed from the world coordinate system to the visible light acquisition device coordinate system, obtaining the three-dimensional coordinates of the actuator in the visible light acquisition device coordinate system. Simultaneously, using the extrinsic parameters of the infrared acquisition device, the same spatial pose information is transformed from the world coordinate system to the infrared acquisition device coordinate system, obtaining the three-dimensional coordinates of the actuator in the infrared acquisition device coordinate system.

[0062] Then, based on the camera intrinsic parameters of the visible light acquisition device, the coordinates in the visible light acquisition device coordinate system are projected to the corresponding pixel coordinate system to obtain the first pixel coordinate mapping of the working part in the visible light image; and based on the camera intrinsic parameters of the infrared acquisition device, the coordinates in the infrared acquisition device coordinate system are projected to the pixel coordinate system to obtain the second pixel coordinate mapping of the working part in the infrared image. The camera intrinsic parameters include focal length, pixel size, and optical center offset, used to describe how a point in three-dimensional space is mapped onto the two-dimensional image plane. In one example, for the three-dimensional coordinates of the center point of the tooth tip in the visible light acquisition device coordinate system, according to the pinhole imaging principle, the pixel position of the point in the visible light image can be calculated by dividing the X-axis and Y-axis coordinates of the point by the Z-axis coordinate (i.e., depth information), multiplying by the focal length, and superimposing the optical center offset. The infrared acquisition device uses the same projection principle, but uses its own camera intrinsic parameters for independent calculation to obtain the pixel coordinates of the tooth tip in the infrared image. In this way, the pixel coordinates of each key point on the working part (such as the center point of the tooth tip, the end point of the tooth seat, etc.) in the visible light image and the infrared image can be calculated, that is, the first pixel coordinate mapping and the second pixel coordinate mapping are obtained.

[0063] Finally, based on the geometric dimensions of the working component and the first and second pixel coordinate mappings, the coverage area of ​​the working component in the visible light and infrared images is determined, respectively, resulting in the visible light target area and the infrared target area. Geometric dimensions refer to the physical dimensions of the working component in real three-dimensional space, such as the length and width of the teeth. Since the pixel coordinates of a key point on the working component (such as the center point of the tooth tip) are known, combined with the actual physical dimensions of the working component and the current depth information, the pixel width and pixel height occupied by the working component in the image can be calculated, thereby determining its lateral and vertical coverage areas, i.e., obtaining a rectangular area as the mapping area. For example, in the visible light image, given the pixel coordinates of the center point of the tooth tip, based on the length and width of the tooth and the current depth, the coverage width and coverage length of the tooth in the image can be calculated. Extending outwards from the center point of the tooth tip as a reference point, a rectangular area is determined, which is the visible light target area. The infrared image uses the same calculation method, but uses the depth information corresponding to the infrared acquisition device to obtain the infrared target area.

[0064] It should be noted that for different types of operating machinery and working parts, geometric parameters can be obtained in advance through measurement and stored in the system. For example, for excavator bucket teeth, their length and width can be measured and stored in advance; similarly, for loader shovel teeth, their length and width can be measured. This application does not limit the specific type and size of the working parts, as long as their geometric parameters can be obtained in advance.

[0065] In this way, by transforming the spatial pose information from the world coordinate system to the acquisition device coordinate system and the pixel coordinate system in sequence, and combining it with the geometric dimension parameters of the working parts, the accurate calculation of the mapping area from the three-dimensional spatial pose information to the two-dimensional image is realized, providing an accurate positional basis for the subsequent determination of the target area and the detection of missing parts.

[0066] In this embodiment of the application, before determining the visible light target region from the visible light image and the infrared target region from the infrared image based on attitude data, the method may further include: Based on the attitude data, determine whether the current attitude of the actuator meets the preset extraction conditions; If the preset extraction conditions are met, the steps of determining the visible light target region from the visible light image and the infrared target region from the infrared image based on the pose data are performed. The preset extraction conditions are that the working part is located within the field of view of the acquisition device, and the angle between the base plate of the actuator and the vertical direction is less than the preset angle threshold.

[0067] In this embodiment, the preset extraction condition refers to a pre-set criterion used to determine whether the current posture of the actuator is suitable for target area extraction. The field of view range refers to the range of angles within which the acquisition device can clearly image the target area. The actuator's base plate refers to the bottom plane of the actuator that supports the working parts, such as the base plate of an excavator bucket. The preset angle threshold refers to a pre-set angle critical value used to determine whether the actuator's base plate is excessively tilted.

[0068] Understandably, during actual operation of machinery, the attitude of the actuator changes in real time, and not all attitudes are suitable for target area extraction and missing area detection. For example, when the bucket tilt angle is too large, the bucket teeth may be obscured by the bucket floor plate and cannot be captured by the acquisition device; when the bucket is raised too high or too low, the bucket teeth may be outside the field of view of the acquisition device, and similarly, the bucket tooth image cannot be effectively acquired. Under these attitudes, even if target area extraction and detection are performed, effective results cannot be obtained, and computational resources will be wasted. Therefore, before performing target area extraction, it is necessary to first determine whether the current attitude of the actuator meets the preset extraction conditions based on the attitude data, and only proceed with subsequent steps if the conditions are met.

[0069] Specifically, the attitude data is used to determine whether the current attitude of the actuator meets the preset extraction conditions. Taking an excavator bucket and its teeth as an example, the attitude data includes the boom angle, stick angle, and bucket angle collected by a tilt sensor. Based on this angle data, the current attitude of the bucket can be determined. The preset extraction conditions include two sub-conditions: First, the working parts are within the field of view of the acquisition equipment. For example, based on spatial pose information, it can be determined whether the teeth are within the vertical field of view of a visible light camera and an infrared thermal imager; if the teeth are outside the camera's field of view, a valid image of the teeth cannot be acquired. Second, the angle between the actuator's bottom plate and the vertical direction is less than a preset angle threshold. For excavator buckets, when the bottom plate is too tilted (e.g., the tilt angle is too large), the teeth may be obscured or overlapped with the bottom plate, making it impossible to clearly distinguish the outline and position of the teeth. The preset angle threshold can be set according to the structural characteristics of the actuator, for example, it can be set to 45°. When the angle between the bucket bottom plate and the vertical direction is less than 45°, the bucket teeth can be clearly observed; when the angle is greater than or equal to 45°, the bucket teeth may be obscured or the observation effect is poor, making it unsuitable for target area extraction.

[0070] If the preset extraction conditions are met, the steps of determining the visible light target region from the visible light image and the infrared target region from the infrared image based on the pose data are executed. That is, the subsequent target region determination and target detection are only performed when both of the aforementioned sub-conditions are met simultaneously; if either sub-condition is not met, the current detection cycle is skipped, and the process waits for the next detection cycle to reacquire image and pose data and make a judgment again.

[0071] It should be noted that the preset angle threshold can be set according to the specific structure and working characteristics of the actuator. The threshold may vary for different types of machinery and actuators. For example, 45° is a preferred threshold proven in practice for excavator buckets; for loader buckets, due to different bucket structures, other angle values, such as 40° or 50°, can be set. This application does not impose any limitations on this, as long as a suitable angle threshold can be determined based on the actual equipment structure and working conditions.

[0072] In another example, when the actuator's current posture does not meet the preset extraction conditions, a prompt message can be generated and output to the operator, prompting the operator to adjust the actuator's posture or wait for the actuator to enter a detectable posture. For example, a prompt message "Bucket posture abnormal, please adjust bucket angle" can be displayed on the human-machine interface in the cab, or a voice prompt can be issued when the operation is not standardized. In this way, it can assist operators in standardizing operations and improve the effective detection rate of the detection system.

[0073] Thus, by setting attitude prediction conditions before the target area is determined, subsequent processing is only performed when the actuator is in a suitable attitude for observation. On the one hand, this avoids unnecessary calculations under invalid attitudes, saves edge computing resources, and reduces power consumption. On the other hand, it reduces false detections and missed detections caused by the working parts being obstructed or out of the field of view, thereby improving the effectiveness and reliability of the detection system.

[0074] In this embodiment of the application, determining the first identification result based on the visible light target region includes: Input the visible light target region into the pre-built visible light target detection model; The visual features of the working parts in the visible light target area are extracted layer by layer by the visible light target detection model, and the missing state of each part of the working parts is identified according to the visual features to obtain the missing state of each part of the working parts and the corresponding confidence level. The missing status of each part of the working component and the corresponding confidence level are used as the first identification result.

[0075] In this embodiment, the visible light target detection model refers to a pre-trained deep learning model used to identify missing states of working parts in visible light images. The visible light target detection model can employ YOLOv8, Faster R-CNN, SSD, etc. The model is pre-trained using a large dataset of visible light images labeled with missing states of working parts, learning the visual features of working parts in visible light images, and can output accurate recognition results based on the input visible light target region.

[0076] Specifically, the visible light target region is first input into a pre-built visible light target detection model. The input layer of this model receives the visible light target region as input data, which is a local visible light image containing the working parts. In one example, taking excavator bucket teeth as an example, the visible light target region is a local image containing multiple bucket teeth.

[0077] Then, the visual features of the working parts in the visible light target region are extracted layer by layer using a visible light target detection model. Based on the visual features, the missing state of each part of the working part is identified, resulting in the missing state of each part and its corresponding confidence score. Specifically, the visible light target detection model can adopt a convolutional neural network structure. The input visible light target region passes through multiple network layers of the model, extracting visual features layer by layer. Shallow convolutional layers extract low-level features such as edges and corners, while deep convolutional layers extract high-level semantic features such as the shape of the teeth, contour integrity, and positional relationships. Finally, the category prediction result and its corresponding confidence score for each working part are output. Finally, the missing state of each part of the working part and its corresponding confidence score are used as the first recognition result.

[0078] Taking excavator bucket teeth as an example, a visible light target area is input into a pre-built visible light target detection model. After identifying each bucket tooth within that area, the model outputs the missing status of each tooth and its corresponding confidence level. For example, the model output "Teeth No. 4 is missing, confidence level 0.92" means that the probability of Teeth No. 4 being missing is 92%; "All teeth are normal, confidence level 0.95" means that the probability of all teeth being present is 95%. If the confidence level of a certain bucket tooth is lower than a preset threshold (e.g., 0.5), it can be marked as "suspected missing" to trigger further verification.

[0079] In this way, by reasoning about the visible light target region through a pre-built visible light target detection model, and making full use of the rich appearance contour information in the visible light image, the rapid identification of missing working parts is achieved.

[0080] In this embodiment of the application, determining the second identification result based on the infrared target region includes: Input the infrared target region into the pre-built infrared target detection model; The temperature distribution features of the working parts in the infrared target area are extracted layer by layer by the infrared target detection model, and the missing state of each part of the working parts is identified according to the temperature distribution features to obtain the missing state of each part of the working parts and the corresponding confidence level. The missing status of each part of the working component and the corresponding confidence level are used as the second identification result.

[0081] In this embodiment, the infrared target detection model refers to a pre-trained deep learning model used to identify missing states of working parts in infrared images. The infrared target detection model can also employ the YOLOv8 target detection network and be pre-trained using a large dataset of infrared images labeled with missing teeth states, enabling the model to learn the temperature distribution characteristics of the missing teeth region in the infrared image.

[0082] Specifically, the infrared target region is first input into a pre-built infrared target detection model. The input layer of this model receives the infrared target region as input data, which is a local infrared image containing the working parts, where the grayscale value or color value of each pixel represents the temperature information of that point. In one example, taking excavator bucket teeth as an example, the infrared target region is an infrared image containing multiple bucket teeth and their tooth holders. In the image, the area of ​​normal bucket teeth appears at high temperature (bright color), while the area of ​​the tooth holder corresponding to missing bucket teeth appears at low temperature (dark color).

[0083] Then, the infrared target detection model extracts the temperature distribution features of the working parts in the infrared target region layer by layer, and identifies the missing state of each part of the working part based on the temperature distribution features, obtaining the missing state and corresponding confidence score of each part of the working part. Specifically, the infrared target detection model can adopt a convolutional neural network structure. The input infrared target region passes through multiple network layers of the model in sequence, extracting temperature distribution features layer by layer. Shallow convolutional layers extract low-level features such as local temperature changes and edge temperature differences, while deep convolutional layers extract high-level semantic features such as the overall temperature pattern of the working part region and temperature anomaly areas. By learning the mapping relationship between temperature distribution features and missing states in a large number of infrared image samples, the model can identify "low temperature areas" or "temperature anomaly areas" caused by missing features, and output the category prediction result and corresponding confidence score of each working part. Finally, the missing state and corresponding confidence score of each part of the working part are used as the second identification result.

[0084] Taking excavator bucket teeth as an example, after inputting the infrared target area into a pre-built infrared target detection model, the model identifies the missing state based on the temperature distribution of the bucket tooth area. For example, the model outputs "Bucket tooth #4 is missing, confidence level 0.88," indicating that the probability of judging that bucket tooth #4 is missing based on temperature difference is 88%. Since infrared thermal imagers are not affected by lighting conditions, this identification result can still maintain high reliability under working conditions such as nighttime, low light, and strong dust.

[0085] In this way, by reasoning about the infrared target area through the infrared target detection model, and making full use of the temperature distribution characteristics in the infrared image for missing detection, a reliable second identification result is provided under conditions where visible light fails, such as at night, in low light, and in strong dust, which complements the first identification result.

[0086] In this embodiment of the application, fusing the first identification result and the second identification result to obtain the target detection result for the missing working part may include: Obtain the current operating status information of the machinery; The fusion weights of the first and second identification results are determined based on the work condition information. The first and second identification results are weighted and fused according to the fusion weights to obtain the comprehensive confidence score. The target detection results are output based on the overall confidence level.

[0087] In this embodiment, the operating condition information refers to the current working environment conditions of the machinery, including lighting conditions and dust concentration. The reliability of each sensor varies under different operating conditions. The fusion weight is a coefficient used to measure the importance of the first and second identification results in the final decision. The overall confidence score is the weighted fusion result, used to characterize the overall reliability of the judgment regarding the missing working part.

[0088] It is understandable that the fusion method of the first and second identification results directly affects the accuracy and reliability of the target detection results. If a fixed weight is used for fusion, it cannot adapt to the dynamic changes in working conditions. In harsh environments such as nighttime or high dust levels, the quality of visible light images deteriorates, leading to a decrease in the reliability of the first identification result. If the weights are not adjusted accordingly, it can easily cause false detections or missed detections. Therefore, the embodiments of this application dynamically determine the fusion weights based on the current working condition information. Under conditions of sufficient light and low dust, the reliability of visible light detection is high, and the visible light mode obtains a higher fusion weight. Under conditions of nighttime or high dust levels, infrared detection is not affected by light and dust, and the infrared mode obtains a higher fusion weight. This allows the fusion strategy to adapt to environmental changes and effectively suppress the negative impact of unreliable modes on the detection results.

[0089] Specifically, the first step is to acquire the current operating condition information of the machinery. The controller uses ambient light sensors and dust concentration sensors installed on the machinery to collect real-time data on the current operating environment's light conditions and dust concentration. In one example, light conditions can be categorized into three levels: daytime, dusk, and nighttime, while dust concentration can be categorized into three levels: normal, medium, and high dust levels.

[0090] Secondly, the fusion weights of the first and second recognition results are determined based on the operating condition information. The controller has a built-in weight mapping table that maps different combinations of operating conditions to corresponding weight ratios. In a specific embodiment, the weight mapping rules are as follows: under daytime conditions with normal dust levels, the visible light image quality is good, so the weight of the first recognition result is set to 0.4, and the weight of the second recognition result is set to 0.6; under nighttime or high-dust conditions, the visible light image quality is severely degraded, so the weight of the first recognition result is reduced to 0.25, and the weight of the second recognition result is increased to 0.75. The weight values ​​can be calibrated and optimized through field experiments according to actual needs.

[0091] Then, the first and second identification results are weighted and fused according to the determined fusion weights to obtain the overall confidence level. In one example, for each part of each working component, the overall confidence level = confidence level of the first identification result × weight of the first identification result + confidence level of the second identification result × weight of the second identification result. For example, if the confidence level of the first identification result of a certain bucket tooth is 0.85 and the confidence level of the second identification result is 0.92, and under the current working condition, the weight of the first identification result is 0.4 and the weight of the second identification result is 0.6, then the overall confidence level = 0.85 × 0.4 + 0.92 × 0.6 = 0.892.

[0092] In another example, the overall confidence level satisfies the following formula: ; ; ; in, It is the confidence level of the first identification result. It is the confidence level of the second identification result. It is the confidence weight of the confidence score of the first identification result. It is the confidence weight of the second identification result. It is the overall confidence level.

[0093] Finally, the target detection result is output based on the overall confidence level. In one example, a single preset threshold can be set and historical state tracking can be introduced. Only when the overall confidence level reaches or exceeds the threshold multiple times consecutively is the working part considered missing; otherwise, the counter is reset to zero to maintain a normal state. This method avoids false alarms caused by single fluctuations in confidence level through continuous confirmation.

[0094] In this way, through fusion processing, the system dynamically adjusts the fusion weight according to the working conditions, automatically increases the weight of infrared recognition results under harsh conditions, gives full play to the advantages of visible light and infrared light respectively, achieves the complementary advantages of the two modes, effectively suppresses false alarms and false negatives of single mode under harsh conditions, and improves the reliability and robustness of detection.

[0095] In this embodiment of the application, outputting the target detection result based on the comprehensive confidence level may include: If the overall confidence level is greater than or equal to the first preset threshold, the target detection result is determined to be a missing working part; If the overall confidence level is less than the first preset threshold and greater than or equal to the second preset threshold, the target detection result is determined to be a suspected missing working part; If the overall confidence level is less than the second preset threshold, the target detection result is determined to be that the working component is normal. The first preset threshold is greater than the second preset threshold.

[0096] In this embodiment, the first preset threshold is a high-confidence threshold for determining whether a working component is missing. The second preset threshold is a low-confidence threshold for determining whether a working component is functioning properly. Both the first and second preset thresholds can be dynamically set based on operating conditions, user configuration, or historical data statistics.

[0097] Understandably, the rationality of the output judgment directly determines the accuracy and reliability of the target detection results. If only a single threshold is set for binary classification, when the overall confidence level is in the critical region, even slight fluctuations in confidence level may cause the output result to repeatedly jump between "missing" and "normal," resulting in misjudgment. To address this, this application embodiment divides the output result into a missing region, a suspected region, and a normal region by setting two thresholds. When the overall confidence level falls into the suspected region, new visible light and infrared images are acquired for re-detection, thereby effectively handling boundary ambiguity, avoiding misjudgments caused by confidence level fluctuations, and improving the reliability of the detection system.

[0098] Specifically, after calculating the overall confidence level, the controller compares it with the first preset threshold and the second preset threshold, and executes the corresponding output logic based on the comparison result.

[0099] In one example, when the overall confidence level is greater than or equal to a first preset threshold, the controller determines that the target detection result is a missing working part. Taking a first preset threshold of 0.8 as an example, if the overall confidence level of a certain bucket tooth is 0.92, the system determines that the bucket tooth is missing. At this time, the controller immediately triggers an early warning, pushing the location number of the missing bucket tooth, the time of the missing part, and the current working condition information to the human-machine interface in the cab and the remote monitoring center, notifying maintenance personnel to handle the situation promptly.

[0100] In another example, when the overall confidence level is less than a first preset threshold but greater than or equal to a second preset threshold, the controller determines the target detection result as a suspected missing working part. Taking a first preset threshold of 0.8 and a second preset threshold of 0.5 as an example, if the overall confidence level of a bucket tooth is 0.67, the system determines that the bucket tooth is suspected of being missing. In this case, the controller does not directly issue a missing alarm. Instead, the system re-acquires a new set of visible light and infrared images of the working part, re-executes the target area extraction, model inference, and fusion processing steps, and recalculates the overall confidence level based on the new data. If two consecutive detection results are both suspected missing or reach the missing threshold, a missing alarm is output; if subsequent detection results return to normal, the suspected status is cleared, and normal monitoring resumes.

[0101] In another example, when the overall confidence level is less than the second preset threshold, the controller determines that the target detection result indicates that the working part is normal. Taking a second preset threshold of 0.5 as an example, if the overall confidence level of a certain bucket tooth is 0.32, the system determines that the bucket tooth is in a normal state, does not perform any warning operation, and continues to monitor in real time.

[0102] Thus, through the dual-threshold three-classification mechanism, the system can effectively handle situations with ambiguous confidence boundaries and avoid misjudgments caused by single fluctuations. When a case falls into a suspected area, resampling and verification are triggered, which not only prevents false alarms from causing unnecessary downtime for inspections, but also avoids missed alarms from causing security risks, significantly improving the reliability and practicality of the detection system.

[0103] In one specific embodiment of this application, taking the detection of missing bucket teeth in excavators as an example, this specific embodiment of the application provides a system for detecting missing bucket teeth in excavators. This system includes a perception layer, an edge computing layer, a cloud processing layer, and an early warning layer. These layers work collaboratively to form a complete detection closed loop of "collection-preprocessing-fusion-identification-verification-early warning," with the specific structure as follows: 1. Perception layer.

[0104] The sensing layer includes a visible light acquisition unit, an infrared thermal imaging acquisition unit, and an auxiliary acquisition unit, providing full coverage of all bucket teeth of the excavator and ensuring no blind spots in complex working conditions.

[0105] The visible light acquisition unit is fundamental to missing tooth detection. Its core component is a high-definition industrial camera with 1080P resolution, 30fps frame rate, wide dynamic range, and IP67 waterproof and dustproof rating. Deployed on the top of the excavator cab, with the lens facing the bucket teeth, it is equipped with an electric wiper for automatic cleaning every 15 minutes to prevent dust obstruction. The visible light acquisition unit captures images of the bucket teeth's outline and position, identifies any gaps in the teeth, and captures the outline of the tooth base corresponding to the missing tooth, serving as the basic basis for missing tooth detection. It is suitable for operating conditions with ample light and low dust levels.

[0106] The infrared thermal imaging acquisition unit serves as an auxiliary tool for missing tooth detection and boasts strong anti-interference capabilities. The core device of this unit is an infrared thermal imager, deployed on the roof of the cab. Working in conjunction with a visible light camera, the overlapping viewpoint covers the bucket teeth, capturing temperature distribution images of the teeth and tooth holders. When a tooth is missing, the temperature at the original tooth location (tooth holder), unaffected by operational friction, creates a significant difference compared to the temperature of surrounding normal teeth. This temperature anomaly helps determine if a tooth is missing, making it suitable for nighttime, low-light, and high-dust operating conditions (compensating for the limitations of visible light nighttime detection).

[0107] The auxiliary acquisition unit includes boom, stick, and bucket tilt sensors, which are used to acquire bucket attitude data (boom angle, stick angle, and bucket angle) to help achieve spatiotemporal alignment of visible light images and infrared images.

[0108] 2. Edge computing layer: Local real-time identification, quickly triggering early warnings.

[0109] The core device of the edge computing layer is an industrial-grade edge computing box (with a built-in GPU chip that supports lightweight AI models). It is installed in the cab electrical control cabinet, electrically connected to all sensors in the perception layer, supports 5G / Wi-Fi communication, and has a local data caching function (caching 7 days of data, and can work offline when the network is interrupted).

[0110] The edge computing layer is used for data preprocessing, ROI extraction, real-time AI inference, and rapid identification of missing teeth to trigger local warnings.

[0111] The data preprocessing includes: using a Gaussian filtering and histogram equalization algorithm to denoise and enhance contrast for visible light images; using a median filtering algorithm to remove temperature noise for infrared images, and performing temperature calibration in conjunction with ambient temperature.

[0112] Extracting the ROI of the bucket teeth: Based on the tilt sensor data from the auxiliary acquisition unit, the ROI of the bucket teeth in visible light images and infrared thermal imaging images can be obtained through "attitude calculation + perspective projection". Taking the visible light image as an example, the pose of the bucket is calculated by combining the angles of the boom, stick, and bucket with mechanical structure parameters. The ROI of the bucket teeth is extracted only when the bucket is within the vertical field of view of the visible light camera and the angle between the bucket bottom plate and the vertical direction is less than 45°. Figure 2 As shown, Figure 2 This diagram illustrates the principle of bucket posture during ROI (Region of Interest) extraction, as provided in a specific embodiment of this application. The end-effector center point can be obtained based on camera imaging principles. The coordinates in the camera coordinate system are Its coordinates in the pixel coordinate system are ,like Figure 3 As shown, Figure 3 This is an example diagram of the ROI of a bucket tooth provided for a specific embodiment of this application. The camera focal length is the offset of the origin (optical center) of the image coordinate system relative to the pixel coordinate system. , express and The physical length of each pixel in the direction on the camera's sensor.

[0113] ; Similarly, the pixel width of the bucket is calculated based on the camera imaging principle. Pixel length of the beak Calculate the coordinates of the top-left vertex:

[0114] Extracting the ROI of the bucket teeth using array slicing: ; If the ROI is outside the image boundary, it means that the bucket teeth are not completely within the field of view (for example, if the bucket is too close to the camera, the outer bucket teeth may not be within the field of view). In this case, detection will not be performed temporarily. Detection will only be performed when all bucket teeth are within the field of view.

[0115] Fusion Recognition: The ROI of the bucket teeth has had most of its background removed. It is then input into the YOLOv8 model for recognition, outputting the recognition result (e.g., "Bucket tooth #4 is missing") and confidence score (e.g., 0.92), with a response time ≤100ms. Visible light and infrared thermal imaging images are each used to train a target detection model. Through a confidence-weighted fusion strategy, the final bucket tooth missing detection result is output (a total score ≥0.8 indicates a missing tooth, a total score <0.5 indicates normal, and 0.5 ≤ total score <0.8 indicates a suspected missing tooth).

[0116] Adaptive weighted fusion strategy: ; ; ; in, It is the confidence level for detecting missing teeth in visible light images. It is the confidence level of infrared thermal imaging in detecting missing teeth. It is the confidence weight for detecting missing teeth in visible light images. It is the confidence weight for detecting missing bucket teeth using infrared thermal imaging. It is the confidence level of the final missing teeth.

[0117] An adaptive weighted fusion strategy can amplify the influence of high-confidence detectors while suppressing the influence of low-confidence detectors. For example, when there is a lot of dust, the visible light image may not clearly show the bucket teeth. The value will be very low, and the system will automatically allocate more weight to infrared thermal imaging.

[0118] Local warning trigger: If the problem is initially identified as missing bucket teeth, a local audible and visual alarm will be triggered immediately, alerting the operator via a pop-up window in the driver's cab HMI, while simultaneously caching the raw data from the visible light camera and the infrared thermal imaging camera.

[0119] 3. Cloud processing layer: deep verification, exception archiving and management.

[0120] The cloud processing layer uses cloud servers deployed in the mine monitoring center, which communicate with the edge computing units. Its core functions include: Data reception: Receives the missing tooth identification results, raw data from the visible light camera and the infrared thermal imaging camera uploaded by the edge computing unit.

[0121] Data archiving and management: Archive the detection results and related data of missing bucket teeth, generate anomaly reports, and realize centralized management of multiple excavators.

[0122] 4. Early warning layer.

[0123] The early warning layer includes a local early warning module and a remote early warning module. The local early warning module is deployed in the excavator cab and uses audible and visual alarms and HMI pop-up reminders. The remote early warning module pushes bucket tooth missing early warning information to maintenance and management personnel through a mobile APP and the mine monitoring center display screen, while also marking the missing location, missing time, and operating conditions to facilitate timely maintenance and handling.

[0124] In summary, by using adaptive weighted fusion of visible light and infrared light, precise target area extraction with attitude assistance, and a dual-threshold three-level judgment mechanism, real-time, accurate, and interference-resistant detection and closed-loop handling of missing excavator bucket teeth under complex working conditions such as mining have been achieved.

[0125] Figure 4 This is a structural block diagram of a device for detecting missing working parts, provided in an embodiment of this application. Figure 4 As shown in the illustration, this application also provides a device 400 for detecting missing working parts, applied to operating machinery. The device 400 may include: The data acquisition module 410 is used to acquire visible light images and infrared images of the working component, as well as the attitude data of the actuator that carries the working component; The first determining module 420 is used to determine a visible light target region from a visible light image and an infrared target region from an infrared image based on attitude data. The second determining module 430 is used to determine the first identification result based on the visible light target area; The third determining module 440 is used to determine the second identification result based on the infrared target area; The fusion processing module 450 is used to fuse the first recognition result and the second recognition result to obtain the target detection result for the missing working part.

[0126] The apparatus for detecting missing working parts provided in this embodiment of the invention can implement all the processes of the method for detecting missing working parts in the method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0127] This application embodiment also provides a working machine, including: Working parts; The apparatus for detecting missing working parts in the above embodiments.

[0128] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the method for detecting missing working components described in the above embodiments.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0137] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting missing working parts, characterized in that, Applied to operating machinery, the method includes: Acquire visible light and infrared images of the working component, as well as attitude data of the actuator carrying the working component; Based on the attitude data, a visible light target region is determined from the visible light image, and an infrared target region is determined from the infrared image; A first identification result is determined based on the visible light target region; A second identification result is determined based on the infrared target region; The first identification result and the second identification result are fused together to obtain the target detection result of the missing working component.

2. The method according to claim 1, characterized in that, Determining the first identification result based on the visible light target region includes: The visible light target region is input into a pre-constructed visible light target detection model; The visual features of the working component in the visible light target region are extracted layer by layer by the visible light target detection model, and the missing state of each part of the working component is identified according to the visual features to obtain the missing state of each part of the working component and the corresponding confidence level. The missing state of each part of the working component and the corresponding confidence level are used as the first identification result.

3. The method according to claim 1, characterized in that, The step of determining the second identification result based on the infrared target region includes: The infrared target region is input into a pre-constructed infrared target detection model; The infrared target detection model extracts the temperature distribution features of the working component in the infrared target area layer by layer, and identifies the missing state of each part of the working component based on the temperature distribution features, so as to obtain the missing state and corresponding confidence level of each part of the working component. The missing status of each part of the working component and the corresponding confidence level are used as the second identification result.

4. The method according to claim 1, characterized in that, The step of determining a visible light target region from the visible light image and an infrared target region from the infrared image based on the attitude data includes: The spatial pose information of the actuator is calculated based on the attitude data. Based on the spatial pose information and the imaging parameters of the acquisition device, the visible light target region of the working component is determined in the visible light image, and the infrared target region of the working component is determined in the infrared image.

5. The method according to claim 4, characterized in that, The step of determining the visible light target region of the working component in the visible light image and the infrared target region of the working component in the infrared image based on the spatial pose information and the imaging parameters of the acquisition device includes: The spatial pose information is transformed from the world coordinate system to the visible light acquisition device coordinate system and the infrared acquisition device coordinate system, respectively. Based on the camera intrinsic parameters of the visible light acquisition device, the coordinates in the visible light acquisition device coordinate system are projected to the corresponding pixel coordinate system to obtain the first pixel coordinate mapping of the working component in the visible light image; and based on the camera intrinsic parameters of the infrared acquisition device, the coordinates in the infrared acquisition device coordinate system are projected to the pixel coordinate system to obtain the second pixel coordinate mapping of the working component in the infrared image. Based on the geometric dimensions of the working component and the first and second pixel coordinate mappings, the coverage area of ​​the working component in the visible light image and the infrared image is determined respectively, thereby obtaining the visible light target area and the infrared target area.

6. The method according to claim 1, characterized in that, Before determining the visible light target region from the visible light image and the infrared target region from the infrared image based on the attitude data, the method further includes: Based on the attitude data, determine whether the current attitude of the actuator meets the preset extraction conditions; If the preset extraction conditions are met, the steps of determining the visible light target region from the visible light image and determining the infrared target region from the infrared image based on the attitude data are performed. The preset extraction conditions are that the working component is located within the field of view of the acquisition device, and the angle between the base plate of the actuator and the vertical direction is less than a preset angle threshold.

7. The method according to claim 1, characterized in that, The step of fusing the first identification result and the second identification result to obtain the target detection result for the missing working component includes: Obtain the current operating status information of the machinery; The fusion weight between the first identification result and the second identification result is determined based on the work condition information. The first identification result and the second identification result are weighted and fused according to the fusion weight to obtain the comprehensive confidence level; The target detection result is output based on the overall confidence level.

8. The method according to claim 7, characterized in that, The step of outputting the target detection result based on the comprehensive confidence score includes: If the overall confidence level is greater than or equal to the first preset threshold, the target detection result is determined to be a missing working component; If the overall confidence level is less than the first preset threshold and greater than or equal to the second preset threshold, the target detection result is determined to be a suspected missing working component; If the overall confidence level is less than the second preset threshold, the target detection result is determined to be that the working component is normal; Wherein, the first preset threshold is greater than the second preset threshold.

9. A device for detecting missing working parts, characterized in that, Applied to operating machinery, including: The data acquisition module is used to acquire visible light images and infrared images of the working component, as well as the attitude data of the actuator that carries the working component; The first determining module is used to determine a visible light target region from the visible light image and an infrared target region from the infrared image based on the attitude data. The second determining module is used to determine a first identification result based on the visible light target region; The third determining module is used to determine the second identification result based on the infrared target region; The fusion processing module is used to fuse the first identification result and the second identification result to obtain the target detection result of the missing working component.

10. A type of operating machinery, characterized in that, include: Working parts; The apparatus for detecting missing working parts according to claim 9.

11. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the method for detecting the absence of a working component according to any one of claims 1 to 8.

Images