Bucket wheel machine health monitoring system and method based on thermal imaging and visible light images

Abstract

The application provides a kind of bucket wheel machine health monitoring system and method based on thermal imaging and visible light image, which first realizes the pixel-level accurate alignment of thermal imaging and visible light image under dynamic field of view through hard trigger timing synchronization and cross-modal edge registration technology based on rigid structure.On this basis, the background interference is stripped using a semantic segmentation network, and the region of interest of the key component is extracted.Further, by establishing the mapping relationship between visible light texture features and physical emissivity and reflected light, a physical perception inversion model is constructed to dynamically correct the radiation temperature deviation caused by dust accumulation and light, and restore the real temperature of the equipment surface.Finally, the multi-dimensional joint diagnosis is carried out by fusing the real thermodynamic characteristics and surface visual defect features, so as to exclude the environmental thermal noise interference, realize the high confidence determination and visual accurate positioning of equipment failure.

Description

Technical Field

[0001] This invention relates to the field of industrial equipment condition monitoring and fault diagnosis technology, specifically to a health monitoring system and method for bucket wheel excavators based on thermal imaging and visible light images. Background Technology

[0002] Bucket wheel stacker-reclaimers, as core material conveying equipment in bulk cargo ports, thermal power plants, and large mines, operate under harsh conditions of long-term exposure, high load, and continuous operation. Their critical components, such as the slewing mechanism, drums, and bearing housings, are highly susceptible to overheating or structural damage due to wear, fatigue, or lubrication failure. Unplanned shutdowns can result in significant economic losses and safety hazards. Therefore, real-time and accurate health monitoring of key components of bucket wheel stacker-reclaimers is crucial for ensuring the continuity and safety of production operations.

[0003] However, existing monitoring technologies typically rely independently on infrared thermal imaging for temperature measurement and early warning or visible light cameras for surface inspection, which presents significant technical bottlenecks in complex outdoor application scenarios. Firstly, infrared thermal imagers collect the radiant temperature of an object's surface, not its true thermodynamic temperature, and their readings are heavily dependent on the surface emissivity and ambient background radiation. At bucket wheel excavator operation sites, equipment surfaces are often covered with unevenly thick coal dust or mineral ash, causing dynamic drift in emissivity. Simultaneously, strong direct sunlight generates significant temperature rise and specular reflection on metal structures. Existing technologies often lack physical-based correction mechanisms, making it difficult to distinguish between heat generated by internal faults and thermal interference caused by external solar radiation or dust accumulation. This leads to monitoring systems easily misreporting normally heated components as overheating faults. Furthermore, because the bucket wheel excavator boom is in a state of dynamic rotation and vibration, it is difficult to maintain real-time and accurate alignment of visible light and infrared images in the spatial field of view, resulting in difficulties in multi-source data fusion and hindering pixel-level accurate fault location and status determination while suppressing ambient light noise.

[0004] Therefore, an optimized health monitoring scheme for bucket wheel excavators based on thermal imaging and visible light images is desired. Summary of the Invention

[0005] The present invention aims to at least solve one of the technical problems existing in the prior art, and provides a health monitoring system and method for bucket wheel excavators based on thermal imaging and visible light images.

[0006] In a first aspect, embodiments of the present invention provide a method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images, comprising: S1: Perform time-series synchronization and distortion correction on the acquired multi-source heterogeneous data to obtain a synchronized dataset. The multi-source heterogeneous data includes the original visible light image stream, the original thermal radiation image stream, and environmental parameter data. S2: Perform cross-modal spatial registration on the synchronized dataset to obtain the registered thermal image and the reference visible light image; S3: Input the reference visible light image into the semantic segmentation network to obtain the component mask, and use the component mask to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain the visible light map and thermal map of the region of interest; S4: Perform temperature correction on the solar azimuth and solar altitude angles, visible light map of region of interest and thermal map of region of interest in the environmental parameter data based on visual perception physical parameters to obtain the real temperature distribution map; S5: Perform multi-dimensional feature deep fusion and anomaly detection on the real temperature distribution map and the visible light map of the region of interest to obtain a fused state vector containing the fault type and location coordinates; S6: Based on the fused state vector, the abnormal information of the reference visible light image is mapped and visualized to obtain the device health monitoring report.

[0007] Secondly, embodiments of the present invention provide a health monitoring system for bucket wheel excavators based on thermal imaging and visible light images, comprising: The timing synchronization and distortion correction module is used to perform timing synchronization and distortion correction on the acquired multi-source heterogeneous data to obtain a synchronized dataset. The multi-source heterogeneous data includes the original visible light image stream, the original thermal radiation image stream, and environmental parameter data. The cross-modal space registration module is used to perform cross-modal space registration on the synchronized dataset to obtain the registered thermal image and the reference visible light image; The visible light image and thermal image acquisition module is used to input the reference visible light image into the semantic segmentation network to obtain the component mask, and use the component mask to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain the visible light image and thermal image of the region of interest. The temperature correction module is used to perform temperature correction on the solar azimuth angle and solar altitude angle, the visible light map of the region of interest and the thermal map of the region of interest based on visual perception physical parameters in order to obtain the real temperature distribution map. The multidimensional feature deep fusion and anomaly detection module is used to perform multidimensional feature deep fusion and anomaly detection on the real temperature distribution map and the visible light map of the region of interest to obtain a fused state vector containing the fault type and location coordinates. The equipment health monitoring report acquisition module is used to map and visualize abnormal information from a reference visible light image based on a fused state vector to obtain an equipment health monitoring report.

[0008] Compared with existing technologies, this invention proposes a health monitoring method for bucket wheel excavators based on thermal imaging and visible light images. It collects multi-source heterogeneous data containing visible light and thermal radiation, performs time-series synchronization and distortion correction, and then utilizes feature matching technology based on rigid structure edges to achieve accurate spatial registration of infrared and visible light images under dynamic operating conditions. Based on this, a semantic segmentation network is used to extract masks of key components to shield against background interference. Furthermore, based on the texture and illumination features of the visible light image of the region of interest, the surface emissivity is dynamically inverted, and a solar radiation reflection model is constructed. Subsequently, the infrared data undergoes physical-level radiation value correction to remove interference from external solar heat sources and surface ash, reconstructing the true thermodynamic temperature of the equipment surface. Finally, multi-dimensional deep fusion discrimination is performed by combining thermodynamic gradients and visible light surface defect features, achieving high-confidence fault diagnosis and visual localization of key bucket wheel excavator components in outdoor environments with strong interference. Attached Figure Description

[0009] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to an embodiment of the present invention; Figure 2 This is a schematic diagram of data flow in a bucket wheel excavator equipment health monitoring method based on thermal imaging and visible light images according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to an embodiment of the present invention, which performs time-series synchronization and distortion correction on acquired multi-source heterogeneous data to obtain a synchronized dataset. The multi-source heterogeneous data includes the original visible light image stream, the original thermal radiation image stream, and environmental parameter data. Figure 4 This is a flowchart illustrating the cross-modal spatial registration of a synchronized dataset to obtain a registered thermal image and a reference visible light image, according to an embodiment of the present invention, for a method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images. Figure 5The flowchart illustrates the process of a health monitoring method for bucket wheel excavator equipment based on thermal imaging and visible light images according to an embodiment of the present invention. This method involves inputting a reference visible light image into a semantic segmentation network to obtain a component mask, and then using the component mask to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain a visible light map and a thermal map of the region of interest. Figure 6 This is a flowchart illustrating the multi-dimensional feature deep fusion and anomaly detection of a bucket wheel excavator equipment health monitoring method based on thermal imaging and visible light images according to an embodiment of the present invention, to obtain a fused state vector containing fault type and location coordinates. Figure 7 This is a block diagram of a bucket wheel excavator equipment health monitoring system based on thermal imaging and visible light images according to an embodiment of the present invention. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0012] Unless otherwise specifically stated, the technical or scientific terms used in the embodiments of this invention should be understood in their ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms "comprising" or "including," as used in the embodiments of this invention, do not limit the shapes, numbers, steps, actions, operations, components, elements, and / or groups thereof mentioned, nor do they exclude the appearance or addition of one or more other different shapes, numbers, steps, actions, operations, components, elements, and / or groups thereof, or the inclusion of these.

[0013] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale, and techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail; however, where appropriate, the illustrated techniques, methods, and apparatus should be considered part of the specification. In all the examples shown and discussed herein, any other specific example may have different values. It should be noted that similar symbols and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0014] In the description of the embodiments of the present invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In the embodiments of the present invention, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in the embodiments of the present invention, as well as the features of different embodiments or examples.

[0015] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention; it should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0016] Existing health monitoring technologies for bucket wheel excavators mainly rely on single thermal imaging or visible light inspection. However, in complex outdoor operating environments, strong solar radiation reflection and uneven dust coverage on the equipment surface can lead to serious temperature measurement deviations and false alarms. Furthermore, the dynamic field of view generated by the equipment's rotation makes accurate spatiotemporal alignment of multi-source data difficult. This limitation of single-modal information and the fragmentation of cross-modal semantics makes it difficult for existing systems to distinguish between internal fault heat and external interference heat, failing to meet the requirements for high-precision fault location. Therefore, this invention proposes a health monitoring method for bucket wheel excavators based on thermal imaging and visible light images. This method first performs hard-triggered temporal synchronization and distortion correction on the acquired visible light and thermal radiation image streams, and calculates the homography matrix based on the edge features of the rigid structure to achieve accurate cross-modal spatial registration in dynamic scenes. Based on this, a semantic segmentation network is introduced to intelligently identify key components and remove background interference, generating a region of interest mask. Subsequently, the scheme innovatively utilizes the texture grayscale features in visible light images to estimate dynamic emissivity, and combines the solar azimuth angle, solar altitude angle, and surface normal vector to eliminate the solar reflected heat component. This allows for the reconstruction of a realistic temperature distribution map through physical inversion, fundamentally eliminating temperature measurement errors caused by ambient light and dust accumulation. Finally, by deeply fusing real thermodynamic gradient features with visible light surface defect features, a fused state vector containing fault type and location coordinates is generated. This vector is then mapped and rendered on the visible image, enabling high-confidence monitoring and visualized closed-loop feedback of equipment health status.

[0017] Figure 1 This is a flowchart of a method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to an embodiment of the present invention. Figure 2This is a schematic diagram of data flow in a bucket wheel excavator equipment health monitoring method based on thermal imaging and visible light images according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to an embodiment of the present invention includes the following steps: S1, performing temporal synchronization and distortion correction processing on the acquired multi-source heterogeneous data to obtain a synchronized dataset, wherein the multi-source heterogeneous data includes the original visible light image stream, the original thermal radiation image stream, and environmental parameter data; S2, performing cross-modal spatial registration on the synchronized dataset to obtain a registered thermal image and a reference visible light image; S3, inputting the reference visible light image into a semantic segmentation network to obtain a component mask, and using the component mask to extract the mask from the registered thermal image and the reference visible light image. / Background culling operation to obtain visible light image and thermal image of region of interest; S4, perform temperature correction based on visual perception physical parameter inversion on solar azimuth angle and solar altitude angle, visible light image and thermal image of region of interest in environmental parameter data to obtain real temperature distribution map; S5, perform multi-dimensional feature deep fusion and anomaly detection on real temperature distribution map and visible light image of region of interest to obtain fused state vector containing fault type and location coordinates; S6, based on fused state vector, perform anomaly information mapping and visualization rendering on reference visible light image to obtain equipment health monitoring report.

[0018] Specifically, in step S1, the acquired multi-source heterogeneous data undergoes time synchronization and distortion correction processing to obtain a synchronized dataset. The multi-source heterogeneous data includes the original visible light image stream, the original thermal radiation image stream, and environmental parameter data. It should be noted that, given that bucket wheel excavators experience cantilever rotation and mechanical vibration during operation, and that different sensors such as visible light cameras and infrared thermal imagers often have independent clock sources and different frame rates, coupled with the inherent distortion characteristics of optical lenses, the directly acquired multi-source data exhibits phase differences in the time dimension and geometric distortions in the spatial dimension, making it difficult to meet the requirements of high-precision fusion. Based on this, the technical solution of this invention first performs time synchronization and distortion correction processing on the acquired multi-source heterogeneous data. The multi-source heterogeneous data encompasses the original visible light image stream, the original thermal radiation image stream, and environmental parameter data. Specifically, by parsing the hardware timestamp in the data stream, using the visible light image as a reference anchor point, a matching thermal radiation image is searched within a preset time tolerance range, and a preset camera intrinsic parameter matrix is ​​invoked to perform geometric correction on the image data. This eliminates the effects of temporal asynchrony and spatial distortion between heterogeneous sensors, ensuring strict physical correspondence and accurate geometric reconstruction of each data set. Through this processing, a synchronized dataset containing time-aligned and distortion-free images can be generated, providing a reliable data foundation for subsequent cross-modal accurate registration and physical inversion, effectively avoiding the risk of misjudgment caused by motion blur or temporal misalignment.

[0019] Figure 3 This document describes a method for health monitoring of bucket wheel excavators based on thermal imaging and visible light images, according to an embodiment of the present invention. It describes the temporal synchronization and distortion correction of acquired multi-source heterogeneous data to obtain a synchronized dataset. The multi-source heterogeneous data includes a raw visible light image stream, a raw thermal radiation image stream, and environmental parameter data. (See flowchart for example.) Figure 3 As shown, step S1 includes: S11, using a frame header parser to unpack and separate the raw sensor data stream received through the multi-threaded parallel interface, and storing the separated raw visible light image, raw thermal radiation image, and environmental parameter data to obtain a buffer queue data packet; S12, using the hardware timestamp of the raw visible light image generated in the buffer queue data packet as the reference anchor point, performing a sliding window search based on a preset temporal tolerance threshold and associating it with the nearest neighbor environmental parameters to obtain a temporal matching data pair; S13, retrieving a preset camera intrinsic parameter matrix and distortion coefficients, and performing geometric correction coordinate calculation and pixel reconstruction on the image data in the temporal matching data pair to obtain a synchronized dataset.

[0020] In step S11, a frame header parser is used to unpack and separate the raw sensor data stream received through the multi-threaded parallel interface, and the separated raw visible light image, raw thermal radiation image, and environmental parameter data are stored to obtain a buffer queue data packet. It should be noted that, due to the high throughput and asynchronous triggering characteristics of multi-source sensor data at the bucket wheel excavator site, direct serial reception and processing are prone to data blocking and frame loss due to bandwidth contention or processing delays. Furthermore, the mixed-transmission raw bit stream lacks structured classification, making it difficult to directly use for subsequent precise alignment. Based on this, the technical solution of this invention first uses a frame header parser to unpack and separate the raw sensor data stream received through the multi-threaded parallel interface, and stores the separated raw visible light image, raw thermal radiation image, and environmental parameter data to obtain a buffer queue data packet. This constructs a parallel, non-blocking data reception channel, achieving real-time decoupling and classification buffering of heterogeneous data. Through the above processing, the risk of data loss in high-concurrency scenarios can be effectively avoided, ensuring that the various types of data stored in the queue maintain integrity and temporal originality, providing a stable and orderly data source for subsequent synchronization operations.

[0021] More specifically, in a concrete example of this invention, the data reception and preprocessing process follows a parallel and pipelined execution logic. First, a multi-threaded parallel reception mechanism is initiated, allocating independent data reception threads to the hardware interfaces connecting the visible light camera, infrared thermal imager, and environmental sensor, capturing raw sensor data streams from different ports in real time. Then, the data stream enters the frame header parsing stage, where a pre-built frame header parser scans the captured binary data packets, identifying protocol header information including data type flags, payload length, and checksums. Based on the parsing results, the data payload encapsulated within the data packets is stripped, distinguishing between high-resolution raw visible light images, raw thermal radiation images containing radiation temperature information, and environmental parameter data containing temperature, humidity, and illumination angle. Finally, the three types of heterogeneous data are written into corresponding first-in-first-out (FIFO) buffer queues. These queues serve as intermediate storage media, dynamically smoothing fluctuations in the data arrival rate, thereby forming clearly structured and categorized buffer queue data packets, awaiting immediate invocation by the subsequent timing alignment module.

[0022] In step S12, using the hardware timestamp of the original visible light image generated in the buffer queue data packet as the reference anchor point, a sliding window search based on a preset time-to-tolerance threshold is performed on the original thermal radiation image, and it is correlated with the nearest neighbor environmental parameters to obtain time-matched data pairs. It should be noted that because heterogeneous sensors such as visible light cameras and thermal imagers typically have independent acquisition frequencies and triggering mechanisms, and because the bucket wheel excavator is in a continuous rotation and pitch motion state during operation, there is a natural nonlinear phase deviation in the time axis of data streams of different modes. If directly paired sequentially, microsecond-level time misalignment will lead to significant spatial position shifts under high-speed motion, causing subsequent fusion analysis to fail. Based on this, the technical solution of this invention further uses the hardware timestamp of the original visible light image generated in the buffer queue data packet as the reference anchor point, and performs a sliding window search based on a preset time-to-tolerance threshold and correlates with the nearest neighbor environmental parameters on the original thermal radiation image to obtain time-matched data pairs. This establishes a hard synchronization mechanism based on absolute physical time, filtering out multi-source data combinations that are highly consistent in the instantaneous physical scene. Through the above processing, invalid frames that exceed the timing tolerance can be effectively eliminated, ensuring that the visible light and thermal radiation images describe the motion posture and thermal state of the bucket wheel machine at the same instant, thus laying a strict time reference for high-precision spatial registration.

[0023] More specifically, in a particular example of the present invention, the timing alignment process strictly follows a time-priority matching logic. First, a frame of raw visible light image is read from the buffer queue, and the precise hardware exposure timestamp recorded in its metadata is extracted as a reference anchor point. Subsequently, a sliding window search mechanism is initiated in the buffer queue of the original thermal radiation images. This window covers a preset time range before and after the reference anchor point. During the search process, the timestamps of the thermal radiation images within the window are calculated one by one. The absolute time difference between the reference anchor point and the reference point. Based on this, a stringent timing tolerance threshold is set. (For example, 10 milliseconds) is used to determine whether two image frames belong to the same physical instant. If the calculated absolute time difference is less than or equal to this time tolerance threshold, the two image frames are considered to be successfully matched; otherwise, if it exceeds the threshold, the thermal radiation image is marked as an invalid frame and discarded until a matching frame is found. Simultaneously, a nearest neighbor search operation is performed on the environmental parameter data stream to select a set of environmental parameters (including temperature, humidity, solar azimuth, and solar altitude angle) closest to the baseline anchor point timestamp. Finally, the successfully matched visible light image, thermal radiation image, and associated environmental parameters are packaged and combined to generate a set of time-series strictly aligned matching data pairs. The matching logic follows the formula... In the formula A timestamp representing a visible light image. A timestamp representing a thermal radiation image. This represents the preset timing tolerance threshold.

[0024] In step S13, a preset camera intrinsic parameter matrix and distortion coefficients are retrieved, and geometric correction coordinate calculations and pixel reconstruction are performed on the image data in the time-series matching data pair to obtain a synchronized dataset. It should be noted that, due to the limitations of optical physics, the wide-angle visible light lens or infrared germanium lens used in the bucket wheel excavator monitoring site inevitably introduces radial and tangential distortion during the imaging process. This causes the projection of rigid straight lines such as the bucket wheel excavator's cantilever and drum to appear as barrel-shaped or pincushion-shaped bends in the image. This geometric deformation severely disrupts the linear mapping relationship between image pixels and physical spatial coordinates, directly hindering subsequent high-precision registration. Based on this, the technical solution of this invention further retrieves a preset camera intrinsic parameter matrix and distortion coefficients, and performs geometric correction coordinate calculations and pixel reconstruction on the image data in the time-series matching data pair to obtain a synchronized dataset. This allows for mathematical correction of the spatial position and grayscale resampling of each pixel in the image based on a rigorous optical imaging model. Through the above processing, the geometric nonlinearity error introduced by the optical system can be effectively eliminated, the true shape characteristics of the equipment components can be restored, and a geometrically accurate and temporally aligned standard dataset can be generated, providing a high-precision spatial reference for pixel-level fusion of cross-modal images.

[0025] More specifically, in a concrete example of the present invention, the geometric correction process follows a logical path from parameter loading to coordinate mapping and then to reconstruction. First, the camera intrinsic parameter matrix and distortion coefficient vector, pre-calibrated using a calibration board, are read from non-volatile memory. This distortion coefficient vector includes radial and tangential distortion coefficients. Then, based on the Brown-Conrady distortion model, for each normalized pixel coordinate (x, y) on the ideal distortion-free image plane, its corresponding normalized coordinates on the actually acquired distorted image plane are calculated. The calculation logic follows the formula:

[0026] in, Represents the radial distance from a pixel to the center of the image, satisfying Represents the radial distortion coefficient, used to correct barrel or pincushion distortion caused by lens curvature. The tangential distortion coefficient is used to correct distortion caused by the lens not being perfectly parallel to the imaging plane. Finally, the calculated distortion coordinates are indexed in the original image, and the pixel grayscale or radiometric value at that location is calculated using a bilinear interpolation algorithm. This value is then filled into the corresponding position in the new image, thus completing the pixel reconstruction of the visible light image and the thermal radiation image, and outputting a synchronized dataset with corrected geometry.

[0027] Specifically, in step S2, cross-modal spatial registration is performed on the synchronized dataset to obtain a registered thermal image and a reference visible light image. It should be noted that, given the inevitable parallax in the physical deployment of visible light cameras and infrared thermal imagers, and the continuous rotation and mechanical vibration of the bucket wheel excavator boom during operation, significant nonlinear offsets exist in the pixel coordinates of the same equipment component in different modal images. Simple time synchronization is insufficient to solve the geometric misalignment problem in the spatial dimension. Therefore, the technical solution of this invention further performs cross-modal spatial registration on the synchronized dataset to obtain a registered thermal image and a reference visible light image, thereby constructing and applying a cross-modal geometric transformation mapping model to strictly project the thermal radiation data into the coordinate system of the visible light image. Through the above processing, precise overlap of thermodynamic information and visual texture information at the sub-pixel level can be achieved, effectively avoiding fault point location drift or false association caused by spatial alignment deviations, and providing a reliable spatial reference for subsequent multi-dimensional feature fusion.

[0028] Figure 4 This is a flowchart illustrating the cross-modal spatial registration of a synchronized dataset to obtain a registered thermal image and a reference visible light image, according to an embodiment of the present invention, for a method of health monitoring of bucket wheel excavators based on thermal imaging and visible light images. Figure 4As shown, step S2 includes: S21, extracting rigid structure contours and calculating binarized gradients from the reference visible light image and the source thermal radiation image obtained from the synchronized dataset to obtain visible light edge maps and thermal radiation edge maps; S22, performing feature key point matching and mismatch point removal calculations on the visible light edge maps and thermal radiation edge maps to obtain homography matrices; S23, based on the homography matrix, performing perspective transformation and bilinear interpolation resampling on the source thermal radiation image projected to the coordinate system space of the reference visible light image to obtain registered thermal images and reference visible light images.

[0029] In step S21, rigid structure contour extraction and binarized gradient calculation are performed on the reference visible light image and source thermal radiation image obtained from the synchronized dataset to obtain visible light edge maps and thermal radiation edge maps. It should be noted that visible light image imaging relies on the light reflection characteristics of the object's surface, presenting color and texture information, while infrared thermal radiation image imaging relies on the object's thermal radiation characteristics, presenting temperature distribution information. There are fundamental modal differences between the two in grayscale distribution and texture details, and matching algorithms based directly on grayscale values ​​are prone to failure under nonlinear differences. Therefore, the technical solution of this invention further performs rigid structure contour extraction and binarized gradient calculation on the reference visible light image and source thermal radiation image obtained from the synchronized dataset to obtain visible light edge maps and thermal radiation edge maps. This is used to strip away the texture and color layers that are susceptible to illumination and temperature differences, and to extract the geometric structural features common to rigid components such as the bucket wheel excavator cantilever and bearing housing under both modalities. The above processing can effectively overcome the semantic gap between cross-modal data, transform heterogeneous images into a shared geometric feature space, and provide a robust structured data foundation for subsequent accurate matching based on feature points.

[0030] More specifically, in a specific example of the present invention, the extraction process of the structural contour follows a processing logic from noise suppression to gradient calculation and then to edge binarization. First, the reference visible light image and the source thermal radiation image are unpacked and separated from the synchronized dataset. Considering the interference of dust and thermal noise in the industrial environment, Gaussian filtering is applied to both images for smoothing preprocessing. Then, the Sobel operator is used to perform convolution operations on the smoothed images in the horizontal and vertical directions to obtain the horizontal gradient component at each pixel location. and vertical gradient components Based on this, the gradient magnitude G of each pixel is calculated using the Euclidean norm, and the calculation logic follows the formula:

[0031] in, The gradient magnitude of a pixel is used to characterize the strength of that point as an edge. Represents the gradient component in the horizontal direction. This represents the gradient component in the vertical direction. Finally, the gradient magnitude map is binarized and truncated according to a preset gradient threshold, retaining pixels with significant gradient magnitudes as edge candidates, thereby generating a visible light edge map and a thermal radiation edge map that only contain the rigid contour information of the bucket wheel excavator.

[0032] In step S22, feature key point matching and mismatch point removal calculations are performed on the visible light edge image and the thermal radiation edge image to obtain the homography matrix. It should be noted that due to the inherent physical parallax in the spatial deployment of the visible light camera and the infrared thermal imager, and the significant differences in perspective distortion of the image at different viewing angles caused by the long distance and depth of the bucket wheel excavator's cantilever, simple linear translation cannot achieve precise overlap of the two heterogeneous coordinate systems. Based on this, the technical solution of this invention further performs feature key point matching and mismatch point removal calculations on the visible light edge image and the thermal radiation edge image to obtain the homography matrix. This allows the use of common feature points on the rigid structure edge of the bucket wheel excavator to solve the mathematical model describing the projection transformation relationship between the two imaging planes. Through the above processing, the geometric transfer function mapping from the thermal infrared coordinate system to the visible light coordinate system can be effectively established, ensuring that subsequent thermal radiation data can be projected onto the corresponding position of the visible light image with sub-pixel accuracy.

[0033] More specifically, in a concrete example of this invention, the homography matrix calculation process follows an execution logic from feature extraction to robust estimation. First, using the ORB feature extraction algorithm, which possesses rotational and scale invariance, corner points and edge feature points are detected on both the visible light edge map and the thermal radiation edge map, generating corresponding binary descriptors. Then, a brute-force matcher combined with the Hamming distance metric is used to perform a preliminary similarity comparison between the two sets of descriptors, filtering out the matching point pairs with the closest feature distance. Considering the potential for mismatches due to repetitive textures in industrial scenarios, a random sampling consensus algorithm is used to iteratively filter the preliminary matching point set. By randomly selecting a subset, calculating the transformation model, and verifying the fitting error of the remaining points, outliers deviating from the model are eliminated, retaining the set of inliers with high confidence. Finally, based on the retained set of inliers, a 3×3 dimension homography matrix is ​​calculated using the least squares method. This matrix describes the coordinates of the source thermal radiation image. To reference visible light image coordinates The projection mapping relationship is calculated according to the following formula:

[0034] in, to The transformation parameters that constitute the homography matrix determine the rotation, translation, scaling, and perspective transformation properties of the image. Represents the coordinates of feature points in the source thermal radiation image. This represents the coordinates of the corresponding feature point in the reference visible light image. This represents the scale factor used for homogeneous coordinate normalization.

[0035] In step S23, based on the homography matrix, the source thermal radiation image is projected onto the reference visible light image coordinate system space through perspective transformation and bilinear interpolation resampling to obtain the registered thermal image and the reference visible light image. It should be noted that since the homography matrix only describes the mathematical mapping relationship between the thermal infrared coordinate system and the visible light coordinate system, and the data of the original thermal radiation image is still stored in the original pixel grid, without substantial pixel resampling and migration, the corresponding temperature data cannot be directly obtained in the visible light coordinate system. Therefore, the technical solution of this invention further uses the homography matrix to project the source thermal radiation image onto the reference visible light image coordinate system space through perspective transformation and bilinear interpolation resampling to obtain the registered thermal image and the reference visible light image, thereby performing pixel-level geometric transformation operations to fill the corresponding pixel positions in the visible light image with thermal radiation values. Through the above processing, a registered thermal image with a resolution and field of view that perfectly match can be generated, achieving strict alignment of the two modalities at the pixel level, thus providing a spatially consistent data carrier for subsequent mask clipping and multi-dimensional feature fusion.

[0036] More specifically, in a specific example of the present invention, the implementation process of perspective transformation and resampling follows the execution logic from coordinate mapping to grayscale interpolation. First, a perspective transformation operator is constructed using the calculated homography matrix. This operator iterates through the coordinates of each target pixel in the reference visible light image coordinate space, finding its corresponding floating-point coordinates in the source thermal radiation image based on the inverse transformation relationship. Since the calculated mapped coordinates typically fall into non-integer sub-pixel positions, direct rounding would lead to significant jagged edges and radiance distortion. Therefore, a bilinear interpolation algorithm is used to calculate the radiance value of the target pixel. For the target coordinates... Based on its four adjacent integer coordinate points The known radiation values ​​are weighted for calculation, and the calculation logic follows the formula:

[0037] in, The registered thermal image obtained after interpolation calculation is located in coordinates. The thermal radiation value at that location, Represents the floating-point coordinates of the target in the source thermal radiation image. Four adjacent integer coordinate points, The image represents the thermal radiation source at the corresponding integer coordinate points. The known pixel values ​​at each location are then calculated. Finally, all the calculated interpolated radiometric data are repackaged into a matrix with the same resolution as the visible light image, and the registered thermal image and the reference visible light image are output.

[0038] Specifically, in step S3, the reference visible light image is input into the semantic segmentation network to obtain a component mask. The component mask is then used to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain the visible light map and thermal map of the region of interest. It should be noted that, given that bucket wheel excavators typically operate in open and complex environments, their monitoring field of view not only includes critical monitoring components such as bearing housings and reducers, but also contains numerous interfering areas such as the sky, ground, conveyor belts, and non-critical steel structures. Directly performing thermal analysis on the entire field of view image could easily introduce false high-temperature alarms due to direct sunlight, heated ground, or other heat sources in the background, while also increasing the unnecessary load on subsequent physical inversion calculations. Based on this, the technical solution of the present invention further inputs a reference visible light image into a semantic segmentation network to obtain a component mask. The component mask is then used to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain a visible light map and a thermal map of the region of interest. This leverages the powerful semantic understanding capabilities of deep learning to intelligently identify and accurately segment the physical contour of the device to be monitored, converging the focus of the heterogeneous data stream from the entire image to the key components themselves. Through this processing, the interference of environmental background noise on temperature interpretation can be shielded from the source, ensuring that subsequent emissivity estimation and temperature correction are performed only on the effective pixels of the key components, thereby improving the signal-to-noise ratio and computational efficiency of fault diagnosis.

[0039] Figure 5 This is a flowchart illustrating the process of a health monitoring method for bucket wheel excavators based on thermal imaging and visible light images, according to an embodiment of the present invention. The method involves inputting a reference visible light image into a semantic segmentation network to obtain a component mask, and then using the component mask to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain a visible light map and a thermal map of the region of interest. Figure 5 As shown, step S3 includes: S31, inputting the reference visible light image into a preset lightweight semantic segmentation network to obtain an original classification probability map; S32, determining pixel category labels from the original classification probability map, and performing morphological hole filling and opening operations to denoise the binary regions corresponding to the key monitoring component set to obtain a component mask; S33, based on the component mask, performing pixel-level Adama product operations with background removal on the reference visible light image and the registered thermal image respectively to obtain the visible light map and thermal map of the region of interest.

[0040] In step S31, the reference visible light image is input into a preset lightweight semantic segmentation network to obtain the original classification probability map. It should be noted that the background environment in the bucket wheel excavator operation scenario is extremely complex. The sky, ground coal piles, and steel structure supports occupy a large number of pixels in the visible light image, and changes in illumination cause nonlinear shifts in the color features of the same component. Traditional threshold- or edge-based segmentation methods are difficult to accurately distinguish key monitoring components from background noise, and cannot provide reliable area delimitation for subsequent accurate temperature measurement. Based on this, the technical solution of this invention further inputs the reference visible light image into a preset lightweight semantic segmentation network to obtain the original classification probability map. Specifically, this includes pixel normalization preprocessing of the reference visible light image to obtain a processed image, and inputting the processed image into the preset lightweight semantic segmentation network for forward inference and posterior probability calculation based on hollow spatial pyramid pooling to obtain the original classification probability map. This utilizes the powerful feature extraction and semantic understanding capabilities of deep convolutional neural networks to map the image from the RGB color space to a high-dimensional semantic probability space. Through the above processing, the probability quantification of the physical category to which each pixel in the image belongs can be achieved, thereby effectively solving the problem of low recognition rate of key components under complex working conditions and laying a solid probabilistic data foundation for generating high-precision component masks.

[0041] More specifically, in a specific example of the present invention, inputting a reference visible light image into a preset lightweight semantic segmentation network to obtain an original classification probability map includes: performing pixel normalization preprocessing on the reference visible light image to obtain a processed image; and inputting the processed image into the preset lightweight semantic segmentation network to perform forward inference and posterior probability calculation based on hollow spatial pyramid pooling to obtain the original classification probability map.

[0042] In other words, more specifically, the implementation process of probabilistic graph generation follows an execution logic from data standardization to deep inference. First, pixel normalization preprocessing is performed on the input reference visible light image, linearly mapping the integer gray values ​​in the original image between 0 and 255 to the floating-point range of 0 to 1, and then performing mean and variance standardization to eliminate the numerical influence of light intensity differences on the model input, improving the network convergence speed and inference stability. Subsequently, the processed image tensor is input into a pre-built lightweight semantic segmentation network (e.g., a DeepLabV3+ architecture using MobileNet as the backbone). This network utilizes a dilated spatial pyramid pooling module to extract multi-scale contextual features of the image in parallel through dilated convolutions at different sampling rates, thereby adapting to the scale changes of bucket wheel excavator components at different viewing distances. In particular, before the network is used for inference, a labeled dataset containing key components of the bucket wheel excavator (such as drums, bearing seats, and support arms) is pre-constructed. This dataset covers historical images collected under different weather conditions (sunny and cloudy) and different levels of dust accumulation, and pixel-level masks of the components are manually labeled. Using this dataset, the DeepLabV3+ network was trained end-to-end with supervised training using the cross-entropy loss function until the average intersection-union ratio on the validation set reached a preset threshold (e.g., above 85%). The network weight parameters were then saved as a pre-defined model. The network output layer generated a 3D feature tensor corresponding to a predefined set of categories C (e.g., background, bearing, roller), i.e., the original logistic values. Finally, the Softmax activation function was used to operate on the feature tensor along the channel dimension to calculate the posterior probability of each pixel (x, y) belonging to the k-th category. .

[0043] In step S32, pixel category labels are determined from the original classification probability map, and morphological hole filling and opening operations are performed on the binary region corresponding to the key monitoring component set to obtain a component mask. It should be noted that since the original classification probability map output by the neural network only represents the statistical confidence of pixels belonging to each category, and is affected by uneven lighting or localized dust accumulation, the directly converted classification map often contains discontinuous noise and internal holes, resulting in rough edges and internal fragmentation of the generated component region, failing to completely cover the physical entity of the monitored device. Based on this, the technical solution of this invention further determines pixel category labels from the original classification probability map and performs morphological hole filling and opening operations on the binary region corresponding to the key monitoring component set to obtain a component mask. This discretizes the continuous probability distribution into definite physical semantic labels and uses morphological geometric operations to repair topological defects caused by recognition errors. Through the above processing, a binary mask with smooth edges, dense internal structure, and removed background stray noise can be generated, ensuring that subsequent thermal analysis is strictly limited to the effective physical boundaries of the key components, effectively improving the spatial accuracy of fault location.

[0044] More specifically, in a specific example of the present invention, the mask generation process follows an execution logic from probabilistic decision-making to binarization filtering and then to morphological optimization. First, a maximum probability indexing operation is performed on the original classification probability map, comparing the posterior probability values ​​of each pixel across all predefined category channels, and selecting the category index with the highest probability as the predicted category label for that pixel. Subsequently, based on a preset set of key monitoring components (e.g., an index list including bearing housings, reducers, and redirecting rollers), a binarization mapping logic is constructed: if the predicted label of a pixel belongs to this set, it is marked as a foreground target (assigned a value of 1); otherwise, it is marked as background (assigned a value of 0). After obtaining the initial binary mask, a morphological closing operation is performed on the mask using structuring elements, i.e., dilation followed by erosion, to fill the tiny holes inside the component caused by reflection or dirt, restoring the connectivity of the region; then, a morphological opening operation is performed, i.e., erosion followed by dilation, to filter out isolated noise points in the background caused by misidentification, finally outputting a high-confidence component mask.

[0045] In step S33, based on the component mask, pixel-level Hadamard product operations with background removal are performed on the reference visible light image and the registered thermal image to obtain the visible light image and thermal image of the region of interest (ROI). It should be noted that in the full-field image of the bucket wheel excavator operation site, the sky, ground, and non-critical structures occupy the majority of the pixel area, and these background areas often contain strong interfering heat sources such as direct sunlight or heated ground. Directly performing subsequent physical inversion and fault diagnosis on the entire image would not only introduce significant computational redundancy but also easily lead to false alarms due to background thermal noise. Therefore, the technical solution of this invention further utilizes the component mask to perform pixel-level Hadamard product operations with background removal on the reference visible light image and the registered thermal image to obtain the visible light image and thermal image of the ROI. This leverages the gating characteristics of the binary mask to perform pixel-level filtering and purification of heterogeneous image data, forcing the pixel values ​​of non-critical areas to zero. Through the above processing, the interference of environmental background on the monitoring target can be completely shielded, ensuring that subsequent processing is only performed on the effective physical pixels of key components, thereby effectively improving the signal-to-noise ratio of fault feature extraction and the robustness of the diagnostic system.

[0046] More specifically, in a concrete example of the present invention, the implementation process of region of interest extraction follows an execution logic from matrix broadcasting to element-wise operations. First, a binarized component mask generated by semantic segmentation is retrieved, in which the pixel values ​​of key component regions are 1, and the pixel values ​​of background regions are 0. Then, the component mask is pixel-aligned with both a reference visible light image and a registered thermal image, and the Hadamard product operation, i.e., element-wise multiplication of the matrix, is performed. Through this mathematical operation, the pixel values ​​of the background regions are completely suppressed by multiplying by 0, while the texture and radiation data of the key component regions are fully preserved by multiplying by 1. Finally, a clean visible light component image and thermal radiation matrix, freed from environmental background interference, are output for use by the subsequent physical correction module.

[0047] Specifically, in step S4, temperature correction based on visual perception physical parameter inversion is performed on the solar azimuth and solar altitude angles, the visible light map of the region of interest (ROI), and the ROI thermal map in the environmental parameter data to obtain a true temperature distribution map. It should be noted that the radiation temperature measured by infrared thermal imagers is inherently significantly affected by the surface emissivity of the measured object and the ambient background radiation. In outdoor operation scenarios of bucket wheel excavators, the equipment surface is often covered with coal dust, causing nonlinear drift in emissivity. Furthermore, strong direct sunlight can produce specular reflection or radiation gain on metal surfaces, resulting in the original thermal map failing to accurately reflect the internal thermodynamic state of the equipment. Therefore, the technical solution of this invention further performs temperature correction based on visual perception physical parameter inversion on the solar azimuth and solar altitude angles, the ROI visible light map of the region of interest (ROI), and the ROI thermal map in the environmental parameter data to obtain a true temperature distribution map. This constructs a physical inversion model that integrates visual texture features and geometric optics, dynamically estimating surface emissivity and eliminating solar reflected radiation components. Through the above processing, the radiation brightness affected by environmental interference can be restored to the true thermodynamic temperature of the object, which does not change with light and dust accumulation. This effectively eliminates false alarms caused by false external heat sources and ensures that fault diagnosis is based on real physical temperature rise data.

[0048] More specifically, in a specific example of the present invention, step S4 includes: determining a dynamic emissivity map based on the visible light map of the region of interest; constructing a solar incident light vector based on environmental parameter data; extracting pixel-level surface normal vectors from the visible light map of the region of interest using a gradient operator, and calculating the incident geometry factor by combining the solar incident light vector and the surface normal vector; performing a radiation value correction calculation on the thermal map of the region of interest to remove the solar reflection component to obtain a corrected radiation intensity map; converting the ambient background temperature in the environmental parameter data into background radiation energy; and performing inverse thermodynamic inversion calculation on the corrected radiation intensity map based on the dynamic emissivity map and the background radiation energy to obtain a true temperature distribution map.

[0049] Accordingly, a dynamic emissivity map is determined based on the visible light map of the region of interest, and a solar incident light vector is constructed based on environmental parameter data. It should be noted that during the stacking and reclaiming operation of the bucket wheel excavator, the surface of its key rotating components is often unevenly covered with coal powder or mineral ash, causing the emissivity of different areas to dynamically fluctuate between clean metal and thick ash layers. Furthermore, the incident angle of outdoor sunlight changes continuously over time. Using a fixed emissivity model or ignoring the geometric relationship of illumination will directly lead to a significant deviation of the measured temperature data from the true value. Therefore, the technical solution of this invention further determines the dynamic emissivity map based on the visible light map of the region of interest and constructs a solar incident light vector based on environmental parameter data. This establishes a quantitative mapping between visual texture features and physical radiation properties, and determines the spatial projection geometry of the external heat source relative to the monitoring field of view. Through the above processing, adaptive dynamic correction of the emissivity of each pixel can be achieved, and a strict vector benchmark is provided for the subsequent accurate removal of the solar directional reflection component, thereby ensuring that the thermal inversion model conforms to the current surface physical state and ambient illumination conditions.

[0050] More specifically, in a specific example of the present invention, determining a dynamic emissivity map based on the visible light map of the region of interest includes: using the pixel brightness values ​​and local texture variance features of the visible light map of the region of interest within the area defined by the component mask to calculate a dust accumulation confidence index characterizing the degree of surface contamination; and performing weighted linear interpolation calculation on the preset emissivity of clean metal and coal dust based on the dust accumulation confidence index to obtain the dynamic emissivity map.

[0051] Furthermore, the construction of physical parameters follows an execution logic from visual feature quantization to geometric vector computation. First, the visible light image of the region of interest is converted to grayscale, and a sliding window is used to calculate the grayscale mean and local texture variance within the pixel neighborhood. Given that dusty areas typically exhibit low brightness and high roughness texture features, while clean metal surfaces exhibit high brightness and low roughness features, a dust accumulation confidence index characterizing the degree of surface contamination is calculated. Subsequently, based on this index at a preset clean metal emissivity... With coal dust emissivity Weighted linear interpolation is performed between the pixels to generate a dynamic emissivity map that corresponds one-to-one with each image pixel. Simultaneously, the geographical location, date, and time information recorded in the environmental parameter data are analyzed, and the current solar azimuth and elevation angles are calculated using a solar position algorithm. These are then converted into unit direction vectors in three-dimensional space, i.e., the solar incident light vector. This is used for subsequent calculations of the light reflection component.

[0052] Accordingly, a gradient operator is used to extract pixel-level surface normal vectors from the visible light image of the region of interest, and the incident geometric factor is calculated by combining the solar incident light vector and the surface normal vector. It should be noted that since key components of the bucket wheel excavator, such as the drum and bearing housing, are mainly composed of curved metal surfaces, their surface reflection characteristics to solar radiation strictly follow the laws of geometric optics. That is, the reflected heat intensity depends on the spatial angle between the microscopic surface normal direction and the solar incident light. A simple two-dimensional image brightness cannot directly resolve this difference in thermal gain caused by the three-dimensional geometry, making it difficult to distinguish between normal radiation-induced heating caused by the surface facing the sun and abnormal heating caused by internal equipment malfunctions. Based on this, the technical solution of this invention further utilizes a gradient operator to extract pixel-level surface normal vectors from the visible light image of the region of interest, and calculates the incident geometric factor by combining the solar incident light vector and the surface normal vector. This allows for the approximate deduction of the microscopic three-dimensional orientation of the object's surface using the grayscale gradient change of the visible light image, and, combined with known astronomical illumination vectors, quantifies the solar radiation reflection and reception capability of each pixel through geometric optics calculations. Through the above processing, a reflection intensity distribution matrix that accurately reflects the geometric response of the device surface to light can be constructed, thereby effectively identifying and marking the pixel set that is in the strong light reflection zone due to the surface geometric orientation, providing a rigorous physical and geometric basis for subsequent accurate removal of solar radiation interference.

[0053] More specifically, in a specific example of the present invention, the calculation of the incident geometric factor follows an execution logic from gradient estimation to vector dot product. First, the luminance component of the visible light image of the region of interest is selected. The Sobel or Scharr operator is used to calculate the gray-level gradient values ​​in the horizontal and vertical directions of the image, respectively. Based on the simplified physical assumption that shape originates from shadow, the rate of change of the gray-level gradient is mapped to the rate of change of surface height, thereby constructing the non-normalized normal vector of that pixel. Subsequently, the vector is normalized to its magnitude to obtain the unit surface normal vector describing the spatial orientation of that point. Based on this, the unit solar incident light vector constructed in the previous steps is retrieved. The cosine of the angle between the two objects is calculated using the dot product formula, which is the incident geometric factor. The larger the value of this factor, the more the surface area tends to face the incident sunlight perpendicularly, and the more significant the impact of solar radiation heat and reflection interference it receives; if the value is zero, it indicates that the area is on the shaded side or in the shadow area and is not affected by direct sunlight.

[0054] Accordingly, the thermal image of the region of interest is corrected by removing the solar reflection component to obtain a corrected radiation intensity map, and the ambient background temperature in the environmental parameter data is converted into background radiation energy. It should be noted that the total radiation energy received by the infrared thermal imager detector does not originate solely from the infrared emission of the device under test. In open-air environments, it is also superimposed with background reflected radiation from the surrounding environment and strong specular reflection from direct sunlight on metal surfaces. In particular, highly reflective metal components can reflect solar heat like a mirror at certain angles, directly causing the thermal image pixel readings to be much higher than their actual temperatures. Based on this, the technical solution of this invention further performs a radiation correction calculation on the thermal image of the region of interest by removing the solar reflection component to obtain a corrected radiation intensity map, and converts the ambient background temperature in the environmental parameter data into background radiation energy. This is done in accordance with the law of conservation of energy and Kirchhoff's radiation law to remove interference from external heat sources at the radiation flux level and quantify the baseline energy of the ambient thermal background. Through the above processing, the mixed radiation signal can be restored to a pure physical quantity that contains only the contribution of the object's own thermal radiation, thereby eliminating false hot spots caused by solar spots or high ambient temperatures, and laying a pure energy data foundation for subsequent high-precision true temperature inversion.

[0055] More specifically, in a concrete example of the present invention, the implementation process of radiation correction and background energy calculation follows an execution logic from component estimation to subtraction correction. First, the original observed radiation values ​​from the heatmap of the region of interest are read, and the real-time solar irradiance values ​​recorded in the environmental parameter data are retrieved. Combining the dynamic emissivity map generated in the previous steps, the surface reflectivity of each pixel is calculated based on the physical relationship that reflectivity equals 1 minus emissivity. Subsequently, using the incident geometry factor (i.e., the geometric relationship between the solar incidence angle and the surface normal vector) calculated in the previous steps, the reflected radiation component caused by direct solar radiation is estimated. A pixel-level subtraction operation is performed to subtract the solar reflection component and the scattering attenuation term corrected by the geometry factor from the original observed radiation values, generating a corrected radiation intensity map that has removed solar thermal interference.

[0056] Accordingly, based on the dynamic emissivity map and background radiation energy, a reverse thermodynamic inversion calculation is performed on the corrected radiation intensity map to obtain the true temperature distribution map. It should be noted that although the corrected radiation intensity map obtained after the previous processing has removed the high-frequency thermal interference from direct sunlight, according to Kirchhoff's thermal radiation law, the radiation value still contains long-wave infrared reflection components caused by the ambient background temperature. Furthermore, the complex dust accumulation on the surface of the bucket wheel excavator components leads to significant spatial differences in radiation efficiency at various points. Directly converting this radiation value into temperature would introduce nonlinear measurement errors. Therefore, the technical solution of this invention further uses the dynamic emissivity map and background radiation energy to perform a reverse thermodynamic inversion calculation on the corrected radiation intensity map to obtain the true temperature distribution map. This allows the construction of a reverse solution model based on physical conservation, removing the environmental reflection term from the total radiation energy and normalizing and compensating for the emission capability of each pixel. Through the above processing, the true surface temperature of the equipment, unaffected by ambient light and heat conditions and surface contamination levels, can be restored, ensuring that subsequent judgments of bearing overheating or roller wear are based on accurate thermodynamic truth values.

[0057] More specifically, in a particular example of the present invention, the thermodynamic inversion process follows an execution logic from energy stripping to temperature mapping. First, the calculated background radiation energy and the dynamic emissivity map corresponding to each image pixel are retrieved. For each pixel coordinate in the corrected radiation intensity map, based on the principle of energy superposition, the environmental reflection component is calculated using the product of the reflectivity (i.e., 1 minus emissivity) and the background radiation energy at that point, and subtracted from the corrected radiation intensity value, thereby separating the net radiant exitance belonging only to the object itself. Subsequently, based on the inverse operation of the fourth root of the Stefan-Boltzmann law, the net radiant exitance is divided by the product of the dynamic emissivity of that pixel and the Stefan-Boltzmann constant, and then the fourth root is taken to obtain the absolute thermodynamic temperature of that pixel.

[0058] Specifically, in step S5, multi-dimensional feature deep fusion and anomaly detection are performed on the real temperature distribution map and the visible light image of the region of interest to obtain a fused state vector containing the fault type and location coordinates. It should be noted that, given that a single thermodynamic parameter can only reflect the current temperature rise of the equipment but cannot independently reveal the deep physical mechanism causing overheating—for example, it is difficult to distinguish between dry friction heating caused by internal bearing wear and lubricating oil leakage heating caused by seal damage—while a simple visible light image can capture surface cracks or oil stains but cannot assess their potential thermal damage risk, this isolation of modal information leads to a lack of sufficient evidence chain support for fault diagnosis. Based on this, the technical solution of this invention further performs multi-dimensional feature deep fusion and anomaly detection on the real temperature distribution map and the visible light image of the region of interest to obtain a fused state vector containing the fault type and location coordinates. This is used to construct a joint decision-making mechanism based on a multi-dimensional feature space, extracting and cross-validating the temperature gradient distribution features of the thermal field and the physical defect texture features in the visible light image. Through the above processing, abstract temperature anomaly data can be accurately mapped to specific mechanical fault modes, thereby effectively eliminating the interference of occasional thermal noise and outputting a fusion diagnostic conclusion that includes a clear fault category determination and precise center coordinates.

[0059] Figure 6 This is a flowchart illustrating the multi-dimensional feature deep fusion and anomaly detection of a bucket wheel excavator equipment health monitoring method based on thermal imaging and visible light images according to an embodiment of the present invention, to obtain a fused state vector containing fault type and location coordinates. For example... Figure 6 As shown, step S5 includes: S51, performing global temperature rise statistics and heat flux density gradient calculation on the real temperature distribution map to construct a thermal feature descriptor containing the highest temperature value, temperature rise rate, and thermal gradient index; S52, performing physical defect texture recognition and hue saturation analysis on the visible light image of the region of interest to obtain a visual feature descriptor; S53, performing multimodal fault probability joint calculation on the thermal feature descriptor and the visual feature descriptor, and combining the highest temperature point location information extracted from the real temperature distribution map to generate a fused state vector.

[0060] In step S51, global temperature rise statistics and heat flux density gradient calculations are performed on the actual temperature distribution map to construct a thermal feature descriptor containing the maximum temperature value, temperature rise rate, and thermal gradient index. It should be noted that simple temperature numerical monitoring is insufficient to comprehensively characterize the operational health status of key components of the bucket wheel excavator. For example, a simple local high temperature may be a false alarm caused by environmental hotspots, while real mechanical faults (such as bearing wear or lack of lubrication) typically manifest as a thermal field with a specific center and diffusion trend, i.e., possessing a significant heat flux density gradient and a continuous temperature rise trend. Based on this, the technical solution of this invention further performs global temperature rise statistics and heat flux density gradient calculations on the actual temperature distribution map to construct a thermal feature descriptor containing the maximum temperature value, temperature rise rate, and thermal gradient index. This quantifies the morphological characteristics and dynamic evolution of the thermal field from a mathematical perspective, transforming discrete pixel temperatures into vector data describing the intensity and diffusion behavior of heat sources. Through the above processing, point noise and real fault heat sources with physical thermal conductivity characteristics can be effectively distinguished, providing physically interpretable high-dimensional feature inputs for subsequent multimodal fault judgment.

[0061] More specifically, in a concrete example of this invention, the thermal feature extraction process follows an execution logic from statistical analysis to field theory calculation. First, a global traversal search is performed on the real temperature distribution map to extract the maximum pixel value in the image matrix as the highest temperature value, and the difference between this value and the ambient background temperature is defined as the temperature rise rate. Simultaneously, historical temperature records are retrieved, and the slope of the temperature change relative to the previous moment is calculated to determine the temperature rise rate. Then, a field theory analysis method is introduced, using the Laplace operator to perform second-order differential convolution on the temperature matrix to calculate the heat flux density gradient of each pixel. This gradient physically characterizes the intensity of heat diffusion from the high-temperature center outwards, and is highly specific for identifying concentric circle heat distributions caused by internal heat sources. Finally, the calculated global highest temperature, temperature rise rate, and average thermal gradient feature values ​​of key regions are normalized and sequentially concatenated and encapsulated to generate a structured thermal feature descriptor.

[0062] In steps S52 and S53, physical defect texture identification and hue / saturation analysis are performed on the visible light image of the region of interest to obtain a visual feature descriptor. Subsequently, the thermal feature descriptor and the visual feature descriptor are used to jointly calculate the multimodal fault probability, and the location information of the highest temperature point extracted from the real temperature distribution map is combined to generate a fused state vector. It should be noted that, since a simple thermal characterization can only reflect the appearance of energy accumulation, it cannot independently distinguish whether the temperature rise is caused by internal mechanical friction, lubrication failure, or external foreign matter entanglement. While a simple visible light image can capture the physical damage texture of the surface, it cannot quantify the current thermodynamic hazard level. The semantic separation between the two leads to a lack of cross-validation of multidimensional evidence for fault diagnosis. Based on this, the technical solution of the present invention further performs physical defect texture recognition and hue saturation analysis on the visible light image of the region of interest to obtain a visual feature descriptor. Subsequently, multimodal fault probability joint calculation is performed on the thermal feature descriptor and the visual feature descriptor, and the location information of the highest temperature point extracted from the real temperature distribution map is combined to generate a fused state vector. This is used to construct a joint inference model of visual-thermal fusion, which logically associates and probabilistically weights the physical defect features of the surface with the internal thermodynamic anomaly features. Through the above processing, the confidence in the recognition of complex fault modes (such as bearing overheating due to oil seal failure) can be improved, thereby outputting a fused state vector containing the exact fault type, comprehensive risk probability, and precise spatial coordinates.

[0063] More specifically, in a concrete example of the present invention, the implementation process of multimodal feature fusion follows an execution logic from visual feature encoding to probabilistic inference fusion. First, the visible light image of the region of interest is input into a pre-set surface defect detection model (e.g., a finely tuned convolutional neural network). The model performs convolutional scanning on minute textures in the image to identify specific physical defects such as oil leaks, metal cracks, or foreign object entanglement, and extracts the confidence level and category label of the defect region. Simultaneously, the image is converted to the HSV color space, and the hue and saturation distribution of key regions are statistically analyzed to quantify the ablation and discoloration characteristics of the surface. The identification results are then encoded as visual feature descriptors. Subsequently, a Bayesian inference framework or a weighted decision tree model is introduced to normalize the thermal feature descriptors (including temperature rise rate and thermal gradient) generated in the previous steps into prior probabilities of thermal failure, and the visual feature descriptors are normalized into prior probabilities of physical damage. These are then jointly calculated based on a pre-set fault mechanism logic table. For example, when a significant temperature rise rate is detected accompanied by high-confidence oil stain features, the joint determination is that overheating is caused by seal failure.

[0064] Specifically, in step S6, based on the fused state vector, the reference visible light image is mapped and visualized to obtain an equipment health monitoring report. It should be noted that while the fused state vector generated in the preceding steps contains precise fault conclusions at a mathematical level, it is essentially abstract numerical data, difficult for on-site maintenance personnel to directly and intuitively read. Furthermore, a simple thermal radiation image lacks sufficient texture details to indicate specific damaged physical components (such as the exact bolt location), and a simple visible light image cannot present hidden temperature rise distributions, resulting in insufficient interpretability and on-site execution capability of the fault data. Therefore, the technical solution of this invention further maps and visualizes the reference visible light image based on the fused state vector to obtain an equipment health monitoring report. This performs a reverse projection from the data space to the visual space, overlaying the diagnosed thermodynamic anomalies and physical defect features onto a high-resolution real-world equipment image in an augmented reality manner. Through the above processing, an intuitive correspondence between fault information and physical entities can be achieved, generating a comprehensive diagnostic report containing a visual evidence chain, thereby effectively reducing the cognitive load of manual interpretation and shortening the fault location and maintenance response cycle.

[0065] More specifically, in a concrete example of the present invention, the implementation process of visualization rendering and report generation follows an execution logic from layer composition to semantic annotation to data encapsulation. First, the fused state vector is parsed, and the fault center coordinates recorded therein are extracted. The process involves several steps: first, determining the radius of influence (R), fault confidence level, and specific type code. Then, a transparent layer with the same size as the reference visible light image is generated. Only within a local area centered on the fault, pseudo-color thermal maps are generated using a reddish-brown gradation based on temperature. To preserve the texture details of the bottom device for easy identification, an alpha blending algorithm is used to weightedly overlay this thermal layer onto the original reference visible light image. Next, a graphics interface is used to draw rectangular bounding boxes and traction lines on the overlaid image, containing the fault name, core temperature reading, and warning level color, generating a semantically annotated diagnostic map. Finally, this diagnostic map is compressed and encoded, then merged and encapsulated with structured text containing timestamps, device IDs, and fault metadata to construct a standardized device health monitoring report. This report is then pushed to a cloud database via an industrial IoT interface to complete the closed loop.

[0066] In summary, the health monitoring method for bucket wheel excavators based on thermal imaging and visible light images according to embodiments of the present invention is explained. First, it utilizes the edge features of rigid structures to perform spatiotemporal reference alignment on the acquired multi-source heterogeneous images, eliminating field-of-view registration deviations caused by equipment vibration and rotation. Then, it uses semantic segmentation to lock key component regions, utilizes texture features in visible light images to estimate surface grayscale state to dynamically correct emissivity, and combines solar azimuth and solar altitude angles with surface normal vectors to accurately eliminate environmental thermal reflection components, thereby obtaining the true temperature distribution of components through physical inversion. Finally, it performs multi-dimensional decision-level fusion based on the true temperature field and surface physical defect features to generate a visualized monitoring report containing precise positioning coordinates, effectively shielding against false alarms caused by solar radiation and dust coverage, and achieving high-confidence diagnosis of equipment health status in complex outdoor environments.

[0067] Furthermore, a health monitoring system for bucket wheel excavators based on thermal imaging and visible light images is also provided.

[0068] Figure 7 This is a block diagram of a bucket wheel excavator equipment health monitoring system based on thermal imaging and visible light images, according to an embodiment of the present invention. Figure 7 As shown, the bucket wheel excavator equipment health monitoring system 100 based on thermal imaging and visible light images according to an embodiment of the present invention includes: a time synchronization and distortion correction module 110, used to perform time synchronization and distortion correction processing on acquired multi-source heterogeneous data to obtain a synchronized dataset, the multi-source heterogeneous data including the original visible light image stream, the original thermal radiation image stream, and environmental parameter data; a cross-modal space registration module 120, used to perform cross-modal space registration on the synchronized dataset to obtain a registered thermal image and a reference visible light image; and a visible light image and thermal image acquisition module 130, used to input the reference visible light image into a semantic segmentation network to obtain a component mask, and use the component mask to mask the registered thermal image and the reference visible light image. The system includes: an extraction / background removal operation to obtain a visible light image and a thermal image of the region of interest (ROI); a temperature correction module 140 to perform temperature correction on the solar azimuth angle, visible light image, and thermal image of the ROI from the environmental parameter data based on visual perception physical parameters to obtain a true temperature distribution map; a multi-dimensional feature deep fusion and anomaly detection module 150 to perform multi-dimensional feature deep fusion and anomaly detection on the true temperature distribution map and the visible light image of the ROI to obtain a fused state vector containing fault type and location coordinates; and an equipment health monitoring report acquisition module 160 to map and visualize anomaly information on a reference visible light image based on the fused state vector to obtain an equipment health monitoring report.

[0069] Furthermore, the visible light image and thermal image acquisition module 130 includes: The original classification probability map acquisition unit is used to input a reference visible light image into a preset lightweight semantic segmentation network to obtain the original classification probability map; The morphological hole filling and opening operation denoising unit is used to determine the pixel category label from the original classification probability map and perform morphological hole filling and opening operation denoising on the binary region corresponding to the key monitoring component set to obtain the component mask. The pixel-level Adama product operation unit is used to perform pixel-level Adama product operations on the reference visible light image and the registered thermal image based on the part mask to obtain the visible light image and the thermal image of the region of interest.

[0070] As described above, the bucket wheel excavator equipment health monitoring system 100 based on thermal imaging and visible light images according to embodiments of the present invention can be implemented in various types of computing devices or control units. For example, it can be a ruggedized industrial computer deployed in the electrical room of the bucket wheel excavator, or an embedded edge computing node installed on the cantilever slewing platform. In one possible implementation, the bucket wheel excavator equipment health monitoring system 100 based on thermal imaging and visible light images according to embodiments of the present invention can be integrated into the computing device as a software module and / or a hardware module. For example, the system 100 can be a resident data processing service in the operating system of the computing device, the software module being configured to perform spatiotemporal synchronization of multi-source heterogeneous image data, physical parameter inversion correction based on visual perception, and multi-dimensional feature deep fusion diagnosis, or it can be a dedicated intelligent monitoring algorithm program developed for the computing device. Of course, the system 100 can also be one of many hardware modules of the computing device or control unit, or it can be embedded in a field-programmable gate array circuit to accelerate image matrix operations and neural network inference in parallel, or it can be a multi-source image synchronous acquisition integrated circuit for a specific application.

[0071] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images, characterized in that, Includes the following steps: S1: Perform time-series synchronization and distortion correction on the acquired multi-source heterogeneous data to obtain a synchronized dataset. The multi-source heterogeneous data includes the original visible light image stream, the original thermal radiation image stream, and environmental parameter data. S2: Perform cross-modal spatial registration on the synchronized dataset to obtain the registered thermal image and the reference visible light image; S3: Input the reference visible light image into the semantic segmentation network to obtain the component mask, and use the component mask to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain the visible light map and thermal map of the region of interest; S4: Perform temperature correction on the solar azimuth and solar altitude angles, visible light map of region of interest and thermal map of region of interest in the environmental parameter data based on visual perception physical parameters to obtain the real temperature distribution map; S5: Perform multi-dimensional feature deep fusion and anomaly detection on the real temperature distribution map and the visible light map of the region of interest to obtain a fused state vector containing the fault type and location coordinates; S6: Based on the fused state vector, the abnormal information of the reference visible light image is mapped and visualized to obtain the device health monitoring report.

2. The method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to claim 1, characterized in that, Step S1 includes: The frame header parser is used to unpack and separate the raw sensor data stream received through the multi-threaded parallel interface, and the separated raw visible light image, raw thermal radiation image and environmental parameter data are stored to obtain a buffer queue data packet. Using the hardware timestamp of the original visible light image generated in the buffer queue data packet as the reference anchor point, a sliding window search based on a preset time tolerance threshold is performed on the original thermal radiation image and associated with the nearest neighbor environmental parameters to obtain time-matched data pairs. The preset camera intrinsic parameter matrix and distortion coefficients are retrieved, and geometric correction coordinates and pixel reconstruction are performed on the image data in the time-series matching data pair to obtain the synchronized dataset.

3. The method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to claim 1, characterized in that, Step S2 includes: Rigid structure contour extraction and binarized gradient calculation are performed on the reference visible light image and source thermal radiation image obtained from the synchronized dataset to obtain the visible light edge map and thermal radiation edge map; The homography matrix is ​​obtained by performing feature key point matching and mismatch point removal on the visible light edge map and the thermal radiation edge map. Based on the homography matrix, the source thermal radiation image is projected onto the reference visible light image coordinate system space by perspective transformation and bilinear interpolation resampling to obtain the registered thermal image and the reference visible light image.

4. The method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to claim 1, characterized in that, Step S3 includes: The reference visible light image is input into a pre-built lightweight semantic segmentation network to obtain the original classification probability map; Pixel category labels are determined from the original classification probability map, and morphological hole filling and opening operations are performed on the binary regions corresponding to the key monitoring component set to obtain the component mask. Based on the component mask, pixel-level Adama product operations with background removal are performed on the reference visible light image and the registered thermal image to obtain the visible light image and thermal image of the region of interest.

5. The method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to claim 4, characterized in that, The reference visible light image is input into a pre-built lightweight semantic segmentation network to obtain the original classification probability map, including: The reference visible light image is preprocessed by pixel normalization to obtain the processed image; The processed image is input into a pre-built lightweight semantic segmentation network to perform forward inference and posterior probability calculation based on hollow spatial pyramid pooling to obtain the original classification probability map.

6. The method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to claim 1, characterized in that, Step S4 includes: Based on the visible light map of the region of interest, determine the dynamic emissivity map; Construct a solar incident light vector based on environmental parameter data; The gradient operator is used to extract pixel-level surface normal vectors from the visible light map of the region of interest, and the incident geometry factor is calculated by combining the solar incident light vector and the surface normal vector. The radiation value correction calculation is performed on the heat map of the region of interest to remove the solar reflection component in order to obtain the corrected radiation intensity map; Convert the ambient background temperature in the environmental parameter data into background radiation energy; Based on the dynamic emissivity map and background radiation energy, the modified radiation intensity map is subjected to inverse thermodynamic inversion calculation to obtain the true temperature distribution map.

7. The method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to claim 1, characterized in that, Based on the visible light map of the region of interest, a dynamic emissivity map is determined, including: By using the visible light map of the region of interest within the part mask-defined area, the pixel brightness value and local texture variance features are used to calculate the dust accumulation confidence index, which characterizes the degree of surface contamination. Based on the ash accumulation confidence index, a weighted linear interpolation calculation is performed on the preset clean metal emissivity and coal dust emissivity to obtain a dynamic emissivity map.

8. The method for health monitoring of bucket wheel excavator equipment based on thermal imaging and visible light images according to claim 1, characterized in that, Step S5 includes: Global temperature rise statistics and heat flux density gradient calculations are performed on the real temperature distribution map to construct a thermal characteristic descriptor that includes the maximum temperature value, temperature rise rate and thermal gradient index. Physical defect texture identification and hue / saturation analysis are performed on the visible light image of the region of interest to obtain visual feature descriptors; Multimodal fault probability is jointly calculated using thermal and visual feature descriptors, and the location information of the highest temperature point extracted from the real temperature distribution map is combined to generate a fused state vector.

9. A health monitoring system for bucket wheel excavators based on thermal imaging and visible light images, characterized in that, include: The timing synchronization and distortion correction module is used to perform timing synchronization and distortion correction on the acquired multi-source heterogeneous data to obtain a synchronized dataset. The multi-source heterogeneous data includes the original visible light image stream, the original thermal radiation image stream, and environmental parameter data. The cross-modal space registration module is used to perform cross-modal space registration on the synchronized dataset to obtain the registered thermal image and the reference visible light image; The visible light image and thermal image acquisition module is used to input the reference visible light image into the semantic segmentation network to obtain the component mask, and use the component mask to perform mask extraction / background removal operations on the registered thermal image and the reference visible light image to obtain the visible light image and thermal image of the region of interest. The temperature correction module is used to perform temperature correction on the solar azimuth angle and solar altitude angle, the visible light map of the region of interest and the thermal map of the region of interest based on visual perception physical parameters in order to obtain the real temperature distribution map. The multidimensional feature deep fusion and anomaly detection module is used to perform multidimensional feature deep fusion and anomaly detection on the real temperature distribution map and the visible light map of the region of interest to obtain a fused state vector containing the fault type and location coordinates. The equipment health monitoring report acquisition module is used to map and visualize abnormal information from a reference visible light image based on a fused state vector to obtain an equipment health monitoring report.

10. The bucket wheel excavator equipment health monitoring system based on thermal imaging and visible light images according to claim 9, characterized in that, The visible light image and thermal image acquisition module includes: The original classification probability map acquisition unit is used to input a reference visible light image into a preset lightweight semantic segmentation network to obtain the original classification probability map; The morphological hole filling and opening operation denoising unit is used to determine the pixel category label from the original classification probability map and perform morphological hole filling and opening operation denoising on the binary region corresponding to the key monitoring component set to obtain the component mask. The pixel-level Adama product operation unit is used to perform pixel-level Adama product operations on the reference visible light image and the registered thermal image based on the part mask to obtain the visible light image and the thermal image of the region of interest.

Images