Coal ship loading equipment fault identification method, device, medium and product

Abstract

This application discloses a method, device, medium, and product for fault identification of coal loading equipment, relating to the field of equipment fault identification. The method includes: acquiring RGB and infrared images of a predetermined part of a target equipment, wherein the target equipment includes coal loading equipment, and the predetermined part includes parts affecting the operating state of the target equipment; extracting texture features from the RGB images and infrared temperature features from the infrared images; fusing the texture features and the infrared temperature features to obtain a comprehensive representation vector; and sending the comprehensive representation vector to a pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type. This application reduces the false negative rate of fault identification in coal loading equipment.

Description

Technical Field

[0001] This application relates to the field of equipment fault identification technology, and in particular to a method, equipment, medium and product for fault identification of coal loading equipment. Background Technology

[0002] As a critical node in the coal transportation chain, the stable operation of core equipment (such as belt conveyors, ship loaders, and stacker-reclaimers) at coal loading terminals directly determines port throughput efficiency and operational safety. With the advancement of intelligent port transformation, equipment fault identification technology is evolving from traditional manual inspection to automation and intelligence. Current fault identification methods are as follows: The device's outer surface is captured using an RGB image by a camera; this RGB image is then used to analyze whether the device has any malfunctions. Problems such as cracks or wear on the device can be detected in the RGB image.

[0003] As can be seen from the above, the existing technology has at least the following technical problems: relying on RGB images can only capture visible faults, such as conveyor belt cracks and gearbox housing wear, but it is powerless to detect other faults, resulting in a high rate of missed fault identification. Summary of the Invention

[0004] The purpose of this application is to provide a method, equipment, medium, and product for identifying faults in coal loading equipment, in order to solve the problem of high false negative rate described in the background art.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for identifying faults in coal loading equipment, including: Acquire RGB and infrared images of predetermined parts of a target device, wherein the target device includes coal loading equipment and the predetermined parts include parts that affect the operating status of the target device; Extract the texture features of the RGB image and the infrared temperature features of the infrared image; By fusing the texture features and the infrared temperature features, a comprehensive representation vector is obtained; The integrated representation vector is sent to a pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type.

[0006] Optionally, if the RGB image is a background-separated RGB image and the infrared image is a background-separated infrared image, then the RGB image and the infrared image are generated by the following method: Acquire initial RGB and initial infrared images of a predetermined portion of the target device; The initial RGB image and the initial infrared image are sent to the pre-trained improved U-net model so that the pre-trained improved U-net model outputs the RGB image and infrared image after removing dust noise, which are denoised as the denoised RGB image and the denoised infrared image, respectively. The denoised RGB image and the denoised infrared image are input into a pre-trained generative adversarial network (GAN) model. The model performs motion blur restoration and illumination enhancement on the input images and outputs the restored images, which are denoted as the restored RGB image and the restored infrared image, respectively. A background separation operation is performed on the repaired RGB image and the repaired infrared image to obtain the background-separated RGB image and the background-separated infrared image.

[0007] Optionally, performing a background separation operation on the repaired RGB image and the repaired infrared image to obtain the background-separated RGB image and the background-separated infrared image includes: The repaired RGB image and the repaired infrared image are sent to the adaptive histogram equalization model so that the adaptive histogram equalization model outputs the contrast-adjusted RGB image and infrared image, which are denoted as the contrast RGB image and the contrast infrared image, respectively. The contrast RGB image and the contrast infrared image are respectively input into the retinal cortex theoretical model so that the retinal cortex theoretical model outputs the separated illumination component map and reflection component map, which are respectively denoted as RGB illumination component, RGB reflection component, infrared illumination component and infrared reflection component. The RGB illumination component and the infrared illumination component are subjected to smoothing filtering, and the processed images are denoted as the RGB illumination smoothing component and the infrared illumination smoothing component; the RGB reflection component and the infrared reflection component are subjected to feature enhancement processing, and the processed images are denoted as the RGB reflection feature component and the infrared reflection feature component. The RGB illumination smoothing component and the RGB reflection feature component are synthesized to obtain the background-separated RGB image; the infrared reflection feature component and the infrared illumination smoothing component are synthesized to obtain the background-separated infrared image.

[0008] Optionally, extracting the texture features of the RGB image and the infrared temperature features of the infrared image includes: The RGB image and the infrared image are sent to a pre-trained dual-branch convolutional neural network, so that the dual-branch convolutional neural network outputs the texture features of the RGB image and the infrared temperature features of the infrared image.

[0009] Optionally, fusing the texture features and the infrared temperature features to obtain a comprehensive representation vector includes: sending the texture features and the infrared temperature features to a cross-modal attention mechanism, so that the cross-modal attention mechanism fuses the texture features and the infrared temperature features to obtain the comprehensive representation vector.

[0010] Optionally, the output of the pre-trained improved YOLOv8 model further includes image coordinate data, device physical coordinate data, and confidence score. After sending the comprehensive representation vector to the pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type, the method further includes: sending the fault type, image coordinate data, device physical coordinate data, and confidence score to a knowledge graph-based fault inference engine so that the fault inference engine outputs the fault level.

[0011] Optionally, after sending the integrated representation vector to the pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type, the method further includes: Acquire the historical and real-time operating data of the target device; A multimodal time-series input sequence is constructed, which includes an RGB image sequence and an infrared image sequence with a continuous predetermined duration window, as well as the historical operating data and the real-time operating data of the target device corresponding to each image timestamp; The multimodal time-series input sequence is input into a pre-trained time-series analysis model so that the time-series analysis model can predict and output the remaining lifespan of the target device.

[0012] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any one of the first aspects above.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0014] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: The coal loading equipment fault identification method provided in this application includes: acquiring RGB and infrared images of a predetermined part of the target equipment, wherein the target equipment includes coal loading equipment, and the predetermined part includes parts affecting the operating state of the target equipment; extracting texture features from the RGB images and infrared temperature features from the infrared images; fusing the texture features and infrared temperature features to obtain a comprehensive representation vector; and sending the comprehensive representation vector to a pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type. As can be seen from the specific process of this method, this application not only acquires the RGB image of the equipment but also the infrared image of a predetermined part of the equipment, thereby determining whether the equipment has a fault and the fault type based on both the RGB and infrared images. In contrast, existing technologies only determine whether the equipment has a fault based on the RGB image, leading to inaccurate judgments. Therefore, this application improves the accuracy of equipment fault judgment compared to existing technologies. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application environment diagram of the coal loading equipment fault identification method in one embodiment of this application; Figure 2 A flowchart illustrating a method for identifying faults in coal loading equipment, provided as an embodiment of this application; Figure 3 This is a schematic flowchart illustrating the generation of a background-separated RGB image and a background-separated infrared image according to an embodiment of this application. Figure 4 A flowchart illustrating a method for generating a background-separated RGB image and a background-separated infrared image, provided in another embodiment of this application; Figure 5 A flowchart illustrating a method for determining the remaining useful life of equipment according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the contents of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] This application provides an environment in which a coal loading equipment fault identification method can be applied. See [link to relevant documentation]. Figure 1 The application environment includes: high-definition industrial cameras, infrared thermal imaging cameras, terminals, and servers.

[0021] The data storage system stores the data that the server needs to process. This system can be set up independently, integrated into the server, or located in the cloud or on other servers. Furthermore, the data storage system stores relevant data required for executing the coal loading equipment fault identification method, such as RGB and infrared images. Of course, the data storage system also stores intermediate data generated during the execution of the coal loading equipment fault identification method, so that it can be retrieved promptly when needed.

[0022] The terminal communicates with the server via a network. The terminal can send access requests to the server, such as requesting to display the current fault identification results. Upon receiving the request, the server can then send the fault identification results back to the terminal.

[0023] The terminals can be, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Servers can be implemented using independent servers, server clusters composed of multiple servers, or cloud servers.

[0024] In one exemplary embodiment, see Figure 2 As shown, a method for identifying faults in coal loading equipment is provided. This method is executed by a computer device. In this embodiment, the method is applied to... Figure 1 Taking the server in the example, the following steps 101 to 104 are used as an example: Step 101: Obtain RGB and infrared images of a predetermined part of the target equipment, wherein the target equipment includes coal loading equipment and the predetermined part includes parts that affect the operating status of the target equipment. Coal loading equipment includes any equipment related to coal loading, such as belt conveyors, ship loaders, stacker-reclaimers, etc.

[0025] The predetermined part can be a key part that affects the operating status of the target equipment. For example, the conveyor belt of a belt conveyor, the gearbox housing, the equipment bearing or bearing inner ring, the gear meshing part, etc.

[0026] Step 102: Extract the texture features of the RGB image and extract the infrared temperature features of the infrared image.

[0027] Step 103: Fuse the texture features and the infrared temperature features to obtain a comprehensive representation vector; Step 104: Send the integrated representation vector to the pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type.

[0028] This improved YOLOv8 model enhances its adaptability to irregular device shapes by introducing deformable convolutions into its backbone and employing a Bidirectional Feature Pyramid Network (BiFPN) structure in the neck to improve the efficiency of multi-scale feature fusion. The optimized model can perform high-precision fault identification while meeting real-time requirements (e.g., inference speed ≥ 25fps), thereby outputting fault type labels.

[0029] In the field of equipment fault identification at coal loading terminals, existing technologies generally suffer from the core drawback of "limited single-modal image acquisition and incomplete equipment status perception." This deficiency directly restricts the completeness of fault identification and early warning capabilities. Most AI image recognition solutions rely solely on single RGB image acquisition, which can only capture visible faults such as conveyor belt cracks and gearbox housing wear. They cannot detect internal temperature changes such as overheating of bearing inner rings and temperature rise at gear meshing points. These temperature anomalies are early signs of latent faults such as lubrication failure and component fatigue, which can easily lead to missed fault detection, resulting in downtime or safety accidents. Furthermore, although some solutions use independent temperature sensors to monitor temperature, the sensors and image acquisition systems are deployed separately, and there is no mechanism for synchronous acquisition and correlation analysis of the two types of data. When the equipment "appears normal but the internal temperature exceeds the standard," the RGB image shows no abnormalities, and the temperature data is difficult to correspond to specific parts of the equipment, resulting in a latent fault missed rate of over 30%.

[0030] Therefore, the aforementioned deficiencies prevent existing solutions from forming a complete perception dimension of equipment status, making it difficult to meet the actual needs of coal terminals for early warning and accurate location of faults. Therefore, this application proposes a fault identification method for coal loading terminal equipment based on AI image recognition, which solves the aforementioned problems by simultaneously acquiring RGB and infrared images of the equipment, thereby reducing the false negative rate of equipment fault identification and improving the accuracy of equipment fault identification.

[0031] Optionally, see Figure 3 In another exemplary embodiment of this application, the RGB image is a background-separated RGB image, and the infrared image is a background-separated infrared image. The RGB image and infrared image in step 101 are generated through the following steps 201 to 204: Step 201: Obtain the initial RGB image and initial infrared image of the predetermined part of the target device; The initial RGB image was captured by a high-definition industrial camera. After capture, the image was sent to a server, which stored the initial RGB image in a database for easy retrieval when needed.

[0032] The initial infrared image is captured by an infrared thermal imaging camera. After capturing the image, the camera sends it to a server, which stores it in a database for easy retrieval when needed.

[0033] Specifically, the high-definition industrial camera is equipped with an explosion-proof and dustproof protective cover and an adjustable color temperature supplementary light device to suppress dust interference and compensate for insufficient light, thereby improving the imaging quality; the infrared thermal imaging camera is equipped with an explosion-proof and dustproof protective cover to reduce the impact of dust on its imaging quality.

[0034] The explosion-proof and dustproof cover has a protection level of not less than IP65, and its built-in automatic dust removal fan starts periodically at intervals of 0.5-2 hours.

[0035] The adjustable color temperature supplementary lighting device has a color temperature adjustment range of 2000K-7000K and a brightness adjustment range of 50-1500lm.

[0036] Furthermore, the high-definition industrial camera and the infrared thermal imaging camera are controlled to acquire initial RGB and initial infrared images of the target device at the same time. These synchronously acquired images provide spatiotemporally aligned input data for the subsequently improved U-Net model, helping to enhance the accuracy of feature fusion and fault identification.

[0037] The high-definition industrial camera is set to a resolution of no less than 10 million pixels and a frame rate of no less than 20fps; the infrared thermal imaging camera is set to a temperature detection range of -20℃ to 500℃ and an image resolution of no less than 320×240 pixels.

[0038] In addition, for how to achieve synchronous acquisition by high-definition industrial cameras and infrared thermal imaging cameras, please refer to the existing relevant content on synchronous or same-frequency data acquisition, which will not be detailed here.

[0039] Step 202: Input the initial RGB image and the initial infrared image into the pre-trained improved U-net model, so that the pre-trained improved U-net model outputs the RGB image and infrared image after removing dust noise, which are denoised as the denoised RGB image and the denoised infrared image, respectively. The pre-trained improved U-net model performs independent and parallel dust and noise removal processing on the initial RGB image and the initial infrared image.

[0040] The U-net model includes an encoder module and a decoder module. The encoder module loads the attention mechanism parameters, and the decoder module loads the residual connection parameters. It also performs dust and noise removal operations on the initial RGB image and the initial infrared image, respectively.

[0041] Specifically, the improved U-Net model is built based on the AttentionU-Net architecture, and its core purpose is to solve the problems of image blurring and detail occlusion caused by high dust coverage in coal terminal environments. The specific settings of the encoder and decoder of the AttentionU-Net architecture are as follows: The encoder employs a multi-level downsampling structure to extract image features at different scales. Crucially, a spatial attention module is embedded at specific levels of the encoder. This module automatically analyzes the importance of each region in the feature map and dynamically generates a spatial attention weight map by fusing global and local statistical information (e.g., average pooling and max pooling features). This weight map is specifically designed to enhance the feature response of dust-covered areas in the image, thereby guiding the model to accurately locate and process noise during denoising, while avoiding mis-suppression of critical details of the equipment (such as cracks and wear).

[0042] The core function of this spatial attention module is to dynamically generate a spatial attention weight map. This weight map can automatically identify and quantify the importance of different spatial locations in the feature map, especially for dust-covered areas. By applying this weight map, the model can accurately enhance feature responses related to dust noise during feature propagation and effectively suppress irrelevant background interference. This mechanism ensures that while removing dust noise, key details of the equipment (such as cracks and wear) are preserved to the greatest extent possible, avoiding indiscriminate attenuation.

[0043] The decoder, responsible for feature reconstruction and image restoration, is designed to fuse multi-scale contextual information and promote the effective utilization of deep features. The decoder gradually restores the spatial resolution of the feature map through upsampling operations and uses skip connections to fuse shallow detail features from the corresponding encoder layer with deep semantic features. To further optimize information flow and detail preservation, the decoder introduces a residual learning structure into the feature fusion path. This composite design combining skip connections and residual connections effectively alleviates gradient problems and promotes efficient reuse of features across different layers. This results in a more complete and accurate reconstruction and preservation of key equipment details such as bearing cracks and gear wear in the final output image, preventing these details from being lost or blurred during denoising.

[0044] Step 203: Input the denoised RGB image and the denoised infrared image into the pre-trained Generative Adversarial Network (GAN) model. The GAN model performs independent and parallel motion blur inpainting and illumination enhancement processing on the two images and outputs the repaired images, which are denoted as the repaired RGB image and the repaired infrared image, respectively.

[0045] The pre-trained GAN model performs forward propagation computation on the input image to complete the restoration operations for both images. The model undergoes no fewer than 800 iterations during the training phase.

[0046] The illumination enhancement process primarily targets low-light areas in the image caused by insufficient ambient light. It uses algorithms to improve the brightness and contrast of these areas in order to restore obscured details.

[0047] The specific algorithms of GAN include the following: Specifically, the GAN model is constructed and optimized based on the ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) architecture to adapt to motion blur caused by vibrations of coal loading equipment and complex lighting conditions resulting from day-night cycles and weather changes. The model includes a generator and a discriminator, as detailed below: The generator is built upon a backbone network containing residual dense blocks. This structure promotes multi-scale feature fusion through dense connections and is specifically designed for restoring motion blur in images. The generator front end integrates an illumination attention mechanism, which can adaptively adjust the restoration intensity for different regions according to the image brightness distribution. This effectively avoids overexposure of highlights and loss of shadow details while improving overall brightness and contrast.

[0048] Discriminator: Employing a local receptive field-based discriminative architecture (such as PatchGAN), the generator's training is guided by evaluating the authenticity of local regions in the image, thereby ensuring the naturalness and realism of the restored image in terms of detail. The model's training incorporates multi-objective loss functions, including those based on the VGG19 network.

[0049] The perceptual loss, calculated from layer features, ensures the consistency of the restored image with the real image at a high-level semantic level; while the adversarial loss drives the generator to output results that are closer to the true distribution. Together, they improve image sharpness while maintaining visual naturalness.

[0050] Step 204: Perform a background separation operation on the repaired RGB image and the repaired infrared image to obtain the background-separated RGB image and the background-separated infrared image.

[0051] Of course, the RGB image and infrared image in step 101 can also be the initial RGB image and initial infrared image mentioned above, but this will affect the effect of fault identification.

[0052] Furthermore, the models, algorithms, and methods involved in steps 202, 203, and 204 can be installed in edge computing nodes.

[0053] The edge computing nodes adopt an embedded computing platform with a CPU core count of no less than 4 cores and a GPU memory capacity of no less than 4GB. The cloud server used for global model optimization training has a CPU core count of no less than 16 cores, a memory capacity of no less than 32GB, and a hard disk storage capacity of no less than 1TB.

[0054] Optionally, see Figure 4 In another exemplary embodiment of this application, step 204 includes the following steps 301 to 304: Step 301: Send the repaired RGB image and the repaired infrared image to the adaptive histogram equalization model so that the adaptive histogram equalization model outputs the contrast-adjusted RGB image and infrared image, which are denoted as the contrast RGB image and the contrast infrared image, respectively. Step 302: Input the contrast RGB image and the contrast infrared image into the retinal cortex theoretical model, so that the retinal cortex theoretical model outputs the separated illumination component map and reflection component map, which are respectively denoted as RGB illumination component, RGB reflection component, infrared illumination component and infrared reflection component.

[0055] Among them, the retinal cortex theoretical model, also known as the Retinex model, can decompose each input image into two independent components, namely, the illumination component that represents the distribution of ambient light and the reflection component that represents the reflection characteristics of the object itself.

[0056] Step 303: Perform smoothing filtering on the RGB illumination component and the infrared illumination component, and record the processed images as the RGB illumination smoothing component and the infrared illumination smoothing component; perform feature enhancement processing on the RGB reflection component and the infrared reflection component, and record the processed images as the RGB reflection feature component and the infrared reflection feature component. Step 304: Perform synthesis processing on the RGB illumination smoothing component and the RGB reflection feature component to obtain the background-separated RGB image; perform synthesis processing on the infrared reflection feature component and the infrared illumination smoothing component to obtain the background-separated infrared image.

[0057] Steps 303 and 304 specifically involve: applying a smoothing filter (such as a Gaussian filter) to the illumination components of the RGB and infrared images to suppress background noise caused by uneven illumination; simultaneously, performing feature enhancement processing (such as sharpening) on ​​the reflection components of the RGB and infrared images to highlight the texture and temperature details of the device. Finally, the processed illumination and reflection components are resynthesized according to Retinex theory to generate a background-separated RGB image and a background-separated infrared image with optimized quality.

[0058] Optionally, in another exemplary embodiment of this application, step 102 is implemented via step 401: Step 401: Send the RGB image and the infrared image to a pre-trained dual-branch convolutional neural network so that the dual-branch convolutional neural network outputs the texture features of the RGB image and the infrared temperature features of the infrared image.

[0059] The texture feature extraction process: The first branch of the convolutional neural network loads extraction parameters for extracting spatial texture features of RGB images; it receives the input RGB image and performs texture feature extraction operation on the RGB image based on the spatial texture feature extraction parameters, thereby obtaining the texture features of the RGB image.

[0060] The process of extracting infrared temperature features: The second branch of the convolutional neural network loads parameters for extracting temperature features from infrared images; it receives the input infrared image and extracts the temperature features of the infrared image based on the temperature feature extraction parameters.

[0061] Furthermore, the first branch of the dual-branch convolutional neural network is based on a deep residual network architecture and incorporates a channel attention mechanism to enhance the extraction capability of key texture features such as conveyor belt cracks and gear wear. The second branch adopts a lightweight network architecture to adapt to edge computing environments and incorporates an attention mechanism that can dynamically focus based on a preset temperature threshold, specifically designed to improve sensitivity to temperature anomalies such as bearing overheating. The features extracted by the two branches are aligned through a dimension transformation layer to facilitate subsequent cross-modal fusion.

[0062] Optionally, in another exemplary embodiment of this application, step 103 includes step 501: sending the texture features and the infrared temperature features to a cross-modal attention mechanism, so that the cross-modal attention mechanism fuses the texture features and the infrared temperature features to obtain the comprehensive representation vector.

[0063] When using a cross-modal attention mechanism, it is necessary to set an attention weight coefficient. In this application, the attention weight coefficient ranges from 0.3 to 0.7.

[0064] This application employs the CMA cross-modal attention algorithm to address the issue of feature separation between RGB and infrared features: Input: RGB features Infrared features Among them, RGB features can be texture features, and infrared features can be infrared humidity features.

[0065] Attention calculation: Calculate the inter-modal similarity matrix (512×512 dimensional), the weight matrix is ​​obtained by Softmax normalization. ; Fusion: ( (As an identity matrix), with weight coefficients of 0.3 to 0.7 to balance the importance of the two modes, a 512-dimensional comprehensive feature vector is generated.

[0066] Optionally, in another exemplary embodiment of this application, the output of the pre-trained improved YOLOv8 model further includes image coordinate data, device physical coordinate data, and confidence score. Then, after step 104, the method further includes step 601: sending the fault type, image coordinate data, device physical coordinate data, and confidence score to a knowledge graph-based fault inference engine so that the fault inference engine outputs the fault level.

[0067] The image coordinate data refers to the location information of the fault area identified by the improved YOLOv8 model in the input image, which is usually represented in the form of bounding box coordinates, such as (x_min, y_min, x_max, y_max).

[0068] The physical coordinate data of the equipment refers to the specific location of the fault point on the real-world equipment, calculated based on the image coordinate data, camera calibration parameters, and the three-dimensional model of the equipment through coordinate mapping relationships, such as "No. 1 belt conveyor drive roller bearing seat".

[0069] The confidence level is a value within the range of [0, 1]. The closer the value is to 1, the higher the confidence level of the improved YOLOv8 model in identifying the fault type and location.

[0070] The knowledge graph-based fault reasoning engine integrates loaded device fault tree structure data, expert rule base, and Bayesian probability model to perform comprehensive logical reasoning and probabilistic verification on the input fault type, image / device physical coordinates, and confidence level. Based on the matching degree between the reasoning rules and historical data, the engine outputs quantified fault level labels, such as Level 1 (urgent), Level 2 (important), or Level 3 (general).

[0071] In addition, after determining the fault level, the following operations can be performed: At the edge, the fault level and type are transmitted to the equipment management system and operation and maintenance scheduling system via mobile networks (e.g., 5G networks). Further, the equipment management system and operation and maintenance scheduling system determine the corresponding handling method based on the fault level and push it to the terminal devices that can be viewed by operation and maintenance personnel. For example, a level 1 fault triggers a shutdown and pushes an emergency task, while levels 2 and 3 push regular tasks. Operation and maintenance personnel accept the task through an app on their terminal device and process it according to the push notification. After the task is completed, relevant AI performs a re-inspection to confirm that the fault has been eliminated and archived.

[0072] The device management system is deployed on a dedicated server with a configuration of at least an Intel Xeon Gold 6330 CPU and 64GB of memory; the operation and maintenance scheduling system is deployed on a cloud server platform with a configuration of at least a 4-core CPU and 8GB of memory.

[0073] The edge node refers to the edge computing node deployed at or near the coal loading equipment site, which is responsible for image acquisition, dynamic adaptive preprocessing, and AI model inference tasks.

[0074] The equipment management system executes a preset differentiated response strategy based on the received fault level label: if the fault level is level one, it automatically sends an equipment shutdown control signal to the central control system and simultaneously sends the highest priority emergency repair task instruction to the operation and maintenance scheduling system; if the fault level is level two or three, it sends a normal priority repair task instruction to the operation and maintenance scheduling system.

[0075] After receiving the task instruction, the operation and maintenance scheduling system calls the personnel matching program, performs the task allocation operation based on the location data and skill tag data of the operation and maintenance personnel, and sends the task data to the assigned personnel through the mobile APP; Maintenance personnel scan the device's QR code using a mobile app to trigger a check-in confirmation process; during the operation, they upload images and video data of the processing process through the app; after the operation is completed, they submit a processing completion application through the app. After receiving and processing the application, the equipment management system sends a secondary imaging command to the acquisition unit; inputs the secondary imaging data into the improved YOLOv8 model and performs a re-inspection operation; when the confidence level of the re-inspection output is <0.1, the fault data and processing data are recorded in the equipment file database; when the confidence level of the re-inspection output is ≥0.1, a re-dispatch command is sent to the operation and maintenance scheduling system.

[0076] Optionally, see Figure 5 In another exemplary embodiment of this application, after step 104, the method further includes steps 701 to 702: Step 701: Construct a multimodal time-series input sequence, which includes an RGB image sequence and an infrared image sequence with a continuous predetermined duration window, as well as the historical operating data and real-time operating data of the target device corresponding to each image timestamp; Each image includes the RGB image sequence and the infrared image sequence.

[0077] Historical operating data refers to the runtime of the target device, such as cumulative runtime, while real-time operating data refers to the current load information.

[0078] The cumulative runtime refers to the total working time of the target equipment since it was put into operation or the last major overhaul was completed.

[0079] The load information includes physical quantities that reflect the operating status of the equipment, such as the voltage and current of the drive motor, and the rotational speed of the transmission components.

[0080] Step 702: Input the multimodal time series input sequence into the pre-trained time series analysis model so that the time series analysis model can predict and output the remaining service life of the target device.

[0081] This time-series analysis model can analyze the degradation trend of the target device's state to predict and output the remaining service life of the target device.

[0082] The length of the continuous time window is preset according to the equipment type and failure mode, and is usually between 6 and 24 hours. For example, for monitoring bearing fatigue, a continuous 12-hour image and operation data sequence can be selected as the model input.

[0083] In this application, the time-series analysis model employs a Long Short-Term Memory (LSTM) network combined with a temporal attention algorithm to predict the remaining useful life (RUL) of the device. Network structure of the temporal analysis model: A multi-layer (2-layer LSTM, 128→64 hidden units) long short-term memory network is adopted, and a temporal attention module is integrated at its output.

[0084] Furthermore, image features and equipment operating parameters (voltage, current, speed) from 6-24 hours are first input into the LSTM, and the output of the LSTM is denoted as... Then The input is fed into the temporal attention algorithm, which outputs RUL.

[0085] Furthermore, the weights at each time step can be calculated internally within the temporal attention algorithm using the following formula. : in, For LSTM output; Used to represent the weight of each time step, highlighting the characteristics of key stages of fault development, representing the quantitative value of the "attention level" of the temporal attention algorithm at time t, or can be regarded as the importance of that time to the prediction of remaining useful life, and the sum of all weights is 1; v T : is a row vector used as part of the temporal attention algorithm to map features after nonlinear transformation to a scalar score; tanh is the hyperbolic tangent activation function, which performs a non-linear transformation on the input, compressing the input value to between -1 and 1, thereby increasing the non-linear expressive power of the model. W: is a weight matrix used to weight the input features. Perform a linear transformation.

[0086] b: This is the bias vector, which is added to the result of the linear transformation to provide some flexibility to the model and help it better fit the data. i: Time step index, representing the i-th time step.

[0087] M: Total number of time steps, i.e., the length of the input sequence (e.g., the number of data points within 6-24 hours).

[0088] In this application, i is used to iterate through all time points, and M represents the length of the historical window that the model focuses on, which corresponds to the "continuous 6-24 hours" input window in the application.

[0089] s i This is the raw attention score calculated for all input elements i, where the value of i ranges from 1 to M.

[0090] Furthermore, the dual-branch convolutional neural network in step 401 and the improved YOLOv8 model in step 104 are optimized using the following method: First, small sample data collection: Obtain a limited number of initial fault image samples from the target device to form a small sample training set. For example, for some rare fault types, the initial sample size may be less than 50 images.

[0091] Then, data augmentation based on GAN is performed: the small sample training set is input into a pre-trained Generative Adversarial Network (GAN) model. This GAN model learns the distribution characteristics of real fault images and generates a large number of visually realistic synthetic fault images, thereby constructing a sufficiently large and reliable dataset of augmented fault images.

[0092] Secondly, model training and updating: using the enhanced fault image dataset, the dual-branch convolutional neural network and the improved YOLOv8 model are trained or incrementally updated.

[0093] Secondly, the online incremental learning mechanism continuously collects newly occurring and confirmed real fault images after the system is deployed and running. When the cumulative number of new samples for any fault category reaches a preset threshold (e.g., 50 images), the incremental learning process is automatically triggered. The new and old samples are mixed and the parameters of the dual-branch convolutional neural network and the improved YOLOv8 model are fine-tuned, enabling the model to continuously adapt to new fault modes and achieve performance evolution.

[0094] Furthermore, visualization and data integration can be achieved in the above process: the human-machine interface displays fault location in a 3D digital twin and generates reports with maintenance suggestions. Through HTTPS, it connects to the enterprise system to realize remote monitoring, tracing, and cloud model optimization of fault data.

[0095] The equipment model used in the three-dimensional digital twin display has an accuracy of no less than millimeters, and the model update frequency is set to once every 1 to 5 minutes; the structured identification report includes information on the time of failure, the location of failure, the personnel handling the failure, the processing time, and the spare parts model, and the report generation format is set to PDF or Excel.

[0096] In addition, in this application, the storage period for the device file database is set to be no less than 3 years, the data backup frequency is set to 1-3 times per day, and the backup storage media adopts both local hard disk and cloud storage methods.

[0097] Furthermore, this application takes a coastal coal loading terminal as an application scenario, and elaborates on the specific implementation process of this application for the fault identification requirements of 3 DTⅡ type belt conveyors and 2 ZJ1400 type ship loaders at the terminal. All equipment and software used are existing mature products to ensure that the process can be implemented.

[0098] I. Implementation Preparation 1.1 Hardware Equipment Selection and Deployment Multimodal acquisition unit deployment: One multimodal acquisition unit is deployed at each of the key components of the belt conveyor, such as the roller bearing, conveyor belt joint, gearbox of the reducer, and the slewing bearing and piston rod of the luffing cylinder of the ship loader. Each unit includes: a high-definition industrial camera (12-megapixel resolution, 25fps frame rate, supporting dynamic exposure), an infrared thermal imaging camera (temperature detection range -10℃ to 450℃, image resolution 384×288 pixels), an IP66-rated explosion-proof and dustproof protective cover (with a built-in automatic dust removal fan that starts once every hour), and an adjustable color temperature supplementary lighting device (color temperature adjustment range 2200K-6800K, brightness adjustment range 80-1200lm). The acquisition units are fixed by brackets to ensure that the lens is directly facing the monitoring part and that the acquisition range of adjacent units does not overlap.

[0099] Deployment of computing and transmission equipment: Edge computing nodes (using the NVIDIA Jetson AGX Xavier embedded platform, 4-core CPU, 8GB GPU memory) are deployed next to the terminal control room; 5G industrial gateways (supporting gigabit Ethernet interfaces) are deployed to enable network connections between the edge computing nodes and the terminal equipment management system and operation and maintenance scheduling system; the equipment management system is deployed on a server with an Intel Xeon Gold 6230 CPU and 64GB of memory, and the operation and maintenance scheduling system is deployed on an Alibaba Cloud server with a 4-core CPU and 8GB of memory.

[0100] 1.2 Software and Model Preparation Software environment setup: Edge computing nodes are equipped with Python 3.7 and PyTorch 1.12 deep learning frameworks, and dynamic adaptive preprocessing module, multi-dimensional feature fusion and AI inference module are deployed; the equipment management system is developed using Java SpringBoot framework, the operation and maintenance scheduling system is developed using Vue.js + MySQL architecture, and the mobile APP supports Android and iOS systems; the 3D digital twin platform is developed based on Unity 3D and imports the 3D model of the dock equipment.

[0101] Model training and deployment: Collect images of dock equipment failures (including 12 types of failures such as conveyor belt cracks and bearing overheating), generate virtual samples using GAN, and build a library of 12,000 labeled samples; train an improved U-Net denoising model (1,200 iterations), an improved YOLOv8 recognition model (50 epochs), and a dual-branch convolutional neural network (80 epochs), and deploy the trained models to edge computing nodes; import dock equipment fault tree data (including 300+ fault nodes) and expert rule base (including 150+ maintenance rules) into the knowledge graph inference engine.

[0102] II. Specific Implementation Steps S1: Multimodal image acquisition The multimodal acquisition unit is activated, with a high-definition industrial camera and an infrared thermal imaging camera simultaneously acquiring images of the equipment at a frequency of one frame every 2 seconds. The explosion-proof and dustproof cover's automatic dust removal fan starts every hour, running for one minute each time to remove dust from the lens surface. The adjustable color temperature supplementary lighting device acquires ambient light intensity through the camera's built-in light sensor, automatically activating when the light intensity is <600 lux, adjusting the color temperature and brightness according to the light intensity (e.g., in cloudy or rainy weather, the color temperature is adjusted to 3000K and the brightness to 800lm) to ensure consistent image clarity. The acquired RGB and infrared images are transmitted to the edge computing node via industrial Ethernet. S2: Dynamic Adaptive Preprocessing S2.1 Dust and Noise Removal: The edge computing node inputs the acquired image into the improved U-Net model. The model encoder locates the dust-covered area through the attention mechanism, and the decoder retains the detailed features of the device through residual connections. The dust and noise removal operation is performed, and the signal-to-noise ratio of the processed image is improved to more than 35dB.

[0103] S2.2 Low-quality image enhancement: Input the image processed by S2.1 into the GAN, start the network iterative training (900 iterations), repair motion blur caused by device vibration, enhance the brightness and contrast of the image in low light environment, and make the gray value distribution of the image more uniform.

[0104] S2.3 Background Separation and Feature Optimization: The adaptive histogram equalization algorithm is called to adjust the overall contrast of the image; then the Retinex algorithm is called to separate the illumination component and the reflection component of the image. Gaussian smoothing filtering is performed on the illumination component, and sharpening processing is performed on the reflection component to highlight the texture features of the RGB image (such as crack edges) and the temperature features of the infrared image (such as overheated areas). The processed image is then transmitted to the multi-dimensional feature fusion and AI inference module.

[0105] S3: Multi-dimensional Feature Fusion and AI Inference S3.1 Dual-branch feature extraction: Start the dual-branch convolutional neural network. The first branch takes the preprocessed RGB image as input and extracts texture features such as conveyor belt cracks and gear wear. The second branch takes the preprocessed infrared image as input and extracts temperature features of bearings, gearboxes and other parts. Both types of feature data are output as 512-dimensional vectors.

[0106] S3.2 Cross-modal feature fusion: Call the cross-modal attention mechanism algorithm, input two types of feature data, set the attention weight coefficient to 0.5 (to balance the importance of visual and temperature features), and perform weighted fusion of feature vectors to generate a 512-dimensional comprehensive representation vector.

[0107] S3.3 Small Sample Size and Incremental Learning Optimization: Periodically screen fault images with fewer than 50 samples (such as latent cracks in conveyor belt joints), input them into the GAN to generate virtual samples (250 virtual samples are generated for each real sample), and update the labeled sample library; when the cumulative number of newly collected fault samples reaches 50, the online incremental learning program is automatically started, updating the parameters of the dual-branch convolutional neural network and improving the YOLOv8 model to enhance the model's ability to identify new types of faults.

[0108] S3.4 Fault Identification and Life Prediction: The comprehensive representation vector is input into the improved YOLOv8 model. The model backbone captures equipment deformation features (such as irregular wear of conveyor belts) through deformable convolution, and the neck fuses multi-scale features through BiFPN. Fault identification is performed at a speed of 28fps, and the fault type (such as "bearing inner ring overheating"), image coordinates (such as (320, 450)), equipment physical coordinates (such as "belt conveyor roller bearing #1") and confidence level (such as 0.96) are output. At the same time, the time series analysis program is called, and the image sequence of 10 consecutive hours is input. The equipment running time (such as "cumulative running time of 1200 hours") and load data (such as "belt conveyor load of 80%)" are imported synchronously to calculate the remaining service life of the equipment (such as "bearing remaining service life of 300 hours").

[0109] S4: Knowledge Graphs and Multi-System Closed Loop S4.1 Fault Reasoning Verification: Input the fault identification results output from S3.4 into the knowledge graph reasoning engine, combine the Bayesian probability model to calculate the fault confidence, match fault tree nodes with expert rules, and output fault classification labels (e.g., "bearing inner ring overheating" is judged as a level 2 fault). S4.2 Multi-system data transmission: Start the 5G gateway transmission program to transmit fault classification tags, fault types, locations, confidence levels, and image data to the equipment management system and operation and maintenance scheduling system. S4.3 Fault Classification Response: After receiving the data, if the fault is classified as a Level 2 fault, the equipment management system sends a routine task push instruction to the operation and maintenance scheduling system; if it is classified as a Level 1 fault (such as "conveyor belt lateral tear"), it automatically sends a shutdown signal to the central control system and pushes an emergency task at the same time. S4.4 Operation and Maintenance Task Allocation: After receiving the task instruction, the operation and maintenance scheduling system calls the personnel matching program. Based on the real-time location of the operation and maintenance personnel (e.g., "Operation and maintenance personnel A is 500 meters away from the faulty equipment") and skill tags (e.g., "proficient in bearing repair"), the system assigns the task to operation and maintenance personnel A and sends the task information (including fault details, handling requirements, and a list of required tools) through the mobile APP. S4.5 Maintenance Process Tracking: Maintenance personnel A scans the QR code of the faulty equipment through the APP to trigger check-in confirmation; during the process, they take photos (such as the bearing disassembly process) and videos through the APP and upload them to the maintenance dispatch system in real time; after the process is completed, they submit a "fault handling completed" application. S4.6 Fault Closed-Loop Confirmation: After receiving the application, the equipment management system sends a secondary imaging command to the acquisition unit to acquire images of the fault-handling area; the secondary imaging data is input into the improved YOLOv8 model for re-examination. If the re-examination confidence level is <0.1 (e.g., "bearing temperature has returned to normal, no overheating area"), the fault data (fault type, processing time) and maintenance data (handling personnel, consumable usage) are recorded in the equipment file database; if the re-examination confidence level is ≥0.1, it is determined that the fault has not been completely eliminated, and a re-dispatch command is sent to the maintenance scheduling system. S5: Visualization and Data Integration The equipment management system transmits fault identification results and remaining equipment lifespan data to a 3D digital twin platform, where fault location is visualized (e.g., marking the location of a faulty bearing in red on a 3D model of a ship loader). A structured identification report is generated (including fault occurrence time, location, handling process, maintenance recommendations, and spare parts model), formatted as PDF and downloadable from the system. Fault data and maintenance data are transmitted to the port enterprise management system via HTTPS, enabling managers to remotely monitor equipment status and trace historical fault data. Simultaneously, edge computing nodes upload newly collected fault samples and model parameters to the cloud, where the global model is optimized and trained, updated monthly to improve overall identification accuracy.

[0110] Therefore, this embodiment addresses the core issue of "high missed detection rate of latent faults due to incomplete single-modal perception" by using multimodal acquisition to adapt to the high-dust environment of coal terminals, dynamic preprocessing to optimize image quality, cross-modal fusion to improve recognition accuracy, multi-system closed-loop to shorten the operation and maintenance cycle, and three-dimensional visualization to enhance management efficiency. It also makes up for the shortcomings of slow response and high operation and maintenance costs of traditional manual inspection. Ultimately, it achieves a significant improvement in fault identification accuracy, a significant reduction in fault response time, and a reduction in the number of unplanned equipment downtimes, balancing identification accuracy and operation and maintenance economy, and fully meeting the actual needs of intelligent management of coal terminal equipment.

[0111] In addition, compared with the prior art, this application provides a method for identifying faults in coal loading terminal equipment, which has the following beneficial effects: 1. This method is based on AI image recognition. It acquires equipment appearance and temperature information simultaneously through multimodal image acquisition (high-definition industrial camera + infrared thermal imaging camera). Combined with cross-modal feature fusion technology, it breaks through the limitations of single-modal perception in existing technologies. It can identify both obvious faults such as conveyor belt cracks and gearbox wear, as well as hidden faults such as bearing overheating and gear meshing temperature rise. After dynamic adaptive preprocessing to optimize image quality, combined with improved AI models and knowledge graph verification, it significantly reduces the rate of missed detection of hidden faults and effectively avoids equipment downtime or safety accidents caused by failure to detect faults in time.

[0112] 2. This method constructs a fully automated closed loop of "fault identification - hierarchical push - operation and maintenance tracking - re-inspection and archiving". Level 1 faults automatically trigger shutdown and emergency response, while level 2 and 3 faults accurately allocate operation and maintenance resources. Compared with the existing manual operation mode, it significantly shortens the fault response and handling cycle. 3D digital twin visualization and structured reports reduce the difficulty of operation and maintenance and improve personnel efficiency. At the same time, it supports remote monitoring of fault data, long-term traceability and prediction of equipment remaining life, helping the terminal to shift from "post-fault maintenance" to "preventive maintenance", reducing unplanned downtime, reducing long-term operation and maintenance costs, and historical data can support equipment health management and operation and maintenance strategy optimization.

[0113] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram can be found in [reference needed]. Figure 6 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to a fault identification method for coal loading equipment. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it can implement a fault identification method for coal loading equipment.

[0114] Those skilled in the art will understand, see Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0116] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0117] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data that have been agreed to by the user or have been fully agreed to by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. In the embodiments provided in this application, any reference to memory, database, or other media can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0120] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units, etc., and are not limited to these.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for fault identification in coal loading equipment, characterized in that, include: Acquire RGB and infrared images of predetermined parts of a target device, wherein the target device includes coal loading equipment and the predetermined parts include parts that affect the operating status of the target device; Extract the texture features of the RGB image and the infrared temperature features of the infrared image; By fusing the texture features and the infrared temperature features, a comprehensive representation vector is obtained; The integrated representation vector is sent to a pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type.

2. The method for fault identification of coal loading equipment according to claim 1, characterized in that, The RGB image is a background-separated RGB image, and the infrared image is a background-separated infrared image. The RGB image and the infrared image are generated using the following method: Acquire initial RGB and initial infrared images of a predetermined portion of the target device; The initial RGB image and the initial infrared image are sent to the pre-trained improved U-net model so that the pre-trained improved U-net model outputs the RGB image and infrared image after removing dust noise, which are denoised as the denoised RGB image and the denoised infrared image, respectively. The denoised RGB image and the denoised infrared image are input into a pre-trained generative adversarial network (GAN) model. The model performs motion blur restoration and illumination enhancement on the input images and outputs the restored images, which are denoted as the restored RGB image and the restored infrared image, respectively. A background separation operation is performed on the repaired RGB image and the repaired infrared image to obtain the background-separated RGB image and the background-separated infrared image.

3. The method for fault identification of coal loading equipment according to claim 2, characterized in that, The step of performing a background separation operation on the repaired RGB image and the repaired infrared image to obtain the background-separated RGB image and the background-separated infrared image includes: The repaired RGB image and the repaired infrared image are sent to the adaptive histogram equalization model so that the adaptive histogram equalization model outputs the contrast-adjusted RGB image and infrared image, which are denoted as the contrast RGB image and the contrast infrared image, respectively. The contrast RGB image and the contrast infrared image are respectively input into the retinal cortex theoretical model so that the retinal cortex theoretical model outputs the separated illumination component map and reflection component map, which are respectively denoted as RGB illumination component, RGB reflection component, infrared illumination component and infrared reflection component. The RGB illumination component and the infrared illumination component are subjected to smoothing filtering, and the processed images are denoted as the RGB illumination smoothing component and the infrared illumination smoothing component; the RGB reflection component and the infrared reflection component are subjected to feature enhancement processing, and the processed images are denoted as the RGB reflection feature component and the infrared reflection feature component. The RGB illumination smoothing component and the RGB reflection feature component are synthesized to obtain the background-separated RGB image; the infrared reflection feature component and the infrared illumination smoothing component are synthesized to obtain the background-separated infrared image.

4. The method for fault identification of coal loading equipment according to claim 1, characterized in that, The extraction of texture features from the RGB image and the extraction of infrared temperature features from the infrared image include: The RGB image and the infrared image are sent to a pre-trained dual-branch convolutional neural network, so that the dual-branch convolutional neural network outputs the texture features of the RGB image and the infrared temperature features of the infrared image.

5. The method for fault identification of coal loading equipment according to claim 1, characterized in that, The process of fusing the texture features and the infrared temperature features to obtain a comprehensive representation vector includes: sending the texture features and the infrared temperature features to a cross-modal attention mechanism so that the cross-modal attention mechanism fuses the texture features and the infrared temperature features to obtain the comprehensive representation vector.

6. The method for fault identification of coal loading equipment according to claim 1, characterized in that, The output of the pre-trained improved YOLOv8 model also includes image coordinate data, device physical coordinate data, and confidence score. After sending the comprehensive representation vector to the pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type, the method further includes: sending the fault type, image coordinate data, device physical coordinate data, and confidence score to a knowledge graph-based fault reasoning engine so that the fault reasoning engine outputs the fault level.

7. The method for fault identification of coal loading equipment according to claim 3, characterized in that, After sending the integrated representation vector to the pre-trained improved YOLOv8 model so that the pre-trained improved YOLOv8 model outputs the fault type, the method further includes: A multimodal time-series input sequence is constructed, which includes an RGB image sequence and an infrared image sequence with a continuous predetermined duration window, as well as the historical and real-time operating data of the target device corresponding to each image timestamp; The multimodal time-series input sequence is input into a pre-trained time-series analysis model so that the time-series analysis model can predict and output the remaining lifespan of the target device.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the coal loading equipment fault identification method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the coal loading equipment fault identification method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the coal loading equipment fault identification method according to any one of claims 1-7.

Images