Coal conveying gallery digital twin inspection method and system

Abstract

This application belongs to the field of intelligent inspection and control technology for coal conveying corridors, and relates to a digital twin inspection method and system for coal conveying corridors. The method includes: establishing a digital twin node library based on the track mileage of the coal conveying corridor; generating inspection frames at the same sampling time and binding them to inspection nodes; correcting the track mileage based on the mileage measurement results of RFID tags and travel encoders and generating field-of-view constraints; inputting the inspection frames and field-of-view constraints into a digital twin constraint multimodal detection model, and outputting candidate anomaly results with spatial attribution; constructing anomaly persistence states based on the candidate anomaly results and confirming the inspection event level; and generating inspection control results based on the inspection event level. The technical solution of this application achieves a closed loop of anomaly identification, continuous confirmation, and inspection handling through the coordination of track mileage constraints, inspection frame synchronization, multimodal detection, and event level control. This can improve the spatial reliability and handling stability of inspection results and meet the intelligent inspection needs in complex corridor environments.

Description

Technical Field

[0001] This application belongs to the field of intelligent inspection and control technology for coal conveying corridors, and specifically relates to a digital twin inspection method and system for coal conveying corridors. Background Technology

[0002] In existing technologies, intelligent inspection of coal conveyor corridors typically relies on inspection robots running along fixed tracks, video acquisition equipment, infrared thermal imaging equipment, and a back-end monitoring platform. This system continuously collects, identifies, and issues alarms regarding the status of belts, idlers, coal drop points, personnel, and the environment to meet the needs of unmanned inspection and operational risk detection in coal conveyor corridors. However, existing coal conveyor corridor inspection methods have some significant shortcomings in terms of data spatial attribution and anomaly handling coordination.

[0003] In practical applications, existing track inspection robots have a certain degree of capabilities in automatic walking, image recognition, infrared temperature measurement, positioning feedback, and alarm output. However, the narrow and long coal conveying corridor, continuous equipment arrangement, and significant dust obstruction mean that images and infrared data collected by the robot between adjacent inspection positions can easily cover adjacent equipment areas simultaneously. Affected by factors such as track positioning deviation, sampling time differences, viewing angle changes, and instantaneous obstruction, the correspondence between anomaly identification results and specific equipment areas is not very stable. When a single identification result directly triggers an alarm or control, the overall coordination between the anomaly confirmation process and subsequent handling actions is also relatively low.

[0004] It is evident that existing technologies often suffer from problems such as weak spatial stability of anomaly identification results during coal conveyor corridor inspections and weak coordination between anomaly confirmation and inspection handling. These are the shortcomings of existing technologies.

[0005] In view of this, it is necessary to provide a digital twin inspection method and system for coal conveying corridors to solve the above-mentioned defects in the existing technology. Summary of the Invention

[0006] The purpose of this application is to provide a digital twin inspection method and system for coal conveying corridors, addressing the shortcomings of the existing technology, such as weak spatial stability of anomaly identification results and weak coordination between anomaly confirmation and inspection handling, in order to solve the aforementioned technical problems.

[0007] To achieve the above objectives, this application provides the following technical solution: Firstly, this application provides a digital twin inspection method for coal conveying corridors, including: Step S1: Establish a digital twin node library including inspection nodes based on the track mileage of the coal conveying corridor; Step S2: Collect the raw data of the inspection frame and generate inspection frames at the same sampling time, and bind the inspection frames to the inspection nodes in the digital twin node library; Step S3: Correct the track mileage of the inspection frame based on the RFID tag and travel encoder mileage measurement results, and generate the field of view constraint of the current inspection node; Step S4: Input the inspection frame and field of view constraints into the digital twin constrained multimodal detection model, and output the candidate anomaly results with spatial attribution; Step S5: Construct an anomaly persistence state based on candidate anomaly results and inspection frames, and confirm the inspection event level according to the anomaly persistence state; Step S6: Generate inspection control results based on the inspection event level and the digital twin node library. The inspection control results include low-speed verification, remote autonomous intervention, safe shutdown, return to base for charging, or mission recovery.

[0008] By adopting the above technical solution, the track position of the coal conveying corridor, inspection data, spatial constraints, continuous confirmation of anomalies, and inspection handling are incorporated into a unified digital twin inspection link. This enables coordinated cooperation between inspection data attribution, anomaly identification and judgment, and control result generation. Under the on-site conditions of narrow coal conveying corridor space, continuous equipment layout, dust obstruction, and robot mobile data collection, inspection results with clear spatial orientation, continuous status confirmation, and stable handling actions can be generated. This meets the requirements of strong spatial stability of anomaly identification results and high coordination capability of anomaly confirmation and inspection handling during coal conveying corridor inspection.

[0009] Specifically, organizing a digital twin node library around track mileage helps transform continuous corridor space into a node base that can be stably attributed to by inspection data, reducing spatial confusion when adjacent equipment areas cross into the same acquisition field of view; inspection frames are formed and bound to inspection nodes at the same sampling time, ensuring that images, infrared, positioning, and operating status are based on a unified judgment benchmark, enhancing the consistency of multi-source data entering the subsequent recognition process; track mileage correction and the current inspection node's field of view constraint work together to ensure that the recognition range converges to the corresponding inspection area as the robot's actual position is adjusted, reducing the impact of positioning drift, viewpoint changes, and instantaneous occlusion on different areas. The interference of constant attribution; the output of the digital twin constrained multimodal detection model with spatial attribution helps to unify visual features and heat source features into the spatial location of the corridor, and improves the correspondence stability between candidate anomaly results and equipment areas; the anomaly persistence state and inspection event level jointly participate in anomaly confirmation, which can reduce the impact of false triggering caused by single frame jitter, dust obstruction and low confidence recognition; the inspection control results are generated with the inspection event level and digital twin node library, so that the review, intervention, shutdown, return charging and task recovery are continuously connected, and promote the formation of a complete closed loop in the coal conveying corridor inspection process.

[0010] Preferably, step S1 specifically includes: The coal conveying corridor is divided into continuous inspection nodes based on the track mileage, and transition mileage segments are generated between adjacent inspection nodes. Based on the overlapping order between the equipment area, the visible light field of view boundary, and the infrared field of view boundary within the transition mileage section, a rule for resolving ownership conflicts is generated. Based on the rules for resolving ownership conflicts, establish the primary ownership identifier, the verification ownership identifier, and the background exclusion identifier corresponding to the overlapping range of adjacent inspection nodes; The inspection nodes, transition mileage segments, rules for resolving ownership conflicts, primary ownership identifiers, verification ownership identifiers, and background exclusion identifiers are written into the digital twin node library.

[0011] Based on the above scheme, an attribution resolution relationship is established around the corridor area where the field of view is likely to overlap between adjacent inspection nodes. This makes the continuously arranged equipment area form a clearer spatial boundary in the digital twin node library, which can reduce the risk of the same abnormal target being repeatedly confirmed or incorrectly excluded by adjacent inspection positions, and improve the stability of abnormal spatial attribution and the consistency of node boundary processing.

[0012] Preferably, step S2 specifically includes: Collect raw data of inspection frames, align the raw data of inspection frames with the same sampling time, and generate inspection frames containing visible light data, infrared thermal image data, positioning status, running status and communication status. Based on the track mileage corresponding to the positioning status in the inspection frame, the ownership conflict resolution rules are called in the digital twin node library to obtain the main ownership identifier, the verification ownership identifier, or the background exclusion identifier corresponding to the inspection frame. When an inspection frame corresponds to a primary home identifier, the inspection frame is bound to the inspection node corresponding to the primary home identifier. When an inspection frame corresponds to a review attribution identifier, the inspection frame is written into the review attribution temporary storage sequence. When an inspection frame corresponds to a background exclusion flag, the inspection frame is written into the background exclusion record sequence.

[0013] In the above scheme, the inspection frames formed at the same sampling time are introduced into the diversion process after the attribution resolution, so that the data before entering the identification first obtains the primary attribution, the verification attribution or the background exclusion processing path. This can reduce the impact of adjacent device overlap, viewpoint offset and background interference on the binding of inspection nodes, and make the data source on which subsequent anomaly identification is based more stable.

[0014] Preferably, step S3 specifically includes: The tag mileage obtained from RFID tag readings and the continuous mileage determined by the travel encoder mileage measurement results are converted to the same track mileage reference. A reliable mileage status is generated based on the reading order of the tag mileage, the direction of change of continuous mileage, and the running status in the inspection frame; When the reading order of the mileage trust status characterization tag mileage, the direction of change of continuous mileage and the running status all correspond to the current inspection node, the visible light field of view boundary and infrared field of view boundary corresponding to the current inspection node are called to generate the field of view constraint. When the reading order of the mileage trust status characterization tag mileage, the direction of change of continuous mileage, or the running status does not correspond to the current inspection node, the field of view constraint is switched to the inspection node level field of view constraint, and a mileage verification mark is generated.

[0015] Based on the above processing, a reliable mileage judgment is formed around tag reading and continuous mileage changes, which enables the field of view of the current inspection node to be constrained and adjusted according to the actual operating state of the robot. This can reduce the probability of misuse of the recognition range when there is positioning deviation, reverse movement or inconsistent operating state, and improve the stability of the correspondence between the field of view constraint and the actual inspection position.

[0016] As a preferred embodiment, the steps for generating mileage verification markers specifically include: The inspection frames that trigger the mileage verification mark are written into the mileage verification sequence according to the sampling time. The mileage verification markers are updated based on the correspondence between the reading order of the mileage tags in the mileage verification sequence and the direction of change of the continuous mileage. When the corresponding relationship points to the same inspection node, remove the mileage verification mark and restore the field of view constraint of the current inspection node; When the correspondence points to different inspection nodes, maintain the mileage verification mark and prohibit candidate abnormal results from forming equipment-level spatial attribution.

[0017] In the above scheme, a continuous verification mechanism is formed around the mileage verification mark, so that the positioning conflict is no longer directly transmitted to the device-level anomaly attribution. It can maintain a more cautious spatial judgment state when there is inconsistency between the tag reading order and continuous mileage changes, reduce the risk of wrong device pointing caused by positioning anomalies, and improve the spatial reliability of inspection results.

[0018] Preferably, step S4 specifically includes: The visible light data in the inspection frame is input into the visible light feature branch of the digital twin constrained multimodal detection model, and visual candidate features are output. The infrared thermal image data in the inspection frame is input into the infrared feature branch of the digital twin constrained multimodal detection model, and the heat source candidate features are output. The field-of-view constraints are input into the node prior branches of the digital twin constrained multimodal detection model to generate candidate region masks; Spatial registration of visual candidate features and heat source candidate features is performed based on the candidate region mask to generate cross-modal candidate results; Candidate anomalies with spatial attribution are generated based on the coverage relationship of cross-modal candidate results in the candidate region mask.

[0019] Based on the above inspection frame and field of view constraint processing logic, visible light features, infrared heat source features and node prior constraints are incorporated into the same multimodal detection process. This allows candidate targets to be restricted by the mask of the current inspection area before spatial assignment is formed, which can suppress interference from background heat sources, adjacent device textures and targets outside the current field of view, and enhance the correspondence stability between candidate abnormal results and real device areas.

[0020] Preferably, step S5, which involves constructing the persistent anomaly state based on the candidate anomaly results and the inspection frame, specifically includes: Based on the spatial attribution of the candidate anomaly results, the cross-modal candidate results within the same inspection node are formed into a risk change sequence according to the sampling time; The risk change sequence and the positioning status, operation status and communication status in the inspection frame are input into the time-series denoising and reconstruction process to generate reconstruction deviation results. An abnormal persistent state is generated based on the correspondence between the reconstruction deviation results and the cross-modal candidate results; Write the persistent abnormal state, the reconstructed deviation result, and the candidate abnormal result into the event evidence chain.

[0021] Based on the above multimodal candidate results, an abnormal and continuous state is formed around the spatial attribution changes within the same inspection node. This allows a single identification result to be included in the continuous judgment along with the positioning, operation, and communication status. This can reduce the impact of dust obstruction, short-term shaking, and instantaneous low-confidence identification on event confirmation, and improve the stability and continuity of evidence in the process of generating inspection event levels.

[0022] Preferably, step S6, which generates inspection control results based on the inspection event level and the digital twin node library, specifically includes: Based on the inspection event level and the abnormal persistence status in the event evidence chain, as well as the reconstruction deviation results, the action boundary and the verification mileage segment are extracted from the digital twin node library; The remote autonomous intervention command is converted into a digital twin candidate action. The digital twin candidate action is verified based on the action boundary, the review mileage segment and the inspection event level, and the execution verification result is generated. When the execution verification result indicates that the digital twin candidate action is within the action boundary and corresponds to the verification mileage segment, the inspection control result is generated according to the inspection event level. When the execution verification result indicates that the digital twin candidate action is not within the action boundary or does not correspond to the verification mileage segment, a verification inspection control result is generated based on the event evidence chain, and the verification inspection control result is written into the event evidence chain.

[0023] After the aforementioned chain of evidence for the event is formed, the abnormal persistence state and the inspection event level are introduced into the control generation process. This allows remote autonomous intervention actions to be jointly constrained by the digital twin space boundary and the review mileage range before execution. This reduces the possibility of the abnormal handling actions being out of sync with the actual risk location and generates a review-type control result when the actions do not match, thereby improving the safety, continuity and traceability of the inspection and handling chain.

[0024] Secondly, this application also provides a digital twin inspection system for coal conveying corridors, including: The node database module is used to build a digital twin node database, including inspection nodes, based on the track mileage of the coal conveying corridor. The inspection binding module is used to collect the raw data of the inspection frame and generate inspection frames at the same sampling time, and bind the inspection frame to the inspection node in the digital twin node library. The mileage constraint module is used to correct the track mileage of the inspection frame based on the mileage measurement results of the RFID tag and the travel encoder, and to generate the field of view constraint of the current inspection node. The multimodal detection module is used to input the inspection frames and field of view constraints into the digital twin constrained multimodal detection model and output candidate anomaly results with spatial attribution; The event confirmation module is used to construct the anomaly persistence status based on candidate anomaly results and inspection frames, and to confirm the inspection event level according to the anomaly persistence status. The control generation module is used to generate inspection control results based on the inspection event level and the digital twin node library. The inspection control results include low-speed verification, remote autonomous intervention, safe shutdown, return to base for charging, or mission recovery. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the embodiments are briefly described below. The following drawings only show some embodiments of this application; those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a digital twin inspection method for coal conveying corridors provided in this application; Figure 2 This is a schematic diagram of a digital twin inspection system for a coal conveying corridor provided in this application.

[0027] The modules include: 1. Node database creation module, 2. Inspection binding module, 3. Mileage constraint module, 4. Multi-mode detection module, 5. Event confirmation module, and 6. Control generation module. Detailed Implementation

[0028] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the following embodiments are only used to illustrate the technical solutions of this application and are not intended to limit the scope of protection of this application. Any adjustments, equivalent substitutions, improvements or other optional implementation methods made by those skilled in the art to the embodiments without departing from the concept and scope of protection of this application should fall within the scope of protection of this application.

[0029] It should be noted in advance that, in order to facilitate understanding of the technical solutions of the embodiments of this application, some terms and related technologies involved in the embodiments of this application will be briefly explained below: 1. Digital twin: refers to the construction of a virtual representation corresponding to a physical object, device or scene in a digital way. It reflects the structure, location, operating status or environmental changes of the actual object through model, data and status updates. It can be used in scenarios such as equipment operation and maintenance, industrial inspection, process monitoring and simulation analysis.

[0030] 2. Radio Frequency Identification: Usually corresponding to RFID (Radio Frequency Identification), it uses radio frequency signals to perform non-contact data identification between the reader and the tag. The tag can record information such as number, location or object attributes. It is often used in scenarios such as item identification, location-assisted calibration and asset management.

[0031] 3. Stroke encoder mileage measurement results: This refers to the change in stroke pulses output by the stroke encoder during the operation of the inspection robot along the coal conveying corridor track, as well as the continuous mileage calculated based on the track displacement corresponding to a single pulse. This result is used to describe the continuous displacement state of the inspection robot along the track mileage direction and is unrelated to the stroke encoding algorithm results in the field of data compression.

[0032] 4. Multimodal detection model: refers to a detection model that simultaneously processes different types of information such as images, thermal images, sound, text, and sensor data. By extracting, aligning, and fusing features from data from different sources, it obtains a more complete target or state judgment result than a single data source. It is often used in scenarios such as intelligent inspection, security monitoring, industrial inspection, and medical image analysis.

[0033] To address the issues of weak spatial stability of anomaly identification results and poor coordination between anomaly confirmation and inspection handling during coal conveyor corridor inspections, which make it difficult to meet the actual needs of coal conveyor corridors for accurate data attribution, continuous confirmation of anomaly states, and stable connection of handling actions, this application discloses a digital twin inspection method and system for coal conveyor corridors. By introducing digital twin node management under track mileage constraints, multi-source inspection frame synchronous processing, multimodal anomaly detection with spatial attribution, and a control generation mechanism driven by inspection event levels, the method enables inspection data, anomaly identification results, and inspection control actions to maintain a correspondence in a unified digital twin space. This achieves stable identification, continuous confirmation, and closed-loop handling of anomalies in coal conveyor corridor operations, thereby improving the spatial reliability of inspection results, the continuity of anomaly judgment, and the coordinated stability of inspection handling in complex corridor environments. This further enhances the on-site adaptability and operational safety management level of the intelligent inspection process for coal conveyor corridors.

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] like Figure 1 As shown in the figure, this embodiment provides a digital twin inspection method for coal conveying corridors, including: Step S1: Establish a digital twin node library including inspection nodes based on the track mileage of the coal conveying corridor; Step S2: Collect the raw data of the inspection frame and generate inspection frames at the same sampling time, and bind the inspection frames to the inspection nodes in the digital twin node library; Step S3: Correct the track mileage of the inspection frame based on the RFID tag and travel encoder mileage measurement results, and generate the field of view constraint of the current inspection node; Step S4: Input the inspection frame and field of view constraints into the digital twin constrained multimodal detection model, and output the candidate anomaly results with spatial attribution; Step S5: Construct an anomaly persistence state based on candidate anomaly results and inspection frames, and confirm the inspection event level according to the anomaly persistence state; Step S6: Generate inspection control results based on the inspection event level and the digital twin node library. The inspection control results include low-speed verification, remote autonomous intervention, safe shutdown, return to base for charging, or mission recovery.

[0036] This embodiment establishes a stable correspondence between continuously distributed belts, idlers, coal drop points, and inspection positions in the coal conveying corridor by employing a digital twin node library based on track mileage. By organizing visible light data, infrared thermal image data, positioning status, and operational status at the same sampling moment into inspection frames and binding them to inspection nodes, multi-source inspection data possesses a consistent temporal reference and spatial attribution basis before entering anomaly identification. The track mileage of the inspection frames is corrected using RFID tags and travel encoder mileage measurement results, and combined with the current inspection node to form a field-of-view constraint, thus mitigating positioning deviations, perspective changes, and overlaps of adjacent equipment during robot movement. The impact on anomaly attribution is controlled; by inputting inspection frames and field-of-view constraints into the digital twin constrained multimodal detection model, visible light features and infrared heat source features can be analyzed within the current inspection area to generate candidate anomaly results with spatial attribution; after the candidate anomaly results are formed, the anomaly persistence state is further constructed by combining the inspection frames and the inspection event level is confirmed, so that short-term occlusion, dust interference and accidental low-confidence identification do not directly enter the formal handling chain; overall, it can keep the data attribution, anomaly confirmation and control and handling in the coal conveying corridor inspection process continuously connected, and improve the spatial credibility of anomaly identification results, event confirmation stability and inspection control closed loop integrity in complex corridor environments.

[0037] The above steps will be specifically described below based on the embodiments of this application.

[0038] In step S1, the core task is to establish a digital twin node library that can express inspection nodes, transition mileage sections, equipment space boundaries, and anomaly attribution rules around the fixed inspection lines continuously arranged along the belt conveyor in the coal conveying corridor.

[0039] In this embodiment, the coal conveying corridor contains idler rollers, guide chutes, drive drums, tensioning mechanisms, coal drop points, dust removal equipment, electrical control cabinets, fire monitoring points, wireless charging stations, and inspection channels. These different devices are densely arranged along the track direction, and when the robot collects images at adjacent locations, it can easily cover the equipment areas of adjacent inspection nodes simultaneously. Therefore, step S1 needs to convert the continuous space in the physical corridor into a digital twin node library that can be called by subsequent inspection frame binding, field-of-view constraint generation, candidate anomaly result attribution, and inspection control result generation. The core outputs include continuous inspection nodes, transition mileage segments, attribution conflict resolution rules, primary attribution identifier, verification attribution identifier, and background exclusion identifier.

[0040] In this embodiment, a digital twin node library including inspection nodes can be established based on the track mileage of the coal conveying corridor. The track mileage starts at the beginning of the fixed track of the inspection robot within the coal conveying corridor and increases along the robot's regular inspection direction. It can be determined jointly by the corridor design drawings, track construction drawings, robot initial calibration data, and on-site distance measurement results. Each inspection node in the digital twin node library corresponds to a track mileage interval and is simultaneously associated with the equipment area, visible light field of view boundary, infrared field of view boundary, parking range, re-shooting location, personnel restricted area boundary, and action boundary within that interval. For ordinary conveyor belt sections, inspection nodes can be divided according to fixed mileage intervals; for the head, tail, coal drop points, turning sections, key idler groups, fire monitoring points, and wireless charging stations, inspection nodes can be more densely divided according to equipment boundaries and verification requirements.

[0041] In some embodiments of this application, the coal conveying corridor can be divided into continuous inspection nodes based on track mileage. Continuous inspection nodes are arranged in ascending order of track mileage. Each continuous inspection node includes a node number, a starting track mileage, an ending track mileage, a target equipment area, a visible light field of view boundary, an infrared field of view boundary, and a node type. Node types may include ordinary inspection nodes, belt misalignment inspection nodes, heat source inspection nodes, smoke inspection nodes, personnel-restricted inspection nodes, and wireless charging inspection nodes. Ordinary inspection nodes are used for routine image and environmental data acquisition; belt misalignment inspection nodes are used for judging belt edges and idler roller outer edges; heat source inspection nodes are used for infrared thermal imaging judgment of idler roller ends, coal accumulation areas, and electrical control cabinets; and wireless charging inspection nodes are used for low battery return and mission recovery.

[0042] Furthermore, transitional mileage segments can be generated between adjacent inspection nodes. These segments address the issue of image and infrared thermal imaging coverage spanning two inspection nodes simultaneously when the robot is near the boundary of adjacent nodes. For example, when the robot is in front of a coal drop point, visible light data may simultaneously cover the leading edge of the drop point and adjacent idler rollers, while infrared thermal imaging data may simultaneously cover the end of the idler roller bearing and the area of ​​interest for the coal heat source. If the inspection frame is directly bound to a single inspection node based on the robot's current track mileage, candidate anomalies are easily misattributed to adjacent equipment areas. Therefore, transitional mileage segments are used to pre-record mileage ranges where such attribution conflicts may occur and participate in inspection frame binding and spatial attribution of candidate anomalies in subsequent steps.

[0043] The transition mileage segment can be calculated based on the equipment area, visible light field of view boundary, and infrared field of view boundary of adjacent inspection nodes. The spatial overlap area corresponding to adjacent inspection nodes can be represented by: ,in, Indicates the first The inspection node and the first The spatial overlap area between inspection nodes Indicates the first The equipment area corresponding to each inspection node. This indicates the equipment area corresponding to the next adjacent inspection node. and They represent the first Each inspection node corresponds to a visible light field of view boundary coverage area and an infrared field of view boundary coverage area. When this spatial overlap area is projected onto the orbital centerline, it forms a transition mileage segment between adjacent inspection nodes.

[0044] Based on this, attribution conflict resolution rules are generated according to the overlap order between the equipment area, visible light field of view boundary, and infrared field of view boundary within the transition mileage segment. The overlap order can be determined according to the projection position of the candidate anomaly result within the visible light field of view boundary, the heat source position within the infrared field of view boundary, and the distance relationship with the equipment area boundary. If the candidate anomaly result falls into the equipment area of ​​both the current inspection node and the adjacent inspection node, and its visible light projection is closer to the equipment centerline of the current inspection node, then the candidate anomaly result corresponds to the primary attribution; if the visible light projection and infrared projection of the candidate anomaly result belong to different inspection nodes, or if the candidate anomaly result is located in a spatially overlapping area. If the candidate abnormal result is near the boundary, then the candidate abnormal result is assigned to the review; if the candidate abnormal result appears at the edge of the image, but its projection position does not fall into the equipment area of ​​any inspection node, or only falls into the background wall, bracket shadow, or lighting reflection area, then the candidate abnormal result is excluded from the background.

[0045] Furthermore, based on the ownership conflict resolution rules, primary ownership identifiers, verification ownership identifiers, and background exclusion identifiers are established for the overlapping ranges of adjacent inspection nodes. The primary ownership identifier indicates that the inspection frame can be directly bound to the inspection node corresponding to the primary ownership identifier; the verification ownership identifier indicates that the inspection frame needs to enter the verification ownership temporary storage sequence; and the background exclusion identifier indicates that the inspection frame needs to be written into the background exclusion record sequence. These three types of identifiers are only used to define the data ownership status. For example, the track mileage of a certain idler roller group is... to The track mileage of adjacent coal drop points is to If the visible light field boundary is in to If there is an overlap between the two and the boundary of the infrared field of view, then... to This is set as a transition mileage section. Heat sources located in the idler end area correspond to the primary attribution identifier, heat sources located at the boundary between the idler end and the coal accumulation area at the coal drop point correspond to the verification attribution identifier, and heat sources located in the background light reflection area correspond to the background exclusion identifier.

[0046] Furthermore, inspection nodes, transition mileage segments, ownership conflict resolution rules, primary ownership identifiers, verification ownership identifiers, and background exclusion identifiers are written into the digital twin node library. During the writing process, each inspection node forms a node index according to the track mileage order, each transition mileage segment is associated with two adjacent inspection nodes, and each ownership conflict resolution rule is associated with the corresponding spatial overlapping area. The primary attribution identifier, the secondary attribution identifier, and the background exclusion identifier are stored as the output state of this rule. The digital twin node library can also store node version numbers, update times, device area outlines, and secondary mileage segments. Before the robot executes an inspection task, it reads the current version of the digital twin node library to prevent the robot from executing a new inspection task based on the old node boundaries.

[0047] Thus, step S1 completes the conversion of the physical track space of the coal conveying corridor into a digital twin node library, forming a node constraint system with track mileage as the main line, continuous inspection nodes and transition mileage sections as the skeleton, and ownership conflict resolution rules as the core. This provides a stable spatial boundary foundation for subsequent inspection frame binding, track mileage correction, field of view constraint generation, and spatial ownership of candidate abnormal results.

[0048] In step S2, the core task is to organize the multi-source data collected during the robot inspection into a unified inspection frame, and bind the inspection frame to the corresponding inspection node or enter the verification or exclusion path according to the ownership conflict resolution rules in the digital twin node library. During this process, visible light data, infrared thermal image data, positioning status, operating status, and communication status within the coal conveying corridor come from different acquisition units, and there may be slight deviations in sampling time. If time alignment is not performed, subsequent track mileage correction, cross-modal registration, and abnormal persistent state construction will be affected. The inputs to step S2 are the original inspection frame data and the digital twin node library, and the outputs are the inspection frames bound to the inspection nodes, the verification ownership temporary storage sequence, and the background exclusion record sequence.

[0049] Specifically, in some embodiments of this application, the raw data of the inspection frame includes visible light data, infrared thermal image data, positioning status, operating status, and communication status. Visible light data is collected by a visible light camera and used to represent visual information such as belt edges, idler roller outer edges, smoke areas, foreign object areas, and personnel targets. Infrared thermal image data is collected by an infrared thermal imager and used to represent temperature information such as idler roller heat sources, coal accumulation heat sources, electrical control cabinet heat sources, and background heat sources. The positioning status may include RFID tag readings, stroke pulse changes output by the stroke encoder, continuous mileage calculated from the stroke pulse changes, and current track mileage estimates. The operating status may include robot speed, gimbal horizontal angle, gimbal pitch angle, obstacle distance, battery status, and task status. The communication status may include wireless signal strength, data transmission delay, packet loss status, and command confirmation status.

[0050] After data acquisition is completed, the raw data of the inspection frames are time-aligned according to the same sampling time. Specifically, the acquisition time of visible light data can be used as the main time reference, and infrared thermal imaging data, positioning status, operating status, and communication status are matched with this main time reference within a preset time window. For example, the preset time window can be set to... Infrared thermal image data, positioning status, operating status, and communication status falling within the time window are merged into the same inspection frame; data exceeding the time window is buffered and re-matched at the next sampling time. The aligned inspection frame can be written as: ,in, Indicates the first Inspection frames at each sampling time, Represents visible light data. Indicates infrared thermal image data, Indicates the location status. Indicates the running status. Indicates the communication status. This inspection frame It serves as a unified data entry point for subsequent orbital mileage correction, field-of-view constraint generation, model inference, and event control.

[0051] Furthermore, after time alignment, an inspection frame containing visible light data, infrared thermal image data, positioning status, operating status, and communication status is generated. This frame not only saves the image and thermal image but also the corresponding track mileage, sampling time, gimbal attitude, robot speed, battery status, wireless communication status, and task status. After the inspection frame is generated, it can be bound to an inspection node in the digital twin node library. The binding process is not only based on the shortest distance in the track mileage but also requires calling the ownership conflict resolution rules in the digital twin node library. Specifically, based on the track mileage corresponding to the positioning status in the inspection frame, the ownership conflict resolution rules in the digital twin node library are called, and combined with the projection positions of the visible light data and infrared thermal image data corresponding to the inspection frame, it is determined whether the inspection frame is located within a normal inspection node or within a transition mileage segment.

[0052] When an inspection frame falls into a transition mileage segment, the digital twin node library returns its attribution status according to the attribution conflict resolution rules. During this process, the primary attribution identifier, the verification attribution identifier, or the background exclusion identifier corresponding to the inspection frame can be obtained. If the inspection frame corresponds to the primary attribution identifier, it is bound to the inspection node corresponding to the primary attribution identifier. After binding, the inspection frame enters the identification process of the inspection node corresponding to the primary attribution identifier and can participate in the inference of the digital twin constrained multimodal detection model. If the inspection frame corresponds to the verification attribution identifier, it is written into the verification attribution temporary storage sequence. The verification attribution temporary storage sequence stores the inspection frame, track mileage, adjacent inspection node numbers, and verification reason. Spatial attribution will be reconfirmed later during low-speed robot verification or node re-capture. If the inspection frame corresponds to the background exclusion identifier, it is written into the background exclusion record sequence. The background exclusion record sequence is mainly used for subsequent model review and false detection statistics and does not enter the formal anomaly confirmation process.

[0053] For example, the robot in When collecting inspection frames, visible light data simultaneously covers the leading edge of the coal drop point and adjacent idler roller groups, while infrared thermal image data shows high-temperature spots at the image edges. If the high-temperature spot is projected onto the background lighting reflection area, the attribution conflict resolution rule returns a background exclusion flag, and the inspection frame is written into the background exclusion record sequence; if the high-temperature spot is located between the idler roller end and the coal accumulation area of ​​the coal drop point, the attribution conflict resolution rule returns a verification attribution flag, and the inspection frame enters the verification attribution temporary storage sequence; if the high-temperature spot clearly falls into the idler roller bearing end area, the attribution conflict resolution rule returns a primary attribution flag, and the inspection frame is bound to the corresponding idler roller group inspection node.

[0054] Thus, step S2 completes the conversion of the original data of the multi-source inspection frames into a unified inspection frame, and realizes the main ownership binding, temporary storage of the verification ownership, and background exclusion record of the inspection frame based on the ownership conflict resolution rule, so that the subsequent track mileage correction, field of view constraint generation and multimodal anomaly identification are all based on the data entry with clear spatial ownership.

[0055] In step S3, the core task is to correct the track mileage of the inspection frame based on the RFID tag and travel encoder mileage measurement results, and generate the field-of-view constraint of the current inspection node based on the corrected track mileage and the operating status in the inspection frame. In this embodiment, vibration, dust, slope, and wheel-rail wear exist in the coal conveying corridor, and the travel encoder mileage measurement results may have cumulative errors; although the RFID tag readings can provide discrete calibration, they are also affected by the reading direction, tag installation position, and instantaneous signal quality. Step S3 needs to convert the two types of positioning sources to the same track mileage reference, generate a reliable mileage state, and decide whether to use the fine field-of-view constraint of the current inspection node or switch to the inspection node-level field-of-view constraint.

[0056] In this embodiment, the track mileage of the inspection frame can be corrected based on the RFID tag and the mileage measurement results of the travel encoder. Specifically, when the robot walks along a fixed track, the change in travel pulse output by the travel encoder is converted into single-pulse track displacement to provide continuous mileage updates, while the RFID tag provides discrete mileage calibration. The continuous mileage determined by the travel encoder mileage measurement results can be obtained using: ,in, Indicates the first The continuous mileage determined by the mileage measurement results of the travel encoder at each sampling time. This indicates the orbital mileage confirmed at the previous sampling time. Indicates the first The change in stroke encoder pulses within each sampling period This represents the orbital displacement corresponding to a single pulse. This continuous mileage... Used to maintain continuous changes in robot track mileage between adjacent RFID tags.

[0057] Meanwhile, the tag mileage obtained from RFID tag readings needs to be corrected based on the reader's installation location and the robot's direction of travel. Tag mileage can be calculated using: ,in, Indicates the first The tag mileage is obtained by correcting the RFID tag at each sampling time. Indicates the first The track mileage recorded by each RFID tag in the digital twin node library. Indicates the first The travel direction factor at each sampling time is taken as positive when the robot is traveling in the forward direction and the card reader is in front of the robot, and as negative when the robot is traveling in the reverse direction and the card reader is in front of the robot. This indicates the installation distance from the card reader to the center of the robot body. (This label mileage) This avoids directly equating the location of the RFID tag with the center of the robot body.

[0058] In some embodiments of this application, tag mileage and continuous mileage can be converted to the same track mileage reference; wherein, the same track mileage reference uses the mileage of the track centerline recorded in the digital twin node library in step S1. After the conversion is completed, a reliable mileage state is generated based on the tag mileage reading order, the direction of change of continuous mileage, and the running status in the inspection frame; wherein, the reliable mileage state can be jointly determined based on the mileage deviation between continuous mileage and tag mileage, whether the tag reading order conforms to the track direction, and whether the running status is in normal inspection or low-speed verification state. For example, the mileage deviation can be: ,in, Indicates the first The mileage deviation between the continuous mileage and the tag mileage at each sampling time is only used to determine whether the inspection frame can generate the spatial attribution of a specific device area. This represents the continuous mileage determined by the mileage measurement results from the travel encoder. This indicates the tag mileage obtained by correcting the RFID tag.

[0059] For example, when mileage deviation Less than Furthermore, the reading order of the tag mileage is consistent with the direction of change of continuous mileage. When the operating status in the inspection frame is autonomous inspection or low-speed verification, the mileage reliability status can be marked as reliable; when the mileage deviation... lie in to If the tag reading order does not jump, it can be marked as a slight deviation; when the mileage deviation... Exceed And continue to exceed If the tag reading order conflicts with the direction of continuous mileage change, the tag can be marked as temporarily deferred. The above values ​​can be adjusted according to the track installation accuracy and the spacing of the RFID tags.

[0060] When the mileage trustworthiness status is trustworthy, the reading order of the tag mileage, the direction of change of continuous mileage, and the running status all correspond to the current inspection node. At this time, the visible light field of view boundary and infrared field of view boundary corresponding to the current inspection node can be invoked. The visible light field of view boundary is mainly determined by the gimbal horizontal angle, gimbal pitch angle, camera intrinsic parameters, and orbital mileage of the current inspection node. The infrared field of view boundary is mainly determined by the infrared thermal imager field of view angle, temperature imaging area, and camera calibration relationship. Based on this, a field of view constraint is generated. This field of view constraint includes a visible light candidate region, an infrared heat source candidate region, a background region outside the node, and a verification boundary region, which are subsequently entered into the node prior branch of the digital twin constraint multimodal detection model.

[0061] When the mileage trust status is temporarily suspended, it indicates that the reading order of the tag mileage, the direction of change of continuous mileage, or the operating status does not correspond to the current inspection node. At this time, the field of view constraint is switched to the inspection node level field of view constraint. Only candidate abnormal results are allowed to be retained at the inspection node level, and it is not allowed to directly form the specific equipment area assignment. At the same time, a mileage verification mark is generated, which includes the trigger time, the triggered inspection frame, the trigger reason, the continuous mileage, the tag mileage, and the current inspection node number.

[0062] During the generation of mileage verification markers, the inspection frames that trigger the mileage verification markers are written into the mileage verification sequence according to their sampling times. The mileage verification sequence is sorted according to the sampling times, recording the tag mileage and continuous mileage in multiple consecutive inspection frames. Then, the mileage verification markers are updated based on the correspondence between the tag mileage reading order and the direction of change of continuous mileage in the mileage verification sequence. If subsequent inspection frames show that the tag mileage reading order and the direction of change of continuous mileage are consistent again, and the mileage deviation returns to a reliable range, the correspondence can be considered to point to the same inspection node. In this case, the mileage verification marker can be removed, and the field of view constraint of the current inspection node can be restored. If subsequent inspection frames show that the tag mileage reading order and the direction of change of continuous mileage still conflict, the correspondence can be considered to point to different inspection nodes. In this case, the mileage verification marker can be maintained, and candidate abnormal results are prohibited from forming device-level spatial attribution. Based on this, candidate abnormal results can be saved as suspected events at the inspection node level, and remapped after the next reliable RFID tag reading appears.

[0063] At this point, step S3 completes the fusion correction of the mileage measurement results of the RFID tag and the travel encoder under the same track mileage benchmark, and forms a positioning reliability control mechanism through mileage reliability status, inspection node-level field of view constraints and mileage verification sequence, so that the subsequent multimodal detection model can generate candidate abnormal results within the verifiable field of view boundary, and block the device-level spatial attribution when positioning is abnormal.

[0064] In step S4, the core task is to input the inspection frame and field-of-view constraints into the digital twin constrained multimodal detection model, forming cross-modal candidate results through visible light feature branches, infrared feature branches, and node prior branches, and outputting candidate anomaly results with spatial attribution. In this embodiment, the visual features, temperature features, and spatial attribution features of belt edges, idler roller edges, coal heat sources, personnel targets, smoke areas, and foreign object areas in the coal conveyor corridor are interdependent. Relying solely on visible light data or infrared thermal image data is easily affected by dust, low illumination, reflection, occlusion, and background heat sources. The digital twin constrained multimodal detection model needs to simultaneously incorporate the visible light data, infrared thermal image data, and field-of-view constraints generated in step S3 into the inference process, outputting candidate anomaly results that can directly participate in the construction of anomaly persistence states.

[0065] In this embodiment, the inspection frame and field of view constraints can be input into DT-MDNet (Digital Twinconstrained Multi-modal Detection Network). This model includes a visible light feature branch, an infrared feature branch, a node prior branch, a cross-modal spatial registration layer, and a spatial attribution output layer. Input data includes visible light data and infrared thermal image data from the inspection frame, as well as the field of view constraints generated by the current inspection node; output data includes visual candidate features, thermal source candidate features, candidate region masks, cross-modal candidate results, and candidate anomaly results with spatial attribution.

[0066] In some embodiments of this application, visible light data from the inspection frame can be input into the visible light feature branch of DT-MDNet. The visible light data can be scaled to a 640×640 pixel three-channel image. The visible light feature branch has five convolutional stages with channel numbers of 32, 64, 128, 256, and 512 respectively. Each convolutional stage includes 3×3 convolution, batch normalization, and the SiLU activation function, with downsampling performed in the intermediate convolutional stages. The visible light feature branch focuses on extracting belt edge texture, idler roller outer contour, personnel contour, smoke texture, and foreign object boundaries. After processing through the convolutional stages, the visible light feature branch outputs visual candidate features, including belt edge candidate features, idler roller outer contour candidate features, personnel candidate features, smoke candidate features, and foreign object candidate features.

[0067] Simultaneously, the infrared thermal image data from the inspection frames is input into the infrared feature branch of DT-MDNet. The infrared thermal image data undergoes field-of-view registration based on the calibration relationship between the infrared thermal imager and the visible light camera, and is scaled to a single-channel thermal image of 640×640 pixels. The infrared feature branch has four convolution stages with channel numbers of 16, 32, 64, and 128 respectively, all using a 3×3 kernel and SiLU activation function. It focuses on extracting heat sources at the roller end, coal accumulation area, electrical control cabinet, and background thermal interference. After processing by the infrared feature branch, candidate heat source features are output, including the heat source center, heat source area, heat source boundary, highest temperature location, and temperature distribution relationship between the background temperature sampling area.

[0068] Simultaneously, the field-of-view constraints are input into the node prior branch of DT-MDNet. The field-of-view constraints are converted into a 640×640 pixel multi-channel node prior map. The channels of the node prior map correspond to candidate regions for belt edges, idler roller edges, heat source concern areas, personnel restricted areas, and background exclusion areas, respectively. The node prior branch has three convolutional stages with 16, 32, and 64 channels respectively, all with 3×3 kernels and ReLU activation. The node prior branch provides the spatial range that the current inspection node is allowed to identify, but it cannot replace the identification of visible light features and infrared thermal imaging features. After processing by the node prior branch, a candidate region mask is generated. Different pixel positions in the candidate region mask correspond to different candidate weights. Positions with higher weights indicate that the position belongs to the equipment area or risk concern area allowed by the current inspection node, while positions with lower weights indicate that the position is more likely to belong to the background area or the overlapping range of adjacent inspection nodes.

[0069] Furthermore, spatial registration of visual candidate features and heat source candidate features can be performed according to the candidate region mask. Spatial registration between infrared thermal image data and visible light data can be achieved using homography. The transformation of heat source points in infrared thermal image data to the visible light image plane can be written as: ,in, This represents the scaling factor of homogeneous coordinates. and These represent the x and y coordinates of the pixels after the heat source point in the infrared thermal image data is mapped onto the visible light image plane, respectively. The homography matrix represents the mapping from infrared thermal image data to visible light data, reflecting the planar mapping relationship between infrared thermal image data and visible light data. and These represent the x-coordinate and y-coordinate of the pixel at the heat source point in the infrared thermal image data, respectively.

[0070] During spatial registration, a candidate region mask limits the range of pixels participating in the registration, and the overlap relationship between the device contour in the visual candidate features and the temperature rise region in the heat source candidate features is calculated. If the heat source candidate features are processed by a homography matrix... If the mapped heat source candidate feature falls into the candidate region of the outer edge of the idler roller, and the corresponding position in the visible light data contains the outline of the outer edge of the idler roller, then the heat source candidate feature can be registered as the idler roller heat source; if the heat source candidate feature falls into the coal accumulation region and the corresponding position in the visible light data contains the outline of local coal accumulation, then the heat source candidate feature can be registered as the coal accumulation heat source; if the heat source candidate feature falls into the background exclusion region, then the heat source candidate feature will not enter the formal candidate anomaly results.

[0071] After spatial registration is completed, cross-modal candidate results can be generated. These results can include candidate target type, candidate target boundary, heat source center, candidate region mask coverage value, visible light confidence level, infrared heat source confidence level, and inspection node affiliation information. Further, based on the coverage relationship of the cross-modal candidate results within the candidate region mask, spatially affixed candidate anomaly results are generated. The candidate region mask coverage relationship can be determined using: ,in, Indicates the first The sampling time of the first sampling moment The coverage relationship value of each cross-modal candidate result relative to the candidate region mask. Indicates the first The sampling time of the first sampling moment The pixel region corresponding to each cross-modal candidate result. Represents pixel area The pixel position in Indicates the first The sampling time of the first sampling moment Candidate region mask at pixel location The mask value at that location, Represents pixel area The number of pixels within. This coverage value. Used to determine whether cross-modal candidate results are primarily located within the candidate region mask corresponding to the current inspection node. For example, coverage relationship values... When the value is greater than 0.7, cross-modal candidate results can be assigned to the equipment area corresponding to the current inspection node; coverage relationship value When the value is between 0.4 and 0.7, cross-modal candidate results can be categorized into the review attribution; coverage relation value When the value is less than 0.4, cross-modal candidate results can be excluded from the background.

[0072] In this embodiment, the training samples of the digital twin constrained multimodal detection model mainly consist of visible light data, infrared thermal image data, and node prior graphs. The training samples cover scenarios such as normal conveyor belt operation, slight belt misalignment, moderate belt misalignment, idler roller temperature rise, coal accumulation heat source, personnel entering restricted areas, smoke, foreign object obstruction, low illumination, dust obstruction, and slight lens damage. For example, the training samples can be no less than 12,000 sets, including approximately 7,000 normal samples, approximately 1,800 conveyor belt misalignment samples, approximately 1,200 heat source samples, approximately 1,000 personnel and foreign object samples, and approximately 1,000 smoke and low-quality image samples. The training set, validation set, and test set can be divided at 70%, 15%, and 15%, respectively.

[0073] Data augmentation is implemented around the coal conveying corridor scenario. Visible light data can be augmented with brightness perturbation, contrast perturbation, motion blur, dust occlusion, and local occlusion. Brightness perturbation can range from 0.6 to 1.4, motion blur convolution kernels can range from 3 to 9, and occlusion area can range from 2% to 12% of the image area. Infrared thermal imaging data can be augmented with background temperature drift, local temperature rise perturbation, and temperature noise simulation. Node prior maps are only subject to boundary perturbations, with the perturbation range not exceeding 2 pixels to avoid altering the spatial relationships of the actual equipment.

[0074] During model training, the localization branch can use CIoU (Complete Intersection over Union) loss, the class determination can use cross-entropy loss, the candidate region mask can use binary cross-entropy loss, the cross-modal candidate results can use coverage consistency loss, and the deviation distance regression can use Huber loss. For example, the Huber loss for deviation distance regression can be written as: , in, This represents the regression loss due to deviation distance. This represents the error between the belt edge distance predicted by the model and the belt edge distance marked manually. This represents the segmented threshold for the deviation distance error. This deviation distance regression loss applies fine-grained constraints to the belt edge distance when the error is small, and reduces the impact of dust occlusion and locally mislabeled samples on model parameter updates when the error is large. For example, the segmented threshold for deviation distance error... It can be set to 0.04 to 0.08 times the belt width.

[0075] The digital twin constrained multimodal detection model can be trained using the AdamW optimizer with an initial learning rate of 0.001, weight decay of 0.0005, batch size of 16, and 160 training epochs. Training is stopped when there is no performance improvement on the validation set for 20 consecutive epochs. Before deployment, INT8 quantization calibration can be performed with at least 500 calibration samples. During online operation, the network weights are not updated in real-time during the inspection process, but updates to the background temperature baseline, belt normal edge reference distance, inspection node image quality threshold, and minor equipment area offsets are allowed. After manually confirmed samples reach 1000 sets, offline retraining and testing are possible before deploying a new version.

[0076] At this point, step S4 completes the deep coupling between the inspection frame, field of view constraints, and digital twin constraints of the multimodal detection model, forming a multimodal anomaly recognition system with candidate region masks as spatial constraints, visual candidate features and heat source candidate features as spatial registration objects, cross-modal candidate results as intermediate expressions, and candidate anomaly results with spatial attribution as outputs. This provides a foundation for the identification results that are localizable, verifiable, and can be included in the event evidence chain for the subsequent construction of anomaly persistence states.

[0077] In step S5, the core task is to construct an abnormal persistence state based on candidate abnormal results and inspection frames, and to confirm the inspection event level according to the abnormal persistence state. In this embodiment, dust obstruction, changes in lighting, lens contamination, robot vibration, communication fluctuations, and positioning deviations in the coal conveying corridor can cause instability in single-frame candidate abnormal results. Directly converting single-frame candidate abnormal results into strong control actions increases the risk of false triggering. Therefore, step S5 needs to organize the cross-modal candidate results within the same inspection node into a risk change sequence according to the sampling time, and combine it with the positioning state, running state, and communication state in the inspection frame for temporal denoising reconstruction, ultimately generating the abnormal persistence state and event evidence chain.

[0078] In this embodiment, a persistent anomaly state can be constructed based on candidate anomaly results and inspection frames. Specifically, the candidate anomaly results come from step S4 and already contain spatial attribution information. The inspection frame contains positioning status, operating status, and communication status, which can be used to determine whether the candidate anomaly results are affected by positioning anomalies, operational jitter, or communication fluctuations. Based on this, according to the spatial attribution corresponding to the candidate anomaly results, cross-modal candidate results within the same inspection node are grouped into a risk change sequence based on the sampling time. The risk change sequence only collects cross-modal candidate results from the same inspection node with consistent spatial attribution, avoiding mixing candidate results from different equipment areas or background areas into continuous anomalies.

[0079] In some embodiments of this application, the risk change sequence may include candidate anomaly types, candidate region mask coverage values, visible light confidence, infrared heat source confidence, relative temperature rise, deviation distance, image quality, positioning status, operating status, and communication status. For example, for a belt misalignment event, the risk change sequence may record the distance change between the belt edge and the outer edge of the idler roller over multiple consecutive sampling times; for a heat source event, the risk change sequence may record changes in the highest temperature of the heat source, the background temperature, and the area of ​​the heat source region; for a personnel entering a restricted area event, the risk change sequence may record the duration and location changes of the personnel target within the restricted area.

[0080] Based on this, the risk change sequence and the positioning, operational, and communication states from the inspection frames are input into the temporal denoising and reconstruction process. This process can be implemented using T-CDAE (Temporal Convolutional Denoising Autoencoder), a model used to analyze recent... The system analyzes the state sequence within the current inspection node to determine whether the candidate anomaly results show a continuous deviation trend. In this embodiment, T-CDAE does not replace the target recognition results of the digital twin constrained multimodal detection model; it only performs denoising and reconstruction on cross-temporal state changes.

[0081] Specifically, the T-CDAE input can be set to The state matrix represents the recent state matrix. One set of status data per second; 12 status features including robot speed, mileage deviation between continuous mileage and tag mileage, visible light image quality, infrared image quality, wireless signal strength, battery current, obstacle distance, carbon monoxide concentration, hydrogen sulfide concentration, smoke concentration, belt misalignment stability value, and relative temperature rise of heat source; Gaussian perturbation with a standard deviation of 0.03 can be added to the input to enable T-CDAE to learn the stable change law of normal inspection state under noise conditions.

[0082] The T-CDAE encoder uses a 3-layer one-dimensional convolution: Layer 1 has 32 channels, a kernel size of 5, and ReLU activation; Layer 2 has 64 channels, a kernel size of 3, and ReLU activation; Layer 3 has 128 channels, a kernel size of 3, and ReLU activation. Max pooling of length 2 is performed after Layer 1 and Layer 2, and the pooled data is then compressed into 32-dimensional latent variables through a fully connected layer. The decoder uses a symmetric structure, first expanding the 32-dimensional latent variables into 128-channel features, then restoring them through two upsampling passes and three layers of one-dimensional convolution. The state matrix.

[0083] The goal of T-CDAE training is to reconstruct normal inspection sequences. Inspection segments that are functioning normally, experience minor environmental fluctuations, and are manually confirmed to be free of abnormalities are used for training; segments involving severe deviations, identified heat sources, personnel entering restricted areas, and obstacle-related stops are excluded from the normal training set. The training sample can contain no fewer than 300 complete inspection sequences, with each sequence containing no fewer than [number missing]. The Adam optimizer was used for training, with an initial learning rate of 0.0008, a batch size of 32, and 120 training epochs. Training was stopped when there was no improvement on the validation set for 15 consecutive epochs.

[0084] Finally, the T-CDAE outputs the reconstruction deviation result, indicating the degree to which the current state sequence deviates from the normal inspection state sequence. This can be achieved using: ,in, Indicates the first The reconstruction deviation results corresponding to each sampling time point Indicates the length of the time window for participating in the reconstruction. Indicates the number of state features. Indicates the first time window One historical sampling location, Indicates the first The sampling location is the first The actual normalized feature value of each state feature This represents the corresponding normalized feature value obtained from T-CDAE reconstruction. For example, the time window length... Set the number to 60, which represents the number of state features. Taking 12, the denominator corresponds to 720 state points.

[0085] Furthermore, anomaly persistence states can be generated based on the correspondence between the reconstruction deviation results and cross-modal candidate results. The anomaly persistence state score can be written as: ,in, Indicates the first The score of the abnormal persistence state at each sampling time. Indicates the first The average coverage relationship value corresponding to cross-modal candidate results within the same inspection node at each sampling time. Indicates the first The state reliability factor at each sampling time is formed by the location state, operation state, and communication state. , and These represent the fusion coefficients of coverage relationship, reconstruction deviation, and state credibility factor, respectively.

[0086] Based on this, the persistent anomaly status, reconstruction deviation results, and candidate anomaly results are written into the event evidence chain. For example, when a single-frame candidate anomaly result appears but the reconstruction deviation result remains within the normal reference range, the event can be recorded as a trend record or a review attribution event; when candidate anomaly results appear consecutively within the same inspection node, and the reconstruction deviation result continues to increase, the persistent anomaly status can be upgraded to a suspected anomaly or a formal anomaly. Simultaneously, the event evidence chain can include the event number, inspection node, track mileage, candidate anomaly type, candidate anomaly result, reconstruction deviation result, persistent anomaly status, image evidence, infrared thermal imaging evidence, positioning status, operational status, and communication status.

[0087] In this embodiment, the inspection event level can be determined according to the duration of the abnormality. The inspection event level can include trend records, suspected abnormal events, formal abnormal events, and high-risk abnormal events. Trend records are used for low-confidence or short-term fluctuations; suspected abnormal events are used for candidate abnormalities requiring slow-speed verification; formal abnormal events are used for abnormalities confirmed after verification; and high-risk abnormal events are used for states requiring strong control actions, such as personnel entering restricted areas, severe heat sources, severe deviations, close-range obstacles, or communication interruptions. For example, a slight deviation occurring in a single frame and the T-CDAE reconstruction deviation result being normal is recorded as a trend record; a slight deviation occurring continuously and the T-CDAE reconstruction deviation result increasing is entered into slow-speed verification; a formal heat source event is generated when the relative temperature rise of a heat source continuously increases and the mask coverage relationship of the candidate area is stable; and a high-risk abnormal event is generated when the personnel target remains continuously in a restricted area.

[0088] Thus, step S5 completes the transformation of candidate anomaly results from single-frame identification results to anomaly persistence states and event evidence chains, forming an event confirmation system based on spatial attribution as the grouping basis, risk change sequence as the temporal expression, T-CDAE reconstruction deviation results as the continuity judgment support, and inspection event level as the control entry point, providing a traceable evidence basis for the subsequent generation of inspection control results.

[0089] In step S6, the core task is to generate inspection control results based on the inspection event level and the digital twin node library, ensuring that low-speed verification, remote autonomous intervention, safe shutdown, return to charging, or task recovery are consistent with the event evidence chain, action boundaries, and verification mileage segments. In this embodiment, the coal conveying corridor inspection control cannot directly issue actions based solely on a single alarm result. When the robot is running in a narrow corridor, it is also necessary to consider the track range, verification location, obstacle distance, battery status, communication status, and task context. Therefore, the inputs to step S6 include the inspection event level, the event evidence chain, and the digital twin node library, and the outputs include inspection control results and verification-type inspection control results.

[0090] In this embodiment, inspection control results, including low-speed verification, remote autonomous intervention, safe shutdown, return to charging, or mission recovery, can be generated based on the inspection event level and the digital twin node library. Specifically, in autonomous inspection mode, the robot performs inspections according to the node sequence in the digital twin node library. For example, the robot's normal inspection speed can be set to... The robot collects data once upon reaching a regular inspection node and twice upon reaching a key inspection node. If the inspection event level is a trend record, the robot continues inspection and writes the event into the event evidence chain; if the inspection event level is a suspected abnormal event, the robot enters low-speed verification; if the inspection event level is a formal abnormal event, the robot enters remote autonomous intervention or maintains low-speed verification; if the inspection event level is a high-risk abnormal event, the robot executes a safe shutdown or controlled retreat.

[0091] In some embodiments of this application, before the control result is generated, action boundaries and verification mileage segments can be extracted from the digital twin node library based on the inspection event level, the abnormal persistence state in the event evidence chain, and the reconstructed deviation result. Action boundaries may include the allowed track range, maximum speed, parking position, gimbal horizontal angle range, gimbal pitch angle range, obstacle safety distance, minimum battery level, and communication status requirements. Verification mileage segments can be jointly determined by the inspection node where the candidate abnormal result is located, its spatial attribution, and the reconstructed deviation result in the event evidence chain. For example, the verification mileage segment for a suspected belt misalignment abnormal event can cover the area before and after the current inspection node. The verification mileage range for suspected abnormal events of heat sources can cover the low-speed parking range before and after the area of ​​the equipment corresponding to the heat source. Events of personnel entering the restricted area can trigger a safety shutdown and restrict the robot from continuing to move forward.

[0092] Based on this, when operators or the host computer input intervention actions, the remote autonomous intervention command is converted into a digital twin candidate action, which may include target track mileage, travel direction, target speed, parking position, gimbal horizontal angle, gimbal pitch angle, number of retakes, visible light retake instructions, infrared thermal image retake instructions, and return-to-home instructions. This digital twin candidate action is validated in the digital twin node library and does not directly enter the robot execution layer. Furthermore, the digital twin candidate action is validated based on action boundaries, verification mileage segments, and inspection event levels, generating execution validation results.

[0093] For example, the execution verification result indicates whether the digital twin candidate action meets the common constraints of action boundary, verification mileage segment, and inspection event level, which can be achieved by: ,in, This indicates the execution verification result of the candidate actions in the digital twin. This represents the action parameters corresponding to the candidate actions in the digital twin. Is it within the action boundary? Inside, Indicates the target orbital mileage corresponding to the candidate actions in the digital twin. Is it located in the mileage section under review? Inside, Indicates the action level corresponding to the candidate action in the digital twin. Is it related to the level of the inspection incident? match, This indicates a matching relationship where the action level does not exceed the allowable range for the inspection event level.

[0094] When the verification result indicates that the candidate action of the digital twin is within the action boundary and corresponds to the verification mileage segment, the inspection control result is generated according to the inspection event level. If the inspection event level is a suspected abnormal event, the inspection control result can be low-speed verification, where the robot reduces its speed and performs node forward and backward micro-movement, gimbal angle switching, visible light supplementary shooting, infrared supplementary shooting, RFID tag rereading, and sensor resampling within the verification mileage segment; if the inspection event level is a formal abnormal event, the inspection control result can be remote autonomous intervention, where the host computer provides candidate verification points in the digital twin scenario, and the robot performs verification according to the verified target track mileage, travel direction, and gimbal attitude; if the inspection event level is a high-risk abnormal event, the inspection control result can be safe shutdown, where the robot decelerates and brakes, saves the task context, and uploads the most recent... The system inspects frames, positioning status, visible light data, and infrared thermal image data. If the battery level is below the low battery return threshold, the inspection control result can be "return to home for charging." If the return to home for charging or a safe shutdown has ended and the mission context is valid, the inspection control result can be "mission recovery."

[0095] When the execution verification result indicates that the digital twin candidate action is not within the action boundary or does not correspond to the verification mileage segment, a verification-type inspection control result is generated based on the event evidence chain. This verification-type inspection control result can restrict the robot to only perform conservative actions such as low-speed reversal, stopping for re-shooting, gimbal repositioning, RFID tag rereading, or returning to charging. For example, if the target track mileage selected by the operator is outside the action boundary, the robot will not execute the action corresponding to that target track mileage, but will instead select the nearest safe stopping point within the verification mileage segment based on the candidate anomaly results and reconstruction deviation results in the event evidence chain. Simultaneously, the verification-type inspection control result is written into the event evidence chain. The written content may include the original remote autonomous intervention command, digital twin candidate actions, action boundaries, verification mileage segments, execution verification results, verification-type inspection control results, and execution feedback. This write-back process enables subsequent task recovery, manual review, and model sample accumulation to trace the anomaly handling chain.

[0096] In this embodiment, task resumption can be triggered by the end of return-to-base charging, communication restoration, manual release of safety shutdown, or the end of low-speed review. Before task resumption, the host computer reads the event evidence chain and task context, and verifies the digital twin node library version, remaining inspection nodes, inspection nodes to be reviewed, inspection event level, and robot battery status. If there are high-risk abnormal events that have not been reviewed, the robot first returns to the review mileage segment to perform low-speed review; if only trend records exist, the robot resumes inspection from the most recently incomplete inspection node; if the digital twin node library version has changed, the robot first downloads the updated digital twin node library, and then recalculates the action boundaries and review mileage segment.

[0097] At this point, step S6 completes the conversion of inspection event levels and digital twin node libraries into inspection control results, forming an inspection control closed loop based on event evidence chains, with action boundaries and review mileage segments as constraints, digital twin candidate actions and execution verification results as control entry points, and low-speed review, remote autonomous intervention, safe shutdown, return to charging or mission recovery as action outputs. This enables anomaly identification, event confirmation and control handling to be executed continuously within the same digital twin node system.

[0098] In summary, this method constructs a digital twin node library that matches the track mileage, inspection nodes, and equipment spatial boundaries of the coal conveying corridor. Combined with inspection frame time alignment, reliable track mileage correction, multimodal anomaly identification, anomaly persistence status discrimination, and inspection control result generation, it achieves a continuous closed loop for coal conveying corridor inspection data from collection, attribution, identification, verification to control and handling. This reduces the impact of dust obstruction, positioning deviation, field of view overlap, and single-frame false detection on anomaly judgment. It provides clear spatial attribution and persistence for risks such as belt misalignment, idler temperature rise, coal accumulation heat source, personnel entry, and smoke and foreign objects, improving the identification stability, verification accuracy, and control response reliability of coal conveying corridor inspections, thereby enhancing the operational safety and intelligent operation and maintenance level of the coal conveying system.

[0099] It should be noted that, although the embodiments in this application are based on... Figure 1 The steps are explained in turn, but this does not mean that the steps must be performed in a strict order. The step numbers are only used to distinguish different steps and do not constitute a restriction on the order of execution. The specific execution order of each step can be appropriately adjusted according to actual needs, functional requirements and the inherent logic in the actual application scenario.

[0100] In some embodiments of this application, a digital twin inspection method for coal conveying corridors is applied to track-type inspection scenarios of coal conveying corridors in thermal power plants. The effective inspection length of the coal conveying corridor is... The fixed track starting point is set as The robot inspects along the increasing track distance. The coal conveying corridor is equipped with idler rollers, coal drop points, material guide chutes, electrical control cabinets, fire monitoring points, wireless charging stations, and personnel restricted areas. The robot is equipped with visible light cameras, infrared thermal imagers, RFID card readers, travel encoders, obstacle avoidance sensors, gas sensors, and wireless communication equipment.

[0101] A complete implementation process takes place around the boundary area between the idler roller assembly and the coal drop point, where the track mileage corresponding to the idler roller assembly is [missing information]. to The track mileage corresponding to the coal drop point is to The two types of equipment areas are in to This overlap can easily lead to problems with the attribution of heat sources.

[0102] The complete implementation process may include the following steps: Step 1: Establish a digital twin node library corresponding to this inspection task. Using track mileage as the main line, divide the coal conveyor corridor into continuous inspection nodes, with each ordinary conveyor belt section... One inspection node is set up, and key areas such as idler roller assembly, coal drop point, electrical control cabinet and wireless charging station are inspected every [number missing]. Set up one inspection node. (For...) to In key areas, establish inspection nodes for idler roller groups, coal drop points, and adjacent transition mileage sections. The robot... When collecting data nearby, the visible light field of view boundary is covered. to Infrared field of view boundary coverage to Therefore, to This section has been designated as a transitional mileage segment.

[0103] In the digital twin node library, the idler group inspection node is defined by the idler outer edge boundary, the heat source concern area at the bearing end, and the belt edge reference belt; the coal drop point inspection node is defined by the coal accumulation heat source concern area, the guide chute boundary, and the smoke concern area. Based on the overlap order between the equipment area, the visible light field of view boundary, and the infrared field of view boundary within the transition mileage section, a conflict resolution rule is established. For candidate results mapped to the idler bearing end and whose visible light profile is the idler outer edge, the digital twin node library returns the primary attribution identifier; for candidate results located at the boundary between the idler end and the coal accumulation area at the coal drop point, the digital twin node library returns a verification attribution identifier; for candidate results located in background lighting reflection, corridor wall reflection, or equipment support shadow positions, the digital twin node library returns a background exclusion identifier. After completion, the continuous inspection nodes, transition mileage sections, conflict resolution rules, primary attribution identifier, verification attribution identifier, and background exclusion identifier are written into the digital twin node library, forming the spatial constraint basis for this inspection task.

[0104] Step two, the robot performs an inspection and generates a specific inspection frame. The robot starts from... Upon entering the key inspection section, the routine inspection speed is: When the robot runs to When nearby, a visible light camera captures data. Visible light data of pixels, acquired by infrared thermal imager The infrared thermal image data of the pixels is read by the RFID reader, which reads the RFID-130 tag. The stroke encoder synchronously outputs the pulse change, and the wireless communication device records the wireless signal strength. The speed of recording running status is The horizontal angle of the gimbal is The gimbal's pitch angle is Using the visible light data acquisition time as the primary time reference, The infrared thermal image data, positioning status, operating status, and communication status within the frame are merged into a single inspection frame.

[0105] The positioning status in this inspection frame points to Nearby, this orbital mileage is located within the transition mileage segment established in step one. The digital twin node library invokes the corresponding attribution conflict resolution rules to determine the spatial attribution of visible light data and infrared thermal image data in the inspection frame. The infrared thermal image data shows a temperature of... A localized heat source was observed, with the outer edge of the idler roller and the edge of the coal drop point appearing simultaneously at the same location in the visible light data. Since this heat source was located at the boundary between the end of the idler roller and the coal accumulation area at the coal drop point, the ownership conflict resolution rule output a verification ownership identifier. The robot wrote this inspection frame into the verification ownership temporary storage sequence, while retaining the original visible light data, infrared thermal image data, track mileage, gimbal attitude, and communication status.

[0106] Step 3: Correct the track mileage corresponding to this inspection frame and generate field-of-view constraints. The track mileage registered by RFID-130 in the digital twin node library is... The RFID reader is installed at the front center of the robot body. At this point, the robot is currently conducting a forward inspection, therefore the corrected mileage of the RFID tag is... The orbital mileage confirmed at the previous sampling time is: The stroke encoder pulse change is 90 within the current sampling period, and the track displacement corresponding to a single pulse is... The continuous mileage determined by the mileage measurement results of the travel encoder is The difference between tagged mileage and continuous mileage is... Smaller than the task setting Confidence threshold.

[0107] Simultaneously, the RFID tag reading order is consistent with the direction of continuous mileage change, the robot is in a low-speed verification preparation state, and the mileage reliability status is determined to be reliable. The robot calls the visible light field of view boundary and infrared field of view boundary corresponding to the current inspection node to generate a fine field of view constraint. This field of view constraint includes the candidate region of the outer edge of the idler roller, the heat source interest area at the bearing end, the heat source interest area of ​​the accumulated coal, and the background exclusion area. Since the current mileage reliability status is reliable, the candidate anomaly results are allowed to form equipment-level spatial attribution. If the tag mileage reading order and the direction of continuous mileage change conflict in subsequent sampling, the robot will write the corresponding inspection frame into the mileage verification sequence and switch the field of view constraint to the inspection node-level field of view constraint, temporarily prohibiting the candidate anomaly results from forming equipment-level spatial attribution.

[0108] Step four involves executing the digital twin constrained multimodal detection model inference for this inspection frame. In this embodiment, the digital twin constrained multimodal detection model employs a three-branch structure, including a visible light feature branch, an infrared feature branch, and a node prior branch. The outputs of these three branches enter the cross-modal spatial registration layer, and then the spatial attribution output layer generates candidate anomaly results with spatial attribution. The visible light feature branch receives the cropped and scaled... Visible light data, comprising 5 convolutional stages with channel numbers of 32, 64, 128, 256, and 512 respectively. Each convolutional stage includes... Convolution, batch normalization, and SiLU activation function are applied, with downsampling performed in the second and fourth convolution stages. This branch outputs visual candidate features corresponding to the outer edge of the idler roller, the edge of the belt, the outline of the person, the smoke texture, and the boundary of the foreign object.

[0109] The infrared feature branch receives the cropped and scaled version. Infrared thermal image data, comprising four convolutional stages with channel numbers of 16, 32, 64, and 128 respectively, each convolutional stage employing... Convolution, batch normalization, and SiLU activation function. This branch outputs candidate heat source features corresponding to the heat source at the idler end, the accumulated coal heat source, the electrical control cabinet heat source, and background thermal interference. The node prior branch receives the features obtained by the field-of-view constraint transformation. The node prior graph has five channels corresponding to the candidate regions of the belt edge, the candidate region of the idler roller outer edge, the region of interest for the heat source, the region of prohibition for personnel, and the region of exclusion for the background. The node prior branch contains three convolutional stages with 16, 32, and 64 channels respectively, and outputs a candidate region mask, which serves as a spatial constraint graph for the current inspection node to participate in the anomaly attribution determination.

[0110] During model inference, the visible light feature branch outputs one visual candidate feature of the outer edge of the idler roller in the current inspection frame, and the infrared feature branch outputs one candidate feature of the heat source, with the center temperature of the heat source being [temperature value missing]. The background temperature is The infrared heat source center, after being mapped from infrared thermal image data to visible light data, falls between the candidate region of the outer edge of the idler roller and the area of ​​interest for the coal accumulation heat source. The candidate region mask spatially registers the visual candidate features and the heat source candidate features, forming a cross-modal candidate result. The candidate target type is the heat source at the end of the idler roller, with a visual confidence score of 0.86 and an infrared heat source confidence score of 0.91.

[0111] In this embodiment, the candidate region mask coverage relationship is calculated using the aforementioned coverage relationship formula. The heat source candidate region contains 430 pixels, of which 362 pixels fall within the candidate region mask along the outer edge of the idler roller, resulting in a coverage relationship value. Since the coverage relationship value is greater than 0.7, the digital twin constrained multimodal detection model outputs candidate anomaly results with spatial attribution. The spatial attribution is the idler end region in the idler group inspection node, and the candidate anomaly type is idler end heat source.

[0112] Step 5: The digital twin constrained multimodal detection model is trained offline and deployed to this inspection task. Training samples consist of 12,600 sets, derived from 30 days of historical coal conveyor corridor inspection data, manually captured supplementary data, and manually constructed abnormal samples. Each set includes visible light data, infrared thermal image data, and node prior maps. Annotations include belt edges, idler roller outer edges, idler roller end heat sources, coal accumulation heat sources, smoke areas, foreign object areas, personnel targets, and background exclusion areas. The samples are divided into training, validation, and test sets, with proportions of 70%, 15%, and 15%, respectively. The training samples include 7,200 normal samples, 1,500 idler roller end heat source samples, 1,600 belt misalignment samples, 900 coal accumulation heat source samples, 700 samples of personnel entering restricted areas, and 700 samples of smoke and foreign object obstruction.

[0113] Data augmentation was implemented based on the actual operating conditions of a coal conveyor corridor. Visible light data underwent brightness perturbation, motion blur, dust occlusion, and local occlusion. The brightness perturbation coefficient was randomly selected between 0.6 and 1.4, the motion blur convolution kernel was randomly selected between 3 and 9, and the occlusion area was controlled between 2% and 12% of the image area. Infrared thermal image data underwent background temperature drift, local temperature rise perturbation, and thermal noise simulation. The background temperature drift range was [insert range here]. to The local temperature rise disturbance range is to The node prior graph only undergoes boundary perturbation, which is limited to no more than 2 pixels to avoid disrupting the spatial relationship between the device area and the inspection nodes. Training uses the AdamW optimizer with an initial learning rate of 0.001, weight decay of 0.0005, a batch size of 16, and 160 training epochs. Training stops when there is no performance improvement on the validation set for 20 consecutive epochs.

[0114] The model training loss includes detection box localization loss, anomaly category loss, candidate region masking loss, cross-modal coverage consistency loss, and deviation distance regression loss. For the idler end heat source scenario, the training focus is on ensuring that the infrared candidate features of the idler end heat source can be stably registered with the visual candidate features of the idler outer edge, and outputting the correct spatial attribution under the constraint of the candidate region mask. After training, the accuracy rate of idler end heat source identification in the test set was 94.2%, the accuracy rate of coal accumulation heat source identification was 91.8%, the accuracy rate of belt deviation identification was 93.5%, and the false alarm rate of background thermal interference was 3.1%. During deployment, the model weights are fixed, and only the background temperature baseline, the image quality threshold of the inspection node, and the slight offset of the equipment area are updated online. When the cumulative number of manually confirmed samples exceeds 1000 sets, a new version of the model is incrementally trained offline, and the deployed version is replaced after testing.

[0115] Step six involves inputting the candidate anomaly results into a temporal convolutional denoising autoencoder model for anomaly persistence state determination, thus enabling the temporal denoising reconstruction process. This model is input from the most recent... The state sequence within, with a sampling frequency of Each sampling point contains 12 state features: robot speed, the difference between tag mileage and continuous mileage, visible light image quality, infrared thermal image quality, wireless signal strength, battery current, obstacle distance, carbon monoxide concentration, hydrogen sulfide concentration, smoke concentration, belt misalignment stability value, and relative temperature rise of the heat source. In this embodiment, the relative temperature rise of the heat source is determined by... Gradually rise to The candidate region mask coverage relationship value is in continuous The internal values ​​are all greater than 0.78, and the wireless signal strength remains at [value missing]. to The positioning difference remains at to .

[0116] The encoder of the temporal convolutional denoising autoencoder model consists of three 1D convolutional layers: the first layer has 32 channels and a kernel size of 5; the second layer has 64 channels and a kernel size of 3; and the third layer has 128 channels and a kernel size of 3. Each layer is followed by a ReLU activation function. Max pooling of length 2 is performed after the first and second layers, and the pooled features are compressed into 32-dimensional latent variables through a fully connected layer. The decoder is symmetrical to the encoder, first expanding the 32-dimensional latent variables into 128-channel features, and then restoring them through two upsampling passes and three 1D convolutional layers. State matrix. During model training, only normal inspection sequences, manually confirmed to be free of anomalies or short-term interference, are used. A total of 300 normal training sequences are provided, each with at least [number missing]. Furthermore, a Gaussian perturbation with a standard deviation of 0.03 was added to enable the model to learn the stable variation patterns under normal inspection conditions.

[0117] In this online inference, the temporal convolutional denoising autoencoder model performs the most recent... The state sequence is reconstructed. The reconstruction deviation result is obtained using the aforementioned formula. The reconstruction deviation result of the current state sequence is 0.186, which is higher than the normal inspection state reference upper limit of 0.080. Combined with continuous... Based on the consistently occurring candidate anomaly results for the heat source at the idler roller end and the mask coverage relationship values ​​of the candidate areas, the current anomaly is determined to be in a continuously developing state. This is because the relative temperature rise of the heat source reaches... Furthermore, the infrared heat source confidence level is 0.91. The event evidence chain includes the candidate anomaly result of the heat source at the end of the idler roller, the coverage relationship value of 0.842, the reconstruction deviation result of 0.186, the anomaly persistence status, and the corresponding inspection frame. The inspection event level is confirmed as a formal anomaly event.

[0118] Step 7: Generate inspection control results based on formal anomaly events. The digital twin node library uses track mileage from the event evidence chain. The spatial allocation is determined by the "idler end area in the idler group inspection node" and the inspection event level, and the action boundary and verification mileage segment are extracted. The verification mileage segment is set as follows: to Action boundaries include robot speed not exceeding The horizontal angle range of the gimbal is to The range of the gimbal's pitch angle is to The distance to the obstacle is not less than The wireless signal strength is not lower than .

[0119] The host computer sends a remote autonomous intervention command to the robot, requesting the robot to move to... The speed is The horizontal angle of the gimbal is The gimbal's pitch angle is It performs visible light and infrared thermal imaging re-capture. This remote autonomous intervention command is translated into digital twin candidate actions. Due to the target orbital mileage... Located within the verification mileage segment, the speed, gimbal angle, obstacle distance, and wireless signal strength all meet the action boundary. The execution verification result indicates that the digital twin candidate action is located within the action boundary and corresponds to the verification mileage segment. The robot generates inspection control results in the form of a combination of low-speed verification and remote autonomous intervention.

[0120] After the robot enters the low-speed verification phase, it moves along the track to... ,exist Two visible light re-images and two infrared thermal image re-images were performed at a certain speed. The highest temperature of the heat source in the verification frame was... The background temperature is The heat source location remains in the candidate area at the end of the idler roller, with a candidate area mask coverage value of 0.858. The reconstructed deviation result recalculated by the temporal convolutional denoising autoencoder model is 0.193. The event evidence chain is updated to a verification and confirmation status, and the inspection control result is updated to formal anomaly reporting and maintained at a low speed. If the wireless signal strength drops below [a certain value] during this verification process... If the target track mileage exceeds the verification mileage segment, the execution of the verification result will trigger the verification inspection control result. The robot will only execute the stop to take a picture or the controlled retreat, and will not directly execute the remote autonomous intervention command that exceeds the action boundary.

[0121] Step 8: Complete event logging and task recovery. The robot uploads the verified event evidence chain to the host computer. The event evidence chain includes the original inspection frame, the verified inspection frame, the tag mileage, the continuous mileage, the mileage credibility status, the candidate region mask coverage relationship value, the reconstruction deviation result, the inspection event level, the inspection control result, and the execution feedback. Since this event is a formal abnormal event rather than a high-risk abnormal event, the robot continues to perform subsequent inspection tasks after passing through the idler group inspection node at low speed. If the battery level is below 20% during subsequent inspections, the robot generates a return-to-home charging control result based on the wireless charging station location in the digital twin node library. After charging is completed, the robot reads the event evidence chain and the task context, resumes inspection from the most recently incomplete inspection node, and marks the idler group inspection node as awaiting manual confirmation.

[0122] Through the complete implementation process described above, this method can continuously link track mileage, inspection nodes, visible light data, infrared thermal image data, candidate area masks, reconstruction deviation results, and inspection control results in a single coal conveying corridor inspection task. This ensures that anomalies such as heat sources at the ends of idlers are no longer limited to single-frame image recognition results, but rather have clear spatial attribution, continuous development evidence, and a control and handling process constrained by a digital twin node library. This can reduce the impact of overlapping fields of view, dust obstruction, background heat sources, positioning deviations, and communication fluctuations on inspection judgment, thereby improving the accuracy of coal conveying corridor anomaly identification, verification stability, and control reliability.

[0123] It should be understood that the step numbers identified in the above embodiments in the form of "Step 1, Step 2" are only used to distinguish different steps, and do not limit the steps to be executed in the order of these numbers. The specific execution order of each step can be adjusted according to its functional requirements and the inherent logic in the actual application scenario.

[0124] like Figure 2 As shown, the following is an embodiment of a digital twin inspection system for coal conveying corridors provided by this application. This digital twin inspection system for coal conveying corridors belongs to the same inventive concept as the digital twin inspection methods for coal conveying corridors in the above embodiments. For details not described in detail in the embodiments of the digital twin inspection system for coal conveying corridors, please refer to the embodiments of the digital twin inspection methods for coal conveying corridors described above.

[0125] Based on the same concept, another embodiment of this application provides a digital twin inspection system for coal conveying corridors, comprising: Node database module 1 is used to establish a digital twin node database including inspection nodes based on the track mileage of the coal conveying corridor. Inspection binding module 2 is used to collect the original data of inspection frames and generate inspection frames at the same sampling time, and bind the inspection frames to the inspection nodes in the digital twin node library. Mileage constraint module 3 is used to correct the track mileage of the inspection frame based on the mileage measurement results of the RFID tag and the travel encoder, and to generate the field of view constraint of the current inspection node. The multimodal detection module 4 is used to input the inspection frame and field of view constraints into the digital twin constrained multimodal detection model and output candidate anomaly results with spatial attribution; Event confirmation module 5 is used to construct an abnormal persistence state based on candidate abnormal results and inspection frames, and to confirm the inspection event level according to the abnormal persistence state; The control generation module 6 is used to generate inspection control results based on the inspection event level and the digital twin node library. The inspection control results include low-speed verification, remote autonomous intervention, safe shutdown, return to base for charging, or mission recovery.

[0126] In some embodiments of this application, the multimodal detection module 4 includes: The visual feature unit is used to input the visible light data in the inspection frame into the visible light feature branch of the digital twin constrained multimodal detection model and output visual candidate features. The infrared feature unit is used to input the infrared thermal image data in the inspection frame into the infrared feature branch of the digital twin constrained multimodal detection model and output heat source candidate features. The node prior unit is used to input the field of view constraints into the node prior branch of the digital twin constrained multimodal detection model to generate candidate region masks; The spatial registration unit is used to spatially register visual candidate features and heat source candidate features according to the candidate region mask, and generate cross-modal candidate results. The attribution generation unit is used to generate candidate anomaly results with spatial attribution based on the coverage relationship of cross-modal candidate results in the candidate region mask.

[0127] The above-disclosed embodiments are merely preferred embodiments of this application, but this application is not limited thereto. Any changes, improvements, and modifications made by those skilled in the art without departing from the principles of this application, without inventive effort, shall fall within the protection scope of this application.

Claims

1. A digital twin inspection method for coal conveying corridors, characterized in that, include: Step S1: Establish a digital twin node library including inspection nodes based on the track mileage of the coal conveying corridor; Step S2: Collect the raw data of the inspection frame and generate inspection frames at the same sampling time, and bind the inspection frames to the inspection nodes in the digital twin node library; Step S3: Correct the track mileage of the inspection frame based on the RFID tag and travel encoder mileage measurement results, and generate the field of view constraint of the current inspection node; Step S4: Input the inspection frame and field of view constraints into the digital twin constrained multimodal detection model, and output the candidate anomaly results with spatial attribution; Step S5: Construct an anomaly persistence state based on candidate anomaly results and inspection frames, and confirm the inspection event level according to the anomaly persistence state; Step S6: Generate inspection control results based on the inspection event level and the digital twin node library. The inspection control results include low-speed verification, remote autonomous intervention, safe shutdown, return to base for charging, or mission recovery.

2. The digital twin inspection method for coal conveying corridors as described in claim 1, characterized in that, Step S1 specifically includes: The coal conveying corridor is divided into continuous inspection nodes based on track mileage, and transition mileage segments are generated between adjacent inspection nodes. Based on the overlapping order between the equipment area, the visible light field of view boundary, and the infrared field of view boundary within the transition mileage section, a rule for resolving ownership conflicts is generated. Based on the rules for resolving ownership conflicts, establish the primary ownership identifier, the verification ownership identifier, and the background exclusion identifier corresponding to the overlapping range of adjacent inspection nodes; The inspection nodes, transition mileage segments, rules for resolving ownership conflicts, primary ownership identifiers, verification ownership identifiers, and background exclusion identifiers are written into the digital twin node library.

3. The digital twin inspection method for coal conveying corridors as described in claim 2, characterized in that, Step S2 specifically includes: Collect raw data of inspection frames, align the raw data of inspection frames with the same sampling time, and generate inspection frames containing visible light data, infrared thermal image data, positioning status, running status and communication status. Based on the track mileage corresponding to the positioning status in the inspection frame, the ownership conflict resolution rules are called in the digital twin node library to obtain the main ownership identifier, the verification ownership identifier, or the background exclusion identifier corresponding to the inspection frame. When an inspection frame corresponds to a primary home identifier, the inspection frame is bound to the inspection node corresponding to the primary home identifier. When an inspection frame corresponds to a review attribution identifier, the inspection frame is written into the review attribution temporary storage sequence. When an inspection frame corresponds to a background exclusion flag, the inspection frame is written into the background exclusion record sequence.

4. The digital twin inspection method for coal conveying corridors as described in claim 3, characterized in that, Step S3 specifically includes: The tag mileage obtained from RFID tag readings and the continuous mileage determined by the travel encoder mileage measurement results are converted to the same track mileage reference. A reliable mileage status is generated based on the reading order of the tag mileage, the direction of change of continuous mileage, and the running status in the inspection frame; When the reading order of the mileage trust status characterization tag mileage, the direction of change of continuous mileage and the running status all correspond to the current inspection node, the visible light field of view boundary and infrared field of view boundary corresponding to the current inspection node are called to generate the field of view constraint. When the reading order of the mileage trust status characterization tag mileage, the direction of change of continuous mileage, or the running status does not correspond to the current inspection node, the field of view constraint is switched to the inspection node level field of view constraint, and a mileage verification mark is generated.

5. The digital twin inspection method for coal conveying corridors as described in claim 4, characterized in that, The steps for generating mileage verification markers include: The inspection frames that trigger the mileage verification mark are written into the mileage verification sequence according to the sampling time. The mileage verification markers are updated based on the correspondence between the reading order of the mileage tags in the mileage verification sequence and the direction of change of the continuous mileage. When the corresponding relationship points to the same inspection node, remove the mileage verification mark and restore the field of view constraint of the current inspection node; When the correspondence points to different inspection nodes, maintain the mileage verification mark and prohibit candidate abnormal results from forming equipment-level spatial attribution.

6. The digital twin inspection method for coal conveying corridors as described in claim 3, characterized in that, Step S4 specifically includes: The visible light data in the inspection frame is input into the visible light feature branch of the digital twin constrained multimodal detection model, and visual candidate features are output. The infrared thermal image data in the inspection frame is input into the infrared feature branch of the digital twin constrained multimodal detection model, and the heat source candidate features are output. The field-of-view constraints are input into the node prior branches of the digital twin constrained multimodal detection model to generate candidate region masks; Spatial registration of visual candidate features and heat source candidate features is performed based on the candidate region mask to generate cross-modal candidate results; Candidate anomalies with spatial attribution are generated based on the coverage relationship of cross-modal candidate results in the candidate region mask.

7. The digital twin inspection method for coal conveying corridors as described in claim 6, characterized in that, Step S5, which involves constructing the persistent anomaly state based on the candidate anomaly results and the inspection frame, specifically includes: Based on the spatial attribution of the candidate anomaly results, the cross-modal candidate results within the same inspection node are formed into a risk change sequence according to the sampling time; The risk change sequence and the positioning status, operation status and communication status in the inspection frame are input into the time-series denoising and reconstruction process to generate reconstruction deviation results. An abnormal persistent state is generated based on the correspondence between the reconstruction deviation results and the cross-modal candidate results; Write the persistent abnormal state, the reconstructed deviation result, and the candidate abnormal result into the event evidence chain.

8. The digital twin inspection method for coal conveying corridors as described in claim 7, characterized in that, Step S6, which generates inspection control results based on the inspection event level and the digital twin node library, specifically includes: Based on the inspection event level and the abnormal persistence status in the event evidence chain, as well as the reconstruction deviation results, the action boundary and the verification mileage segment are extracted from the digital twin node library; The remote autonomous intervention command is converted into a digital twin candidate action. The digital twin candidate action is verified based on the action boundary, the review mileage segment and the inspection event level, and the execution verification result is generated. When the execution verification result indicates that the digital twin candidate action is within the action boundary and corresponds to the verification mileage segment, the inspection control result is generated according to the inspection event level. When the execution verification result indicates that the digital twin candidate action is not within the action boundary or does not correspond to the verification mileage segment, a verification inspection control result is generated based on the event evidence chain, and the verification inspection control result is written into the event evidence chain.

9. A digital twin inspection system for coal conveying corridors, characterized in that, include: The node database module is used to build a digital twin node database, including inspection nodes, based on the track mileage of the coal conveying corridor. The inspection binding module is used to collect the raw data of the inspection frame and generate inspection frames at the same sampling time, and bind the inspection frame to the inspection node in the digital twin node library. The mileage constraint module is used to correct the track mileage of the inspection frame based on the mileage measurement results of the RFID tag and the travel encoder, and to generate the field of view constraint of the current inspection node. The multimodal detection module is used to input the inspection frames and field of view constraints into the digital twin constrained multimodal detection model and output candidate anomaly results with spatial attribution; The event confirmation module is used to construct the anomaly persistence status based on candidate anomaly results and inspection frames, and to confirm the inspection event level according to the anomaly persistence status. The control generation module is used to generate inspection control results based on the inspection event level and the digital twin node library. The inspection control results include low-speed verification, remote autonomous intervention, safe shutdown, return to base for charging, or mission recovery.

10. The digital twin inspection system for coal conveying corridors as described in claim 9, characterized in that, The multi-mode detection module includes: The visual feature unit is used to input the visible light data in the inspection frame into the visible light feature branch of the digital twin constrained multimodal detection model and output visual candidate features. The infrared feature unit is used to input the infrared thermal image data in the inspection frame into the infrared feature branch of the digital twin constrained multimodal detection model and output heat source candidate features. The node prior unit is used to input the field of view constraints into the node prior branch of the digital twin constrained multimodal detection model to generate candidate region masks; The spatial registration unit is used to spatially register visual candidate features and heat source candidate features according to the candidate region mask, and generate cross-modal candidate results. The attribution generation unit is used to generate candidate anomaly results with spatial attribution based on the coverage relationship of cross-modal candidate results in the candidate region mask.

Images