Material safe transportation method for deep-buried station

Abstract

The application discloses a kind of deep-buried station material safe transport method, the multimodal image of deep-buried station transport scene is collected, the multimodal image is preprocessed and fused, fusion image is generated, the fusion image is input into multi-task joint perception network and is perceived, and perception result is output, according to the continuous frame sequence of the multimodal image, time series image sequence is extracted, according to the time series image sequence, the motion trend of each transport equipment is predicted, the collision risk between equipment is calculated, the structural anomaly of transport path is detected according to the real-time collected path image, and path health assessment result is output, according to the perception result, the collision risk and the path health assessment result, generate safety situation map, and early warning is carried out according to risk level.The application is through image acquisition, fusion enhancement, multi-task perception, situation prediction to path health assessment and comprehensive early warning, and the safety of deep-buried station material transport is comprehensively guaranteed.

Description

Technical Field

[0001] This invention belongs to the field of urban rail transit construction technology, and in particular relates to a method for safe transportation of materials in deeply buried stations. Background Technology

[0002] Deep-buried stations typically refer to underground stations with significant burial depths, such as Xiaoshizi Station and Qixinggang Station in a certain urban rail transit system, where the deepest burial depth reaches 92.94 meters. Material transportation in these stations faces unique challenges, including confined spaces, rugged passageways, large vertical differences, and overlapping operations by multiple trades. Therefore, material transportation safety is a critical issue in construction management.

[0003] Currently, material transport safety technologies in deeply buried railway stations mainly rely on single sensor data (such as track pressure sensors and speed sensors) or manual inspection methods. Some locomotives are equipped with safety management systems such as automatic track gauge measurement and automatic speed limiting on slopes, which can provide early warnings for speeding, runaway, and other situations. BIM technology is also used to plan material transport routes in advance and to mitigate known risks by simulating the construction process. In terms of visual monitoring, there are solutions that use visible light and infrared image fusion for foreign object detection, but these are mainly designed for ground-based rail transit scenarios.

[0004] However, the extreme environmental conditions of deeply buried railway stations present unique challenges to the safety monitoring of material transportation. Deeply buried tunnels experience extremely low illumination (often below 50 lux) and high dust concentrations, resulting in low signal-to-noise ratios and significant loss of detail in conventional visible light images. Existing fusion methods are not specifically optimized for these extreme conditions. Furthermore, transport equipment such as belt conveyors and railcars may experience abnormal conditions during operation, such as material imbalances, spillage, and exceeding limits. The overlapping operation of multiple types of transport equipment in confined spaces also poses a risk of collisions and interference. In addition, sudden obstacles along the transport path (such as fallen materials, equipment displacement, water accumulation, and track deformation) cannot be detected in real time using static BIM planning, and automated inspection methods for the health status of the path structure are lacking. These issues make it difficult for existing technologies to comprehensively guarantee the safety of material transportation in deeply buried railway stations. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for safe transportation of materials in deeply buried railway stations, addressing the shortcomings of the prior art.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for safe transportation of materials in deeply buried railway stations, comprising the following steps: Step 1: Acquire multimodal images of the transportation scene at the deep-buried station, including visible light images, infrared thermal imaging images, and depth images; Step 2: Preprocess and fuse the multimodal images to generate a fused image; Step 3: Input the fused image into the multi-task joint perception network for perception, and output the perception results, which include material status information, equipment spatial pose information, path obstacle information, and personnel behavior information; Step 4: Extract time-series image sequences from the continuous frame sequences of the multimodal images, predict the motion trends of each transportation device based on the time-series image sequences, and calculate the collision risk between devices; Step 5: Detect structural anomalies in the transportation route based on the real-time acquired route images, and output the route health assessment results; Step 6: Generate a safety situation map based on the perception results, the collision risk, and the path health assessment results, and issue warnings according to the risk level.

[0007] Furthermore, in step 2, the multimodal images are preprocessed and fused, including: The visible light image is subjected to low-light enhancement processing; The enhanced visible light image, infrared thermal image, and depth image are input into a multi-scale feature extraction network; An attention-guided mechanism is used to dynamically allocate fusion weights based on the quality of modal features at each pixel location, enabling adaptive pixel-level fusion.

[0008] Furthermore, the low-light enhancement process includes: The visible light image is decomposed into an illumination layer and a reflection layer; The texture structure of the infrared thermal imaging image is used as a guiding prior to guide the illumination layer to make adaptive adjustments in the dark area; The attention guidance mechanism includes: The fusion weights of each modality are dynamically determined based on the sharpness and information richness of each modal feature at each pixel location; Increase the fusion weight of infrared thermal imaging features in areas obscured by dust; Increase the fusion weight of visible light features in areas with rich texture.

[0009] Furthermore, the preprocessing and fusion of the multimodal images also includes: Real-time quality scoring of the fused image is performed using a no-reference image quality assessment subnetwork. When the score falls below the set threshold, the collection parameters are automatically adjusted.

[0010] Furthermore, in step 3, the material status information is obtained through the material status recognition branch of the multi-task joint sensing network. The material status recognition branch detects the material outline on the transport equipment and calculates the area difference ratio between the left and right sides of the material. When the difference ratio exceeds a preset safety threshold, it is determined to be an off-center load risk. The material status recognition branch also detects whether the material exceeds the boundary of the transport container and whether there is any scattered material falling into the transport channel.

[0011] Furthermore, in step 3, the spatial pose information of the equipment is obtained through the three-dimensional spatial positioning branch of the multi-task joint perception network. The three-dimensional spatial positioning branch combines the depth image to perform three-dimensional bounding box regression on each transportation device, and outputs the position and orientation of each transportation device in the station spatial coordinate system. The path obstacle information is obtained through the path obstacle detection branch of the multi-task joint perception network. The path obstacle detection branch performs pixel-level semantic segmentation on the ground area of ​​the transportation path, and identifies and locates fallen materials, equipment displacement, water accumulation and landslide obstacles.

[0012] Furthermore, in step 3, the personnel behavior information is obtained through the personnel safety behavior analysis branch of the multi-task joint perception network. The personnel safety behavior analysis branch extracts the skeletal posture of the workers based on human key point detection and identifies unsafe behaviors such as illegally crossing transportation channels and entering dangerous areas.

[0013] Furthermore, in step 4, the movement trend of each transportation device is predicted, including: Motion features of each transportation device are extracted from the time-series image sequence, and the motion features are input into the time-series model to predict the position of each device at a future time. The motion features include position, speed, acceleration, and direction of travel. The time series model is a lightweight time series model based on the attention mechanism. Spatial constraint rules are added during the prediction process to force the predicted trajectory to conform to the physical boundaries of the station passage.

[0014] Furthermore, in step 4, the collision risk is determined based on the minimum expected distance and expected approach time between any two devices. When the minimum expected distance is lower than the safety interval or the expected approach time is shorter than the threshold, a graded warning signal is generated according to the risk level.

[0015] Furthermore, in step 5, structural anomalies in the transportation route are detected, including: The real-time acquired path image is input into the path state anomaly detection network to obtain the reconstructed image, and the difference between the real-time path image and the reconstructed image is calculated. When the difference exceeds the set threshold, the path is determined to be abnormal and a difference heatmap is generated to display the abnormal area. The abnormal regions detected are identified by a classification subnetwork to determine the type of abnormality, and the severity level is assessed based on the area and morphology of the abnormal regions. The path state anomaly detection network adopts a generative adversarial network architecture, which reconstructs the path images collected in real time into reconstructed images under normal conditions.

[0016] Compared with the prior art, the present invention has the following advantages: This invention deploys an enhanced low-light visible light camera, an infrared thermal imaging camera, and a depth camera, acquiring complementary information through a multimodal acquisition system. The visible light image is decomposed into an illuminance layer and a reflection layer. The texture structure of the infrared thermal imaging image is introduced as a guiding prior, guiding the illuminance layer to adaptively adjust in dark areas, effectively improving brightness in dark areas while avoiding over-enhancement that could amplify noise. During the fusion stage, an attention-guided mechanism dynamically allocates fusion weights based on the sharpness and information richness of each modal feature at each pixel location. In dust-covered areas, the weight of infrared thermal imaging features is automatically increased to penetrate dust and obtain the target outline; in texture-rich areas, the weight of visible light features is automatically increased to preserve details. The combined use of an active-passive acquisition strategy and adaptive fusion methods enables this invention to maintain stable sensing capabilities even in extreme environments with illuminance as low as 10 lux and high dust concentrations.

[0017] This invention also incorporates a closed-loop feedback step for fusion quality. A no-reference image quality assessment subnetwork evaluates the brightness, contrast, and structural fidelity of the fusion result in real time. When the score falls below a threshold, the camera exposure, gain, and other acquisition parameters are automatically adjusted, forming a dynamic closed loop of acquisition, fusion, and optimization. This ensures that the quality of the fused image continuously meets the requirements of subsequent analysis, thereby improving the engineering robustness of the method at the system level.

[0018] This invention constructs a multi-task joint sensing network. Through an architecture design that shares a backbone network and branches out into multiple task-specific branches, a single network can simultaneously execute four tasks. The material status recognition branch detects the material outline, identifies off-center loading risks by calculating the area difference ratio between the left and right sides, and simultaneously detects whether the material exceeds the container boundary and whether there is any scattered material falling. The 3D spatial positioning branch combines depth images to perform 3D bounding box regression, outputting the precise position and orientation of each device in the station's spatial coordinate system. The path obstacle detection branch performs pixel-level semantic segmentation of the ground along the transport path, identifying and locating obstacles such as fallen materials, equipment displacement, water accumulation, and landslides. The personnel safety behavior analysis branch extracts the skeleton posture of workers in real time based on human key point detection, identifying unsafe behaviors such as unauthorized crossings and entering dangerous areas. The four branches share basic feature calculations, balancing the comprehensiveness of the sensing dimensions with computational efficiency, achieving integrated real-time monitoring of material off-center loading, equipment orientation, path obstacles, and personnel behavior.

[0019] This invention extracts motion features such as position, velocity, acceleration, and direction of travel of each transportation device from continuous frame sequences of multimodal images. It then predicts the future position of each device using a lightweight temporal model based on an attention mechanism. Spatial constraint rules are incorporated into the prediction process to force the predicted trajectory to conform to the physical boundaries of the station passageway, avoiding unreasonable paths. The minimum estimated distance and estimated approach time between any two devices are calculated based on the predicted positions. When warning conditions are met, graded warning signals are generated according to risk level. This technique enables the system to issue warnings seconds before a collision occurs, providing dispatchers with time to intervene in advance.

[0020] This invention constructs a path state anomaly detection network based on generative adversarial networks (GANs). A normal state reference image library is established in the initial stage of construction when the path is intact. During real-time detection, the path image is input into the GAN to reconstruct a normal state image. Anomalies are detected by calculating pixel-level differences between the reconstructed image and the original image, and a difference heatmap is generated to visually display the specific location of the anomaly area. Furthermore, a classification subnetwork identifies anomaly types such as track deformation, road surface cracking, water accumulation, and foreign object coverage. Based on the area and morphology of the anomaly area, it is assessed into three levels: minor, moderate, and severe, providing a basis for maintenance decisions. This achieves automated, all-weather detection of transportation path structural anomalies without human intervention.

[0021] This invention forms a complete technical closed loop with clear data flow and close module collaboration, encompassing multimodal image acquisition, adaptive fusion enhancement, multi-task joint perception, temporal situation prediction, path health assessment, and comprehensive decision-making early warning. The comprehensive decision-making module integrates the outputs of each module to generate a safety situation map, providing graded early warnings based on four levels: green, yellow, orange, and red. It also synchronously outputs specific scheduling suggestions such as deceleration, avoidance, or shutdown according to the risk type and level. The various technical means support each other and are organically connected, jointly constituting a full-chain protection system from environmental perception to situation prediction, and from anomaly detection to decision output.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0024] like Figure 1 As shown, the present invention provides a method for safe transportation of materials in a deeply buried railway station, comprising the following steps: Step 1: Acquire multimodal images of the transportation scene at the deep-buried station, including visible light images, infrared thermal imaging images, and depth images; Step 2: Preprocess and fuse the multimodal images to generate a fused image; Step 3: Input the fused image into the multi-task joint perception network for perception, and output the perception results, which include material status information, equipment spatial pose information, path obstacle information, and personnel behavior information; Step 4: Extract time-series image sequences from the continuous frame sequences of the multimodal images, predict the motion trends of each transportation device based on the time-series image sequences, and calculate the collision risk between devices; Step 5: Detect structural anomalies in the transportation route based on the real-time acquired route images, and output the route health assessment results; Step 6: Generate a safety situation map based on the perception results, the collision risk, and the path health assessment results, and issue warnings according to the risk level.

[0025] Before implementing the method of this invention, a system deployment is first required in the transportation scenario of a deeply buried railway station. Multimodal image acquisition units are deployed at key locations in the station's transportation channels, including material loading and unloading areas, track intersections, turning sections, ramps, and equipment cross-operation areas. Each acquisition unit includes three cameras: an enhanced low-light visible light camera (minimum illumination down to 0.01 lux) for acquiring visible light images; an infrared thermal imaging camera (preferably with a resolution of 640×512) for acquiring infrared thermal imaging images; and a depth camera (which can be a ToF camera or a structured light camera, with a detection distance of 0.5~20 meters) for acquiring depth images. Each camera achieves frame-synchronized acquisition through a time synchronization trigger to ensure that the multimodal images acquired at the same time are strictly aligned in time, with the acquisition frame rate set to 30 frames per second. Each acquisition unit is connected via an industrial Ethernet connection to a computing server deployed in the station's monitoring room. This server is equipped with a GPU accelerator card for running image processing and deep learning models. The following example, using a material transportation scenario at a deeply buried subway station (approximately 50 meters deep, with a passage illuminance of approximately 30 lux and high dust concentration during construction), will be used to illustrate the method of this invention in detail.

[0026] Step 1: Acquire multimodal images of the transportation scene at the deep-buried station. The multimodal images include visible light images, infrared thermal imaging images, and depth images (multimodal image acquisition).

[0027] After system deployment, the multimodal image acquisition unit is activated for synchronous acquisition, obtaining visible light images, infrared thermal imaging images, and depth images. In the environment of deeply buried railway stations, the illuminance of the passageway is usually below 50 lux, and in some areas even below 10 lux. Simultaneously, construction dust further reduces the quality of visible light imaging, making it difficult to obtain clear images relying solely on a visible light camera. Therefore, this invention employs a multimodal acquisition strategy—the visible light camera is responsible for acquiring the texture details of the scene, the infrared thermal imaging camera obtains the target contour and position information by detecting the thermal radiation of objects, and the depth camera directly measures the distance from each point in the scene to the camera. The information from these three modalities complements each other, providing a rich data foundation for subsequent fusion.

[0028] By simultaneously acquiring multimodal images, this method can supplement environmental perception data with infrared thermal imaging and depth information even under poor visible light imaging conditions. This solves the problem of insufficient information from a single visual sensor in the extreme environment of deeply buried stations from the source, laying a data foundation for subsequent image fusion and intelligent perception.

[0029] Step 2: Preprocess and fuse the multimodal images to generate a fused image (image preprocessing and fusion).

[0030] The acquired multimodal images suffer from quality issues due to the extreme environmental conditions of the deeply buried station—visible light images have low illumination and high noise, infrared images lack texture details, and depth images contain holes and noise. Directly using any single modality image is insufficient to meet the requirements of subsequent perception tasks. Therefore, this step integrates the information from the three modalities through preprocessing and fusion to generate a fused image containing rich texture information and accurate spatial information, providing high-quality input for subsequent multi-task perception.

[0031] Image fusion combines the advantages of visible light images in terms of texture detail, infrared thermal imaging in terms of dust penetration resistance, and depth images in terms of spatial distance information. It is significantly superior to any single modality image in terms of illumination, contrast, and information richness, thus ensuring the processing quality of subsequent steps from the source.

[0032] The preprocessing and fusion of the multimodal images includes: The visible light image is subjected to low-light enhancement processing; The enhanced visible light image, infrared thermal image, and depth image are input into a multi-scale feature extraction network; An attention-guided mechanism is used to dynamically allocate fusion weights based on the quality of modal features at each pixel location, enabling adaptive pixel-level fusion.

[0033] The low-light enhancement process includes: The visible light image is decomposed into an illumination layer and a reflection layer; The texture structure of the infrared thermal image is used as a guiding prior to guide the illumination layer to make adaptive adjustments in the dark area.

[0034] The attention guidance mechanism includes: The fusion weights of each modality are dynamically determined based on the sharpness and information richness of each modal feature at each pixel location; Increase the fusion weight of infrared thermal imaging features in areas obscured by dust; Increase the fusion weight of visible light features in areas with rich texture.

[0035] For low-illuminance visible light images, direct fusion would suffer from insufficient dark area information, affecting fusion quality. Therefore, this embodiment first performs low-illuminance enhancement processing. Specifically, an improved Retinex decomposition method is used to decompose the visible light image into an illuminance layer and a reflection layer. The core assumption of Retinex theory is that an image can be represented as the product of an illuminance component and a reflection component, where the illuminance component reflects the lighting conditions and the reflection component reflects the reflective properties of the object's surface. After decomposition, the illuminance layer reflects the distribution of ambient light in different areas of the image, while the reflection layer contains the texture and edge information of the object. Conventional Retinex enhancement methods only globally adjust the illuminance layer, which can easily lead to excessive amplification of noise in dark areas under extremely low-illuminance conditions in deeply buried stations. To overcome this deficiency, this embodiment extracts texture structure information (such as edge position, gradient direction, etc.) from the infrared thermal imaging image as a guiding prior. In areas where structural edges are visible in the infrared image, the illuminance layer is guided to moderately increase the illuminance while maintaining edge clarity. In uniform dark areas, the illuminance is smoothly increased to avoid noise amplification.

[0036] After enhancement, the enhanced visible light image, infrared thermal image, and depth image are input into a multi-scale feature extraction network. This network employs three parallel encoder branches, each extracting feature maps of its corresponding modality at the original, half-size, and quarter-size dimensions, thereby simultaneously capturing both detailed texture information and overall structural information of the image. After obtaining the multi-scale features of each modality, the attention-guided fusion stage begins. The attention-guided mechanism dynamically determines the fusion weights based on the sharpness (characterized by calculating local variance, with larger local variance indicating richer texture) and information richness (characterized by signal strength) of each modality's features at each pixel location. In dust-occluded areas, the visible light image is severely degraded due to dust scattering, with both sharpness and information richness significantly reduced. The attention mechanism automatically increases the fusion weight of the infrared thermal imaging features because infrared thermal imaging can penetrate a certain concentration of dust to obtain the target's thermal radiation information. In texture-rich areas with relatively sufficient illumination and less dust, the high weight of visible light features is maintained to preserve detailed texture. Depth images have an irreplaceable advantage in providing spatial distance information and always maintain an appropriate weight in tasks involving device localization and 3D obstacle perception.

[0037] The aforementioned technical solution combines infrared texture-guided low-light enhancement with attention-driven multimodal adaptive fusion to address the issues of poor image quality and insufficient single-modal information in conventional visible light images under extreme low-light and high-dust environments in deeply buried railway stations. Infrared-guided priors ensure that the enhanced visible light image effectively improves brightness in dark areas while avoiding excessive noise amplification. Attention-guided adaptive fusion allows the system to rely on infrared features to penetrate dust and obtain target contours in dust-covered areas, while relying on visible light features to preserve details in texture-rich areas, achieving optimal fusion results under different environmental conditions. Through this step, the fused image significantly outperforms the single-modal image in terms of illumination, contrast, and detail richness, enabling the method to maintain stable sensing capabilities even in extreme environments with illumination as low as 10 lux and high dust concentrations. The high-quality fused image provides reliable input for the subsequent multi-task joint sensing in step 3, serving as the fundamental guarantee for the entire technical chain.

[0038] Meanwhile, the preprocessing and fusion of the multimodal images also includes: Real-time quality scoring of the fused image is performed using a no-reference image quality assessment subnetwork. When the score falls below the set threshold, the collection parameters are automatically adjusted.

[0039] After obtaining the fused image, this embodiment also uses a no-reference image quality assessment subnetwork to perform real-time quality evaluation of the fused image. "No-reference" means that this assessment subnetwork can score the brightness suitability, contrast, and structural fidelity of the fused image without referencing a real, high-quality image, enabling it to operate in real-time at the construction site. The scoring uses a 100-point scale, and this embodiment pre-sets a quality threshold of 75 points. When the score is below this threshold, it indicates that the fused image quality does not meet the requirements of subsequent sensing tasks, and the system automatically triggers adjustments to the acquisition parameters—prioritizing adjustments to the exposure time of the visible light camera to increase light intake and adjusting the gain to moderately enhance signal strength; if the score still does not meet the standard after adjustments, the gain setting of the infrared camera is further adjusted. For example, in this embodiment, a sudden dust storm during construction caused the fused image quality score to drop to 68 points. The system extended the camera exposure time from 10ms to 15ms, and after the dust dissipated, the score rebounded to 79 points.

[0040] Technical Results: Through the aforementioned closed-loop feedback of fusion quality, the acquisition parameters are dynamically adjusted according to environmental changes, forming a dynamic closed loop of acquisition-fusion-optimization. This mechanism ensures that even with drastic changes in environmental conditions (such as sudden construction dust storms or lighting equipment malfunctions), the quality of the fused image continues to meet the requirements of subsequent analysis. This improves the engineering robustness of the method at the system level, making it more adaptable and reliable in complex and ever-changing construction sites.

[0041] Step 3: Input the fused image into the multi-task joint perception network for perception and output the perception results, which include material status information, equipment spatial pose information, path obstacle information and personnel behavior information (multi-task joint perception).

[0042] The fused image generated in step 2 is input into a multi-task joint perception network. This network is built on a target detection framework (such as the YOLO series). Its core architectural feature is that a shared backbone network extracts basic features, and then multiple task-specific branches are generated. The shared backbone network ensures that the basic features only need to be calculated once, significantly reducing computational costs. At the same time, each task branch performs specialized learning based on the shared features, forming implicit information sharing between tasks, which helps improve the detection accuracy of each branch. Through this network, a single model can simultaneously output four types of information: material status, equipment pose, path obstacles, and personnel behavior, achieving comprehensive perception of all elements of transportation safety.

[0043] The multi-task joint sensing network completes four sensing tasks simultaneously using a single network, balancing comprehensive sensing dimensions with computational efficiency, and avoiding the resource waste and increased latency of deploying models separately for each task. The structured sensing results output by this network will be transmitted to the integrated decision-making module in step 6, and will also provide device detection results as a basis for situation prediction in step 4.

[0044] The material status information is obtained through the material status recognition branch of the multi-task joint sensing network. The material status recognition branch detects the material outline on the transport equipment and calculates the area difference ratio between the left and right sides of the material. When the difference ratio exceeds a preset safety threshold, it is determined to be an off-center load risk. The material status recognition branch also detects whether the material exceeds the boundary of the transport container and whether there is any scattered material falling into the transport channel.

[0045] In the material state recognition branch, the slag on the conveyor belt is used as the detection target. The network outputs the detection bounding box and segmentation mask of the material. The detection bounding box is used to locate the position of the material in the image, while the segmentation mask precisely outlines the contour range of the material. Based on the segmentation mask, the pixel area difference ratio of the material on the left and right sides of the conveyor belt is calculated. In this embodiment, the off-center load safety threshold is set to 20%. When the difference ratio exceeds 20%, it is judged as an off-center load risk. In a certain detection, the calculated difference ratio was 23%, triggering an off-center load warning. In addition, this branch also detects whether the material exceeds the container boundary by comparing the positional relationship between the material contour and the boundary of the transport container (over-limit detection), and determines whether there is any scattered material falling into the transport channel by detecting whether there is any material in the ground area below the conveyor belt (scattering detection).

[0046] Through a material status recognition branch, this method achieves simultaneous real-time visual detection of three abnormal material states: off-center loading, exceeding limits, and spillage. Off-center loading is a common safety hazard in material transportation, which can lead to the overturning of transport equipment in severe cases. Real-time detection can issue early warnings at the initial stage of off-center loading. Oversized materials may scrape against fixed facilities on both sides of the passage during transportation; spilled materials may cause subsequent equipment to bump or even derail, or trip up on-site workers. This branch fills the technical gap of lacking visual perception of material status in existing belt conveyor systems, effectively ensuring loading safety during material transportation.

[0047] The spatial pose information of the equipment is obtained through the three-dimensional spatial positioning branch of the multi-task joint perception network. The three-dimensional spatial positioning branch combines the depth image to perform three-dimensional bounding box regression on each transportation device and outputs the position and orientation of each transportation device in the station spatial coordinate system. The path obstacle information is obtained through the path obstacle detection branch of the multi-task joint perception network. The path obstacle detection branch performs pixel-level semantic segmentation on the ground area of ​​the transportation path to identify and locate fallen materials, equipment displacement, water accumulation, and landslide obstacles.

[0048] The 3D spatial localization branch combines depth images to perform 3D bounding box regression for each transport device. This predicts the center point coordinates, length, width, height, and spatial orientation of the smallest cuboid enclosing the device in 3D space, thereby determining the device's precise position (x, y, z coordinates) and attitude (pitch, yaw, roll) in the station's spatial coordinate system. The depth image provides the distance from each pixel to the camera, enabling the network to recover 3D spatial information from 2D images. To further improve positioning accuracy, this embodiment also utilizes geometric constraints between multi-angle cameras for error correction—the same device may be observed simultaneously by multiple acquisition units. Perspective projection consistency verification verifies whether the 3D bounding box projected onto the imaging plane of each camera under each viewpoint matches the actual detection result, effectively correcting positioning deviations caused by occlusion or depth measurement errors under a single viewpoint. In this embodiment, the positioning output of a certain railcar is (X=125.3m, Y=8.7m, Z=-42.5m), and its attitude is (pitch 2.3°, yaw -1.5°, roll 0.8°). The obstacle detection branch performs pixel-level semantic segmentation on the ground area of ​​the transportation path, classifying each pixel in the fused image to determine whether it belongs to an obstacle and what type of obstacle it is. The advantage of semantic segmentation over object detection is its ability to accurately depict the boundary contours of obstacles, which is particularly effective for irregularly shaped obstacles (such as scattered piles of slag). In this embodiment, this branch identifies a fallen pipe segment approximately 15 meters ahead of the track, outputting its category as "scattered material," with a spatial range of approximately 0.8m × 0.6m × 0.4m.

[0049] The 3D spatial positioning branch overcomes the limitation of existing technologies that can only sense the existence of devices but cannot accurately locate them, providing precise pose input for the collision risk calculation in the subsequent step 4. The path obstacle detection branch achieves real-time pixel-level identification of sudden obstacles on the transportation path, making up for the deficiency of BIM static planning in dealing with dynamic sudden obstacles. The two branches share the backbone network calculation with the material status recognition branch and the personnel safety behavior analysis branch, taking into account both the comprehensiveness of the perception dimensions and the computational efficiency.

[0050] The personnel behavior information is obtained through the personnel safety behavior analysis branch of the multi-task joint perception network. The personnel safety behavior analysis branch extracts the skeletal posture of the workers based on human key point detection and identifies unsafe behaviors such as illegally crossing transportation channels and entering dangerous areas.

[0051] The personnel safety behavior analysis branch extracts the skeletal posture of workers based on human keypoint detection technology. Specifically, an improved HRNet architecture is used to detect 17 key points of the human body (including head, shoulders, elbows, wrists, hips, knees, and ankles) from the fused image. The connections between these key points form a human skeletal model. Based on the skeletal model, the posture and movement intention of the workers can be determined. When a worker's skeleton is detected entering the restricted area of ​​the transport channel (i.e., within the operating range of the track or conveyor belt) and its direction of movement intersects with the direction of travel of the transport equipment, it is determined as an unauthorized crossing and an immediate alarm is triggered; an alarm is also triggered when a person enters a restricted area for equipment operation. No unauthorized personnel behavior was detected during the operation of this embodiment.

[0052] This branch, based on the human skeleton model, achieves a refined understanding of the behavior of workers, can distinguish between normal operations and dangerous behaviors, provides effective protection for personnel safety in multi-equipment cross-operation environments, and fills the gap in existing technologies that lack intelligent monitoring of personnel behavior.

[0053] Step 4: Extract time-series image sequences from the continuous frame sequences of the multimodal images, predict the motion trends of each transportation device based on the time-series image sequences, and calculate the collision risk between devices (time-series situation prediction and collision risk calculation).

[0054] Step 3 provides the current perception results, but safe transportation requires not only knowing "what is happening now" but also predicting "what will happen next." Therefore, this embodiment designs a temporal situation prediction module, which uses continuous frame sequences of multimodal images to extract temporal information, predict the future movement trend of the equipment, and calculate the collision risk, thus achieving a leap from passive perception to active early warning.

[0055] By predicting the situation in time series, this method can issue an early warning seconds before a collision occurs, buying valuable time for dispatchers to intervene in advance and achieving a fundamental improvement from "post-event alarm" to "pre-event prediction". The prediction result will be transmitted to the integrated decision-making module in step 6 as an important basis for risk warning.

[0056] The prediction of the movement trends of each transportation device includes: Motion features of each transportation device are extracted from the time-series image sequence, and the motion features are input into the time-series model to predict the position of each device at a future time. The motion features include position, speed, acceleration, and direction of travel. The time series model is a lightweight time series model based on the attention mechanism. Spatial constraint rules are added during the prediction process to force the predicted trajectory to conform to the physical boundaries of the station passage.

[0057] The temporal situation prediction module takes a continuous 30-frame image sequence (i.e., data from the past second) as input. By tracking the detected devices in each frame, the module obtains the positions of each device at different times, and then calculates their velocity (rate of change of position), acceleration (rate of change of velocity), and direction of travel (direction vector of position change). For example, the position sequence of a certain railcar in the past second is [(120.5, 8.7, -42.5), (121.0, 8.7, -42.5), (121.5, 8.7, -42.5), ...], from which its velocity is calculated to be 0.5 m / s, acceleration to be 0, and direction of travel to be the positive X-axis. The motion feature sequence is then input into a lightweight temporal model based on an attention mechanism (such as an improved Transformer decoder architecture). This model adaptively weights the importance of each time step in the historical motion feature sequence to predict the positions of each device in the next few seconds. To ensure that the prediction results conform to physical reality, spatial constraint rules are added to the prediction process—the predicted trajectory must conform to the physical boundaries of the station passage (such as the track direction, clearance range, and the location of inaccessible walls), to avoid generating unreasonable prediction paths (such as the track car drifting laterally out of the track range).

[0058] Attention-based temporal models can adaptively focus on the most predictable time steps in the motion feature sequence, achieving higher prediction accuracy compared to traditional temporal models. The inclusion of spatial constraint rules further ensures the physical rationality of the prediction results, avoiding false alarms caused by predicted trajectories deviating from the actual feasible region. Since the 3D spatial localization branch in step 3 provides the precise poses of each device, the motion feature extraction in this step has higher accuracy.

[0059] The collision risk is determined based on the minimum expected distance and expected approach time between any two devices. When the minimum expected distance is lower than the safety interval or the expected approach time is shorter than the threshold, a graded warning signal is generated according to the risk level.

[0060] Based on the predicted location, the minimum expected distance and the expected approach time (i.e., the time elapsed from the current moment until the two devices reach the minimum distance) between any two devices are calculated. In this embodiment, the prediction results show that the minimum distance between a certain railcar and a certain AGV trolley will be shortened to 1.2 meters in the next 2 seconds, which is lower than the safe interval (3 meters). The expected approach time is 2 seconds, which is shorter than the warning threshold (5 seconds). The system generates a "warning" level collision risk signal.

[0061] By comprehensively judging the minimum expected distance and the expected approach time, this method can accurately identify collision risks of different urgency levels and generate graded early warning signals in three levels: attention, warning, and emergency. This provides dispatchers with differentiated decision-making basis and avoids the problems of false alarms or missed alarms caused by a single threshold judgment.

[0062] Step 5: Detect structural anomalies in the transportation path based on the real-time acquired path images, and output the path health assessment result (path health status assessment).

[0063] The structural health of transportation routes is directly related to transportation safety. Problems such as track deformation, road surface cracking, and water accumulation, if not detected in time, can lead to serious accidents. However, current technologies rely on manual inspections, which are inefficient and time-consuming. This step utilizes deep learning-based image anomaly detection technology to automate the assessment of transportation route structural health, enabling continuous monitoring of route status without human intervention.

[0064] Through route health status assessment, this method achieves automated, 24 / 7 detection of structural anomalies in transportation routes without human intervention. It can detect problems in their early stages, significantly improving the timeliness and efficiency of route maintenance. The assessment results will be transmitted to the integrated decision-making module in step 6, serving as an important component of the safety situation map.

[0065] Among these, detecting structural anomalies in transportation routes includes: The real-time acquired path image is input into the path state anomaly detection network to obtain the reconstructed image, and the difference between the real-time path image and the reconstructed image is calculated. When the difference exceeds the set threshold, the path is determined to be abnormal and a difference heatmap is generated to display the abnormal area. The detected abnormal areas are identified by a classification subnetwork to identify the abnormality type, which includes track deformation, road surface cracking, water accumulation and foreign object coverage, and the severity level is assessed based on the area and shape of the abnormal area. The path state anomaly detection network adopts a generative adversarial network architecture, which reconstructs the path images collected in real time into reconstructed images under normal conditions.

[0066] This embodiment constructs a path state anomaly detection network based on a generative adversarial network (GAN) architecture. In the initial construction phase, when the path is intact, images of each transportation path are first collected to establish a "normal state" reference image library. This image library is then used to train the GAN, enabling it to learn the distribution characteristics of normal path images. In other words, the network can "reconstruct" any input image into an image conforming to a normal distribution pattern. During the real-time detection phase, the currently collected track area image is input into the trained GAN, which attempts to reconstruct the input as an image in a normal state. Since the network is trained on normal path images, when the input image itself is normal, the difference between the reconstructed image and the input image is small. However, when the input image has anomalies (such as track deformation, water accumulation, etc.), the network attempts to "correct" it to a normal state, and the pixel-level difference between the reconstructed image and the input image increases significantly. Based on this principle, the pixel-level difference between the real-time path image and the reconstructed image is calculated, and an overall difference score is obtained. For detected abnormal areas, a classification sub-network is used to identify the anomaly type, and the severity level is assessed based on the area and morphology of the abnormal area (such as water depth estimation, crack width, etc.).

[0067] In this embodiment, the difference score is 28.6, exceeding the anomaly threshold (20 points), indicating an anomaly in the path. The difference heatmap shows that the differences are mainly concentrated in an area of ​​approximately 0.5m × 0.4m, about 3 meters to the right of the track. The classification subnetwork determines the anomaly type as "water accumulation" and the severity level as "moderate." Through the above-described path health assessment method based on generative adversarial network image reconstruction, this method can not only automatically detect path anomalies but also generate a difference heatmap to intuitively indicate the location of anomalies, identify the anomaly type, and assess the severity level. This achieves automated and accurate detection of transportation path structural anomalies without human intervention, operating 24 / 7. Since step 2 has already provided high-quality input to the path image through multimodal fusion, the anomaly detection in this step has higher sensitivity and accuracy.

[0068] Step 6: Generate a safety situation map based on the perception results, the collision risk, and the path health assessment results, and issue warnings according to the risk level (comprehensive decision-making and early warning).

[0069] The integrated decision-making module combines the perception results output from step 3 (material off-center loading warning handled, no overloading or spillage, AGV position, no path obstacles, no personnel violations), the collision risk prediction output from step 4 (warning-level collision risk between the railcar and AGV), and the path health assessment output from step 5 (moderate water accumulation on the right side of the track, location X=130~130.5m). First, the output information from each module is aligned and overlaid in the same spatial coordinate system to form a unified safety situation map—using the station's floor plan as the base map, collision risk areas are marked in yellow, and water accumulation areas are marked in orange. The real-time location, direction of movement, and distribution of path obstacles for each transport device are also marked, allowing dispatchers to intuitively grasp the overall safety situation. Based on this, graded warnings are issued according to risk level. This invention sets four levels: green, yellow, orange, and red: green indicates normal operation with no risk; yellow indicates a warning level, with minor risk but not affecting current operations; orange indicates a warning level, with moderate risk and recommended preventative measures; and red indicates an emergency level, with high risk requiring immediate action. Based on the risk type and level, the system synchronously outputs specific scheduling suggestions—in this embodiment, it suggests that the AGV trolley decelerate to below 0.3 m / s and that maintenance personnel proceed to X=130m to handle the accumulated water. The aforementioned warning information and suggestions are simultaneously pushed to the dispatcher's monitoring terminal and the mobile terminals of on-site workers in both text and voice formats. When the integrated decision-making module determines that there is an emergency-level risk (such as predicting that two devices will collide within 1 second), the system also automatically sends a stop command to the relevant transport equipment, achieving automated safety interlock control.

[0070] Through the aforementioned comprehensive decision-making and early warning mechanism, this method transforms the perception, prediction, and detection results of each module into specific and operable early warning signals and scheduling instructions. The various technical means support and are organically connected—the multimodal fusion in step 2 provides high-quality fused image input for perception in step 3; the precise device pose output from the 3D spatial positioning branch in step 3 provides accurate data for motion feature extraction in step 4; collision risk prediction in step 4 and path health assessment in step 5 together constitute the risk basis for comprehensive decision-making; step 6 summarizes the analysis results of each module and transforms them into actual action instructions. Steps 1 to 6 are interconnected and progressively build upon each other, jointly forming a full-chain intelligent safety transportation assurance system from environmental perception to situation prediction, from anomaly detection to decision output. This systematically solves the technical challenges in the safety monitoring of material transportation in deeply buried stations, such as poor environmental adaptability, single perception dimension, lack of proactive early warning, and reliance on manual path inspection.

[0071] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for safe transportation of materials in a deeply buried railway station, characterized in that, Includes the following steps: Step 1: Acquire multimodal images of the transportation scene at the deep-buried station, including visible light images, infrared thermal imaging images, and depth images; Step 2: Preprocess and fuse the multimodal images to generate a fused image; Step 3: Input the fused image into the multi-task joint perception network for perception, and output the perception results, which include material status information, equipment spatial pose information, path obstacle information, and personnel behavior information; Step 4: Extract time-series image sequences from the continuous frame sequences of the multimodal images, predict the motion trends of each transportation device based on the time-series image sequences, and calculate the collision risk between devices; Step 5: Detect structural anomalies in the transportation route based on the real-time acquired route images, and output the route health assessment results; Step 6: Generate a safety situation map based on the perception results, the collision risk, and the path health assessment results, and issue warnings according to the risk level.

2. The method for safe transportation of materials in a deeply buried railway station according to claim 1, characterized in that, In step 2, the multimodal images are preprocessed and fused, including: The visible light image is subjected to low-light enhancement processing; The enhanced visible light image, infrared thermal image, and depth image are input into a multi-scale feature extraction network; An attention-guided mechanism is used to dynamically allocate fusion weights based on the quality of modal features at each pixel location, enabling adaptive pixel-level fusion.

3. The method for safe transportation of materials in a deeply buried railway station according to claim 2, characterized in that, The low-light enhancement process includes: The visible light image is decomposed into an illumination layer and a reflection layer; The texture structure of the infrared thermal imaging image is used as a guiding prior to guide the illumination layer to make adaptive adjustments in the dark area; The attention guidance mechanism includes: The fusion weights of each modality are dynamically determined based on the sharpness and information richness of each modal feature at each pixel location; Increase the fusion weight of infrared thermal imaging features in areas obscured by dust; Increase the fusion weight of visible light features in areas with rich texture.

4. A method for safe transportation of materials in a deeply buried railway station according to claim 2, characterized in that, The preprocessing and fusion of the multimodal images further includes: Real-time quality scoring of the fused image is performed using a no-reference image quality assessment subnetwork. When the score falls below the set threshold, the collection parameters are automatically adjusted.

5. A method for safe transportation of materials in a deeply buried railway station according to claim 1, characterized in that, In step 3, the material status information is obtained through the material status recognition branch of the multi-task joint sensing network. The material status recognition branch detects the material outline on the transport equipment and calculates the area difference ratio between the left and right sides of the material. When the difference ratio exceeds a preset safety threshold, it is determined to be an off-center load risk. The material status recognition branch also detects whether the material exceeds the boundary of the transport container and whether there is any scattered material falling into the transport channel.

6. A method for safe transportation of materials in a deeply buried railway station according to claim 1, characterized in that, In step 3, the spatial pose information of the equipment is obtained through the three-dimensional spatial localization branch of the multi-task joint perception network. The three-dimensional spatial localization branch combines the depth image to perform three-dimensional bounding box regression on each transportation equipment, and outputs the position and orientation of each transportation equipment in the station spatial coordinate system. The path obstacle information is obtained through the path obstacle detection branch of the multi-task joint perception network. The path obstacle detection branch performs pixel-level semantic segmentation on the ground area of ​​the transportation path, and identifies and locates fallen materials, equipment displacement, water accumulation and landslide obstacles.

7. A method for safe transportation of materials in a deeply buried railway station according to claim 1, characterized in that, In step 3, the personnel behavior information is obtained through the personnel safety behavior analysis branch of the multi-task joint perception network. The personnel safety behavior analysis branch extracts the skeletal posture of the workers based on human key point detection and identifies unsafe behaviors such as illegally crossing transportation channels and entering dangerous areas.

8. A method for safe transportation of materials in a deeply buried railway station according to claim 1, characterized in that, In step 4, the movement trend of each transportation device is predicted, including: Motion features of each transportation device are extracted from the time-series image sequence, and the motion features are input into the time-series model to predict the position of each device at a future time. The motion features include position, speed, acceleration, and direction of travel. The time series model is a lightweight time series model based on the attention mechanism. Spatial constraint rules are added during the prediction process to force the predicted trajectory to conform to the physical boundaries of the station passage.

9. A method for safe transportation of materials in a deeply buried railway station according to claim 8, characterized in that, In step 4, the collision risk is determined based on the minimum expected distance and expected approach time between any two devices. When the minimum expected distance is lower than the safety interval or the expected approach time is shorter than the threshold, a graded warning signal is generated according to the risk level.

10. A method for safe transportation of materials in a deeply buried railway station according to claim 1, characterized in that, In step 5, structural anomalies in the transportation route are detected, including: The real-time acquired path image is input into the path state anomaly detection network to obtain the reconstructed image, and the difference between the real-time path image and the reconstructed image is calculated. When the difference exceeds the set threshold, the path is determined to be abnormal and a difference heatmap is generated to display the abnormal area. The abnormal regions detected are identified by a classification subnetwork to determine the type of abnormality, and the severity level is assessed based on the area and morphology of the abnormal regions. The path state anomaly detection network adopts a generative adversarial network architecture, which reconstructs the path images collected in real time into reconstructed images under normal conditions.

Images