Railway train roof maintenance operation safety detection system and method

Abstract

The invention discloses a railway train roof maintenance operation safety detection system and method, which are used for solving the technical problem of poor detection effect caused by incapability of realizing real-time monitoring of a three-dimensional state by existing detection equipment. The system is composed of a multi-modal data acquisition module, an edge calculation processing module, an intelligent detection analysis module and a safety early warning module which are connected in sequence. The multi-modal data acquisition module acquires a visual image, an infrared image and a sound signal of a roof maintenance area; the edge calculation processing module preprocesses the data and outputs a target visual image, a target infrared image and a target sound feature; the intelligent detection analysis module adopts a preset railway train roof maintenance operation safety detection model to detect the three types of target data, and outputs a visual detection result, a temperature detection result and a sound detection result; and the safety early warning module determines risk levels according to the three types of detection results, and gives an alarm according to the corresponding levels.

Description

Technical Field

[0001] This invention relates to the field of railway traffic safety technology, and in particular to a safety inspection system and method for maintenance operations on the roof of railway trains. Background Technology

[0002] As a core pillar and key infrastructure of the national comprehensive transportation system, railway transportation bears the strategic task of passenger and freight transport. Railway rooftop equipment (pantographs, air conditioning units, high-voltage disconnect switches, etc.) are core components for train power transmission and environmental control; their operational status directly determines whether a train can run normally. Improper maintenance leading to malfunctions can result in major safety accidents such as train stoppages, short circuits in the overhead contact system, or even derailments.

[0003] Railway car roof maintenance is a crucial link in ensuring the reliability of train equipment. However, due to the limitations of the working environment and equipment characteristics, its safety management faces many practical challenges. On the one hand, car roof work is a high-altitude and dangerous operation. The working space is narrow, and there are potential hazards such as high-voltage overhead contact lines nearby. Workers must complete equipment disassembly, inspection, and installation on the top of the train, which poses a risk of electric shock, falls from heights, and injuries from falling tools. On the other hand, the equipment on the car roof is diverse in type and complex in structure, and is exposed to wind, sun, and rain for extended periods, making it prone to aging, loosening, and damage of components. Accurate identification of equipment status and adherence to operating procedures are essential during maintenance.

[0004] Existing detection equipment mostly uses single sensor technology, but it is difficult to balance the comprehensiveness of monitoring and cost control in terms of sensor placement and number. Especially in the complex structure of the roof area, conventional sensors cannot achieve real-time monitoring of the three-dimensional state, resulting in poor detection results. Summary of the Invention

[0005] This invention provides a safety inspection system and method for maintenance operations on railway train roofs, which solves the technical problem that existing inspection equipment cannot achieve real-time monitoring of three-dimensional status, resulting in poor inspection results.

[0006] The first aspect of the present invention provides a safety detection system for maintenance operations on the roof of a railway train, the system comprising a multimodal data acquisition module, an edge computing processing module, an intelligent detection and analysis module, and a safety early warning module connected in sequence;

[0007] The multimodal data acquisition module is used to acquire visual images, infrared images, and sound signals of the roof repair area;

[0008] The edge computing processing module is used to preprocess the visual image, infrared image and sound signal of the roof maintenance area, and output the target visual image, target infrared image and target sound features;

[0009] The intelligent detection and analysis module is used to detect the target visual image, the target infrared image and the target sound features respectively using a preset railway train roof maintenance operation safety detection model, and output visual detection results, temperature detection results and sound detection results;

[0010] The safety warning module is used to determine the risk level based on the visual detection results, the temperature detection results, and the sound detection results, and to issue an alarm according to the risk level.

[0011] Optionally, the edge computing processing module is specifically used for:

[0012] The visual image is denoised to output the target visual image;

[0013] The infrared image is calibrated for temperature, and the target infrared image is output.

[0014] Feature extraction is performed on the sound signal to generate target sound features.

[0015] Optionally, the preset railway train roof maintenance operation safety detection model includes a backbone network, a neck network, and a multi-task detection head; the intelligent detection and analysis module is specifically used for:

[0016] The backbone network is used to extract enhancement features from the target visual image, the target infrared image, and the target sound features, respectively, and output visual enhancement features, infrared enhancement features, and sound enhancement features;

[0017] The neck network is used to perform multi-scale feature fusion on the visual enhancement features, the infrared enhancement features, and the sound enhancement features, respectively, and output visual fusion multi-scale features, infrared fusion multi-scale features, and sound fusion multi-scale features.

[0018] Multimodal feature alignment is performed on the visual fusion multi-scale features, the infrared fusion multi-scale features, and the sound fusion multi-scale features respectively to generate aligned visual fusion multi-scale features, aligned infrared fusion multi-scale features, and aligned sound fusion multi-scale features;

[0019] Cross-modal attention weights are calculated for the aligned visual fusion multi-scale features, the aligned infrared fusion multi-scale features, and the aligned audio fusion multi-scale features, and the cross-modal weights are output.

[0020] The alignment visual fusion multi-scale features, the alignment infrared fusion multi-scale features, the alignment audio fusion multi-scale features, and the cross-modal weights are fused to output multi-modal fusion features.

[0021] The multimodal fusion features are used as input to the multi-task detection head, which outputs visual detection results, temperature detection results, and sound detection results.

[0022] Optionally, the data processing procedure of the backbone network specifically includes:

[0023] The target sound features are converted into a two-dimensional spectrogram;

[0024] The target visual image, the target infrared image, and the two-dimensional acoustic spectrogram are sequentially subjected to feature embedding and basic feature extraction to output a first visual feature, a first infrared feature, and a first acoustic feature.

[0025] The first visual feature, the first infrared feature, and the first sound feature are sequentially embedded and basic feature extracted to output the second visual feature, the second infrared feature, and the second sound feature.

[0026] The second visual feature, the second infrared feature, and the second sound feature are respectively subjected to feature embedding and basic feature extraction, and the third visual feature, the third infrared feature, and the third sound feature are output.

[0027] The third visual feature, the third infrared feature, and the third sound feature are respectively subjected to feature embedding and basic feature extraction to output the fourth visual feature, the fourth infrared feature, and the fourth sound feature;

[0028] The fourth visual feature, the fourth infrared feature, and the fourth sound feature are respectively used as inputs to the context prior module, and the visual context prior feature, infrared context prior feature, and sound context prior feature are output.

[0029] Dynamic convolution is performed on the visual context prior features, the infrared context prior features, and the sound context prior features respectively to output the first visual dynamic convolution feature, the first infrared dynamic convolution feature, and the first sound dynamic convolution feature;

[0030] The first visual dynamic convolution feature, the first infrared dynamic convolution feature, and the first sound dynamic convolution feature are respectively embedded to output the fifth visual feature, the fifth infrared feature, and the fifth sound feature.

[0031] The fifth visual feature and the visual context prior feature are spliced ​​together, the fifth infrared feature and the infrared context prior feature are spliced ​​together, and the fifth sound feature and the sound context prior feature are spliced ​​together to generate visual splicing features, infrared splicing features and sound splicing features.

[0032] Dynamic convolution is performed on the visual stitching features, the infrared stitching features, and the sound stitching features respectively to generate visual enhancement features, infrared enhancement features, and sound enhancement features.

[0033] Optionally, the data processing procedure of the neck network specifically includes:

[0034] The visual enhancement feature, the infrared enhancement feature, and the sound enhancement feature are continuously upsampled respectively to output multiple visual upsampled features, multiple infrared upsampled features, and multiple sound upsampled features;

[0035] Multi-scale feature extraction is performed on multiple visual upsampling features and visual enhancement features to output multiple visual multi-scale features;

[0036] Multi-scale feature extraction is performed on multiple infrared upsampling features and infrared enhancement features to output multiple infrared multi-scale features;

[0037] Multi-scale feature extraction is performed on multiple sound upsampling features and sound enhancement features to output multiple sound multi-scale features;

[0038] Multiple visual multi-scale features, multiple infrared multi-scale features, and multiple sound multi-scale features are spliced ​​together to output visual fusion multi-scale features, infrared fusion multi-scale features, and sound fusion multi-scale features;

[0039] The plurality of visual upsampling features include a first visual upsampling feature, a second visual upsampling feature, and a third visual upsampling feature; the plurality of visual multi-scale features include a first visual multi-scale feature, a second visual multi-scale feature, a third visual multi-scale feature, and a fourth visual multi-scale feature; the process of extracting multi-scale features from the plurality of visual upsampling features and the visual enhancement features to output the plurality of visual multi-scale features is specifically as follows:

[0040] The visual enhancement features are subjected to a 1×1 convolution operation, then processed sequentially by a high-frequency perception module and a spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation. Finally, an upsampling operation is performed to output the first visual multi-scale features.

[0041] The first visual upsampled features are subjected to a 1×1 convolution operation, then processed sequentially by the high-frequency perception module and the spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally an upsampling operation is performed to output the second visual multi-scale features.

[0042] The second visual upsampled features are subjected to a 1×1 convolution operation, then processed sequentially by the high-frequency perception module and the spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally a downsampling operation is performed to output the third visual multi-scale features.

[0043] The upsampled features of the third vision are subjected to a 1×1 convolution operation, then processed sequentially by the high-frequency perception module and the spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally a downsampling operation to output the multi-scale features of the fourth vision.

[0044] Optionally, the multi-task detection head includes a shared feature layer, a personnel violation detection head, an equipment temperature anomaly detection head, and a structural defect detection head; the data processing procedure of the multi-task detection head is as follows:

[0045] The multimodal fusion features are used as input to the shared feature layer, and the shared feature representation is output.

[0046] The shared feature representations are respectively input into the personnel violation detection head, the equipment temperature anomaly detection head, and the structural defect detection head for detection, generating visual detection results, temperature detection results, and sound detection results.

[0047] Optionally, the model training process for the preset railway train roof maintenance operation safety detection model is as follows:

[0048] The system acquires visual images, infrared images, and sound signals for model training, preprocesses these images and signals, and outputs the preprocessed visual images, infrared images, and sound features.

[0049] The initial railway train roof maintenance operation safety detection model is used to generate visual detection results, temperature detection results, and sound detection results for model training based on the preprocessed visual images, infrared images, and sound features.

[0050] The initial railway train roof maintenance operation safety detection model is trained using a preset loss function based on the visual detection results, temperature detection results, and sound detection results used for model training, and the trained preset railway train roof maintenance operation safety detection model is determined.

[0051] The second aspect of this invention provides a safety inspection method for maintenance operations on the roof of a railway train, comprising:

[0052] Acquire visual, infrared, and audio signals of the rooftop repair area;

[0053] The visual image, infrared image, and sound signal of the roof maintenance area are preprocessed to output the target visual image, target infrared image, and target sound features.

[0054] A preset railway train roof maintenance operation safety detection model is used to detect the target visual image, the target infrared image, and the target sound features respectively, and outputs visual detection results, temperature detection results, and sound detection results;

[0055] Based on the visual detection results, the temperature detection results, and the sound detection results, the risk level is determined, and an alarm is issued according to the risk level.

[0056] A computer device provided in a third aspect of the present invention includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the railway train roof maintenance operation safety detection system as described above.

[0057] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the steps of the railway train roof maintenance operation safety detection system as described above.

[0058] As can be seen from the above technical solutions, the present invention has the following advantages:

[0059] The first aspect of the technical solution of the present invention provides a safety detection system for railway train roof maintenance operations. This system includes a multimodal data acquisition module, an edge computing processing module, an intelligent detection and analysis module, and a safety early warning module connected in sequence. The multimodal data acquisition module is used to acquire visual images, infrared images, and sound signals from the roof maintenance area. The edge computing processing module is used to preprocess the visual images, infrared images, and sound signals from the roof maintenance area, outputting target visual images, target infrared images, and target sound features. The intelligent detection and analysis module is used to detect the target visual images, target infrared images, and target sound features using a preset railway train roof maintenance operation safety detection model, outputting visual detection results, temperature detection results, and sound detection results. The safety early warning module is used to determine the risk level based on the visual detection results, temperature detection results, and sound detection results, and issue an alarm according to the risk level. Based on the above solution, the present invention, through the synchronous acquisition and real-time processing of visual, infrared, and sound three-dimensional data, and through the preset railway train roof maintenance operation safety detection model, can accurately extract risk information from multimodal data, effectively improving the comprehensiveness and accuracy of detection, thereby enhancing the detection effect.

[0060] The second aspect of the above-mentioned technical solution of the present invention provides a safety detection method for railway train roof maintenance operations, which acquires visual images, infrared images, and sound signals of the roof maintenance area; preprocesses the visual images, infrared images, and sound signals of the roof maintenance area to output target visual images, target infrared images, and target sound features; uses a preset railway train roof maintenance operation safety detection model to detect the target visual images, target infrared images, and target sound features respectively, and outputs visual detection results, temperature detection results, and sound detection results; determines the risk level based on the visual detection results, temperature detection results, and sound detection results, and issues an alarm according to the risk level; based on the above solution, the present invention can accurately mine risk information in multimodal data through the synchronous acquisition and real-time processing of visual, infrared, and sound three-dimensional data, and through the preset railway train roof maintenance operation safety detection model, which can effectively improve the comprehensiveness and accuracy of detection, thereby improving the detection effect. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of a safety inspection system for maintenance operations on the roof of a railway train, provided in Embodiment 1 of the present invention.

[0063] Figure 2 This is a schematic diagram of the structure of the pre-set railway train roof maintenance operation safety detection model provided in Embodiment 1 of the present invention;

[0064] Figure 3 This is a schematic diagram of the backbone network provided in Embodiment 1 of the present invention;

[0065] Figure 4 This is a schematic diagram of the neck network provided in Embodiment 1 of the present invention;

[0066] Figure 5 This is a schematic diagram of the neck network provided in Embodiment 1 of the present invention;

[0067] Figure 6 This is an overall framework diagram of a railway train roof maintenance operation safety detection system provided in Embodiment 1 of the present invention;

[0068] Figure 7 This is a flowchart illustrating the steps of a safety inspection method for maintenance operations on the roof of a railway train, as provided in Embodiment 2 of the present invention. Detailed Implementation

[0069] This invention provides a safety inspection system and method for railway train roof maintenance operations, which solves the technical problem that existing inspection equipment cannot achieve real-time monitoring of three-dimensional status, resulting in poor inspection results.

[0070] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0071] Please see Figure 1 , Figure 1 This is a structural schematic diagram of a railway train roof maintenance operation safety detection system provided in Embodiment 1 of the present invention.

[0072] This invention provides a safety detection system for maintenance operations on the roof of a railway train. The system includes a multimodal data acquisition module, an edge computing processing module, an intelligent detection and analysis module, and a safety early warning module connected in sequence.

[0073] A multimodal data acquisition module is used to acquire visual images, infrared images, and sound signals from the roof repair area;

[0074] The edge computing processing module is used to preprocess the visual images, infrared images and sound signals of the roof maintenance area, and output the target visual image, target infrared image and target sound features;

[0075] The intelligent detection and analysis module is used to detect the target visual image, target infrared image and target sound features using a preset railway train roof maintenance operation safety detection model, and output visual detection results, temperature detection results and sound detection results respectively.

[0076] The safety warning module is used to determine the risk level based on visual detection results, temperature detection results, and sound detection results, and to issue an alarm according to the risk level.

[0077] It should be noted that the multimodal data acquisition module is used to simultaneously acquire multi-source sensing data of the railway roof maintenance scenario. It consists of a high-definition vision submodule, an infrared thermal imaging submodule, and an acoustic sensing submodule. Among them, the high-definition vision submodule uses a 4K resolution industrial camera with a frame rate of no less than 30fps and is equipped with an adaptive infrared fill light to acquire color image data of the roof maintenance area; the infrared thermal imaging submodule has a resolution of no less than 640×512 and a temperature measurement range of 20℃~150℃ to acquire equipment temperature distribution data; the acoustic sensing submodule has a sampling rate of no less than 44.1kHz and is equipped with an anti-noise microphone array to acquire characteristic sound signals such as abnormal vibrations and arc discharges.

[0078] The multimodal data acquisition module employs a "3+1" distributed sensor layout: three sets of high-definition visual-infrared composite sensors are deployed around the rooftop maintenance platform, with each set spaced 5 meters apart; an additional acoustic sensor array is installed near key equipment such as the pantograph. All sensors are connected to the edge computing unit via the MIPICSI-2 interface and feature an industrial-grade protection design (IP65, Ingress Protection Rating 65), adapting to an operating temperature range of -40℃ to 70℃.

[0079] As a further improvement, the edge computing processing module is specifically used for:

[0080] Denoise the visual image and output the target visual image;

[0081] Perform temperature calibration on the infrared image and output the target infrared image;

[0082] Feature extraction is performed on the sound signal to generate target sound features.

[0083] It should be noted that the edge computing processing module consists of an RK3576 edge computer, a data preprocessing unit, and a local storage unit. The RK3576 edge computer integrates a Neural Processing Unit (NPU) with 6 TOPS (Tera Operations Per Second) computing power and a quad-core Cortex-A72 processor for parallel preprocessing of multimodal data. The data preprocessing unit includes an image denoising module based on Wavelet Pooling, an infrared temperature calibration module, and an acoustic feature extraction module. The local storage unit uses an encrypted storage chip to cache raw data and detection results, with a storage capacity of no less than 128GB.

[0084] The edge computing processing module uses the Rockchip RK3576 development board, equipped with 2GB LowPower Double Data Rate 4 (LPDDR4) memory and 128GB embedded MultiMediaCard (eMMC) storage. Through the Rockchip Neural Network Toolkit2 (RKNN-Toolkit2) toolchain, the improved Train-RT-DETR model is quantized to 8-bit Integer (INT8) precision, compressing the model size to 1 / 4 of the original while maintaining over 99% accuracy. The power module adopts a wide-voltage input design (9-36 Volts Direct Current, VDC) and supports battery backup, ensuring data retention for up to 30 minutes in the event of a power outage.

[0085] As a further improvement, the pre-set safety inspection model for railway train roof maintenance operations includes a backbone network, a neck network, and a multi-task inspection head; the intelligent inspection and analysis module is specifically used for:

[0086] A backbone network is used to extract enhancement features from the target visual image, target infrared image and target sound features respectively, and output visual enhancement features, infrared enhancement features and sound enhancement features;

[0087] The neck network is used to perform multi-scale feature fusion on visual enhancement features, infrared enhancement features and sound enhancement features respectively, and output visual fusion multi-scale features, infrared fusion multi-scale features and sound fusion multi-scale features.

[0088] Multimodal feature alignment is performed on the visual fusion multi-scale features, infrared fusion multi-scale features, and sound fusion multi-scale features respectively to generate aligned visual fusion multi-scale features, aligned infrared fusion multi-scale features, and aligned sound fusion multi-scale features;

[0089] Cross-modal attention weights are calculated for aligned visual fusion multi-scale features, aligned infrared fusion multi-scale features, and aligned audio fusion multi-scale features, and the cross-modal weights are output.

[0090] The multimodal fusion features are output by fusing aligned visual multi-scale features, aligned infrared multi-scale features, aligned audio multi-scale features, and cross-modal weights.

[0091] The multimodal fusion features are used as input to the multi-task detection head, which outputs visual detection results, temperature detection results, and sound detection results.

[0092] It should be noted that you should refer to [link / reference]. Figure 2The proposed safety detection model for railway train roof maintenance operations is an improved RT-DETR (Real-Time Detection Transformer) network. Specifically, for backbone network optimization: the original backbone is replaced with an OverLock architecture RepConvBlock (Repetitive Convolution Block) combined with CSP (Cross Stage Partial Network) to improve the backbone network. Pinwheel-shaped convolution is introduced to optimize small target feature extraction, thereby increasing the Intensity Enhancement Layer to improve feature extraction capabilities under low-light conditions. For the Neck part improvement: the HFP and SDP modules from the AAAI2025 (Association for the Advancement of Artificial Intelligence 2025) HS-FPN (Hybrid Spatial Feature Pyramid Network) are used to improve the feature pyramid network, enhancing the feature representation of small targets (such as loose bolts).

[0093] Furthermore, this invention proposes a dynamic cross-modal attention mechanism based on the dynamic routing mechanism in the ICML2025 (International Conference on Machine Learning 2025) multimodal technology white paper. It calculates the correlation matrix of visual, infrared, and acoustic features and adaptively adjusts the modal weights through a gating function. The visual fusion multi-scale features, infrared fusion multi-scale features, and acoustic fusion multi-scale features are fused to obtain multimodal fusion features. Specifically, let the visual fusion multi-scale features be... Infrared fusion multi-scale features are The multi-scale features of sound fusion are then processed by FFT (Fast Fourier Transform) to obtain... Where H is the height of the feature map and W is the width of the feature map. , , The number of channels corresponding to visual fusion multi-scale features, infrared fusion multi-scale features, and sound fusion multi-scale features.

[0094] For multimodal feature alignment: ,in, The m-th modal feature after alignment (such as aligned visual fusion multi-scale features); , These are learnable weight matrices and biases used for feature transformation; is the original m-th modal feature (such as visual fusion multi-scale features); m represents the modal set (v for visual, i for infrared, a for sound).

[0095] For cross-modal attention weight calculation: ,in, For cross-modal weights, m and n correspond to visual (v), infrared (i), and audio (a) modalities, respectively (e.g., ...). The weights representing the association between visual and infrared modal features are used to measure the strength of the association between aligned visual fusion multi-scale features, aligned infrared fusion multi-scale features, and aligned audio fusion multi-scale features. Let m be the query matrix for the m-th modality; The dimension of the key matrix (the key matrices of visual, infrared, and audio modalities are aligned to have the same dimension) is used to scale the attention scores and avoid the impact of dimensional differences on weight calculation. To query the learnable weight matrix; Let n be the key matrix of the nth modality. When n is the visual, infrared, or audio modality, it is formed by aligning the visual fusion multi-scale features, aligning the infrared fusion multi-scale features, and aligning the audio fusion multi-scale features, respectively, through the key weight matrix. Obtained by linear transformation, used in conjunction with the query matrix. Matching to generate cross-modal attention weights; The key is a learnable weight matrix; For the input features of the nth modality, when n is a visual, infrared, or audio modality, the corresponding features are: aligned visual fusion multi-scale features, aligned infrared fusion multi-scale features, and aligned audio fusion multi-scale features, respectively. Collaborative participation in cross-modal attention weight calculation; T is the transpose.

[0096] For dynamic weighted fusion output: , , ,in, The final feature after multimodal dynamic weight fusion (i.e., multimodal fusion feature) is the result of fusing visual, infrared, and sound multimodal features after dynamic weight adjustment, and is used for subsequent detection or analysis tasks; The self-modal weights for the m-th modality (visual, infrared, sound) control the contribution of the modality's own features to the fusion process, determined by the importance index of the modality's features. through Calculated; The weighting coefficients for the m-th mode are used to further adjust the weights of different features; The interaction terms between the m-th mode and the n-th mode (i.e., the interaction terms between different features) are derived from... After pooling and The processed data measures the correlation strength between the features of the m-mode and the n-mode. For pooling layers; This is a softmax activation function layer; This is a temperature coefficient (controlling the effect of temperature in feature fusion, ranging from 0.1 to 0.5). is the modal feature importance index for the m-th mode.

[0097] It is worth mentioning that the dynamic cross-modal attention mechanism employs a dual-channel processing strategy: for the visual and infrared modalities, pixel-level alignment and fusion are used; for the acoustic modalities, spectral features are extracted through FFT transformation and then correlated with visual semantic features. Temperature coefficient An adaptive adjustment strategy is adopted, which dynamically adjusts the modal entropy value between 0.1 and 0.5, effectively reducing the semantic mismatch rate to below 12%.

[0098] As a further improvement, the data processing procedure for the backbone network is as follows:

[0099] Convert the target sound features into a two-dimensional spectrogram;

[0100] Feature embedding and basic feature extraction are performed on the target visual image, target infrared image and two-dimensional acoustic spectrogram respectively, and the first visual feature, first infrared feature and first acoustic feature are output.

[0101] The first visual feature, the first infrared feature, and the first sound feature are respectively embedded and extracted into basic features to output the second visual feature, the second infrared feature, and the second sound feature.

[0102] The second visual feature, the second infrared feature, and the second sound feature are respectively embedded and basic feature extracted to output the third visual feature, the third infrared feature, and the third sound feature;

[0103] The third visual feature, the third infrared feature, and the third sound feature are respectively embedded and basic feature extracted to output the fourth visual feature, the fourth infrared feature, and the fourth sound feature;

[0104] The fourth visual feature, the fourth infrared feature, and the fourth sound feature are used as inputs to the context prior module, and the outputs are visual context prior features, infrared context prior features, and sound context prior features.

[0105] Dynamic convolution is performed on the visual context prior features, infrared context prior features, and sound context prior features respectively, and the output is the first visual dynamic convolution feature, the first infrared dynamic convolution feature, and the first sound dynamic convolution feature.

[0106] Feature embedding is performed on the first visual dynamic convolution feature, the first infrared dynamic convolution feature and the first sound dynamic convolution feature respectively, and the fifth visual feature, the fifth infrared feature and the fifth sound feature are output;

[0107] The fifth visual feature and the visual context prior feature are concatenated, the fifth infrared feature and the infrared context prior feature are concatenated, and the fifth sound feature and the sound context prior feature are concatenated to generate visual concatenated features, infrared concatenated features, and sound concatenated features.

[0108] Dynamic convolution is performed on visual stitching features, infrared stitching features, and sound stitching features respectively to generate visual enhancement features, infrared enhancement features, and sound enhancement features.

[0109] It should be noted that you should refer to [link / reference]. Figure 3 First, the target sound features are converted into a two-dimensional spectrogram (i.e., a two-dimensional spectrogram adapted to the image feature dimensions (3×H×W) is generated through spectrogram transformation (such as short-time Fourier transform). Then, in the base network, feature embedding and basic feature extraction are performed sequentially on the target visual image, target infrared image, and two-dimensional spectrogram (processed through a feature embedding layer and a basic feature extraction module), outputting the first to fourth visual features, the first to fourth infrared features, and the first to fourth sound features in sequence. Next, the fourth visual feature, the fourth infrared feature, and the fourth sound feature are respectively input into the context prior module. Prior to, the system outputs visual context prior features, infrared context prior features, and sound context prior features. Then, it performs dynamic convolution on these context prior features (through a dynamic module) to output the first visual dynamic convolution feature, the first infrared dynamic convolution feature, and the first sound dynamic convolution feature. These are then processed by a feature embedding layer to obtain the fifth visual feature, the fifth infrared feature, and the fifth sound feature. Finally, the fifth visual feature is concatenated with the visual context prior features, the fifth infrared feature is concatenated with the infrared context prior features, and the fifth sound feature is concatenated with the sound context prior features to generate visual concatenated features, infrared concatenated features, and sound concatenated features. These concatenated features are then dynamically convolved to finally generate visual enhancement features, infrared enhancement features, and sound enhancement features.

[0110] As a further improvement, the data processing procedure for the neck network is as follows:

[0111] The visual enhancement features, infrared enhancement features, and sound enhancement features are continuously upsampled respectively, and multiple visual upsampled features, multiple infrared upsampled features, and multiple sound upsampled features are output.

[0112] Multi-scale feature extraction is performed on multiple visual upsampling features and visual enhancement features to output multiple visual multi-scale features;

[0113] Multi-scale feature extraction is performed on multiple infrared upsampling features and infrared enhancement features to output multiple infrared multi-scale features;

[0114] Multi-scale feature extraction is performed on multiple sound upsampling features and sound enhancement features to output multiple sound multi-scale features;

[0115] Multiple visual multi-scale features, multiple infrared multi-scale features, and multiple sound multi-scale features are spliced ​​together to output visual fusion multi-scale features, infrared fusion multi-scale features, and sound fusion multi-scale features.

[0116] The multiple visual upsampling features include a first visual upsampling feature, a second visual upsampling feature, and a third visual upsampling feature; the multiple visual multi-scale features include a first visual multi-scale feature, a second visual multi-scale feature, a third visual multi-scale feature, and a fourth visual multi-scale feature; the process of extracting multi-scale features from the multiple visual upsampling features and visual enhancement features, and outputting multiple visual multi-scale features, is as follows:

[0117] The visual enhancement features are processed by a 1×1 convolution, then processed by a high-frequency perception module and a spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused. After that, a 3×3 convolution is performed, and finally an upsampling operation is performed to output the first visual multi-scale features.

[0118] The first visual upsampled features are subjected to a 1×1 convolution operation, then processed by the high-frequency perception module and the spatial dependency perception module in sequence. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally an upsampling operation is performed to output the second visual multi-scale features.

[0119] The upsampled features of the second vision are subjected to a 1×1 convolution operation, then processed by the high-frequency perception module and the spatial dependency perception module in sequence. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally a downsampling operation is performed to output the multi-scale features of the third vision.

[0120] The upsampled features of the third vision are subjected to a 1×1 convolution operation, then processed sequentially by the high-frequency perception module and the spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally a downsampling operation to output the multi-scale features of the fourth vision.

[0121] It should be noted that you should refer to [link / reference]. Figures 4-5 Taking the processing of multiple visual upsampling features and visual enhancement features as an example, the third visual upsampling feature C2 is a high-resolution feature (e.g., 800×600 pixels), the second visual upsampling feature C3 is a medium-high resolution feature (e.g., 500×400 pixels), C4 is a medium-resolution feature (e.g., 300×225 pixels), and C5 is a low-resolution feature (e.g., 100×75 pixels). For the C2 branch, a 1×1 convolution operation is first performed on C2, followed by sequential processing through the HFP (High-Frequency Perception Module) and SDP (Spatial Dependency Perception) modules. The P2 feature is processed by the spatial dependency awareness module (SFP) followed by feature fusion, then a 3×3 convolution operation, and finally upsampling. For branch C3, a 1×1 convolution operation is first performed, followed by processing through the HFP and SDP modules, then feature fusion, then a 3×3 convolution operation, and finally upsampling. For branch C4, a 1×1 convolution operation is first performed, followed by processing through the HFP and SDP modules, then feature fusion, and finally a 3×3 convolution operation. Finally, a downsampling operation is performed to output the P4 feature. For the C5 branch, a 1×1 convolution operation is first performed on C5, followed by processing through the HFP module and the SDP module in sequence, then feature fusion is performed, followed by a 3×3 convolution operation, and finally a downsampling operation is performed to output the P5 feature. Finally, the P2, P3, P4, and P5 features are spliced ​​and fused to obtain visual fusion multi-scale features, which take into account both fine-grained detail detection and global semantic recognition, providing multi-resolution fusion feature support for subsequent detection tasks such as personnel violations, equipment abnormalities, and sound abnormalities in railway roof maintenance operations.

[0122] Based on the above, similarly, infrared fusion multi-scale features and sound fusion multi-scale features can be obtained. Among them, multiple infrared upsampling features include a first infrared upsampling feature, a second infrared upsampling feature, and a third infrared upsampling feature; multiple sound upsampling features include a first sound upsampling feature, a second sound upsampling feature, and a third sound upsampling feature; multiple infrared multi-scale features include a first infrared multi-scale feature, a second infrared multi-scale feature, a third infrared multi-scale feature, and a fourth infrared multi-scale feature; and multiple sound multi-scale features include a first sound multi-scale feature, a second sound multi-scale feature, a third sound multi-scale feature, and a fourth sound multi-scale feature.

[0123] As a further improvement, the multi-task detection head includes a shared feature layer, a personnel violation detection head, an equipment temperature anomaly detection head, and a structural defect detection head; the data processing procedure of the multi-task detection head is as follows:

[0124] Multimodal fusion features are used as input to the shared feature layer, and the output is a shared feature representation;

[0125] The shared feature representations are input into the personnel violation detection head, the equipment temperature abnormality detection head, and the structural defect detection head respectively for detection, generating visual detection results, temperature detection results, and sound detection results.

[0126] It should be noted that the multimodal fusion features (comprehensive features integrating visual, infrared, and acoustic multimodal information, covering visual details, equipment temperature distribution, and operational acoustic features of railway rooftop operations) are used as input to the shared feature layer (the core feature layer responsible for unifying dimensions, aggregating information, and reducing noise of the multimodal fusion features, providing general and effective basic feature support for various specialized detection heads). After lightweight convolution and adaptive pooling operations, the shared feature representation (a standardized feature vector obtained after processing by the shared feature layer, containing both multimodal global semantic information and local key details) is output. Subsequently, the shared feature representation is input to the personnel violation detection head (focused on extracting human posture, action sequences, and operational behavior features from the shared features to determine whether workers are engaging in unsafe behaviors such as not wearing protective equipment or illegally touching equipment). The system comprises a detection unit, an equipment temperature anomaly detection head (a specialized detection unit that combines infrared modal feature information to analyze the temperature distribution patterns of different areas of the equipment and identify abnormal conditions such as high temperatures exceeding safety thresholds and uneven local temperatures), and a structural defect detection head (a specialized detection unit that focuses on the structural morphological characteristics of railway roof equipment and detects physical defects such as bolt loosening, shell cracks, and component deformation). Through the built-in classifiers and regressors of each detection head, targeted feature extraction and calculation are performed, ultimately generating visual detection results (including detection frames, positioning positions, categories, and visually relevant detection conclusions corresponding to personnel violations and equipment structural defects), temperature detection results (including detection frames, positioning positions, categories, and specialized detection conclusions corresponding to equipment temperature anomalies), and sound detection results (segmentation masks, and acoustically relevant detection conclusions corresponding to abnormal equipment operation noises and noises accompanying violations).

[0127] As a further improvement, the model training process for the pre-set railway train roof maintenance operation safety detection model is as follows:

[0128] The system acquires visual images, infrared images, and sound signals for model training, preprocesses these images and signals, and outputs the preprocessed visual images, infrared images, and sound features.

[0129] An initial railway train roof maintenance operation safety detection model is used to generate visual detection results, temperature detection results, and sound detection results for model training based on preprocessed visual images, infrared images, and sound features.

[0130] An initial railway train roof maintenance operation safety detection model is trained using a preset loss function based on visual detection results, temperature detection results, and sound detection results used for model training, thus determining the trained preset railway train roof maintenance operation safety detection model.

[0131] It should be noted that the preset loss function consists of a personnel violation detection branch (IoU loss function, Intersection over Union Loss Function), an equipment temperature anomaly detection branch (Dice Loss Function), and a structural defect detection branch (Focal Loss Function). The preset loss function is as follows:

[0132] ;

[0133] in, The loss value corresponding to the preset loss function is combined with the losses of the three task branches of vision, temperature and sound to optimize the joint learning effect of the model on the three types of detection results; The final weight of the i-th task branch is used to balance the importance of the three types of detection tasks (e.g., visual detection has a higher weight because it contains category information, while sound segmentation can be given a medium weight due to mask complexity). Let i be the current loss for the i-th task branch. The task branch indices correspond to 1 (visual detection branch), 2 (temperature detection branch), and 3 (sound detection branch), respectively. When i=1 (visual branch): it includes detection box regression loss (such as IoU loss, which measures the deviation between the predicted box and the real box), localization error loss (such as Euclidean distance loss), and category classification loss (such as cross-entropy loss, which distinguishes categories such as "personnel violation" and "equipment defect"). When i=2 (temperature branch): it includes temperature anomaly region detection box regression loss, localization error loss, and temperature category loss (such as "high temperature overload" and "normal temperature" classification loss). When i=3 (sound branch): it includes segmentation mask-based loss (such as Dice loss, which measures the overlap between the predicted mask and the real sound anomaly region mask). The adaptive weights for the i-th task branch; Let be the course learning weight for the i-th task branch; Let be the hyperparameters of the i-th task; Let be the difficulty coefficient of the i-th task branch; For the Sigmoid function; Let be the exponential moving average of the loss of the i-th task branch; This is the current training round number; This represents the total number of training rounds.

[0134] It is worth mentioning that the safety early warning module consists of a local early warning submodule, a remote communication submodule, and an emergency response unit. The local early warning submodule uses an audible and visual alarm device to issue alarm signals of different frequencies according to the risk level. The remote communication submodule supports the 5G / MQTT protocol and uploads the detection results and raw data fragments to the cloud management platform. The emergency response unit integrates an emergency stop signal interface, which can trigger an emergency braking process when a fatal risk is detected.

[0135] Specifically, the safety early warning module simultaneously acquires three types of core detection results: visual detection results (including target bounding boxes to locate the position, target categories such as "personnel illegally crossing", "abnormal equipment deformation", "structural cracks", etc.), infrared detection results (only segmentation masks, used to mark the pixel range of abnormal areas, such as high-temperature areas of equipment, areas covered by structural defects), and acoustic detection results (including defect categories such as "pipeline leakage sound" and "abnormal noise of equipment bearings", and defect location information). Then, the real risk event is confirmed through a dual verification of "location association + category matching": first, the physical location of the acoustic detection is mapped to image pixel coordinates through scene calibration, and then the overlap area ratio between the visual bounding box and the infrared segmentation mask is calculated (overlap ≥ 50% is considered location association), while determining whether the acoustic location falls within the overlapping area; if the location is associated and the category logic is consistent (such as visual label "structural cracks" and acoustic recognition "pipeline leakage sound", or visual label "abnormal equipment deformation" and acoustic recognition "abnormal noise of equipment"), it is determined to be the same real risk event; if there is only a single modality or category contradiction, false detection is excluded. After confirming the actual risks, the risk level is determined based on three dimensions: "risk type and severity, abnormal area range, and emergency impact timeliness." High risk corresponds to situations that directly threaten personal / equipment safety, have a large impact range, and are urgent—risk types include personnel falls, equipment overheating (potentially causing fires), and large-scale structural collapse hazards. Simultaneously, the abnormal area marked by the infrared segmentation mask exceeds 5 square meters (or the visual boundary box covers core equipment components / densely populated areas), and an accident may be triggered within 10 minutes. For example, visual detection of "unprotected personnel working at height" (boundary box positioned on a platform above 10 meters), infrared segmentation mask showing a high-temperature area on the equipment circuit board exceeding 3 square meters, and acoustic detection of short-circuit sparks are all considered high risk. Medium risk corresponds to situations that affect normal operation but do not currently endanger safety, have a moderate impact range, and a moderate timeliness—risk types include structural cracks, minor equipment damage, and other hazards. For malfunctions, the abnormal area marked by infrared / visual markers is 0.5-5 square meters, which may cause a decrease in efficiency but pose no immediate danger within 1-2 hours. Examples include visual markings of "1.5 square meters of wall cracks", infrared segmentation masks covering the corresponding area, no acoustic detection of wall compression sounds, or abnormal equipment vibration (visual detection of equipment displacement), no high temperature detected by infrared, and slight bearing noise detected by acoustics. These are classified as medium risk. Low risk corresponds to small-scale minor anomalies with no immediate impact and can be delayed in handling. Risk types include small-area wall peeling and brief personnel crossing the line. The abnormal area marked by infrared / visual markers is less than 0.5 square meters and can be handled within 24 hours. Examples include visual detection of "0.3 square meters of wall peeling" (boundary box measurement area), no high temperature detected by infrared, no acoustic anomalies, or brief personnel crossing the line (crossing distance < 1 meter, duration < 10 seconds). These are classified as low risk.Finally, the corresponding audible and visual alarms are triggered according to the risk level: high risk triggers a 5Hz alarm (5 beeps per second and LED lights flash 5 times simultaneously, with a sharp sound), medium risk triggers a 3Hz alarm (3 beeps per second and LED lights flash 3 times simultaneously, with moderate warning intensity), and low risk only logs the data without triggering an alarm. Subsequently, the three-modal detection results are refreshed every 2-5 seconds. If the risk escalates (e.g., the area of ​​the medium-risk zone expands to 6 square meters, or a dangerous abnormal sound is detected acoustically), the alarm frequency is immediately increased. If the risk degrades (e.g., the high-risk equipment cools down, or the abnormal area shrinks to 2 square meters) or disappears, the frequency is reduced or the alarm is turned off to ensure that the alarm accurately matches the real-time risk status.

[0136] Furthermore, the system also includes a cloud management platform for receiving, storing, and analyzing data uploaded by edge devices. This platform consists of three parts: 1) Federated learning update unit: enabling collaborative optimization of cross-line detection models while protecting data privacy; 2) Historical data traceability unit: establishing a maintenance safety database that supports multi-dimensional queries by time, location, equipment type, etc.; 3) Report generation unit: automatically generating monthly safety inspection reports to identify high-frequency risk areas and typical violations.

[0137] In this invention, performance tests were performed on the proposed system:

[0138] In actual testing on a certain high-speed railway section, this system demonstrated excellent performance:

[0139] 1. Detection speed: The detection time for a single frame of 4K image is 32ms, which meets the real-time requirements;

[0140] 2. Accuracy: In 1000 tests, it correctly identified 995 cases, had 3 false alarms, and missed 2 cases.

[0141] 3. Environmental adaptability: The system accuracy remains above 95% under complex conditions such as rain, snow, and night.

[0142] 4. Power consumption: The edge computing module has an average power consumption of 1.2W and supports continuous operation for 72 hours.

[0143] For comparison of technical effects, existing technologies can be used as a reference. As a vital national infrastructure, the operational safety of railway transportation is directly related to the safety of people's lives and property and socio-economic development. Regular maintenance of railway car roof equipment (such as pantographs and air conditioning units) is a crucial link in ensuring the normal operation of trains, but the maintenance work environment is complex and poses many safety hazards.

[0144] Currently, railway car roof maintenance and safety inspection mainly rely on manual inspections and traditional automated monitoring methods, which have the following technical bottlenecks:

[0145] 1. Limitations of Manual Inspection: Traditional manual monitoring relies heavily on the skill level and work attitude of operators, resulting in strong subjectivity and a high error rate, making it difficult to guarantee data reliability and consistency. Furthermore, railway lines are long and have numerous maintenance points, making manual inspections labor-intensive and inefficient, unable to achieve real-time monitoring and early warning, and difficult to detect sudden safety hazards in a timely manner. In high-risk work areas such as car roofs, manual inspections also face safety risks such as falls and electric shock.

[0146] 2. Limitations of Single-Mode Sensors: Existing automated monitoring equipment often employs single-sensor technology, which suffers from accuracy and stability issues. For example, vision sensors are susceptible to changes in lighting and weather conditions such as rain and snow; temperature sensors are affected by vibration, leading to large data fluctuations; and the placement and number of traditional sensors make it difficult to balance comprehensive monitoring with cost control, especially in complex structures like the roof area of ​​a vehicle, where conventional sensors cannot achieve real-time monitoring of the three-dimensional state.

[0147] 3. Insufficient performance of detection algorithms: Existing target detection algorithms suffer from low accuracy and poor real-time performance in railway vehicle roof scenarios. Safety hazards (such as loose bolts or lost tools) and violations (such as not wearing seat belts) in roof maintenance scenarios often manifest as small-scale targets, resulting in a false negative rate of over 15% for traditional algorithms. Furthermore, single-modal data cannot comprehensively characterize the safety status of complex maintenance scenarios, necessitating multi-modal fusion technology to improve detection robustness.

[0148] 4. Conflict between real-time performance and edge deployment: Traditional detection solutions relying on cloud computing suffer from data transmission latency issues, often exceeding 500ms, which cannot meet the real-time early warning requirements of maintenance operations. Meanwhile, existing edge computing devices still have shortcomings in multimodal data parallel processing capabilities and energy consumption control.

[0149] To address the above problems, this invention proposes a safety inspection system for maintenance operations on the roof of railway trains. Please refer to [link / reference]. Figure 6The system comprises a multimodal data acquisition module, an edge computing processing module, an intelligent detection and analysis module, and a safety early warning module. The multimodal data acquisition module integrates a high-definition visual camera, an infrared thermal imaging sensor, and an acoustic sensor to simultaneously acquire visual images, temperature distribution, and abnormal sounds from the rooftop maintenance scene. The edge computing processing module uses an RK3576 edge computer to achieve real-time preprocessing of multimodal data through hardware acceleration. The intelligent detection and analysis module, based on an improved Train-RT-DETR algorithm combined with a dynamic cross-modal attention mechanism, achieves high-precision detection of maintenance personnel's violations, abnormal equipment status, and safety hazards. The safety early warning module constructs a multi-level safety response mechanism through local audible and visual alarms and remote data transmission. This invention effectively solves the problems of poor real-time performance, low accuracy, and weak environmental adaptability in traditional railway rooftop maintenance and inspection, significantly improving the safety management level of railway maintenance operations.

[0150] Specifically, the system workflow includes the following steps:

[0151] 1. Data acquisition phase: After the system is powered on, the multimodal sensor is automatically activated, and a 10-second warm-up calibration is performed. At the same time, visual images (4K / 30fps), infrared thermal images (640×512 / 15fps) and audio signals (44.1kHz / 16bit) are acquired.

[0152] 2. Preprocessing stage: The edge computing module processes the raw data in real time, including image denoising, infrared temperature calibration (to eliminate the influence of ambient temperature), audio noise reduction, and feature extraction;

[0153] 3. Detection and Analysis Stage: The improved Train-RT-DETR model performs inference on the preprocessed data. First, it detects the position of personnel and equipment components through visual modality, and then it identifies abnormal states by fusing infrared temperature data and acoustic features through cross-modal attention.

[0154] 4. Decision-making and early warning stage: When a violation or abnormal state is detected, the system determines the risk level (general / serious / fatal) based on a preset threshold, triggers the corresponding audible and visual alarm, and uploads the early warning information through the 5G module;

[0155] 5. Model update phase: The cloud platform summarizes the detection data of each edge node every week, updates the model parameters using federated learning, and pushes them to the edge devices via OTA.

[0156] Compared with the prior art, the present invention has the following beneficial effects:

[0157] 1. Significantly Improved Detection Accuracy: By employing an improved Train-RT-DETR algorithm and a dynamic cross-modal attention mechanism, the problems of missed detections in small target detection and modal hegemony are resolved. Real-world testing shows that the accuracy rate for detecting maintenance personnel violations reaches 99.5%, and the equipment anomaly recognition rate reaches 98.7%, representing a 42% improvement compared to traditional single-modal solutions.

[0158] 2. Real-time performance meets field requirements: Based on hardware acceleration and model optimization of the RK3576 edge computer, the system end-to-end latency is controlled within 50ms, supporting real-time analysis of 4K video streams, reducing latency by 90% compared to cloud solutions.

[0159] 3. Strong environmental adaptability: The multimodal data fusion strategy overcomes the environmental limitations of a single sensor and can still maintain stable operation under complex conditions such as heavy rain, strong light, and vibration, adapting to the harsh working environment on the roof of a railway vehicle.

[0160] 4. Complies with industry safety standards: The system design fully complies with the requirements of the National Railway Administration's "Implementation Rules for Quality and Safety Supervision and Management of Railway Equipment Use and Maintenance", and has established a full-chain safety management mechanism from real-time detection to emergency response, meeting the regulatory requirements of "safety first, prevention foremost".

[0161] 5. Controllable deployment costs: The edge computing architecture reduces cloud transmission and storage costs, and the optimized sensor layout reduces single-point monitoring costs by 30%, while supporting the reuse and upgrading of existing equipment.

[0162] In this embodiment of the invention, a safety detection system for railway train roof maintenance operations is provided. The system includes a multimodal data acquisition module, an edge computing processing module, an intelligent detection and analysis module, and a safety early warning module connected in sequence. The multimodal data acquisition module acquires visual images, infrared images, and sound signals from the roof maintenance area. The edge computing processing module preprocesses the visual images, infrared images, and sound signals from the roof maintenance area, outputting target visual images, target infrared images, and target sound features. The intelligent detection and analysis module uses a preset railway train roof maintenance operation safety detection model to detect the target visual images, target infrared images, and target sound features, outputting visual detection results, temperature detection results, and sound detection results. The safety early warning module determines the risk level based on the visual detection results, temperature detection results, and sound detection results, and issues an alarm according to the risk level. Based on the above scheme, this invention, through the synchronous acquisition and real-time processing of visual, infrared, and sound three-dimensional data, and the use of a preset railway train roof maintenance operation safety detection model, can accurately extract risk information from multimodal data, effectively improving the comprehensiveness and accuracy of detection, thereby enhancing the detection effect.

[0163] Please see Figure 7 , Figure 7 This is a flowchart illustrating the steps of a safety inspection method for maintenance operations on the roof of a railway train, as provided in Embodiment 2 of the present invention.

[0164] This invention provides a safety inspection method for maintenance operations on the roof of railway trains, comprising:

[0165] Step 701: Acquire visual images, infrared images, and sound signals of the roof repair area;

[0166] Step 702: Preprocess the visual image, infrared image and sound signal of the roof repair area, and output the target visual image, target infrared image and target sound features;

[0167] Step 703: Use the preset railway train roof maintenance operation safety detection model to detect the target visual image, target infrared image and target sound features respectively, and output the visual detection results, temperature detection results and sound detection results;

[0168] Step 704: Determine the risk level based on the visual detection results, temperature detection results, and sound detection results, and issue an alarm according to the risk level.

[0169] In this embodiment of the invention, a safety detection method for railway train roof maintenance operations is provided. The method involves acquiring visual images, infrared images, and sound signals of the roof maintenance area; preprocessing the visual images, infrared images, and sound signals of the roof maintenance area to output target visual images, target infrared images, and target sound features; using a pre-set railway train roof maintenance operation safety detection model to detect the target visual images, target infrared images, and target sound features respectively, outputting visual detection results, temperature detection results, and sound detection results; determining the risk level based on the visual detection results, temperature detection results, and sound detection results, and issuing an alarm according to the risk level. Based on the above scheme, this invention, through the synchronous acquisition and real-time processing of visual, infrared, and sound three-dimensional data, and through the pre-set railway train roof maintenance operation safety detection model, can accurately mine risk information in multimodal data, effectively improving the comprehensiveness and accuracy of detection, thereby enhancing the detection effect.

[0170] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the safety inspection method for railway train roof maintenance operations as described in Embodiment 2 above.

[0171] This invention also provides a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the steps of the safety inspection method for maintenance operations on the roof of a large railway train as described in Embodiment 2 above.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A safety inspection system for maintenance operations on the roof of a railway train, characterized in that, The system includes a multimodal data acquisition module, an edge computing processing module, an intelligent detection and analysis module, and a security early warning module connected in sequence. The multimodal data acquisition module is used to acquire visual images, infrared images, and sound signals of the roof repair area; The edge computing processing module is used to preprocess the visual image, infrared image and sound signal of the roof maintenance area, and output the target visual image, target infrared image and target sound features; The intelligent detection and analysis module is used to detect the target visual image, the target infrared image and the target sound features respectively using a preset railway train roof maintenance operation safety detection model, and output visual detection results, temperature detection results and sound detection results; The safety warning module is used to determine the risk level based on the visual detection results, the temperature detection results, and the sound detection results, and to issue an alarm according to the risk level.

2. The railway train roof maintenance operation safety detection system according to claim 1, characterized in that, The edge computing processing module is specifically used for: The visual image is denoised to output the target visual image; The infrared image is calibrated for temperature, and the target infrared image is output. Feature extraction is performed on the sound signal to generate target sound features.

3. The railway train roof maintenance operation safety detection system according to claim 1, characterized in that, The preset railway train roof maintenance operation safety detection model includes a backbone network, a neck network, and a multi-task detection head; the intelligent detection and analysis module is specifically used for: The backbone network is used to extract enhancement features from the target visual image, the target infrared image, and the target sound features, respectively, and output visual enhancement features, infrared enhancement features, and sound enhancement features; The neck network is used to perform multi-scale feature fusion on the visual enhancement features, the infrared enhancement features, and the sound enhancement features, respectively, and output visual fusion multi-scale features, infrared fusion multi-scale features, and sound fusion multi-scale features. Multimodal feature alignment is performed on the visual fusion multi-scale features, the infrared fusion multi-scale features, and the sound fusion multi-scale features respectively to generate aligned visual fusion multi-scale features, aligned infrared fusion multi-scale features, and aligned sound fusion multi-scale features; Cross-modal attention weights are calculated for the aligned visual fusion multi-scale features, the aligned infrared fusion multi-scale features, and the aligned audio fusion multi-scale features, and the cross-modal weights are output. The alignment visual fusion multi-scale features, the alignment infrared fusion multi-scale features, the alignment audio fusion multi-scale features, and the cross-modal weights are fused to output multi-modal fusion features. The multimodal fusion features are used as input to the multi-task detection head, which outputs visual detection results, temperature detection results, and sound detection results.

4. The railway train roof maintenance operation safety detection system according to claim 3, characterized in that, The data processing procedure of the backbone network is as follows: The target sound features are converted into a two-dimensional spectrogram; The target visual image, the target infrared image, and the two-dimensional acoustic spectrogram are sequentially subjected to feature embedding and basic feature extraction to output a first visual feature, a first infrared feature, and a first acoustic feature. The first visual feature, the first infrared feature, and the first sound feature are sequentially embedded and basic feature extracted to output the second visual feature, the second infrared feature, and the second sound feature. The second visual feature, the second infrared feature, and the second sound feature are respectively subjected to feature embedding and basic feature extraction, and the third visual feature, the third infrared feature, and the third sound feature are output. The third visual feature, the third infrared feature, and the third sound feature are respectively subjected to feature embedding and basic feature extraction to output the fourth visual feature, the fourth infrared feature, and the fourth sound feature; The fourth visual feature, the fourth infrared feature, and the fourth sound feature are respectively used as inputs to the context prior module, and the visual context prior feature, infrared context prior feature, and sound context prior feature are output. Dynamic convolution is performed on the visual context prior features, the infrared context prior features, and the sound context prior features respectively to output the first visual dynamic convolution feature, the first infrared dynamic convolution feature, and the first sound dynamic convolution feature; The first visual dynamic convolution feature, the first infrared dynamic convolution feature, and the first sound dynamic convolution feature are respectively embedded to output the fifth visual feature, the fifth infrared feature, and the fifth sound feature. The fifth visual feature and the visual context prior feature are spliced ​​together, the fifth infrared feature and the infrared context prior feature are spliced ​​together, and the fifth sound feature and the sound context prior feature are spliced ​​together to generate visual splicing features, infrared splicing features and sound splicing features. Dynamic convolution is performed on the visual stitching features, the infrared stitching features, and the sound stitching features respectively to generate visual enhancement features, infrared enhancement features, and sound enhancement features.

5. The railway train roof maintenance operation safety detection system according to claim 3, characterized in that, The data processing procedure for the neck network is as follows: The visual enhancement feature, the infrared enhancement feature, and the sound enhancement feature are continuously upsampled respectively to output multiple visual upsampled features, multiple infrared upsampled features, and multiple sound upsampled features; Multi-scale feature extraction is performed on multiple visual upsampling features and visual enhancement features to output multiple visual multi-scale features; Multi-scale feature extraction is performed on multiple infrared upsampling features and infrared enhancement features to output multiple infrared multi-scale features; Multi-scale feature extraction is performed on multiple sound upsampling features and sound enhancement features to output multiple sound multi-scale features; Multiple visual multi-scale features, multiple infrared multi-scale features, and multiple sound multi-scale features are spliced ​​together to output visual fusion multi-scale features, infrared fusion multi-scale features, and sound fusion multi-scale features; The plurality of visual upsampling features include a first visual upsampling feature, a second visual upsampling feature, and a third visual upsampling feature; the plurality of visual multi-scale features include a first visual multi-scale feature, a second visual multi-scale feature, a third visual multi-scale feature, and a fourth visual multi-scale feature; the process of extracting multi-scale features from the plurality of visual upsampling features and the visual enhancement features to output the plurality of visual multi-scale features is specifically as follows: The visual enhancement features are subjected to a 1×1 convolution operation, then processed sequentially by a high-frequency perception module and a spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation. Finally, an upsampling operation is performed to output the first visual multi-scale features. The first visual upsampled features are subjected to a 1×1 convolution operation, then processed sequentially by the high-frequency perception module and the spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally an upsampling operation is performed to output the second visual multi-scale features. The second visual upsampled features are subjected to a 1×1 convolution operation, then processed sequentially by the high-frequency perception module and the spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally a downsampling operation is performed to output the third visual multi-scale features. The upsampled features of the third vision are subjected to a 1×1 convolution operation, then processed sequentially by the high-frequency perception module and the spatial dependency perception module. The outputs of the high-frequency perception module and the spatial dependency perception module are then fused, followed by a 3×3 convolution operation, and finally a downsampling operation to output the multi-scale features of the fourth vision.

6. The railway train roof maintenance operation safety detection system according to claim 3, characterized in that, The multi-task detection head includes a shared feature layer, a personnel violation detection head, an equipment temperature anomaly detection head, and a structural defect detection head; the data processing procedure of the multi-task detection head is as follows: The multimodal fusion features are used as input to the shared feature layer, and the shared feature representation is output. The shared feature representations are respectively input into the personnel violation detection head, the equipment temperature anomaly detection head, and the structural defect detection head for detection, generating visual detection results, temperature detection results, and sound detection results.

7. The railway train roof maintenance operation safety detection system according to claim 1, characterized in that, The model training process for the preset railway train roof maintenance operation safety detection model is as follows: The system acquires visual images, infrared images, and sound signals for model training, preprocesses these images and signals, and outputs the preprocessed visual images, infrared images, and sound features. The initial railway train roof maintenance operation safety detection model is used to generate visual detection results, temperature detection results, and sound detection results for model training based on the preprocessed visual images, infrared images, and sound features. The initial railway train roof maintenance operation safety detection model is trained using a preset loss function based on the visual detection results, temperature detection results, and sound detection results used for model training, and the trained preset railway train roof maintenance operation safety detection model is determined.

8. A safety inspection method for maintenance operations on the roof of a railway train, characterized in that, include: Acquire visual, infrared, and audio signals of the rooftop repair area; The visual image, infrared image, and sound signal of the roof maintenance area are preprocessed to output the target visual image, target infrared image, and target sound features. A preset railway train roof maintenance operation safety detection model is used to detect the target visual image, the target infrared image, and the target sound features respectively, and outputs visual detection results, temperature detection results, and sound detection results; Based on the visual detection results, the temperature detection results, and the sound detection results, the risk level is determined, and an alarm is issued according to the risk level.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the safety inspection method for maintenance operations on the roof of a railway train as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the safety detection method for railway train roof maintenance operations as described in claim 8.

Images