Rail train environment sensing system based on ARM platform AI application deployment

Abstract

A rail train environment sensing system based on ARM platform AI application deployment comprises a data acquisition module, an AI application module, a multi-modal algorithm module and an application display module, the data acquisition module acquires to-be-detected image data and preprocesses the to-be-detected image data through an ARM platform, the preprocessed image is transmitted to an edge computing server, the AI application module is called to recognize a target image, and the application display module displays the target image. The application display module sends a processing result of the multi-modal algorithm module to a display terminal platform; the edge computing platform based on the ARM architecture can support real-time reasoning of a lightweight deep learning model by means of the characteristics of low power consumption and high parallel processing, provides a reliable carrier for on-orbit processing of multi-modal sensing data, and meets the strict requirements of a train operation control system for safety standards.

Description

Technical Field

[0001] This invention relates to the field of track safety detection technology, and in particular to a track train environmental perception system based on ARM platform AI application deployment. Background Technology

[0002] Railway dedicated lines and freight yard transportation are rapidly evolving towards intelligent assisted driving and unmanned driving. However, traditional manual inspection and basic sensor solutions suffer from bottlenecks such as low efficiency, high missed detection rates, and insufficient real-time performance. Current track environment perception requires the integration of data from multiple sensors, including vision and lidar, to achieve accurate identification of targets such as track obstacles and signal status. However, the dynamic perception and high reliability requirements in complex scenarios pose challenges to the computing power and energy efficiency of hardware platforms, making it difficult for traditional industrial control equipment to balance computing power and deployment costs. Summary of the Invention

[0003] This invention aims to address the shortcomings of existing technologies by providing a rail train environmental perception system based on ARM platform AI application deployment.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A rail train environment perception system based on ARM platform AI application deployment includes a data acquisition module, an AI application module, a multimodal algorithm module, and an application display module. The data acquisition module acquires image data to be detected, preprocesses it through the ARM platform, and transmits the preprocessed image to the edge computing server. The AI ​​application module is called to identify the target image. The application display module sends the processing results of the multimodal algorithm module to the display terminal platform.

[0006] The data acquisition module refers to the use of sensors in a multimodal intelligent sensing terminal to acquire image data from the site. Under the time-series control of the video, the acquired images are preprocessed by the ARM platform and then transmitted to the edge computing server for further processing. The multimodal image data includes data acquired by telephoto cameras, short-focus cameras, infrared cameras, and millimeter-wave radar.

[0007] The AI ​​application module is a module that uses a deep learning framework to call and verify the constructed edge AI application model, detects real-time acquired images, and outputs detection results. The edge AI application model includes an orbital semantic segmentation model and a regional intrusion detection model.

[0008] The multimodal algorithm module refers to processing the multi-sensor results detected by the AI ​​model, fusing the target detection and orbit segmentation results from telephoto cameras, short-focus cameras, and infrared cameras with the ranging results from the ranging radar. This includes multiple processing algorithms such as multimodal fusion calculation, radar distance processing algorithm, and safety threshold partitioning algorithm.

[0009] The application display module refers to displaying the results of information fusion of images collected by multiple sensor terminals in the train cab, allowing the driver or safety officer to easily observe the results. The results display includes various fused information such as detection target bounding boxes, category information, distance information, and warning level information.

[0010] The edge AI application models include a track semantic segmentation model and a region intrusion detection model;

[0011] The track semantic segmentation model is characterized by segmenting train tracks using a Yolov8 network structure. The backbone adopts an improved CSPDarknet53 structure, which integrates cross-stage local connections to optimize feature extraction efficiency. The neck uses PAN-FPN for multi-scale feature fusion to enhance the perception of small targets. The head outputs a segmentation mask and bounding box (BBox) to support end-to-end instance segmentation. An SE module is added before the output features of the backbone network to strengthen global channel dependencies. An SE module is embedded in the upsampling path of PAN-FPN to optimize the multi-scale feature fusion weights.

[0012] The regional intrusion detection model uses the pre-defined geofence of the area segmented by the aforementioned track to determine whether there is any foreign object intrusion within the area, and determines the alarm level of the foreign object based on the distance and the range of the area.

[0013] The CSPDarknet stages consist of four stages, from Stage 1 to Stage 4. Each stage comprises multiple improved C3 modules. These C3 modules reduce redundant computation by splitting the feature stream through cross-stage local connections.

[0014] The formula is: Output = Concat(Xmain, Conv(Xside)) + Xmain, where Xmain is the main branch and Xside is the bypass branch;

[0015] Neck's improved PAN-FPN incorporates an SE module within the upsampling path to optimize multi-scale feature fusion weights. Specifically, deep features (20×20) are upsampled and fused with mid-level features (40×40). Shallow features (80×80) are downsampled and fused with mid-level features. SE module insertion: An SE module is embedded in the upsampled feature map to dynamically adjust the fusion weights.

[0016] Fused_Feature = SE(UP) sample (F deep ))⊙F shallow

[0017] Among them, Fdeep Deep feature map, F shallow For shallow feature maps, UP sample For upsampling, the resolution of deep features is adjusted to be consistent with that of shallow features. SE is the Squeeze-and-Excitation module, which dynamically generates channel attention weights. ⊙ is for channel-by-channel multiplication, that is, weighting the feature map of each channel.

[0018] The SE module is a channel attention mechanism that enhances the network's sensitivity to important features by dynamically adjusting the weights of each channel. It performs global information compression on each channel of the feature map, then learns the dependencies between channels through fully connected layers, and finally strengthens key channels through weighted scaling.

[0019]

[0020]

[0021] Wherein, feature map X∈R C×H×W c is the number of feature channels, H×W is the spatial size of the feature map, and z c W1 represents the global response intensity of the c-th channel, where W1∈R C / r×C The weights of the first fully connected layer compress the dimension from C to C / r, W2∈R C×C / r The weights of the second fully connected layer are restored to dimension C, where δ is the activation function and σ is the sigmoid function. The weights are normalized to the range [0,1], and the learned channel weights s are... c Applying this to the original feature map enhances key channels to obtain features.

[0022] The multimodal algorithm module fuses data acquired from different sensors to obtain comprehensive detection information. The fusion process uses a single mapping matrix to calculate the position mapping relationship, as shown in the following equation:

[0023] p′ i =Hp i

[0024] Where, p i p represents the pixel coordinates of a telephoto camera or an infrared camera. i =(x i ,y i ,1) T , p′ i Let p′ be the pixel coordinates mapped to the short focal length interface. i =(x′) i ,y′ i ,1) T H is a 3×3 simple mapping matrix. Calculate matrix H based on the mapped coordinate points.

[0025] The application display module is used to display the detection results of the multimodal system. The step of classifying the detection results into warning levels is to classify the warning levels according to preset level classification rules based on factors such as the distance, type, and number of detected targets.

[0026] The detection results include whether there are safety hazards in the area approved by the track, whether there are personnel or vehicles threatening driving safety in the danger zone and warning zone, and timely detection of dangerous behaviors and issuance of alarm information.

[0027] The edge computing server adopts a multi-source heterogeneous data access architecture: it supports parallel access of LiDAR point clouds, video streams, and wireless sensor network data; it achieves interoperability of industrial bus, 5G NR, and LoRa communication protocols through a protocol conversion layer; it supports GPUs for AI model inference; and it achieves dynamic allocation of computing resources through a microservice architecture.

[0028] The beneficial effects of this invention are: the edge computing platform based on the ARM architecture of this invention, with its low power consumption and high parallel processing characteristics, can support real-time inference of lightweight deep learning models, provide a reliable carrier for on-orbit processing of multimodal sensing data, and meet the stringent safety standards of the train operation control system. Attached Figure Description

[0029] Figure 1 This is a structural framework diagram of the present invention;

[0030] Figure 2 This invention relates to an ARM platform multi-sensor terminal;

[0031] Figure 3 This is a diagram of the edge computing server of the present invention;

[0032] Figure 4 The display terminal device of the present invention;

[0033] Figure 5 It is an auxiliary operating system combination (using 5G / WiFi connection between display terminal equipment and host);

[0034] Figure 6 It is a combination of driver assistance systems (using a wired connection to connect the display terminal device and the host);

[0035] Figure 7 It is a combination of mapless navigation systems (without a display terminal);

[0036] Figure 8 Advanced driver assistance systems for complex scenarios;

[0037] Figure 9Combination of unmanned driving systems in underground mines;

[0038] The following will describe in detail, with reference to the accompanying drawings, embodiments of the invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0040] A rail train environment perception system based on ARM platform AI application deployment includes a data acquisition module, an AI application module, a multimodal algorithm module, and an application display module. The data acquisition module acquires image data to be detected, preprocesses it through the ARM platform, and transmits the preprocessed image to the edge computing server. The AI ​​application module is called to identify the target image. The application display module sends the processing results of the multimodal algorithm module to the display terminal platform.

[0041] The data acquisition module refers to the use of sensors in a multimodal intelligent sensing terminal to acquire image data from the site. For example, under the time-series control of video, the acquired images are preprocessed by an ARM platform and then transmitted to an edge computing server for further processing. Multimodal image data includes data acquired by telephoto cameras, short-focus cameras, infrared cameras, and millimeter-wave radar. Specifically, the front-end data acquisition uses multimodal data terminal equipment, such as… Figure 2 As shown, the data acquisition terminal uses an ARM platform, which can preprocess the acquired data and directly deploy AI models with lower computing power for real-time monitoring of AI inference.

[0042] The AI ​​application module is a module that uses a deep learning framework to call and verify the constructed edge AI application model, detects real-time acquired images, and outputs detection results. The edge AI application model includes an orbital semantic segmentation model and a regional intrusion detection model.

[0043] The multimodal algorithm module refers to processing the multi-sensor results detected by the AI ​​model, fusing the target detection and orbit segmentation results from telephoto cameras, short-focus cameras, and infrared cameras with the ranging results from the ranging radar. This includes multiple processing algorithms such as multimodal fusion calculation, radar distance processing algorithm, and safety threshold partitioning algorithm.

[0044] The application display module refers to displaying the results of information fusion of images collected by multiple sensor terminals in the train cab, allowing the driver or safety officer to easily observe the results. The results display includes various fused information such as the detected target bounding box, category information, distance information, and warning level information. Figure 4 As shown.

[0045] The edge AI application models include a track semantic segmentation model and a region intrusion detection model;

[0046] The track semantic segmentation model is characterized by segmenting train tracks using a Yolov8 network structure. The backbone adopts an improved CSPDarknet53 structure, which integrates cross-stage local connections to optimize feature extraction efficiency. The neck uses PAN-FPN for multi-scale feature fusion to enhance the perception of small targets. The head outputs a segmentation mask and bounding box (BBox) to support end-to-end instance segmentation. An SE module is added before the output features of the backbone network to strengthen global channel dependencies. An SE module is embedded in the upsampling path of PAN-FPN to optimize the multi-scale feature fusion weights.

[0047] The regional intrusion detection model uses the pre-defined geofence of the area segmented by the aforementioned track to determine whether there is any foreign object intrusion within the area, and determines the alarm level of the foreign object based on the distance and the range of the area.

[0048] The CSPDarknet stages consist of four stages, from Stage 1 to Stage 4. Each stage comprises multiple improved C3 modules. These C3 modules reduce redundant computation by splitting the feature stream through cross-stage local connections.

[0049] The formula is: Output = Concat(Xmain, Conv(Xside)) + Xmain, where Xmain is the main branch and Xside is the side branch;

[0050] Neck's improved PAN-FPN incorporates an SE module within the upsampling path to optimize multi-scale feature fusion weights. Specifically, deep features (20×20) are upsampled and fused with mid-level features (40×40). Shallow features (80×80) are downsampled and fused with mid-level features. SE module insertion: An SE module is embedded in the upsampled feature map to dynamically adjust the fusion weights.

[0051] Fused_Feature = SE(UP) sample (F deep ))⊙F shallow

[0052] Among them, F deep Deep feature map, F shallow For shallow feature maps, UP sample For upsampling, the resolution of deep features is adjusted to be consistent with that of shallow features. SE is the Squeeze-and-Excitation module, which dynamically generates channel attention weights. ⊙ is for channel-by-channel multiplication, that is, weighting the feature map of each channel.

[0053] The SE module is a channel attention mechanism that enhances the network's sensitivity to important features by dynamically adjusting the weights of each channel. It performs global information compression on each channel of the feature map, then learns the dependencies between channels through fully connected layers, and finally strengthens key channels through weighted scaling.

[0054]

[0055] Wherein, feature map X∈R C×H×W c is the number of feature channels, H×W is the spatial size of the feature map, and z c W1 represents the global response intensity of the c-th channel, where W1∈R C / r×C The weights of the first fully connected layer compress the dimension from C to C / r, W2∈R C×C / r The weights of the second fully connected layer are restored to dimension C, where δ is the activation function and σ is the sigmoid function. The weights are normalized to the range [0,1], and the learned channel weights s are... c Applying this to the original feature map enhances key channels to obtain features.

[0056] The multimodal algorithm module fuses data acquired from different sensors to obtain comprehensive detection information. The fusion process uses a single mapping matrix to calculate the position mapping relationship, as shown in the following equation:

[0057] p′ i =Hp i

[0058] Where, p i p represents the pixel coordinates of a telephoto camera or an infrared camera. i =(x i ,y i ,1) T , p′ i Let p′ be the pixel coordinates mapped to the short focal length interface. i =(x′) i ,y′ i ,1) t H is a 3×3 simple mapping matrix. Calculate matrix H based on the mapped coordinate points.

[0059] The application display module is used to display the detection results of the multimodal system. The step of classifying the detection results into warning levels is to classify the warning levels according to preset level classification rules based on factors such as the distance, type, and number of detected targets.

[0060] The detection results include whether there are safety hazards in the area approved by the track, whether there are personnel or vehicles threatening driving safety in the danger zone and warning zone, and timely detection of dangerous behaviors and issuance of alarm information.

[0061] The edge computing server adopts a multi-source heterogeneous data access architecture: it supports parallel access of LiDAR point clouds, video streams, and wireless sensor network data; it achieves interoperability of industrial bus, 5G NR, and LoRa communication protocols through a protocol conversion layer; it supports GPUs for AI model inference; and it achieves dynamic allocation of computing resources through a microservice architecture. The edge computing server, such as... Figure 3 As shown, it can be configured according to actual use. For scenarios with high computing power requirements, edge computing servers and various sensors can be connected and used.

[0062] This system offers more than one combination style; you can choose the option that best suits your needs by trying the system itself. (For reference...) Figures 5-9 Suitable for system combinations in various complex scenarios.

[0063] Example 1: Intelligent Molten Iron Transportation System of a Steel Plant

[0064] Inside a steel plant, an intelligent molten iron transportation system plays a crucial role. Molten iron transport vehicles shuttle along tracks, carrying hot molten iron from the blast furnace to the steelmaking workshop.

[0065] The data acquisition module plays a fundamental role. Sensors from the multimodal intelligent terminal are distributed at key locations on the molten iron transport vehicle. During the vehicle's movement, telephoto cameras constantly monitor the distant track conditions, while short-focus cameras focus on the surrounding environment, and infrared cameras can clearly capture thermal images in low light or at night. Under the time-series control of the video, the large amount of image data collected by these cameras is pre-processed and rapidly transmitted to the edge computing server.

[0066] The AI ​​application module, utilizing a deep learning framework, calls a pre-built edge AI application model to clearly delineate the danger zone within the track and the warning zone extending beyond it, based on the segmented area of ​​the track. During molten iron transportation, when a worker approaches the track for operational purposes, or when other vehicles accidentally enter the transportation track area, the model quickly identifies and detects the target within the area. Based on the detection model's identification results and the segmented area, the system accurately classifies the warning level of the identified target. For example, if a worker steps into the danger zone, the system immediately determines it as a high-risk warning, issuing an alarm to remind relevant personnel to pay attention to safety and avoid injury or death due to the potential threat of high-temperature molten iron.

[0067] The invention has been described above with reference to the accompanying drawings. Obviously, the specific implementation of the invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution, or direct application to other situations without modification, are all within the scope of protection of the invention.

Claims

1. A rail train environmental perception system based on ARM platform AI application deployment, characterized in that, It includes a data acquisition module, an AI application module, a multimodal algorithm module, and an application display module. The data acquisition module collects the image data to be detected, preprocesses it through the ARM platform, and transmits the preprocessed image to the edge computing server. The AI ​​application module is called to identify the target image. The application display module sends the processing results of the multimodal algorithm module to the display terminal platform. The data acquisition module refers to the use of sensors in a multimodal intelligent sensing terminal to acquire image data from the site. Under the time-series control of the video, the acquired images are preprocessed by the ARM platform and then transmitted to the edge computing server for further processing. The multimodal image data includes data acquired by telephoto cameras, short-focus cameras, infrared cameras, and millimeter-wave radar. The AI ​​application module is a module that uses a deep learning framework to call and verify the constructed edge AI application model, detects real-time acquired images, and outputs detection results. The edge AI application model includes an orbital semantic segmentation model and a regional intrusion detection model. The multimodal algorithm module refers to processing the multi-sensor results detected by the AI ​​model, fusing the target detection and orbit segmentation results from telephoto cameras, short-focus cameras, and infrared cameras with the ranging results from the ranging radar. This includes multiple processing algorithms such as multimodal fusion calculation, radar distance processing algorithm, and safety threshold partitioning algorithm. The application display module refers to displaying the results of information fusion of images collected by multiple sensor terminals in the train driver's cab. The results display includes a variety of fused information, such as the detection target bounding box, category information, distance information, and warning level information.

2. The rail train environmental perception system based on ARM platform AI application deployment according to claim 1, characterized in that, The edge AI application models include a track semantic segmentation model and a region intrusion detection model; The track semantic segmentation model is characterized by segmenting train tracks using a Yolov8 network structure. The backbone adopts an improved CSPDarknet53 structure, which integrates cross-stage local connections to optimize feature extraction efficiency. The neck uses PAN-FPN for multi-scale feature fusion to enhance the perception of small targets. The head outputs a segmentation mask and bounding box (BBox) to support end-to-end instance segmentation. An SE module is added before the output features of the backbone network to strengthen global channel dependencies. An SE module is embedded in the upsampling path of PAN-FPN to optimize the multi-scale feature fusion weights. The regional intrusion detection model uses the pre-defined geofence of the area segmented by the aforementioned track to determine whether there is any foreign object intrusion within the area, and determines the alarm level of the foreign object based on the distance and the range of the area.

3. The rail train environmental perception system based on ARM platform AI application deployment according to claim 2, characterized in that, The CSPDarknet stages consist of four stages, from Stage 1 to Stage 4. Each stage comprises multiple improved C3 modules. These C3 modules reduce redundant computation by splitting the feature stream through cross-stage local connections. The formula is: Output = Concat(Xmain, Conv(Xside)) + Xmain, where Xmain is the main branch and Xside is the bypass branch; Neck's improved PAN-FPN incorporates an SE module in the upsampling path to optimize multi-scale feature fusion weights; This involves upsampling deep features and fusing them with mid-level features; downsampling shallow features and fusing them with mid-level features; and inserting an SE module: embedding an SE module into the upsampled feature map to dynamically adjust the fusion weights. Fused_Feature=SE(UP sample (F deep ))⊙F shallow Among them, F deep Deep feature map, F shallow For shallow feature maps, UP sample For upsampling, the resolution of deep features is adjusted to be consistent with that of shallow features. SE is the Squeeze-and-Excitation module, which dynamically generates channel attention weights. ⊙ is for channel-by-channel multiplication, that is, weighting the feature map of each channel. The SE module is a channel attention mechanism that enhances the network's sensitivity to important features by dynamically adjusting the weights of each channel. It performs global information compression on each channel of the feature map, then learns the dependencies between channels through fully connected layers, and finally strengthens key channels through weighted scaling. s=σ(W2·δ(W1·z)) Wherein, feature map X∈R C×H×W c is the number of feature channels, H×W is the spatial size of the feature map, and z c W1 represents the global response intensity of the c-th channel, where W1∈R C / r×C The weights of the first fully connected layer compress the dimension from C to C / r, W2∈R C×C / r The weights of the second fully connected layer are restored to dimension C, where δ is the activation function and σ is the sigmoid function. The weights are normalized to the range [0,1], and the learned channel weights s are... c Applying this to the original feature map enhances key channels to obtain features.

4. The rail train environmental perception system based on ARM platform AI application deployment according to claim 3, characterized in that, The multimodal algorithm module fuses data acquired from different sensors to obtain comprehensive detection information. The fusion process uses a single mapping matrix to calculate the position mapping relationship, as shown in the following equation: p′ i =Hp i Where, p i p represents the pixel coordinates of a telephoto camera or an infrared camera. i =(x i ,y i ,1) T , p′ i Let p′ be the pixel coordinates mapped to the short focal length interface. i =(x′) i ,y′ i ,1) T H is a 3×3 simple mapping matrix. Calculate matrix H based on the mapped coordinate points.

5. A rail train environmental perception system based on ARM platform AI application deployment according to claim 4, characterized in that, The step of classifying warning levels in the application display module is to classify warning levels according to the distance, type, and quantity of the detected targets, and in accordance with preset level classification rules.

6. A rail train environmental perception system based on ARM platform AI application deployment according to claim 5, characterized in that, The edge computing server adopts a multi-source heterogeneous data access architecture: it supports parallel access of LiDAR point clouds, video streams, and wireless sensor network data; it achieves interoperability of industrial bus, 5G NR, and LoRa communication protocols through a protocol conversion layer; it supports GPUs for AI model inference; and it achieves dynamic allocation of computing resources through a microservice architecture.

Images