Rail transit obstacle detection method, device and medium

Abstract

The present invention relates to a rail transit obstacle detection method, device, and medium. The method comprises the following steps: Step S1: capturing real-time images of the area in front of a train using infrared and visible light cameras; Step S2: performing feature-level fusion of the infrared and visible light images using the multi-source image fusion target detection model (MSOD-MFF) with multi-scale feature fusion, outputting obstacle detection results; and Step S3: issuing graded warnings based on obstacle categories. Compared with existing technologies, this invention effectively avoids the limited performance of single-sensor obstacle detection methods, improves obstacle detection accuracy at a low equipment cost, and further ensures train operation safety.

Description

Technical Field

[0001] The present invention relates to rail transit obstacle detection technology, and in particular to a rail transit obstacle detection method, device and medium based on infrared and visible light fusion. Background Art

[0002] With the continuous advancement of information technology, rail transit is experiencing a period of rapid development in intelligent technology. A new generation of urban rail transit systems, based on modern computer, communication, control, and system integration technologies, that fully automate train operations, is also being established. As a key safety measure for fully automated rail transit, train obstacle detection has become an essential function for intelligent trains. It utilizes various sensor devices to obtain information about objects near the train's current operating area and detects obstacles using traditional filtering algorithms or deep learning algorithms. Common obstacles in rail transit scenarios include pedestrians and construction workers, animals, fallen rocks, equipment and tools left on the tracks, and garbage. Real-time, high-precision detection of these objects and the implementation of appropriate feedback measures are attracting the attention of increasing numbers of scholars and rail transit industry professionals.

[0003] Common sensors used for rail transit obstacle detection include cameras, millimeter-wave radar, and lidar. Millimeter-wave radar has strong anti-interference capabilities and is not easily affected by visibility and lighting conditions, but its object detection accuracy is relatively weak and it cannot distinguish between target categories. Lidar has higher recognition accuracy and can accurately model the surrounding environment in 3D. However, environmental factors (such as rain, snow, and smoke) can disperse the lidar's beam, affecting its detection accuracy. Compared to other sensors, cameras offer intuitive imaging and are inexpensive, making them the preferred choice for rail transit obstacle detection. However, their disadvantage is poor imaging performance under extreme conditions, especially at night or in low-light conditions underground, where obstacle information cannot be obtained.

[0004] Therefore, how to further improve the obstacle detection accuracy in various complex scenarios has become a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a rail transit obstacle detection method, equipment and medium based on the fusion of infrared and visible light.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for detecting obstacles in rail transit is provided, the method comprising the following steps:

[0008] Step S1, capturing real-time images in front of the train using infrared and visible light cameras respectively;

[0009] Step S2: Based on the multi-source image fusion target detection model MSOD-MFF with multi-scale feature fusion, feature-layer fusion is performed on the infrared and visible light images to output obstacle detection results.

[0010] Step S3: Perform graded warning based on obstacle categories.

[0011] As an optimal technical solution, the multi-source image fusion target detection model MSOD-MFF performs deep feature extraction on visible light and infrared images, and performs multi-scale fusion on the intermediate features of visible light and infrared images to fully utilize the information contained in shallow and deep features to obtain the final target detection results.

[0012] As a preferred technical solution, the multi-source image fusion target detection model MSOD-MFF includes a single-source feature extraction module SSD, a progressive channel fusion module PCF and a detection output head DOH.

[0013] As a preferred technical solution, the single source feature extraction module SSD is based on the visible light image I RGB and infrared image I I As input, the intermediate feature layers of each level are obtained by passing them through the classic single-stage target detection model SSD with the same network parameters.

[0014] As a preferred technical solution, the single-source feature extraction module SSD uses a modified VGG-16 network as the basic model and adds 6 convolutional layers thereafter for further feature extraction.

[0015] As a preferred technical solution, the progressive channel fusion module PCF is used to progressively fuse the features of each level extracted by the single-source feature extraction module SSD at the channel layer.

[0016] As a preferred technical solution, the specific processing process of the progressive channel fusion module PCF is as follows:

[0017] First, the visible light features F at each level are obtained by the single source feature extraction module SSD. RGB and infrared signature F I As input, first set F RGB and F I Perform element-by-element addition to obtain F RGB +F I , and then processed by a residual structure, where the first branch of the residual structure converts F RGB +F I The two weights of different channels are obtained by Maximum Pooling and Multi-layer Perceptron MLP and Average Pooling and Multi-layer Perceptron MLP respectively;

[0018] Then, these two weights are added together and the final channel weights are obtained through Min Max Normalization. The second branch of the residual structure is F RGB +F I ;

[0019] Finally, the channel weights obtained from the first branch and F RGB +F I Multiply element by element to get the fused feature F of this layer. fused .

[0020] As an optimal technical solution, the detection output head DOH takes the six fusion feature layers of the progressive channel fusion module PCF as input, and corrects the default candidate box through six convolutional layers respectively.

[0021] As a preferred technical solution, the specific processing process of the detection output head DOH is as follows:

[0022] First, for the fusion feature layer i, input it into the convolution layer with a convolution kernel size of 3×3, and output the score and boundary offset of the default candidate box generated at each pixel, where i = 1, 2, ..., 6;

[0023] Then, the 8732 default candidate frames are corrected through the processing of the convolution layer, and the redundant frames are suppressed and eliminated through the non-maximum method, retaining the target frame that is most likely to be the object, and the final obstacle detection result is obtained.

[0024] As a preferred technical solution, the training and deployment process of the multi-source image fusion target detection model MSOD-MFF is as follows:

[0025] Step 1: Configure the environment and build the training and deployment environment for the obstacle detection model;

[0026] Step 2: Image acquisition: Place and fix the visible light and infrared cameras at the center of the upper end of the train head to capture the track images under different lighting conditions;

[0027] Step 3: Image annotation: Use labelme open source image annotation software to create obstacle labels for the collected images;

[0028] Step 4: Image enhancement: Perform data augmentation on the acquired images to expand the number of obstacle images and improve the generalization ability of the detection model.

[0029] Step 5: Model training: pre-train MSOD-MEFE on a public dataset and fine-tune it using the collected track obstacle images.

[0030] Step 6: Model deployment: deploy the trained MSOD-MFF on the vehicle-mounted device and configure the corresponding environment;

[0031] Step 7: Real-time detection: Turn on the camera and send the real-time captured visible light and infrared images to MSOD-MFF for real-time obstacle detection.

[0032] As a preferred technical solution, the model training is divided into two stages. In the first stage, the MSOD-MFF is trained using a public infrared fusion detection dataset. In the second stage, the weights of the model trained in the first stage are fine-tuned using the data obtained from steps one to four.

[0033] Phase 1 and phase 2 were trained for 500 and 40 epochs respectively. The optimizer used the ADAM optimizer with a weight decay of 0.0001. The initial learning rate was 1e-4 and decayed to 1e-5 after 400 epochs. The input infrared and visible light images were both cropped to 300×300 size. The loss function of both phases was the sum of the classification loss and the regression loss, where the classification loss was the softmax loss of the target category and the regression loss was the Smooth L1 loss of the bounding box regression coefficient.

[0034] As a preferred technical solution, the graded warning based on obstacle category is specifically as follows:

[0035] According to the category of the target in the detection results, different warning forms are used to remind the train staff.

[0036] According to a second aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method when executing the program.

[0037] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1) This invention is a vision-based automatic detection method that can eliminate manual inspections, improve detection efficiency, and ensure train operation safety;

[0040] 2) This paper designs a multi-source image fusion object detection model based on multi-scale feature fusion, MSOD-MFF, which takes advantage of the advantages of visible light and infrared imaging. Compared with obstacle detection algorithms based on a single source, it has higher detection accuracy and greater reliability in various complex scenes (such as insufficient lighting conditions and extreme conditions such as rain and fog);

[0041] 3) The present invention designs a multi-scale feature layer fusion strategy CF for multi-source images, which fuses the shallow and deep features of visible light and infrared imaging at the channel layer. It not only retains the rich image texture information in the shallow features but also utilizes the semantic information in the deep features, thereby improving the detection rate of obstacles of different pixel sizes.

[0042] 4) The present invention designs an early warning system based on the obstacle danger level, which can take different early warning measures according to the detection results of MSOD-MFF, that is, the type of obstacle, and effectively reduce the waste of manpower and material resources caused by sudden stops of trains due to any obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the rail transit obstacle detection method based on visible light and infrared fusion of the present invention;

[0044] Figure 2 This is a network structure diagram of the multi-source image fusion target detection model MSOD-MFF based on multi-scale feature fusion of the present invention;

[0045] Figure 3 Detailed diagram of the CF module in the multi-source image fusion target detection model based on multi-scale feature fusion of the present invention;

[0046] Figure 4 This is a flow chart for training and deploying the rail transit obstacle detection model of the present invention;

[0047] Figure 5 This is an example diagram of the interface for labeling track obstacles using labelme of the present invention;

[0048] Figure 6 This is a schematic diagram of the graded warning system based on obstacle hazard levels of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0050] To improve obstacle detection accuracy in various complex scenarios, the present invention designs a rail transit obstacle detection method based on visible light and infrared fusion. Taking images acquired by a visible light camera and an infrared camera as input, the multi-source image fusion target detection model MSOD-MFF based on multi-scale feature fusion designed by the present invention is used to fuse features at all levels of the two images. This effectively avoids the limited performance of single-sensor obstacle detection methods, improves obstacle detection accuracy at a low equipment cost, and further ensures train operation safety.

[0051] Unlike visible light imaging, which is easily affected by lighting and weather conditions, infrared imaging cannot form color image information, but can produce grayscale images based on the thermal radiation of objects. It has the advantages of not relying on external lighting, being able to work at night, and being less affected by electromagnetic interference. Figure 1 As shown, the present invention provides a rail transit obstacle detection method based on visible light and infrared fusion, which includes the following steps:

[0052] Step S1: The visual sensor acquires images and uses infrared and visible light cameras to capture real-time images in front of the train.

[0053] Step S2: obstacle detection based on multi-source images, using the multi-source image fusion target detection model MSOD-MFF based on multi-scale feature fusion to fuse the infrared and visible light images at the feature level and output obstacle detection results;

[0054] Step S3: Perform graded warnings based on obstacle categories. Different warning methods are used to remind train staff based on the category of the target in the detection results.

[0055] In the above obstacle detection process, step S2, obstacle detection based on multi-source images, and step S3, graded warning based on obstacle categories, are the key points of the present invention, and their details are described in detail below.

[0056] For the obstacle detection based on multi-source images in step S2, since there are many types of obstacles in rail transit and their sizes are unknown, in order to effectively utilize infrared and visible light information and deal with the problems of low accuracy and high false alarm rate of small target detection, the present invention designs a multi-source image fusion object detection model based on multi-scale feature fusion (MSOD-MFF), whose network structure is as follows: Figure 2As shown in the figure, MSOD-MFF extracts deep features from visible and infrared images and performs multi-scale fusion of intermediate features of visible and infrared images to fully utilize the information contained in shallow and deep features to obtain the final target detection results. It includes three modules: (1) single-source feature extraction module SSD, (2) progressive channel fusion module PCF, and (3) detection output head DOH. The details of each module are described below.

[0057] Module 1: Single source feature extraction module. Using visible light image I RGB (ie RGB image) and infrared image I I The input is passed through the classic single-stage object detection model SSD with the same network parameters to obtain the intermediate feature layers at each level. The SSD model uses a modified VGG-16 network as the base model and adds six convolutional layers to further extract features.

[0058] Module 2: Progressive channel fusion module PCF. For deep neural networks, different levels of intermediate feature layers will be generated in the process of extracting image features. The shallow features are closer to the input, and their receptive fields and the overlapping areas of the receptive fields are relatively small, containing more image color, texture, edge, corner point and other information; the deep features are farther away from the input, and their receptive fields are constantly increasing, and the overlapping areas between the receptive fields are also increasing, containing more abstract semantic information. Combining shallow features and deep features can effectively improve target detection performance and help improve the detection accuracy of targets with different pixel sizes. The present invention designs a progressive channel fusion module PCF to progressively fuse the features of each level extracted by module one at the channel layer. The specific structure is as follows: Figure 3 As shown, the visible light features F at each level obtained by module 1 RGB and infrared signature F I As input, first set F RGB and F I Perform element-by-element addition to obtain F RGB +F I , and then processed by a residual structure, where the first branch of the residual structure converts F RGB +F I The two weights of different channels are obtained by Maximum Pooling and Multi-layer Perceptron MLP and Average Pooling and Multi-layer Perceptron MLP respectively, and then the two weights are added together and the final channel weights are obtained by Min Max Normalization. The second branch of the residual structure is F RGB +F I . By combining the channel weights obtained from the first branch and F RGB +F IMultiply element by element to get the fused feature F of this layer. fused .

[0059] Module 3: Detect output head DOH. Figure 2 As shown in Figure 2, MSOD-MFF obtains 6 fused intermediate feature layers after module 2. The shallow fusion feature map contains more image detail information and is more suitable for small obstacle detection; the deep fusion feature map contains more global semantic information and is more suitable for large obstacle detection. Therefore, for each pixel point of each fusion feature layer, several default candidate boxes are given as the potential location box of the target. The properties of each fusion feature layer and the settings of the default candidate boxes are shown in Table 1. According to this setting, a total of 8732 default candidate boxes can be generated.

[0060] Table 1

[0061]

[0062] The detection output head (DOH) takes the six fused feature layers of module 2 as input and corrects the default candidate boxes through six convolutional layers. Specifically, for fused feature layer i (i=1, 2, ..., 6), it is input into a convolutional layer with a convolution kernel size of 3×3, and the score and boundary offset (i.e., bounding box regression parameters) of the default candidate box generated at each pixel are output. The 8732 default candidate boxes are corrected through the convolutional layer processing, and then the non-maximum method is used to suppress and eliminate redundant boxes, retaining the target box that is most likely to be the object, thus obtaining the final obstacle detection result.

[0063] The training and deployment processes of the obstacle detection model MSOD-MFF introduced above are as follows Figure 4 The specific explanation is as follows:

[0064] Step 1: Configure the environment. Build a training and deployment environment for the obstacle detection model. The training environment hardware includes an NVIDIA GeForce RTX 3090 graphics card, an Intel Core i7 CPU, a monitor, a power supply, and other common computer equipment. The deployment environment hardware includes an NVIDIA Jetson AGX Xavier graphics card, an Intel Core i7 CPU, a monitor, a power supply, and other common computer equipment. The training and deployment software uses the PyTorch deep learning framework, the PyCharm IDE interface, the Python language, and common software packages such as NumPy and Pandas.

[0065] Step 2: Image Acquisition: Place and secure visible light and infrared cameras at the center of the upper end of the train head to capture images of the track under different lighting conditions.

[0066] Step 3: Image annotation. Use the open-source labeling software LabelMe to label obstacles in the captured images. The annotation format here is a rectangular box, which records the pixel coordinates of the top and bottom right corners of the obstacle. Figure 5 The figure shows an interface for labelme to mark track obstacles.

[0067] Step 4: Image Enhancement. Data augmentation is performed on the acquired images to increase the number of obstacle images and improve the generalization capability of the detection model. Image augmentation operations include horizontal and vertical flipping, random rotation, and random cropping of the images.

[0068] Step 5: Model Training. The multi-source image fusion object detection model, MSOD-MFF, based on multi-scale feature fusion, is trained in two stages. Stage 1 uses publicly available infrared fusion detection datasets (OTCBVS, INO, TNO, etc.) to train MSOD-MFF. Stage 2 uses the data from steps 1 to 4 to fine-tune the weights of the trained model in stage 1 for more accurate rail transit obstacle detection. Stages 1 and 2 are trained for 500 and 40 epochs, respectively. The optimizer uses the ADAM optimizer with a weight decay of 0.0001. The learning rate is initialized at 1e-4 and decays to 1e-5 after 400 epochs. The input infrared and visible light images are cropped to 300×300 pixels. The loss function for both stages is the sum of the classification loss and the regression loss: the classification loss is a softmax loss for the target class, and the regression loss is a smooth L1 loss for the bounding box regression coefficient.

[0069] Step 6: Model deployment. Place the NVIDIA Jetson AGX Xavier graphics card in the vehicle-mounted device and connect it to infrared and visible light cameras, displays, and other devices. Migrate the detection model trained in Step 6 to the NVIDIA Jetson AGX Xavier graphics card for later real-time detection.

[0070] Step 7: Real-time Detection. The infrared and visible light cameras are turned on, and the CPU transmits the captured images to the AGX image processing unit. This is where the inference phase of the MSOD-MFF detection model is performed, enabling real-time detection of rail transit obstacles.

[0071] For the graded warning based on the obstacle danger level in step S3, different types of obstacles have different impacts on train safety. For example, pedestrians who maliciously intrude are high-risk obstacles, which seriously affect the normal operation of the train and require the train to stop immediately. Some small equipment and tools left on the rails include screwdrivers, nails, etc., which may affect the operation of the train. If the train is stopped immediately, it will cause panic among passengers. Such items are medium-risk obstacles and can be immediately fed back to the driver's cab. The train management and control personnel will determine whether to stop the vehicle urgently and deal with it later. The specific judgment and warning process of the graded warning system based on the obstacle danger level is as follows: Figure 6 shown.

[0072] Specifically, the present invention divides common obstacles into three categories according to their danger levels: (1) Category I is humans or animals, which is a high-risk level and a red alarm is issued, that is, an emergency braking request is immediately made to the vehicle to prevent serious safety accidents and the detection records are saved in time; (2) Category II is objects such as wrenches or nails that may affect driving safety, which is a medium-risk level and a yellow alarm is issued, that is, the obstacle category and pixel size are reported to the train management and control personnel based on the detection results of MSOD-MFF, and the train management and control personnel decide whether to take emergency braking and other safety defense measures; (3) Category III is objects such as tiny pieces of paper on the roadside that have no impact on driving safety and a low-risk level and a green alarm is issued, that is, only the obstacle detection report is saved without timely feedback, and a dedicated person conducts regular inspections to clear the obstacles.

[0073] The above is an introduction to a method embodiment. The following further illustrates the solution of the present invention through an electronic device and a storage medium embodiment.

[0074] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0075] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0076] The processing unit performs the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S3 by any other appropriate means (e.g., by means of firmware).

[0077] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0078] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0079] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A rail transit obstacle detection method, characterized in that: The method comprises the following steps: Step S1, capturing real-time images of the front of the train using infrared and visible light cameras respectively; Step S2: Based on the multi-source image fusion target detection model MSOD-MFF with multi-scale feature fusion, feature-layer fusion is performed on the infrared and visible light images to output obstacle detection results. Step S3, performing graded warning based on obstacle categories; The multi-source image fusion target detection model MSOD-MFF includes a single-source feature extraction module SSD, a progressive channel fusion module PCF and a detection output head DOH; The progressive channel fusion module PCF is used to progressively fuse the features of each level extracted by the single source feature extraction module SSD at the channel layer; The specific processing process of the progressive channel fusion module PCF is as follows: First, the visible light features F at each level are obtained by the single source feature extraction module SSD. RGB and infrared signature F I As input, first set F RGB and F I Perform element-by-element addition to obtain F RGB +F I , and then processed by a residual structure, where the first branch of the residual structure converts F RGB +F I The two weights of different channels are obtained by Maximum Pooling and Multi-layer Perceptron MLP and Average Pooling and Multi-layer Perceptron MLP respectively; Then, these two weights are added together and the final channel weights are obtained through Min Max Normalization. The second branch of the residual structure is F RGB +F I ; Finally, the channel weights obtained from the first branch and F RGB +F I Multiply element by element to get the fused feature F of this layer. fused .

2. A rail transit obstacle detection method according to claim 1, characterized in that: The multi-source image fusion target detection model MSOD-MFF extracts deep features from visible light and infrared images, and performs multi-scale fusion of intermediate features from visible light and infrared images to fully utilize the information contained in shallow and deep features to obtain the final target detection result.

3. A rail transit obstacle detection method according to claim 1, characterized in that: The single source feature extraction module SSD is based on the visible light image I RGB and infrared image I I As input, the intermediate feature layers of each level are obtained by passing them through the classic single-stage target detection model SSD with the same network parameters.

4. A rail transit obstacle detection method according to claim 1, characterized in that: The single-source feature extraction module SSD uses a modified VGG-16 network as the base model and adds 6 convolutional layers to it for further feature extraction.

5. The rail transit obstacle detection method according to claim 1, characterized in that: The detection output head DOH takes the six fusion feature layers of the progressive channel fusion module PCF as input, and corrects the default candidate box through six convolutional layers respectively.

6. A rail transit obstacle detection method according to claim 5, characterized in that: The specific processing process of the detection output head DOH is as follows: First, for the fusion feature layer i, input it into the convolution layer with a convolution kernel size of 3×3, and output the score and boundary offset of the default candidate box generated at each pixel, where i = 1, 2, ..., 6; Then, the 8732 default candidate frames are corrected through the processing of the convolution layer, and the redundant frames are suppressed and eliminated through the non-maximum method, retaining the target frame that is most likely to be the object, and the final obstacle detection result is obtained.

7. A rail transit obstacle detection method according to claim 1, characterized in that: The training and deployment process of the multi-source image fusion target detection model MSOD-MFF is as follows: Step 1: Configure the environment and build the training and deployment environment for the obstacle detection model; Step 2: Image acquisition: Place and fix the visible light and infrared cameras at the center of the upper end of the train head to capture the track images under different lighting conditions; Step 3: Image annotation: Use labelme open source image annotation software to create obstacle labels for the collected images; Step 4: Image enhancement: Perform data augmentation on the acquired images to expand the number of obstacle images and improve the generalization ability of the detection model. Step 5: Model training: pre-train MSOD-MFF on a public dataset and fine-tune it using collected track obstacle images. Step 6: Model deployment: deploy the trained MSOD-MFF on the vehicle-mounted device and configure the corresponding environment; Step 7: Real-time detection, turn on the camera and send the real-time captured visible light and infrared images to MSOD-MFF, performs real-time obstacle detection.

8. A rail transit obstacle detection method according to claim 7, characterized in that: The model training is divided into two stages. In the first stage, the public infrared fusion detection dataset is used to train MSOD-MFF. In the second stage, the data obtained from steps 1 to 4 are used to fine-tune the model weights trained in the first stage. Phase 1 and phase 2 were trained for 500 and 40 epochs respectively. The optimizer used the ADAM optimizer with a weight decay of 0.0001. The initial learning rate was 1e-4 and decayed to 1e-5 after 400 epochs. The input infrared and visible light images were both cropped to 300×300 size. The loss function of both phases was the sum of the classification loss and the regression loss, where the classification loss was the softmax loss of the target category and the regression loss was the Smooth L1 loss of the bounding box regression coefficient.

9. The rail transit obstacle detection method according to claim 1, characterized in that: The specific steps of the graded warning based on obstacle category are as follows: According to the category of the target in the detection results, different warning forms are used to remind the train staff.

10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images