Urban rail train autonomous obstacle detection method and system based on long and short focus image fusion

Abstract

The invention provides an urban rail train autonomous obstacle detection method and system based on long and short focus image fusion, and belongs to the technical field of urban rail transit train obstacle detection and environmental perception, and the method comprises the steps: obtaining a long focus image and a short focus image collected in an urban rail train scene; processing the obtained long-focus image and short-focus image by using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; processing the short-focus image by using a short-focus wide-view target detection model to obtain a target detection result of the short-focus wide-view image; and performing decision-level fusion on the target detection result of the short-focus wide-view image and the long-distance small target detection result to obtain a final detection result. According to the method, information complementation of images after long and short focus fusion is fully utilized, the long and short focus image boundary long-distance small target detection capability is improved, large-view long-distance small target obstacle detection is realized, and technical support is provided for enhancing urban rail vehicle-mounted autonomous perception capability.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle detection and environment perception of urban rail transit trains, and in particular to an autonomous obstacle detection method and system for urban rail transit trains based on a long-short focal length image fusion model. Background Art

[0002] Urban rail transit is a large-capacity public transportation infrastructure and the backbone transportation network of large cities. Its safety and operating efficiency are related to the life, health and travel convenience of passengers. Active obstacle detection of trains is an important research and development direction for realizing the automated operation of urban rail transit trains. It is of great significance to effectively detect abnormal intrusive obstacles outside the train, improve the obstacle detection performance of unmanned trains, and enhance the on-board autonomous perception capability.

[0003] Considering that subway trains are heavy, have large inertia, and have long braking distances, there are high requirements for the detection distance of obstacle detection. Urban rail transit usually requires the ability to detect abnormal obstacles that affect train travel at a distance of no less than 200 meters, and effective detection of larger obstacles within 100 meters is required. Since the obstacle detection method may be directly related to the emergency braking request, the obstacle detection algorithm must have high accuracy and low false alarm rate.

[0004] Traditional obstacle detection methods mainly rely on manual or ground-based obstacle detection. The former has low detection efficiency and high labor intensity; the latter has a limited detection range, and if the entire line is covered by monitoring, the deployment cost is high. Train active obstacle detection is an important research and development direction for realizing the automated operation of urban rail transit trains and a key technology for improving train operation safety. On-board active obstacle detection installs environmental perception sensors, such as visible light cameras, and computing resources on the vehicle side, and detects obstacles ahead of the vehicle by collecting and processing images.

[0005] There are many challenges for a single camera in capturing small obstacles in front of the train. When capturing images, a single camera can either optimize distant details or expand the field of view, but not both. Long-distance shooting requires a telephoto lens, which sacrifices the field of view, while wide-angle shooting requires a short focal length lens, which compromises distant details. Long-short focal image fusion refers to the fusion of an image with clear local details taken by a telephoto lens with an image of the overall scene taken by a short focal lens to achieve the effect of retaining both the overall scene and local details.

[0006] In the application of autonomous obstacle detection in urban rail trains, considering the heavy body, large inertia and long braking distance of subway trains, there are high requirements for the detection distance, long-distance small targets and detection field of view of obstacle detection. Urban rail transit usually requires the ability to detect abnormal intrusive obstacles that affect train travel and are not less than 40cm*40cm in 200 meters. Since the obstacle detection method may be directly related to the emergency braking request, in the train transportation scenario with high safety requirements, the shorter detection distance means that potential obstacles in the distance cannot be detected in time, which greatly reduces the system's early warning capability and response time. Therefore, long-distance, small target and large field of view obstacle detection technology is of great significance.

[0007] Active obstacle detection on trains is an important research and development direction for realizing the automated operation of urban rail transit trains. It is of great significance for realizing the effective detection of abnormal intrusive obstacles outside the train, improving the obstacle detection capability of unmanned trains, and enhancing the autonomous perception capability of vehicles. In terms of acquiring small obstacles at a distance ahead of the train, a single camera either focuses on small obstacles at a distance or expands the field of view to focus on close targets, but it cannot take into account both large field of view and small obstacle detection at a distance. Long-distance shooting requires a telephoto lens, which sacrifices the field of view, while wide-angle shooting requires a short focal length lens, which loses details at a distance. Summary of the invention

[0008] The object of the present invention is to provide an urban rail train autonomous obstacle detection method and system based on a long-short focal length image fusion model to solve at least one technical problem existing in the above-mentioned background technology.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides an autonomous obstacle detection method for an urban rail train, comprising:

[0011] Obtaining the collected long-focus image and short-focus image in the urban rail train scene;

[0012] The acquired long-focus image and short-focus image are processed by using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a plurality of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image by the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result;

[0013] Processing the short-focus image using a short-focus-length and wide-field-of-view target detection model to obtain a target detection result of the short-focus-length and wide-field-of-view image;

[0014] The target detection results of the short focal length and wide field of view image are fused with the long-distance small target detection results at the decision level to obtain the final target detection results.

[0015] As a further limitation of the first aspect, the target detection results of the short focal length and wide field of view image are fused at the decision level to obtain the final target detection results, including: applying non-maximum suppression to the detection results of small targets at a long distance to eliminate redundant repetitions in the detection results, NMS removes the detection frames with lower confidence in the short focal length image by comparing the confidence of the detection frames in the overlapping areas, and retains the most representative target frames; using non-maximum suppression, the same target detected in the long and short focal length images is screened according to the confidence of the target frame, leaving the detection frames with higher confidence.

[0016] As a further limitation of the first aspect, the features of the telephoto image and the short-focus image are fused, and the feature information of the small target with partial feature missing at the edge of the telephoto image is supplemented by the complete small target feature information of the short-focus image, including: introducing a self-attention mechanism to calculate the correlation of different positions in the feature sequence, dynamically assigning weights, focusing on the small target area with missing features in the telephoto image, and extracting relevant information from the short-focus image for supplement; introducing a position encoding mechanism to embed spatial position information into the feature sequence, determine the source and spatial position of each feature, and achieve feature alignment and integration between modalities.

[0017] As a further limitation of the first aspect, convolution feature extraction is performed on the telephoto image and the short-focus image respectively to obtain their feature maps; the features of the feature maps of the telephoto image and the short-focus image are spliced ​​and position-encoded; the spliced ​​and position-encoded features are divided into multiple non-overlapping local windows using a sliding window, and their local window attention is calculated, the self-attention weight matrix is ​​calculated, and the value matrix is ​​weighted using the self-attention weight to obtain an updated feature representation; the standard multi-head self-attention of the shifted window is calculated again, and the result is recalibrated to the same feature image as the input, and it is added to the original modal branch as supplementary information; multi-scale feature fusion is realized using weighted bidirectional feature pyramid networks respectively, and finally feature maps of multiple sizes are obtained.

[0018] As a further limitation of the first aspect, in the classification unit, all detected target frames are sorted according to their confidence levels, with the bounding frame with the highest confidence level at the front; starting from the bounding frame with the highest confidence level, the remaining bounding frames are traversed in sequence; for the currently traversed bounding frame, if it overlaps with the previously retained bounding frame to a certain degree, that is, the IOU value is calculated, if the IOU is greater than a certain threshold, it is suppressed and not retained; continue to traverse the next target frame, and repeat the above process until all target frames have been processed; the target frame remaining at the end is the target frame of the distant small target captured by the telephoto image that integrates the short-focus image to supplement the missing edge feature information of the telephoto image.

[0019] As a further limitation of the first aspect, the edge area of ​​the telephoto image is filled with a number of pixel values ​​of 0 to complete the small target information with some features missing at the edge of the telephoto image, including: filling the edge area of ​​the telephoto image with p pixel values ​​of 0; assuming that the original image is I with a size of m×n, the filled image is I′ with a size of (m+2×p)×(n+2×p), where p is determined based on the size of the identified small target.

[0020] In a second aspect, the present invention provides an urban rail train autonomous obstacle detection system, comprising:

[0021] An acquisition module, used to acquire a long-focus image and a short-focus image captured in a city rail train scene;

[0022] The first detection module is used to process the acquired long-focus image and short-focus image using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a number of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image with the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result;

[0023] A second detection module is used to process the short-focus image using a short-focus wide-field-of-view target detection model to obtain a target detection result of the short-focus wide-field-of-view image;

[0024] The fusion module is used to perform decision-level fusion on the target detection results of the short focal length and wide field of view image and the long-distance small target detection results to obtain the final target detection results.

[0025] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the autonomous obstacle detection method for urban rail trains as described in the first aspect is implemented.

[0026] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the autonomous obstacle detection method for urban rail trains as described in the first aspect.

[0027] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the autonomous obstacle detection method for urban rail trains as described in the first aspect.

[0028] The beneficial effects of the present invention are as follows: the information complementarity of the images after long- and short-focus fusion is fully utilized, the long-distance small target detection capability at the boundary of long- and short-focus images is improved, the small target obstacle detection at a large field of view and a long distance is realized, and technical support is provided for enhancing the autonomous perception capability of urban rail vehicles.

[0029] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0031] Figure 1 The present invention is a flowchart of an autonomous obstacle detection method for urban rail trains based on a long- and short-focus image fusion model according to an embodiment of the present invention.

[0032] Figure 2 This is a diagram of the detection results of a short-focus camera in a subway tunnel according to an embodiment of the present invention.

[0033] Figure 3 This is a diagram of the detection results of long- and short-focus fusion with the same field of view in a subway tunnel according to an embodiment of the present invention.

[0034] Figure 4 This is a diagram of the detection results of small targets with a wide field of view using a fusion of long and short focal lengths in a subway tunnel according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0036] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0037] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0038] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0039] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0040] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0041] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0042] The present invention proposes a long-short focal length image fusion network model (SwinTransformer Fuse with Different Focal Length Cameras, STFDFL) based on the SwinTrans model, which makes full use of the information complementarity of the image after long-short focal length fusion, improves the ability to detect small targets at a long distance at the boundary of long-short focal length images, realizes the detection of small targets with a large field of view and obstacles at a long distance, and provides technical support for enhancing the autonomous perception capability of urban rail vehicles. The present invention combines the images of long-short focal length cameras, and avoids the incompleteness of small targets at a long distance by padding the pixels of the long-short focal length images for the same field of view of the long-short focal length cameras. The complementary information provided by the bidirectional interpolation of the short focal length images with the same field of view is combined, and a long-short focal length dual-path fusion network STFDFL based on the SwinTransformer model is proposed to realize the information fusion within the same field of view of the long-short focal length cameras, and improves the detection of small targets at a distance, especially the detection accuracy of incomplete small targets at the edge of the image. At the same time, compared with the computational complexity of the traditional Transfomer model, which is compared with the square of the image size, the SwinTransfomer obtains a computational complexity that is linear with the image size, which effectively improves the computational efficiency of the algorithm. Finally, the advantage of the short-focus camera's large field of view is combined to achieve accurate detection of small targets at a large viewing angle and a long distance of autonomous vehicle-mounted obstacles, meet the requirements of abnormal intrusion obstacle detection of no less than 200 meters and small target detection of 40cm*40cm, and provide support for improving the efficiency and accuracy of autonomous obstacle detection of urban rail trains. The present invention uses a telephoto camera to achieve long-distance, small target detection and recognition, and uses a short-focus camera to achieve close-range, wide-view target detection and recognition. For the same field of view of long and short focal lengths, the detection accuracy of incomplete small targets at long distances at the image boundary is improved by filling in image pixels and fusing the Swin Tranformer multi-sensor attention mechanism. The method includes: (1) long-distance small target detection and recognition; in order to more effectively utilize the image information of different viewing angles of the same area of ​​the long and short focal length cameras (due to different camera installation positions), bilinear interpolation is used to interpolate and enlarge the same area in the short focal length camera image to the same size as the telephoto camera. In order to detect incomplete small targets at long distances at the image boundary, the telephoto camera is filled so that it can include complete targets.Combined with the long- and short-focus dual-path fusion network based on SwinTransformer, the complete small target feature information of the short-focus image is used to supplement the small target feature information with some missing features at the edge of the long-focus image, so as to realize the information fusion of the long- and short-focus images in the same area and the detection of long-distance small targets, especially the detection of incomplete small targets at a long distance at the image boundary; (2) Short-distance wide-field-of-view target detection and recognition: Use commonly used deep learning algorithms such as SSD or YOLOvx to complete the target recognition of short-focus camera images; (3) Long- and short-focus camera detection fusion: Combine the long-distance small target detection of the long-focus image and the wide field of view of the short-focus camera to realize the accurate detection of small targets with a large viewing angle and a long distance, meet the detection of abnormal intrusion obstacles of not less than 200 meters and small targets of 40cm*40cm, and provide a reference for improving the efficiency and accuracy of autonomous obstacle detection of urban rail trains.

[0043] Example 1

[0044] In this embodiment 1, firstly, an urban rail train autonomous obstacle detection system is provided, including: an acquisition module, which is used to acquire a collected long-focus image and a short-focus image in an urban rail train scene. The first detection module is used to process the acquired long-focus image and short-focus image using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to preprocess the short-focus image by bilinear interpolation and enlargement; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a number of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image through the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result. The second detection module is used to process the short-focus image using the short-focus wide-field target detection model to obtain the target detection result of the short-focus wide-field image. The fusion module is used to perform decision-level fusion on the target detection result of the short-focus wide-field image and the long-distance small target detection result to obtain the final target detection result.

[0045] In this embodiment, the above-mentioned system is used to implement an autonomous obstacle detection method for urban rail trains, including: using an acquisition module to acquire a captured long-focus image and a short-focus image in an urban rail train scene; based on a first detection module, using a pre-trained long-distance small target detection model to process the acquired long-focus image and short-focus image to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on a normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a plurality of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image through the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result. Based on the second detection module, the short-focus image is processed using the short-focus wide-field target detection model to obtain the target detection result of the short-focus wide-field image. The target detection result of the short-focus wide-field image is fused at the decision level with the long-distance small target detection result using the fusion module to obtain the final target detection result.

[0046] In this embodiment, the target detection results of the short focal length and wide field of view image are fused at the decision level to obtain the final target detection results, including: applying non-maximum suppression to the long-distance small target detection results to eliminate redundant repetitions in the detection results, NMS compares the confidence of the detection frames in the overlapping areas, removes the detection frames with lower confidence in the short focal length image, and retains the most representative target frames; using non-maximum suppression, the same target detected in the long and short focal length images is screened according to the confidence of the target frame, leaving the detection frames with higher confidence.

[0047] In this embodiment, the telephoto image and the short-focus image are feature fused, and the small target feature information with some missing features at the edge of the telephoto image is supplemented by the complete small target feature information of the short-focus image, including: long-distance dependency modeling and multimodal feature interaction are realized through the self-attention mechanism, and the global information fusion of long-distance dependency is better handled. The self-attention mechanism calculates the correlation of different positions in the feature sequence, dynamically assigns weights, focuses on the small target area with missing features in the telephoto image, and extracts relevant information from the short-focus image for supplementation. In addition, by introducing a position encoding mechanism to embed spatial position information into the feature sequence, the model can accurately perceive the source and spatial position of each feature, thereby achieving efficient feature alignment and integration between modalities.

[0048] In this embodiment, the process of fusing the long-focus image with the short-focus image specifically includes:

[0049] In the first step, convolution feature extraction is performed on the long focus image and the short focus image respectively to obtain their feature maps F L and F S These feature maps have the size H×W×C, where H and W are the height and width of the image, and C is the number of channels.

[0050] The second step is to concatenate and position-code the features of the feature map of the long-focus image and the feature map of the short-focus image. Specifically, in order to combine the features of the two images with different focal lengths, the feature sequences X of the two images are L and X S Concatenate into a new sequence X: Among them, concat is a concatenation operation, N = H × W represents the total number of pixels in the image, and C is the number of feature channels for each pixel. In order to embed the position of each feature, a position code P is added, and its size is the same as the concatenated sequence X. The position code P is to ensure that the Swin Transformer can perceive the spatial information of each position. Finally, the image after adding the position code is φ = X + P.

[0051] The third step is to use a sliding window to divide the concatenated and position-encoded features into multiple non-overlapping local windows, calculate the local window attention, calculate the self-attention weight matrix, and use the self-attention weight to weight the value matrix to obtain the updated feature representation. Specifically: Use an N*N sliding window to divide the input into non-overlapping n=HW / N 2 local windows, calculate their local window attention, where N 2 is the local sliding window size. For the local window φ p , Q = φ p W Q , K = φ p W K , V = φ p W V ,in, is the weight matrix obtained by model learning and training, d k are the dimensions of query, key, and value. Next, the self-attention weight matrix A is calculated, where A represents the correlation between each two positions in the local window input feature sequence. Through the softmax operation, the weight values ​​are normalized so that the attention distribution of each position can reflect its attention to other positions. Then, the value matrix V is weighted by the self-attention weight A to obtain the updated feature representation Z. This process assigns different weights to each position, thereby enhancing the features of small target areas in long-focus images while supplementing the detailed information in short-focus images.

[0052] In the fourth step, the standard multi-head self-attention of the shifted window is calculated again, the result is recalibrated to the same feature image as the input, and it is added to the original modality branch as supplementary information; multi-scale feature fusion is achieved using a weighted bidirectional feature pyramid network, and finally a multi-size feature map is obtained.

[0053] In this embodiment, the classification unit includes: sorting all detected target frames according to their confidence levels, with the bounding frame with the highest confidence level at the front; starting from the bounding frame with the highest confidence level, traversing the remaining bounding frames in turn; for the currently traversed bounding frame, if it overlaps with the previously retained bounding frame to a certain degree, that is, calculating the IOU value, if the IOU is greater than a certain threshold, then it is suppressed and not retained; continue to traverse the next target frame, and repeat the above process until all target frames have been processed; the target frame remaining at the end is the target frame of the small target at a long distance captured by the telephoto image that integrates the short-focus image to supplement the missing information of the edge features of the telephoto image.

[0054] The registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method, specifically including:

[0055] The registration area of ​​the long- and short-focus images is obtained based on the Normalized Cross-correlation Matching Algorithm (NCC). The information cross-correlation algorithm first preprocesses the image by denoising or grayscale conversion, while NCC needs to calculate the mean of the information value of the image after preprocessing. After calculating the means of the two images respectively, the correlation between the images is calculated by sliding windows. In the NCC algorithm, in order to eliminate the influence of the absolute intensity value on the cross-correlation value, the cross-correlation is normalized. After comparing the cross-correlation values ​​of the two images in different areas, the cross-correlation value is compared with the preset threshold. The cross-correlation value exceeding the threshold will be selected, and the area with the maximum cross-correlation value will be determined by the maximum comparison method. This area is determined as the best matching position, that is, the short-focus image registration area to be extracted.

[0056] In this embodiment, p pixels with a value of 0 are filled in the edge area of ​​the telephoto image; assuming that the original image is I with a size of m×n, the filled image is I′ with a size of (m+2×p)×(n+2×p), where p is determined based on the size of the identified small target.

[0057] Example 2

[0058] like Figure 1As shown, in this embodiment 2, an autonomous obstacle detection method for urban rail trains based on a long-short focus dual-path fusion network (STFDFL) is proposed. In terms of long-distance small target detection, the short-focus image is firstly enlarged by bilinear interpolation according to the focal length multiple relationship of the long-short focus camera, and then image registration is performed with the long-focus image; finally, the matching area in the short-focus image is extracted as one input image by image cropping. Although the long-focus camera can form a high-resolution image of a small target at a distance, the disadvantage is that the field of view is narrow. When the small target is at the edge of the long-focus image, a large area of ​​features will be lost. Based on this point, in this embodiment, it is proposed to fill the edge of the long-focus image with pixels. Through pixel filling, the integrity of the small target in the long-focus image can be maintained to avoid the situation where the image area is completely lost. Even if there is information loss in part of the long-distance small target area, the entire small target image can be kept in a complete matrix format through pixel filling. Finally, the SwinTransformer module is combined to realize the image feature fusion of the long- and short-focus matching areas. The small target feature information with complete short-focus image is supplemented with the small target feature information with some missing features at the edge of the long-focus image. The dual-channel image information is fused to realize the detection of abnormal intrusion obstacles no less than 200 meters in front of the train and the detection of small targets of 40cm*40cm. In terms of short-distance and wide-field target detection, deep learning algorithms such as SSD or YOLOvx (such as Figure 1 Finally, through the decision-level fusion of the long-distance detection of the telephoto image and the wide field of view of the short-focus camera, accurate detection of small targets at a wide viewing angle and long distance is achieved, meeting the detection requirements of small targets of 40cm*40cm at a distance of 200 meters, greatly improving the detection efficiency and accuracy.

[0059] like Figure 1 As shown, the autonomous obstacle detection method for urban rail trains based on the long-short focus dual-path fusion network (STFDFL) described in this embodiment specifically includes the following steps: First, the acquired long-focus image and short-focus image are subjected to the SwinTransformer model (long-distance small target detection model) to achieve the long-distance small target detection result combining the long-short focus images. Secondly, for the acquired short-focus image, a deep learning algorithm such as SSD or YOLOvx (i.e., a short-focus wide-field target detection model) is used to achieve the short-distance wide-field target detection result. Finally, the fusion module is used to perform decision-level fusion on the long-distance small target detection result and the short-distance wide-field target detection result to achieve accurate detection of small targets with a wide viewing angle and a long distance.

[0060] Specifically, the Swin Transformer model includes a preprocessing unit, which uses the preprocessing unit to perform bilinear interpolation magnification preprocessing on the short-focus image (matching area bilinear interpolation magnification). Due to the large imaging differences between lenses with different focal lengths, the short-focus image can only be bilinearly interpolated and magnified according to the relationship between focal length multiples, as shown in the following formula (4-1):

[0061] k=f l / f s (4-1)

[0062] where f l is the long focal length, f s is the short focal length. Then, according to the obtained focal length multiple relationship, the magnified target size of the short focal length image is obtained, and then the short focal length image is interpolated and magnified by the bilinear interpolation method. The specific bilinear interpolation method is shown in the following formula (4-2).

[0063]

[0064] The target point is (x, y), which is the point to be interpolated and located at the non-integer coordinates of the original image. The four surrounding pixels are: (x1, y1) upper left pixel, value Q1; (x2, y2) upper right pixel, value Q2; (x3, y3) lower left pixel, value Q3; (x3, y3) lower right pixel, value Q4; where x1≤x≤x2 and y1≤y≤y2, these pixels form a rectangular area, the target point (x, y) is located inside the rectangle, and its pixel value is obtained by interpolation. It can be seen from formula (4-2) that the pixel value closer to the target point has a greater impact on the result of bilinear interpolation, ensuring that the features of the image are not distorted after interpolation and amplification.

[0065] The Swin Transformer model also includes a registration unit, which obtains the registration area of ​​the long- and short-focus images based on the Normalized Cross-correlation Matching Algorithm (NCC) method. The information cross-correlation algorithm first preprocesses the image by denoising or grayscale conversion, while NCC needs to calculate the mean of the information value of the image after preprocessing, as shown in formula (4-3).

[0066]

[0067] in, is the mean value of the image pixels, Img(x,y) is the pixel value of the image at (x,y), w is the width of the image, and h is the height of the image. After calculating the mean of the two images respectively, the correlation between the images is calculated by sliding the window, as shown in formula (4-4).

[0068]

[0069] Where Rel is the cross-correlation value, w2 is the width of the template image, h2 is the height of the template image, u is the x-direction position offset of the template image in the background image, and its value range is 0≤u≤w1-w2; v is the y-direction position offset of the template image in the background image, and its value range is 0≤v≤h1-h2.

[0070] In order to eliminate the influence of absolute intensity value on cross-correlation value in NCC algorithm, cross-correlation is normalized, as shown in formula (4-5).

[0071]

[0072] Where NR is the normalized cross-correlation value. After comparing the cross-correlation values ​​of the two images in different regions, the cross-correlation values ​​are compared with the preset threshold. Those cross-correlation values ​​exceeding the threshold will be selected, and the region with the maximum cross-correlation value will be determined by the maximum comparison method. This region is determined as the best position for matching, that is, the short-focus image registration region to be extracted.

[0073] The Swin Transformer model also includes a supplementary unit, which is used to fill the edge area of ​​the telephoto image with a number of pixel values ​​​​of 0 to complete the small target information with some missing features at the edge of the telephoto image, and fuse the telephoto image with the short-focus image to supplement the small target feature information with some missing features at the edge of the telephoto image with the complete small target feature information of the short-focus image.

[0074] Among them, the edge area of ​​the telephoto image is filled with several pixels with a value of 0 (the filling size depends on the size of the small target), such as Figure 1 The edge padding shown is used to complete the small target information where some features are missing at the edge of the telephoto image. When detecting small obstacles in front of the track in the telephoto image, the problem of missing features in the edge area is particularly prominent. Due to the inherent characteristics of the telephoto lens, the details of the edge of the image are often blurred or lost, which poses a challenge to the recognition of small targets. In order to address this problem, an effective image preprocessing method is proposed in the supplementary unit of this embodiment: padding p pixels with a value of 0 in the edge area of ​​the telephoto image. Assume that the original image is I with a size of m×n, and the padded image is I′ with a size of (m+2×p)×(n+2×p), where p is determined based on the size of the identified small target. Specifically, it is shown in the following formulas (4-6)-(4-7).

[0075]

[0076]

[0077] Wherein formula (4-6) is the original image matrix, formula (4-7) is the filled image matrix table, where the pixel value of the filled area P is 0. The filling operation can be described by the following formula (4-8):

[0078]

[0079] The pixel value of the filled area is 0, which means that the pixel values ​​of the filled areas at the top, bottom, left and right of the image are all 0; the pixel value of the original image area remains unchanged, which means that in the filled image I′, except for the filled area, its pixel value is the same as the pixel value of the corresponding position of the original image I.

[0080] The core idea of ​​this pixel filling method is to artificially fill in the missing feature information due to the limitation of the optical system by expanding the edge area of ​​the image. Specifically, in this embodiment, a filling area with a width of p pixels is added to the top, bottom, left and right of the image, and the pixel values ​​of these areas are set to 0. This operation not only expands the effective range of the image, ensures that no additional noise is introduced, maintains the integrity of the original image data, but also provides more contextual information for the subsequent obstacle detection algorithm, thereby improving the recognition ability of small targets in the edge area of ​​the telephoto image.

[0081] Among them, the feature fusion of long-focus and short-focus images is also realized in the supplementary unit, specifically: based on the SwinTransformer architecture, the features of the long-focus image and the short-focus image are fused, and the feature information of the small target with some missing features at the edge of the long-focus image is supplemented by the complete small target feature information of the short-focus image, so as to realize the fusion of the same area of ​​the long-focus and short-focus images and the detection of small targets at a distance. In the feature fusion part of the long-focus and short-focus images, the introduction of Swin Transformer aims to realize long-distance dependency modeling and multimodal feature interaction through the self-attention mechanism, so as to better handle the global information fusion of long-distance dependencies. The self-attention mechanism calculates the correlation between different positions in the feature sequence, dynamically assigns weights, focuses on the small target area with missing features in the long-focus image, and extracts relevant information from the short-focus image for supplementation. Figure 1In the model, the main function of the head layer is to map the features processed by the neck layer to the final output space and generate the final prediction results of the network. The head layer is the top-level part of the model, usually a classifier or regressor, responsible for performing specific tasks such as classification, object detection, or semantic segmentation. The structure of the head layer varies depending on the task. For example, in image classification tasks, the head layer may use a softmax classifier; in object detection tasks, the head layer may contain a bounding box regressor and a classifier. In addition, the position encoding mechanism of the Swin Transformer embeds spatial position information into the feature sequence, enabling the model to accurately perceive the source and spatial position of each feature, thereby achieving efficient feature alignment and integration between modalities. The specific process of this step is as follows:

[0082] The first step is to extract and preprocess the long and short focus images. First, perform convolution feature extraction on the long focus image and the short focus image respectively to obtain their feature maps F L and F S The size of these feature maps is H×W×C, where H and W are the height and width of the image, and C is the number of channels.

[0083] The second step is to concatenate the image features and encode their positions. In order to combine the features of two images with different focal lengths, we concatenate the feature sequences X of the two images. L and X S Splice into a new sequence X, as shown in the following formula (4-9):

[0084]

[0085] Among them, concat is a concatenation operation, N = H × W represents the total number of pixels in the image. C is the number of feature channels for each pixel. In order to embed the position of each feature, a position code P is added, and its size is the same as the concatenated sequence X. The position code P is to ensure that the Swin Transformer can perceive the spatial information of each position. Finally, the input after adding the position code is shown in the following formula (4-10):

[0086] φ=X+P (4-10)

[0087] Among them, φ is the input of Swin Transformer, which will contain all the features and position information.

[0088] The third step is to use an N*N sliding window to divide the input into non-overlapping n=HW / N 2 local windows, calculate their local window attention, where N 2 is the local sliding window size. For the local window φ p, the calculation of Q, K, V is shown in the following formulas (4-11)-(4-13):

[0089] Q=φ p W Q (4-11)

[0090] K=φ p W K (4-12)

[0091] V=φ p W V (4-13)

[0092] in, is the weight matrix obtained by model learning and training, d k is the dimension of query, key and value. Next, calculate the self-attention weight matrix A as shown in the following formula (4-14):

[0093]

[0094] Here, A represents the correlation between every two positions in the local window input feature sequence. Through the softmax operation, the weight value will be normalized so that the attention distribution of each position can reflect its attention to other positions. Then, the value matrix V is weighted by the self-attention weight A to obtain the updated feature representation Z, as shown in the following formula (4-15):

[0095] Z=A×V (4-15)

[0096] This process assigns different weights to each location, thereby enhancing the features of small target areas in the long-focus image while supplementing the detailed information in the short-focus image.

[0097] In the fourth step, the Swin Transformer layer again calculates the standard multi-head self-attention (MSA) of the shifted window, which consists of window-based self-attention (W-MSA) and moving window multi-head attention (SWMSA), followed by a multi-layer perceptron (MLP) with a Gaussian error linear unit (GELU) nonlinearity between them. The LayerNorm layer is applied before both MSA and NLP, and each module uses a residual connection to construct the multi-head attention mechanism of the Transformer. Finally, the result is recalibrated to the same feature image as the input and added to the original modality branch as supplementary information. Then, BiFPN is used to implement multi-scale feature fusion, and finally a multi-size feature map can be obtained, i.e. Figure 1 Middle NECK layer.

[0098] The Swin Transformer model in this embodiment also includes a classification unit (such as Figure 1 HEAD layer), which uses non-maximum suppression (NMS) technology to process the detection results, see 4-1HEAD module, to ensure that the final obstacle detection effect can be clearly displayed on the telephoto image. The specific process of NMS is as follows: (1) All detected target boxes are sorted according to their confidence, with the bounding box with the highest confidence at the front. (2) Starting from the bounding box with the highest confidence, traverse the remaining bounding boxes in turn. (3) For the currently traversed bounding box, if it overlaps with the previously retained bounding box, the IOU value is calculated, as shown in the following formulas (4-16)-(4-17):

[0099] IOU=insec / (w1×h1+w2×h2) (4-16)

[0100] insec=max(0,min(x1+w1,x2+w2)-max(x1,x2))×max(0,min(y1+h1,y2+h2)-max(y1,y2))(4-17)

[0101] Where insec is the intersection area of ​​the target frame of the telephoto image and the target frame of the short-focus image, w1 is the normalized width of the target frame of the short-focus image, h1 is the normalized height of the target frame of the short-focus image, w2 is the normalized width of the target frame of the telephoto image, h2 is the normalized height of the target frame of the telephoto image, and IOU is the intersection-over-union ratio of the target frames of the telephoto image and the short-focus image. If the IOU is greater than a certain threshold (such as 0.5), it will be suppressed and not retained. (4) Continue to traverse the next target frame and repeat the above process until all target frames have been processed. The target frame left at the end is the target frame of the small target at a long distance captured by the telephoto image that integrates the short-focus image to supplement the missing edge feature information of the telephoto image.

[0102] Finally, in this embodiment, the target detection results of the short focal length and wide field of view image based on the YOLOvx target detection model are fused at the decision level with the target detection results of the long focal length and long-distance field of view image mentioned above through the fusion module. First, in order to eliminate the redundant repetitions in the detection results, non-maximum suppression (NMS) is also applied to the detection results of the short focal field of view. NMS removes the detection frames with low confidence in the short focal image by comparing the confidence of the detection frames in the overlapping areas, and retains the most representative target frames. After that, the long-distance small target detection results based on the long focal image obtained in the previous step are fused at the decision level with the detection results of the short focal length and wide field of view image. Using the principle of non-maximum suppression (NMS), the same target detected in the long and short focal images is screened according to the confidence of the target frame, leaving the detection frame with higher confidence. The results of the fusion of the short focal image detection in this step make up for the shortcomings of the blind spot in the obstacle detection outside the telephoto lens due to the narrow field of view of the telephoto camera.

[0103] In summary, the autonomous obstacle detection method for urban rail trains based on the long- and short-focus image fusion model provided in this embodiment 2 effectively utilizes the image information of the same area from different perspectives of the long- and short-focus cameras (due to the different camera installation positions), and obtains the same area in the short-focus camera image based on the normalized cross-correlation matching method; the short-focus area image is interpolated and enlarged to the same size as the long-focus camera by a bilinear interpolation method; a prescribed number of pixels are filled at the edge of the long-focus image to ensure the integrity of the small target image, and even if part of the area information is lost, the overall matrix format can still be maintained to avoid complete loss of the image area; based on the SwinTransformer long- and short-focus dual-path fusion network, the fusion of the long- and short-focus images of the same area and the fusion and detection of the long-distance small target information are realized, which solves the problem of missed obstacle detection due to the lack of incomplete small target features in the edge area of ​​the long-focus image, and effectively completes the detection of abnormal intrusive obstacles of not less than 200 meters and the detection of small targets of 40cm*40cm. Based on the decision-level fusion of short-focus wide-field target detection and long-focus long-distance small target detection based on the YOLOvx model, non-maximum suppression (NMS) is performed on the short-focus image detection results to remove redundancy and retain high-confidence target frames. Subsequently, combined with the NMS principle, the same target in the long-focus and short-focus images is screened according to the target frame confidence, and the detection frame with higher confidence is selected. This fusion strategy effectively makes up for the detection blind spot caused by the narrow field of view of the telephoto camera, ensuring the ability to detect small target obstacles at a wide viewing angle and long distance.

[0104] The method described in this embodiment performs target detection and recognition tests on the actual data set collected from the subway tunnel. It can effectively detect a 200-meter moving pedestrian, a 40 cm*40 cm box, and an incomplete long-distance small target (box) with some missing features. Figures 2 to 4As shown in the figure, the detection accuracy (True Positive) at a distance of 150m is 85%, 78% and 82% respectively, which basically meets the needs of autonomous obstacle target recognition for urban rail.

[0105] Example 3

[0106] This embodiment 3 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned urban rail train autonomous obstacle detection method is implemented. The method includes:

[0107] Obtaining the collected long-focus image and short-focus image in the urban rail train scene;

[0108] The acquired long-focus image and short-focus image are processed by using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a plurality of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image by the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result;

[0109] Processing the short-focus image using a short-focus-length and wide-field-of-view target detection model to obtain a target detection result of the short-focus-length and wide-field-of-view image;

[0110] The target detection results of the short focal length and wide field of view image are fused with the long-distance small target detection results at the decision level to obtain the final target detection results.

[0111] Example 4

[0112] This embodiment 4 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, the processor calls the program instructions to execute the above-mentioned urban rail train autonomous obstacle detection method, the method comprising:

[0113] Obtaining the collected long-focus image and short-focus image in the urban rail train scene;

[0114] The acquired long-focus image and short-focus image are processed by using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a plurality of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image by the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result;

[0115] Processing the short-focus image using a short-focus-length and wide-field-of-view target detection model to obtain a target detection result of the short-focus-length and wide-field-of-view image;

[0116] The target detection results of the short focal length and wide field of view image are fused with the long-distance small target detection results at the decision level to obtain the final target detection results.

[0117] Example 5

[0118] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the above-mentioned urban rail train autonomous obstacle detection method, the method comprising:

[0119] Obtaining the collected long-focus image and short-focus image in the urban rail train scene;

[0120] The acquired long-focus image and short-focus image are processed by using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a plurality of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image by the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result;

[0121] Processing the short-focus image using a short-focus-length and wide-field-of-view target detection model to obtain a target detection result of the short-focus-length and wide-field-of-view image;

[0122] The target detection results of the short focal length and wide field of view image are fused with the long-distance small target detection results at the decision level to obtain the final target detection results.

[0123] In summary, the method provided by the embodiment of the present invention solves the two major problems of short detection distance and narrow detection field of view in obstacle detection technology based on long-short focus images to the greatest extent. By using long-focus images to detect small targets at a long distance, the early warning capability of the system is greatly improved and the reaction time is reduced. For example, when a train is running at high speed, if the obstacle detection system can detect obstacles in advance at a sufficiently long distance, it can provide the train control system with sufficient time to implement braking or take other evasive measures, which undoubtedly reduces the risk of accidents. First, a new method of filling edge pixels of long-focus images is introduced. By filling a number of pixel points with a pixel value of 0 at the edge of the image, the problem of edge feature loss caused by the narrow field of view of the long-focus camera is solved, and the integrity of the small target image is ensured. Even if the information of some areas of the long-distance small target is lost, the overall matrix format can be maintained by filling, and the reliability of detection is enhanced. The bilinear interpolation of the same field of view area of ​​the near-focus image is used to supplement the incomplete small target information, avoiding the problem of missing small target features caused by ignoring the image edge when using the long-distance target detection by using the long-focus image. Secondly, a dual-channel image fusion method for long-distance small targets is proposed. The short-focus camera image is interpolated and enlarged to match the resolution of the telephoto camera with the help of bilinear interpolation technology to achieve detailed reconstruction of the same area. This interpolated and enlarged image is combined with the telephoto image and deeply fused through a long-short-focus dual-channel fusion network based on SwinTransformer. The network uses the characteristics of SwinTransformer to achieve the fusion of long-short focal length images in the same area and the detection of long-distance small targets, effectively solving the problem of missed obstacle detection caused by the lack of small target features in the edge area of ​​the telephoto image, and reducing the computational complexity to a linear ratio with the image size. This dual-channel fusion strategy not only significantly improves the accuracy of long-distance small target detection, but also effectively avoids the problem of relying only on a single perspective of a telephoto or short-focus camera in traditional methods. Traditional methods are limited by the narrow field of view of telephoto cameras when facing long-distance targets, while the detection of close-range targets relies on the wide-angle characteristics of short-focus cameras, resulting in the failure to fully integrate and utilize image information in the same scene. Through the above strategy, the overall efficient capture of long-distance small targets is achieved, and the comprehensiveness and accuracy of the detection system are enhanced. Finally, for the detection and recognition of short-distance and wide-field-of-view targets, mature deep learning algorithms such as YOLOv5 and SSD are used. By combining the high-resolution detection of long-distance small targets by the telephoto camera and the wide-field-of-view advantage of the short-focus camera, accurate detection of long-distance small targets within a wide viewing angle is achieved. This fusion method can not only meet the detection requirements of abnormal intrusive obstacles within a distance of more than 200 meters, but also detect small targets with a minimum size of 40cm*40cm, significantly improving the efficiency and accuracy of autonomous obstacle detection of urban rail trains.

[0124] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. An autonomous obstacle detection method for urban rail trains based on long- and short-focus image fusion, characterized in that: include: Obtaining the collected long-focus image and short-focus image in the urban rail train scene; The acquired long-focus image and short-focus image are processed by using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a plurality of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image by the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result; Processing the short-focus image using a short-focus-length and wide-field-of-view target detection model to obtain a target detection result of the short-focus-length and wide-field-of-view image; The target detection results of the short focal length and wide field of view image are fused with the long-distance small target detection results at the decision level to obtain the final target detection results.

2. The method for autonomous obstacle detection of urban rail trains based on long- and short-focus image fusion according to claim 1 is characterized in that: The target detection results of the short-focus and wide-field-of-view images are fused at the decision level to obtain the final target detection results, including: applying non-maximum suppression to the detection results of small targets at a long distance to eliminate redundant repetitions in the detection results. NMS removes the detection frames with lower confidence in the short-focus image by comparing the confidence of the detection frames in the overlapping areas, and retains the most representative target frames. Using non-maximum suppression, the same target detected in the long-focus and short-focus images is screened according to the confidence of the target frame, leaving the detection frames with higher confidence.

3. The method for autonomous obstacle detection of urban rail trains based on long- and short-focus image fusion according to claim 1 is characterized in that: The features of the telephoto image and the short-focus image are fused, and the feature information of the small target with partial feature missing at the edge of the telephoto image is supplemented by the complete small target feature information of the short-focus image, including: introducing a self-attention mechanism to calculate the correlation of different positions in the feature sequence, dynamically assigning weights, focusing on the small target area with missing features in the telephoto image, and extracting relevant information from the short-focus image for supplement; introducing a position encoding mechanism to embed spatial position information into the feature sequence, determine the source and spatial position of each feature, and realize feature alignment and integration between modalities.

4. The method for autonomous obstacle detection of urban rail trains based on long- and short-focus image fusion according to claim 3 is characterized in that: Perform convolution feature extraction on the long-focus image and the short-focus image respectively to obtain their feature maps; splice and position encode the features of the feature map of the long-focus image and the feature map of the short-focus image; The concatenated and position-encoded features are divided into multiple non-overlapping local windows using a sliding window, and their local window attention is calculated. The self-attention weight matrix is ​​calculated, and the value matrix is ​​weighted using the self-attention weight to obtain the updated feature representation; the standard multi-head self-attention of the shifted window is calculated again, and the result is recalibrated to the same feature image as the input, and it is added to the original modality branch as supplementary information; The weighted bidirectional feature pyramid network is used to realize multi-scale feature fusion, and finally multi-size feature maps are obtained.

5. The method for autonomous obstacle detection of urban rail trains based on long- and short-focus image fusion according to claim 1, characterized in that: In the classification unit, all detected target frames are sorted according to their confidences, with the bounding frame with the highest confidence being at the front; starting from the bounding frame with the highest confidence, the remaining bounding frames are traversed in sequence; for the currently traversed bounding frame, if it overlaps with the previously retained bounding frame to a certain extent, that is, the IOU value is calculated, if the IOU is greater than a certain threshold, it is suppressed and not retained; continue to traverse the next target frame, and repeat the above process until all target frames are processed; the target frame remaining at the end is the target frame of the small target at a long distance captured by the telephoto image that integrates the short-focus image to supplement the missing information of the edge features of the telephoto image.

6. The method for autonomous obstacle detection of urban rail trains based on long- and short-focus image fusion according to claim 1, characterized in that: The edge area of ​​the telephoto image is filled with a number of pixels with 0 values ​​to complete the small target information with some missing features at the edge of the telephoto image, including: filling the edge area of ​​the telephoto image with p pixels with 0 values; assuming that the original image is I with a size of m×n, the filled image is I′ with a size of (m+2×p)×(n+2×p), where p is determined according to the size of the identified small target.

7. An autonomous obstacle detection system for urban rail trains based on long- and short-focus image fusion, characterized in that: include: An acquisition module, used to acquire a long-focus image and a short-focus image captured in a city rail train scene; The first detection module is used to process the acquired long-focus image and short-focus image using a pre-trained long-distance small target detection model to obtain a long-distance small target detection result; wherein the long-distance small target detection model includes a preprocessing unit, a registration unit, a supplementing unit and a classification unit; the preprocessing unit is used to perform bilinear interpolation magnification preprocessing on the short-focus image; the registration unit is used to obtain the registration area of ​​the long-focus image and the short-focus image based on the normalized cross-correlation matching method; the supplementing unit is used to fill the edge area of ​​the long-focus image with a number of pixel values ​​of 0 to complete the small target information with some feature missing at the edge of the long-focus image, perform feature fusion on the long-focus image and the short-focus image, and supplement the small target feature information with some feature missing at the edge of the long-focus image with the complete small target feature information of the short-focus image; the classification unit is used to process the image after feature fusion to obtain a long-distance small target detection result; A second detection module is used to process the short-focus image using a short-focus wide-field-of-view target detection model to obtain a target detection result of the short-focus wide-field-of-view image; The fusion module is used to perform decision-level fusion on the target detection results of the short focal length and wide field of view image and the long-distance small target detection results to obtain the final target detection results.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the urban rail train autonomous obstacle detection method based on long- and short-focus image fusion as described in any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the urban rail train autonomous obstacle detection method based on long-short focal length image fusion as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the method for autonomous obstacle detection of urban rail trains based on long- and short-focus image fusion as described in any one of claims 1 to 6.

Images