Rail transit foreign matter detection method and device

Abstract

The application discloses a rail transit foreign matter detection method and device, and relates to the field of image recognition; wherein, the method comprises: acquiring a video stream of a to-be-detected area, collecting a current environment image of the to-be-detected area from the video stream, and obtaining a first image; performing image enhancement and global motion compensation on the first image, and obtaining a second image; determining a region of interest from the second image based on a currently configured dynamic electronic fence, and obtaining a third image; the dynamic electronic fence is adaptively constructed based on a dynamically updated semantic segmentation result, and the semantic segmentation result is dynamically updated based on the change of the to-be-detected area environment image over time; and judging whether the to-be-detected area has foreign matters based on the third image and a first model, and obtaining a detection result. The rail transit foreign matter detection method and device provided in the application embodiment can improve the timeliness and accuracy of foreign matter detection.

Description

Technical Field

[0001] This application relates to the fields of image recognition and image processing, and in particular to a method and apparatus for detecting foreign objects in rail transit. Background Technology

[0002] Foreign object detection in rail transit has a significant impact on railway maintenance and safe operation, while timely detection of leftover luggage in carriages is crucial to the service experience. Current technologies include regular inspections by train crews or the installation of red light strips for foreign object or luggage detection; however, both manual inspections and red light strips are highly reactive and prone to missed detections, resulting in inaccurate and untimely detection of foreign objects, threatening rail transit safety and impacting the service experience.

[0003] Therefore, overcoming the aforementioned technical problems and defects has become a key issue that needs to be addressed. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method and apparatus for detecting foreign objects in rail transit, which can improve the timeliness and accuracy of foreign object detection.

[0005] According to one aspect of this application, a method for detecting foreign objects in rail transit is provided, the method comprising: A video stream of the area to be tested is acquired, and an environmental image of the current area to be tested is collected from the video stream to obtain a first image; the area to be tested includes the activity area of ​​the rail transit machine or the luggage storage area of ​​the carriage; The first image is enhanced and global motion compensation is performed to obtain the second image; Based on the currently configured dynamic electronic fence, the Region of Interest (ROI) is determined from the second image to obtain the third image; the dynamic electronic fence is adaptively constructed based on dynamically updated semantic segmentation results, which are dynamically updated based on the changes in the environmental image of the area under test over time; Based on the third image and the first model, it is determined whether there are foreign objects in the area to be tested, and the detection result is obtained.

[0006] In the above scheme, global motion compensation is performed on the first image, including: Based on the currently configured mask image, determine the background region of the first image; Extract FAST feature points from the background region; Based on the displacement of FAST feature points in the first image and the previous frame image, the matching point pairs in the first image are determined. Based on the matching point pairs, construct the global homography matrix of the first image; The first image is subjected to perspective transformation based on the global homography matrix.

[0007] In the above scheme, image enhancement of the first image includes: The first image is converted from the RGB color space to the HSV color space to obtain an HSV image; The luminance component is extracted from the HSV image, and the extracted luminance component is enhanced based on the Multi-Scale Retinex (MSR) algorithm.

[0008] The method in the above scheme further includes: Sample frames are obtained by sampling from the video stream according to a preset sampling frequency; Calculate the average displacement of feature points in the background region of the sample frame relative to the initial frame; Determine whether the average displacement exceeds a preset drift threshold, and if it does, update the mask image and the dynamic electronic fence.

[0009] In the above scheme, updating the mask image and the dynamic electronic fence includes: The sample frames are semantically segmented using the second model to obtain semantic segmentation results; the second model is a YOLOv11 network constructed based on a lightweight semantic segmentation branch connected in parallel to the backbone output layer, and the semantic segmentation branch is generated based on the BiSeNet structure. Based on the semantic segmentation results, each pixel in the sample frame is classified to generate an updated mask image; the classification results include skateboard or background. The maximum connected component contour of the updated mask image is extracted using the findContours operator of OpenCV, and the electronic fence is reconstructed based on the extraction results to generate the updated dynamic electronic fence. In the process of extracting the maximum connected component contour of the mask image, the RDP algorithm is used to approximate the contour as a polygon.

[0010] In the above scheme, determining whether there is a foreign object in the area to be tested based on the third image and the first model, and obtaining the detection result, includes: Based on the first model, determine whether there is a foreign object in the third image and generate detection information; When the detection information includes the presence of foreign objects, the tracking target in the third image is determined based on the detection information to obtain at least one candidate target; Based on the motion characteristics of each candidate target in consecutive frames, it is determined whether there is a real foreign object in the at least one candidate target, and the detection result is obtained.

[0011] In the above scheme, determining whether there is a real foreign object in the at least one candidate target based on the motion characteristics of each candidate target in consecutive frames, and obtaining the detection result, includes: The sparse optical flow feature tracking method is used to determine whether each candidate target is successfully tracked in consecutive frames, and at least one first screening target is determined from the at least one candidate target; The motion trajectory of each tracked target in the preset target tracking table is smoothed using a Kalman filter, and at least one second-selected target matching the reference target is selected from the at least one first-selected target based on the Hungarian matching result of the intersection-over-union (IoU) ratio of each first-selected target and the reference target; the target tracking table is constructed based on the tracking information of the tracked targets in each frame of the video stream, the tracking information including tracking identifiers (IDs), and the reference target being the tracked target in the adjacent frame; Based on the difference between the absolute pixel displacement of each second screening target and the global motion vector of the first image, the true motion vector magnitude of each second screening target is determined. Based on the relationship between the true motion vector magnitude and a preset static threshold, it is determined whether the corresponding second screening target has actually moved, and the detection result is obtained.

[0012] In the above scheme, the first model is constructed based on a YOLOv11 network with a coordinate attention (CA) mechanism embedded in the C2f module, and the first model includes the WIoU v3 loss function.

[0013] In the above scheme, when the detection result includes the presence of a real foreign object, the method further includes: For each real foreign object, a spatiotemporal logic verification is performed; when the spatiotemporal logic verification passes, foreign object information is generated, sent, and an alarm is triggered; the foreign object information includes a foreign object image and its location; wherein, The spatiotemporal logic verification includes: Based on the positional changes of the real foreign object in consecutive frames, the spatiotemporal consistency of the real foreign object is verified. Based on the retention status of the real foreign object at the same position in consecutive frames, a retention time window criterion is applied to the real foreign object. Based on the oriented gradient histogram (HOG) features of the real foreign object, the similarity between the real foreign object and the corresponding position in the preset background model is determined, and a pixel-level histogram is performed on the real foreign object based on the similarity. Figure 2 Secondary verification.

[0014] According to one aspect of this application, a foreign object detection device for rail transit is provided, the device comprising: An image acquisition unit is used to acquire a video stream of the area to be tested and to acquire an environmental image of the area to be tested from the video stream to obtain a first image; the area to be tested includes the activity area of ​​the rail transit machine or the luggage storage area of ​​the carriage; the number of image acquisition units is multiple and they are deployed at different locations of the machine. The preprocessing unit is used to perform image enhancement and global motion compensation on the first image to obtain the second image; An image detection unit is used to determine the Region of Interest (ROI) from the second image based on the currently configured dynamic electronic fence to obtain a third image; the dynamic electronic fence is adaptively constructed based on dynamically updated semantic segmentation results, which are dynamically updated based on the changes in the environmental image of the region to be tested over time. The judgment unit is used to determine whether there is a foreign object in the area to be tested based on the third image and the first model, and to obtain the detection result.

[0015] The rail transit foreign object detection method and apparatus provided in this application improve the timeliness and accuracy of foreign object detection by acquiring video streams of the area to be tested and detecting foreign objects in real time based on on-site images extracted from the video streams. Furthermore, by performing image enhancement and motion compensation on the acquired images, the impact of complex lighting conditions on the detection results at the railway site can be reduced during the target detection process, and the impact of high-frequency camera vibration on the target detection results under scenarios such as train vibration or severe weather can be reduced, thereby improving the accuracy of target detection results. Furthermore, by dynamically performing semantic segmentation and constructing adaptive dynamic electronic fences, the accuracy of ROI calibration results can be improved, thereby enabling precise locking and judgment of tracking targets, and further improving the accuracy of target detection results.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0018] Figure 1 This is a schematic flowchart of a foreign object detection method for rail transit provided in an embodiment of this application; Figure 2 This is a flowchart of the method for image enhancement of the first image in the foreign object detection method for rail transit according to an embodiment of this application; Figure 3 This is a flowchart of the method for global motion compensation of the first image in the rail transit foreign object detection method according to an embodiment of this application; Figure 4 This is a flowchart of the method for updating the mask image and the dynamic electronic fence in the rail transit foreign object detection method of this application embodiment; Figure 5 This is a flowchart of step S104 in the foreign object detection method for rail transit according to an embodiment of this application; Figure 6 This is a flowchart of step S503 in the foreign object detection method for rail transit according to an embodiment of this application; Figure 7 This is a flowchart of the spatiotemporal logic verification method in the foreign object detection method for rail transit according to an embodiment of this application; Figure 8 This is a schematic diagram of a foreign object detection device for rail transit provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In rail transit scenarios, turnouts are the weakest link and critical part of railway lines. If hard foreign objects (such as ballast, bolts, tools, etc.) fall into the turnout switch rail area, they are very likely to be loose and produce a red light band during the switch rail switching process, which seriously affects the safety and efficiency of railway transportation. Large track maintenance machinery, also known as large machines, may have their auxiliary parts loosened and fall off due to their own vibration when they work continuously on the railway line. Screening machines also pose a risk that stones that have not been thoroughly cleaned from the car body may fall onto the turnout, seriously threatening the safety of rail transit.

[0021] In existing technologies, the detection of foreign objects falling into the switch rail area mainly relies on the visual observation of the train crew and the red light band generated by the switch. However, large locomotives typically operate continuously for 24 hours, with significant variations in lighting conditions (such as day-night cycles and changes inside and outside tunnels). Existing detection methods struggle to adapt to these complex and changing environments, resulting in low accuracy. Furthermore, the limited attention span of train crew members, coupled with the small visual range of the switch rail movement area, makes it difficult for existing detection methods to achieve continuous and effective monitoring of the switch area. Therefore, there is an urgent need for a technical solution that can automatically, in real-time, and accurately detect foreign object intrusion in the switch area.

[0022] Meanwhile, to ensure a smooth train service experience, it is necessary to promptly clean and inspect the carriages where passengers are active. One of the most common checks is to ensure there is no lost luggage in the luggage storage areas. However, current technology often relies on manual inspections to check for leftover items, which also suffers from the problems of being labor-intensive and prone to missed checks.

[0023] Based on this, in various embodiments of this application, by acquiring video streams of the area to be tested and detecting foreign objects in real time based on on-site images extracted from the video streams, the timeliness and accuracy of foreign object detection are improved. Furthermore, by performing image enhancement and motion compensation on the acquired images, the influence of complex lighting conditions at the railway site on the detection results can be reduced during the target detection process, and the influence of high-frequency camera vibration on the target detection results under scenarios such as train vibration or severe weather can be reduced, thereby improving the accuracy of target detection results. Furthermore, by dynamically performing semantic segmentation and constructing adaptive dynamic electronic fences, the accuracy of ROI calibration results can be improved, thereby enabling precise locking and judgment of tracking targets, and further improving the accuracy of target detection results.

[0024] This application provides a method for detecting foreign objects in rail transit, such as... Figure 1 As shown, the method may include: S101: Acquire a video stream of the area to be tested, and collect the current environmental image of the area to be tested from the video stream to obtain a first image; the area to be tested includes the activity area of ​​the rail transit machine or the luggage storage area of ​​the carriage; S102: Perform image enhancement and global motion compensation on the first image to obtain the second image; S103: Based on the currently configured dynamic electronic fence, determine the ROI from the second image to obtain the third image; the dynamic electronic fence is adaptively constructed based on dynamically updated semantic segmentation results, and the semantic segmentation results are dynamically updated based on the changes in the environmental image of the area to be tested over time; S104: Based on the third image and the first model, determine whether there are foreign objects in the area to be tested, and obtain the detection result.

[0025] In practical applications, when the monitoring area is the external environment, i.e. the area where the train operates, the foreign objects detected can include those in the railway tracks that affect train operation, such as stones and bolts. The detection frequency can be configured according to the train's speed and the camera's field of view to ensure that the detection results can completely cover the train's operating area. When the monitoring area is the internal environment, i.e. the luggage storage area of ​​the carriage, the foreign objects detected can include items left behind by passengers, including suitcases and handbags. The detection frequency can be configured to be detected upon arrival at the terminal station, or it can be flexibly configured according to actual application needs.

[0026] In practical applications, the term "large machine" can also be referred to as "large road maintenance machinery." For the purpose of clearly illustrating the embodiments of this application, it will be uniformly referred to as "large machine" below.

[0027] In practical applications, when the area to be monitored is the activity area of ​​the large machine, i.e., the railway track maintenance area such as turnouts, multiple sets of image acquisition units can be deployed at different positions at the head and tail of the large machine to collect video streams of the external operating environment of the train in real time, and to obtain and analyze the current on-site images from the video streams. Here, as the large machine runs on the track, it collects images of the railway track from all directions, which can achieve complete and reliable foreign object detection across the entire track section, thereby effectively reducing labor and time costs while improving the accuracy, reliability and comprehensiveness of the detection results. When the area to be monitored is the luggage storage area of ​​the carriage, multiple sets of image acquisition units can be deployed inside the carriage to collect video streams of different luggage storage areas inside the carriage in real time, in order to detect luggage left in the luggage storage area.

[0028] In practical applications, a video stream can also be called a raw video stream; the parameters of a video stream can be configured to a resolution of no less than 1920×1080 and a frame rate of 25fps.

[0029] In one embodiment, such as Figure 2 As shown, image enhancement of the first image may include: S201: Convert the first image from the RGB color space to the HSV color space to obtain an HSV image; S202: Extract the luminance component from the HSV image and enhance the extracted luminance component based on the MSR algorithm.

[0030] In practical applications, after acquiring the video stream in real time, each frame of the video stream can be enhanced by converting each frame from the RGB color space to the HSV color space and extracting the luminance component for enhancement using the MSR algorithm; the luminance component can also be called the v component.

[0031] In practical applications, there are many interfering factors at railway sites, such as oil stains reflecting off switch slides and uneven lighting at night. These factors greatly affect the accuracy of target detection results. Here, by introducing the MSR algorithm, the image can be filtered using Gaussian kernel functions of different scales to calculate the reflection component, thereby removing the influence of the illumination component on image details. Finally, it is converted back to RGB space, which significantly improves the texture clarity of stones in shadow areas and improves the accuracy of target detection results under complex environments or lighting conditions.

[0032] In one embodiment, such as Figure 3As shown, performing global motion compensation on the first image may include: S301: Determine the background region of the first image based on the currently configured mask image; S302: Extract FAST feature points from the background region; S303: Based on the displacement of FAST feature points in the first image and the previous frame image, determine the matching point pair in the first image; S304: Based on the matching point pairs, construct the global homography matrix of the first image; S305: Perform perspective transformation on the first image based on the global homography matrix.

[0033] In practical applications, the global motion compensation process can be understood as the global motion vector calculation process. Here, camera shake can be eliminated by extracting FAST feature points in the background area of ​​each frame image. The background area can also be called the non-skateboard area, such as ballast, sleepers, etc.

[0034] Specifically, the previous frame is defined as I_prev, and the current frame as I_curr. The Lucas-Kanade (LK) optical flow method can be used to track background feature points, obtaining a set of matching point pairs. Then, the Random Sample Consensus (RANSAC) algorithm can be used to remove mismatched points, and the global homography matrix H from I_prev to I_curr is calculated. Matrix H contains the camera's translation, rotation, and scaling parameters in the plane. I_curr is then subjected to a perspective transformation (warping) using matrix H to align its coordinate system with I_prev. The difference between the aligned image and the previous frame no longer contains camera shake components; any residual pixel displacement can be considered as the real motion of objects in the scene, thus eliminating camera shake. Global motion compensation lays the geometric foundation for subsequent judgment of stationary foreign objects.

[0035] In practical applications, at railway sites, the passing of trains or strong winds often cause high-frequency vibrations in surveillance cameras. This application's embodiment introduces a global motion compensation mechanism based on feature point matching, which can effectively eliminate the impact of camera shake on target detection results and improve detection accuracy.

[0036] In practical applications, traditional electronic fences often rely on manual calibration. Since railway operation sites are often accompanied by vibrations, the camera is prone to angular displacement. Once the camera's viewing angle shifts slightly, the manually calibrated fixed rectangle will no longer fit the target to be monitored, leading to missed detections. The embodiments of this application use a dynamic electronic fence that is adaptively constructed based on dynamically updated semantic segmentation results, which can more accurately and reliably lock the ROI and achieve accurate detection and tracking of the target.

[0037] In practical applications, dynamic electronic fences and mask images can be dynamically updated based on the offset of feature points. In order to achieve a balance between detection accuracy and computational load, a drift detection mechanism can be introduced to analyze whether semantic segmentation and electronic fence reconstruction should be performed at a certain frequency.

[0038] Based on this, in one embodiment, the method further includes: Sample frames are obtained by sampling from the video stream according to a preset sampling frequency; Calculate the average displacement of feature points in the background region of the sample frame relative to the initial frame; Determine whether the average displacement exceeds a preset drift threshold, and if it does, update the mask image and the dynamic electronic fence.

[0039] In practical applications, the mask image can also be called a mask or a mask. This application does not limit the term, as long as it can achieve its function.

[0040] In practical applications, sampling can be performed every 100 frames. The preset drift threshold can be configured according to the actual application scenario, for example, it can be configured to 5 pixels. Here, by introducing a drift detection mechanism, semantic segmentation and ROI reconstruction are only triggered when the average displacement exceeds the set drift threshold. This ensures the accuracy of the fence and minimizes the computational load, achieving a balance between monitoring accuracy and computational load.

[0041] In practical applications, the initial frame can be the first image captured at the corresponding location, or it can be a pre-configured image of the corresponding location in a state free of foreign objects.

[0042] In practical applications, when the area to be measured is the activity area of ​​a large machine, i.e., an external rail transit scenario, the corresponding initial frames can be pre-configured for different positions based on the field of view of the image acquisition unit at different positions of the large machine, and the correspondence between the position and the initial frame can be established. When it is necessary to compare the sample frame and the initial frame, the corresponding initial frame is retrieved based on the current position of the large machine for comparison to determine the average displacement.

[0043] In practical applications, when the area to be tested is the luggage storage area inside the carriage, i.e. the inside of the train, the initial frame can be generated based on the image collected when the area is empty and the target area is within the field of view of the camera.

[0044] In one embodiment, such as Figure 4 As shown, updating the mask image and the dynamic electronic fence may include: S401: Use the second model to perform semantic segmentation on the sample frame to obtain the semantic segmentation result; the second model is constructed based on a YOLOv11 network with a lightweight semantic segmentation branch connected in parallel to the backbone output layer, and the semantic segmentation branch is generated based on the BiSeNet structure; S402: Based on the semantic segmentation result, classify each pixel in the sample frame to generate an updated mask image; the classification result includes skateboard or background; S403: Use OpenCV's findContours operator to extract the maximum connected component contour of the updated mask image, and reconstruct the electronic fence based on the extraction result to generate the updated dynamic electronic fence; wherein, when extracting the maximum connected component contour of the mask image, the RDP algorithm is used to approximate the contour as a polygon.

[0045] In practical applications, updating the mask image and dynamic electronic fence can also be called dynamically configuring the mask image and dynamic electronic fence; dynamic electronic fence can also be called Auto-ROI.

[0046] In practical applications, when using the second model to perform semantic segmentation on sample frames, a lightweight semantic segmentation branch (based on the BiSeNet structure) can be connected in parallel to the output layer of the YOLOv11 detection network backbone. This branch is specifically used for pixel-level classification, classifying each pixel in the image as either a "skateboard" or "background". The YOLOv11 in the second model is trained on a specific dataset and can adapt to skateboard surfaces with different levels of wear and oil stains. After the second model completes semantic segmentation, the output is a binary mask image, where "1" can represent a skateboard and "0" can represent the background.

[0047] In practical applications, in order to eliminate jagged edges and noise at the segmentation edges, morphological operations can be performed on the generated mask image. Specifically, a 5×5 kernel-size closing operation is first performed to fill the holes inside the slide plate, and then an opening operation is performed to smooth the edges.

[0048] In practical applications, when extracting the maximum connected component contour of a mask image, the RDP algorithm can be used to approximate the contour with polygons to solve the problem of excessive original contour points and large computational load. Specifically, a distance threshold can be preset, for example, 0.01×arcLength, to simplify complex pixel edges into polygons composed of several key points. Then, this polygon is expanded outward by N pixels to form the final dynamic ROI. Here, N can be configured according to the detection accuracy, for example, N can be configured as 20. This dynamic ROI only includes the skateboard and its very small surrounding area, excluding interference from the ballast area. In practical applications, RDP stands for Ramer-Douglas-Peucker.

[0049] In practical applications, after obtaining the ROI region, it can be cropped and input into the object detection network for detection, i.e., S104 is executed; the first model can also be called the object detection model.

[0050] In practical applications, an improved version of YOLOv11 can be introduced to reduce the risk of misjudgment.

[0051] Based on this, in one embodiment, the first model can be constructed based on a YOLOv11 network with a CA mechanism embedded in the C2f module, and the first model includes the WIoU v3 loss function.

[0052] Specifically, when dealing with easily confused objects on a skateboard, the native YOLO is easily misled by non-foreign objects with similar shapes. For example, it can easily confuse small stones with similarly shaped oil stains. In this embodiment, a CA mechanism module is embedded after the C2f module of YOLOv11. Compared with the traditional SE (Squeeze-and-Excitation) module, the CA mechanism module not only focuses on channel information but also on position information. Specifically, it performs global average pooling on the input feature map along the x and y directions to generate a pair of direction-aware representation maps. Therefore, by introducing the CA mechanism module, position information can be embedded into the channel attention. Then, an attention weight map is generated through convolution transformation and multiplied with the original feature map. This enables the model to more accurately locate the spatial coordinates of foreign objects, suppress the weight of messy textures in the background (such as rust, oil stains, etc.), significantly improve the recall rate of small target foreign objects, and effectively distinguish between foreign objects and non-foreign objects with similar shapes.

[0053] In practical applications, this embodiment of the application optimizes the loss function by replacing the original CIoU loss function with the WIoU v3 loss function. Since WIoU introduces a dynamic non-monotonic focusing mechanism, it can dynamically allocate gradient gain according to outlier, making the model pay more attention to samples of average quality, rather than over-focusing on simple samples or extremely difficult samples (such as mislabeled samples). Therefore, by replacing the loss function with the WIoU v3 loss function, the regression accuracy of bounding boxes can be improved, and the detection accuracy of irregularly shaped samples can be improved, especially for the detection of irregularly shaped ballast stones.

[0054] In practical applications, after inputting the ROI image into the first model, multiple detection box information Bi can be output, indicating that multiple candidate targets have been detected; this can be expressed by the formula: Bi=(x,y,w,h,conf); Where (x, y) represents the center coordinates, w represents the width, h represents the height, and conf represents the confidence level. Here, the confidence level can be configured based on experience or actual application scenarios. The confidence threshold can be configured to 0.45, that is, the confidence level is greater than or equal to 0.45, in order to filter low-quality targets.

[0055] In practical applications, the detection results of a single frame often exhibit flickering or false alarms. Therefore, in order to improve the accuracy of the detection results, motion features in the time dimension can be combined to determine the final detection result.

[0056] Based on this, in one embodiment, such as Figure 5 As shown, the step of determining whether there is a foreign object in the area to be tested based on the third image and the first model, and obtaining the detection result, i.e., S104, may include: S501: Based on the first model, determine whether there is a foreign object in the third image and generate detection information; S502: When the detection information includes the presence of foreign objects, the tracking target in the third image is determined based on the detection information to obtain at least one candidate target; S503: Based on the motion characteristics of each candidate target in consecutive frames, determine whether there is a real foreign object in the at least one candidate target, and obtain the detection result.

[0057] In practical applications, when the area to be tested is the activity area of ​​a large machine, S501 to S503 can be executed to introduce the motion characteristics of the tracked target to determine whether the tracked target is a real foreign object and prevent misjudgment. When the area to be tested is the luggage storage area inside the carriage, only S501 to S502 can be executed and each candidate target can be treated as a real foreign object.

[0058] In one embodiment, such as Figure 6 As shown, the step of determining whether there is a real foreign object in the at least one candidate target based on the motion characteristics of each candidate target in consecutive frames, and obtaining the detection result, i.e., S503, may include: S601: Use the sparse optical flow feature tracking method to determine whether each candidate target is successfully tracked in consecutive frames, and determine at least one first screening target from the at least one candidate target; S602: The motion trajectory of each tracked target in the preset target tracking table is smoothed using a Kalman filter, and at least one second-selected target matching the reference target is selected from the at least one first-selected target based on the IoU Hungarian matching result between each first-selected target and the reference target; the target tracking table includes the ID of the tracked target in each frame of the video stream, and the reference target is the tracked target in the adjacent frame; S603: Based on the difference between the absolute pixel displacement of each second screening target and the global motion vector of the first image, determine the true motion vector magnitude of each second screening target, and based on the relationship between the true motion vector magnitude and the preset static threshold, determine whether the corresponding second screening target has actually moved, and obtain the detection result.

[0059] In practical applications, sparse optical flow feature tracking methods can be used to continuously track detected targets. Specifically, for each target box Bi detected by the first model, Shi-Tomasi corner points can be extracted as feature points within it. Then, using the pyramid LK optical flow method, the positions of these feature points in the next frame are calculated to achieve target tracking. To prevent optical flow drift, a forward-backward error check algorithm can be introduced. This algorithm calculates the feature point Pt_next tracked from frame t to frame t+l, and then tracks it back to frame t to obtain Pt_back, which is the reverse prediction position. Then, the Euclidean distance between the initial position and the reverse prediction position is calculated, which is the forward-backward (FB) error. If the FB error is greater than a set threshold (e.g., 1 pixel), the feature point is considered to have failed to track and is discarded. Points with small errors are retained as reliable matches to improve the robustness of target tracking.

[0060] In practical applications, by introducing target ID association and Kalman filtering, duplicate counting caused by flicker can be effectively reduced. Specifically, a target tracking list (Track List) can be constructed. For a newly detected target in the current frame, its IoU with the tracked target in the previous frame is calculated and Hungarian matching is performed. If the match is successful, the target ID is inherited; if the match fails and the target is located at the edge of the image, it is considered a newly entered target and a new ID is assigned to it. For each tracked target, a Kalman filter is used to smooth its trajectory. The state variables of the Kalman filter are set as (x, y, vx, vy), i.e., position and velocity. When YOLO misses a detection in a frame, the predicted value of the Kalman filter is used for blind navigation to keep the target ID from being lost and prevent duplicate counting caused by detection flicker.

[0061] In practical applications, by introducing the calculation of the real motion vector, it is possible to effectively distinguish whether the movement of the foreign object in the detection result is its own real movement or the image movement caused by camera shake. Specifically, combined with the global motion vector V_global determined in the global motion compensation process, the real motion vector V_real of the tracked target is calculated, and the calculation formula is as follows: V_real = V_optical - V_global; Where V_optical represents the absolute pixel displacement measured by optical flow; Then, the magnitude of the real motion vector is defined as |V_real|. It is determined whether |V_real| is greater than the static threshold. If it is greater than the static threshold, it is determined that the object is moving, such as a small stone rolling. Otherwise, it is determined that the object is stationary relative to the skateboard, such as the image displacement caused by camera vibration. Here, the introduction of the real motion vector can effectively distinguish between the real movement of the object and the image displacement caused by camera vibration, so as to effectively determine whether the object is moving or stationary.

[0062] In practical applications, to prevent false alarms, such as those about flying insects or fallen leaves, the spatiotemporal logic of the recognition results can be used to determine whether the object is a real foreign object.

[0063] Based on this, in one embodiment, when the detection result includes the presence of a real foreign object, the method may further include: Perform spatiotemporal logic verification on each real foreign object; When the spatiotemporal logic verification passes, foreign object information is generated, sent, and an alarm is triggered; the foreign object information includes a foreign object image and the location of the foreign object.

[0064] In one embodiment, such as Figure 7 As shown, the spatiotemporal logic verification may include: S701: Verify the spatiotemporal consistency of the real foreign object based on its positional changes in consecutive frames; S702: Based on the retention status of the real foreign object at the same position in consecutive frames, a retention time window criterion is applied to the real foreign object; S703: Based on the HOG features of the real foreign object, determine the similarity between the real foreign object and the corresponding position in the preset background model, and perform pixel-level histogram analysis on the real foreign object based on the similarity. Figure 2 Secondary verification.

[0065] In practical applications, when executing S701, to verify the spatiotemporal consistency of the real foreign object based on its positional changes across consecutive frames, a state machine can be established for each tracked target ID, containing three states: entering, moving, and stationary. The criteria for determining each state include: When the target first appears in the state, the lifecycle timer is initialized, i.e., T_ife=0; In the moving state, if |V_real|>0, the state remains moving. At this time, since the target may continue to move away from the currently monitored area, no alarm is triggered. In a stationary state, if |V_real|≈0 and the target is still within the dynamic electronic fence, it is determined that the system has entered a stationary timing state.

[0066] In practical applications, for each identified real foreign object, corresponding spatiotemporal features can be generated based on the positional changes of the real foreign object in consecutive frames, and based on these spatiotemporal features, it can be determined whether the trajectory of the real foreign object can pass the spatiotemporal consistency check.

[0067] In practical applications, for real foreign objects that pass the spatiotemporal consistency check, their persistence in consecutive frames can be further checked, i.e., S702 is executed. During the execution of S702, a time window threshold T_thresh can be preset, for example, it can be configured to 25 frames, i.e. 1 second. When the tracked target is in a stationary state and the duration T_stay is greater than T_thresh, the tracked target is marked as a dangerous foreign object. Here, by introducing the persistence time window criterion, it is possible to effectively filter out fast-moving insects that do not persist, as well as fallen leaves with irregular trajectories that are usually blown away by the wind after landing, and also to accurately filter out non-stone textures, thereby reducing the probability of false alarms and improving detection accuracy.

[0068] In practical applications, when performing pixel-level histograms... Figure 2During the second verification, i.e. when executing S703, a region map containing the area where the tracking target is located can be captured, the HOG features of the region map can be extracted, and the similarity can be compared with the corresponding position of the background model established during model initialization. If the similarity is higher than the preset similarity threshold, it means that the tracking target is a background stain and is removed; otherwise, it means that the difference is significant and the tracking target is determined to be a new foreign object. Here, the background model can be a model map of the monitored object in a state without foreign objects.

[0069] In practical applications, when the spatiotemporal logic verification passes, i.e. when the tracking target is determined to be a newly added foreign object, foreign object information can be generated, sent, and an alarm can be triggered. Specifically, a foreign object image can be generated and sent to the management terminal. On the monitoring screen of the management terminal, the foreign object can be marked with a red border or other highlight colors, and its ID and confidence level can be labeled. The coordinates containing the foreign object can also be mapped back to the original panoramic image, and on-site photos and 5-second video clips before and after the object can be captured and uploaded to the server.

[0070] In practical applications, when a foreign object is detected, its specific location can be determined based on its coordinates. Specifically, it can be determined whether the foreign object is located on the side of the switch rail or the side of the main rail, and specific foreign object information can be generated based on the location determination result. For example, in turnout W1, a foreign object is detected on the third slide block on the left side of the switch rail.

[0071] In summary, the foreign object detection method for rail transit provided in this application improves the timeliness and accuracy of foreign object detection by acquiring video streams of the area to be tested and detecting foreign objects in real time based on on-site images extracted from the video streams. Furthermore, by performing image enhancement and motion compensation on the acquired images, the influence of complex lighting conditions at the railway site on the detection results can be reduced during the target detection process, and the influence of high-frequency camera vibrations in scenarios such as train vibrations or severe weather on the target detection results can be reduced, thereby improving the accuracy of target detection results. Furthermore, by dynamically performing semantic segmentation and constructing adaptive dynamic electronic fences, the accuracy of ROI calibration results can be improved, thereby enabling precise locking and judgment of tracking targets, and further improving the accuracy of target detection results.

[0072] To implement the foreign object detection method for rail transit as described in this application, embodiments of this application also provide a foreign object detection device for rail transit, such as... Figure 8 As shown, the device may include...

[0073] Image acquisition unit 801 is used to acquire a video stream of the area to be tested and to acquire a current environmental image of the area to be tested from the video stream to obtain a first image; the area to be tested includes the activity area of ​​the rail transit machine or the luggage storage area of ​​the carriage. Preprocessing unit 802 is used to perform image enhancement and global motion compensation on the first image to obtain a second image; Image detection unit 803 is used to determine the ROI from the second image based on the currently configured dynamic electronic fence to obtain a third image; the dynamic electronic fence is adaptively constructed based on dynamically updated semantic segmentation results, and the semantic segmentation results are dynamically updated based on the changes in the environmental image of the area to be tested over time; The judgment unit 804 is used to determine whether there is a foreign object in the area to be tested based on the third image and the first model, and to obtain the detection result.

[0074] In practical applications, the image acquisition unit 801 may include a camera module, which may include multiple sets of cameras. When the area to be measured is the activity area of ​​a large locomotive, the multiple sets of cameras are respectively installed at the front of the head car and the rear car of the locomotive, and are located directly above the two rails. The camera lenses are aimed at the switch activity area below, and the acquisition range of the two sets of cameras located at the same locomotive head includes the entire activity area of ​​the switch located on the side of the locomotive head. The camera installation angle can be adjusted in advance to ensure full coverage of the activity area. When the area to be measured is the luggage storage area inside the car, the multiple sets of cameras can be installed at positions opposite to the luggage storage area. The field of view of the multiple sets of cameras covers all the luggage storage areas to achieve comprehensive and reliable monitoring of the area.

[0075] In practical applications, the image acquisition unit 801 may also include a supplementary lighting module; the supplementary lighting module can be installed in conjunction with the camera module to provide stable and sufficient illumination for the camera when the ambient light is insufficient, ensuring that the acquired image quality is clear and stable, and meeting the requirements of subsequent analysis.

[0076] In practical applications, the preprocessing unit 802, the image detection unit 803, and the judgment unit 804 can be located on the processor, specifically on the processor of the vehicle-mounted industrial computer.

[0077] In one embodiment, the preprocessing unit 802 may specifically be used for: Based on the currently configured mask image, determine the background region of the first image; Extract FAST feature points from the background region; Based on the displacement of FAST feature points in the first image and the previous frame image, the matching point pairs in the first image are determined. Based on the matching point pairs, construct the global homography matrix of the first image; The first image is subjected to perspective transformation based on the global homography matrix.

[0078] In one embodiment, the preprocessing unit 802 may specifically be used for: The first image is converted from the RGB color space to the HSV color space to obtain an HSV image; The luminance component is extracted from the HSV image, and the extracted luminance component is enhanced based on the MSR algorithm.

[0079] In one embodiment, the image detection unit 803 can also be used for: Sample frames are obtained by sampling from the video stream according to a preset sampling frequency; Calculate the average displacement of feature points in the background region of the sample frame relative to the initial frame; Determine whether the average displacement exceeds a preset drift threshold, and if it does, update the mask image and the dynamic electronic fence.

[0080] In one embodiment, the image detection unit 803 may specifically be used for: The sample frames are semantically segmented using the second model to obtain semantic segmentation results; the second model is a YOLOv11 network constructed based on a lightweight semantic segmentation branch connected in parallel to the backbone output layer, and the semantic segmentation branch is generated based on the BiSeNet structure. Based on the semantic segmentation results, each pixel in the sample frame is classified to generate an updated mask image; the classification results include skateboard or background. The maximum connected component contour of the updated mask image is extracted using the findContours operator of OpenCV, and the electronic fence is reconstructed based on the extraction results to generate the updated dynamic electronic fence. In the process of extracting the maximum connected component contour of the mask image, the RDP algorithm is used to approximate the contour as a polygon.

[0081] In one embodiment, the determination unit 804 can be specifically used for: Based on the first model, determine whether there is a foreign object in the third image and generate detection information; When the detection information includes the presence of foreign objects, the tracking target in the third image is determined based on the detection information to obtain at least one candidate target; Based on the motion characteristics of each candidate target in consecutive frames, it is determined whether there is a real foreign object in the at least one candidate target, and the detection result is obtained.

[0082] In one embodiment, the determination unit 804 can be specifically used for: The sparse optical flow feature tracking method is used to determine whether each candidate target is successfully tracked in consecutive frames, and at least one first screening target is determined from the at least one candidate target; The motion trajectory of each tracked target in the preset target tracking table is smoothed using a Kalman filter, and at least one second-selected target matching the reference target is selected from the at least one first-selected target based on the Hungarian matching result of the IoU between each first-selected target and the reference target; the target tracking table is constructed based on the tracking information of the tracked targets in each frame of the video stream, the tracking information including the tracking ID, and the reference target being the tracked target in the adjacent frame; Based on the difference between the absolute pixel displacement of each second screening target and the global motion vector of the first image, the true motion vector magnitude of each second screening target is determined. Based on the relationship between the true motion vector magnitude and a preset static threshold, it is determined whether the corresponding second screening target has actually moved, and the detection result is obtained.

[0083] In one embodiment, the first model is built on a YOLOv11 network with a CA mechanism embedded in the C2f module, and the first model includes the WIoU v3 loss function.

[0084] In one embodiment, the device further includes: The verification unit is used to perform spatiotemporal logic verification on each real foreign object when the detection result includes the presence of a real foreign object; when the spatiotemporal logic verification passes, foreign object information is generated, sent, and an alarm is triggered; the foreign object information includes a foreign object image and the location of the foreign object; wherein... The verification unit is specifically used for: Based on the positional changes of the real foreign object in consecutive frames, the spatiotemporal consistency of the real foreign object is verified. Based on the retention status of the real foreign object at the same position in consecutive frames, a retention time window criterion is applied to the real foreign object. Based on the HOG features of the real foreign object, the similarity between the real foreign object and the corresponding position in the preset background model is determined, and a pixel-level histogram is performed on the real foreign object based on the similarity. Figure 2 Secondary verification.

[0085] It should be noted that the foreign object detection device for rail transit provided in the above embodiments is only illustrated by the division of the above-described program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the foreign object detection device for rail transit provided in the above embodiments and the foreign object detection method embodiments for rail transit belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0086] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0087] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0088] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A method for detecting foreign objects in rail transit, characterized in that, The method includes: A video stream of the area to be tested is acquired, and an environmental image of the current area to be tested is collected from the video stream to obtain a first image; the area to be tested includes the activity area of ​​the rail transit machine or the luggage storage area of ​​the carriage; The first image is enhanced and global motion compensation is performed to obtain the second image; Based on the currently configured dynamic electronic fence, the region of interest is determined from the second image to obtain the third image; the dynamic electronic fence is adaptively constructed based on dynamically updated semantic segmentation results, which are dynamically updated based on the changes in the environmental image of the region under test over time. Based on the third image and the first model, it is determined whether there are foreign objects in the area to be tested, and the detection result is obtained.

2. The method according to claim 1, characterized in that, Perform global motion compensation on the first image, including: Based on the currently configured mask image, determine the background region of the first image; Extract FAST feature points from the background region; Based on the displacement of FAST feature points in the first image and the previous frame image, the matching point pairs in the first image are determined. Based on the matching point pairs, construct the global homography matrix of the first image; The first image is subjected to perspective transformation based on the global homography matrix.

3. The method according to claim 2, characterized in that, Image enhancement of the first image includes: The first image is converted from the RGB color space to the HSV color space to obtain an HSV image; The luminance component is extracted from the HSV image, and the extracted luminance component is enhanced based on a multi-scale retinal enhancement algorithm.

4. The method according to claim 2, characterized in that, The method further includes: Sample frames are obtained by sampling from the video stream according to a preset sampling frequency; Calculate the average displacement of feature points in the background region of the sample frame relative to the initial frame; Determine whether the average displacement exceeds a preset drift threshold, and if it does, update the mask image and the dynamic electronic fence.

5. The method according to claim 4, characterized in that, Updating the mask image and the dynamic electronic fence includes: The sample frames are semantically segmented using the second model to obtain semantic segmentation results; the second model is a YOLOv11 network constructed based on a lightweight semantic segmentation branch connected in parallel to the backbone output layer, and the semantic segmentation branch is generated based on the BiSeNet structure. Based on the semantic segmentation results, each pixel in the sample frame is classified to generate an updated mask image; the classification results include skateboard or background. The maximum connected component contour of the updated mask image is extracted using the findContours operator of OpenCV, and the electronic fence is reconstructed based on the extraction results to generate the updated dynamic electronic fence. In the process of extracting the maximum connected component contour of the mask image, the RDP algorithm is used to approximate the contour as a polygon.

6. The method according to claim 2, characterized in that, The step of determining whether there is a foreign object in the area to be tested based on the third image and the first model, and obtaining the detection result, includes: Based on the first model, determine whether there is a foreign object in the third image and generate detection information; When the detection information includes the presence of foreign objects, the tracking target in the third image is determined based on the detection information to obtain at least one candidate target; Based on the motion characteristics of each candidate target in consecutive frames, it is determined whether there is a real foreign object in the at least one candidate target, and the detection result is obtained.

7. The method according to claim 6, characterized in that, The step of determining whether there is a real foreign object in at least one candidate target based on the motion features of each candidate target in consecutive frames, and obtaining the detection result, includes: The sparse optical flow feature tracking method is used to determine whether each candidate target is successfully tracked in consecutive frames, and at least one first screening target is determined from the at least one candidate target; The motion trajectory of each tracked target in the preset target tracking table is smoothed using a Kalman filter, and at least one second-selected target matching the reference target is selected from the at least one first-selected target based on the Hungarian matching result of the intersection-union ratio of each first-selected target and the reference target; the target tracking table is constructed based on the tracking information of the tracked targets in each frame of the video stream, the tracking information including tracking identifiers, and the reference target being the tracked target in the adjacent frame; Based on the difference between the absolute pixel displacement of each second screening target and the global motion vector of the first image, the true motion vector magnitude of each second screening target is determined. Based on the relationship between the true motion vector magnitude and a preset static threshold, it is determined whether the corresponding second screening target has actually moved, and the detection result is obtained.

8. The method according to claim 6, characterized in that, The first model is built on a YOLOv11 network with a coordinate attention mechanism embedded in the C2f module, and the first model includes the WIoU v3 loss function.

9. The method according to claim 6, characterized in that, When the detection result includes the presence of a real foreign object, the method further includes: For each real foreign object, a spatiotemporal logic verification is performed; when the spatiotemporal logic verification passes, foreign object information is generated, sent, and an alarm is triggered; the foreign object information includes a foreign object image and its location; wherein, The spatiotemporal logic verification includes: Based on the positional changes of the real foreign object in consecutive frames, the spatiotemporal consistency of the real foreign object is verified. Based on the retention status of the real foreign object at the same position in consecutive frames, a retention time window criterion is applied to the real foreign object. Based on the directional gradient histogram features of the real foreign object, the similarity between the real foreign object and the corresponding position in the preset background model is determined, and a second pixel-level histogram verification is performed on the real foreign object based on the similarity.

10. A foreign object detection device for rail transit, characterized in that, The device includes: An image acquisition unit is used to acquire a video stream of the area to be tested and to acquire a current environmental image of the area to be tested from the video stream to obtain a first image; the area to be tested includes the activity area of ​​a rail transit machine or the luggage storage area of ​​a carriage. The preprocessing unit is used to perform image enhancement and global motion compensation on the first image to obtain the second image; An image detection unit is used to determine the region of interest from the second image based on the currently configured dynamic electronic fence to obtain a third image; the dynamic electronic fence is adaptively constructed based on dynamically updated semantic segmentation results, which are dynamically updated based on the changes in the environmental image of the region under test over time. The judgment unit is used to determine whether there is a foreign object in the area to be tested based on the third image and the first model, and to obtain the detection result.

Images