Railway foreign body intrusion identification method, system, equipment and medium

Abstract

The present invention discloses a method, system, device, and medium for identifying foreign objects intruding on railway tracks. The method comprises the following steps: acquiring multimodal data, the multimodal data including video and / or original railway images; preprocessing the video and / or original railway images to obtain a preprocessed image; performing image enhancement processing on the preprocessed image to obtain a target-enhanced image; performing image segmentation prediction on the target-enhanced image to obtain a segmentation prediction result; and post-processing the segmentation prediction result to obtain a foreign object identification result. Post-processing involves retaining the identifiers corresponding to abnormal targets within the track area. This method can quickly and effectively identify foreign objects intruding on railway tracks in real time, effectively detecting foreign objects intruding on the track area, and reducing the occurrence of algorithmic false detections and missed detections.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, equipment and medium for identifying foreign body intrusion on railways. Background Art

[0002] With the rapid development of railway transportation, the threat posed by foreign objects to railway safety has become increasingly prominent. Ensuring train operation safety has become a critical task in the railway sector, prompting the continuous development and research of foreign object identification technology. Currently, AI technologies have been successfully applied in various fields, and several deep learning-based railway foreign object detection algorithms have emerged. However, due to the unique operating conditions of the railway industry and the complex actual field environment, general AI technologies rely on large amounts of supervisory data, resulting in high hardware requirements and training costs. Furthermore, the cost of actual data annotation is high, and the data processing efficiency cannot meet the needs of railway business.

[0003] In practical applications, railway foreign object intrusion monitoring generates a large amount of multi-source data, including videos, images, and text. This data is subject to various interference factors, such as background, weather, and lighting. Furthermore, there are relatively few real-world cases of railway foreign object intrusion, and a small amount of labeled data can easily lead to model overfitting. In summary, the unique characteristics of the railway industry and its diverse scenarios necessitate customized algorithms for railway scenarios, placing high demands on the real-time and robustness of the algorithms. Summary of the Invention

[0004] The purpose of the present invention is to provide a railway foreign object intrusion identification method, system, equipment and medium to solve the problems of high manual labeling cost, small amount of label data, high model calculation complexity and poor generalization.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is to provide a method for identifying foreign objects intruding on railways, comprising the following steps: acquiring multimodal data, wherein the multimodal data includes video and / or original railway images; preprocessing the video and / or original railway images to obtain preprocessed images; performing image enhancement processing on the preprocessed images to obtain target enhanced images; performing image segmentation prediction on the target enhanced images using a segmentation prediction network to obtain segmentation prediction results; post-processing the segmentation prediction results to obtain foreign object identification results; wherein the post-processing refers to retaining the identification corresponding to the abnormal targets in the track area.

[0006] In some embodiments, the multimodal data also includes multi-source prompt information, and the method also includes the steps of: constructing a segmentation prediction network, the segmentation prediction network including a target segmentation model and a prompt detection model; wherein, when the multi-source prompt information includes text prompt information, the prompt detection model includes an open domain target detection model, and the open domain target detection model performs preliminary recognition on the target enhanced image based on the text prompt information to obtain an initial detection box and a foreign object category; the target segmentation model performs target segmentation on the target enhanced image based on the multi-source prompt information, the initial detection box and the foreign object category to obtain a segmentation prediction result; using a pre-training dataset, the segmentation prediction network is pre-trained; wherein, the target segmentation model is first used as a student model, and the reference backbone network Hiera is used as a teacher model; then, the pre-training dataset is input into the teacher model, and a knowledge distillation strategy is adopted for the student model to extract knowledge from the teacher model to the student model.

[0007] In some embodiments, the total loss function of the segmentation prediction network is:

[0008] ;

[0009] in, and are the weights corresponding to the mask loss and the detection box loss respectively; The mask loss between the true mask and the predicted mask output by the target segmentation model; The detection box loss between the real detection box and the predicted detection box output by the target segmentation model;

[0010] The mask loss and the detection box loss They are:

[0011] ;

[0012] ;

[0013] in, represents the mask cross entropy loss; represents the mask Dice loss; and They are respectively the mask cross entropy loss and masked Dice loss The corresponding weight; Indicates the detection box Smooth L1 loss; represents the category cross entropy loss; and They are detection box Smooth L1 loss and categorical cross entropy loss The corresponding weight.

[0014] In some embodiments, the image enhancement processing is performed on the preprocessed image to obtain a target enhanced image, including the steps of: performing image illumination clustering on the preprocessed image to obtain a three-channel mean; the image illumination clustering refers to performing mean calculations on the image according to the RGB color space; judging whether the three-channel mean exceeds a first threshold, the first threshold being a critical value for distinguishing the illumination conditions corresponding to the preprocessed image; if the three-channel mean does not exceed the first threshold, the preprocessed image is a dark light image, and the image enhancement processing is performed on the preprocessed image to obtain a target enhanced image; if the three-channel mean exceeds the first threshold, the preprocessed image is other light images, and there is no need to perform the image enhancement processing on the preprocessed image, and the preprocessed image is directly made the target enhanced image.

[0015] In some embodiments, the method also includes the steps of: constructing an adaptive segmentation network, constructing an adaptive segmentation network, the adaptive segmentation network includes a first branch network, a second branch network and a third branch network, the first branch network, the second branch network and the third branch network are respectively used to perform strong image enhancement, weak image enhancement and image pre-enhancement on the pre-processed image, and then perform target segmentation to obtain the first target segmentation result, the second target segmentation result and the third target segmentation result respectively; then the first branch network is also used to post-process the first target segmentation result and the third target segmentation result to obtain the foreign body recognition result.

[0016] In some embodiments, the method further includes the steps of: using a training data set to train the adaptive segmentation network, the training data set including labeled images and unlabeled images; wherein, in the first branch network, the labeled image is weakly enhanced, and the unlabeled image is strongly enhanced to obtain a first weakly enhanced image and a strongly enhanced image, respectively; the first weakly enhanced image and the strongly enhanced image are input into a first segmentation model for target segmentation to obtain a first segmentation result, the first segmentation result including a true pseudo label corresponding to the first weakly enhanced image and a first pseudo label corresponding to the strongly enhanced image; a self-supervised loss is used between the true pseudo label and the true label, and the first pseudo label is used to compare with the third The third segmentation result output by the branch network is post-processed to obtain a target segmentation result corresponding to the unlabeled image; in the second branch network, the unlabeled image is weakly enhanced to obtain a second weakly enhanced image; the second weakly enhanced image is input into the second segmentation model for target segmentation to obtain a second segmentation result; the second segmentation result is adaptively screened to obtain a second pseudo label; in the third branch network, the unlabeled image is pre-enhanced to obtain a pre-enhanced image; the pre-enhanced image is input into the target segmentation model for target segmentation to obtain a third segmentation result; wherein, the first pseudo label adopts corresponding consistency loss functions with the second pseudo label and the third segmentation result respectively.

[0017] In some embodiments, the total loss function of the adaptive segmentation network is:

[0018] ;

[0019] in, is the supervision loss of the first segmentation model, is the consistency loss between the second segmentation model and the first segmentation model, is the consistency loss between the target segmentation model and the first segmentation model, 、 and are the corresponding loss weights respectively.

[0020] The present invention also provides a railway foreign object intrusion identification system, which includes: a data acquisition unit for acquiring multimodal data, wherein the multimodal data includes video and / or original railway images; a preprocessing unit for preprocessing the video and / or original railway images to obtain a preprocessed image; an image enhancement unit for performing image enhancement processing on the preprocessed image to obtain a target enhanced image; a segmentation prediction unit for performing image segmentation prediction on the target enhanced image to obtain a segmentation prediction result; a post-processing unit for post-processing the segmentation prediction result to obtain a foreign object identification result; wherein the post-processing refers to retaining the identification corresponding to the abnormal target in the track area.

[0021] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0022] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0023] The beneficial effects of the present invention are as follows: the present invention discloses a method, system, device, and medium for identifying foreign objects intruding on railway tracks, the method comprising the following steps: acquiring multimodal data, the multimodal data comprising video and / or original railway images; preprocessing the video and / or original railway images to obtain preprocessed images; performing image enhancement processing on the preprocessed images to obtain target enhanced images; performing image segmentation prediction on the target enhanced images to obtain segmentation prediction results; post-processing the segmentation prediction results to obtain foreign object identification results; wherein post-processing refers to retaining the identifiers corresponding to abnormal targets within the track area. This method can quickly and effectively identify foreign objects in massive amounts of unlabeled data, and can effectively detect foreign objects intruding on railway tracks in real time, reducing the occurrence of algorithmic false detections and missed detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is an overall flow chart of an embodiment of a method for identifying foreign body intrusion on railways according to the present invention;

[0025] Figure 2 This is a block diagram of the composition of the segmentation prediction network in an embodiment of a railway foreign object intrusion identification method of the present invention;

[0026] Figure 3 This is a specific flow chart of an embodiment of a method for identifying foreign objects intruding into a railway boundary according to the present invention;

[0027] Figure 4This is a specific flow chart of step S3 in an embodiment of a method for identifying foreign body intrusion on railways according to the present invention;

[0028] Figure 5 1 is a schematic diagram of the principle of an adaptive segmentation network in another embodiment of a railway foreign object intrusion identification method of the present invention;

[0029] Figure 6 This is a block diagram of the composition of an image encoder used in an embodiment of a method for identifying foreign objects intruding into a railway boundary of the present invention;

[0030] Figure 7 It is a pre-processed image obtained in an embodiment of a method for identifying foreign objects intruding into a railway according to the present invention;

[0031] Figure 8 It is a target enhanced image obtained in an embodiment of a method for identifying foreign matter intrusion on railways according to the present invention;

[0032] Figure 9 This is a segmentation prediction result diagram obtained in an embodiment of a railway foreign object intrusion identification method of the present invention;

[0033] Figure 10 This is a foreign object recognition result diagram obtained in an embodiment of a railway foreign object intrusion recognition method of the present invention;

[0034] Figure 11 This is a structural block diagram of an embodiment of a railway foreign object intrusion identification system of the present invention;

[0035] Figure 12 This is a schematic diagram of the architecture of an electronic device according to an embodiment of the present invention;

[0036] Figure 13 It is a schematic block diagram of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0037] To facilitate understanding of the present invention, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present invention.

[0038] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification are intended solely for the purpose of describing specific embodiments and are not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0039] like Figure 1-6 As shown, a method for identifying foreign body intrusion on railways according to the present invention is shown, comprising the following steps:

[0040] Step S1: Acquire multimodal data, where the multimodal data includes videos and / or original railway images.

[0041] It should be noted that this application supports multimodal data input from cameras used to monitor tracks and nearby areas, or from other backend interfaces. Multimodal data includes other data types such as video and images. In some embodiments, multimodal data also includes multi-source prompt information. Accordingly, this application establishes a segmentation prediction network 4 for foreign object recognition in images. The segmentation prediction network 4 and multi-source prompt information will be further described below. In other embodiments, this application establishes an adaptive segmentation network for foreign object recognition in images.

[0042] As an example, before step S1, it is necessary to construct a segmentation prediction network 4 and pre-train the segmentation prediction network 4 using a pre-training dataset.

[0043] Combine Figure 2 and Figure 3 The segmentation prediction network 4 supports input of multimodal data, including videos, images, and multi-source prompt information such as rectangles, points, masks, and text. The segmentation prediction network 4 includes a target segmentation model 32 and a prompt detection model 41. The prompt detection model 41 can quickly and accurately detect foreign objects intruding into image and video data based on multi-source prompt information, and input the detection results into the target segmentation model 32. The target segmentation model 32 performs the target segmentation task and marks the target foreign objects in the image.

[0044] like Figure 3 As shown, the prompt detection model 41 includes an open domain target detection model. The open domain target detection model adopts the open vocabulary target detector Mamba-YOLO-World, which can identify the initial detection box and foreign object category of the foreign object in the image based on the input image and any text prompt information. The initial detection box and foreign object category will be input into the target segmentation model 32 as new supervision information, further improving the recognition accuracy of the segmentation prediction network 4.

[0045] In some other embodiments, the multi-source prompt information input into the segmentation prediction network 4 also includes other types of prompt information such as rectangular boxes, points, masks, etc., and the prompt detection model 41 also includes corresponding types of target detection models to realize different types of target prompt detection according to different types of target detection models.

[0046] Combine Figure 5 and Figure 6The object segmentation model 32 includes an image encoder 34 and a mask decoder 35. The image encoder 34 is used to perform feature encoding on the received input image Input image and output the image encoding feature Output; the mask decoder 35 is used to receive the image encoding feature Output, as well as the initial detection box and foreign object category output by the open vocabulary object detector Mamba-YOLO-World, and perform image decoding based on the image encoding feature Output, the initial detection box and the foreign object category.

[0047] like Figure 6 As shown in the figure, leveraging the high-efficiency and low-parameter network characteristics of RWKV, an image encoder 34 is designed. Image encoder 34 includes a convolutional neural network module (Stem), a fused convolution module (Fused-MBConv), and a VRWKV (Vision-RWKV) model. The convolutional neural network module (Stem) first performs dimensionality and size reduction on the input image. It then uses a two-layer fused convolution module (Fused-MBConv) for further fused convolution. Finally, the VRWKV model performs image encoding to obtain the corresponding image encoding features (Output).

[0048] After receiving the input image, image encoder 34 performs feature encoding based on its structural configuration and outputs the encoded image features (Output). Before inputting the encoded image features (Output) from image encoder 34 into mask decoder 35, the open vocabulary object detector Mamba-YOLO-World is introduced to perform preliminary foreign object recognition, obtaining an initial detection bounding box and foreign object category.

[0049] In this embodiment, compared with the network model using MBConv, the contextual semantics extracted by the fused convolution module Fused-MBConv is richer, which can reduce the amount of computation and improve performance; the VRWKV model can efficiently and scalably process visual perception tasks, especially in image classification and dense prediction tasks, making the composed image encoder 34 lightweight.

[0050] Furthermore, the open vocabulary object detector Mamba-YOLO-World, together with the image encoder 34 and the mask decoder 35, forms a segmentation prediction network 4. The open vocabulary object detector Mamba-YOLO-World can identify the initial detection box and foreign object category of the foreign object in the image based on the input image and any text prompts. The initial detection box and foreign object category will be input into the mask decoder 35 as new supervision information, further improving the recognition accuracy of the object segmentation model 32.

[0051] Secondly, in the process of pre-training the segmentation prediction network 4, the target segmentation model 32 is first used as the student model, and the reference backbone network Hiera is used as the teacher model; then, the pre-training dataset is input into the teacher model, and the knowledge distillation strategy is adopted for the student model to extract knowledge from the teacher model to the student model.

[0052] Among them, using the reference backbone network Hiera to efficiently distill the target segmentation model 32 can greatly improve the efficiency of the target segmentation model 32 without losing the segmentation performance.

[0053] In this embodiment, the loss function of the target segmentation model 32 adopts the pixel-by-pixel mean square error (MSE) loss function to align the reference backbone network Hiera with the target segmentation model 32. The loss function between the target segmentation model 32 and the reference backbone network Hiera can be expressed as:

[0054] ;

[0055] in, i represents the input of the target segmentation model 32, X represents the output of the target segmentation model 32, Represents the output of the reference backbone network Hiera.

[0056] Furthermore, the mask loss between the true mask and the predicted mask output by the target segmentation model 32 is:

[0057] ;

[0058] in, represents the mask cross entropy loss, represents the mask Dice loss, and They are respectively the mask cross entropy loss and masked Dice loss The corresponding weight.

[0059] Furthermore, the detection box loss between the true detection box and the predicted detection box output by the target segmentation model 32 is:

[0060] ;

[0061] in, Indicates the detection box Smooth L1 loss, represents the category cross entropy loss, and They are detection box Smooth L1 loss and categorical cross entropy loss The corresponding weight.

[0062] In summary, the total loss function of the segmentation prediction network 4 is:

[0063] ;

[0064] in, and The mask loss is and detection box loss The corresponding weight.

[0065] Step S2: Preprocess the video and / or the original railway image to obtain a preprocessed image.

[0066] In this embodiment, after receiving input data, the data type is first determined. If it is video data, the video is decoded according to preset parameters and converted into video frames in a unified format. Each video frame corresponds to a preprocessed image. If it is image data (i.e., the original railway image), the decoding step is omitted and the preprocessed image is obtained by simply converting the data into a unified format.

[0067] Step S3: Perform image enhancement processing on the pre-processed image to obtain a target enhanced image.

[0068] It should be noted that the image enhancement methods used here include cropping, flipping, translation, scaling, photometric distortion, color conversion, grayscaling, noise addition, MixUp, CutMix, and RandAugment. These methods enhance the robustness of railway foreign object intrusion detection by adding strong interference to the image. Furthermore, environmental interference and weather disturbances in railway scenes can easily cause image blur, making accurate identification of intrusions difficult, especially in low-light conditions. Therefore, a new low-light image enhancement strategy is proposed for these low-light images.

[0069] Specifically, combined Figure 3 and Figure 4 , step 3 also includes the following sub-steps:

[0070] Step S31: setting a first threshold for distinguishing the illumination conditions corresponding to the pre-processed image.

[0071] The RGB mean values ​​of historical images under different lighting conditions (night, morning, afternoon, and evening) need to be calculated. A three-dimensional scatter plot analysis reveals that night images are more susceptible to spectral shifts at the lower end of the spectrum. A first threshold, T = (A, B, C), is preset. The RGB mean value is composed of three values ​​obtained by calculating the mean of each historical image in the RGB color space. The first threshold, T = (A, B, C), is the critical value for distinguishing the lighting conditions corresponding to railway images. A represents the threshold for the red space, B for the green space, and C for the blue space.

[0072] Step S32: performing image illumination clustering on the preprocessed image to obtain the three-channel mean.

[0073] Among them, image illumination clustering refers to calculating the mean of the preprocessed image according to the RGB color space to obtain the three-channel mean I(i)=(a, b, c), where a represents the mean of the red space, b represents the mean of the green space, and c represents the mean of the blue space.

[0074] Step S33: Determine whether the three-channel mean exceeds a first threshold.

[0075] Among them, it is judged whether the three-channel mean I(i)=(a, b, c) exceeds the first threshold T=(A, B, C); if the three-channel mean I(i)=(a, b, c) does not exceed the first threshold T=(A, B, C), the preprocessed image is a dark light image, and the process goes to step S24; if the three-channel mean I(i)=(a, b, c) exceeds the first threshold T=(A, B, C), the preprocessed image is other light image, and there is no need to perform image enhancement processing on the preprocessed image, and the process goes to step S25.

[0076] Step S34: performing image enhancement processing on the pre-processed image to obtain a target enhanced image.

[0077] Step S35: The pre-processed image is used as the target enhanced image.

[0078] Among them, the target enhanced image is a railway image with obvious image clarity and detail information, in which the influence of lighting is small, and it can be clearly seen that there are foreign objects in the railway.

[0079] In step S34, the Illumination Adaptive Transformer (IAT) is used to enhance the preprocessed image, transforming the night scene into a daytime-like scene. This improves image clarity and detail, thereby reducing the negative impact of low light on the subsequent recognition process.

[0080] In this embodiment, the dark light image corresponds to the lighting condition of night. By judging the size of the three-channel mean I(i)=(a, b, c) and the first threshold T=(A, B, C), the two lighting conditions of night and other lighting can be distinguished.

[0081] In this embodiment, the above process primarily addresses the difficulty of identifying foreign objects in low light. First, the average values ​​of the preprocessed image are calculated in the RGB color space. A first threshold is then used to determine whether the preprocessed image is a low-light image or another type of image. If the image is a low-light image, the low-light image enhancement network (IAT) is invoked to transform the nighttime scene into a daytime-like scene, improving image clarity and detail, thereby reducing illumination variations that could negatively impact the performance of the segmentation prediction network 4. For other types of images, the image enhancement process is omitted.

[0082] like Figure 8 As shown, it shows Figure 7 The target enhanced image is obtained by image enhancement processing of the pre-processed image shown. Figure 8 Compared to Figure 7 It is clearer and you can clearly see that there is a foreign object in the middle of the left track.

[0083] Step S4: Perform segmentation prediction on the target enhanced image to obtain a segmentation prediction result.

[0084] Combine Figure 2 and Figure 3 , determine whether the segmentation prediction network 4 has input text prompt information. If there is text prompt information, the open domain target detection model uses the initial detection box and foreign object category to preliminarily mark the target area in the target enhanced image according to the text prompt information to obtain a preliminary detection result; then input the preliminary detection result into the target segmentation model 32 for target segmentation, and the target segmentation model 32 directly outputs the segmentation prediction result.

[0085] like Figure 9 As shown in FIG. 3 , the segmentation prediction results output by the target segmentation model 32 are displayed. The figure uses multiple segmentation regions of different colors and text corresponding to the object categories to mark different objects, including multiple segmentation regions corresponding to people, foreign objects, and tracks. Objects of the same category are marked with the same text category, and foreign objects of a specific type that can be identified are marked with text of a specific category. For example, the text corresponding to the category of people (located in the red and brown segmentation regions in the figure) is marked with "person", and the text corresponding to the category of tracks (located in the yellow segmentation region in the figure) is marked with "railway". Foreign objects of an unidentifiable category (located in the cyan segmentation region in the figure) are marked with the text of the category of objects. Multiple objects of the same category are distinguished by different color labels. For example, the two different people on the left side of the figure are marked with red and brown segmentation regions, respectively.

[0086] Step S5: Post-process the segmentation prediction results to obtain foreign body recognition results.

[0087] In this embodiment, post-processing involves removing the identifiers corresponding to abnormal objects outside the track area from the segmentation prediction results, retaining only the identifiers corresponding to abnormal objects within the track area. The foreign object recognition results are the segmentation and detection results of abnormal objects within the track area of ​​the image, and include files in various formats, such as a JSON-formatted prediction label file and a test result image.

[0088] Combine Figure 3 、 Figure 9 and Figure 10 , the segmentation prediction results output by the target segmentation model 32 are post-processed. Figure 9 The identification of the corresponding parts of the person and track in the image is removed, so that the segmented area corresponding to the person and track is restored to Figure 8 The corresponding area in the railway track area is finally obtained to obtain the foreign object recognition result. Figure 10 In the figure, the target foreign object is marked with a blue segmentation area and the text object.

[0089] like Figure 3 As shown, after obtaining the foreign object recognition result, it is necessary to determine whether the foreign object recognition task has been completed; if the input multimodal data is a video frame and there are subsequent frame recognition tasks that have not been completed, it indicates that the task has not been completed, and the predicted foreign object recognition result is updated to the memory bank. The target segmentation model 32 will refer to the content of this memory bank for information exchange during subsequent frame recognition. If the task has been completed, the foreign object recognition result is output and the collection step is entered. In the collection step, it is determined whether the foreign object recognition result is a high-confidence result. If it is a high-confidence result, the high-confidence result is used as its target pseudo-label, and the preprocessed image and the target pseudo-label are included in the pseudo-label database. Subsequently, semi-supervised technology can be used to quickly fine-tune the target segmentation model 32 to improve the utilization rate of unlabeled data.

[0090] As another example, before step S1, an adaptive segmentation network needs to be constructed first.

[0091] like Figure 5 As shown, the adaptive segmentation network includes a first branch network 1, a second branch network 2 and a third branch network 3. The first branch network 1 includes a first enhancement module 11, a first segmentation model 12 and a post-processing module 13. The second branch network 2 includes a second enhancement module 21, a second segmentation model 22 and an adaptive screening module 23. The third branch network 3 includes a third enhancement module 31, a target segmentation model 32 and a prompt encoder 33.

[0092] Secondly, after building the adaptive segmentation network, it is necessary to use the training data set to train the adaptive segmentation network. The training data set includes labeled images and unlabeled images.

[0093] like Figure 5As shown, in the first branch network 1, the first enhancement module 11 is used to perform weak image enhancement on the labeled image and strong image enhancement on the unlabeled image, respectively obtaining a first weakly enhanced image and a strongly enhanced image; the first weakly enhanced image and the strongly enhanced image are input into the first segmentation model 12 for target segmentation to obtain a first segmentation result. The first segmentation result includes a true pseudo-label 14 corresponding to the first weakly enhanced image and a first pseudo-label 15 corresponding to the strongly enhanced image, and a self-supervised loss is used between the true pseudo-label 14 and the true label 17; the post-processing module 13 is used to perform post-processing operations on the first pseudo-label 15 and the third segmentation result 36 to filter out small segmentation results, thereby obtaining a more accurate target segmentation result 16. The post-processing here refers to the fusion of the segmentation results output by the first segmentation model 12 and the segmentation results output by the target segmentation model 32, retaining the labels corresponding to abnormal targets in the track area.

[0094] In the second branch network 2, the second enhancement module 21 is used to weakly enhance the unlabeled image to obtain a second weakly enhanced image; the second weakly enhanced image is input into the second segmentation model 22 for target segmentation to obtain a second segmentation result 24; the adaptive screening module 23 adaptively screens the second segmentation result 24 to obtain a second pseudo label 25.

[0095] In the third branch network 3, the third enhancement module 31 is used to pre-enhance the unlabeled image to obtain a pre-enhanced image; the pre-enhanced image is input into the target segmentation model 32 for target segmentation to obtain a third segmentation result 36; wherein, corresponding consistency losses are used between the first pseudo label 15 and the second pseudo label 25 and the third segmentation result 36 respectively.

[0096] It should be noted that the image pre-enhancement is similar to the image enhancement process in the above step S3, and will not be described in detail here.

[0097] Furthermore, during training, the first segmentation model 12 receives the first weakly enhanced image and the strongly enhanced image, obtains the true pseudo-label 14 corresponding to the labeled image and the first pseudo-label 15 corresponding to the unlabeled image, and performs semi-supervised training on the first segmentation model 12. The second segmentation model 22 adds the second weakly enhanced image and, in combination with an adaptive screening strategy, obtains the second pseudo-label 25 corresponding to the unlabeled image, and performs unsupervised training on the second segmentation model 22. Simultaneously, the parameters of the second segmentation model 22 are updated using the exponential moving average (EMA) of the parameters of the first segmentation model 12.

[0098] In this embodiment, adaptive screening adopts a dynamic threshold process class-wise GMM, which regards high-quality and low-quality pseudo-label segmentation samples as positive samples (target category) and negative samples (background), respectively. The maximum expectation EM algorithm is used to iteratively learn the optimal parameters of the Gaussian distribution corresponding to positive and negative samples. The Gaussian mixture model GMM can update the threshold of each category in each iterative training, dynamically filter out high-quality samples, and solve the pseudo-label inconsistency problem caused by traditional hard thresholds.

[0099] Among them, since the input of the second branch network 2 is an unlabeled image, the obtained second segmentation result 24 may have inaccurate predictions. By using adaptive screening to filter the second segmentation result 24, a more reliable second pseudo label 25 can be obtained.

[0100] like Figure 5 As shown, the target segmentation model 32 includes an image encoder 34 and a mask decoder 35. After the third branch network 3 inputs the unlabeled image, the third enhancement module 31 first performs image pre-enhancement; then the obtained pre-enhanced image is input to the image encoder 34 for encoding. At the same time, the first pseudo label 15 predicted by the first segmentation model 12 is input as a mask prompt to the prompt encoder 33 (corresponding to Figure 2 The prompt detection model 41 may include a prompt encoder 33, which can encode the preprocessed image to realize the target prompt detection of the mask prompt type; finally, the mask decoder 35 decodes the encoding results of the image encoder 34 and the prompt encoder 33 to obtain a third segmentation result 36, thereby realizing direct prediction of the unlabeled image under multi-source input. After that, the third segmentation result 36 is fused with the first pseudo label 15 of the first segmentation model, further improving the segmentation prediction effect of the first branch network 1.

[0101] Furthermore, the total loss function of the adaptive segmentation network is:

[0102] ;

[0103] in, is the self-supervised loss of the first segmentation model 12, is the consistency loss between the second segmentation model 22 and the first segmentation model 12, is the consistency loss between the target segmentation model 32 and the first segmentation model 12, 、 and are the corresponding loss weights respectively.

[0104] Furthermore, when using the trained adaptive segmentation network to perform railway foreign object intrusion identification, corresponding steps S1-S5 are executed.

[0105] The processes of steps S1 and S2 are similar to those of the above steps S1 and S2 and are not described in detail here. However, instead of receiving multi-source prompt information, only video and / or image data is received and corresponding preprocessing operations are performed on the video and / or image data to obtain corresponding preprocessed images.

[0106] It should be noted that the pre-processed image is an unlabeled image. Therefore, steps S3 to S5 are similar to the training process of the unlabeled image described above and will not be repeated here. However, here, the image to be detected is subjected to strong image enhancement, weak image enhancement, and image pre-enhancement, respectively, and then target segmentation is performed. The corresponding segmentation prediction results obtained are the first target segmentation result, the second target segmentation result, and the third target segmentation result; then, the first target segmentation result and the third target segmentation result are post-processed, and finally the target segmentation result 16 corresponding to the pre-processed image is obtained, which is the foreign body recognition result.

[0107] In this application, under the condition of limited labeled data, semi-supervised technology is used to achieve rapid and continuous optimization of the adaptive segmentation network. The obtained adaptive segmentation network has excellent foreign object detection capabilities and can effectively detect foreign objects invading the railway track area in real time.

[0108] It should be noted that the process of constructing and training the adaptive segmentation network above adopts a semi-supervised optimization algorithm, and other solutions such as self-supervised learning algorithms and pseudo-label algorithms can also be used as substitutes. If there is a problem of insufficient data on railway foreign object intrusion during training, image generation and video generation technologies can be used to expand the data. The RWKV module used in the above-mentioned target segmentation model 32 can also be replaced by a new Mamba module or other solutions. The open vocabulary image target algorithm used in the above-mentioned open domain target detection model can also be replaced by other algorithms such as the GroundedDINO algorithm and the DetClipV3 algorithm, and the same effect can be achieved.

[0109] Compared with the prior art, this application has the following beneficial effects:

[0110] By combining semi-supervised technology, video image automatic recognition technology, and image enhancement technology, the present invention can effectively detect foreign objects invading the railway track area in real time, reducing the occurrence of algorithm false detection and missed detection.

[0111] This application uses multimodal data input to design a more efficient object segmentation model 32. Knowledge distillation is used to make this model high-performance and lightweight. Furthermore, a prompt detection model 41 is combined with additional supervisory information to provide this model with real-time, accurate identification of any foreign object within the railway track area. Furthermore, based on multi-source prompt information, this system can automatically, in real time, and accurately identify railway track areas and intruding objects within them in videos and images, improving data processing speed, significantly reducing manual annotation costs, and forming a valuable railway foreign object intrusion database.

[0112] In addition, this application also introduces semi-supervised segmentation technology and designs an adaptive segmentation network in combination with the target segmentation model 32. In the case of limited labeled data, the semi-supervised technology is used to achieve rapid and continuous optimization of the adaptive segmentation network, which greatly improves the detection performance and generalization ability of the adaptive segmentation network.

[0113] Finally, to address the challenge of difficult image recognition in low light, this application proposes a set of image enhancement strategies that significantly improve the quality and detail of low-light images, further enhancing the accuracy of model recognition. Using low-light image enhancement technology, we provide a new and effective solution to the problem of missed detections and false detections that image algorithms are prone to in low light conditions.

[0114] In summary, this application can significantly improve the safety of railway transportation in practical applications, reduce the risk of accidents caused by foreign objects intruding, improve the utilization rate of massive railway monitoring data, and provide a set of practical and usable semi-supervised adaptive segmentation solutions based on railway scenarios, which has broad application prospects and market value.

[0115] Based on the same inventive concept, Figure 11 As shown, the present invention also provides a railway foreign object intrusion identification system, the system comprising:

[0116] The data acquisition unit 101 is configured to acquire multimodal data, where the multimodal data includes video and / or original railway images.

[0117] The preprocessing unit 102 is configured to preprocess the video and / or the original railway image to obtain a preprocessed image.

[0118] The image enhancement unit 103 is configured to perform image enhancement processing on the pre-processed image to obtain a target enhanced image.

[0119] The segmentation prediction unit 104 is configured to perform image segmentation prediction on the target enhanced image to obtain a segmentation prediction result.

[0120] The post-processing unit 105 is used to perform post-processing on the segmentation prediction result to obtain a foreign object recognition result; wherein, post-processing refers to retaining the identification corresponding to the abnormal target in the track area.

[0121] In this application, other technical features of the above-mentioned railway foreign object intrusion identification system are the same as the features disclosed in the above-mentioned method embodiment and will not be repeated here.

[0122] Based on the same inventive concept, the present application also provides an electronic device, which includes a processor, a memory and a communication circuit, and the processor is connected to the memory and the communication circuit respectively; wherein the communication circuit is used for communication connection, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above method.

[0123] See also Figure 12 The electronic device described in the embodiment of the present application may specifically include a processor 210 and a memory 220 . The memory 220 is coupled to the processor 210 .

[0124] Processor 210 is used to control the operation of the electronic device. Processor 210 may also be referred to as a CPU (Central Processing Unit). Processor 210 may be an integrated circuit chip with signal processing capabilities. Processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor, or processor 210 may be any conventional processor.

[0125] The memory 220 is used to store computer programs and can be RAM, ROM, or other types of storage terminals. Specifically, the memory 220 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 220 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage terminals and flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 220 is used to store at least one program code.

[0126] The processor 210 is configured to execute the computer program stored in the memory 220 to implement the methods described in the various method embodiments of the present application.

[0127] In some embodiments, the electronic device may further include a peripheral terminal interface 230 and at least one peripheral terminal. The processor 210, memory 220, and peripheral terminal interface 230 may be connected via a bus or signal lines. Each peripheral terminal may be connected to the peripheral terminal interface 230 via a bus, signal lines, or circuit boards. Specifically, the peripheral terminal includes at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260, and a power supply 270.

[0128] The peripheral terminal interface 230 can be used to connect at least one peripheral terminal related to I / O (Input / Output) to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral terminal interface 230 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 210, the memory 220, and the peripheral terminal interface 230 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0129] The RF circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 240 communicates with communication networks and other IoT devices via electromagnetic signals, and is therefore the communication circuitry of the electronic device. The RF circuit 240 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 240 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identification module card, and the like. The RF circuit 240 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 240 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0130] The display screen 250 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 250 is a touch screen, it is also capable of collecting touch signals on or above the surface of the display screen 250. These touch signals can be input as control signals to the processor 210 for processing. In this case, the display screen 250 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 250, disposed on the front panel of the electronic device; in other embodiments, there can be at least two display screens 250, disposed on different surfaces of the electronic device or in a foldable design; in still other embodiments, the display screen 250 can be a flexible display, disposed on a curved or foldable surface of the electronic device. Furthermore, the display screen 250 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0131] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals to be input into the processor 210 for processing, or input into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signals from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0132] Power supply 270 is used to power various components in the electronic device. Power supply 270 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0133] For a detailed description of the functions and execution processes of the various functional modules or components in the electronic device embodiments of the present application, reference can be made to the descriptions in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0135] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0136] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0137] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above method.

[0138] See also Figure 13 If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, as well as electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. that have the above-mentioned storage media.

[0139] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0140] It can be seen that the present invention discloses a method, system, device and medium for identifying foreign objects intruding on railway tracks. The method includes the following steps: acquiring multimodal data, which includes multi-source prompt information, video and / or original railway images; preprocessing the video and / or original railway images to obtain preprocessed images; performing image enhancement processing on the preprocessed images to obtain target enhanced images; performing image segmentation prediction on the target enhanced images to obtain segmentation prediction results; post-processing the segmentation prediction results using a segmentation prediction network to obtain foreign object identification results; wherein post-processing refers to retaining the identifiers corresponding to abnormal targets in the track area. This method can quickly and effectively identify foreign objects in massive amounts of unlabeled data, and can effectively detect foreign objects intruding on railway tracks in real time, reducing the occurrence of algorithm false detection and missed detection.

[0141] The above are merely embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any equivalent structural transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of protection of the present invention.

Claims

1. A railway foreign body intrusion identification method, characterized in that: The method comprises the following steps: Acquiring multimodal data, wherein the multimodal data includes video and / or original railway images; Preprocessing the video and / or the original railway image to obtain a preprocessed image; Performing image enhancement processing on the preprocessed image to obtain a target enhanced image; Performing image segmentation prediction on the target enhanced image to obtain a segmentation prediction result; Post-processing the segmentation prediction results to obtain foreign object recognition results; wherein the post-processing refers to retaining the identifiers corresponding to abnormal targets in the track area; The multimodal data also includes multi-source prompt information, and the method further includes the steps of: Constructing a segmentation prediction network, the segmentation prediction network including a target segmentation model and a prompt detection model; wherein, when the multi-source prompt information includes text prompt information, the prompt detection model includes an open domain target detection model, the open domain target detection model performs preliminary recognition on the target enhanced image based on the text prompt information to obtain an initial detection box and a foreign object category; the target segmentation model performs target segmentation on the target enhanced image based on the multi-source prompt information, the initial detection box, and the foreign object category to obtain a segmentation prediction result; The segmentation prediction network is pre-trained using a pre-training dataset; wherein, the target segmentation model is first used as a student model, and the reference backbone network Hiera is used as a teacher model; then, the pre-training dataset is input into the teacher model, and a knowledge distillation strategy is adopted for the student model to extract knowledge from the teacher model to the student model.

2. The railway foreign object intrusion identification method according to claim 1 is characterized in that: The total loss function of the segmentation prediction network is: ； in, and are the weights corresponding to the mask loss and the detection box loss respectively; The mask loss between the true mask and the predicted mask output by the target segmentation model; The detection box loss between the real detection box and the predicted detection box output by the target segmentation model; The mask loss and the detection box loss They are: ； ； in, represents the mask cross entropy loss; represents the mask Dice loss; and They are respectively the mask cross entropy loss and masked Dice loss The corresponding weight; Indicates the detection box Smooth L1 loss; represents the category cross entropy loss; and They are detection box Smooth L1 loss and categorical cross entropy loss The corresponding weight.

3. The railway foreign object intrusion identification method according to claim 1 is characterized in that: The method of performing image enhancement processing on the pre-processed image to obtain a target enhanced image comprises the steps of: Performing image illumination clustering on the preprocessed image to obtain a three-channel mean; The image illumination clustering refers to performing mean calculation on the image according to the RGB color space; Determining whether the three-channel mean exceeds a first threshold, where the first threshold is a critical value for distinguishing the illumination conditions corresponding to the preprocessed image; If the three-channel mean does not exceed the first threshold, the pre-processed image is a dark-light image, and the image enhancement process is performed on the pre-processed image to obtain a target enhanced image; If the three-channel mean exceeds the first threshold, the pre-processed image is another light image, and there is no need to perform the image enhancement processing on the pre-processed image. The pre-processed image is directly set as the target enhanced image.

4. The railway foreign object intrusion identification method according to claim 1 is characterized in that: The method further comprises the steps of: Constructing an adaptive segmentation network, the adaptive segmentation network comprising a first branch network, a second branch network, and a third branch network, wherein the first branch network, the second branch network, and the third branch network are respectively used to perform strong image enhancement, weak image enhancement, and image pre-enhancement on the preprocessed image, and then perform target segmentation, thereby correspondingly obtaining a first target segmentation result, a second target segmentation result, and a third target segmentation result; Then the first branch network is also used to post-process the first target segmentation result and the third target segmentation result to obtain the foreign object recognition result.

5. The railway foreign object intrusion identification method according to claim 4 is characterized in that: The method further comprises the steps of: Training the adaptive segmentation network using a training data set, wherein the training data set includes labeled images and unlabeled images; Among them, in the first branch network, the labeled image is weakly enhanced, and the unlabeled image is strongly enhanced to obtain a first weakly enhanced image and a strongly enhanced image, respectively; the first weakly enhanced image and the strongly enhanced image are input into the first segmentation model for target segmentation to obtain a first segmentation result, which includes a true pseudo label corresponding to the first weakly enhanced image and a first pseudo label corresponding to the strongly enhanced image; a self-supervised loss is used between the true pseudo label and the true label, and the first pseudo label is used for post-processing with the third segmentation result output by the third branch network to obtain a target segmentation result corresponding to the unlabeled image; In the second branch network, weakly enhancing the unlabeled image to obtain a second weakly enhanced image; inputting the second weakly enhanced image into a second segmentation model to perform target segmentation to obtain a second segmentation result; and adaptively screening the second segmentation result to obtain a second pseudo label; In the third branch network, the unlabeled image is pre-enhanced to obtain a pre-enhanced image; the pre-enhanced image is input into the target segmentation model for target segmentation to obtain a third segmentation result; wherein, the first pseudo label is respectively subjected to corresponding consistency loss functions with the second pseudo label and the third segmentation result.

6. The railway foreign object intrusion identification method according to claim 5, characterized in that: The total loss function of the adaptive segmentation network is: ； in, is the supervision loss of the first segmentation model, is the consistency loss between the second segmentation model and the first segmentation model, is the consistency loss between the target segmentation model and the first segmentation model, 、 and are the corresponding loss weights respectively.

7. A railway foreign object intrusion identification system, characterized in that: The system comprises: A data acquisition unit, configured to acquire multimodal data, wherein the multimodal data includes multi-source prompt information, video and / or original railway image; a preprocessing unit, configured to preprocess the video and / or the original railway image to obtain a preprocessed image; An image enhancement unit, configured to perform image enhancement processing on the preprocessed image to obtain a target enhanced image; A segmentation prediction unit, configured to perform image segmentation prediction on the target enhanced image to obtain a segmentation prediction result; A post-processing unit, configured to perform post-processing on the segmentation prediction result to obtain a foreign object recognition result; wherein the post-processing refers to retaining the identification corresponding to the abnormal target in the track area; The multimodal data also includes multi-source prompt information, and further includes: constructing a segmentation prediction network, the segmentation prediction network including a target segmentation model and a prompt detection model; wherein, when the multi-source prompt information includes text prompt information, the prompt detection model includes an open domain target detection model, the open domain target detection model performs preliminary recognition on the target enhanced image based on the text prompt information to obtain an initial detection box and a foreign object category; the target segmentation model performs target segmentation on the target enhanced image based on the multi-source prompt information, the initial detection box and the foreign object category to obtain a segmentation prediction result; The segmentation prediction network is pre-trained using a pre-training dataset; wherein, the target segmentation model is first used as a student model, and the reference backbone network Hiera is used as a teacher model; then, the pre-training dataset is input into the teacher model, and a knowledge distillation strategy is adopted for the student model to extract knowledge from the teacher model to the student model.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images