Railway foreign matter invasion identification method, system, equipment and medium

Abstract

The invention discloses a railway foreign matter invasion identification method, system and device and a medium, and the method comprises the following steps: obtaining multi-modal data, the multi-modal data comprising a video and / or an original railway image; preprocessing the video and / or the original railway image to obtain a preprocessed image; performing image enhancement processing on the preprocessed image to obtain a target enhanced image; performing image segmentation prediction on the target enhanced image to obtain a segmentation prediction result; performing post-processing on the segmentation prediction result to obtain a foreign matter recognition result; wherein the post-processing refers to reserving an identifier corresponding to the abnormal target in the rail-mounted area. According to the method, rapid and effective foreign matter recognition processing can be carried out on mass label-free data, the foreign matter invasion target in the railway track area can be effectively detected in real time, and the situations of false detection and missing detection of an algorithm are reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a railway foreign object intrusion recognition method, system, equipment and medium. Background Art

[0002] With the rapid development of railway transportation, the threat of foreign objects to railway safety has become increasingly prominent. Ensuring the safety of train operation has become an important task in the railway field, which has prompted the continuous development and research of foreign object intrusion identification technology. At present, AI intelligent technology has been successfully applied in various fields, and some railway foreign object intrusion algorithms based on deep learning have emerged. However, due to the special working conditions of the railway industry and the complex actual on-site environment, general AI intelligent technology relies on a large amount of supervision data, with high hardware requirements and training costs, and the actual data annotation cost is high, and the data processing efficiency cannot meet the needs of railway business.

[0003] In practical applications, railway foreign object intrusion monitoring generates a large amount of multi-source data, including videos, images, texts, etc., and there are many different interferences in the data, such as background, weather, light, etc. On the other hand, there are few real cases of railway foreign object intrusion, and a small amount of labeled data can easily cause overfitting of the model. In summary, the particularity of the railway industry and the diversity of its scenarios indicate the necessity of customized railway scenario algorithms, and high requirements are placed on the real-time and robustness of the algorithms. Summary of the invention

[0004] The purpose of the present invention is to provide a railway foreign object intrusion identification method, system, equipment and medium to solve the problems of high manual labeling cost, small amount of label data, high model calculation complexity and poor generalization.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is to provide a method for identifying foreign object intrusion on railways, comprising the following steps: acquiring multimodal data, wherein the multimodal data includes video and / or original railway images; preprocessing the video and / or original railway images to obtain preprocessed images; performing image enhancement processing on the preprocessed images to obtain target enhanced images; performing image segmentation prediction on the target enhanced images using a segmentation prediction network to obtain segmentation prediction results; post-processing the segmentation prediction results to obtain foreign object identification results; wherein the post-processing refers to retaining the identification corresponding to the abnormal targets in the track area.

[0006] In some embodiments, the multimodal data also includes multi-source prompt information, and the method also includes the steps of: constructing a segmentation prediction network, the segmentation prediction network including a target segmentation model and a prompt detection model; wherein, when the multi-source prompt information includes text prompt information, the prompt detection model includes an open domain target detection model, and the open domain target detection model performs preliminary recognition on the target enhanced image according to the text prompt information to obtain an initial detection box and a foreign object category; the target segmentation model performs target segmentation on the target enhanced image according to the multi-source prompt information, the initial detection box and the foreign object category to obtain a segmentation prediction result; using a pre-training data set, pre-training the segmentation prediction network; wherein, firstly, the target segmentation model is used as a student model, and the reference backbone network Hiera is used as a teacher model; then, the pre-training data set is input into the teacher model, and a knowledge distillation strategy is adopted for the student model to extract knowledge from the teacher model to the student model.

[0007] In some embodiments, the total loss function of the segmentation prediction network is: ; in, and They are the weights corresponding to the mask loss and the detection box loss respectively; The mask loss between the true mask and the predicted mask output by the target segmentation model; The detection box loss between the real detection box and the predicted detection box output by the target segmentation model; The mask loss and the detection box loss They are: ; ; in, represents the mask cross entropy loss; represents the masked Dice loss; and They are respectively the mask cross entropy loss and masked Dice loss The corresponding weights; Indicates the Smooth L1 loss of the detection box; represents the category cross entropy loss; and They are the detection box Smooth L1 loss and categorical cross entropy loss The corresponding weight.

[0008] In some embodiments, the image enhancement processing is performed on the preprocessed image to obtain a target enhanced image, including the steps of: performing image illumination clustering on the preprocessed image to obtain a three-channel mean; the image illumination clustering refers to performing mean calculations on the image according to the RGB color space; judging whether the three-channel means exceed a first threshold, wherein the first threshold is a critical value for distinguishing the illumination conditions corresponding to the preprocessed image; if the three-channel means does not exceed the first threshold, the preprocessed image is a dark light image, and the image enhancement processing is performed on the preprocessed image to obtain a target enhanced image; if the three-channel means exceeds the first threshold, the preprocessed image is an image of other light, and there is no need to perform the image enhancement processing on the preprocessed image, and the preprocessed image is directly made the target enhanced image.

[0009] In some embodiments, the method also includes the steps of: constructing an adaptive segmentation network, constructing an adaptive segmentation network, the adaptive segmentation network includes a first branch network, a second branch network and a third branch network, the first branch network, the second branch network and the third branch network are respectively used to perform strong image enhancement, weak image enhancement and image pre-enhancement on the pre-processed image, and then perform target segmentation to obtain the first target segmentation result, the second target segmentation result and the third target segmentation result respectively; then the first branch network is also used to post-process the first target segmentation result and the third target segmentation result to obtain the foreign object recognition result.

[0010] In some embodiments, the method further includes the steps of: using a training data set to train the adaptive segmentation network, the training data set including labeled images and unlabeled images; wherein, in the first branch network, the labeled images are weakly enhanced, and the unlabeled images are strongly enhanced to obtain a first weakly enhanced image and a strongly enhanced image, respectively; the first weakly enhanced image and the strongly enhanced image are input into a first segmentation model for target segmentation to obtain a first segmentation result, the first segmentation result including a true pseudo-label corresponding to the first weakly enhanced image and a first pseudo-label corresponding to the strongly enhanced image; a self-supervised loss is used between the true pseudo-label and the true label, and the first pseudo-label is used to compare with the third The third segmentation result output by the branch network is post-processed to obtain a target segmentation result corresponding to the unlabeled image; in the second branch network, the unlabeled image is weakly enhanced to obtain a second weakly enhanced image; the second weakly enhanced image is input into the second segmentation model for target segmentation to obtain a second segmentation result; the second segmentation result is adaptively screened to obtain a second pseudo-label; in the third branch network, the unlabeled image is pre-enhanced to obtain a pre-enhanced image; the pre-enhanced image is input into the target segmentation model for target segmentation to obtain a third segmentation result; wherein, the first pseudo-label uses corresponding consistency loss functions with the second pseudo-label and the third segmentation result respectively.

[0011] In some embodiments, the total loss function of the adaptive segmentation network is: ; in, is the supervised loss of the first segmentation model, is the consistency loss between the second segmentation model and the first segmentation model, is the consistency loss between the target segmentation model and the first segmentation model, , and are the corresponding loss weights respectively.

[0012] The present invention also provides a railway foreign object intrusion identification system, the system comprising: a data acquisition unit, used to acquire multimodal data, the multimodal data including video and / or original railway images; a preprocessing unit, used to preprocess the video and / or original railway images to obtain a preprocessed image; an image enhancement unit, used to perform image enhancement processing on the preprocessed image to obtain a target enhanced image; a segmentation prediction unit, used to perform image segmentation prediction on the target enhanced image to obtain a segmentation prediction result; a post-processing unit, used to post-process the segmentation prediction result to obtain a foreign object identification result; wherein the post-processing refers to retaining the identification corresponding to the abnormal target in the track area.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0014] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0015] The beneficial effects of the present invention are as follows: the present invention discloses a method, system, device and medium for identifying foreign objects intruding on railways, the method comprising the following steps: acquiring multimodal data, the multimodal data comprising video and / or original railway images; preprocessing the video and / or original railway images to obtain preprocessed images; performing image enhancement processing on the preprocessed images to obtain target enhanced images; performing image segmentation prediction on the target enhanced images to obtain segmentation prediction results; post-processing the segmentation prediction results to obtain foreign object identification results; wherein post-processing refers to retaining the identification corresponding to the abnormal targets in the track area. The present method can perform fast and effective foreign object identification processing on massive amounts of unlabeled data, can effectively detect foreign object intruding targets in the railway track area in real time, and reduce the occurrence of algorithm misdetection and missed detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is an overall flow chart of an embodiment of a railway foreign body intrusion identification method of the present invention; Figure 2 It is a block diagram of the composition of the segmentation prediction network in an embodiment of a railway foreign body intrusion identification method of the present invention; Figure 3 It is a specific flow chart of an embodiment of a railway foreign body intrusion identification method of the present invention; Figure 4 This is a specific flow chart of step S3 in an embodiment of a railway foreign object intrusion identification method of the present invention; Figure 5 It is a schematic diagram of the principle of an adaptive segmentation network in another embodiment of a railway foreign object intrusion identification method of the present invention; Figure 6 It is a block diagram of the composition of an image encoder used in an embodiment of a railway foreign object intrusion identification method of the present invention; Figure 7 It is a pre-processed image obtained in an embodiment of a railway foreign body intrusion identification method of the present invention; Figure 8 It is a target enhanced image obtained in an embodiment of a railway foreign object intrusion identification method of the present invention; Fig. 9It is a segmentation prediction result diagram obtained in an embodiment of a railway foreign body intrusion identification method of the present invention; Fig.10 It is a foreign object recognition result diagram obtained in an embodiment of a railway foreign object intrusion recognition method of the present invention; Fig.11 It is a structural block diagram of an embodiment of a railway foreign object intrusion identification system of the present invention; Fig.12 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention; Fig.13 It is a schematic block diagram of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0017] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0018] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0019] like Figure 1-6 As shown, a railway foreign body intrusion identification method of the present invention is shown, comprising the following steps: Step S1: Acquire multimodal data, where the multimodal data includes videos and / or original railway images.

[0020] It should be noted that the present application supports multimodal data input from cameras for on-site monitoring of tracks and nearby areas or other interfaces in the background, and the multimodal data includes other data types such as videos and images. In some embodiments, the multimodal data also includes multi-source prompt information, and the present application establishes a segmentation prediction network 4 for foreign body recognition in images. The segmentation prediction network 4 and multi-source prompt information will be further introduced in the following content. In other embodiments, the present application establishes an adaptive segmentation network for foreign body recognition in images.

[0021] As an example, before step S1, it is necessary to construct a segmentation prediction network 4 and pre-train the segmentation prediction network 4 using a pre-training dataset.

[0022] Combination Figure 2 and Figure 3 , the segmentation prediction network 4 supports input of multimodal data, which includes videos, images, and multi-source prompt information such as rectangular boxes, points, masks, and texts. The segmentation prediction network 4 includes a target segmentation model 32 and a prompt detection model 41. The prompt detection model 41 can realize fast and accurate foreign body intrusion detection on image and video data according to multi-source prompt information, and input the detection results into the target segmentation model 32; the target segmentation model 32 performs the target segmentation task and marks the target foreign body in the image.

[0023] like Figure 3 As shown, the prompt detection model 41 includes an open domain target detection model. The open domain target detection model adopts the open vocabulary target detector Mamba-YOLO-World, which can identify the initial detection box and foreign object category of the foreign object in the image according to the input image and any text prompt information. The initial detection box and foreign object category will be input into the target segmentation model 32 as new supervision information, further improving the recognition accuracy of the segmentation prediction network 4.

[0024] In some other embodiments, the multi-source prompt information input into the segmentation prediction network 4 also includes other types of prompt information such as rectangular boxes, points, masks, etc., and the prompt detection model 41 also includes corresponding types of target detection models to realize different types of target prompt detection according to different types of target detection models.

[0025] Combination Figure 5 and Figure 6 The target segmentation model 32 includes an image encoder 34 and a mask decoder 35. The image encoder 34 is used to perform feature encoding on the received input image Input image and output the image encoding feature Output; the mask decoder 35 is used to receive the image encoding feature Output, as well as the initial detection box and foreign object category output by the open vocabulary target detector Mamba-YOLO-World, and perform image decoding according to the image encoding feature Output, the initial detection box and the foreign object category.

[0026] like Figure 6 As shown in the figure, the image encoder 34 is designed by utilizing the high-efficiency and low-parameter network characteristics of RWKV. The image encoder 34 includes a convolutional neural network module Stem, a fused convolution module Fused-MBConv and a VRWKV (Vision-RWKV) model. The convolutional neural network module Stem first increases the dimension and reduces the size of the input image Input image; then a two-layer fused convolution module Fused-MBConv is used for further fused convolution. Finally, the VRWKV model performs image encoding to obtain the corresponding image encoding feature Output.

[0027] After the image encoder 34 receives the input image Input image, it performs corresponding feature encoding according to the structural setting of the image encoder 34 and outputs the image encoding feature Output. Before the image encoding feature Output output by the image encoder 34 is input to the mask decoder 35, the open vocabulary object detector Mamba-YOLO-World is introduced to realize the preliminary recognition of foreign objects and obtain the initial detection box and foreign object category.

[0028] In this embodiment, compared with the network model using MBConv, the context semantics extracted by the fused convolution module Fused-MBConv is richer, which can reduce the amount of calculation and improve performance; the VRWKV model can efficiently and scalably process visual perception tasks, especially in image classification and intensive prediction tasks, making the composed image encoder 34 lightweight.

[0029] Furthermore, the open vocabulary object detector Mamba-YOLO-World, together with the image encoder 34 and the mask decoder 35, forms a segmentation prediction network 4. The open vocabulary object detector Mamba-YOLO-World can identify the initial detection box and foreign object category of the foreign object in the image according to the input image and any text prompts. The initial detection box and foreign object category will be input into the mask decoder 35 as new supervision information, further improving the recognition accuracy of the object segmentation model 32.

[0030] Secondly, in the process of pre-training the segmentation prediction network 4, the target segmentation model 32 is first used as the student model, and the reference backbone network Hiera is used as the teacher model; then, the pre-training data set is input into the teacher model, and the knowledge distillation strategy is adopted for the student model to extract knowledge from the teacher model to the student model.

[0031] Among them, using the reference backbone network Hiera to efficiently distill the target segmentation model 32 can greatly improve the efficiency of the target segmentation model 32 without losing the segmentation performance.

[0032] In this embodiment, the loss function of the target segmentation model 32 adopts a pixel-by-pixel mean square error MSE loss function to align the reference backbone network Hiera and the target segmentation model 32. The loss function between the target segmentation model 32 and the reference backbone network Hiera can be expressed as: ; in, i represents the input of the target segmentation model 32, X represents the output of the target segmentation model 32, Represents the output of the reference backbone network Hiera.

[0033] Furthermore, the mask loss between the true mask and the predicted mask output by the target segmentation model 32 is: ; in, represents the mask cross entropy loss, represents the masked Dice loss, and They are respectively the mask cross entropy loss and masked Dice loss The corresponding weight.

[0034] Furthermore, the detection box loss between the real detection box and the predicted detection box output by the target segmentation model 32 is: ; in, Indicates the detection box Smooth L1 loss, represents the category cross entropy loss, and They are the detection box Smooth L1 loss and categorical cross entropy loss The corresponding weight.

[0035] In summary, the total loss function of segmentation prediction network 4 is: ; in, and The mask loss is and detection box loss The corresponding weight.

[0036] Step S2: preprocessing the video and / or the original railway image to obtain a preprocessed image.

[0037] In this embodiment, after receiving the input data, the data type is first determined. If it is video data, the video is decoded according to preset parameters and converted into video frames in a unified format, each of which corresponds to a preprocessed image. If it is image data (i.e., the original railway image), the decoding step is omitted and the preprocessed image can be obtained by simply unifying the format.

[0038] Step S3: Perform image enhancement processing on the preprocessed image to obtain a target enhanced image.

[0039] It should be noted that the image enhancement processing methods used here include cropping, flipping, translation, scaling, photometric distortion, color conversion, grayscale, adding noise, MixUp, CutMix, RandAugment, etc., which improve the robustness of railway foreign object intrusion identification by adding strong interference to the image. At the same time, the surrounding environment interference and weather interference of railway scenes are easy to cause image blur, especially under the challenge of low light, it is difficult to accurately identify the intrusion target, so here a new image dark light enhancement strategy is proposed for this type of low-light image.

[0040] Specifically, combined Figure 3 and Figure 4 , step 3 also includes the following sub-steps: Step S31: setting a first threshold for distinguishing the illumination conditions corresponding to the preprocessed image.

[0041] Among them, it is necessary to calculate the RGB mean of historical images under different lighting conditions (night, morning, afternoon and evening). Through the three-dimensional scatter plot analysis, it can be found that night images are more likely to shift at the low end of the spectrum, and the first threshold T=(A, B, C) is preset. Among them, the RGB mean is composed of three values ​​obtained by calculating the mean of historical images according to the RGB color space; the first threshold T=(A, B, C) is the critical value for distinguishing the corresponding lighting conditions of railway images, A represents the threshold of the red space, B represents the threshold of the green space, and C represents the threshold of the blue space.

[0042] Step S32: performing image illumination clustering on the preprocessed image to obtain the three-channel mean.

[0043] Among them, image illumination clustering refers to calculating the mean of the preprocessed image according to the RGB color space to obtain the three-channel mean I(i)=(a, b, c), where a represents the mean of the red space, b represents the mean of the green space, and c represents the mean of the blue space.

[0044] Step S33: Determine whether the three-channel mean exceeds a first threshold.

[0045] Among them, it is judged whether the three-channel mean I(i)=(a, b, c) exceeds the first threshold T=(A, B, C); if the three-channel mean I(i)=(a, b, c) does not exceed the first threshold T=(A, B, C), the preprocessed image is a dark light image, and the process goes to step S24; if the three-channel mean I(i)=(a, b, c) exceeds the first threshold T=(A, B, C), the preprocessed image is other light images, and there is no need to perform image enhancement processing on the preprocessed image, and the process goes to step S25.

[0046] Step S34: performing image enhancement processing on the pre-processed image to obtain a target enhanced image.

[0047] Step S35: Let the pre-processed image be the target enhanced image.

[0048] Among them, the target enhanced image is a railway image with obvious image clarity and detail information, in which the lighting effect is small and it can be clearly seen that there are foreign objects in the railway.

[0049] For step S34, the low-light image enhancement network IAT (Illumination Adaptive Transformer) is ultra-lightweight and has fast inference speed. Therefore, the low-light image enhancement network IAT is used here to perform image enhancement processing on the preprocessed image, converting the night scene into a daytime scene, improving the image clarity and detail information, and thus reducing the negative impact of low light on the subsequent recognition process.

[0050] In this embodiment, the lighting condition corresponding to the dark light image is night. By judging the size of the three-channel mean I(i)=(a, b, c) and the first threshold T=(A, B, C), the two lighting conditions of night and other lighting can be distinguished.

[0051] In this embodiment, the above process mainly solves the problem of difficult identification of foreign objects under low light. First, the average values ​​of the pre-processed images are calculated according to the RGB color space, and then the pre-processed images are judged as dark light images or other images according to the first threshold. If it is a dark light image, the low light image enhancement network IAT is called to convert the night scene into a daytime scene, improve the image clarity and detail information, and thus reduce the illumination changes that may have a negative impact on the performance of the segmentation prediction network 4; if it is other images, the image enhancement processing operation is omitted.

[0052] like Figure 8 As shown, it shows Figure 7 The preprocessed image shown is processed by image enhancement to obtain the target enhanced image. Figure 8 Compared to Figure 7 It is clearer and you can clearly see that there is a foreign object in the middle of the left track.

[0053] Step S4: Perform segmentation prediction on the target enhanced image to obtain a segmentation prediction result.

[0054] Combination Figure 2 and Figure 3 , determine whether the segmentation prediction network 4 has text prompt information input. If there is text prompt information, the open domain target detection model uses the initial detection box and foreign body category to initially mark the target area in the target enhanced image according to the text prompt information to obtain a preliminary detection result; then input the preliminary detection result into the target segmentation model 32 for target segmentation, and the target segmentation model 32 directly outputs the segmentation prediction result.

[0055] like Fig. 9 As shown, the segmentation prediction result output by the target segmentation model 32 is shown. In the figure, different objects are marked with multiple segmentation areas of different colors and texts corresponding to the object categories, including multiple segmentation areas corresponding to people, foreign objects and tracks. Among them, objects of the same category are marked with texts of the same category, and foreign objects of specific types that can be identified are marked with texts of specific categories, such as: the texts corresponding to the categories of people (located in the red segmentation area and the brown segmentation area in the figure) are all marked with person, and the texts corresponding to the categories of tracks (located in the yellow segmentation area in the figure) are all marked with railway; the texts corresponding to the categories of foreign objects that cannot be identified in a specific category (located in the cyan segmentation area in the figure) are all marked with object. Multiple objects of the same category are distinguished by different color labels. For example, the two different people on the left side of the figure are marked with red segmentation areas and brown segmentation areas respectively.

[0056] Step S5: Post-process the segmentation prediction result to obtain a foreign body recognition result.

[0057] In this embodiment, post-processing refers to removing the identifiers corresponding to abnormal targets outside the track area in the segmentation prediction results, and only retaining the identifiers corresponding to abnormal targets in the track area. The foreign object recognition result is the segmentation and detection result of abnormal targets in the track area of ​​the image, including files in json format, prediction label files, test result images, and other formats.

[0058] Combination Figure 3 , Fig. 9 and Fig.10 , post-process the segmentation prediction results output by the target segmentation model 32, Fig. 9 The marks of the corresponding parts of the person and track in the image are removed, so that the segmented areas corresponding to the person and track are restored to Figure 8 The corresponding area in the railway track area is finally obtained. Fig.10 In the figure, the target foreign body is marked with a blue segmentation area and text object.

[0059] like Figure 3As shown, after obtaining the foreign body recognition result, it is necessary to determine whether the foreign body recognition task has been completed; if the input multimodal data is a video frame and there are subsequent frame recognition tasks that have not been completed, it means that the task has not been completed, and the predicted foreign body recognition result is updated to the memory bank. The target segmentation model 32 will refer to the content of this memory bank for information exchange during subsequent frame recognition. If the task has been completed, the foreign body recognition result is output and the collection step is entered. In the collection step, it is determined whether the foreign body recognition result is a high-confidence result. If it is a high-confidence result, the high-confidence result is used as its target pseudo-label, and the preprocessed image and the target pseudo-label are included in the pseudo-label database. Semi-supervised technology can also be used to quickly fine-tune the target segmentation model 32 in the future to improve the utilization rate of unlabeled data.

[0060] As another example, before step S1, an adaptive segmentation network needs to be constructed first.

[0061] like Figure 5 As shown, the adaptive segmentation network includes a first branch network 1, a second branch network 2 and a third branch network 3. The first branch network 1 includes a first enhancement module 11, a first segmentation model 12 and a post-processing module 13. The second branch network 2 includes a second enhancement module 21, a second segmentation model 22 and an adaptive screening module 23. The third branch network 3 includes a third enhancement module 31, a target segmentation model 32 and a prompt encoder 33.

[0062] Secondly, after the adaptive segmentation network is constructed, it is necessary to use the training data set to train the adaptive segmentation network. The training data set includes labeled images and unlabeled images.

[0063] like Figure 5 As shown, in the first branch network 1, the first enhancement module 11 is used to perform weak image enhancement on the labeled image and strong image enhancement on the unlabeled image, respectively obtaining a first weakly enhanced image and a strongly enhanced image; the first weakly enhanced image and the strongly enhanced image are input into the first segmentation model 12 for target segmentation to obtain a first segmentation result. The first segmentation result includes a true pseudo-label 14 corresponding to the first weakly enhanced image and a first pseudo-label 15 corresponding to the strongly enhanced image, and a self-supervised loss is used between the true pseudo-label 14 and the true label 17; the post-processing module 13 is used to perform post-processing operations on the first pseudo-label 15 and the third segmentation result 36, and small segmentation results are filtered out, so as to obtain a more accurate target segmentation result 16. Among them, the post-processing here refers to the fusion of the segmentation result output by the first segmentation model 12 and the segmentation result output by the target segmentation model 32, and retains the identification corresponding to the abnormal target in the track area.

[0064] In the second branch network 2, the second enhancement module 21 is used to weakly enhance the unlabeled image to obtain a second weakly enhanced image; the second weakly enhanced image is input into the second segmentation model 22 for target segmentation to obtain a second segmentation result 24; the adaptive screening module 23 adaptively screens the second segmentation result 24 to obtain a second pseudo label 25.

[0065] In the third branch network 3, the third enhancement module 31 is used to pre-enhance the unlabeled image to obtain a pre-enhanced image; the pre-enhanced image is input into the target segmentation model 32 for target segmentation to obtain a third segmentation result 36; wherein, corresponding consistency losses are respectively used between the first pseudo label 15 and the second pseudo label 25 and the third segmentation result 36.

[0066] It should be noted that the image pre-enhancement is similar to the image enhancement process in the above step S3, and will not be described in detail here.

[0067] Furthermore, during the training process, the first segmentation model 12 receives the first weakly enhanced image and the strongly enhanced image, obtains the true pseudo label 14 corresponding to the labeled image and the first pseudo label 15 corresponding to the unlabeled image, and performs semi-supervised training on the first segmentation model 12; the second segmentation model 22 adds the second weakly enhanced image, and combines the adaptive screening strategy to obtain the second pseudo label 25 corresponding to the unlabeled image, and performs unsupervised training on the second segmentation model 22. At the same time, the parameters of the second segmentation model 22 are updated by the exponential moving average (EMA) of the parameters of the first segmentation model 12.

[0068] In this embodiment, adaptive screening adopts a dynamic threshold process class-wise GMM, which regards high-quality and low-quality pseudo-label segmentation samples as positive samples (target category) and negative samples (background), respectively, and iteratively learns the optimal parameters of the Gaussian distribution corresponding to the positive and negative samples through the maximum expectation EM algorithm. The Gaussian mixture model GMM can update the threshold of each category in each iterative training, dynamically filter out high-quality samples, and solve the pseudo-label inconsistency problem caused by the traditional hard threshold.

[0069] Among them, since the input of the second branch network 2 is an unlabeled image, the obtained second segmentation result 24 may have inaccurate predictions. By filtering the second segmentation result 24 using adaptive screening, a more reliable second pseudo label 25 can be obtained.

[0070] like Figure 5As shown, the target segmentation model 32 includes an image encoder 34 and a mask decoder 35. After the third branch network 3 inputs the unlabeled image, the third enhancement module 31 first performs image pre-enhancement; then the obtained pre-enhanced image is input to the image encoder 34 for encoding, and at the same time, the first pseudo label 15 predicted by the first segmentation model 12 is input as a mask prompt to the prompt encoder 33 (corresponding to Figure 2 The prompt detection model 41 may include a prompt encoder 33, which can encode the preprocessed image to realize the target prompt detection of the mask prompt type; finally, the mask decoder 35 decodes the encoding results of the image encoder 34 and the prompt encoder 33 to obtain the third segmentation result 36, so as to realize the direct prediction of the unlabeled image under multi-source input, and then the third segmentation result 36 is fused with the first pseudo label 15 of the first segmentation model, so as to further improve the segmentation prediction effect of the first branch network 1.

[0071] Furthermore, the total loss function of the adaptive segmentation network is: ; in, is the self-supervised loss of the first segmentation model 12, is the consistency loss between the second segmentation model 22 and the first segmentation model 12, is the consistency loss between the target segmentation model 32 and the first segmentation model 12, , and are the corresponding loss weights respectively.

[0072] Furthermore, when using the trained adaptive segmentation network to perform railway foreign object intrusion identification, corresponding steps S1-S5 are executed.

[0073] Among them, step S1 and step S2 are similar to the above-mentioned step S1 and step S2, and will not be described here. However, here, multi-source prompt information is not received, only video and / or image data is received, and corresponding preprocessing operations are performed on the video and / or image data to obtain corresponding preprocessed images.

[0074] It should be noted that the preprocessed image is an unlabeled image. Therefore, step S3-step S5 is similar to the training process of the unlabeled image mentioned above, and will not be repeated here. However, here, the image to be detected is subjected to strong image enhancement, weak image enhancement and image pre-enhancement respectively, and then target segmentation is performed, and the corresponding segmentation prediction results obtained are the first target segmentation result, the second target segmentation result and the third target segmentation result; then the first target segmentation result and the third target segmentation result are post-processed, and finally the target segmentation result 16 corresponding to the preprocessed image is obtained, which is the foreign body recognition result.

[0075] In this application, under the condition of limited labeled data, semi-supervised technology is used to achieve rapid and continuous optimization of the adaptive segmentation network. The obtained adaptive segmentation network has excellent foreign object detection capabilities and can effectively detect foreign objects in the railway track area in real time.

[0076] It should be noted that the process of constructing and training the adaptive segmentation network adopts a semi-supervised optimization algorithm, and other solutions such as self-supervised learning algorithms and pseudo-label algorithms can also be used as substitutes. If there is a problem of insufficient railway foreign object intrusion data during the training process, image generation and video generation technology can be used for data expansion. The RWKV module is used in the above-mentioned target segmentation model 32, and it can also be replaced by a new Mamba module and other solutions. The open vocabulary image target algorithm is used in the above-mentioned open domain target detection model, and it can also be replaced by other algorithms such as GroundedDINO algorithm and DetClipV3 algorithm, and the same effect can be obtained.

[0077] Compared with the prior art, this application has the following beneficial effects: By combining semi-supervised technology, video image automatic recognition technology, and image enhancement technology, the present invention can effectively detect foreign objects invading the railway track area in real time, reducing the occurrence of algorithm misdetection and missed detection.

[0078] This application uses multimodal data input to design a more efficient target segmentation model 32, uses knowledge distillation to make the target segmentation model 32 have the characteristics of high performance and light weight, and combines the prompt detection model 41 to introduce more supervision information to the target segmentation model 32, so as to realize the real-time and accurate recognition of any foreign objects in the railway track area. At the same time, based on multi-source prompt information, it can automatically, in real time and accurately identify the railway track area and the intrusion targets in the area in the video and image, improve the data processing speed, greatly reduce the cost of manual annotation, and form a valuable railway foreign object intrusion database.

[0079] In addition, this application also introduces semi-supervised segmentation technology and designs an adaptive segmentation network in combination with the target segmentation model 32. In the case of limited labeled data, semi-supervised technology is used to achieve rapid and continuous optimization of the adaptive segmentation network, which greatly improves the detection performance and generalization ability of the adaptive segmentation network.

[0080] Finally, in response to the challenge of difficult image recognition under low light, this application proposes a set of image enhancement strategies, which significantly improves the quality and detail information of low-light images, and further improves the accuracy of model recognition. The use of low-light image enhancement technology provides a new and effective solution to the dilemma of image algorithms that are prone to missed detection and false detection under low light.

[0081] In summary, this application can significantly improve the safety of railway transportation in practical applications, reduce the risk of accidents caused by foreign objects intruding, improve the utilization rate of massive railway monitoring data, and provide a set of practical and usable semi-supervised adaptive segmentation solutions based on railway scenarios, which has broad application prospects and market value.

[0082] Based on the same inventive concept, Fig.11 As shown, the present invention also provides a railway foreign body intrusion identification system, the system comprising: The data acquisition unit 101 is used to acquire multimodal data, where the multimodal data includes video and / or original railway images.

[0083] The preprocessing unit 102 is used to preprocess the video and / or the original railway image to obtain a preprocessed image.

[0084] The image enhancement unit 103 is used to perform image enhancement processing on the pre-processed image to obtain a target enhanced image.

[0085] The segmentation prediction unit 104 is used to perform image segmentation prediction on the target enhanced image to obtain a segmentation prediction result.

[0086] The post-processing unit 105 is used to perform post-processing on the segmentation prediction result to obtain a foreign object recognition result; wherein the post-processing refers to retaining the identification corresponding to the abnormal target in the track area.

[0087] In the present application, other technical features of the above-mentioned railway foreign object intrusion identification system are the same as the features disclosed in the above-mentioned method embodiment, and will not be repeated here.

[0088] Based on the same inventive concept, the present application also provides an electronic device, which includes a processor, a memory and a communication circuit, and the processor is connected to the memory and the communication circuit respectively; wherein the communication circuit is used for communication connection, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above method.

[0089] See also Fig.12 The electronic device described in the embodiment of the present application may specifically include a processor 210 and a memory 220. The memory 220 is coupled to the processor 210.

[0090] The processor 210 is used to control the operation of the electronic device. The processor 210 may also be referred to as a CPU (Central Processing Unit). The processor 210 may be an integrated circuit chip with signal processing capabilities. The processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor 210 may also be any conventional processor, etc.

[0091] The memory 220 is used to store computer programs, and may be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 220 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 220 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage terminals, flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 220 is used to store at least one program code.

[0092] The processor 210 is used to execute the computer program stored in the memory 220 to implement the methods described in the various method embodiments of the present application.

[0093] In some embodiments, the electronic device may further include: a peripheral terminal interface 230 and at least one peripheral terminal. The processor 210, the memory 220 and the peripheral terminal interface 230 may be connected via a bus or a signal line. Each peripheral terminal may be connected to the peripheral terminal interface 230 via a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes: at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260 and a power supply 270.

[0094] The peripheral terminal interface 230 may be used to connect at least one peripheral terminal related to I / O (Input / output) to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral terminal interface 230 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 210, the memory 220, and the peripheral terminal interface 230 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0095] The radio frequency circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 240 communicates with the communication network and other IoT devices through electromagnetic signals, and the radio frequency circuit 240 is the communication circuit of the electronic device. The radio frequency circuit 240 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 240 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identity module card, and the like. The radio frequency circuit 240 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 240 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0096] The display screen 250 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 250 is a touch display screen, the display screen 250 also has the ability to collect touch signals on the surface or above the surface of the display screen 250. The touch signal can be input to the processor 210 as a control signal for processing. At this time, the display screen 250 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 250 can be one, set on the front panel of the electronic device; in other embodiments, the display screen 250 can be at least two, respectively set on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 250 can be a flexible display screen, set on a curved surface or a folding surface of the electronic device. Even, the display screen 250 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0097] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 210 for processing, or input them into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0098] The power supply 270 is used to power various components in the electronic device. The power supply 270 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0099] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0100] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above method.

[0104] See also Fig.13 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0105] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0106] It can be seen that the present invention discloses a method, system, device and medium for identifying foreign objects intruding on railways, and the method includes the following steps: acquiring multimodal data, the multimodal data including multi-source prompt information, video and / or original railway images; preprocessing the video and / or original railway images to obtain preprocessed images; performing image enhancement processing on the preprocessed images to obtain target enhanced images; performing image segmentation prediction on the target enhanced images to obtain segmentation prediction results; post-processing the segmentation prediction results using a segmentation prediction network to obtain foreign object identification results; wherein post-processing refers to retaining the identifiers corresponding to abnormal targets in the track area. This method can perform fast and effective foreign object identification processing on massive amounts of unlabeled data, and can effectively detect foreign object intrusion targets in the railway track area in real time, reducing the occurrence of algorithm misdetection and missed detection.

[0107] The above are only embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent structural transformations made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are included in the protection scope of the present invention.

Claims

1. A railway foreign body intrusion identification method, characterized in that: The method comprises the following steps: Acquiring multimodal data, wherein the multimodal data includes video and / or original railway images; Preprocessing the video and / or the original railway image to obtain a preprocessed image; Performing image enhancement processing on the preprocessed image to obtain a target enhanced image; Performing image segmentation prediction on the target enhanced image to obtain a segmentation prediction result; The segmentation prediction result is post-processed to obtain a foreign object recognition result; wherein the post-processing refers to retaining the identification corresponding to the abnormal target in the track area.

2. The railway foreign object intrusion identification method according to claim 1 is characterized in that: The multimodal data also includes multi-source prompt information, and the method further includes the steps of: Constructing a segmentation prediction network, the segmentation prediction network includes a target segmentation model and a prompt detection model; wherein, when the multi-source prompt information includes text prompt information, the prompt detection model includes an open domain target detection model, the open domain target detection model performs preliminary recognition on the target enhanced image according to the text prompt information to obtain an initial detection box and a foreign object category; the target segmentation model performs target segmentation on the target enhanced image according to the multi-source prompt information, the initial detection box and the foreign object category to obtain a segmentation prediction result; The segmentation prediction network is pre-trained using a pre-training data set; wherein the target segmentation model is first used as a student model, and the reference backbone network Hiera is used as a teacher model; then, the pre-training data set is input into the teacher model, and a knowledge distillation strategy is adopted for the student model to extract knowledge from the teacher model to the student model.

3. The railway foreign object intrusion identification method according to claim 2 is characterized in that: The total loss function of the segmentation prediction network is: ； in, and They are the weights corresponding to the mask loss and the detection box loss respectively; The mask loss between the true mask and the predicted mask output by the target segmentation model; The detection box loss between the real detection box and the predicted detection box output by the target segmentation model; The mask loss and the detection box loss They are: ； ； in, represents the mask cross entropy loss; represents the masked Dice loss; and They are respectively the mask cross entropy loss and masked Dice loss The corresponding weights; Indicates the Smooth L1 loss of the detection box; represents the category cross entropy loss; and They are the detection box Smooth L1 loss and categorical cross entropy loss The corresponding weight.

4. The railway foreign object intrusion identification method according to claim 1 is characterized in that: The method of performing image enhancement processing on the pre-processed image to obtain a target enhanced image comprises the steps of: Performing image illumination clustering on the preprocessed image to obtain a three-channel mean; The image illumination clustering refers to performing mean calculation on the image according to the RGB color space; Determine whether the three-channel mean exceeds a first threshold, where the first threshold is a critical value for distinguishing the illumination condition corresponding to the preprocessed image; If the mean of the three channels does not exceed the first threshold, the pre-processed image is a dark light image, and the image enhancement process is performed on the pre-processed image to obtain a target enhanced image; If the three-channel mean exceeds the first threshold, the preprocessed image is another light image, and there is no need to perform the image enhancement processing on the preprocessed image, and the preprocessed image is directly made the target enhanced image.

5. The railway foreign object intrusion identification method according to claim 1 is characterized in that: The method further comprises the steps of: Constructing an adaptive segmentation network, the adaptive segmentation network comprising a first branch network, a second branch network and a third branch network, wherein the first branch network, the second branch network and the third branch network are respectively used to perform strong image enhancement, weak image enhancement and image pre-enhancement on the preprocessed image, and then perform target segmentation to obtain a first target segmentation result, a second target segmentation result and a third target segmentation result respectively; Then the first branch network is also used to post-process the first target segmentation result and the third target segmentation result to obtain the foreign body recognition result.

6. The railway foreign object intrusion identification method according to claim 5 is characterized in that: The method further comprises the steps of: Using a training data set to train the adaptive segmentation network, the training data set includes labeled images and unlabeled images; Among them, in the first branch network, the labeled image is weakly enhanced, and the unlabeled image is strongly enhanced to obtain a first weakly enhanced image and a strongly enhanced image respectively; the first weakly enhanced image and the strongly enhanced image are input into the first segmentation model for target segmentation to obtain a first segmentation result, and the first segmentation result includes a true pseudo label corresponding to the first weakly enhanced image and a first pseudo label corresponding to the strongly enhanced image; a self-supervised loss is used between the true pseudo label and the true label, and the first pseudo label is used for post-processing with the third segmentation result output by the third branch network to obtain a target segmentation result corresponding to the unlabeled image; In the second branch network, weakly enhance the unlabeled image to obtain a second weakly enhanced image; input the second weakly enhanced image into a second segmentation model to perform target segmentation to obtain a second segmentation result; and adaptively screen the second segmentation result to obtain a second pseudo label; In the third branch network, the unlabeled image is pre-enhanced to obtain a pre-enhanced image; the pre-enhanced image is input into the target segmentation model to perform target segmentation to obtain a third segmentation result; wherein, the first pseudo-label and the second pseudo-label and the third segmentation result respectively use corresponding consistency loss functions.

7. The railway foreign object intrusion identification method according to claim 6 is characterized in that: The total loss function of the adaptive segmentation network is: ； in, is the supervised loss of the first segmentation model, is the consistency loss between the second segmentation model and the first segmentation model, is the consistency loss between the target segmentation model and the first segmentation model, , and are the corresponding loss weights respectively.

8. A railway foreign object intrusion identification system, characterized in that: The system comprises: A data acquisition unit, used to acquire multimodal data, wherein the multimodal data includes multi-source prompt information, video and / or original railway image; A preprocessing unit, used for preprocessing the video and / or the original railway image to obtain a preprocessed image; An image enhancement unit, used for performing image enhancement processing on the preprocessed image to obtain a target enhanced image; A segmentation prediction unit, used to perform image segmentation prediction on the target enhanced image to obtain a segmentation prediction result; A post-processing unit is used to post-process the segmentation prediction result to obtain a foreign object recognition result; wherein the post-processing refers to retaining the identification corresponding to the abnormal target in the track area.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images