Foreign object intrusion detection method and system for rail transit environment based on visual model

Abstract

The present invention belongs to the field of rail transit technology, and discloses a method and system for detecting foreign body intrusion in a rail transit environment based on a visual model. The method first extracts suspicious area sub-images from the original image through preprocessing, and uses an image encoder to extract coarse-grained global feature vectors and fine-grained local feature vectors respectively; at the same time, text prompt information matching the image content is generated and the corresponding text feature vector is extracted; by calculating the multi-level similarity between the image feature vector and the text feature vector, a comprehensive anomaly score is obtained by combining their respective weights; finally, the distribution of anomaly scores is dynamically modeled based on a mixed Gaussian model to realize foreign body intrusion judgment. The present invention integrates multimodal information of images and texts, improves the detection accuracy of foreign bodies in complex scenes through multi-scale feature association, and adopts an adaptive threshold strategy to enhance dynamic environment adaptability, effectively solving the problems of high false detection rate and insufficient generalization ability of traditional methods in complex railway backgrounds.

Description

Technical Field

[0001] The present invention relates to the field of rail transit technology, and in particular to a method and system for detecting foreign body intrusion in a rail transit environment based on a visual model. Background Art

[0002] In recent years, my country's high-speed railway network has rapidly developed, with operating mileage continuing to grow, placing higher demands on environmental safety monitoring along the lines. The intrusion of foreign objects (such as gravel, branches, and lost tools) along railway lines can cause major safety incidents, necessitating an urgent need for efficient and reliable detection methods. Traditional detection methods rely primarily on fixed sensors or manual inspections, which suffer from large blind spots and poor environmental adaptability. With the development of deep learning technology, vision-based detection methods have gradually become mainstream, but they still face numerous challenges: supervised learning methods rely on large amounts of labeled data and struggle to cover the complex and diverse types of foreign objects; unsupervised methods rely on autoencoder reconstruction errors, resulting in high false alarm rates in dynamic lighting and complex background environments; existing large-scale visual models lack optimization for railway scenarios, making it difficult to balance the correlation between global environmental information and local foreign object characteristics. Furthermore, existing technologies often use fixed thresholds to determine anomalies, which cannot adapt to the dynamic changes in train operation scenarios. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides a method and system for detecting foreign object intrusion in a rail transit environment based on a visual model.

[0004] In a first aspect, the present invention provides a method for detecting foreign body intrusion in a rail transit environment based on a visual model, comprising the following steps:

[0005] S1. Preprocessing the original image captured by the high-speed railway onboard camera to identify and extract suspicious areas containing potential foreign objects, thereby obtaining the original complete image and at least one suspicious area sub-image;

[0006] S2. Input the original complete image and the suspicious area sub-image into an image encoder to extract a coarse-grained global feature vector and a fine-grained local feature vector respectively;

[0007] S3, generating corresponding text prompt information for the original complete image and the suspicious area sub-image respectively, and extracting text feature vectors through a text encoder;

[0008] S4, calculating the multi-level similarity between the coarse-grained global feature vector and the complete image text feature vector, and the multi-level similarity between the fine-grained local feature vector and the suspicious area text feature vector;

[0009] S5. Perform weighted fusion on the two similarities according to the preset weights to obtain a comprehensive anomaly score;

[0010] S6. Perform distribution modeling on the comprehensive abnormality score based on a dynamic threshold model to determine whether there is an intruding foreign object in the image.

[0011] Optionally, the multi-level similarity calculation in S4 includes:

[0012] At the 6th, 9th and 12th layers of the image encoder, the first-level feature vector, the second-level feature vector and the third-level feature vector of the complete image and the corresponding-level feature vector of the suspicious region sub-image are extracted respectively;

[0013] Calculate the similarity Sim between the feature vectors of each level of the complete image and the feature vector of the complete image text Di , the formula is:

[0014] ;

[0015] in, represents the image feature vector output by the kth layer of the image encoder, where k∈{6,9,12} represents the network depth level, The complete image text feature vector representing the corresponding level is generated by the text prompt information containing the railway scene description;

[0016] Calculate the similarity Sim between the feature vectors of each level of the suspicious area sub-image and the feature vector of the suspicious area text Di j , the formula is:

[0017] ;

[0018] in, The k-th layer image feature vector representing the suspicious area sub-image, The text feature vector representing the suspicious area at the corresponding level is generated by the text prompt information containing the feature description of the foreign object.

[0019] Optionally, the weighted fusion in S5 satisfies:

[0020] ;

[0021] Among them, An_score represents the comprehensive anomaly score, represents the complete image anomaly score, defined as ; Represents the abnormal score of the suspicious area sub-graph, defined as , α and β are preset weight coefficients, satisfying α+β=1.

[0022] Optionally, the dynamic threshold model in S6 is a mixed Gaussian model, and its probability density function is:

[0023] ;

[0024] Where x is the input comprehensive anomaly score vector, μ is the mean vector, Σ is the covariance matrix, n is the dimension of x, T represents the transpose sign, and π is the circumference of the circle.

[0025] Optionally, the Gaussian mixture model optimizes parameters by an expectation maximization algorithm and iteratively calculates the sample x i The posterior probability γ(z ij )for:

[0026] ;

[0027] Where K is the number of Gaussian distributions, π j is the mixing coefficient of the jth distribution, and satisfies , Represents the sample point x i The probability density value under the j-th Gaussian distribution, Represents the sample point x i The probability density value under the k-th Gaussian distribution.

[0028] Optionally, the model parameters are updated to satisfy:

[0029] ;

[0030] ;

[0031] ;

[0032] Where N is the total number of training samples, μ j is the updated j-th Gaussian distribution mean vector, π j is the updated mixing coefficient, Σ j is the updated covariance matrix, and T represents the transposed sign.

[0033] Optionally, the image encoder adopts a hierarchical feature extraction structure, and the image feature vector I output by the kth layer is fk satisfy:

[0034] ;

[0035] in, represents the k-th layer network of the encoder, is the image feature vector output by the previous layer.

[0036] Optionally, the suspicious area subgraph is generated to satisfy:

[0037] ;

[0038] Among them, S is the image segmentation model, D i represents the original input image, is the jth suspicious area sub-graph output.

[0039] In a second aspect, the present invention further provides a rail transit environment foreign body intrusion detection system based on a visual model, comprising:

[0040] A preprocessing module is used to preprocess the original image captured by the high-speed railway onboard camera, identify and extract suspicious areas containing potential foreign objects, and obtain the original complete image and at least one suspicious area sub-image;

[0041] A first extraction module is configured to input the original complete image and the suspicious region sub-image into an image encoder, and extract a coarse-grained global feature vector and a fine-grained local feature vector respectively;

[0042] A second extraction module is used to generate corresponding text prompt information for the original complete image and the suspicious area sub-image respectively, and extract text feature vectors through a text encoder;

[0043] A calculation module, configured to calculate the multi-level similarity between the coarse-grained global feature vector and the complete image text feature vector, and the multi-level similarity between the fine-grained local feature vector and the suspicious area text feature vector;

[0044] The fusion module is used to perform weighted fusion of the two similarities according to the preset weights to obtain a comprehensive anomaly score;

[0045] The determination module is used to perform distribution modeling on the comprehensive anomaly score based on a dynamic threshold model to determine whether there is an intruding foreign object in the image.

[0046] The present invention has the following technical effects:

[0047] The present invention significantly improves the comprehensiveness and accuracy of foreign object detection in complex railway scenarios by fusing the multimodal association of coarse-grained global feature vectors and fine-grained local feature vectors, combined with the semantic information generated by text prompt information. Based on the multi-level feature vectors extracted by the image encoder at different network depths, similarity matching is performed with the corresponding text feature vectors, enhancing the synergy between global environmental anomaly perception and local foreign object detail capture, avoiding the missed detection of tiny foreign objects or large-scale anomalies at a single feature level. Preset weights are used to weightedly fuse the complete image anomaly score and the suspicious area sub-graph anomaly score to optimize the sensitivity of foreign object detection at different scales, taking into account the overall environmental stability of the track and the collaborative discrimination of local area mutation characteristics. The comprehensive anomaly score is dynamically distributed modeled using a mixed Gaussian model, and the judgment threshold is adaptively adjusted based on real-time data, effectively adapting to complex environmental fluctuations such as lighting changes and weather interference, reducing the risk of misjudgment caused by traditional fixed thresholds. The pre-screening mechanism of the suspicious area sub-graph is combined with the image segmentation model to reduce the interference feature extraction of non-critical areas and improve the pertinence of local foreign object analysis. The scene description feature vectors and foreign object description feature vectors generated by the text encoder provide semantic guidance for image content, enhancing the ability to differentiate between normal scenes and abnormal areas. In an unsupervised framework, training solely on normal samples, the system discovers potential abnormal patterns through cross-modal feature comparison, addressing the practical detection challenges of scarce foreign object samples and unknown types in railway scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A schematic flow chart of a method for detecting foreign body intrusion in a rail transit environment based on a visual model according to an embodiment of the present invention;

[0050] Figure 2 A schematic structural diagram of a rail transit environment foreign object intrusion detection system based on a visual model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0052] Figure 1 The schematic flow chart of the method for detecting foreign body intrusion in a rail transit environment based on a visual model provided by an embodiment of the present invention includes the following steps:

[0053] S1. Preprocessing the original image captured by the high-speed railway onboard camera to identify and extract suspicious areas containing potential foreign objects, thereby obtaining the original complete image and at least one suspicious area sub-image;

[0054] In the preprocessing phase, an image segmentation model is used to perform preliminary image analysis on raw images captured by high-speed railway onboard cameras, identifying and extracting suspicious areas containing potential foreign objects. Based on deep learning methods, the image segmentation model locates candidate anomalies through multi-level feature fusion and an attention mechanism, generating the original complete image and multiple sub-images of suspicious areas. This process uses a sliding window and region clustering algorithm to select candidate boxes that meet the size and location characteristics of foreign objects, reducing computational redundancy in subsequent processing. The preprocessed image data is divided into a complete scene view and a local detail view, respectively representing the overall track environment and the microscopic characteristics of suspected foreign objects.

[0055] S2, input the original complete image and the suspicious area sub-image into the image encoder to extract the coarse-grained global feature vector and the fine-grained local feature vector respectively;

[0056] The original full image and the suspicious region sub-image are fed into an image encoder in parallel for feature vector extraction. The image encoder employs a hierarchical feature extraction architecture: the bottom layer captures basic visual elements such as lines and textures, the middle layer aggregates spatial context, and the deep layer extracts global semantic features. The encoder processes the original full image to produce a coarse-grained global feature vector representing macroscopic scene information such as track alignment and catenary layout. The suspicious region sub-image is then fed into the same encoder to produce fine-grained local feature vectors, focusing on subtle differences such as foreign object edge morphology and material reflectivity. During feature vector extraction, the network uses residual connections to preserve cross-layer detail, preventing the loss of local information due to over-abstraction of deep features.

[0057] S3, generate corresponding text prompt information for the original complete image and the suspicious area sub-image respectively, and extract the text feature vector through the text encoder;

[0058] The text prompt generation module dynamically constructs semantic descriptions based on image content. The full image corresponds to a text template representing the normal state of a railway scene, such as "unobstructed straight rails" or "normally operating overhead contact line equipment." The suspicious area subimage corresponds to a foreign object feature description template, such as "foreign objects hanging above the track" or "scattered obstacles on the track surface." The text prompts are converted into embedding vectors using a learnable encoder, sharing a high-dimensional semantic space with the image features. The text encoder employs a sequence modeling approach based on an attention mechanism to capture semantic associations between keywords and generate a text feature vector aligned with the image feature vector. The text encoder also employs Context Optimization for Prompting (CoOP), optimizing text feature generation through a learnable prompt vector.

[0059] S4, calculating the multi-level similarity between the coarse-grained global feature vector and the complete image text feature vector, and the multi-level similarity between the fine-grained local feature vector and the suspicious area text feature vector;

[0060] The multi-level similarity calculation module uses matrix operations to measure the degree of match between image feature vectors and text feature vectors. Cosine similarity is calculated between the coarse-grained global feature vector and the scene description text vector, reflecting the overall consistency of the current scene with the normal railway environment. Similarly, similarity is calculated between the fine-grained local feature vector and the foreign object description text vector, quantifying the degree to which the suspicious area deviates from normal conditions. The similarity calculation results are normalized and mapped to anomaly probability values. The anomaly probability of the full image reflects the risk of anomalies in the larger environment, while the anomaly probability of a local sub-image indicates the likelihood of foreign objects in a specific area.

[0061] S5. Perform weighted fusion on the two similarities according to the preset weights to obtain a comprehensive anomaly score;

[0062] S6. Based on the dynamic threshold model, the comprehensive anomaly score is distributed modeled to determine whether there is an intruding foreign object in the image.

[0063] The weighted fusion module presets weighting parameters based on the characteristics of railway monitoring scenarios. The full image anomaly score focuses on detecting track structural integrity, while the suspicious area sub-image anomaly score focuses on identifying the risk of foreign object intrusion. The fused comprehensive anomaly score integrates multi-scale detection results, avoiding the one-sidedness of judgments at a single feature level. The dynamic threshold model constructs a probability density function based on the anomaly score distribution of historical normal samples. Model parameters are updated through an online learning mechanism, adapting in real time to score fluctuations caused by environmental factors such as diurnal lighting changes and seasonal vegetation differences. When the comprehensive anomaly score exceeds the confidence interval of the normal distribution, a foreign object intrusion alarm signal is triggered, and the location information of the suspicious area is simultaneously output for manual review.

[0064] This method enhances the distinguishability between normal and abnormal conditions through collaborative analysis of image and text modalities, overcoming the false positives often associated with traditional visual inspection methods in complex backgrounds. A hierarchical feature extraction structure balances global environmental stability assessment with the precise localization of local foreign objects. A dynamic threshold mechanism enhances system robustness under varying climatic conditions and operating time periods. A pre-screening strategy for suspicious areas effectively reduces computational resource consumption, meets the real-time processing requirements of onboard equipment, and provides reliable technical support for the safe operation of high-speed railways.

[0065] In some embodiments, the multi-level similarity calculation in S4 includes:

[0066] At the 6th, 9th and 12th layers of the image encoder, the first-level feature vector, the second-level feature vector and the third-level feature vector of the complete image and the corresponding-level feature vector of the suspicious region sub-image are extracted respectively;

[0067] Calculate the similarity Sim between the feature vectors of each level of the complete image and the feature vector of the complete image text Di , the formula is:

[0068] ;

[0069] in, represents the image feature vector output by the kth layer of the image encoder, where k∈{6,9,12} represents the network depth level, The complete image text feature vector representing the corresponding level is generated by the text prompt information containing the railway scene description;

[0070] Calculate the similarity Sim between the feature vectors of each level of the suspicious area sub-image and the feature vector of the suspicious area text Di j , the formula is:

[0071] ;

[0072] in, The k-th layer image feature vector representing the suspicious area sub-image, The text feature vector representing the suspicious area at the corresponding level is generated by the text prompt information containing the feature description of the foreign object.

[0073] In order to make full use of the feature vectors of different levels extracted by the image encoder, this method extracts the image feature vectors at three different depths (6, 9, and 12 layers respectively). For the convenience of representation, the entire image encoder is divided into three parts, namely 、 and , for the entire input image, the image feature vectors at different levels are expressed as 、 and , but:

[0074] ,

[0075] ,

[0076] ;

[0077] Among them, D i is the i-th complete original image. For the sub-image that may contain abnormal objects, the image feature vectors at three different levels are:

[0078] ,

[0079] ,

[0080] ;

[0081] Since the image encoder has a strong feature extraction capability, the original complete image and the suspicious area sub-image can share the same image encoder. ( 、 and ).

[0082] For two different text prompt information, they are defined as and , and the corresponding generated text feature vectors are and .

[0083] During the multi-level similarity calculation process, the image encoder extracts image feature vectors for the full image and the suspicious region subimage at the 6th, 9th, and 12th network depths, respectively. These three layers correspond to different stages of feature abstraction in the image encoder: the image feature vector output by the 6th layer primarily contains low-level geometric structure information such as track edges and catenary supports; the image feature vector output by the 9th layer incorporates local texture and spatial context (for example, the relationship between the reflective properties of the rail surface and the surrounding vegetation); and the image feature vector output by the 12th layer represents semantically abstract information about the global scene (such as the layout of the track direction and the background skyline). The image feature vectors at each layer undergo dimensionality reduction and normalization through fully connected layers to ensure dimensionality alignment with the text feature vector.

[0084] For a complete image, its multi-level similarity calculation is implemented by the following formula:

[0085] ;

[0086] in, Represents the image feature vector output by the 6th, 9th, and 12th layers of the image encoder, is the text feature vector corresponding to the complete image. This text feature vector is generated from a predefined railway scene description template, such as "straight and unobstructed rails" or "normally operating contact network equipment," and is converted into a high-dimensional vector through a text encoder. The dot product operation in the formula measures the projection consistency of the image feature vector and the text feature vector in the semantic space, and the summation and averaging operation balances the contribution weights of feature vectors at different levels. For example, the feature vector at the sixth level is sensitive to the details of the track edge, and the feature vector at the 12th level is sensitive to the overall scene layout. The mean calculation can avoid misjudgment of a single level due to environmental interference (such as local reflections or shadows).

[0087] For the suspicious area subgraph, the similarity calculation formula is:

[0088] ;

[0089] in, Represents the image feature vector output by the suspicious area sub-image in the same three network layers, A text feature vector describing the characteristics of a foreign object (e.g., "hanging foreign objects" or "scattered obstacles"). The dot product operation reflects the degree of match between the suspicious region sub-image area and the semantic description of the foreign object, while the mean calculation integrates the detection results at different levels. For example, the sixth-layer feature vector captures the sharpness of the foreign object's edge, the ninth-layer feature vector analyzes the texture contrast between the foreign object's surface and the surrounding environment, and the twelfth-layer feature vector assesses the spatial disharmony between the foreign object and the overall scene. If the suspicious domain sub-image area is a flying bird or fallen leaves in a normal environment, the deep features will have a low match with the text description, thereby reducing the probability of false alarms.

[0090] The dot product operation in the formula essentially maps the image feature vector and the text feature vector to the same semantic space, and measures the similarity by the cosine of the angle between the vectors. If the image content is highly consistent with the text description, the vector directions tend to be consistent, and the dot product result approaches the maximum value; if there is a significant difference, the dot product value decreases. The averaging operation avoids the deviation of the feature vectors of a single layer due to the different network depths: shallow feature vectors (such as the 6th layer) have high resolution but are easily affected by noise, while deep feature vectors (such as the 12th layer) are highly abstract but may ignore local details. For example, for smaller foreign objects, the high-resolution feature vectors of the 6th layer can capture their contours, while the global feature vectors of the 12th layer may weaken their response due to the large receptive field. The mean calculation can balance the contributions of the two.

[0091] In implementation, the image encoder and text encoder are jointly optimized through contrastive learning. During the training phase, the similarity between the full image of a normal sample and the scene description text is maximized, while the similarity between the normal region subimage and the foreign object description text is minimized. In this invention, only images of normal samples are required during the training phase, eliminating the need for detailed annotation of abnormal objects or the introduction of abnormal samples. This unsupervised training approach significantly reduces the cost of data collection and annotation, while also avoiding performance degradation caused by inaccurate or omitted annotations. This makes the invention more efficient in practical applications. During the inference phase, an anomaly score is dynamically generated based on the degree of similarity deviation. For example, when a foreign object appears suspended on the track, the similarity between the coarse-grained global feature vector of the full image and the scene description text decreases (reflecting the overall scene anomaly), while the similarity between the fine-grained local feature vector of the suspicious region subimage and the foreign object description text increases (confirming the presence of a foreign object). By comparing hierarchical features from multiple angles, the model can distinguish between real foreign objects and environmental interference (such as temporary flying birds or dynamic shadows), improving detection robustness in complex scenes.

[0092] This method enhances the detection of foreign objects of varying scales, such as large-scale landslides and small gravel, by fusing multi-layered image feature vectors with cross-modal text semantics. Dynamic weight allocation and a layered feature complementarity mechanism effectively address complex conditions such as changing lighting and weather disturbances, avoiding the limitations of traditional single-layer feature detection and providing a highly accurate, adaptive solution for railway environmental safety monitoring.

[0093] In some implementations, the weighted fusion in S5 satisfies:

[0094] ;

[0095] Among them, An_score represents the comprehensive anomaly score, represents the complete image anomaly score, defined as ; Represents the abnormal score of the suspicious area sub-graph, defined as , α and β are preset weight coefficients, satisfying α+β=1.

[0096] In the anomaly score fusion stage, the anomaly score of the complete image and the anomaly score of the suspicious area sub-image are weighted by the preset weights. The anomaly score of the complete image is defined as the complement of the mean of the multi-level similarity of the complete image, and the formula is expressed as: .

[0097] in, It is the mean similarity between the 6th, 9th, and 12th layer features of the complete image and the scene description text features, reflecting the consistency between the current scene and the normal railway environment.

[0098] The abnormal score of the suspicious region sub-graph is defined as the complement of the mean similarity of the suspicious region sub-graph: .in, The weighted fusion formula combines the two types of anomaly scores (the complete image anomaly score and the suspicious region subimage anomaly score) through a linear combination to obtain the comprehensive anomaly score An_score.

[0099] Here, α and β are preset weight coefficients, satisfying α + β = 1. The physical significance of this formula lies in dynamically balancing the sensitivity of global environmental anomalies with that of local foreign object detection by adjusting the ratio of α and β. For example, when monitoring straight track sections, if it is necessary to prioritize the identification of scattered foreign objects on the track surface, increasing β will increase the contribution of the anomaly score of the suspicious area sub-image; in curved sections or bridge sections, to prevent the risk of landslides or structural deformation, increasing α will strengthen the weight of the anomaly score of the complete image.

[0100] in the formula The calculation is based on the degree of similarity attenuation between the complete image and the normal scene text description. When the overall track environment deviates (such as large-scale contact network shedding), Significantly reduced, Increased, triggering a global abnormal warning. The matching degree between the suspicious area sub-image and the description of the foreign object reflects the probability of local abnormality. For example, when there is a hanging foreign object in the suspicious area sub-image area, reduce, Elevated, indicating local invasion risk.

[0101] The weight coefficients α and β are optimized based on the actual monitoring scenario requirements. During the training phase, the optimal weight combination is determined through grid search and cross-validation. For example, in areas with complex backgrounds and multiple curves, α=0.6 and β=0.4 are set to suppress local false alarms caused by swaying vegetation. In linear track monitoring, α=0.4 and β=0.6 are used to enhance the detection of small foreign objects on the track surface. The inference phase supports dynamic weight adjustment. For example, in rainy and foggy weather, α can be temporarily increased to compensate for the ambiguity in local feature extraction caused by reduced visibility.

[0102] The normalization constraint (α + β = 1) in the formula ensures a stable range for the composite anomaly score, preventing drift in the score scale due to uneven weight distribution. For example, when α = 0.7 and β = 0.3, the composite score is more likely to reflect large-scale anomalies such as catenary breakage; while when α = 0.3 and β = 0.7, it is more sensitive to small foreign objects such as track surface debris and loose bolts. The fused composite anomaly score is modeled using a mixed Gaussian model to dynamically distinguish the score distribution boundaries between normal and abnormal states, improving robustness in complex environments.

[0103] This method achieves flexible adaptation of detection strategies through a configurable weighting mechanism. In tunnel monitoring, global weighting prioritizes mountain structural stability assessments, while local weighting monitors track surface rockfalls. In elevated bridge sections, global weighting focuses on bridge deformation trends, while local weighting detects loose or corroded bolts. The dynamic adjustment of weight coefficients enhances the system's adaptability to varying line characteristics and operating conditions, providing multi-dimensional, refined anomaly detection support for safe railway operations.

[0104] In some embodiments, the dynamic threshold model in S6 is a mixed Gaussian model, and its probability density function is:

[0105] ;

[0106] Where x is the input comprehensive anomaly score vector, μ is the mean vector, Σ is the covariance matrix, n is the dimension of x, T represents the transpose sign, and π is the circumference of the circle. The input variable x of the mixed Gaussian model is essentially an extended form of the weighted comprehensive anomaly score (An_score) mentioned above. In the single-frame detection scenario, x is a scalar, that is, x=An_score, which directly inherits the linear fusion result of the above An_score; in multi-frame time series analysis or multi-region joint monitoring scenarios, x is expanded to a vector, for example, x=[An_score t , An_score t−1 , An_score 轨道 , An_score 接触网 ], where An_score t An_score is the comprehensive anomaly score at the current moment. t−1 An_score is the comprehensive anomaly score of the previous moment. 轨道 An_score is the independent anomaly score of the track area. 接触网 An independent anomaly score for the catenary area is obtained by fusing the output of the aforementioned An_score across time and space. The core difference between the two lies in their application granularity: An_score focuses on real-time anomaly scoring for a single frame, while x improves robustness in complex scenarios through probabilistic modeling. Both share the same anomaly score generation logic.

[0107] This formula describes the difference in score distribution between normal and abnormal states through the linear superposition of multiple Gaussian distributions, where the abnormal scores of normal samples are concentrated in the low-value area, while the abnormal samples are distributed in the high-value area.

[0108] During the model training phase, the parameters of the mixed Gaussian model are initialized using the anomaly score data of historical normal samples. The expectation maximization algorithm maximizes the log-likelihood function of the sample data by iteratively optimizing the mean vector, covariance matrix, and mixing coefficients. , L is the likelihood function, K is the number of Gaussian distributions, and N is the total number of training samples. During the model training phase, only normal railway scene image data is used to optimize the network parameters. The similarity between the coarse-grained global feature vector of the complete image and the feature vector of the scene description text is maximized through a contrastive learning strategy ( ), while minimizing the similarity between the fine-grained local feature vector of the suspicious area sub-graph and the feature vector of the foreign object description text ( ). Specifically, define the contrast loss function for: ; Among them, λ1 and λ2 are balance coefficients. The parameters of the image encoder and text encoder are jointly optimized through back propagation, forcing the model to align the semantics of the global image and scene text in normal scenes, and suppress the mismatch between local areas and foreign text.

[0109] In some embodiments, in the posterior probability calculation step, the sample x i The probability γ(z ij ) is determined by the following formula:

[0110] ;

[0111] Where K is the number of Gaussian distributions, π j is the mixing coefficient of the jth distribution, and satisfies , Represents the sample point x i The probability density value under the j-th Gaussian distribution, Represents the sample point x i The probability density value under the kth Gaussian distribution. This formula calculates the probability of each sample belonging to each Gaussian distribution and is used for subsequent parameter updates.

[0112] During the parameter update process, the mean vector μ j Calculated by weighted average, the weight is the posterior probability of the sample for the jth distribution:

[0113] ;

[0114] Covariance matrix Σ j Updated to the weighted covariance of the sample deviations from the mean:

[0115] ;

[0116] Mixing coefficient π j Update to the sample proportion of the corresponding distribution:

[0117] ;

[0118] Where N is the total number of training samples, μ jis the updated j-th Gaussian distribution mean vector, π j is the updated mixing coefficient (same as the “π j is the mixing coefficient of the jth distribution" is the same parameter, that is, π j is updateable), Σ j is the updated covariance matrix, and T represents the transposed sign.

[0119] The iterative optimization process continues until the log-likelihood function converges and stable Gaussian distribution parameters are obtained.

[0120] Finally, the conditional probability of each sample is obtained according to the optimized model parameters, and then its maximum value j is obtained, that is, , argmax means finding the function γ(z ij ) obtains the maximum value of the input parameter j, that is, the i-th sample is assigned to the most likely Gaussian distribution, λ i is a classification label with a value of 1 or 2, representing whether the i-th sample is classified as normal or abnormal, respectively. In this embodiment of the present invention, the obtained anomaly scores can be assumed to follow two distributions (the anomaly scores of normal and abnormal images follow different distributions). These anomaly scores are used as input to the GMM model, which learns parameters such as the mean and variance of the two different distributions (the mean of the anomaly score distribution for normal images is smaller than the mean of the anomaly score distribution for abnormal images). For each anomaly score, the GMM model then determines the probability of it belonging to each distribution, and thus determines the category to which the anomaly score belongs, thereby determining whether the corresponding image is normal or abnormal.

[0121] In the initialization stage of the mixed Gaussian model, the comprehensive anomaly score data of historical normal samples is used to estimate the initial parameters, including the mean vector μ, the covariance matrix Σ and the mixing coefficient π j After model training is complete, an online update mechanism is supported: a sliding window is used to collect the anomaly scores of the latest normal samples, and the expectation-maximization algorithm is periodically executed to update model parameters. This dynamically adapts to anomaly score distribution shifts caused by environmental factors such as changes in light intensity and seasonal differences in vegetation cover. When the detected comprehensive anomaly score exceeds the current Gaussian distribution confidence interval, a foreign object intrusion alarm is triggered.

[0122] During the inference phase, the real-time calculated composite anomaly score is fed into a trained Gaussian mixture model to calculate the probability of belonging to each Gaussian distribution. If the probability of a sample belonging to the normal distribution (Gaussian components with low mean and small variance) falls below a preset confidence threshold, it is considered an anomaly. For example, when track visibility is reduced due to heavy rain, the anomaly score distribution of normal samples will shift to the right. The model adaptively adjusts the distribution parameters to dynamically expand the confidence interval of the normal distribution, thus avoiding false positives due to environmental interference.

[0123] This method uses probability density modeling to distinguish the dynamic boundary between normal and abnormal states. For example, in a day-night scenario, normal scores are concentrated in low-value areas during the day when there is ample sunlight. At night, the score distribution may shift slightly to the right due to the activation of supplemental lighting equipment. The mixed Gaussian model captures these distribution changes through multi-component fitting, avoiding the imbalance in detection sensitivity between day and night caused by a fixed threshold. The covariance matrix update mechanism can characterize the fluctuation characteristics of abnormal scores, such as the impact of seasonal vegetation growth on coarse-grained global eigenvectors, and reduce the risk of misjudgment by expanding the covariance range of the normal distribution.

[0124] The multi-distribution nature of the Gaussian mixture model allows for simultaneous modeling of multimodal data in complex scenarios. For example, the normal scores for straight tracks and curved areas may exhibit different distributions. The model characterizes these scores using two Gaussian components, avoiding missed detections due to insufficient fitting of a single distribution. During anomaly determination, an alarm is triggered if the score deviates from all normal distributions. This approach overcomes the traditional single Gaussian model's inability to model asymmetric distributed data and is particularly applicable to railway scenarios, where score distribution diversity arises from differences in line sections and weather changes.

[0125] In technical implementation, the number of mixed Gaussian models, K, is set based on the complexity of the historical data distribution. For linear track scenarios with a stable background, K=2 is sufficient to account for day-night variations. For complex routes with multiple curves and bridges, K=3 or higher is used to fit the normal state distribution across different sections. The model's online update mechanism regularly incorporates the latest normal sample data, dynamically revising distribution parameters to accommodate gradual environmental changes (such as vegetation growth and equipment aging) during long-term operation, ensuring the continued effectiveness of the detection system.

[0126] In some embodiments, the image encoder adopts a hierarchical feature extraction structure, and the image feature vector I output by the kth layer is fk satisfy:

[0127] ;

[0128] in, represents the k-th layer network of the encoder, is the image feature vector output by the previous layer.

[0129] This formula reveals the hierarchical transmission mechanism of feature vectors in the encoder: the kth layer performs convolution, normalization, and nonlinear activation on the input feature vector, gradually extracting higher-level semantic information. For example, the first layer (shallow layer) extracts basic visual features such as rail edges and ballast texture; the sixth layer (middle layer) aggregates the relative position of the track and the overhead line; and the twelfth layer (deep layer) captures the global semantics of the scene, such as the spatial layout of the track direction and the background skyline.

[0130] Each layer of the encoder network module is composed of a residual connection structure. The input feature vector is added to the convolution output feature vector through a skip connection to alleviate the gradient vanishing problem in deep network training. Taking the 6th layer network as an example, the input feature vector After the 3×3 convolution layer extracts local details, and then processes it through batch normalization and ReLU activation function, it is added to the original input feature vector and output This design ensures that shallow-layer detail information is preserved in the deep network, preventing the loss of local feature vectors due to increased network depth. For example, edge information of tiny foreign objects on the track surface is prominent in shallow-layer features and is transferred to the deep network through residual connections, assisting in global semantic analysis while taking into account local anomalies.

[0131] Feature vectors at different levels have different resolutions and receptive fields. Shallow feature vectors (e.g., layers 1-3) retain high spatial resolution and can accurately locate the edge contours of foreign objects. Mid-level feature vectors (e.g., layers 4-9) reduce resolution through downsampling and expand the receptive field to capture the contextual association between foreign objects and their surroundings. Deeper feature vectors (e.g., layers 10-12) further abstract and characterize the global structural stability of the scene. For example, when detecting a suspended foreign object, the sixth-level feature vector combines the local structural information of the contact network support to determine the rationality of the foreign object's hanging position. The 12th-level feature vector uses global layout analysis to identify large-scale anomalies caused by overall track deviation or landslides.

[0132] The progressive relationship between feature vector layers is optimized through parameter sharing and weight linkage. During training, the encoder synchronously updates the parameters of each network layer through backpropagation, forcing the shallow network to focus on basic feature extraction and the deep network to focus on high-level semantic modeling. For example, in normal scenarios, the deep network learns to encode patterns such as the straight track and the regular layout of the catenary into low-dimensional vectors. However, when track deformation occurs, the deep feature vector deviates significantly from the normal semantic description, triggering an increase in the complete image anomaly score.

[0133] This hierarchical structure supports multi-scale feature fusion, enhancing detection robustness in complex scenarios. During the inference phase, feature maps output by layers 6, 9, and 12 are each similarly calculated with the corresponding text descriptions, and mean fusion is used to balance the contributions of different layers. For example, when detecting gravel on the track surface, the high-resolution feature vectors of the 6th layer capture the edges of the gravel, the feature vectors of the 9th layer analyze the texture differences between the gravel and ballast, and the feature vectors of the 12th layer assess whether the gravel distribution conforms to the normal state of the track. These three elements work together to reduce missed detections or false alarms caused by misjudgments at a single layer.

[0134] In its technical implementation, the image encoder, based on the Vision Transformer architecture, divides the input image into fixed-size image blocks and generates a serialized token vector through linear embedding and positional encoding. Each network layer consists of a multi-head self-attention mechanism and a feedforward neural network, calculating the global dependencies between image blocks using self-attention weights. For example, the sixth layer uses the self-attention mechanism to strengthen the spatial correlation between the rails and the overhead line, suppressing interference from irrelevant background areas; the 12th layer models the global consistency of the track's overall orientation with the geographical environment. This architecture fully leverages the Transformer's global modeling capabilities, compensating for the shortcomings of traditional convolutional networks in capturing long-range dependencies and improving the accuracy of detecting anomalies in large-scale environments.

[0135] This method achieves a balance between preserving local details and extracting global semantics through a hierarchical progression and residual connection design. Under complex conditions such as changing lighting and partial occlusion, shallow feature vectors provide stable edge and texture information, while deep feature vectors ensure the overall rationality of the scene. The two work together to enhance the model's adaptability to unknown foreign object types and environmental interference, providing reliable technical support for high-speed railway safety monitoring.

[0136] In some implementations, the suspicious region subgraph is generated to satisfy:

[0137] ;

[0138] Among them, S is the image segmentation model, among them, D i represents the original input image, is the jth suspicious area sub-graph output.

[0139] The image segmentation model uses a pre-trained semantic segmentation architecture, extracting multi-scale image features through an encoder-decoder structure and generating pixel-level segmentation masks. The encoder uses convolutional layers and residual modules to gradually downsample, capturing semantic information about areas such as the track, catenary, and background vegetation. The decoder restores spatial resolution through upsampling and skip connections, outputting highly accurate target area boundaries.

[0140] The segmentation model uses a sliding window strategy to generate candidate regions on the original image. Each window region undergoes feature extraction and classifier analysis, calculating a probability score for its inclusion in the foreign object region. Windows with probability scores above a preset threshold are retained as candidate regions, and redundant candidate frames with high overlap are removed using a non-maximum suppression algorithm. For example, in scenarios with scattered gravel on the track surface, the model identifies candidate regions based on local texture contrast and edge irregularities. In scenarios with suspended foreign objects, the model locates abnormal areas based on structural continuity interruptions in the contact network support.

[0141] During the candidate region post-processing stage, the segmentation results are screened based on prior knowledge. Typical characteristics of railway foreign object intrusion (such as object size range, aspect ratio threshold, and relative position to the track) are used to filter out falsely detected areas that do not meet the criteria. For example, transient interference objects such as flying birds are excluded due to being too small or positioned off the track centerline. However, areas that meet the preset foreign object characteristics are retained as suspicious sub-images. The screened sub-images are cropped and normalized to ensure consistent size for the input image encoder.

[0142] The segmentation model S in the formula uses a segmentation network based on an attention mechanism. Its encoder outputs multi-level feature maps, and the decoder uses a channel attention module to enhance the salient features of the foreign object area. For example, in the encoder's deep feature maps, spatial attention weights are used to focus on the spatial location of the foreign object hanging above the track; in the shallow feature maps, channel attention is used to enhance the contrast between the foreign object's edge and the background. The multi-scale feature fusion module stitches the feature maps from different levels and generates the final segmentation mask through convolutional layers to accurately delineate the suspicious area.

[0143] The advantage of this method lies in its dual filtering mechanism, combining semantic segmentation with a priori rules. The segmentation model learns the common characteristics of foreign object areas in railway scenes (such as irregular shapes and material differences from the background) through end-to-end training, while rule-based filtering uses domain knowledge to eliminate false detections that do not conform to physical laws. For example, the model may mistakenly identify the shadow of a ballast pile as a foreign object, but through aspect ratio and position constraints, it can be identified as belonging to the normal trackbed structure, thus avoiding false positives. The dynamic threshold adjustment mechanism adaptively adjusts the segmentation probability threshold based on the ambient light intensity, appropriately lowering the threshold in low-light conditions to compensate for feature extraction ambiguity.

[0144] In its technical implementation, the segmentation model supports multi-task joint optimization, simultaneously outputting a segmentation mask and confidence score for the foreign object region. The confidence score reflects the model's confidence in the presence of a foreign object in the current region and is used for subsequent sub-image sorting and selective processing. For example, in real-time monitoring, only the top five confidence-ranked sub-images undergo fine-grained feature extraction and anomaly score calculation, significantly reducing computational resource consumption. The confidence score works synergistically with the anomaly score from multi-level similarity calculation to further improve the efficiency and accuracy of the detection system.

[0145] This method significantly reduces the computational effort required for subsequent image encoding and cross-modal matching by accurately pre-screening suspicious areas. For example, local feature extraction is performed on only a few high-confidence sub-images within the complete image, avoiding indiscriminate processing of all regions of the full image. Simultaneously, the segmentation model's attention mechanism enhances the salience of foreign object regions, allowing local feature extraction to focus more on key details and improving the discriminative power of fine-grained similarity calculations. This design not only ensures detection accuracy but also meets the stringent real-time and computational efficiency requirements of onboard equipment, providing efficient and reliable technical support for the safe operation of high-speed railways.

[0146] In order to conduct experimental research and comparison, the present invention collects and organizes a real high-speed railway abnormal image dataset. First, videos of trains running on different sections of high-speed railways are taken. Then 2,000 images are extracted, of which 1,600 images do not contain abnormal objects as a training set; 400 images are used as a test set, of which 200 do not contain abnormal objects and 200 contain abnormal objects, and the dataset is named RailwayAnomaly. When evaluating the detection performance of the model on this dataset, the general image anomaly detection standards are followed, and precision and recall are used as key indicators. Precision indicates the proportion of instances that the model predicts to be positive samples that are actually positive samples. The recall rate indicates the proportion of actual positive samples that the model correctly identifies as positive samples. In order to provide a comprehensive evaluation, we use the F1 score, which combines precision and recall into a single indicator:

[0147] .

[0148] In addition, AUROC is used as another evaluation metric. It evaluates the overall performance of a model at different thresholds by calculating the area under the ROC curve. The ROC curve shows the relationship between the true positive rate (TPR) and the false positive rate (FPR) of a model at different threshold settings. AUROC values ​​range from 0 to 1, with higher values ​​indicating better classification performance. A perfect classifier has an AUROC value of 1, while a random guessing classifier has an AUROC value close to 0.5. Therefore, AUROC can intuitively reflect the classification effect of a model and facilitate quantitative comparison of model performance.

[0149] The present invention and other existing abnormal object detection methods are compared in experiments on the RailwayAnomaly dataset. The experimental results are shown in Table 1.

[0150] Table 1 - Comparison of experimental results of different models

[0151]

[0152] Table 1 shows the results of different models on the RailwayAnomaly dataset, among which the method RailWay-VLM proposed in the present invention achieved the best level in both F1-score and AUROC score, which are 0.9267 and 0.9725 respectively. The reason why the present invention is superior to other methods is that although other methods have achieved good results in other fields, they cannot achieve good results in the railway operation environment. Abnormal object detection based on the railway environment is carried out in a real environment with complex backgrounds and drastic changes in weather and lighting. Therefore, methods such as PaDim cannot effectively complete the task of abnormal object detection in such a complex and changeable environment. There are two reasons why RailWay-VLM can achieve good results. First, the use of the SAM segmentation network to extract suspicious abnormal areas reduces the difficulty of detection; second, the multimodal-based method proposed in the present invention not only improves the detection accuracy, but also reduces the false alarm rate.

[0153] This paper also validates the effectiveness of the various modules proposed in RailWay-VLM. "Raw" represents the use of only the basic image encoder and a standard text encoder; "Feat3" represents the output of features at all three layers of the image encoder at different depths, and their similarity calculations with text features; and "Prompt Learner" represents the use of a learnable text encoder. The experimental results are shown in Table 2.

[0154] Table 2 - Effects of different modules in the model

[0155]

[0156] The experimental results in Table 2 show the performance improvement of the model when different modules are added to the basic model. The experiments in the table above show that both Feat3 and Prompt Learner modules improve the performance of the model.

[0157] When preprocessing an image, it's necessary to identify areas in the image that may contain abnormal objects. This can be achieved using either traditional image processing methods (Selective Search) or a large-scale model-based approach (SAM). This paper compares these two methods based on the number of candidate regions and their performance on the Railway Anomaly Dataset. The experimental results are shown in Table 3.

[0158] Table 3 - Comparison of experimental results of different candidate region selection algorithms

[0159]

[0160] As shown in Table 3, both the Selective Search-based method and the SAM-based method can detect anomaly regions in the image. In terms of the F1-score and AUROC metrics for the anomaly detection task, the SAM-based method outperforms Selective Search when the number of valid candidate regions is small. Selective Search performs segmentation based on low-level features and lacks understanding of semantic information, resulting in redundant detected candidate regions. The SAM-based model, however, learns more semantic information from multiple large-scale datasets and is therefore more effective in detecting potential anomaly regions while avoiding redundant information.

[0161] After obtaining the anomaly scores for the complete image and the subimage containing the suspected anomaly, we need to sum these two scores. In this paper, we try to use different weights for summing to achieve higher anomaly detection accuracy. The experimental results are shown in Table 4.

[0162] Table 4 - Comparison of experimental results of summing two anomaly scores with different weights

[0163]

[0164] From the experimental results in Table 4, we can see that when the weight ratio of the two is 4:1, the model achieves the best effect, with an F1-score of 0.9152 and an AUROC of 0.9725.

[0165] Figure 2 The schematic diagram of the structure of the foreign body intrusion detection system for rail transit environment based on the visual model provided by the embodiment of the present invention includes:

[0166] The preprocessing module 201 is used to preprocess the original image captured by the high-speed railway onboard camera, identify and extract suspicious areas containing potential foreign objects, and obtain the original complete image and at least one suspicious area sub-image;

[0167] A first extraction module 202 is configured to input the original complete image and the suspicious region sub-image into an image encoder to extract coarse-grained global features and fine-grained local features respectively;

[0168] The second extraction module 203 is used to generate corresponding text prompt information for the original complete image and the suspicious area sub-image respectively, and extract text feature vectors through a text encoder;

[0169] A calculation module 204 is used to calculate the multi-level similarity between the coarse-grained global features and the complete image text features, and the multi-level similarity between the fine-grained local features and the suspicious area text features;

[0170] Fusion module 205, for performing weighted fusion of the two types of similarities according to a preset weight to obtain a comprehensive anomaly score;

[0171] The determination module 206 is configured to perform distribution modeling on the comprehensive anomaly score based on a dynamic threshold model to determine whether there is an intruding foreign object in the image.

[0172] The system provided by the embodiment of the present invention has the same technical features as the above-mentioned method embodiment, and therefore can achieve the same technical effects, which will not be described in detail here.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A foreign body intrusion detection method for rail transit environment based on a visual model, characterized in that: The following steps are involved: S1. Preprocessing the original image captured by the high-speed railway onboard camera, using the SAM segmentation network to identify and extract suspicious areas containing potential foreign objects, to obtain the original complete image and at least one suspicious area sub-image; S2. Input the original complete image and the suspicious area sub-image into an image encoder to extract a coarse-grained global feature vector and a fine-grained local feature vector respectively; S3, generating corresponding text prompt information for the original complete image and the suspicious area sub-image respectively, and extracting text feature vectors through a text encoder; S4, calculating the multi-level similarity between the coarse-grained global feature vector and the complete image text feature vector, and the multi-level similarity between the fine-grained local feature vector and the suspicious area text feature vector; S5. Perform weighted fusion on the two similarities according to the preset weights to obtain a comprehensive anomaly score; S6. Based on a dynamic threshold model, the distribution of the comprehensive anomaly score is modeled to determine whether there is an intruding foreign object in the image. During the model training phase, the image encoder and the text encoder are jointly optimized based on an unsupervised training method. Specifically, a comparative learning strategy is used to maximize the similarity between the coarse-grained global feature vector of the complete image and the feature vector of the scene description text, while minimizing the similarity between the fine-grained local feature vector of the suspicious area sub-image and the feature vector of the foreign object description text; The multi-level similarity calculation in S4 includes: At the 6th, 9th and 12th layers of the image encoder, the first-level feature vector, the second-level feature vector and the third-level feature vector of the complete image and the corresponding-level feature vector of the suspicious region sub-image are extracted respectively; Calculate the similarity Sim between the feature vectors of each level of the complete image and the feature vector of the complete image text Di , the formula is: ； in, represents the image feature vector output by the kth layer of the image encoder, where k∈{6,9,12} represents the network depth level, The complete image text feature vector representing the corresponding level is generated by the text prompt information containing the railway scene description; Calculate the similarity Sim between the feature vectors of each level of the suspicious area sub-image and the feature vector of the suspicious area text Di j , the formula is: ； in, The k-th layer image feature vector representing the suspicious area sub-image, The text feature vector representing the suspicious area at the corresponding level is generated by the text prompt information containing the feature description of the foreign object.

2. The method according to claim 1, characterized in that The weighted fusion in S5 satisfies: ； Among them, An_score represents the comprehensive anomaly score, represents the complete image anomaly score, defined as ; Represents the abnormal score of the suspicious area sub-graph, defined as , α and β are preset weight coefficients, satisfying α+β=1.

3. The method according to claim 1, characterized in that The dynamic threshold model in S6 is a mixed Gaussian model, and its probability density function is: ； Where x is the input comprehensive anomaly score vector, μ is the mean vector, Σ is the covariance matrix, n is the dimension of x, T represents the transpose sign, and π is the circumference of the circle.

4. The method according to claim 3, characterized in that The Gaussian mixture model optimizes parameters by the expectation maximization algorithm and iteratively calculates the sample x i The posterior probability γ(z ij )for: ； Where K is the number of Gaussian distributions, π j is the mixing coefficient of the jth distribution, and satisfies , Represents the sample point x i The probability density value under the j-th Gaussian distribution, Represents the sample point x i The probability density value under the k-th Gaussian distribution.

5. The method according to claim 4, characterized in that The model parameter update satisfies: ； ； ； Where N is the total number of training samples, μ j is the updated j-th Gaussian distribution mean vector, π j is the updated mixing coefficient, Σ j is the updated covariance matrix, and T represents the transposed sign.

6. The method according to claim 1, wherein The image encoder adopts a hierarchical feature extraction structure, and the image feature vector I output by the kth layer is fk satisfy: ； in, represents the k-th layer network of the encoder, is the image feature vector output by the previous layer.

7. The method according to claim 1, characterized in that The generation of the suspicious area subgraph satisfies: ； Among them, S is the image segmentation model, D i represents the original input image, is the jth suspicious area sub-graph output.

8. The rail transit environment foreign body intrusion detection system based on visual model is characterized by: include: A preprocessing module is used to preprocess the original image captured by the high-speed railway onboard camera, using the SAM segmentation network to identify and extract suspicious areas containing potential foreign objects, thereby obtaining the original complete image and at least one suspicious area sub-image; A first extraction module is configured to input the original complete image and the suspicious region sub-image into an image encoder, and extract a coarse-grained global feature vector and a fine-grained local feature vector respectively; A second extraction module is used to generate corresponding text prompt information for the original complete image and the suspicious area sub-image respectively, and extract text feature vectors through a text encoder; A calculation module, configured to calculate the multi-level similarity between the coarse-grained global feature vector and the complete image text feature vector, and the multi-level similarity between the fine-grained local feature vector and the suspicious area text feature vector; The fusion module is used to perform weighted fusion of the two similarities according to the preset weights to obtain a comprehensive anomaly score; A determination module is used to perform distribution modeling of the comprehensive anomaly score based on a dynamic threshold model to determine whether there is an intruding foreign object in the image. During the model training phase, the image encoder and text encoder are jointly optimized based on an unsupervised training method. Specifically, a comparative learning strategy is used to maximize the similarity between the coarse-grained global feature vector of the complete image and the feature vector of the scene description text, while minimizing the similarity between the fine-grained local feature vector of the suspicious area subimage and the feature vector of the foreign object description text; Among them, the multi-level similarity calculation includes: At the 6th, 9th and 12th layers of the image encoder, the first-level feature vector, the second-level feature vector and the third-level feature vector of the complete image and the corresponding-level feature vector of the suspicious region sub-image are extracted respectively; Calculate the similarity Sim between the feature vectors of each level of the complete image and the feature vector of the complete image text Di , the formula is: ； in, represents the image feature vector output by the kth layer of the image encoder, where k∈{6,9,12} represents the network depth level, The complete image text feature vector representing the corresponding level is generated by the text prompt information containing the railway scene description; Calculate the similarity Sim between the feature vectors of each level of the suspicious area sub-image and the feature vector of the suspicious area text Di j , the formula is: ； in, The k-th layer image feature vector representing the suspicious area sub-image, The text feature vector representing the suspicious area at the corresponding level is generated by the text prompt information containing the feature description of the foreign object.

Images