A multi-modal based track foreign object intrusion event capturing system and method

Abstract

The present application belongs to the technical field of track safety monitoring, and more particularly relates to a track foreign matter intrusion event capturing system and method based on multi-modal. The method comprises periodically collecting image sequences of a track line; performing change detection on each pair of adjacent images in the image sequences to identify change regions; performing multi-modal feature extraction on the change regions in the images; using the image features and text features extracted by the multi-modal to determine the change type; then generating a text answer and a corresponding visual mask to further verify the change type; the event capturing module determines whether it is a foreign matter intrusion event according to the change type, and if it is a foreign matter intrusion event, triggers an alarm and records event information. The present application solves the problem that the prior art cannot accurately distinguish the change type when capturing foreign matter intrusion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of track safety monitoring, and more particularly relates to a multi-modal based track foreign object intrusion event capturing system and method. BACKGROUND

[0002] In the track, timely and accurate capture of foreign object intrusion events is crucial to ensure train operation safety. Existing technologies mainly achieve foreign object intrusion event capture by comparing the changed areas between two frames of images. However, this method has certain limitations and cannot accurately distinguish the types of changes, such as being unable to determine whether the change is due to the appearance of a new foreign object, the disappearance of a foreign object, or a change in the type of a foreign object, resulting in a high false positive rate.

[0003] Chinese patent document CN118759590A discloses an airport runway foreign object detection and removal system and method. The system includes a detector distributedly arranged on the airport runway for acquiring detection data of the airport runway; a data processing terminal connected with the detector and a removal device respectively, for determining the foreign object position and type based on the detection data, and generating a foreign object removal instruction according to the foreign object position, the foreign object type and the current position of the removal device; and a removal device for removing foreign objects based on the foreign object removal instruction. The detection data of the airport runway is acquired by the detector, and the data processing terminal processes the detection data to determine the foreign object position, thereby realizing automatic detection of foreign objects on the airport runway. The data processing terminal also generates a foreign object removal instruction according to the foreign object position, the foreign object type and the current position of the removal device to guide the removal device to remove foreign objects. However, this patent has the following shortcomings:

[0004] 1. Lack of semantic reasoning ability

[0005] The detection method of CN118759590A is mainly based on signal processing and simple image analysis, lacking semantic understanding of the detection results. Its system mainly relies on image signal analysis when determining the type of foreign objects, and cannot perform deeper semantic reasoning on the types of changes.

[0006] 2. Lack of intuitive explanation and interpretability

[0007] The detection method of CN118759590A mainly outputs the position and type of foreign objects, lacking intuitive explanation of the detection results. Its system cannot provide detailed semantic description or visual mask of the changed area, making it difficult for operators to quickly understand the detection results.

[0008] 3. Lack of comprehensive judgment on the type of change

[0009] The detection method of CN118759590A mainly focuses on the presence and location of foreign matter, and lacks comprehensive judgment of change types. For example, it cannot accurately distinguish whether the foreign matter is newly appeared, disappeared, or the type has changed.

[0010] Therefore, the present application designs a multi-modal based track foreign matter intrusion event capturing system and method. SUMMARY

[0011] The present application aims to overcome at least one of the above-mentioned defects of the prior art, and provides a multi-modal based track foreign matter intrusion event capturing system.

[0012] The present application also discloses a multi-modal based track foreign matter intrusion event capturing method to solve the problem that the prior art cannot accurately distinguish the change type when capturing foreign matter intrusion.

[0013] The detailed technical solutions of the present application are as follows:

[0014] A multi-modal based track foreign matter intrusion event capturing system, the system comprising: an image acquisition module, a change detection module, a multi-modal feature extraction module, a change type judgment module, an event capturing module;

[0015] The image acquisition module is used to acquire an image sequence of a track line, and construct a detection data set of line patrol scene foreign matter intrusion; the detection data set contains image pairs, descriptive problem texts, text answers, and corresponding visual masks, which are used to support multi-modal feature extraction and change type judgment;

[0016] The change detection module is used to perform change detection on adjacent two frames of images in the image sequence, and mark out the change area;

[0017] The multi-modal feature extraction module is used to extract image features and text features of the change area;

[0018] The change type judgment module is used to judge the change type based on the multi-modal features, and distinguish newly appeared foreign matter, disappeared foreign matter, or changed foreign matter type;

[0019] The event capturing module is used to determine whether it is a foreign matter intrusion event according to the change type, and perform corresponding alarm or record.

[0020] According to the present application, the multi-modal feature extraction module comprises an image feature extraction unit and a text feature extraction unit.

[0021] The image feature extraction unit is used for extracting image features of the change area, and includes a residual network and a 1x1 convolution layer, in addition, the image feature extraction unit further includes a normalization layer, a pooling layer, an activation function and a feature fusion module, which are used for extracting multi-scale features and enhancing the expression ability of the model;

[0022] The text feature extraction unit is used for generating text questions describing the change area and extracting text features, and includes a question generation module, a text encoder, a feature extraction module, a feature alignment module and a feature enhancement module;

[0023] The question generation module is used for presetting a question template and automatically generating descriptive questions according to the image content of the change area;

[0024] The text encoder is used for text input preprocessing, performs word segmentation processing on the text describing the change area, and converts it into an embedding vector;

[0025] The feature extraction module is used for extracting semantic features in the text based on a pre-trained model CLIP Transformer of comparison and image pairs;

[0026] The feature alignment module is used for aligning the text features with the image features to ensure the consistency of the two in space and semantics;

[0027] The feature enhancement module is used for further processing the extracted text features to enhance the expression ability of the key information.

[0028] According to the application, the change type judgment module includes:

[0029] A multi-stage semantic reasoning unit is used for combining the image features and the text features, judging the change type through multi-stage semantic reasoning;

[0030] A text-visual answer decoding unit is used for generating a text answer and a corresponding visual mask to further verify the change type.

[0031] A multi-modal based track foreign object intrusion event capturing method, the method includes:

[0032] S1, image acquisition: using a high-resolution camera to periodically acquire an image sequence of the track line, ensuring that the image can clearly reflect the track and its surrounding environment. The acquired image sequence includes a panorama and a local close-up of the track line to cover different ranges of change situations.

[0033] Detection dataset construction: labeling and processing the acquired image sequence to generate a multi-modal detection dataset for model training and verification; the labeling content includes a pixel-level visual mask of the change area, and question texts and corresponding text answers generated for the change area;

[0034] S2, change detection: change detection is performed on each pair of adjacent images in the image sequence to identify a change region;

[0035] S3, multi-modal feature extraction on the change region in the image, first using an image feature extraction unit to extract image features of the change region; then using a text feature extraction unit to generate a text question describing the change region and extracting text features;

[0036] S4, change type judgment using the image features and text features extracted by the multi-modal: first, combining the image features and text features, generating a multi-modal feature representation through a multi-stage semantic reasoning module, determining the change type according to the multi-modal feature representation, then generating a text answer and a corresponding visual mask, further verifying the change type;

[0037] S5, the event capture module determines whether it is a foreign object intrusion event according to the change type, if it is a foreign object intrusion event, triggering an alarm and recording event information.

[0038] According to the application, the specific steps of the change detection in step S2 are as follows:

[0039] S2.1, image preprocessing: grayscale, normalization, denoising and other operations are performed on the collected image sequence to improve the contrast and clarity of the image.

[0040] S2.2, change detection algorithm: first, difference operation is performed on the current acquired image frame and the background image to obtain a gray image of the motion region in the image, and then thresholding is performed on the gray image to extract the motion region, and then an edge-based inter-frame difference method is used to obtain the change region of the motion target;

[0041] S2.3, change region marking: according to the change detection result, the change region is marked as the region of interest for the subsequent steps.

[0042] According to the application, the image features of the change region extracted by the image feature extraction unit are as follows:

[0043] S3.1.1, multi-scale feature extraction: using a shared weight residual network as a basic feature extraction network, multi-scale feature maps of the image are extracted layer by layer through multiple residual blocks;

[0044] S3.1.2, channel dimension adjustment: the multi-scale feature maps output by the residual network are adjusted in channel dimension through a 1x1 convolutional layer to optimize the feature expression ability;

[0045] S3.1.3, feature fusion and output: the multi-scale feature maps adjusted in channel dimension are fused to generate the final visual feature representation for subsequent processing.

[0046] According to the application, preferably, the text feature extraction unit generates a text question describing the change region, and the specific processing process of extracting the text feature is as follows:

[0047] S3.2.1, the question generation module presets a question template, and automatically generates descriptive question text according to the image content of the change region;

[0048] S3.2.2, input the generated descriptive question text into the text encoder, and the text encoder performs word segmentation processing on the descriptive question text, and converts each word or subword in the text into a corresponding embedding vector.

[0049] S3.2.3, text feature extraction:

[0050] The text processed by the text encoder is input into the pre-trained CLIP Transformer, and the CLIP Transformer encodes the text through a multi-layer Transformer architecture to extract the semantic features of the text; each layer of the Transformer captures long-distance dependencies in the text through a self-attention mechanism, and further processes the features through a feedforward network; finally, the CLIP Transformer outputs a high-dimensional feature representation of the text;

[0051] S3.2.4, the feature alignment module aligns the high-dimensional feature representation of the text with the image features to ensure consistency in space and semantics;

[0052] S3.2.5, the feature enhancement module further processes the extracted text features, and enhances the expression ability of key information through an attention mechanism.

[0053] According to the application, preferably, in step S4, the visual features and the text features are combined to generate a multi-modal feature representation through a multi-stage semantic reasoning module, and the change type is determined according to the multi-modal feature representation, which specifically includes the following steps:

[0054] S4.1, the multi-stage semantic reasoning module receives image features and text features from the multi-modal feature extraction module. The image features include visual information such as color, texture, shape, etc. of the change region, while the text features are text questions describing the change region and their corresponding feature representations;

[0055] S4.2, the image features and the text features are fused through convolution layers and fully connected layers to generate a multi-modal feature representation;

[0056] S4.3, use multi-head self-attention mechanism MHSA to perform self-attention calculation on the multi-modal feature representation to enhance the expression ability of the features, specifically as follows:

[0057] MHSA(Q,K,V) = Concat(head1, head2, head3,...,head h )W O (1)

[0058] where,

[0059] head i = Attention(QW iQ ,KW iK ,VW iv ) (2)

[0060]

[0061] In formulas (1)-(3), MHSA represents a multi-head self-attention mechanism; Q represents a query matrix, representing the query relationship of each element in the input sequence to other elements; K represents a key matrix, representing the key information of each element in the input sequence, used for matching with the query matrix; V represents a value matrix, representing the value information of each element in the input sequence, used for weighted summation according to the matching results of the query and the key; W O represents an output linear transformation matrix, used for mapping the outputs of all heads back to the original dimension after concatenation; head i represents the self-attention output of the i-th head; W iQ , W iK and W iV are linear transformation matrices corresponding to the i-th head, respectively, for mapping the input sequence to different subspaces for self-attention calculation; d k represents the dimension of the key matrix K, used to scale the dot product result to prevent the dot product result from being too large to cause the gradient of the softmax function to disappear;

[0062] S4.4, further enhances the interaction between image features and text features using a cross-attention mechanism CA, the input is two sequences of different modalities, the dimensions of the two sequences must be the same, one sequence is input Q, which defines the output sequence length, the other sequence provides input K and V, as follows:

[0063] CA(Q,K,V) = softmax((W Q S2)*(W K S1) T ))*W V S1 (4)

[0064] In formula (4), S1 represents the feature sequence of the first modality; S2 represents the feature sequence of the second modality; W Q , W K and W Vlinear transformation matrix of query, key and value respectively, used to map the feature sequence of different modalities to the query, key and value space of cross attention mechanism; Q represents the query matrix, which is calculated by W Q S2 is calculated, which represents the query relationship of the feature sequence of the second modality to the first modality; K represents the key matrix, which is calculated by W K S1 is calculated, which represents the key information of the feature sequence of the first modality; V represents the value matrix, which is calculated by W V S1 is calculated, which represents the value information of the feature sequence of the first modality.

[0065] S4.5, using a feedforward network FFN to further process the multi-modal features processed by the cross attention mechanism, to enhance the non-linear expression ability of the features, as follows:

[0066] FFN(x) = max(0, xW1 + b1)W2 + b2 (5)

[0067] In formula (5), x represents the multi-modal features input to the feedforward network; W1 represents the weight matrix of the first layer of the feedforward network, used for linear transformation of the input features; b1 represents the bias vector of the first layer of the feedforward network, used to increase the flexibility of the model; max(0, ·) represents the ReLU activation function, used to introduce nonlinearity, setting all negative values to 0 and retaining positive values; W2 represents the weight matrix of the second layer of the feedforward network, used for further linear transformation of the features after ReLU activation; b2 represents the bias vector of the second layer of the feedforward network, used for the flexibility of the second layer of the model;

[0068] S4.6, the multi-modal features processed by the feedforward network pass through a feature selection module to dynamically select the feature representation most relevant to the current problem, as follows:

[0069] α = σ(F vl *F w ), β = 1 - α, F s = αF vl + βF w (6)

[0070] In formula (6), σ is a sigmoid function used to map the input value to the interval (0, 1) as a weight coefficient; F vl is the multi-modal feature representation processed by the feedforward network; F w is the text feature; α is the weight coefficient calculated by the sigmoid function, representing the importance of the visual feature in the current problem; β is the complement of α, representing the importance of the text feature in the current problem; F s represents the final selected multi-modal feature representation, which is the weighted sum of the visual feature and the text feature, used for subsequent processing;

[0071] S4.7, based on the generated multi-modal feature representation, using a classifier to determine the change type, specifically as follows:

[0072] Change Type = argmax(softmax(F s W+b))(7)

[0073] In formula (7), F s is the multi-modal feature representation processed by the multi-stage semantic reasoning module, W and b are the weights and bias of the classifier.

[0074] According to the application, the generated text answer and the corresponding visual mask further verify the change type, which means that first, the multi-modal feature is processed by the multi-stage semantic reasoning module to obtain F s Then, a linear transformation layer is applied to map F s to a new space to obtain a weight matrix W and a bias vector b, then W and b are segmented and reshaped to adapt to the subsequent visual mask generation, and finally, the processed weight matrix W is multiplied by the feature representation processed by the Mask Decoder and the bias vector b is added to generate the final visual mask M, the formula is as follows:

[0075] W,b = S&R(Linear(F s ))(8)

[0076]

[0077] In formula (8) and formula (9), S&R represents the segmentation and reshaping operation, which is used to adjust the shape of the weight matrix and the bias vector to facilitate the calculation with the subsequent feature representation, Linear is a linear transformation, F s is the multi-modal feature representation processed by the multi-stage semantic reasoning module, is the feature representation processed by the Mask Decoder, M is the final visual mask, W represents the weight matrix after linear transformation, which is used to generate the visual mask, and b represents the bias vector after linear transformation, which is used to adjust the generation of the visual mask.

[0078] The Mask Decoder processing means that the rough visual mask M c and the original visual feature F v are input into the Mask Decoder, and the Mask Decoder uses two consecutive bidirectional attention blocks to establish the mapping relationship between M c and F v at the pixel level, so that the model can more accurately understand which visual features are related to the question-answer feature F sMost relevant, the formula is expressed as follows:

[0079]

[0080] In formula (10), M c represents a rough visual mask: this is a preliminary visual feature map indicating the region that may contain the target, generated by F vl After two layers of convolution layers, pixel decoding is obtained; F v represents the original visual features: this is the original feature directly extracted from the image, which contains rich visual information.

[0081] Compared with the prior art, the beneficial effects of the present application are:

[0082] (1) The present application provides multi-modal feature extraction and fusion: not only detects changes, but also generates text answers and visual masks in combination with natural language questions, providing more intuitive explanations, combining image and text information, and improving the accuracy of change type judgment. Introduce multi-modal feature extraction module, not only extract the deep features of the image, but also generate text features describing the change area in combination with the text information. Use the ResNet network with shared weights to extract multi-scale visual features of the two images, and adjust the channel dimension through the 1×1 convolution layer to generate multi-scale change features; use the pre-trained CLIP Transformer to extract the text features of the question and generate sentence-level representations. Through the fusion of multi-modal features, the system can comprehensively understand the change area from the visual and semantic levels, so as to more accurately judge the change type. For example, for a new foreign object appearing in the image, the system can not only recognize its shape and color through image features, but also understand its semantic information through text features, thereby improving the accuracy and reliability of the judgment.

[0083] (2) Multi-stage semantic reasoning of the present application: through the multi-stage semantic reasoning module, the judgment of change type is gradually refined to reduce false positives. This module uses multi-head self-attention mechanism, cross-attention mechanism and feedforward network to gradually refine and verify the interaction between image and text features, enhancing the expression ability of the features. At the same time, through the feature selection module, the most relevant feature representation to the current question is dynamically selected, further improving the accuracy of reasoning. For example, when judging whether the change area is a foreign object disappearance, the multi-stage semantic reasoning module can consider the pixel changes in that area of the image and the semantic explanation of that area in the text description, so as to make a more accurate judgment.

[0084] (3) Text-Visual Answer Decoding: Generate intuitive visual masks to verify the correctness of the text answer and enhance the reliability of the system. Generate intuitive visual masks and corresponding text answers to further verify the change type. This module not only provides more intuitive visual feedback for the system, but also enhances the explainability of the system. For example, when the system judges that a change area is the appearance of a new foreign matter, the text-visual answer decoding module can generate a visual mask marking the new foreign matter area and output the text answer "yes, new foreign matter appears", so that the operator can quickly and intuitively understand the change situation, and improve the practicality and credibility of the system. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 is the system architecture diagram of the present application.

[0086] Figure 2 is the method flow chart of the present application. DETAILED DESCRIPTION

[0087] The present disclosure will be further described below in conjunction with the drawings and examples.

[0088] Example 1,

[0089] Referring to Figure 1 The embodiment provides a multi-modal based track foreign matter intrusion event capturing system, which comprises:

[0090] An image acquisition module is configured to acquire an image sequence of a track line; and construct a line patrol scene foreign matter intrusion detection dataset.

[0091] The detection dataset comprises image pairs, question text descriptions, text answers, and corresponding visual masks, which are used to support multi-modal feature extraction and change type judgment.

[0092] Image acquisition: a high-resolution camera is used to periodically acquire an image sequence of a track line, ensuring that the images can clearly reflect the track and its surrounding environment. The acquired images should include panoramic and close-up views of the track line to cover different ranges of changes.

[0093] Dataset construction: the acquired image sequence is labeled and processed to generate a multi-modal dataset for model training and verification. The labeling content includes pixel-level masks of the change area, as well as natural language questions and corresponding answers generated for the change area.

[0094] Change detection module: change detection is performed on adjacent two frames of images in the image sequence to mark the change area.

[0095] The multi-modal feature extraction module is used to extract image features and text features of the changed area. The introduction of the multi-modal feature extraction module not only extracts deep features of the image, but also generates text features describing the changed area in combination with text information. A ResNet network with shared weights is used to extract multi-scale visual features of the two images, and a 1x1 convolution layer is used to adjust the channel dimension to generate multi-scale change features. A pre-trained CLIP Transformer is used to extract text features of the question and generate sentence-level representations.

[0096] The multi-modal feature extraction module includes an image feature extraction unit and a text feature extraction unit.

[0097] The image feature extraction unit is used to extract visual features of the changed area, including a residual network ResNet and a 1x1 convolution layer. In addition, the unit also includes a normalization layer, a pooling layer, a ReLU activation function, and a feature fusion module, which is used to extract multi-scale features and enhance the expression ability of the model.

[0098] The text feature extraction unit is used to generate text questions describing the changed area and extract text features, including a question generation module Question Generation Module: used to preset a question template and automatically generate descriptive questions based on the image content of the changed area, such as "Has a new foreign object appeared in this area?"

[0099] The text encoder Text Encoder: is used for text input preprocessing, performs word segmentation processing on the text describing the changed area, and converts it into an embedding vector.

[0100] The feature extraction module: a pre-trained CLIP Transformer model based on contrast and image pairs is used to extract semantic features in the text.

[0101] The feature alignment module Feature Alignment Module: is used to align the text features with the image features to ensure their consistency in space and semantics.

[0102] The feature enhancement module Feature Enhancement Module: is used to further process the extracted text features, such as enhancing the expression ability of key information through attention mechanisms.

[0103] The change type judgment module is used to judge the change type based on multi-modal features, and to distinguish between new foreign objects, disappearance of foreign objects, or change of foreign object type.

[0104] The change type judgment module includes:

[0105] A multi-stage semantic reasoning unit is used to combine image features and text features, and determine the change type through multi-stage semantic reasoning.

[0106] A text-visual answer decoding unit is used to generate a text answer and a corresponding visual mask, and further verify the change type.

[0107] An event capture module is used to determine whether it is a foreign object intrusion event according to the change type, and to perform corresponding alarm or recording.

[0108] Embodiment 2,

[0109] Part Figure 2 The embodiment provides a multi-modal based track foreign object intrusion event capture method, comprising the following steps:

[0110] S1, image acquisition: periodically acquiring image sequences of track lines;

[0111] S2, change detection: performing change detection on each pair of adjacent images in the image sequence to identify a change region, and the specific steps are as follows:

[0112] S21, image preprocessing: performing grayscale, normalization, denoising and other operations on the acquired image sequence to improve the contrast and clarity of the image.

[0113] S22, change detection algorithm: first, difference operation is performed on the current acquired image frame and the background image to obtain a grayscale image of the motion region in the image, and then thresholding is performed on the grayscale image to extract the motion region, and then an edge-based inter-frame difference method is used to obtain the change region of the motion target;

[0114] S23, change region marking: according to the change detection result, the change region is marked as a region of interest for subsequent steps.

[0115] S3, multi-modal feature extraction on the change region in the image:

[0116] First, the image feature extraction unit is used to extract the image features of the change region, such as color, texture, shape, etc., and the specific steps are as follows:

[0117] S3.1.1, multi-scale feature extraction: using a shared weight residual network ResNet as a basic feature extraction network, and extracting multi-scale feature maps of the image through multiple residual blocks layer by layer;

[0118] S3.1.2, channel dimension adjustment: adjusting the channel dimension of the multi-scale feature map output by the residual network through a 1x1 convolutional layer to optimize the feature expression ability;

[0119] S3.1.3, Feature Fusion and Output: The multi-scale feature maps after adjusting the channel dimension are fused to generate the final visual feature representation for subsequent processing.

[0120] Then the text feature extraction unit is used to generate a text question describing the changed area and extract text features, as follows:

[0121] S3.2.1, The Question Generation Module predefines a question template and automatically generates descriptive question text based on the image content of the changed area. The question text includes but is not limited to the following forms:

[0122] "Has a new foreign object appeared in this area?"

[0123] "Has the foreign object in this area disappeared?"

[0124] "Has the type of foreign object in this area changed?"

[0125] S3.2.2, The generated descriptive question text is input into the Text Encoder, which processes the text by tokenizing and converts each word or subword in the text into a corresponding embedding vector.

[0126] S3.2.3, Text Feature Extraction:

[0127] The text processed by the text encoder is input into the pre-trained CLIP Transformer, which encodes the text through a multi-layer Transformer architecture to extract the semantic features of the text. Each layer of the Transformer captures long-range dependencies in the text through self-attention mechanisms and further processes the features through feedforward networks. Finally, the CLIP Transformer outputs a high-dimensional feature representation of the text.

[0128] S3.2.4, The Feature Alignment Module aligns the high-dimensional feature representation of the text with the image features to ensure consistency in space and semantics.

[0129] S3.2.5, The Feature Enhancement Module further processes the extracted text features, such as through attention mechanisms to enhance the expressiveness of key information.

[0130] S4. Change type judgment using image and text features extracted by multi-modal extraction: First, combine image and text features, pass through multi-stage semantic reasoning module to judge change type; then generate text answer and corresponding visual mask to further verify change type, as follows:

[0131] 1. Input feature extraction

[0132] The multi-stage semantic reasoning module receives image and text features from the multi-modal feature extraction module. Image features include color, texture, shape, and other visual information of the changed area, while text features are the text problems describing the changed area and their corresponding feature representations.

[0133] 2. Multi-stage reasoning process

[0134] The multi-stage semantic reasoning module gradually refines and verifies the change type through multiple stages. Each stage includes the following key steps:

[0135] 2.1 Feature fusion

[0136] Fuse image and text features to generate multi-modal feature representation. The specific method is as follows:

[0137] Image features: visual features of the changed area obtained from the change detection module.

[0138] Text features: text features describing the changed area obtained from the text feature extraction unit.

[0139] Fusion method: fuse image and text features through convolutional and fully connected layers to generate multi-modal feature representation.

[0140] 2.2 Multi-head self-attention mechanism

[0141] Use multi-head self-attention mechanism (Multi-Head Self-Attention, MHSA) to perform self-attention calculation on multi-modal features to enhance the expression ability of features. The specific steps are as follows:

[0142] Input: multi-modal feature representation.

[0143] Output: feature representation after self-attention mechanism processing.

[0144] Formula:

[0145] MHSA(Q, K, V) = Concat(head1, head2, head3,..., head h )W O

[0146] Where,

[0147] head i = Attention(Q iQ , K iQ , V iQ )

[0148]

[0149] where MHSA(Q, K, V) denotes the Multi-Head Self-Attention mechanism; Q denotes the query matrix, which represents the query relationship of each element in the input sequence to other elements; K denotes the key matrix, which represents the key information of each element in the input sequence, used for matching with the query matrix; V denotes the value matrix, which represents the value information of each element in the input sequence, used for weighted summation according to the matching results of the query and the key; W O denotes the output linear transformation matrix, which is used to map the outputs of all heads back to the original dimension after concatenation; head i denotes the self-attention output of the i-th head; W iQ , W iK and W iV denote the linear transformation matrices of the query, key and value corresponding to the i-th head, respectively, which are used to map the input sequence to different subspaces for self-attention calculation; d k denotes the dimension of the key matrix K, which is used to scale the dot product result to prevent the dot product result from being too large to cause the gradient of the softmax function to disappear.

[0150] 2.3 Cross-Attention mechanism

[0151] The cross-attention mechanism (CA) is used to further enhance the interaction between image features and text features. The specific steps are as follows:

[0152] Input: image features and text features processed by the self-attention mechanism, input as two different modal sequences, the dimensions of the two sequences must be the same, one sequence as input Q, defining the output sequence length, the other sequence providing input K and V.

[0153] Output: multi-modal feature representation processed by the cross-attention mechanism.

[0154] Formula:

[0155] CA(Q, K, V) = softmax((W Q S2) * (W K S1) T ) * W V S1

[0156] where S1 denotes the feature sequence of the first modality (e.g., text); S2 denotes the feature sequence of the second modality (e.g., image); W Q , W K , and W V are linear transformation matrices of query, key and value respectively, which are used to map the feature sequences of different modalities to the query, key and value spaces of cross-attention mechanism; Q denotes the query matrix, which is calculated by W Q S2, representing the query relationship of the feature sequence of the second modality to the first modality; K denotes the key matrix, which is calculated by W K S1, representing the key information of the feature sequence of the first modality; V denotes the value matrix, which is calculated by W V S1, representing the value information of the feature sequence of the first modality.

[0157] 2.4 Feed-Forward Network

[0158] A Feed-Forward Network (FFN) is used to further process the multi-modal features and enhance the non-linear representation ability of the features. The specific steps are as follows:

[0159] Input: multi-modal feature representation processed by cross-attention mechanism.

[0160] Output: multi-modal feature representation processed by FFN.

[0161] Formula:

[0162] FFN(x) = max(0, xW1 + b1)W2 + b2

[0163] where x denotes the multi-modal feature input to the FFN; W1 denotes the weight matrix of the first layer of FFN, which is used to perform linear transformation on the input feature; b1 denotes the bias vector of the first layer of FFN, which is used to increase the flexibility of the model; max(0, ·) denotes the ReLU activation function, which is used to introduce nonlinearity, setting all negative values to 0 and preserving positive values; W2 denotes the weight matrix of the second layer of FFN, which is used to perform further linear transformation on the feature after ReLU activation; b2 denotes the bias vector of the second layer of FFN, which is used for the flexibility of the second layer of the model.

[0164] 2.5 Feature Selection

[0165] A selection module is used to dynamically select the feature representation most relevant to the current problem. The specific steps are as follows:

[0166] Input: multi-modal feature representation processed by FFN.

[0167] Output: selected feature representation.

[0168] Formula:

[0169] a = s(F vl ), b = 1 - a, F w = aF s + bF vl w

[0170] where s is a sigmoid function used to map input values to the interval (0, 1) as a weight coefficient; F vl is the multi-modal feature representation after processing by the feedforward network; F w is the text feature; a is the weight coefficient calculated by the sigmoid function, representing the importance of visual features in the current problem; b is the complement of a, representing the importance of text features in the current problem; F s represents the final selected multi-modal feature representation, which is the weighted sum of visual features and text features, used for subsequent processing.

[0171] 3. Output feature representation

[0172] After processing by the multi-stage semantic reasoning module, the generated multi-modal feature representation will be used for subsequent change type judgment and text-visual answer decoding module.

[0173] 4. Change type judgment

[0174] Based on the generated multi-modal feature representation, a classifier is used to judge the change type. The specific steps are as follows:

[0175] Input: multi-modal feature representation.

[0176] Output: change type, including change, new foreign object, foreign object disappearance, and foreign object type change.

[0177] Formula:

[0178] Change type = argmax(softmax(F s W + b))

[0179] where F s is the multi-modal feature representation after processing by the multi-stage semantic reasoning module, and W and b are the weights and bias of the classifier.

[0180] 5. Text-visual answer decoding

[0181] Generate a visual mask corresponding to the text answer to further verify the change type. The specific steps are as follows:

[0182] Input: multi-modal feature representation.

[0183] ​Output: Visual mask corresponding to the text answer.

[0184] Formula:

[0185] W, b = S & R(Linear(F s ))

[0186]

[0187] Where S&R represents the Split and Reshape operation, used to adjust the shape of the weight matrix and bias vector, so as to calculate with the subsequent feature representation, Linear is linear transformation, F s is the multi-modal feature representation processed by the multi-stage semantic reasoning module, is the feature representation processed by Mask Decoder, M is the final visual mask, W represents the weight matrix after linear transformation, used to generate visual mask, b represents the bias vector after linear transformation, used to adjust the generation of visual mask;

[0188] The processing process of Mask Decoder is as follows:

[0189]

[0190] Input feature integration:

[0191] Rough visual mask M c : This is a preliminary visual feature map, indicating the area that may contain the target, obtained by F vl through two layers of convolutional layers for pixel decoding.

[0192] Original visual feature F v : This is the original feature extracted directly from the image, containing rich visual information.

[0193] Feature processing: input M c and F v into Mask Decoder. Mask Decoder contains two consecutive bidirectional attention blocks, and the design purpose of these blocks is to establish the mapping relationship between M c and F v at the pixel level. This mapping helps the model to more accurately understand which visual features are most relevant to the question-answer feature F s .

[0194] S5, the event capture module determines whether it is a foreign object intrusion event according to the change type, if it is a foreign object intrusion event, triggers an alarm and records the event information.

[0195] Example 3,

[0196] This embodiment provides a track line patrol scene dataset construction:

[0197] 1. Dataset construction target:

[0198] A multi-modal change detection dataset for track line patrol scenes is constructed to support the capture and analysis of track intrusion events. The dataset will contain image pairs, descriptive question texts, text answers, and corresponding visual masks to support multi-modal feature extraction and change type judgment.

[0199] 2. Dataset composition:

[0200] The dataset will contain the following content:

[0201] Image pairs: Each image pair represents the state of the track line at different time points, used for change detection.

[0202] Question text description: Generate natural language questions for the change area in the image pair, such as "Is there a change in the second image compared to the first image?" "Is there a new foreign object in the change area of the second image?" etc.

[0203] Text answer: Accurate answer to the question text description, such as "Yes" "No" "New foreign object appears" "Foreign object disappears" etc.

[0204] Visual mask: Pixel-level mask marking the change area, used to visually display the change location.

[0205] 3. Data collection:

[0206] Image collection:

[0207] Use high-resolution cameras to regularly collect image sequences of the track line, ensuring that the images can clearly reflect the track and its surrounding environment.

[0208] The image collection frequency is determined according to the actual operation of the track line and safety requirements, such as collecting once a minute or once an hour.

[0209] The collected images should include panoramic and close-up views of the track line to cover different ranges of changes.

[0210] Annotation tool:

[0211] Use professional image annotation tools such as Labelme or custom annotation software to annotate the collected images.

[0212] The annotation content includes the boundary box and pixel-level mask of the change area, and the annotators need to have professional knowledge of track line patrol to ensure the accuracy of the annotation.

[0213] 4. Descriptive question text generation:

[0214] Question template design:

[0215] According to the characteristics of the track line patrol scene, the following question templates are designed:

[0216] "Is there a change in the second image compared to the first image?"

[0217] "Is there a new foreign object in the changed area of the second image?"

[0218] "Is the changed area of the second image a foreign object disappearance?"

[0219] "Is there a change in the foreign object in the changed area of the second image?"

[0220] "Is the changed area of the second image a change in foreign object type?"

[0221] "Is the changed area of the second image a change in foreign object size?"

[0222] Question generation:

[0223] For each pair of images, according to the annotated change area and change type, automatically generate questions in the above question templates.

[0224] During question generation, combine the annotation information and change type to ensure that the questions are consistent with the actual situation.

[0225] 5. Text answer generation

[0226] Answer rules:

[0227] According to the annotated change type and change area, automatically generate corresponding text answers. For example:

[0228] If the change area is the appearance of a new foreign object, the answer is "Yes, new foreign object appears";

[0229] If the change area is the disappearance of a foreign object, the answer is "Yes, foreign object disappears".

[0230] If the change area is a change in foreign object type, the answer is "Yes, foreign object type changes".

[0231] If there is no change, the answer is "No".

[0232] 6. Visual mask generation:

[0233] Mask annotation:

[0234] For the change area in each pair of images, annotate the pixel-level mask to ensure that the mask accurately reflects the change location.

[0235] Mask annotation should cover all types of changes, including the appearance of new foreign objects, the disappearance of foreign objects, and changes in the type of foreign objects.

[0236] 7. Dataset statistics and analysis:

[0237] Dataset size:

[0238] The dataset should contain enough image pairs and question-answer pairs to support the training and validation of the model. For example, a dataset containing thousands of image pairs and tens of thousands of question-answer pairs can be constructed.

[0239] Statistical analysis of the types of changes in the dataset should be conducted to ensure that the dataset covers a variety of common changes, such as the appearance of new foreign objects, the disappearance of foreign objects, and changes in the type of foreign objects.

[0240] 8. Dataset example:

[0241] The following is an example of a dataset:

[0242]

[0243]

Claims

1. A multimodal orbital foreign object intrusion event detection system, characterized in that, The system includes: an image acquisition module, a change detection module, a multimodal feature extraction module, a change type judgment module, and an event capture module; The image acquisition module is used to acquire image sequences of the track line and construct a detection dataset for foreign object intrusion in the track patrol scene; the detection dataset includes image pairs, descriptive question text, text answers and corresponding visual masks, which are used to support multimodal feature extraction and change type judgment; The change detection module is used to detect changes between two adjacent frames in an image sequence and mark the changed areas. The multimodal feature extraction module is used to extract image features and text features from the changing regions; The multimodal feature extraction module includes an image feature extraction unit and a text feature extraction unit; The image feature extraction unit is used to extract image features from the changing regions. It includes a residual network and a 1×1 convolutional layer. In addition, the image feature extraction unit also includes a normalization layer, a pooling layer, an activation function, and a feature fusion module to extract multi-scale features and enhance the expressive power of the model. The text feature extraction unit is used to generate text questions describing the changing regions and extract text features, including a question generation module, a text encoder, a feature extraction module, a feature alignment module, and a feature enhancement module; The question generation module is used to preset question templates and automatically generate descriptive questions based on the image content of the changing area. The text encoder is used for text input preprocessing, which segments the text describing the changing regions and converts it into an embedding vector. The feature extraction module extracts semantic features from the text based on the CLIP Transformer pre-trained model of contrast and image pairs. The feature alignment module is used to align text features with image features to ensure their spatial and semantic consistency. Feature enhancement module: Used to further process the extracted text features to enhance the expressive power of key information; The change type determination module is used to determine the change type based on multimodal features, and to distinguish between newly appearing foreign objects, disappearance of foreign objects, and changes in the type of foreign objects. The event capture module is used to determine whether it is a foreign object intrusion event based on the type of change, and to issue an alarm or record accordingly.

2. The multimodal orbital foreign object intrusion event detection system according to claim 1, characterized in that, The change type determination module includes: A multi-stage semantic reasoning unit is used to combine image features and text features to determine the type of change through multi-stage semantic reasoning; The text-visual answer decoding unit is used to generate text answers and corresponding visual masks to further verify the change type.

3. A method for detecting foreign object intrusion events based on multimodality, using the multimodal foreign object intrusion event detection system based on any one of claims 1 to 2, characterized in that, The method includes: S1. Image Acquisition: High-resolution cameras are used to periodically acquire image sequences of the track line. The acquired image sequences include panoramic views and close-up details of the track line. Detection dataset construction: The collected image sequences are labeled and processed to generate a multimodal detection dataset for model training and validation; the labeled content includes pixel-level visual masks of changing regions, as well as question text and corresponding text answers generated for the changing regions; S2. Perform change detection on each pair of adjacent images in the image sequence to identify the changed regions; S3. Perform multimodal feature extraction on the changing regions in the image. First, use the image feature extraction unit to extract the image features of the changing regions; then use the text feature extraction unit to generate text questions describing the changing regions and extract the text features. S4. Use the image and text features extracted from the multimodal model to determine the change type: First, combine the image and text features and generate a multimodal feature representation through the multi-stage semantic reasoning module. Determine the change type based on the multimodal feature representation, and then generate the text answer and the corresponding visual mask to further verify the change type. S5. The event capture module determines whether it is a foreign object intrusion event based on the type of change. If it is a foreign object intrusion event, an alarm is triggered and the event information is recorded.

4. The method for capturing orbital foreign object intrusion events based on multimodal analysis according to claim 3, characterized in that, The specific steps for change detection in step S2 are as follows: S2.1 Image preprocessing: Perform grayscale conversion, normalization, and noise reduction on the acquired image sequence to improve image contrast and clarity; S2.2 Change Detection Algorithm: First, the grayscale image of the moving region in the image is obtained by performing a difference operation between the currently acquired image frame and the background image. The moving region is then extracted by thresholding the grayscale image. Finally, the change region of the moving target is obtained by using an edge-based inter-frame difference method. S2.3, Change Region Marking: Based on the change detection results, the changed regions are marked as regions of interest for subsequent steps.

5. The method for capturing orbital foreign object intrusion events based on multimodal analysis according to claim 3, characterized in that, The specific meaning of using the image feature extraction unit to extract image features of the changing region is: S3.1.1 Multi-scale feature extraction: A residual network with shared weights is used as the basic feature extraction network to extract multi-scale feature maps of the image layer by layer through multiple residual blocks; S3.1.2 Channel Dimension Adjustment: The channel dimension of the multi-scale feature map output by the residual network is adjusted by using a 1×1 convolutional layer to optimize the feature representation capability; S3.1.3 Feature Fusion and Output: The multi-scale feature maps after adjusting the channel dimensions are fused to generate the final visual feature representation for subsequent processing.

6. The method for detecting foreign object intrusion events based on multimodality according to claim 3, characterized in that, The specific process of generating a text question describing the changed region using the text feature extraction unit and extracting text features is as follows: S3.2.1 The problem generation module presets a problem template and automatically generates descriptive problem text based on the image content of the changed area; S3.2.2 Input the generated descriptive question text into the text encoder. The text encoder performs word segmentation on the descriptive question text and converts each word or subword in the text into a corresponding embedding vector. S3.2.3 Text Feature Extraction: The text processed by the text encoder is input into the pre-trained CLIP Transformer. The CLIP Transformer encodes the text through a multi-layer Transformer architecture to extract the semantic features of the text. Each Transformer layer captures long-distance dependencies in the text through a self-attention mechanism and further processes the features through a feedforward network. Finally, the CLIP Transformer outputs a high-dimensional feature representation of the text. S3.2.4 The feature alignment module aligns the high-dimensional feature representation of the text with the image features to ensure their spatial and semantic consistency. S3.2.5 The feature enhancement module further processes the extracted text features and enhances the expressive power of key information through an attention mechanism.

7. The method for detecting foreign object intrusion events based on multimodality according to claim 3, characterized in that, In step S4, combining visual and textual features, a multimodal feature representation is generated through a multi-stage semantic reasoning module. Determining the change type based on the multimodal feature representation specifically includes the following steps: S4.1 The multi-stage semantic reasoning module receives image features and text features from the multimodal feature extraction module; image features include visual information of the changing regions, while text features are textual questions describing the changing regions and their corresponding feature representations; S4.

2. Image features and text features are fused through convolutional layers and fully connected layers to generate multimodal feature representations; S4.

3. Use the multi-head self-attention mechanism MHSA to perform self-attention calculation on multimodal feature representations to enhance the expressive power of features, as detailed below: （1） in, （2） （3） In formulas (1) to (3), MHSA represents multi-head self-attention mechanism; Q represents query matrix, which represents the query relationship of each element in the input sequence with other elements; K represents key matrix, which represents the key information of each element in the input sequence, used to match with the query matrix; V represents value matrix, which represents the value information of each element in the input sequence, used to perform weighted summation based on the matching results of query and key. This represents the output linear transformation matrix, used to concatenate the outputs of all heads and map them back to the original dimension; This represents the self-attention output of the i-th head; , and These represent the linear transformation matrices corresponding to the query, key, and value of the i-th head, respectively, and are used to map the input sequence to different subspaces for self-attention computation; The dimension of the key matrix K is used to scale the dot product result to prevent the dot product result from becoming too large and causing the gradient of the softmax function to vanish. S4.

4. The cross-attention (CA) mechanism is used to further enhance the interaction between image and text features. The input consists of two sequences of different modalities, both with the same dimension. One sequence serves as the input Q, defining the length of the output sequence, while the other sequence provides the input K and V, as detailed below: （4） In equation (4), The feature sequence representing the first mode; The feature sequence representing the second mode; , and These are linear transformation matrices for query, key, and value, respectively, used to map feature sequences of different modalities to the query, key, and value space of the cross-attention mechanism; S4.

5. The feedforward network (FFN) is used to further process the multimodal features after the cross-attention mechanism to enhance the non-linear expressive power of the features, as detailed below: （5） In equation (5), x represents the multimodal features input to the feedforward network; This represents the weight matrix of the first-layer feedforward network, used to perform linear transformations on the input features; This represents the bias vector of the first layer of the feedforward network; max(0, () represents the ReLU activation function, used to introduce nonlinearity, setting all negative values ​​to 0 and retaining positive values; This represents the weight matrix of the second-layer feedforward network, used to perform further linear transformations on the features after ReLU activation; This represents the bias vector of the second-layer feedforward network; S4.6 After processing by the feedforward network, the multimodal features are dynamically selected by a feature selection module to represent the features most relevant to the current problem, as follows: , , （6） In equation (6), It is the sigmoid function, used to map input values ​​to the (0,1) interval as weight coefficients; It is a multimodal feature representation after processing by a feedforward network; It is a text feature; These are weight coefficients calculated using the sigmoid function, representing the importance of visual features in the current problem; yes The complement of represents the importance of text features in the current problem; The final selected multimodal feature representation is a weighted sum of visual and textual features, used for subsequent processing; S4.7 Based on the generated multimodal feature representation, a classifier is used to determine the type of change, as follows: （7） In equation (7), It is a multimodal feature representation after processing by a multi-stage semantic reasoning module, where W and b are the weights and biases of the classifier.

8. The method for detecting foreign object intrusion events based on multimodality according to claim 7, characterized in that, The generated text answer and corresponding visual mask, further verifying the change type, refer to, firstly, processing multimodal features through a multi-stage semantic reasoning module to obtain... Then, a linear transformation layer is applied to... Mapping to a new space yields a weight matrix W and a bias vector b. Then, W and b are segmented and reshaped to accommodate subsequent visual mask generation. Finally, the processed weight matrix W is combined with the feature representation processed by the Mask Decoder. Multiply the results and add the bias vector b to generate the final visual mask M, as shown in the following formula: （8） （9） In equations (8) and (9), Representing segmentation and reshaping operations, used to adjust the shape of the weight matrix and bias vector for computation with subsequent feature representations. `Linear` stands for linear transformation. It is a multimodal feature representation after processing by a multi-stage semantic reasoning module. It is the feature representation after processing by the Mask Decoder. M is the final visual mask, W represents the weight matrix after linear transformation, which is used to generate the visual mask, and b represents the bias vector after linear transformation, which is used to adjust the generation of the visual mask.

9. A method for detecting foreign object intrusion events based on multimodality according to claim 8, characterized in that... The Mask Decoder processing refers to the process of converting a coarse visual mask... and original visual features The input is fed into the Mask Decoder, which uses two consecutive bidirectional attention blocks to build up pixel-level density. and The mapping relationship between them allows the model to more accurately understand which visual features are associated with question-answer features. Most relevant, the formula is as follows: （10） In equation (10), Represents a coarse visual mask: This is a preliminary visual feature map used to indicate regions that may contain the target, by... The pixel is obtained by decoding through two convolutional layers; Representing raw visual features: These are raw features extracted directly from the image and contain rich visual information.

Images