Training method for railway intrusion target semantic segmentation model based on cycle consistency

Abstract

The present application provides a training method for a railway intrusion target semantic segmentation model based on cycle consistency, wherein the method includes extracting query sample features of a query sample image and k supporting sample features of k supporting sample images through a feature extraction network; performing self-alignment processing on the query sample features and multiple supporting sample features through a feature alignment module, and cross-aligning the self-aligned query sample features and the kth supporting sample features; determining a supporting prototype based on the k query cross features and n mask suggestions through a mask matching module; determining n query prototypes based on the k supporting cross features and corresponding standard mask results through the mask matching module; and determining a predicted mask result corresponding to the query sample image through the mask matching module based on the supporting prototypes, n query prototypes and n mask suggestions.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for a railway intrusion target semantic segmentation model based on cycle consistency. Background Art

[0002] Perimeter intrusions, such as those caused by natural disasters, illegal pedestrian access, and animal activity, are common occurrences in railway areas. These can severely impact train operations, causing derailments and rollovers, posing a serious threat to life and property. Therefore, accurate perception of railway perimeter intrusions can assist personnel in promptly resolving potential threats and reducing the occurrence of such accidents. Machine vision technology has advanced rapidly in recent years and has emerged as a promising technology for detecting railway intrusion targets. However, existing semantic segmentation technologies offer relatively crude detection of intrusion targets in railway scenarios, and the diverse and uneven distribution of intrusion target categories hinders the generalization capabilities of traditional models. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a training method for a railway intrusion target semantic segmentation model, so as to improve the refined perception of the railway intrusion target semantic segmentation model and the generalization ability of the model in the case of a large number of sparse sample categories.

[0004] In the first aspect, the present invention provides a training method for a railway intrusion target semantic segmentation model based on cycle consistency. The railway intrusion target semantic segmentation model includes a feature extraction network, a segmentation module, a feature alignment module and a mask matching module. The method includes extracting query sample features of a query sample image and k support sample features of k support sample images through the feature extraction network; inputting the query sample into the segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module; performing self-alignment processing on the query sample features and multiple support sample features through the feature alignment module, and matching the query sample features after self-alignment with the kth support sample image. The support sample features are cross-aligned to obtain the kth support cross feature and query cross feature; the support prototype is determined based on the k query cross features and n mask suggestions through the mask matching module; the n query prototypes are determined based on the k support cross features and the corresponding standard mask results through the mask matching module; the predicted mask result corresponding to the query sample image is determined based on the support prototype, n query prototypes and n mask suggestions through the mask matching module; based on the predicted mask result corresponding to the query sample image, the standard mask result and the target loss function, the model parameters are optimized to train and generate a railway intrusion target semantic segmentation model.

[0005] In an optional embodiment, the feature extraction network includes an input layer, a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer, and the input layer includes a convolution sublayer, a batch normalization sublayer, an activation sublayer, and a pooling sublayer.

[0006] In an optional embodiment, the query sample feature includes a first query sample feature output by the first residual layer, a second query sample feature output by the second residual layer, a third query sample feature output by the third residual layer, and a fourth query sample feature output by the fourth residual layer; the support sample feature includes a first support sample feature output by the first residual layer, a second support sample feature output by the second residual layer, a third support sample feature output by the third residual layer, and a fourth support sample feature output by the fourth residual layer.

[0007] The segmentation module outputs n mask proposals in the following way:

[0008] The second query sample feature, the third query sample feature, the fourth query sample feature and the first learnable parameter are input into the first decoding layer to obtain the second learnable parameter; the second query sample feature, the third query sample feature and the fourth query sample feature and the second learnable parameter are input into the second decoding layer to obtain the third learnable parameter; the second query sample feature, the third query sample feature and the fourth query sample feature and the third learnable parameter are input into the third decoding layer to obtain the fourth learnable parameter; the product of the fourth learnable parameter and the first query sample feature is calculated, and n mask suggestions are output through the normalization unit.

[0009] In an optional embodiment, the feature alignment module performs self-alignment processing on the features in the following manner:

[0010] For each feature of the second query sample feature, the third query sample feature, and the fourth query sample feature, average them in the channel dimension to obtain the corresponding average feature; calculate the product between the average feature and the second query sample feature / the third query sample feature / the fourth query sample feature as the corresponding weighted coefficient feature; multiply the feature value on each channel of the second query sample feature / the third query sample feature / the fourth query sample feature by the corresponding coefficient value in the weighted coefficient feature to obtain the self-aligned second query sample feature / the third query sample feature / the fourth query sample feature.

[0011] In an optional embodiment, the sizes of the query sample feature and the support sample feature are both H×w×d, and the feature alignment module can obtain the query cross feature by the following method:

[0012] Cut the query sample features and the kth support sample features after self-alignment into multiple first query feature slices of hw×d, N p×d multiple second query feature slices, hw×d multiple first support feature slices and N p ×d multiple second supporting feature slices, N p =HW / (hw); for each first query feature slice, a local multi-head Transformer unit based on cycle consistency is used to compare it with the corresponding first support feature slice to obtain the first query feature slice after local information interaction; for each second query feature slice, a global multi-head Transformer unit based on cycle consistency is used to compare it with the corresponding second support feature slice to obtain the second query feature slice after global information interaction; based on all the first query feature slices after local information interaction, the local reconstructed query features are recombined to obtain the local reconstructed query features; based on all the second query feature slices after global information interaction, the global reconstructed query features are recombined to obtain the global reconstructed query features; based on the local reconstructed query features and the global reconstructed query features, the query cross features are coupled to obtain.

[0013] In an optional embodiment, based on a cycle-consistent global / local multi-head Transformer unit, the query feature slice after information interaction is output through the following steps:

[0014] Perform a linear transformation on the query feature slice to obtain the query sub-features; perform a linear transformation on the support feature slice corresponding to the query feature slice to obtain the key sub-features and value sub-features; calculate the similarity matrix between the query sub-features and the key sub-features; revise the similarity matrix through element-level cyclic consistent attention to obtain the similarity optimization matrix; based on the similarity optimization matrix, the query sub-features and the value sub-features, calculate the query feature slice after the output information interaction.

[0015] In an optional embodiment, the similarity optimization matrix is ​​obtained by:

[0016] Based on the size of the similarity value in each column of the similarity matrix, multiple vertical coordinate indexes are determined; based on the size of the similarity value in the row corresponding to the multiple vertical coordinate indexes in the similarity matrix, multiple horizontal coordinate indexes are determined; based on the multiple vertical coordinate indexes and the multiple horizontal coordinate indexes, multiple feature positions are determined; for each feature position, it is determined whether the eigenvalue at the feature position of the query sub-feature and the eigenvalue at the feature position of the supporting feature indicate the same semantic category; if so, the value at the corresponding feature position in the weight matrix is ​​determined to be a preset multiple of the eigenvalue at the feature position in the similarity matrix; if not, the value at the corresponding feature position in the weight matrix is ​​determined to be negative infinity; the remaining positions of the weight matrix are filled with zeros to obtain a weight matrix of the same size as the similarity matrix; the sum between the similarity matrix and the weight matrix is ​​calculated through an activation function to obtain a similarity optimization matrix.

[0017] In an optional embodiment, the method further includes inputting the railway monitoring image into a railway intrusion target semantic segmentation model to obtain a railway intrusion target prediction mask result output by the railway intrusion target semantic segmentation model.

[0018] In a second aspect, the present invention provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the training method of the railway intrusion target semantic segmentation model based on a cycle-consistent embodiment as described in any of the foregoing embodiments.

[0019] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the training method of the railway intrusion target semantic segmentation model based on a cycle-consistent embodiment are executed.

[0020] The present application provides a training method for a railway intrusion target semantic segmentation model based on cycle consistency, wherein the railway intrusion target semantic segmentation model includes a feature extraction network, a segmentation module, a feature alignment module and a mask matching module. The method includes extracting query sample features of a query sample image and k support sample features of k support sample images through the feature extraction network; inputting the query sample into the segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module; performing self-alignment processing on the query sample features and multiple support sample features through the feature alignment module, and matching the query sample features and the kth support sample features after self-alignment. The sample features are cross-aligned to obtain the kth supporting cross-feature and the query cross-feature. The mask matching module determines the supporting prototype based on the k query cross-features and n mask suggestions. The mask matching module determines the n query prototypes based on the k supporting cross-features and the corresponding standard mask results. The mask matching module determines the predicted mask result corresponding to the query sample image based on the supporting prototypes, n query prototypes, and n mask suggestions. The model parameters are optimized based on the predicted mask result, the standard mask result, and the target loss function corresponding to the query sample image to train and generate a railway intrusion target semantic segmentation model. A small-sample segmentation algorithm is designed to address the uneven distribution of multi-category intrusion targets. It not only focuses on common targets such as pedestrians, which are commonly used algorithms, but also performs well on other rare intrusion categories, ensuring that the model still has good generalization capabilities under class imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a training method for a cycle-consistent railway intrusion target semantic segmentation model provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of the steps of feature cross alignment provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a process for establishing a cycle-consistent similarity constraint provided in an embodiment of the present application;

[0025] Figure 4 A flowchart of the steps for interacting with feature information provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In vast railway areas, perimeter intrusions such as those left over from natural disasters, illegal entry of people, and animal activities often occur. These can severely impact train operations, causing derailments and rollovers, and seriously threaten people's lives and property safety. Therefore, accurate perception of railway perimeter intrusions can assist personnel in promptly eliminating potential threats and reducing the occurrence of such accidents.

[0028] Machine vision technology has developed rapidly in recent years and has made a name for itself in the field of railway intrusion detection. Most current work employs two granularity approaches: object-level detection and pixel-level semantic segmentation. Object detection aims to detect and locate specific objects in images or videos. A feature extraction network extracts features from an image, generates regions that potentially contain objects, and predicts the category and bounding box for each candidate region. Semantic segmentation aims to assign a category label to each pixel, providing a more refined approach than object detection. After extracting image features, it uses contextual information to classify each pixel, resulting in pixel-level classification results.

[0029] The aforementioned methods all focus on relatively common categories such as pedestrians. However, the actual types of railway intrusion targets are numerous and unevenly distributed, with some categories having very few samples compared to common categories. For semantic segmentation, acquiring large quantities of high-quality semantic segmentation annotation data is time-consuming and labor-intensive.

[0030] This application mainly solves the following two problems:

[0031] (1) Refined perception of intrusion targets in railway scenarios. Railway scenarios are often vast, and intrusion targets sometimes appear small due to their distance, making them difficult to effectively identify using traditional detection algorithms. This lack of refined perception capabilities can lead to missed detections and false alarms, especially when facing high-speed trains. Accurate and refined intrusion perception is crucial.

[0032] (2) Perception of categories with sparse sample sizes. Railway scenarios often present numerous and unevenly distributed intrusion targets. Common algorithms mostly target common categories such as pedestrians, paying less attention to other categories that can also cause major safety incidents. Specifically, intrusion targets in railway scenarios may include people, animals, vehicles, and foreign objects. These categories differ significantly in visual features, motion patterns, and spatial distribution. Furthermore, some categories of targets appear less frequently, resulting in data imbalance and, in turn, affecting the generalization ability of traditional models.

[0033] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0034] In one embodiment of the present application, a railway intrusion target semantic segmentation model is provided, which includes a feature extraction network, a segmentation module, a feature alignment module, and a mask matching module.

[0035] Since there are many categories of railway intrusion targets and their distribution is uneven, the number of samples in some categories is very scarce compared to common categories. For semantic segmentation, obtaining a large amount of high-quality semantic segmentation annotation data is time-consuming and labor-intensive. To address these problems, this application provides a small sample model training method.

[0036] Specifically, small-sample semantic segmentation training adopts an episodic paradigm approach to training and evaluation, that is, training on a large number of simulated tasks. Given a training image set D train and the test image set D test , their respective category sets are C train and C test , where C train and Q test Disjoint in object categories In D trainThe model trained on D is directly used test Each simulation task consists of:

[0037] one k-shotepisode={{I s} k ,i q};

[0038] Each simulation task consists of k support sample images {I s} and a query sample image I q They all contain objects of the same type. So when the number of simulation tasks in the training set and the test set is N train 、N test When , the training set and test set can be expressed as:

[0039]

[0040] It should be noted that the mask M of the image and query image is supported during training s 、M q All are accessible and used to calculate losses to optimize network parameters. During testing, only the mask M of the supported image can be accessed. s .

[0041] In one embodiment of the present application, the segmentation and matching problems are decoupled, and a simple mask matching is used to operate on the support samples and the obtained set of query mask suggestions to obtain the query segmentation result. The advantages of this are:

[0042] (1) This few-to-few matching paradigm frees the matching task from the segmentation module, allowing us to focus on the matching problem itself;

[0043] (2) It reduces the training complexity and only requires simple “few-to-few” matching to solve the small sample segmentation problem;

[0044] (3) Because the matching and segmentation modules are separated, the learning of the two modules will not affect each other, and it is also more convenient to examine the performance of matching and segmentation.

[0045] Figure 1 The flowchart of a training method for a railway intrusion target semantic segmentation model based on cycle consistency is provided in an embodiment of the present application. Figure 1 As shown, the present application provides a training method for a cycle-consistent railway intrusion target semantic segmentation model, which completes each simulation task through the following steps:

[0046] S1. Extract query sample features of the query sample image and k support sample features of the k support sample images through the feature extraction network.

[0047] Here, we provide a ResNet network. The feature extraction network includes an input layer, a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer. The input layer includes a convolution sublayer, a batch normalization sublayer, an activation sublayer, and a pooling sublayer. The first residual layer, the second residual layer, the third residual layer, and the fourth residual layer have different bottleneck blocks in each layer, but the basic composition is 1×1 convolution for dimensionality reduction, general convolution for feature processing, and finally 1×1 convolution for dimensionality increase. The output of each layer can be recorded as {F i}, i = 1, 2, 3, 4. F i The superscript of is the number of special residual layers.

[0048] The query sample feature may include the first query sample feature output by the first residual layer The second query sample features output by the second residual layer The third query sample feature output by the third residual layer The fourth query sample feature output by the fourth residual layer The support sample features include the first support sample features output by the first residual layer The second support sample features output by the second residual layer The third support sample features output by the third residual layer The fourth support sample features output by the fourth residual layer

[0049] S2. Input the query sample into the segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module.

[0050] The segmentation module is used to segment all objects in an image. The segmentation module mainly consists of a transformer decoder. Specifically, the segmentation module can output n mask suggestions in the following way:

[0051] The second query sample feature The third query sample feature And the fourth query sample feature The first learnable parameter is input into the first decoding layer to obtain the second learnable parameter. The second query sample feature, the third query sample feature, and the fourth query sample feature, along with the second learnable parameter, are input into the second decoding layer to obtain the third learnable parameter. The second query sample feature, the third query sample feature, and the fourth query sample feature, along with the third learnable parameter, are input into the third decoding layer to obtain the fourth learnable parameter. The product of the fourth learnable parameter and the first query sample feature is calculated, and n mask suggestions are outputted through the normalization unit.

[0052] First, use the standard transformer decoder to calculate the second query sample features The third query sample feature And the fourth query sample feature The cross attention between N learnable vectors. The transformer decoder consists of three consecutive transformer decoding layers, each of which uses the output of the previous layer as input. Each transformer decoding layer can be expressed as:

[0053]

[0054] Where, E l and E l+1 Represent the N learnable embedding vectors before and after the transformer. Tlayer represents the transformer decoding layer. Finally, the output of the transformer decoder is combined with Multiply them together to get N mask proposals, and use the sigmoid function to normalize all mask proposals to the grayscale output of [0, 1].

[0055] S3. Perform self-alignment processing on the query sample feature and multiple support sample features respectively through the feature alignment module, and cross-align the self-aligned query sample feature and the k-th support sample feature to obtain the k-th support cross feature and the query cross feature.

[0056] The feature alignment module may include a self-alignment module and a cross-alignment module.

[0057] In step S3, the self-alignment module can perform self-alignment processing on the features in the following ways:

[0058] For each feature in the second query sample feature, the third query sample feature, and the fourth query sample feature, average them in the channel dimension to obtain the corresponding average feature. Calculate the product of the average feature and the second query sample feature / the third query sample feature / the fourth query sample feature as the corresponding weighted coefficient feature. Multiply the feature value on each channel of the second query sample feature / the third query sample feature / the fourth query sample feature by the corresponding coefficient value in the weighted coefficient feature to obtain the self-aligned second query sample feature / the third query sample feature / the fourth query sample feature.

[0059] Here, we take the query sample feature as an example to explain the steps of how the self-alignment module implements feature self-alignment. The self-alignment module is a parameter-free module that regularizes the input features. The input can be the second query sample feature, the third query sample feature, and the fourth query sample feature. For each layer of the input query sample feature First average it in the channel dimension to get the corresponding average feature Then multiply the two together to get the weighted coefficient feature in the channel dimension

[0060]

[0061] Finally, according to A i Input query sample features Multiply the corresponding weighted coefficient features on different channels to obtain the self-aligned

[0062] The steps for self-alignment of support sample features are similar to those for self-alignment of query sample features and will not be repeated here.

[0063] The cross alignment module is used to reduce the deviation between different images. Two weight-sharing transformer decoders are input in parallel, into which the proposed novel global-local coupling based cycle-consistent lightweight Transformer unit is introduced.

[0064] like Figure 2 As shown in Figure 2, the cross alignment module can obtain query cross features in the following ways:

[0065] S100, cutting the self-aligned query sample features and the kth support sample features into multiple first query feature slices of hw×d, N p ×d multiple second query feature slices, hw×d multiple first support feature slices and N p ×d multiple second supporting feature slices, N p =HW / (hw).

[0066] The cross-alignment module uses a global-local coupling based on cycle consistency lightweight transformer unit, introduces cycle consistency constraints in the attention mechanism, and is used in feature cross-alignment. Take as an example to explain the implementation steps of the cross alignment module:

[0067] against Divide equally into N p =HW / (hw) feature sub-blocks, and then reorganized into N p ×hw×d feature blocks, and the calculation is carried out in two ways: local and global.

[0068] S101 : For each first query feature slice, compare it with the corresponding first support feature slice through a local multi-head Transformer unit based on cycle consistency to obtain the first query feature slice after local information interaction.

[0069] In the local area, several hw×d feature slices are passed through the multi-head Transformer unit based on cycle consistency. Since these feature slices correspond to the adjacent areas of the original input features, the local information interaction within each feature sub-block is enhanced, so they belong to the local area.

[0070] S102 : For each second query feature slice, compare it with the corresponding second support feature slice through a global multi-head Transformer unit based on cycle consistency to obtain the second query feature slice after global information interaction.

[0071] In the global, several shapes are N p The ×d feature slices are processed by a multi-head Transformer unit based on cycle consistency. These feature slices are equivalent to the features at the same position of each original feature sub-block. The interaction of this information can enhance the expression of global information, so it belongs to the global.

[0072] The number of feature slice combinations can be used as a hyperparameter to adjust the complexity and performance of the module.

[0073] S103 : Reconstruct the first query feature fragments after all local information interactions to obtain local reconstructed query features.

[0074] S104: Reconstruct the second query feature fragments based on all global information interactions to obtain a global reconstructed query feature.

[0075] The reorganization here can sequentially splice the first / second query feature slices after each interaction.

[0076] S105 , coupling and obtaining query cross features based on the local reconstructed query features and the global reconstructed query features.

[0077] Finally, after addition and convolution, the local reconstructed query features and the global reconstructed query features are coupled to obtain the output features.

[0078] Self-aligned Two weight-sharing transformer decoders are input in parallel, into which the proposed novel global-local coupling based cycle-consistent lightweight transformer unit is introduced. It can be expressed as:

[0079]

[0080]

[0081] Where, express The query cross features after cross alignment, express Cross-aligned support cross features, MLP represents multi-layer perceptron, and LcycMHAtten represents the proposed new global-local coupling based on cycle-consistent lightweight transformer unit operation.

[0082] in, The cross alignment step is similar to the above steps, except that in step S101 and step S102, the support feature slice is used as input to be compared with the query feature slice.

[0083] In this way, the query cross features can be obtained in step S3 And get support cross features

[0084] S4. Determine the supporting prototype based on the k query cross features and n mask suggestions through the mask matching module.

[0085] The mask matching module can cross-feature each query Combine the corresponding mask proposals to perform mask global average pooling (MGAP), and superimpose the results of the 2nd, 3rd, and 4th layers together to obtain a supporting prototype.

[0086] S5. Determine n query prototypes based on the k supporting cross features and the corresponding standard mask results through the mask matching module.

[0087] The mask matching module can be used on each cross-feature Combined with the corresponding standard mask results, mask global average pooling (MGAP) is performed, and the results of the 2nd, 3rd, and 4th layers are superimposed together to finally obtain n query prototypes where d is dimension.

[0088] S6. Determine a predicted mask result corresponding to the query sample image based on the supporting prototypes, the n query prototypes, and the n mask suggestions through the mask matching module.

[0089] Measure using cosine distance The similarity between them is calculated using a fully connected layer to fuse the corresponding masks. The detailed relationship can be expressed as:

[0090]

[0091]

[0092] in, The final predicted mask result is represented by M, N mask proposals, and S, which represents the similarity. The MLP is a fully connected layer. The N similarities are used as the input of the fully connected layer, and its output is used to weighted average the N mask proposals to obtain the predicted mask result.

[0093] S7. Based on the predicted mask results, standard mask results and target loss function corresponding to the query sample image, the model parameters are optimized to train and generate a railway intrusion target semantic segmentation model.

[0094] Since the algorithm is divided into two stages: potential target segmentation and mask matching, in step S7, the loss function here also consists of two parts.

[0095] In the potential target segmentation stage, the segmentation loss function proposed by Mask2Former is adopted, denoted as L p The Hungarian algorithm is used to match the predicted multiple mask proposals with the true value, and only the mask with the best matching effect is supervised by Dice Loss.

[0096] In the mask matching stage, the prediction results Find Dice Loss as the main loss, denoted as L M . And designed the loss function L for reverse prediction back and the loss function L based on cycle consistency cycle Serves as an auxiliary loss to constrain the cross alignment module.

[0097] Loss function L for reverse prediction back The aligned query features are averaged using the true labels and concatenated with the support features to predict the support segmentation map, which is then compared with the true value of the support mask to obtain the loss.

[0098] Based on the cycle consistent loss function L cycle The process of establishing cycle consistent constraints is as follows Figure 3 shown.

[0099] For a set of support and query features {F s ,F q}, first flatten each of them into a HW×d shape, and then calculate the cosine similarity between each pair of feature points:

[0100]

[0101] Get the similarity matrix D∈R WH×WH , then the cyclic consistency between the support features and the query features is expressed as follows:

[0102] (a) For each row m in D, obtain the index set N={n1,n2,…,n k};

[0103] (b) For each column in D Find the index corresponding to the maximum similarity value in its column Then the maximum values ​​found by k indexes constitute the index set

[0104] (c) If the cycle consistency is satisfied, the starting feature point k transition points and k target feature points Have the same semantics.

[0105] Similarly, starting from each column, we can obtain the corresponding cycle-consistent relationship. For feature point pairs that satisfy cycle consistency, we need to explicitly guide the network model to enhance the similarity between them, thereby aligning the query and support features. Using pixel-level label information, we propose the following cycle-consistent loss function to achieve this goal:

[0106]

[0107] The first fraction in the logarithm corresponds to the circular consistency starting from the query feature, and the second fraction corresponds to the circular consistency starting from the support feature.

[0108] Therefore, the final loss function can be expressed as:

[0109] L=L p +λ1L M +λ2L back +λ3L cycle

[0110] Among them, λ1, λ2 and λ3 are constants, which can be adjusted to make the algorithm show better results.

[0111] The present invention provides a training method for a cycle-consistent railway intrusion target semantic segmentation model, which achieves pixel-level refined perception of intrusion targets. By leveraging pixel-level semantic segmentation labels for supporting images, the similarity relationship between supporting samples and query samples is mined, thereby achieving pixel-level mask prediction of intrusion targets. This refined perception capability effectively captures targets at low resolution and long distances, significantly reducing the missed detection rate and improving the system's stability and safety in complex railway environments.

[0112] Secondly, the small sample segmentation algorithm is designed to address the uneven distribution of multi-category intrusion targets. It not only focuses on common targets such as pedestrians targeted by commonly used algorithms, but also performs well on other scarce intrusion categories, enabling the model to still have good generalization capabilities even in the case of category imbalance.

[0113] At the same time, based on cycle consistency theory, we designed a global-local lightweight Transformer module based on a cycle-consistent attention mechanism and a cycle-consistent loss constraint to further explore the similarity relationship between support samples and query samples. We also used the idea of ​​mask matching to decouple the segmentation and matching problems, focusing on the matching problem itself. This reduced training complexity and made it easier to evaluate the performance of the segmentation and matching modules.

[0114] The model training method provided in this application can perceive intrusion targets at the pixel level in a refined manner, and can well distribute unevenly multi-category targets, reduce the training complexity in module and network design, and significantly improve the overall performance of the intrusion target detection system in railway scenarios, achieving the effect of ensuring the safety of railway operations.

[0115] In one embodiment of the present application, a novel global-local coupled cycle-consistent lightweight transformer unit is proposed. The following uses a query feature slice as an example to illustrate the steps of implementing global or local information interaction using a cycle-consistent lightweight transformer unit:

[0116] Specifically, such as Figure 4 As shown in Figure 2, based on the cycle-consistent global / local multi-head Transformer unit, the query feature slice after information interaction can be output through the following steps:

[0117] S200: Perform linear transformation on the query feature slice to obtain query sub-features.

[0118] Here the size of the query feature slice is hw×d. The input feature is transformed into the query sub-feature F through three different linear transformations. Q .

[0119] S201, perform linear transformation on the support feature slice corresponding to the query feature slice to obtain the key sub-feature F K Sum value sub-feature F V .

[0120] Among them, F Q =XW Q , F K =XW K , F V =XW V .

[0121] Among them, X is the input feature, W Q 、W K 、W V is a trainable weight matrix.

[0122] S202: Calculate the similarity matrix between the query sub-features and the key sub-features.

[0123] The common method is to calculate their dot product, which is calculated as follows:

[0124]

[0125] To prevent gradients from exploding or vanishing, the dot product result is usually divided by a scaling factor where d k It is the dimension of the Key.

[0126] S203. Revise the similarity matrix through element-level cyclic consistent attention to obtain a similarity optimization matrix.

[0127] Here, the similarity optimization matrix can be obtained as follows:

[0128] Based on the size of the similarity value of each column in the similarity matrix, multiple vertical coordinate indexes are determined. Based on the size of the similarity value in the row corresponding to the multiple vertical coordinate indexes in the similarity matrix, multiple horizontal coordinate indexes are determined. Based on the multiple vertical coordinate indexes and the multiple horizontal coordinate indexes, multiple feature positions are determined. For each feature position, determine whether the eigenvalue at the feature position of the query sub-feature and the eigenvalue at the feature position of the supporting feature indicate the same semantic category. If so, determine that the value at the corresponding feature position in the weight matrix is ​​a preset multiple of the eigenvalue at the feature position in the similarity matrix. If not, determine that the value at the corresponding feature position in the weight matrix is ​​negative infinity. Fill the remaining positions of the weight matrix with zeros to obtain a weight matrix of the same size as the similarity matrix. Calculate the sum between the similarity matrix and the weight matrix through the activation function to obtain a similarity optimization matrix.

[0129] For each column j in the similarity matrix D, select the index corresponding to the value with the highest similarity before P Then find the index j′ with the greatest similarity in each row corresponding to these indices. p , p∈[1,P]. If the feature obtained after the loop With the start feature If the semantic category of the supporting feature point is the same, it is considered to meet the cycle consistency and should make more contributions to the linear representation of the feature; otherwise, it is not satisfied and its influence should be weakened or eliminated. Therefore, the corresponding cases can be divided into three types. If the supporting feature point does not participate in the establishment of cycle consistency, then ΔD i,j =0; if the supporting feature points satisfy cyclic consistency, then ΔDi,j =η|D i,j |; If the supporting feature points do not satisfy cyclic consistency, then ΔD i,j =-∞.

[0130] S204: Calculate query feature slices after outputting information interaction based on the similarity optimization matrix, the query sub-features, and the value sub-features.

[0131] Finally, the weight modification matrix ΔD can be obtained. Its modification process is:

[0132] D′=softmax(D+ΔD)

[0133] In this way, the unvisited supporting feature points will not be affected, while the contribution of the feature points that meet the cycle consistency can be amplified and the similarities that do not meet the cycle consistency can be removed.

[0134] This lightweight, globally coupled Transformer design eliminates the O(HW × HW) computational complexity required by the naive Transformer for pairwise calculations of each feature point, significantly reducing computational overhead. In actual network construction, multiple consecutive calculations can be performed end-to-end, balancing the network's representational capabilities and computational complexity.

[0135] In one embodiment of the present application, the railway monitoring image may be input into a railway intrusion target semantic segmentation model to obtain a railway intrusion target prediction mask result output by the railway intrusion target semantic segmentation model.

[0136] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in FIG, the electronic device 500 includes a processor 510, a memory 520 and a bus 530.

[0137] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 through the bus 530. When the machine-readable instructions are executed by the processor 510, the steps of a training method based on a cycle-consistent railway intrusion target semantic segmentation model in the above-mentioned method embodiment can be executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0138] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a training method based on a cycle-consistent railway intrusion target semantic segmentation model in the above-mentioned method embodiment can be executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0139] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0143] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0144] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0145] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for a railway intrusion target semantic segmentation model based on cycle consistency, characterized by: The railway intrusion target semantic segmentation model includes a feature extraction network, a segmentation module, a feature alignment module and a mask matching module. The method includes: Extract query sample features of the query sample image and k support sample features of the k support sample images through the feature extraction network; Input the query sample into the segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module; The query sample feature and multiple support sample features are self-aligned by the feature alignment module, and the query sample feature and the k-th support sample feature are cross-aligned to obtain the k-th support cross feature and the query cross feature; Determining a supporting prototype based on the k query cross features and the n mask suggestions through a mask matching module; Determining n query prototypes based on the k supporting cross features and corresponding standard mask results through a mask matching module; Determining, by a mask matching module, a predicted mask result corresponding to the query sample image based on the supporting prototypes, the n query prototypes, and the n mask suggestions; Based on the predicted mask results, standard mask results and target loss function corresponding to the query sample image, the model parameters are optimized to train and generate a railway intrusion target semantic segmentation model.

2. The method according to claim 1, characterized in that The feature extraction network includes an input layer, a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer. The input layer includes a convolution sublayer, a batch normalization sublayer, an activation sublayer, and a pooling sublayer.

3. The method according to claim 2, characterized in that The query sample features include the first query sample features output by the first residual layer, the second query sample features output by the second residual layer, the third query sample features output by the third residual layer, and the fourth query sample features output by the fourth residual layer. The support sample features include the first support sample features output by the first residual layer, the second support sample features output by the second residual layer, the third support sample features output by the third residual layer, and the fourth support sample features output by the fourth residual layer. The segmentation module outputs n mask proposals in the following way: Inputting the second query sample feature, the third query sample feature, the fourth query sample feature and the first learnable parameter into the first decoding layer to obtain a second learnable parameter; Inputting the second query sample feature, the third query sample feature, the fourth query sample feature and the second learnable parameter into the second decoding layer to obtain a third learnable parameter; Inputting the second query sample feature, the third query sample feature, the fourth query sample feature and the third learnable parameter into the third decoding layer to obtain a fourth learnable parameter; The product of the fourth learnable parameter and the first query sample feature is calculated, and n mask suggestions are output through a normalization unit.

4. The method according to claim 2, characterized in that The feature alignment module performs self-alignment processing on features in the following manner: For each of the second query sample feature, the third query sample feature, and the fourth query sample feature, averaging the feature in the channel dimension to obtain a corresponding average feature; Calculating the product of the average feature and the second query sample feature / the third query sample feature / the fourth query sample feature as the corresponding weighted coefficient feature; The feature value on each channel of the second query sample feature / the third query sample feature / the fourth query sample feature is multiplied by the corresponding coefficient value in the weighted coefficient feature to obtain the self-aligned second query sample feature / the third query sample feature / the fourth query sample feature.

5. The method according to claim 2, characterized in that The size of query sample features and support sample features are both H×w×d. The feature alignment module can obtain query cross features in the following way: Cut the query sample features and the kth support sample features after self-alignment into multiple first query feature slices of hw×d, N p ×d multiple second query feature slices, hw×d multiple first support feature slices and N p ×d multiple second supporting feature slices, N p =HW / (hw); For each first query feature slice, compare it with the corresponding first support feature slice through a local multi-head Transformer unit based on cycle consistency to obtain the first query feature slice after local information interaction; For each second query feature slice, compare it with the corresponding second support feature slice through a global multi-head Transformer unit based on cycle consistency to obtain the second query feature slice after global information interaction; Based on the first query feature fragment after all local information interactions, reconstruct the local reconstructed query feature; Based on the second query feature slice after all global information interactions, a global reconstructed query feature is obtained by recombining; The query cross feature is obtained by coupling based on the local reconstructed query feature and the global reconstructed query feature.

6. The method according to claim 5, characterized in that Based on the cycle-consistent global / local multi-head Transformer unit, the query feature slice after information interaction is output through the following steps: Perform linear transformation on the query feature slice to obtain the query sub-feature; Perform linear transformation on the support feature slice corresponding to the query feature slice to obtain key sub-features and value sub-features; Calculating a similarity matrix between the query sub-features and the key sub-features; The similarity matrix is ​​revised through element-level cyclic consistent attention to obtain the similarity optimization matrix; Based on the similarity optimization matrix, the query sub-features and the value sub-features, the query feature slices after the output information interaction are calculated.

7. The method according to claim 6, characterized in that The similarity optimization matrix is ​​obtained by: Determining a plurality of vertical coordinate indexes based on the magnitude of the similarity value of each column in the similarity matrix; Determine multiple horizontal coordinate indexes based on the similarity values ​​in the rows corresponding to the multiple vertical coordinate indexes in the similarity matrix; Determine a plurality of feature positions based on a plurality of ordinate indices and a plurality of abscissa indices; For each feature position, determine whether the feature value of the query sub-feature at that feature position and the feature value of the support feature at that feature position indicate the same semantic category; if so, determine that the value at the corresponding feature position in the weight matrix is ​​a preset multiple of the feature value at that feature position in the similarity matrix; if not, determine that the value at the corresponding feature position in the weight matrix is ​​negative infinity; Fill the remaining positions of the weight matrix with zeros to obtain a weight matrix of the same size as the similarity matrix; The sum of the similarity matrix and the weight matrix is ​​calculated by an activation function to obtain a similarity optimization matrix.

8. The method according to claim 1, characterized in that It also includes inputting the railway monitoring image into a railway intrusion target semantic segmentation model to obtain a railway intrusion target prediction mask result output by the railway intrusion target semantic segmentation model.

9. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the training method of the cycle-consistent railway intrusion target semantic segmentation model according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for training a railway intrusion target semantic segmentation model based on cycle consistency as claimed in any one of claims 1 to 7.

Images