Railway intrusion target semantic segmentation model training method based on circulation consistency

Abstract

The invention provides a training method of a railway intrusion target semantic segmentation model based on circulation consistency, and the method comprises the steps: extracting a query sample feature of a query sample image and k support sample features of k support sample images through a feature extraction network; performing self-alignment processing on the query sample feature and the plurality of support sample features through a feature alignment module, and performing cross alignment on the query sample feature after self-alignment and the kth support sample feature; determining a support prototype through a mask matching module based on the k query cross features and the n mask suggestions; determining n query prototypes based on the k support cross features and the corresponding standard mask results through a mask matching module; and determining a prediction mask result corresponding to the query sample image based on the support prototype, the n query prototypes and the n mask suggestions through a mask matching module.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for a railway intrusion target semantic segmentation model based on cycle consistency. Background Art

[0002] In the vast railway area, perimeter intrusion phenomena such as natural disasters, illegal entry of people and animal activities often occur, which may cause serious impacts on train operation such as derailment and overturning, and seriously threaten the safety of people's lives and property. Therefore, accurate perception of railway perimeter intrusion phenomena can assist staff to eliminate potential threats in time and reduce the occurrence of such accidents. Machine vision technology has developed rapidly in recent years and has emerged in the field of railway intrusion target perception. However, the existing semantic segmentation technology has a relatively rough perception of intrusion targets in railway scenes, and the categories of railway intrusion targets are numerous and unevenly distributed, which affects the generalization ability of traditional models. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a training method for a railway intrusion target semantic segmentation model, so as to improve the refined perception of the railway intrusion target semantic segmentation model and the generalization ability of the model in the case of numerous rare sample categories.

[0004] In a first aspect, the present invention provides a training method for a railway intrusion target semantic segmentation model based on cycle consistency, wherein the railway intrusion target semantic segmentation model comprises a feature extraction network, a segmentation module, a feature alignment module and a mask matching module, and the method comprises extracting query sample features of a query sample image and k supporting sample features of k supporting sample images respectively through the feature extraction network; inputting the query sample into the segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module; performing self-alignment processing on the query sample features and multiple supporting sample features respectively through the feature alignment module, and matching the query sample features after self-alignment with the kth supporting sample images; The support sample features are cross-aligned to obtain the kth support cross feature and the query cross feature; the support prototype is determined based on the k query cross features and the n mask suggestions through the mask matching module; the n query prototypes are determined based on the k support cross features and the corresponding standard mask results through the mask matching module; the predicted mask result corresponding to the query sample image is determined based on the support prototype, the n query prototypes and the n mask suggestions through the mask matching module; based on the predicted mask result corresponding to the query sample image, the standard mask result and the target loss function, the model parameters are optimized to train and generate a railway intrusion target semantic segmentation model.

[0005] In an optional embodiment, the feature extraction network includes an input layer, a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer, and the input layer includes a convolution sublayer, a batch normalization sublayer, an activation sublayer, and a pooling sublayer.

[0006] In an optional embodiment, the query sample feature includes a first query sample feature output by the first residual layer, a second query sample feature output by the second residual layer, a third query sample feature output by the third residual layer, and a fourth query sample feature output by the fourth residual layer; the support sample feature includes a first support sample feature output by the first residual layer, a second support sample feature output by the second residual layer, a third support sample feature output by the third residual layer, and a fourth support sample feature output by the fourth residual layer.

[0007] The segmentation module outputs n mask proposals in the following way:

[0008] The second query sample feature, the third query sample feature, the fourth query sample feature and the first learnable parameter are input into the first decoding layer to obtain the second learnable parameter; the second query sample feature, the third query sample feature and the fourth query sample feature and the second learnable parameter are input into the second decoding layer to obtain the third learnable parameter; the second query sample feature, the third query sample feature and the fourth query sample feature and the third learnable parameter are input into the third decoding layer to obtain the fourth learnable parameter; the product of the fourth learnable parameter and the first query sample feature is calculated, and n mask suggestions are output through the normalization unit.

[0009] In an optional embodiment, the feature alignment module performs self-alignment processing on the features in the following manner:

[0010] For each feature of the second query sample feature, the third query sample feature and the fourth query sample feature, average them in the channel dimension to obtain the corresponding average feature; calculate the product of the average feature and the second query sample feature / the third query sample feature / the fourth query sample feature as the corresponding weighted coefficient feature; multiply the feature value on each channel of the second query sample feature / the third query sample feature / the fourth query sample feature by the corresponding coefficient value in the weighted coefficient feature to obtain the self-aligned second query sample feature / the third query sample feature / the fourth query sample feature.

[0011] In an optional implementation, the sizes of the query sample features and the support sample features are both H×w×d, and the feature alignment module can obtain the query cross features in the following manner:

[0012] Cut the query sample features and the kth support sample features after self-alignment into multiple first query feature slices of hw×d, N p×d multiple second query feature slices, hw×d multiple first support feature slices and N p ×d multiple second supporting feature slices, N p =HW / (hw); for each first query feature slice, compare it with the corresponding first support feature slice based on cycle consistency through a local multi-head Transformer unit to obtain the first query feature slice after local information interaction; for each second query feature slice, compare it with the corresponding second support feature slice based on cycle consistency through a global multi-head Transformer unit to obtain the second query feature slice after global information interaction; based on the first query feature slice after all local information interactions, reorganize to obtain local reconstructed query features; based on the second query feature slice after all global information interactions, reorganize to obtain global reconstructed query features; based on the local reconstructed query features and the global reconstructed query features, couple to obtain query cross features.

[0013] In an optional implementation, based on a cycle-consistent global / local multi-head Transformer unit, the query feature slice after information interaction is outputted through the following steps:

[0014] Perform linear transformation on the query feature slice to obtain query sub-features; perform linear transformation on the support feature slice corresponding to the query feature slice to obtain key sub-features and value sub-features; calculate the similarity matrix between the query sub-features and the key sub-features; revise the similarity matrix through element-level cyclic consistent attention to obtain a similarity optimization matrix; calculate the query feature slice after output information interaction based on the similarity optimization matrix, query sub-features and value sub-features.

[0015] In an optional implementation, the similarity optimization matrix is ​​obtained in the following manner:

[0016] Based on the size of the similarity value of each column in the similarity matrix, multiple ordinate indexes are determined; based on the size of the similarity value in the row corresponding to the multiple ordinate indexes in the similarity matrix, multiple abscissa indexes are determined; based on the multiple ordinate indexes and the multiple abscissa indexes, multiple feature positions are determined; for each feature position, it is determined whether the feature value at the feature position of the query sub-feature and the feature value at the feature position of the supporting feature indicate the same semantic category, if so, it is determined that the value at the corresponding feature position in the weight matrix is ​​a preset multiple of the feature value at the feature position in the similarity matrix, if not, it is determined that the value at the corresponding feature position in the weight matrix is ​​negative infinity; the remaining positions of the weight matrix are filled with zeros to obtain a weight matrix of the same size as the similarity matrix; the sum between the similarity matrix and the weight matrix is ​​calculated by an activation function to obtain a similarity optimization matrix.

[0017] In an optional implementation, the method further includes inputting the railway monitoring image into a railway intrusion target semantic segmentation model to obtain a railway intrusion target prediction mask result output by the railway intrusion target semantic segmentation model.

[0018] In a second aspect, the present invention provides an electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the training method of the railway intrusion target semantic segmentation model based on a cycle-consistent embodiment as described in any of the foregoing embodiments.

[0019] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a training method for a railway intrusion target semantic segmentation model based on a cycle-consistent semantic segmentation model as described in any of the aforementioned embodiments are executed.

[0020] The present application provides a training method for a railway intrusion target semantic segmentation model based on cycle consistency, wherein the railway intrusion target semantic segmentation model includes a feature extraction network, a segmentation module, a feature alignment module and a mask matching module. The method includes extracting query sample features of a query sample image and k supporting sample features of k supporting sample images respectively through the feature extraction network; inputting the query sample into the segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module; performing self-alignment processing on the query sample features and multiple supporting sample features respectively through the feature alignment module, and matching the query sample features after self-alignment with the kth supporting sample images. The sample features are cross-aligned to obtain the kth support cross-feature and query cross-feature; the support prototype is determined based on the k query cross-features and n mask suggestions through the mask matching module; the n query prototypes are determined based on the k support cross-features and the corresponding standard mask results through the mask matching module; the predicted mask result corresponding to the query sample image is determined through the mask matching module based on the support prototype, n query prototypes and n mask suggestions; based on the predicted mask result, standard mask result and target loss function corresponding to the query sample image, the model parameters are optimized to train and generate the railway intrusion target semantic segmentation model. The small sample segmentation algorithm is designed for the uneven distribution of multi-category intrusion targets. It not only focuses on common targets such as pedestrians targeted by common algorithms, but also performs well on other sparse intrusion categories, so that the model still has good generalization ability under the condition of imbalanced categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of a training method for a cycle-consistent railway intrusion target semantic segmentation model provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a feature cross alignment step provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a process for establishing a cycle-consistent similarity constraint provided in an embodiment of the present application;

[0025] Figure 4 A flowchart of the steps of feature information interaction provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In the vast railway areas, perimeter intrusion phenomena such as natural disasters, illegal entry of people and animal activities often occur, which may cause serious impacts on train operation such as derailment and overturning, and seriously threaten people’s life and property safety. Therefore, accurate perception of railway perimeter intrusion phenomena can assist staff to eliminate potential threats in time and reduce the occurrence of such accidents.

[0028] Machine vision technology has developed rapidly in recent years and has emerged in the field of railway intrusion target perception. At present, most works use two granularity methods, namely object-level target detection and pixel-level image semantic segmentation. Object detection aims to detect and locate specific target objects from images or videos. The features in the image are extracted through the feature extraction network, and the regions that potentially contain the target are generated in the image, and the categories and bounding boxes of the candidate regions are predicted. Semantic segmentation aims to assign a category label to each pixel, which is more precise than target detection. After extracting the image features, each pixel is classified using the captured context information to obtain pixel-level classification results.

[0029] The above methods all focus on relatively common categories such as pedestrians. However, there are many categories of railway intrusion targets in reality, and their distribution is uneven. The number of samples of some categories is very scarce compared to common categories. For semantic segmentation, it is time-consuming and laborious to obtain a large number of high-quality semantic segmentation annotation data.

[0030] This application mainly solves the following two problems:

[0031] (1) Refined perception of intrusion targets in railway scenarios. Railway scenarios are often vast, and intrusion targets sometimes appear small due to their long distance, making them difficult to effectively identify in traditional detection algorithms. This lack of refined perception capabilities may result in missed detections and false alarms, especially when facing high-speed trains. Accurate and refined intrusion perception is crucial.

[0032] (2) Perception of categories with sparse sample sizes. In railway scenarios, there are often many categories of intrusion targets with uneven distribution. Most general algorithms target common categories such as pedestrians, and pay less attention to other categories that may also cause major safety accidents. Specifically, intrusion targets in railway scenarios may include people, animals, vehicles, and foreign objects. They have obvious differences in visual features, motion patterns, and spatial distribution. In addition, some categories of targets appear less frequently, resulting in data imbalance, which in turn affects the generalization ability of traditional models.

[0033] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0034] In one embodiment of the present application, a railway intrusion target semantic segmentation model is provided, which includes a feature extraction network, a segmentation module, a feature alignment module and a mask matching module.

[0035] Since there are many categories of actual railway intrusion targets and their distribution is uneven, the number of samples of some categories is very scarce compared to common categories. For semantic segmentation, it is time-consuming and laborious to obtain a large number of high-quality semantic segmentation annotation data. To solve these problems, this application provides a small sample model training method.

[0036] Specifically, small sample semantic segmentation training adopts an episodic paradigm approach to training and evaluation, that is, training on a large number of simulated tasks. Given a training image set D train and the test image set D test , their respective category sets are C train and C test , where C train and Q test Disjoint in object categories In D trainThe model trained on D is directly used test Each simulation task consists of:

[0037] one k-shotepisode={{I s} k ,i q};

[0038] Each simulation task consists of k support sample images {I s} and a query sample image I q They all contain objects of the same type. In this way, when the number of simulation tasks in the training set and the test set is N train 、N test When , the training set and test set can be expressed as:

[0039]

[0040] It should be noted that the mask M of the image and query image is supported during training. s 、M q All are accessible and used to calculate losses to optimize network parameters. During testing, only the mask M that supports the image can be accessed s .

[0041] In one embodiment of the present application, the segmentation and matching problems are decoupled, and a simple mask matching is used to operate on the support samples and a set of query mask suggestions to obtain the query segmentation result. The advantages of this are:

[0042] (1) This few-to-few matching paradigm frees the matching task from the segmentation module, allowing us to focus on the matching problem itself;

[0043] (2) It reduces the training complexity and only requires simple “few-to-few” matching to solve the small sample segmentation problem;

[0044] (3) Because the matching and segmentation modules are separated, the learning of the two modules will not affect each other, and it is also more convenient to examine the matching and segmentation performance.

[0045] Figure 1 A flowchart of a training method for a railway intrusion target semantic segmentation model based on cycle consistency is provided in an embodiment of the present application. Figure 1 As shown, the present application provides a training method for a cycle-consistent railway intrusion target semantic segmentation model, which completes each simulation task through the following steps:

[0046] S1. Extract query sample features of the query sample image and k support sample features of the k support sample images through a feature extraction network.

[0047] Here, a ResNet network is provided. The feature extraction network includes an input layer, a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer. The input layer includes a convolution sublayer, a batch normalization sublayer, an activation sublayer, and a pooling sublayer. The first residual layer, the second residual layer, the third residual layer, and the fourth residual layer have different bottleneck blocks in each layer, but the basic composition is 1×1 convolution for dimensionality reduction, general convolution for feature processing, and finally 1×1 convolution for dimensionality increase. The output of each layer can be recorded as {F i}, i = 1, 2, 3, 4. F i The superscript of is the number of special residual layers.

[0048] The query sample feature may include the first query sample feature output by the first residual layer The second query sample features output by the second residual layer The third query sample features output by the third residual layer The fourth query sample features output by the fourth residual layer The support sample features include the first support sample features output by the first residual layer The second support sample features output by the second residual layer The third support sample features output by the third residual layer The fourth support sample features output by the fourth residual layer

[0049] S2. Input the query sample into the segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module.

[0050] The segmentation module is used to segment all objects in an image. The segmentation module is mainly composed of a transformer decoder. Specifically, the segmentation module can output n mask suggestions in the following ways:

[0051] The second query sample features The third query sample features And the fourth query sample features The first query sample feature and the first learnable parameter are input into the first decoding layer to obtain the second learnable parameter. The second query sample feature, the third query sample feature and the fourth query sample feature and the second learnable parameter are input into the second decoding layer to obtain the third learnable parameter. The second query sample feature, the third query sample feature and the fourth query sample feature and the third learnable parameter are input into the third decoding layer to obtain the fourth learnable parameter. The product of the fourth learnable parameter and the first query sample feature is calculated, and n mask suggestions are output through the normalization unit.

[0052] First, use the standard transformer decoder to calculate the second query sample features The third query sample features And the fourth query sample features The transformer decoder consists of three consecutive transformer decoding layers, each of which uses the output of the previous layer as input. Each transformer decoding layer can be expressed as:

[0053]

[0054] In the formula, E l and E l+1 Represent the N learnable embedding vectors before and after the transformer. Tlayer represents the transformer decoding layer. Finally, the output of the transformer decoder is combined with Multiply them together to get N mask proposals, and use the sigmoid function to normalize all mask proposals to the grayscale output of [0, 1].

[0055] S3. Perform self-alignment processing on the query sample feature and multiple supporting sample features respectively through the feature alignment module, and cross-align the query sample feature after self-alignment with the k-th supporting sample feature to obtain the k-th supporting cross feature and the query cross feature.

[0056] The feature alignment module may include a self-alignment module and a cross-alignment module.

[0057] In step S3, the self-alignment module can perform self-alignment processing on the features in the following manner:

[0058] For each feature of the second query sample feature, the third query sample feature, and the fourth query sample feature, average them in the channel dimension to obtain the corresponding average feature. Calculate the product between the average feature and the second query sample feature / the third query sample feature / the fourth query sample feature as the corresponding weighted coefficient feature. Multiply the feature value on each channel of the second query sample feature / the third query sample feature / the fourth query sample feature by the corresponding coefficient value in the weighted coefficient feature to obtain the self-aligned second query sample feature / the third query sample feature / the fourth query sample feature.

[0059] Here, we take the query sample feature as an example to explain the steps of implementing feature self-alignment in the self-alignment module. The self-alignment module is a parameter-free module that regularizes the input features. The input can be the second query sample feature, the third query sample feature, and the fourth query sample feature. For each layer of the input query sample feature First, average it in the channel dimension to get the corresponding average feature Then multiply the two together to get the weighted coefficient feature in the channel dimension

[0060]

[0061] Finally, according to A i The input query sample features Multiply the corresponding weighted coefficient features on different channels to obtain the self-aligned

[0062] The steps for self-alignment of support sample features are similar to those for self-alignment of query sample features and will not be described in detail here.

[0063] The cross alignment module is used to reduce the deviation between different images. Two weight-shared transformer decoders are input in parallel, into which the proposed novel global-local coupling based cycle-consistent lightweight Transformer unit is introduced.

[0064] like Figure 2 As shown, the cross alignment module can obtain query cross features in the following ways:

[0065] S100, cutting the self-aligned query sample features and the kth support sample features into a plurality of first query feature slices of hw×d, N p ×d multiple second query feature slices, hw×d multiple first support feature slices and N p ×d multiple second supporting feature slices, N p =HW / (hw).

[0066] The cross-alignment module uses a global-local coupling based on cycle consistency lightweight transformer unit, introduces cycle consistency constraints in the attention mechanism, and is used in feature cross-alignment. Take as an example to explain the implementation steps of the cross alignment module:

[0067] against Divide into N p =HW / (hw) feature sub-blocks, and then reorganized into N p ×hw×d feature blocks, and the calculation is carried out in two ways: local and global.

[0068] S101 . For each first query feature slice, compare it with the corresponding first support feature slice through a local multi-head Transformer unit based on cycle consistency to obtain the first query feature slice after local information interaction.

[0069] In the local area, several hw×d feature slices are passed through a multi-head Transformer unit based on cycle consistency. Since these feature slices correspond to the adjacent areas of the original input features, the local information interaction within each feature sub-block is enhanced, so they belong to the local area.

[0070] S102 . For each second query feature slice, compare it with the corresponding second support feature slice through a global multi-head Transformer unit based on cycle consistency to obtain a second query feature slice after global information interaction.

[0071] Globally, several shapes are N p The feature slices of ×d are processed by multi-head Transformer units based on cycle consistency. These feature slices are equivalent to the features at the same position of each original feature sub-block. The interaction of these information can enhance the expression of global information, so they belong to the global.

[0072] The number of feature slice combinations can be used as a hyperparameter to adjust the complexity and performance of the module.

[0073] S103: Reconstruct the first query feature fragments after all local information interactions to obtain local reconstructed query features.

[0074] S104: Reconstruct the second query feature fragments based on all global information interactions to obtain a global reconstructed query feature.

[0075] The reorganization here can sequentially splice the first / second query feature slices after each interaction.

[0076] S105 , coupling and obtaining query cross features based on the local reconstructed query features and the global reconstructed query features.

[0077] Finally, after addition and convolution, the local reconstruction query feature and the global reconstruction query feature are coupled to obtain the output feature.

[0078] Self-aligned Two weight-sharing transformer decoders are input in parallel, into which the proposed novel global-local coupling based cycle-consistent lightweight Transformer unit is introduced. It can be expressed as:

[0079]

[0080]

[0081] In the formula, express The query cross features after cross alignment, express The cross-aligned support cross features, MLP represents multi-layer perceptron, and LcycMHAtten represents the proposed new global-local coupling based on cycle-consistent lightweight transformer unit operation.

[0082] in, The cross alignment step is similar to the above steps, except that in step S101 and step S102, the supporting feature slices need to be used as input to be compared with the query feature slices.

[0083] In this way, the query cross features can be obtained in step S3 And get support cross-features

[0084] S4. Determine the supporting prototype based on k query cross features and n mask suggestions through the mask matching module.

[0085] The mask matching module can be used to cross-feature each query The corresponding mask proposal is combined with mask global average pooling (MGAP), and the results of the 2nd, 3rd, and 4th layers are superimposed together to finally obtain a supporting prototype.

[0086] S5. Determine n query prototypes based on k supporting cross features and corresponding standard mask results through a mask matching module.

[0087] The mask matching module can be used on each cross-feature Combined with the corresponding standard mask results, mask global average pooling (MGAP) is performed, and the results of the 2nd, 3rd, and 4th layers are superimposed together to finally obtain n query prototypes. where d is Dimension.

[0088] S6. Determine a predicted mask result corresponding to the query sample image based on the supporting prototypes, the n query prototypes and the n mask suggestions through a mask matching module.

[0089] Measuring with cosine distance The similarity between them. A fully connected layer is used to fuse the corresponding masks. The detailed relationship can be expressed as:

[0090]

[0091]

[0092] in, Represents the final predicted mask result, M represents N mask suggestions, S represents similarity, and MLP is the fully connected layer. The N similarities are used as the input of the fully connected layer, and its output is used to weighted average the N mask suggestions to obtain the predicted mask result.

[0093] S7. Based on the predicted mask results, standard mask results and target loss function corresponding to the query sample image, the model parameters are optimized to train and generate a railway intrusion target semantic segmentation model.

[0094] Since the algorithm is divided into two stages: potential target segmentation and mask matching, in step S7, the loss function here also consists of two parts.

[0095] In the potential target segmentation stage, the segmentation loss function proposed by Mask2Former is adopted, denoted as L p The Hungarian algorithm is used to match the predicted multiple mask proposals with the true value, and only the mask with the best matching effect is supervised by Dice Loss.

[0096] In the mask matching stage, the prediction results are obtained Find Dice Loss as the main loss, denoted by L M . And designed the loss function L for reverse prediction back and the loss function L based on cycle consistency cycle It is used as an auxiliary loss to constrain the cross alignment module.

[0097] The loss function L for reverse prediction back The aligned query features are averaged using the true labels and concatenated with the support features to predict the support segmentation map, which is then compared with the true value of the support mask to obtain the loss.

[0098] Based on the cycle consistent loss function L cycle The process of establishing the cycle consistent constraint is as follows Figure 3 shown.

[0099] For a set of support and query features {F s ,F q}, first flatten them into a HW×d shape, and then calculate the cosine similarity between the two feature points:

[0100]

[0101] Get the similarity matrix D∈R WH×WH , then the cyclic consistency between the support features and the query features is expressed as follows:

[0102] (a) For each row m in D, obtain the index set N = {n1,n2,…,n k};

[0103] (b) For each column in D Find the index corresponding to the maximum similarity value in its column Then the maximum values ​​found by k indexes constitute the index set

[0104] (c) If the cycle consistency is satisfied, the starting feature point k transition points With k target feature points Have the same semantics.

[0105] Similarly, the corresponding cycle-consistent relationship can be obtained from each column. For feature point pairs that satisfy cycle consistency, it is necessary to explicitly guide the network model to enhance the similarity between them, so as to align the query and support features. Using pixel-level label information, the following cycle-consistent loss function is proposed to achieve the above purpose:

[0106]

[0107] Among them, the first fraction in the logarithm corresponds to the circular consistency starting from the query feature, and the second fraction corresponds to the circular consistency starting from the support feature.

[0108] Therefore, the final loss function can be expressed as:

[0109] L=L p +λ1L M +λ2L back +λ3L cycle

[0110] Among them, λ1, λ2 and λ3 are constants, which can be adjusted to make the algorithm show better results.

[0111] The embodiment of the present application provides a training method for a cycle-consistent railway intrusion target semantic segmentation model, which realizes pixel-level refined perception of the intrusion target. By using the pixel-level support image semantic segmentation labels, the similarity relationship between the support samples and the query samples is mined, thereby realizing the pixel-level mask prediction of the intrusion target. This fine perception capability can effectively capture the target under low-resolution and long-distance conditions, thereby significantly reducing the missed detection rate and improving the stability and safety of the system in complex railway environments.

[0112] Secondly, the small sample segmentation algorithm is designed to address the uneven distribution of multi-category intrusion targets. It not only focuses on common targets such as pedestrians targeted by common algorithms, but also performs well on other scarce intrusion categories, allowing the model to still have good generalization capabilities even in the case of category imbalance.

[0113] At the same time, starting from the cycle consistency theory, a global-local lightweight Transformer module based on the cycle consistent attention mechanism and cycle consistent loss constraints are designed to further explore the similarity relationship between the support samples and the query samples. The idea of ​​mask matching is used to decouple the segmentation and matching problems, focusing on the matching problem itself, reducing the training complexity and making it easier to examine the performance of the segmentation and matching modules.

[0114] The model training method provided in this application can perceive intrusion targets at the pixel level in a refined manner, and can well distribute uneven multi-category targets, reduce the training complexity in module and network design, and significantly improve the overall performance of the intrusion target detection system in railway scenarios, achieving the effect of ensuring the safety of railway operations.

[0115] In one embodiment of the present application, a novel global-local coupling based cycle-consistent lightweight transformer unit is proposed. The following takes the query feature slice as an example to illustrate the steps of implementing global or local information interaction based on cycle-consistent lightweight transformer unit:

[0116] Specifically, Figure 4 As shown in Figure 1, based on the cycle-consistent global / local multi-head Transformer unit, the query feature slice after information interaction can be output through the following steps:

[0117] S200: Perform linear transformation on the query feature slice to obtain query sub-features.

[0118] Here the size of the query feature slice is hw×d. The input feature is transformed into the query sub-feature F through three different linear transformations. Q .

[0119] S201, perform linear transformation on the support feature slice corresponding to the query feature slice to obtain the key sub-feature F K Sum value sub-feature F V .

[0120] Among them, F Q =XW Q , F K =XW K , F V =XW V .

[0121] Among them, X is the input feature, W Q , W K , W V is a trainable weight matrix.

[0122] S202: Calculate a similarity matrix between the query sub-features and the key sub-features.

[0123] The commonly used method is to calculate their dot product, the calculation formula is:

[0124]

[0125] To prevent gradients from exploding or vanishing, the dot product result is usually divided by a scaling factor where d k It is the dimension of Key.

[0126] S203, revising the similarity matrix through element-level cyclic consistent attention to obtain a similarity optimization matrix.

[0127] Here, the similarity optimization matrix can be obtained as follows:

[0128] Based on the size of the similarity value of each column in the similarity matrix, multiple ordinate indexes are determined. Based on the size of the similarity value in the row corresponding to the multiple ordinate indexes in the similarity matrix, multiple abscissa indexes are determined. Based on the multiple ordinate indexes and the multiple abscissa indexes, multiple feature positions are determined. For each feature position, determine whether the feature value at the feature position of the query sub-feature and the feature value at the feature position of the supporting feature indicate the same semantic category. If so, determine that the value at the corresponding feature position in the weight matrix is ​​a preset multiple of the feature value at the feature position in the similarity matrix. If not, determine that the value at the corresponding feature position in the weight matrix is ​​negative infinity. Fill the remaining positions of the weight matrix with zeros to obtain a weight matrix of the same size as the similarity matrix. Calculate the sum between the similarity matrix and the weight matrix through the activation function to obtain a similarity optimization matrix.

[0129] For each column j in the similarity matrix D, select the index corresponding to the top P values ​​with the highest similarity Then find the index j′ with the greatest similarity in each row corresponding to these indices. p , p∈[1,P]. If the feature obtained after the loop Start with feature If the semantic category of the supporting feature point is the same, it is considered to satisfy the circular consistency and should make more contributions to the linear representation of the feature; otherwise, it is not satisfied and its influence should be weakened or eliminated. Therefore, the corresponding situations can be divided into three types. If the supporting feature point does not participate in the establishment of circular consistency, then ΔD i,j =0; if the supporting feature points satisfy cyclic consistency, then ΔDi,j =η|D i,j |; If the supporting feature points do not satisfy the cycle consistency, then ΔD i,j =-∞.

[0130] S204: Calculate the query feature slice after outputting information interaction based on the similarity optimization matrix, the query sub-features and the value sub-features.

[0131] Finally, the weight modification matrix ΔD can be obtained. Its modification process is:

[0132] D′=softmax(D+ΔD)

[0133] In this way, the unvisited supporting feature points will not be affected, while the contribution of the feature points that meet the cycle consistency can be amplified and the similarities that do not meet the cycle consistency can be removed.

[0134] This globally-locally coupled lightweight Transformer design does not require the original naive Transformer to calculate each feature point pairwise to reach the computational complexity of O(HW×HW), which can greatly reduce the computational overhead. In actual network construction, multiple consecutive operations can be performed end to end to balance the network's representation ability and computational complexity.

[0135] In one embodiment of the present application, the railway monitoring image may be input into a railway intrusion target semantic segmentation model to obtain a railway intrusion target prediction mask result output by the railway intrusion target semantic segmentation model.

[0136] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .

[0137] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the steps of a training method based on a cycle-consistent railway intrusion target semantic segmentation model in the above method embodiment can be executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0138] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a training method based on a cycle-consistent railway intrusion target semantic segmentation model in the above-mentioned method embodiment can be executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0141] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0143] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM) random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0144] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0145] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a cycle-consistent railway intrusion target semantic segmentation model, characterized in that: The railway intrusion target semantic segmentation model includes a feature extraction network, a segmentation module, a feature alignment module and a mask matching module. The method includes: Extract query sample features of the query sample image and k support sample features of the k support sample images respectively through a feature extraction network; Inputting the query sample into a segmentation module to obtain n mask suggestions corresponding to the query sample image output by the segmentation module; The query sample feature and the multiple support sample features are self-aligned by the feature alignment module, and the query sample feature after self-alignment is cross-aligned with the k-th support sample feature to obtain the k-th support cross feature and the query cross feature; Determining a supporting prototype based on the k query cross features and the n mask suggestions by a mask matching module; Determining n query prototypes based on the k supporting cross features and corresponding standard mask results through a mask matching module; Determine, by a mask matching module, a predicted mask result corresponding to the query sample image based on the supporting prototypes, the n query prototypes and the n mask suggestions; Based on the predicted mask results, standard mask results and target loss function corresponding to the query sample image, the model parameters are optimized to train and generate a railway intrusion target semantic segmentation model.

2. The method according to claim 1, characterized in that: The feature extraction network includes an input layer, a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer. The input layer includes a convolution sublayer, a batch normalization sublayer, an activation sublayer, and a pooling sublayer.

3. The method according to claim 2, characterized in that The query sample features include a first query sample feature output by the first residual layer, a second query sample feature output by the second residual layer, a third query sample feature output by the third residual layer, and a fourth query sample feature output by the fourth residual layer; the support sample features include a first support sample feature output by the first residual layer, a second support sample feature output by the second residual layer, a third support sample feature output by the third residual layer, and a fourth support sample feature output by the fourth residual layer. The segmentation module outputs n mask proposals in the following way: Inputting the second query sample feature, the third query sample feature, the fourth query sample feature and the first learnable parameter into the first decoding layer to obtain a second learnable parameter; Inputting the second query sample feature, the third query sample feature, the fourth query sample feature and the second learnable parameter into the second decoding layer to obtain a third learnable parameter; Inputting the second query sample feature, the third query sample feature, the fourth query sample feature and the third learnable parameter into the third decoding layer to obtain a fourth learnable parameter; The product of the fourth learnable parameter and the first query sample feature is calculated, and n mask suggestions are output through a normalization unit.

4. The method according to claim 2, characterized in that: The feature alignment module performs self-alignment processing on features in the following manner: For each feature of the second query sample feature, the third query sample feature, and the fourth query sample feature, average the feature in the channel dimension to obtain a corresponding average feature; Calculate the product of the average feature and the second query sample feature / the third query sample feature / the fourth query sample feature as the corresponding weighted coefficient feature; The feature value on each channel of the second query sample feature / the third query sample feature / the fourth query sample feature is multiplied by the corresponding coefficient value in the weighted coefficient feature to obtain the self-aligned second query sample feature / the third query sample feature / the fourth query sample feature.

5. The method according to claim 2, characterized in that: The size of the query sample feature and the support sample feature are both H×w×d. The feature alignment module can obtain the query cross feature in the following way: Cut the query sample features and the kth support sample features after self-alignment into multiple first query feature slices of hw×d, N p ×d multiple second query feature slices, hw×d multiple first support feature slices and N p ×d multiple second supporting feature slices, N p =HW / (hw); For each first query feature slice, compare it with the corresponding first support feature slice through a local multi-head Transformer unit based on cycle consistency to obtain the first query feature slice after local information interaction; For each second query feature slice, compare it with the corresponding second support feature slice through a global multi-head Transformer unit based on cycle consistency to obtain the second query feature slice after global information interaction; Based on the first query feature piece after all local information interactions, reorganize to obtain a local reconstructed query feature; Based on the second query feature pieces after all global information interactions, reorganize to obtain the global reconstructed query feature; Based on the local reconstruction query feature and the global reconstruction query feature, the query cross feature is obtained through coupling.

6. The method according to claim 5, characterized in that Based on the cycle-consistent global / local multi-head Transformer unit, the query feature slice after information interaction is output through the following steps: Perform linear transformation on the query feature slice to obtain the query sub-feature; Perform linear transformation on the support feature slice corresponding to the query feature slice to obtain key sub-features and value sub-features; Calculating a similarity matrix between the query sub-features and the key sub-features; The similarity matrix is ​​revised through element-level cyclic consistent attention to obtain a similarity optimization matrix; Based on the similarity optimization matrix, the query sub-features and the value sub-features, the query feature slices after the output information interaction are calculated.

7. The method according to claim 6, characterized in that The similarity optimization matrix is ​​obtained by: Determining a plurality of ordinate indexes based on the size of the similarity value of each column in the similarity matrix; Determine multiple horizontal coordinate indexes based on the magnitudes of similarity values ​​in the rows corresponding to the multiple vertical coordinate indexes in the similarity matrix; Determine a plurality of feature positions based on a plurality of ordinate indices and a plurality of abscissa indices; For each feature position, determine whether the feature value of the query sub-feature at the feature position and the feature value of the supporting feature at the feature position indicate the same semantic category; if so, determine that the value at the corresponding feature position in the weight matrix is ​​a preset multiple of the feature value at the feature position in the similarity matrix; if not, determine that the value at the corresponding feature position in the weight matrix is ​​negative infinity; Fill the remaining positions of the weight matrix with zeros to obtain a weight matrix of the same size as the similarity matrix; The sum of the similarity matrix and the weight matrix is ​​calculated through an activation function to obtain a similarity optimization matrix.

8. The method according to claim 1, characterized in that: The method also includes inputting the railway monitoring image into a railway intrusion target semantic segmentation model to obtain a railway intrusion target prediction mask result output by the railway intrusion target semantic segmentation model.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the training method of the railway intrusion target semantic segmentation model based on cycle consistency as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the training method of the railway intrusion target semantic segmentation model based on cycle consistency are performed as claimed in any one of claims 1 to 7.

Images