Rail transit train fault sample data generation method and device, equipment and medium

Abstract

The application discloses a rail transit train fault sample data generation method, device, equipment and medium, and the method steps comprise: obtaining positive sample data and negative sample data of a target train as original image data; the original image data is segmented and cropped to form an image data set; the image data set is input into a cyclic generative adversarial network model for training, the cyclic generative adversarial network model is formed by two GAN networks, and each GAN network comprises a generator and a discriminator; target positive sample data and a fault position to be generated are obtained, a background image and a target region image are segmented and cropped from the positive sample data, are input into the trained model to generate a corresponding negative sample target region image, and are fused with the segmented background image to obtain final fault sample data. The application has the advantages of simple implementation method, high generation efficiency and quality, strong flexibility and the like.

Description

Technical Field

[0001] This invention relates to the field of rail transit technology, and in particular to a method, apparatus, equipment and medium for generating fault sample data of rail transit trains. Background Technology

[0002] In fault identification in the rail transit field, due to the scarcity of real faults, the performance of deep learning detection algorithms cannot achieve the expected results. This invention constructs a neural network to learn the correlation between positive and negative samples. Only a small number of fault samples need to be trained to realize the conversion between positive and negative samples, thereby expanding the amount of scarce sample data for training deep learning detection models and improving the detection performance of the models.

[0003] In the rail transit sector, trains may develop safety hazards due to prolonged operation, requiring regular inspections by maintenance personnel. Intelligent inspection robots, utilizing AGV robots as carriers and combining cameras with machine learning algorithms, process and analyze the data acquired through perception using machine learning models. This enables intelligent real-time monitoring of equipment operating status and timely detection of potential faults. Machine learning algorithms rely on a large amount of labeled training data, and testing and validating algorithm performance also requires numerous data samples. However, in rail transit trains, only positive sample data is typically available, making it difficult to collect real fault sample data. Current technologies typically employ data augmentation methods such as flipping, rotating, translating, cropping, deforming, adding noise, and blurring to obtain negative samples. However, these methods generate a limited variety of negative samples, failing to meet the requirements for data diversity. Summary of the Invention

[0004] The technical problem to be solved by this invention is: In view of the technical problems existing in the prior art, this invention provides a method, apparatus, equipment and medium for generating fault sample data of rail transit trains that is simple to implement, efficient and of high quality, and highly flexible. It can make full use of positive sample data in rail transit trains to generate fault sample data, effectively ensure the diversity of sample data, reduce the difficulty of sample generation, and improve the quality of sample data generation.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0006] A method for generating fault sample data for rail transit trains, comprising the following steps:

[0007] The positive and negative sample data of the target train are acquired and used as the raw image data, wherein the negative sample data corresponds to the fault sample data.

[0008] Each image in the original image data is segmented and cropped to extract image slices with negative sample fault locations and corresponding positive sample locations, forming an image dataset;

[0009] The image dataset is input into a pre-built recurrent generative adversarial network (GAN) model for training, resulting in a trained GAN model. The GAN model is constructed from two GANs, each consisting of a generator and a discriminator. The two generators are used to learn the mapping relationship between positive and negative samples, respectively, to convert positive samples to negative samples and vice versa. The two discriminators are used to distinguish between real positive and negative sample data and positive and negative sample data generated by the generator to predict the authenticity of the sample data.

[0010] Obtain the required target positive sample data and the fault locations to be generated. Based on the fault locations to be generated, crop and segment the background image and target region image from the target positive sample data, input them into the trained recurrent generative adversarial network model to generate the corresponding negative sample target region image, and fuse the generated negative sample target region image with the segmented background image to obtain the final fault sample data.

[0011] Furthermore, in each generator of the recurrent generative adversarial network model, a self-attention module is set in series before the residual module. The feature map extracted by the encoder in the generator is weighted and summed with the feature map transformed by the residual module after passing through the self-attention module to obtain the final feature map.

[0012] Furthermore, the execution steps of the self-attention module in the recurrent generative adversarial network model include:

[0013] Multiply each feature map by W. q W k W v The coefficients correspond to Q, K, and V, where Q, K, and V represent the query, key, and value, respectively.

[0014] Based on Q, K, and V, an attention matrix A is calculated for each feature map, where A = K. T Q, where each value in the attention matrix A is the attention size α(i,j) of the two corresponding input feature maps;

[0015] The attention matrix A is normalized using the Softmax function to obtain matrix A';

[0016] Calculate the output matrix O of the self-attention layer corresponding to each input feature map based on the obtained matrices A' and V;

[0017] The output matrix O is weighted with the input feature map to obtain a feature map output with global dependencies at any two positions.

[0018] Furthermore, each discriminator in the recurrent generative adversarial network model is constructed from a PatchGAN network and a self-attention module. When each discriminator receives an input image, it first calculates the attention weight at each position through the attention module, and then uses an activation function to normalize each attention weight into a probability distribution so that the sum of the attention weights at each position is 1. Based on the calculated attention weights at each position and the target position, a weight mask is generated and weighted and output to segment the target region position. The segmented image is then input into the PatchGAN network to calculate the predicted probability value of the sample's authenticity. If the probability value is close to 1, it is a real sample; if the probability value is close to 0, it is a fake sample.

[0019] Furthermore, the overall optimization objective function of the recurrent generative adversarial network model during training includes adversarial loss and cycle consistency loss. The cycle consistency loss includes positive cycle loss and inverse cycle loss. The positive cycle loss is the difference between the image G(x) generated by generator G and the original image x after being transformed by generator F. The inverse cycle loss is the difference between the image F(y) generated by generator F and the original image y after being transformed by generator G. Generator G is a generator used to transform positive sample X into negative sample Y, and generator F is a generator used to transform negative sample Y into positive sample X.

[0020] Furthermore, the overall objective function is calculated as follows:

[0021] Loss = Loss GAN (G,D Y (X,Y)+Loss GAN (F,D X Loss cyc (G,F)

[0022]

[0023] Where Loss is the overall optimization objective function, Loss GAN Let G be the adversarial loss function, x be an image from a positive sample X, y be an image from a negative sample Y, G be the generator that transforms a positive sample X into a negative sample Y, F be the generator that transforms a negative sample Y into a positive sample X, and D be the generator that transforms a negative sample Y into a positive sample X. X and D Y For the corresponding discriminator, Losscyc The cycle-consistent loss function is expressed as:

[0024] Loss cyc =E|F(G(x)-x)|+E|G(F(y)-y)|.

[0025] Further, the step of inputting the image dataset into a pre-built recurrent generative adversarial network (RGAN) model for training to obtain the trained RGAN model includes:

[0026] First, lock two discriminator networks, Dx and Dy, and train two generator networks, G and F, such that the difference between the probability that the images generated by the two generators are judged as real by the discriminator networks Dx and Dy and 1 is less than a preset threshold.

[0027] Then, open two discriminator networks, Dx and Dy, and train the two discriminator networks until they determine that the real data sample is true.

[0028] Then re-lock the two discriminator networks Dx and Dy, and train the two generator networks G and F until the difference between the probability that the output image is judged as real by the discriminator networks Dx and Dy and 1 is less than the preset threshold.

[0029] Repeatedly train the two generator networks G and F and the two discriminator networks Dx and Dy until the two discriminators can no longer distinguish between the true and false outputs of the two generators. The training is then complete, and the weight file of the currently trained recurrent generative adversarial network model is output.

[0030] Furthermore, in the process of locking the two discriminator networks Dx and Dy and training the two generator networks G and F, the optimization objective function used is:

[0031]

[0032] The two discriminator networks Dx and Dy are opened. The optimization objective function used during the training of these two discriminator networks is:

[0033]

[0034] Among them, Loss GAN Let G be the adversarial loss function, x be an image from a positive sample X, y be an image from a negative sample Y, G be the generator that transforms a positive sample X into a negative sample Y, F be the generator that transforms a negative sample Y into a positive sample X, and D be the generator that transforms a negative sample Y into a positive sample X. X and D Y This is the corresponding discriminator.

[0035] Furthermore, the step of fusing the generated negative sample target region map with the segmented background map to obtain the final fault sample data includes:

[0036] The generated negative sample target region map is converted into a negative sample image patch with the same resolution as the target region map after inverse size normalization.

[0037] The negative sample image blocks obtained after conversion are stitched together with the background image obtained by cropping and segmentation to form a complete fault sample data output.

[0038] Furthermore, it also includes using the FID metric to evaluate the quality of the generated fault sample data, so as to obtain fault sample data whose FID (Fréchet Inception Distance) metric meets the preset requirements as the final generated fault sample data.

[0039] A device for generating fault sample data for rail transit trains, comprising:

[0040] The data acquisition module is used to acquire the positive sample data and negative sample data of the target train and use them as raw image data. The negative sample data corresponds to the fault sample data.

[0041] The data cropping module is used to segment and crop each image in the original image data to crop image slices with negative sample fault locations and corresponding positive sample locations, forming an image dataset;

[0042] The model training module is used to input the image dataset into a pre-built recurrent generative adversarial network (GAN) model for training, thereby obtaining the trained GAN model. The GAN model is constructed from two GAN networks, each of which includes a generator and a discriminator. The two generators are used to learn the mapping relationship between positive and negative samples, respectively, to realize the conversion from positive to negative samples and from negative to positive samples. The two discriminators are used to distinguish between real positive and negative sample data and positive and negative sample data generated by the generator in order to predict the authenticity of the sample data.

[0043] The sample generation module is used to obtain the required target positive sample data and the fault location to be generated. Based on the fault location to be generated, the background image and target region image are cropped and segmented from the target positive sample data and input into the trained recurrent generative adversarial network model to generate the corresponding negative sample target region image. The generated negative sample target region image is then fused with the segmented background image to obtain the final fault sample data.

[0044] An electronic device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0045] A computer-readable storage medium storing a computer program that, when executed, implements the method described above.

[0046] Compared with the prior art, the advantages of the present invention are as follows:

[0047] 1. This invention utilizes two generators in a recurrent generative adversarial network to learn the mapping relationship between positive and negative samples, thereby realizing the conversion between positive and negative samples. This makes it applicable to the field of rail transit to realize the conversion from positive to negative samples, and can also realize the process of restoring positive samples from negative samples. In this way, it can make full use of positive sample data to quickly generate fault sample data, effectively increase the diversity of negative samples, and solve the problem of scarce negative sample data in traditional rail transit trains.

[0048] 2. This invention adopts a method of generating local images based on positive samples. By segmenting the target region, converting it into negative samples through a model, and then fusing it with the background before segmentation, it can quickly and accurately generate the required fault sample data while preserving background information to the greatest extent, thus solving the problem of data distortion caused by traditional data generation methods.

[0049] 3. This invention further enhances the detail and overall consistency of image generation by introducing an attention mechanism into a recurrent generative adversarial network. This allows for the modeling of long-distance, multi-level dependencies across image regions. It also enables the fine details of the generated image at each location to be coordinated with the fine details of the surrounding parts of the image, rather than being limited to the local neighborhood. Furthermore, by weighting the attention mechanism into the residual module in the generator, the dependence on external information can be reduced, making it better able to capture the internal correlations of data or features. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the implementation process of the rail transit train fault sample data generation method in this embodiment.

[0051] Figure 2 This is a schematic diagram illustrating the principle of the recurrent generative adversarial network constructed in this embodiment.

[0052] Figure 3 This is a schematic diagram illustrating the implementation principle of the generator model used in this embodiment.

[0053] Figure 4 This is a schematic diagram illustrating the implementation principle of the discriminator model used in this embodiment.

[0054] Figure 5This is a schematic diagram illustrating the implementation process of generating fault images based on positive samples in this embodiment.

[0055] Figure 6 This is a schematic diagram illustrating the experimental results obtained in a specific application embodiment of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0057] Images acquired during the monitoring of rail transit trains typically feature high resolution, small targets, and complex and variable backgrounds. Considering the scarcity and difficulty in acquiring real fault samples, this invention employs a recurrent generative adversarial network (RBAN) consisting of two generative adversarial networks (GANs) to achieve the transformation between positive and negative samples. The two generators in the RBAN learn the mapping relationship between positive and negative samples, enabling the transformation. This ensures that the image after the forward transformation (source → target → source) is as close as possible to the original source image, and the image after the reverse transformation (target → source → target) is as close as possible to the original target image. This achieves both the transformation of positive to negative samples in rail transit and... This method enables the process of restoring positive samples from negative samples, thereby fully utilizing positive sample data to quickly generate fault sample data. It transforms positive samples into fault samples, effectively increasing the diversity of negative samples and solving the problem of scarce negative sample data in traditional rail transit trains. At the same time, considering the characteristics of images acquired by rail transit trains, it first determines the target region map by cropping the positive sample data, and then combines the trained recurrent generative adversarial network model to generate the corresponding negative sample target region map. Finally, it restores the original image, which can generate local images based on positive samples, thereby accurately generating the required fault sample data. It can retain background information to the greatest extent and solve the problem of data distortion caused by traditional data generation methods.

[0058] like Figure 1 As shown, the steps of the method for generating fault sample data for rail transit trains in this embodiment include:

[0059] Step S01. Acquire the positive and negative sample data of the target train and use them as the original image data. The negative sample data corresponds to the fault sample data.

[0060] In this embodiment, historical monitoring data of the target rail transit train is obtained as raw image data, which includes positive sample data and a small amount of fault sample data. For example, it can be normal sample images of the train undercarriage and a small amount of negative sample image data collected by an intelligent inspection robot.

[0061] Step S02. Segment and crop each image in the original image data to crop image slices with negative sample fault locations and corresponding positive sample locations, forming an image dataset.

[0062] In this embodiment, the original image is segmented and cropped by cropping the fault locations of negative samples and the corresponding positive sample locations to form image slices, thereby establishing an image dataset that can be used for model training.

[0063] Since there are few real fault samples, alternatively, traditional data augmentation methods (rotation, translation) can be used to add some negative sample data, and then the positive sample data and negative sample data are stored in different locations to serve as an unpaired image dataset.

[0064] Step S03. Input the image dataset into the pre-built CycleGAN model for training to obtain the trained CycleGAN model. The CycleGAN model is constructed from two GAN networks. Each GAN network includes a generator and a discriminator. The two generators are used to learn the mapping relationship between positive and negative samples, respectively, to realize the conversion from positive samples to negative samples and the conversion from negative samples to positive samples. The two discriminators are used to distinguish between real positive and negative sample data and positive and negative sample data generated by the generator to predict the authenticity of the sample data.

[0065] In this embodiment, the image dataset is input into the network model for training. The model is optimized based on the loss functions of the generator and the discriminator. The generator and the discriminator evolve together in an alternating iterative game until the generator learns the data distribution of the real samples and the discriminator can no longer distinguish between the data generated by the generator and the corresponding real data, thus obtaining the trained weight file.

[0066] In this embodiment, the CycleGAN network adopts a generative adversarial network based on the PyTorch framework, such as... Figure 2As shown, the system consists of two GAN (Generative Adversarial Network) networks used for cyclically constrained generators. Each GAN network contains a generator network, and each generator is associated with a discriminator. The two GANs share two generators: a generator G that maps positive samples X to negative samples Y, and a generator F that maps negative samples Y to positive samples X. These two generators learn the mapping relationship between two positive and negative samples and convert the input image into the target image. The discriminator network includes a discriminator Dy that distinguishes between real negative sample images Y and negative sample images generated by generator G, and a discriminator Dx that distinguishes between positive sample images X and positive sample images generated by generator F. These two discriminators evaluate the authenticity of the generated images and predict the probability that the input is a real image. In other words, generator G represents the process of generating corresponding negative samples from positive samples, generator F represents the process of generating corresponding positive samples from negative samples, and the Dx and Dy discriminator networks respectively determine the authenticity of the generated positive and negative samples. X and Y represent the positive sample image and the corresponding negative sample image, respectively. The above network is a recurrent network. The entire network has a dual structure, which can realize the transformation from positive sample X to negative sample Y, and also realize the transformation from negative sample Y to positive sample X.

[0067] In this embodiment, each generator in the recurrent generative adversarial network model has a self-attention module cascaded before the residual module. The feature map extracted by the encoder in the generator is weighted and summed with the feature map transformed by the residual module after passing through the self-attention module to obtain the final feature map. That is, adding a self-attention module before the residual module in the generator enables the model to "focus" on the most important parts of the image, just like a human, and give these regions more weight during the transformation process. Furthermore, the self-attention module uses the ReLU function as the activation function for weight allocation to achieve weight normalization and focusing effect. Since the ReLU function is sparsity, all x values ​​less than 0 will become y values ​​after being activated by the ReLU function, which ensures that the features of irrelevant background regions remain unchanged when the generator's feature map is transformed by the residual network.

[0068] As an optional implementation method, such as Figure 3As shown, the generator model adopts the UNet (a convolutional neural network for image segmentation) structure, with a self-attention module cascaded before the residual module. The input data is image data, and the goal is to generate sufficiently realistic target samples to deceive the discriminator. The generator aims to generate samples that will cause the discriminator to output a probability value close to 1. To achieve this goal, the generator receives feedback from the discriminator (i.e., the discriminator's judgment probability of its generated samples) and adjusts its parameters accordingly. In other words, the generator tries to minimize the probability that its generated samples will be identified as fake by the discriminator.

[0069] Specifically, the generator comprises an encoder and a decoder, connected in a U-shaped structure. The encoder employs CIL (Cov convolution, IN regularization, Leaky ReLU activation function) and uses an image enhancement module (ReflectionPad2d) to symmetrically enhance the image along its edges, increasing resolution. Residual blocks are used in the connection regions between the two structures, employing nine repeated blocks to restore and enhance the data, addressing the vanishing gradient problem in deep neural networks. A self-attention module is cascaded with the nine residual blocks. The decoder uses deconvolution, IN normalization, and ReLU activation function to restore the image size. Finally, ReflectionPad2d further enhances the image resolution, and convolution restores the image to its original size, effectively resolving the issue of object edge information.

[0070] As an optional implementation, the image enhancement module (ReflectionPad2d) can employ PyTorch's function for two-dimensional data reflection padding, expanding the data size by mirroring boundary pixel values. For the input image, the image enhancement module performs reflection padding around the data according to a specified padding size. Specifically, the padding pixel values ​​can be obtained by mirroring the corresponding boundary pixel values ​​with the boundary as the axis of symmetry. This method maintains boundary continuity and helps improve the performance of convolution operations.

[0071] Because rail transit images require high resolution and detail clarity, traditional GAN ​​discriminators are not suitable. As an alternative implementation method, such as... Figure 4As shown, in this embodiment, each discriminator of the recurrent generative adversarial network model is constructed from a PatchGAN network and a self-attention module. The PatchGAN network is introduced as a fully convolutional network to determine the realism of the input image through local perception. An attention mechanism is introduced to enhance specific target regions of interest while weakening irrelevant background regions. When each discriminator receives an input image, it first calculates the attention weight at each position using the attention module. Then, an activation function is used to normalize each attention weight into a probability distribution so that the sum of the attention weights at each position is 1. Based on the calculated attention weights at each position, a weight mask is generated at the target position and weighted, thus segmenting the target region. The segmented image is then input into the PatchGAN network to calculate the predicted probability value of the sample's realism. If the probability value is close to 1, it is a real sample; if the probability value is close to 0, it is a fake sample.

[0072] Specifically, a self-attention module is added to the discriminator. The input image first passes through the self-attention module to obtain an attention weight. The attention weight of each position is calculated, and then the activation function (Softmax) is used to normalize these attention scores into a probability distribution to ensure that the sum of the attention weights at each position is 1. These weights are then applied to the original image, and a weight mask is generated based on the weight magnitude and target position. The weighted output is then used to segment the target region, thereby enhancing the specific target region of interest while weakening irrelevant background regions. The segmented image is then fed into the PatchGAN network for judgment. The discriminator only performs convolution on the segmented target region positions, dividing them into image blocks of a specified size (e.g., 70×70) and inputting them into the PatchGAN network for discrimination. This reduces the impact of the complex background of the rail transit system on the discriminator. At the same time, by reducing the number of image blocks processed by the PatchGAN network, the processing efficiency of the discriminator can also be improved.

[0073] As an optional implementation, the PatchGAN network is a fully convolutional network used to determine the realism of an input image through local perception. During training, PatchGAN receives input from two sources: real samples from a real dataset and fake samples generated by the generator. It attempts to output a probability value close to 1 for real samples and a probability value close to 0 for fake samples. PatchGAN divides the input image into a series of patches. For each patch, the network provides a binary prediction indicating whether the patch belongs to the real image. The final prediction is determined by a matrix composed of the predictions for all patches.

[0074] This embodiment introduces an attention mechanism into a recurrent generative adversarial network (GAN). In the generator, the feature map of the image after passing through the encoder in UNet first passes through a self-attention module and then enters the residual module. In the discriminator, the generated image passes through the self-attention module, the target region is segmented, and then it enters the PatchGAN network. Using the self-attention module as a supplement to convolution helps to model long-distance, multi-level dependencies across image regions. Furthermore, by setting the self-attention module, the fine details of the image generated by the generator at each location can be coordinated with the fine details of the surrounding areas of the image, rather than being limited to local neighborhoods, thus effectively enhancing the detail and overall consistency of the generated image. Further, by weighting the attention mechanism into the residual module in the generator—that is, by weighting the feature map extracted by the encoder in the generator with the feature map transformed by the residual module after passing through the self-attention module—the dependence on external information can be reduced, making it better able to capture the internal correlations of data or features.

[0075] As an optional implementation, the self-attention module in a recurrent generative adversarial network model can be executed using the following steps:

[0076] Multiply each feature map by W. q W k W v The coefficients correspond to Q, K, and V, where Q, K, and V represent the query, key, and value, respectively.

[0077] For each feature map, an attention matrix A is computed based on Q, K, and V, where A = K. T • Q, where each value in the attention matrix A is the attention size α(i,j) of the two corresponding input feature maps, and the sum of the weights of all j pairs at position i is 1;

[0078] After normalizing the attention matrix A using the softmax function, the resulting matrix becomes A'.

[0079] Calculate the output matrix O of the self-attention layer corresponding to each input feature map based on the obtained matrices A' and V, i.e., O = V·A';

[0080] The output matrix O is weighted with the input feature map to obtain the feature map output with global dependencies at any two positions, i.e., y. i =λo i +x i λ is a learnable scalar initialized to 0, and the output is the original feature map. As learning progresses, a weighted attention layer is added to the original feature map X, resulting in a feature map containing global dependencies between any two locations.

[0081] This embodiment introduces a self-attention mechanism during image conversion, which can accelerate the model's learning speed, improve the quality of generated images, and effectively solve problems such as background contamination during traditional negative sample generation.

[0082] As an optional implementation, the overall optimization objective function of the recurrent generative adversarial network (RGAN) model during training includes adversarial loss and cycle consistency loss. The consistency loss ensures that the transformed image remains consistent with the original image. The cycle consistency loss includes positive cycle loss and inverse cycle loss. The positive cycle loss is the difference between the image G(x) generated by generator G and the original image x after transformation by generator F. The inverse cycle loss is the difference between the image F(y) generated by generator F and the original image y after transformation by generator G. Generator G is used to transform positive samples X into negative samples Y, and generator F is used to transform negative samples Y into positive samples X.

[0083] For example, the overall optimization objective function can be expressed as:

[0084] Loss = Loss GAN (G,D Y (X,Y)+Loss GAN (F,D X Loss cyc (G,F) (1)

[0085] The adversarial loss function between the generator and discriminator of two GAN networks can be defined as:

[0086]

[0087] Where Loss is the overall optimization objective function, Loss GAN Let G be the adversarial loss function, x be an image from a positive sample X, y be an image from a negative sample Y, G be the generator that transforms a positive sample X into a negative sample Y, F be the generator that transforms a negative sample Y into a positive sample X, and D be the generator that transforms a negative sample Y into a positive sample X. X and D Y For the corresponding discriminator, Loss represents the mathematical expectation of random variables x and y belonging to the true distribution of samples X and Y. cyc The cycle-consistent loss function is expressed as:

[0088] Loss cyc =E|F(G(x)-x)|+E|G(F(y)-y)| (4)

[0089] Where E represents the expectation function.

[0090] This embodiment introduces cycle consistency loss into the CycleGAN network. The positive cycle is x→G(x)→F(G(x))≈x, which calculates the difference between the image G(x) generated by generator G and the original image x after transformation by generator F. The negative cycle is y→F(y)→G(F(y))≈y, which calculates the difference between the image F(y) generated by generator F and the original image y after transformation by generator G. This can maintain the consistency of image transformation and ensure the preservation of information during the transformation process.

[0091] As an optional implementation, during the training process of inputting the image dataset into a pre-built recurrent generative adversarial network (RGAN) model, the generator and discriminator interact to continuously optimize the model parameters, resulting in the trained RGAN model. Specifically, the following steps can be taken:

[0092] Step S301. First, lock two discriminator networks Dx and Dy, and train two generator networks G and F, such that the difference between the probability that the images generated by the two generators are judged as real by the discriminator networks Dx and Dy and 1 is less than a preset threshold.

[0093] Step S302. Then open two discriminator networks Dx and Dy, and train the two discriminator networks Dx and Dy until they determine that the real data sample is true;

[0094] Step S303. Re-lock the two discriminator networks Dx and Dy, and train the two generator networks G and F until the difference between the probability that the output image is judged as real by the discriminator networks Dx and Dy and 1 is less than the preset threshold.

[0095] Step S304. Repeat steps S301 to S303 to re-train the two generator networks G and F and the two discriminator networks Dx and Dy until the two discriminators can no longer determine whether the outputs of the two generators are true or false. Complete the training and output the weight file of the currently trained recurrent generative adversarial network model.

[0096] Specifically, during model training, the discriminator networks Dx and Dy are first locked, while the generator networks G and F are trained. The goal is to generate images that can fool the discriminator networks Dx and Dy; that is, the more realistic the generated images, the better, and the closer the probability of the discriminator networks Dx and Dy classifying them as real is to 1. Then, the Dx and Dy networks are opened, and the discriminator networks Dx and Dy are trained to identify fake images (generated images). The discriminator's goal is to judge real data samples as real and images generated by the generator as fake. Then, the Dx and Dy networks are locked again, and the generator networks G and F are trained again, so that the output images can fool the discriminator networks Dx and Dy. This process is repeated, involving continuous adversarial analysis, optimization, iteration, and updates, until the discriminator networks can no longer distinguish between real and fake generator outputs, ultimately achieving a highly realistic effect, thus completing the training.

[0097] As an alternative implementation, during the training of two generator networks, G and F, with two discriminator networks Dx and Dy locked, the optimization objective function can be:

[0098]

[0099] As an optional implementation, two discriminator networks, Dx and Dy, are provided. The optimization objective function used during the training of these two discriminator networks is:

[0100]

[0101] Among them, Loss GAN Let G be the adversarial loss function, x be an image from a positive sample X, y be an image from a negative sample Y, G be the generator that transforms a positive sample X into a negative sample Y, F be the generator that transforms a negative sample Y into a positive sample X, and D be the generator that transforms a negative sample Y into a positive sample X. X and D Y This is the corresponding discriminator.

[0102] In the training process of a recurrent generative adversarial network (GAN) model, positive samples X are repeatedly input in the forward loop, and the positive samples X are transformed into negative samples Y by the generator G. ’ Then, the discriminator Dy is used to identify it, and then the generator F converts it into the generated X. ’ Then use the discriminator Dx to distinguish X. ’ Similarly, the negative sample Y is input in reverse loop, and the generator generates negative samples Y twice. ’ The discriminator Dy is used to determine the authenticity of the generated negative sample Y. After the optimizer continuously adjusts the optimization parameters, the discriminator Dy eventually becomes unable to distinguish the generated negative sample Y. ’ Dx cannot identify the generated positive sample X. ’ The authenticity is verified, and finally the trained generator models Dx and Dy are obtained.

[0103] Understandably, besides using the UNet architecture, the generator model can also employ other backbone network structures, such as UNet++, TransUNet, and SegNet. Similarly, the discriminator model can use other structures besides PatchGAN, such as LR, SVM, LDA, and KNN. Model training can utilize the Adam (Adaptive Moment Estimation) optimization algorithm, or other optimization algorithms such as BGD, SGD, and AdaGrad. The generative adversarial network (GAN) can be built using PyTorch or other frameworks such as TensorFlow. The specific configuration can be tailored to the specific needs.

[0104] Step S04. Obtain the required target positive sample data and the fault location to be generated. Based on the fault location to be generated, crop and segment the background image and target region image from the target positive sample data, input them into the trained recurrent generative adversarial network model to generate the corresponding negative sample target region image, and fuse the generated negative sample target region image with the segmented background image to obtain the final fault sample data.

[0105] Due to the high safety requirements of rail transit trains and the high resolution of the acquired image data while the targets are often small, traditional image generation methods struggle to handle large-size images. This embodiment addresses this by inputting the desired fault location, segmenting image patches from the original image as input data, and feeding this into a trained recurrent generative adversarial network (RGAN) model. This solves the model's inability to handle high-resolution data. Simultaneously, by preserving a large area of ​​positive sample background region to generate negative sample target regions, and finally fusing these negative sample target regions with the preserved positive sample background region, the required fault sample data can be generated quickly and accurately based on the local positive sample image. This significantly reduces the amount of data processing while improving the accuracy of the generated sample data.

[0106] It is understandable that a trained network model can perform the function of converting positive and negative samples. Inputting positive sample data and passing it through generator D can generate negative sample data, and at the same time, inputting negative samples and passing them through generator F can generate positive sample data.

[0107] As an optional implementation, the final fault sample data is obtained by fusing the generated negative sample target region map with the segmented background map, including:

[0108] The generated negative sample target region map is converted into a negative sample image patch with the same resolution as the target region map after inverse size normalization.

[0109] The transformed negative sample image patch is stitched together with the cropped and segmented background image to form a complete fault sample data output.

[0110] Specifically, such as Figure 5 As shown, the positive sample data of the images collected by the rail transit system are input, and the fault location to be generated is input. Image patches are segmented from the original image as input data. After loading the trained model weights and parameters, the input data is scaled and normalized to be consistent with the size of the training image and then input into the trained network model. The corresponding negative sample data can be generated by the generator network. The generated data is restored in the original image to obtain the overall negative sample data.

[0111] In this embodiment, the quality of the generated fault sample data is evaluated using the FID (Firmware ID) metric to obtain fault sample data whose FID metric meets preset requirements as the final generated fault sample data. A pre-trained Inception network is used to transform the real image and the generated image into feature vector sets, respectively. Then, the mean and covariance matrix of the two feature vector sets are calculated. Finally, the FID metric is obtained by calculating the square root of the sum of the squares of the differences between the two covariance matrices. The lower the FID score, the closer the distribution of the generated image is to the real image, and the higher the quality of the generated image.

[0112] The formula for calculating the FID index can be expressed as:

[0113]

[0114] Where, μ r and μ g The feature mean of the real image and the generated image is ∑ r and ∑ g Let the covariance matrix be the real image and the generated image. It is the L2 norm of the square of the difference between the mean vectors, and Tr represents the trace of the matrix (i.e., the sum of the diagonal elements of the matrix). It is the covariance matrix ∑ r and ∑ g The square root of the product of two matrices represents the matrix obtained by taking the square root of the eigenvalues ​​of the product of the two matrices.

[0115] It is understandable that, in addition to using the FID evaluation index, other commonly used indicators, such as IS, KID, SSIM, PSNR, etc., can also be used.

[0116] The rail transit train fault sample data generation device in this embodiment includes:

[0117] The data acquisition module is used to acquire the positive and negative sample data of the target train and use them as the raw image data. The negative sample data corresponds to the fault sample data.

[0118] The data cropping module is used to segment and crop each image in the original image data to crop image slices with negative sample fault locations and corresponding positive sample locations, forming an image dataset;

[0119] The model training module is used to input the image dataset into a pre-built recurrent generative adversarial network (GAN) model for training, resulting in a trained GAN model. The GAN model is constructed from two GAN networks, each of which includes a generator and a discriminator. The two generators are used to learn the mapping relationship between positive and negative samples, respectively, to realize the conversion from positive to negative samples and from negative to positive samples. The two discriminators are used to distinguish between real positive and negative sample data and positive and negative sample data generated by the generator to predict the authenticity of the sample data.

[0120] The sample generation module is used to obtain the required target positive sample data and the fault location to be generated. Based on the fault location to be generated, the background image and target region image are cropped and segmented from the target positive sample data and input into the trained recurrent generative adversarial network model to generate the corresponding negative sample target region image. The generated negative sample target region image is then fused with the segmented background image to obtain the final fault sample data.

[0121] The following example, using the method described above to generate negative samples from positive samples of rail transit temperature labels, further illustrates the present invention. The detailed steps are as follows:

[0122] Step 1) Data Acquisition: Obtain raw image data of rail transit, including positive sample data and fault sample data.

[0123] The train undercarriage was sampled using 160 normal temperature tags collected by an intelligent inspection robot as positive samples, and 20 faulty temperature tags with excessive temperatures as negative samples. The original data was 2048×1536 pixel BMP format image data.

[0124] Step 2) Image Dataset Creation: By segmenting and cropping the original images, a dataset that can be used for model training is created.

[0125] In creating the image dataset for model training, faulty temperature labels were cropped to their corresponding normal temperature label locations, resulting in 256×256 pixel image slices. Since there were relatively few real faulty samples, traditional data augmentation methods such as rotation and translation were used to expand the number of negative samples to 160, corresponding to 160 positive sample images. The positive and negative samples were placed in two folders, trainX and trainY, respectively. A new folder, testX, containing 10 positive sample images, was then created for model testing.

[0126] Step 3) Model building: Build a generative adversarial neural network model, including a generator model and a discriminator model based on the self-attention mechanism.

[0127] Using the PyTorch deep learning framework to construct network models, and employing the CycleGAN network to generate adversarial neural network models, such as... Figure 2 As shown, the model contains two GAN networks, which share two generators. The generator model and discriminator model are as follows: Figure 3 , 4 As shown.

[0128] Step 4) Model Training: Input the image dataset into the network model for training. Based on the loss functions of the generator and discriminator, the generator and discriminator evolve together in an alternating iterative game until the generator learns the data distribution of the real samples and the discriminator can no longer distinguish between the data generated by the generator and the corresponding real data, thus obtaining the trained weight file.

[0129] Specifically, the model loads the dataset through the data input interface, defines the batch size for training as 1, the number of training epochs as 200, sets λ = 10 in Equation 4, calculates the loss for each training iteration, optimizes the network weights using the Adam solver, sets the initial learning rate to 0.0002, keeps it unchanged for the first 100 epochs, and then linearly decays it to 0 for the next 100 epochs. The specific training steps are as follows:

[0130] Step 4.1) First, lock the discriminator network Dx and Dy, and train the generator network G and F so that the generated image can fool the discriminator network Dx and Dy. That is, the more realistic the generated image is, the better, and the closer the probability of the discriminator network Dx and Dy judges it as real is to 1, the better.

[0131] Step 4.2) Then open the Dx and Dy networks and train the discriminator Dx and Dy networks to identify whether the output image is a fake image, i.e., a generated image. The goal of the discriminator is to judge whether the real data sample is real and whether the image generated by the generator is fake.

[0132] Step 4.3) Re-lock the Dx and Dy networks and train the generator G and F networks so that the output image can fool the discriminator Dx and Dy networks.

[0133] Step 4.4) Repeat steps 4.1) and 4.2) continuously, optimizing, iterating, and updating the model weight parameters until the discriminator network can no longer determine whether the generator network outputs true or false.

[0134] Step 5) Image data generation: After loading the trained model weights, the model can realize the conversion function of positive and negative samples. Input positive sample data, and output the corresponding negative sample data through the generation network. The quality of the generated data is evaluated by the evaluation model.

[0135] Step 5.1: Input positive sample data X1 of rail transit image and input the fault location (x, y, w, h) to be generated. Crop and segment it into background image M and target area image x1. The resolution of target area image x1 is (w, h). After size normalization, the image x2 is normalized to 256×256 pixels and used as input data for the generation model.

[0136] Step 5.2: Load the trained model weights into the network model, input the image data x2 into the generator network model through the interface, and generate the corresponding 256x256 pixel negative sample image y2 through the generator network G.

[0137] Step 5.3 generates the corresponding 256x256 pixel negative sample image y2, which is then reverse-sized and normalized to become a negative sample image block with a resolution of (w, h). Finally, it is stitched together with the background image M that was cropped and segmented in step 5.1 to form the complete fault image data Y1.

[0138] The negative samples generated by the generator are evaluated using the FID metric to assess the quality of the generated data. This metric is used to quantify the similarity between the images generated by the generator model and real images, until the FID metric reaches the preset requirements.

[0139] like Figure 6 The image shows a model that generates temperature labels exceeding the temperature limit based on normal temperature labels. Figure 6 (a) shows the generated result without the self-attention mechanism, where the temperature label's color-changing area is discontinuous and lacks clarity. Figure 6 (b) shows the result of adding the self-attention mechanism. As can be seen from the result, the result generated after the method of the present invention is clearer and more realistic.

[0140] The rail transit train fault sample data generation device in this embodiment includes:

[0141] The data acquisition module is used to acquire the positive sample data and negative sample data of the target train and use them as raw image data. The negative sample data corresponds to the fault sample data.

[0142] The data cropping module is used to segment and crop each image in the original image data to crop image slices with negative sample fault locations and corresponding positive sample locations, forming an image dataset;

[0143] The model training module is used to input the image dataset into a pre-built recurrent generative adversarial network (GAN) model for training, thereby obtaining the trained GAN model. The GAN model is constructed from two GAN networks, each of which includes a generator and a discriminator. The two generators are used to learn the mapping relationship between positive and negative samples, respectively, to realize the conversion from positive to negative samples and from negative to positive samples. The two discriminators are used to distinguish between real positive and negative sample data and positive and negative sample data generated by the generator in order to predict the authenticity of the sample data.

[0144] The sample generation module is used to acquire the required positive sample data and the fault locations to be generated. Based on the fault locations to be generated, the background image and target region image are cropped and segmented from the target positive sample data and input into the trained recurrent generative adversarial network model to generate the corresponding negative sample target region image. The generated negative sample target region image is then fused with the segmented background image to obtain the final fault sample data.

[0145] The rail transit train fault sample data generation device in this embodiment corresponds one-to-one with the rail transit train fault sample data generation method described above, and will not be described in detail here.

[0146] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the method described above.

[0147] It is understood that the method described in this embodiment can be executed by a single device, such as a computer or server, or it can be applied to a distributed scenario where multiple devices cooperate to complete the task. In a distributed scenario, one of the multiple devices may execute only one or more steps of the method described in this embodiment, and the multiple devices interact to complete the method. The processor can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the method described in this embodiment. The memory can be implemented using read-only memory (ROM), random access memory (RAM), static storage devices, and dynamic storage devices. The memory can store the operating system and other applications. When the method described in this embodiment is implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0148] This embodiment further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0149] Those skilled in the art will understand that the above embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create an implementation for the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for generating fault sample data for rail transit trains, characterized in that the steps include: include: The positive and negative sample data of the target train are acquired and used as the raw image data, wherein the negative sample data corresponds to the fault sample data. Each image in the original image data is segmented and cropped to extract image slices with negative sample fault locations and corresponding positive sample locations, forming an image dataset; The image dataset is input into a pre-built recurrent generative adversarial network (GAN) model for training, resulting in a trained GAN model. The GAN model is constructed from two GAN networks, each including a generator and a discriminator. The two generators are used to learn the mapping relationship between positive and negative samples, realizing the conversion from positive to negative samples and from negative to positive samples. The two discriminators are used to distinguish between real positive and negative sample data and positive and negative sample data generated by the generator, in order to predict the authenticity of the sample data. Obtain the required target positive sample data and the fault locations to be generated. Based on the fault locations to be generated, crop and segment the background image and target region image from the target positive sample data, input them into the trained recurrent generative adversarial network model to generate the corresponding negative sample target region image, and fuse the generated negative sample target region image with the segmented background image to obtain the final fault sample data.

2. The method for generating fault sample data for rail transit trains according to claim 1, characterized in that, In each generator of the recurrent generative adversarial network model, a self-attention module is set in series before the residual module. The feature map extracted by the encoder in the generator is weighted and summed with the feature map transformed by the residual module after passing through the self-attention module to obtain the final feature map.

3. The method for generating fault sample data for rail transit trains according to claim 2, characterized in that, The execution steps of the self-attention module in the recurrent generative adversarial network model include: Multiply each feature map by W. q W k W v The coefficients correspond to Q, K, and V, where Q, K, and V represent the query, key, and value, respectively. Based on Q, K, and V, an attention matrix A is calculated for each feature map, where A = K. T Q, where each value in the attention matrix A is the attention size α(i,j) of the two corresponding input feature maps; The attention matrix A is normalized using the Softmax function to obtain matrix A'; Calculate the output matrix O of the self-attention layer corresponding to each input feature map based on the obtained matrices A' and V; The output matrix O is weighted with the input feature map to obtain a feature map output with global dependencies at any two positions.

4. The method for generating fault sample data for rail transit trains according to claim 1, characterized in that, Each discriminator in the recurrent generative adversarial network model is constructed from a PatchGAN network and a self-attention module. When each discriminator receives an input image, it first calculates the attention weight at each position through the attention module, and then uses an activation function to normalize each attention weight into a probability distribution so that the sum of the attention weights at each position is 1. Based on the calculated attention weights at each position and the target position, a weight mask is generated and weighted and output to segment the target region. The segmented image is then input into the PatchGAN network to calculate the predicted probability value of the sample's authenticity. If the probability value is close to 1, it is a real sample; if the probability value is close to 0, it is a fake sample.

5. The method for generating fault sample data for rail transit trains according to claim 1, characterized in that, The overall optimization objective function of the recurrent generative adversarial network model during training includes adversarial loss and cycle consistency loss. The cycle consistency loss includes positive cycle loss and inverse cycle loss. The positive cycle loss is the difference between the image G(x) generated by generator G and the original image x after being transformed by generator F. The inverse cycle loss is the difference between the image F(y) generated by generator F and the original image y after being transformed by generator G. Generator G is used to transform positive sample X into negative sample Y, and generator F is used to transform negative sample Y into positive sample X.

6. The method for generating fault sample data for rail transit trains according to claim 5, characterized in that, The overall objective function is expressed as follows: Loss=Loss GAN (G,D Y ,X,Y)+Loss GAN (F,D X ,Y,X)+λLoss cyc (G,F) Where Loss is the overall optimization objective function, Loss GAN Let G be the adversarial loss function, x be an image from a positive sample X, y be an image from a negative sample Y, G be the generator that transforms a positive sample X into a negative sample Y, F be the generator that transforms a negative sample Y into a positive sample X, and D be the generator that transforms a negative sample Y into a positive sample X. X and D Y For the corresponding discriminator, Loss represents the mathematical expectation of random variables x and y belonging to the true distribution of samples X and Y. cyc The cycle-consistent loss function is expressed as: Loss cyc =E|F(G(x)-x)|+E|G(F(y)-y)| Where E represents the expectation function.

7. The method for generating fault sample data for rail transit trains according to any one of claims 1 to 6, characterized in that, The step of inputting the image dataset into a pre-built recurrent generative adversarial network (RGAN) model for training to obtain the trained RGAN model includes: First, lock two discriminator networks, Dx and Dy, and train two generator networks, G and F, such that the difference between the probability that the images generated by the two generators are judged as real by the discriminator networks Dx and Dy and 1 is less than a preset threshold. Then, open two discriminator networks, Dx and Dy, and train the two discriminator networks until they determine that the real data sample is true. Then re-lock the two discriminator networks Dx and Dy, and train the two generator networks G and F until the difference between the probability that the output image is judged as real by the discriminator networks Dx and Dy and 1 is less than the preset threshold. Repeatedly train the two generator networks G and F and the two discriminator networks Dx and Dy until the two discriminators can no longer distinguish between the true and false outputs of the two generators. The training is then complete, and the weight file of the currently trained recurrent generative adversarial network model is output.

8. The method for generating fault sample data for rail transit trains according to claim 7, characterized in that, The optimization objective function used in the process of locking the two discriminator networks Dx and Dy and training the two generator networks G and F is: The two discriminator networks Dx and Dy are opened. The optimization objective function used during the training of these two discriminator networks is: Among them, Loss GAN Let G be the adversarial loss function, x be an image from a positive sample X, y be an image from a negative sample Y, G be the generator that transforms a positive sample X into a negative sample Y, F be the generator that transforms a negative sample Y into a positive sample X, and D be the generator that transforms a negative sample Y into a positive sample X. X and D Y This is the corresponding discriminator.

9. The method for generating fault sample data for rail transit trains according to any one of claims 1 to 6, characterized in that, The process of fusing the generated negative sample target region map with the segmented background map to obtain the final fault sample data includes: The generated negative sample target region map is converted into a negative sample image patch with the same resolution as the target region map after inverse size normalization. The negative sample image blocks obtained after conversion are stitched together with the background image obtained by cropping and segmentation to form a complete fault sample data output.

10. The method for generating fault sample data for rail transit trains according to any one of claims 1 to 6, characterized in that, It also includes using the FID metric to evaluate the quality of the generated fault sample data, so as to obtain fault sample data that meets the preset requirements of the FID metric as the final generated fault sample data.

11. A device for generating fault sample data for rail transit trains, characterized in that, include: The data acquisition module is used to acquire the positive sample data and negative sample data of the target train and use them as raw image data. The negative sample data corresponds to the fault sample data. The data cropping module is used to segment and crop each image in the original image data to crop image slices with negative sample fault locations and corresponding positive sample locations, forming an image dataset; The model training module is used to input the image dataset into a pre-built recurrent generative adversarial network (GAN) model for training, thereby obtaining the trained GAN model. The GAN model is constructed from two GAN networks, each of which includes a generator and a discriminator. The two generators are used to learn the mapping relationship between positive and negative samples, respectively, to realize the conversion from positive to negative samples and from negative to positive samples. The two discriminators are used to distinguish between real positive and negative sample data and positive and negative sample data generated by the generator in order to predict the authenticity of the sample data. The sample generation module is used to obtain the required target positive sample data and the fault location to be generated. Based on the fault location to be generated, the background image and target region image are cropped and segmented from the target positive sample data and input into the trained recurrent generative adversarial network model to generate the corresponding negative sample target region image. The generated negative sample target region image is then fused with the segmented background image to obtain the final fault sample data.

12. An electronic device comprising a processor and a memory, the memory being used to store a computer program, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the method as described in any one of claims 1 to 10.

Images