Multi-modal image sample generation method for intelligent operation of high-speed railway

Abstract

The invention provides a multi-modal image sample generation method for intelligent operation of a high-speed railway, and relates to the technical field of railway monitoring based on machine vision, and the method comprises a generator and a discriminator, the generator is a conditional diffusion network, the discriminator is a deep convolutional network, and the discriminator is a multi-modal image sample. The generator comprises a condition coding module and a diffusion generation module, and the condition coding module is used for receiving task category labels and scene description texts as input; a conditional diffusion network is introduced as a generator, railway operation scenes and conditional information are fused into a model, and railway operation scene distribution is efficiently sampled in a submerged space, so that the limitation that an image generation method based on a generative adversarial network is insufficient in sample diversity and insufficient in small sample category generation capability is overcome, and the image generation efficiency is improved. A discriminator of the generative adversarial network is used to supervise the generation process of the conditional diffusion network, and adversarial loss optimization is combined, so that the detail quality of the generated image is improved, and the defect in details is made up.

Description

Technical Field

[0001] The present invention relates to the field of railway monitoring technology based on machine vision, and in particular to a method for generating multimodal image samples for intelligent operation of high-speed railways. Background Art

[0002] In railway operation scenarios, visual recognition and detection tasks are widely used in the fields of abnormal behavior recognition of passenger station personnel, track disease detection, key component detection of locomotives and vehicles, and key component fault detection of contact lines. However, due to insufficient samples of actual data sets, single scenarios, and uneven distribution of sample categories, visual tasks in railway operation scenarios have brought significant challenges. In recent years, computer vision large model technologies based on attention mechanisms and Transformer structures have developed rapidly. However, these technologies have put forward higher requirements on the scale and diversity of data sets, and it is difficult to achieve ideal recognition accuracy in railway operation scenarios with small samples and unbalanced data sets. In response to insufficient sample size and uneven category distribution, traditional methods mainly include collecting additional data, using Copy-Paste data enhancement, and artificial image synthesis based on image editing software. The data collection and annotation process usually requires additional manpower and time investment, which is not only costly and time-consuming, but also difficult to supplement samples of specified categories in a targeted manner. The Copy-Paste-based data enhancement method has the problems of limited scene diversity, abnormal sample size ratio and depth relationship, and poor authenticity of the generated data. The artificial image synthesis method based on image editing software needs to rely on professional operation, with high labor costs and is difficult to use to quickly generate a large number of images. In recent years, generative adversarial networks and diffusion models have emerged. Neural network technology has shown great potential in the field of automated image generation. However, the application of these technologies in the field of railway operations still has many deficiencies. The image generation method based on generative adversarial networks has a mode collapse problem, the diversity of generated data is insufficient, and it is difficult to cover all scene distributions; although the image generation method based on the diffusion model generates samples with a wider distribution, the quality of its image details is often relatively poor, and there are still similar proportional distortion and depth error problems as traditional methods. The above methods all have certain limitations and cannot be well applied to image sample generation in railway operation scenarios. Therefore, the present invention proposes a multimodal image sample generation method for high-speed railway intelligent operation to solve the problems existing in the prior art. Summary of the invention

[0003] In response to the above problems, the present invention proposes a multimodal image sample generation method for high-speed railway intelligent operation. This method combines the advantages of diffusion model and generative adversarial network, and can generate high-quality image samples of specific types under specified railway operation scenarios, thereby effectively expanding the scale of the data set, solving the problems of too few samples and unbalanced categories, and improving the applicability and performance of the model.

[0004] To achieve the purpose of the present invention, the present invention is implemented through the following technical solutions: a multimodal image sample generation method for high-speed railway intelligent operation, including a generator and a discriminator, the generator is a conditional diffusion network, the discriminator is a deep convolutional network, the generator includes a conditional encoding module and a diffusion generation module, the conditional encoding module is used to accept task category labels and scene description texts as inputs, generate corresponding conditional embedding vectors as inputs of the diffusion generation module, the diffusion generation module uses a U-Net structure suitable for deformable convolution as a basic framework, introduces adaptive sampling point positions, and generates railway operation scene images under different camera installation angles; The discriminator is used to determine whether an input image is a real image or a generated virtual image. The generator and the discriminator are trained alternately, and both the generator and the discriminator have built-in loss functions.

[0005] Further improvements are as follows: the task category label is encoded in one-hot format and converted into task category embedding through an embedding layer, the generated image samples are characterized as railway operation scenarios to be actually applied, the scene description text is used to describe the scene label of the generated image, and is converted into the corresponding scene description embedding through CLIP, the task category label and the scene description text are spliced ​​together as conditional coding embedding, and together with the sinusoidally modulated time step embedding, are used as the input of the diffusion generation module.

[0006] Further improvements are: when training the discriminator, the discrimination ability is improved by minimizing the probability of misjudgment, and when training the generator, the generation quality is improved by maximizing the probability that the discriminator judges the generated samples as real samples.

[0007] A further improvement is that the loss function of the generator is:

[0008] Among them, diffusion loss Used to describe the quality of diffusion-generated samples and the denoising effect, adversarial loss Used for adversarial training of the generator to guide the generator to generate more realistic samples, hyperparameters Used to reconcile the importance of the two parts of loss.

[0009] A further improvement is that the loss function of the discriminator is:

[0010] Among them, the loss of resistance Used to describe the discriminative ability of the discriminator, regular term Used to limit the gradient range of the discriminator, hyperparameter Used to reconcile the importance of the two parts of loss.

[0011] A method for generating multimodal image samples for high-speed railway intelligent operation includes the following steps: S1: Use a general visual dataset to pre-train the generator and discriminator separately, so that they have specified performance; S2: Use proprietary datasets in the railway field to conduct targeted training for railway operation scenarios, and iteratively train the generator and discriminator alternately; S3: At each iteration, the discriminator classification result in the previous round of generator output is lower than the threshold The high-quality rare scene samples are used as the input for the next round of iterative generator training to adjust the distribution of training data; S4: For specific railway operation scenarios, select corresponding task category labels, design scenario labels based on different scenario elements, and randomly combine them as conditional embeddings for the input model; S5: In the diffusion generation module, the conditional embedding and random noise sampling in the latent space are used as input to complete the generation of image samples; S6: The generated image samples are input into the discriminator to judge the quality of the generated samples. By randomly combining scene labels, high-quality image samples that meet the task quantity requirements are generated to complete the expansion of the sample library.

[0012] Further improvements are as follows: In S1, for the generator, the COCO Image Captioning dataset is used for training, the text description of the image in the dataset is used as the input scene description text, the task category label is set to 0, indicating a non-railway operation scenario, and the input and output of the model at each time step are obtained through a diffusion process. The hyperparameters of the loss function in this stage are Take 0, that is, no adversarial loss is calculated; for the discriminator, a generator based on a convolutional neural network and the discriminator of the original network are used for generative adversarial training, so that the discriminator has the ability to discriminate images generated by general scenes.

[0013] A further improvement is that in S2, the output of the generator is used as the input of the discriminator, and the generator and the discriminator are iteratively trained alternately according to the normal training process of the generative adversarial network until the training process converges.

[0014] Further improvements are: in S3, the generated virtual data is used to expand the rare scene training samples in the railway operation scene. In order to prevent model drift during training, the proportion of real data in the generator input of each sample category is controlled to be no less than the threshold .

[0015] A further improvement is that in S6, when the discriminator output is higher than the threshold, it is considered that the authenticity of the generated sample is low, the generated result is discarded and the process goes to S5.

[0016] The beneficial effects of the present invention are: 1. The present invention introduces a conditional diffusion network as a generator, integrates railway operation scenarios and condition information into the model, and efficiently samples the distribution of railway operation scenarios in the latent space, thereby overcoming the limitations of the image generation method based on the generative adversarial network in terms of insufficient sample diversity and insufficient ability to generate small sample categories. At the same time, the generation process of the conditional diffusion network is supervised by the discriminator of the generative adversarial network, and combined with adversarial loss optimization, the detail quality of the generated image is significantly improved, making up for the shortcomings of the image generation method based on the diffusion model in terms of detail presentation.

[0017] 2. The present invention introduces railway operation scene labels and categories as conditional embedding, which further improves the stability and directionality of sample generation. The generated image dataset has high quality and authenticity, thereby better meeting the needs of railway operation scene visual artificial intelligence model training, thereby providing strong data support for railway operation-related visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the network architecture of the present invention. DETAILED DESCRIPTION

[0019] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention. Embodiment 1

[0020] according to Figure 1 As shown, this embodiment proposes a multimodal image sample generation method for high-speed railway intelligent operation. The overall structure of the system is a generative adversarial network, which is divided into two parts: a generator and a discriminator. Among them, the generator model is a conditional diffusion network, and the discriminator is a deep convolutional network.

[0021] The generator consists of two parts: a conditional encoding module and a diffusion generation module. The conditional encoding module accepts task category labels and scene description text as inputs to generate corresponding conditional embedding vectors. Among them, the task category label is encoded in one-hot format and converted into a task category embedding through the embedding layer, which characterizes the railway operation scenarios in which the generated image samples will be actually applied, such as track disease detection, vehicle number recognition, and contact network fault detection. The scene description text is used to describe the scene labels of the generated image, such as day and night, weather, and elements in the scene, and is converted into the corresponding scene description embedding through CLIP. The two are spliced ​​together as conditional coding embeddings, and together with the sinusoidally modulated time step embeddings, they are used as the input of the diffusion generation module.

[0022] The diffusion generation module uses an improved U-Net structure as the basic framework. Compared with the original U-Net structure, the deformable convolution is used to replace the traditional convolution operation in the original structure, and adaptive sampling point positions are introduced to enhance the model's ability to model geometric deformations and improve the flexibility of the receptive field, which is conducive to better generating railway operation scene images under different camera installation perspectives.

[0023] The discriminator uses a deep convolutional network to determine whether the input image is a real image or a generated virtual image. The generator and the discriminator are trained alternately. When the discriminator is trained, it is similar to the ordinary binary classification network training, and the discrimination ability is improved by minimizing the probability of misjudgment. When the generator is trained, the generation quality is improved by maximizing the probability that the discriminator judges the generated sample as a real sample.

[0024] Loss function design The loss function of the generator is designed as follows:

[0025] Among them, diffusion loss Describes the quality and denoising effect of diffusion-generated samples, adversarial loss Used for adversarial training of GAN generators, guiding the generator to generate more realistic samples, hyperparameters Used to reconcile the importance of the two parts of loss.

[0026] The loss function of the discriminator is designed as follows:

[0027] Among them, the loss of resistance Describes the discriminative ability of the discriminator, the regular term Used to limit the gradient range of the discriminator, hyperparameter Used to reconcile the importance of the two parts of loss. Embodiment 2

[0028] according to Figure 1 As shown, this embodiment proposes a multimodal image sample generation method for high-speed railway intelligent operation, including the following steps: Three-stage training method: The training of the image generation network is divided into three stages. In the first stage, the generator and discriminator are pre-trained using a general visual dataset to give them certain performance. For the generator, the COCO ImageCaptioning dataset is used for training. The text description of the image in the dataset is used as the input scene description text, and the task category label is set to 0, indicating a non-railway operation scene. The input and output of the model at each time step are obtained through a diffusion process. The loss function hyperparameters in this stage Take 0, that is, no adversarial loss is calculated. For the discriminator, a generator based on a convolutional neural network is used to perform generative adversarial training with the discriminator of the original network, so that the discriminator has a certain ability to discriminate images generated by general scenes.

[0029] In the second phase, a proprietary data set in the railway field is used for targeted training for railway operation scenarios. At this point, the output of the generator is used as the input of the discriminator, and the generator and discriminator are iteratively trained alternately according to the normal training process of the generative adversarial network. Until the training process converges. However, at this time, the distribution generated by the generator still has the problem of small number of sparse scene training data and poor generation quality. Therefore, it is necessary to continue the third round of training, using the generated virtual data to expand the sparse scene training samples in the railway operation scenario and improve the generation quality of the generator.

[0030] In the third stage of training, at each iteration, the discriminator classification result in the previous round of generator output is lower than the threshold The high-quality rare scene samples are used as the input for the next round of iterative generator training to adjust the distribution of training data and enrich the diversity of the generator's generation results. In order to prevent model drift during training, the proportion of real data in the generator input of each sample category is guaranteed to be no less than the threshold By adding the generated data to the next round of training, the problem of uneven sample distribution during training can be improved and the generation quality can be improved.

[0031] Sample generation process: Generate conditional embeddings For specific railway operation scenarios, select the corresponding task category labels. Design a series of scenario labels from aspects such as time, weather, viewing angle, and scenario elements, and randomly combine them as conditional embeddings for the input model.

[0032] Image sample generation In the diffusion generation module, conditional embedding and random noise sampling in the latent space are used as input to complete the generation of image samples.

[0033] Optimize the generated results The generated image samples are input into the discriminator to judge the quality of the generated samples. If the discriminator output is higher than the threshold , the authenticity of the generated sample is considered to be low, the generated result is discarded and the image sample generation step is turned to.

[0034] Get expanded sample library By randomly combining scene labels, high-quality image samples that meet the number of task requirements are generated to expand the sample library.

[0035] The present invention proposes a high-quality visual image generation algorithm for expanding the railway operation scene image sample library, solving the problem of insufficient number of samples and uneven category distribution in this field. For different railway operation visual tasks and scenes, samples for different tasks and scenes can be generated by adjusting the task category labels and scene description texts, avoiding repeated training and deployment. By generating high-quality samples to expand the sample library, the problem of low accuracy caused by insufficient number of samples in railway operation scene visual tasks can be solved.

[0036] The present invention introduces a conditional diffusion network as a generator, integrates railway operation scenarios and condition information into the model, and efficiently samples the distribution of railway operation scenarios in the latent space, thereby overcoming the limitations of the image generation method based on the generative adversarial network in terms of insufficient sample diversity and insufficient ability to generate small sample categories. At the same time, the generation process of the conditional diffusion network is supervised by the discriminator of the generative adversarial network, and combined with adversarial loss optimization, the detail quality of the generated image is significantly improved, making up for the shortcomings of the image generation method based on the diffusion model in terms of detail presentation. The present invention introduces railway operation scene labels and categories as conditional embeddings, further improving the stability and directionality of sample generation, and the generated image data set has high quality and authenticity, thereby better meeting the needs of railway operation scene visual artificial intelligence model training, thereby providing strong data support for railway operation-related visual tasks.

[0037] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multimodal image sample generation method for high-speed railway intelligent operation, including a generator and a discriminator, characterized in that: The generator is a conditional diffusion network, the discriminator is a deep convolutional network, the generator includes a conditional encoding module and a diffusion generation module, the conditional encoding module is used to accept the task category label and the scene description text as input, generate the corresponding conditional embedding vector as the input of the diffusion generation module, the diffusion generation module adopts the U-Net structure applicable to deformable convolution as the basic framework, introduces adaptive sampling point positions, and generates railway operation scene images under different camera installation angles; The discriminator is used to determine whether an input image is a real image or a generated virtual image. The generator and the discriminator are trained alternately, and both the generator and the discriminator have built-in loss functions.

2. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 1, characterized in that: The task category label is encoded in one-hot format and converted into task category embedding through an embedding layer, representing the generated image sample as a railway operation scene to be actually applied. The scene description text is used to describe the scene label of the generated image and is converted into the corresponding scene description embedding through CLIP. The task category label and the scene description text are spliced ​​together as a conditional coding embedding, which is used as the input of the diffusion generation module together with the sinusoidally modulated time step embedding.

3. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 2, characterized in that: When training the discriminator, the discrimination ability is improved by minimizing the probability of misjudgment. When training the generator, the generation quality is improved by maximizing the probability that the discriminator judges the generated samples as real samples.

4. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 3 is characterized in that: The loss function of the generator is: ， Among them, diffusion loss Used to describe the quality of diffusion-generated samples and the denoising effect, adversarial loss Used for adversarial training of the generator to guide the generator to generate more realistic samples, hyperparameters Used to reconcile the importance of the two parts of loss.

5. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 4, characterized in that: The loss function of the discriminator is: ， Among them, the loss of resistance Used to describe the discriminative ability of the discriminator, regular term Used to limit the gradient range of the discriminator, hyperparameter Used to reconcile the importance of the two parts of loss.

6. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 5, characterized in that: The following steps are involved: S1: Use a general visual dataset to pre-train the generator and discriminator separately, so that they have specified performance; S2: Use proprietary datasets in the railway field to conduct targeted training for railway operation scenarios, and iteratively train the generator and discriminator alternately; S3: At each iteration, the discriminator classification result in the previous round of generator output is lower than the threshold The high-quality rare scene samples are used as the input for the next round of iterative generator training to adjust the distribution of training data; S4: For specific railway operation scenarios, select corresponding task category labels, design scenario labels based on different scenario elements, and randomly combine them as conditional embeddings for the input model; S5: In the diffusion generation module, the conditional embedding and random noise sampling in the latent space are used as input to complete the generation of image samples; S6: The generated image samples are input into the discriminator to judge the quality of the generated samples. By randomly combining scene labels, high-quality image samples that meet the task quantity requirements are generated to complete the expansion of the sample library.

7. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 6, characterized in that: In S1, for the generator, the railway operation scene dataset is used for training, the text description of the image in the dataset is used as the input scene description text, the task category label is set to 0, indicating a non-railway operation scene, and the input and output of the model at each time step are obtained through the diffusion process. The loss function hyperparameters in this stage Take 0, that is, no adversarial loss is calculated; for the discriminator, a generator based on a convolutional neural network and the discriminator of the original network are used for generative adversarial training, so that the discriminator has the ability to discriminate images generated by general scenes.

8. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 6, characterized in that: In S2, the output of the generator is used as the input of the discriminator, and the generator and the discriminator are iteratively trained alternately according to the normal training process of the generative adversarial network until the training process converges.

9. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 6, characterized in that: In S3, the generated virtual data is used to expand the rare scene training samples in the railway operation scene. In order to prevent the model drift during training, the proportion of real data in the generator input of each sample category is controlled to be no less than the threshold .

10. The method for generating multimodal image samples for high-speed railway intelligent operation according to claim 6, characterized in that: In S6, when the discriminator output is higher than the threshold, it is considered that the authenticity of the generated sample is low, the generated result is discarded and the process goes to S5.

Images