Rail transit image generation method

Abstract

The embodiment of the invention provides a rail transit image generation method, and the method comprises the steps: obtaining a rail transit image, inputting the rail transit image into an image generation network, preliminarily processing the image into hidden space features, and fusing the hidden space features with feature prompts generated by a feature extraction module to generate auxiliary features; and according to the auxiliary features, guiding a diffusion model to execute a denoising process on the preliminarily processed image to obtain generated image features, and decoding the generated image features into an output image. According to the generation method provided by the invention, fine adjustment of the diffusion model is realized by using a small number of images, and diversified images are generated; the image features are used for replacing text features, so that the problems that text and graph data annotation is tedious and a model is difficult to train are solved; and by using the hidden space features, the resource consumption of the diffusion model in the image generation process is reduced, and the rail transit image generation task is quickly and efficiently completed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for generating a rail transit image and a storage medium. Background Art

[0002] In the field of rail transit, computer vision systems must rely on a large amount of data for training to achieve accurate perception of the environment. However, in the field of rail transit, there is a lack of high-quality, large-scale data sets. It is very difficult to obtain high-quality visual data for rail transit. For operational safety and collection costs, it is impossible to manually collect diverse sample data. Artificial intelligence models trained with low-quality, low-data samples cannot make accurate judgments and decisions, thereby increasing the probability of accidents and risks, and hindering the further development of intelligence in the field of rail transit.

[0003] Image generation technology has received widespread attention because it can generate diverse images with a small amount of data and achieve safe and fast rich track data sets. The emergence of Generative Adversarial Networks (GANs) has made breakthrough progress in the field of image generation, but the images it generates have low resolution and cannot be used in subsequent tasks. The CLIP model is an image generation model guided by text and images, which can generate high-quality images, but its generation efficiency is low and it cannot generate a large number of images simply and quickly. Compared with traditional methods, the diffusion model generation method based on Diffusion has a large number of parameters and model structures. After training with a large amount of data, it can generate diverse high-resolution images and achieve good results. However, the diffusion model generation method based on Diffusion requires high computing resources and a large amount of training data.

[0004] In order to solve the problem that the generation method of the diffusion model requires a large amount of training data and high resource consumption, many scholars have proposed different diffusion model fine-tuning algorithms, such as freezing parameters when training the diffusion model or training the diffusion model through image features. However, these methods need to be fine-tuned and adapted when applied to a specific field for image generation. In order to generate high-quality and diverse rail transit images using a small number of rail transit images, designing a specific fine-tuning method for the diffusion model is still an urgent problem to be solved. Summary of the invention

[0005] The purpose of an embodiment of the present invention is to provide a method for generating rail transit images. The technical problem to be solved by the method is that the existing diffusion model uses text prompts to generate image features during fine-tuning. This method requires a large amount of data and cumbersome manual annotation, and consumes a lot of computing resources.

[0006] In order to solve the above technical problems, the first aspect of an embodiment of the present invention provides a method for generating a rail transit image, the method comprising: acquiring a rail transit image; and inputting the rail transit image into a rail transit image generation network, and performing the following operations in the rail transit image generation network: performing preliminary processing on the rail transit image to generate latent space features, compressing the latent space features, and encoding them into latent space features of different dimensions; directly extracting the explicit features of the rail transit image, extracting the deep features of the preliminary processed image, combining the display features with the deep features, and upsampling them to the dimension of the rail transit image to obtain feature prompts; fusing the feature prompts with the latent space features of different dimensions through a ControlNet module to generate auxiliary features; based on the auxiliary features, guiding the diffusion model to perform a denoising process on the preliminary processed image to obtain generated image features; and decoding the generated image features into an output image.

[0007] Optionally, the preliminary processing of the rail transit image to generate latent space features includes: using a VAE module to transform the features of the rail transit image from high dimension to low dimension, and extracting the most significant feature information of the rail transit image; and randomly extracting features of the most significant feature information and restoring it to the original feature dimension of the rail transit image to obtain the latent space features; wherein the VAE module includes an encoder part, an implicit distribution representation part, and a decoder part.

[0008] Optionally, compressing the latent space features and encoding them into latent space features of different dimensions includes: using a compression encoder module to gradually encode the latent space features from large to small, encoding them into latent space features of different dimensions to correspond to the dimensions of the feature prompts; the compression encoder module is formed by cascading processing blocks of different dimensions, the processing blocks include a Conv structure and two PixelNorms, and a Downsample module and a convolution module are used between every two dimensions to ensure dimensional matching.

[0009] Optionally, the feature hint is generated by a feature extraction module; the feature extraction module includes an explicit feature extraction part and a deep feature extraction part; the explicit feature extraction part includes 4 canny functions, which are used to extract image edge features of different sizes, and upsample the edge features of different sizes to the original image resolution to obtain texture detail features at different levels; the deep feature extraction part includes 4 groups of feature extractors for extracting image features of different dimensions, each group of extractors includes three groups of cascaded Conv+Relu+Maxpool operation blocks, which are used to extract image features corresponding to the explicit feature output dimensions, and combine them into feature hints of different dimensions.

[0010] Optionally, the explicit features are edge features directly extracted with the same dimension as the rail transit image and of different sizes; the deep features are features of different dimensions obtained by continuously convolving and downsampling the initially processed image.

[0011] Optionally, the ControlNet module includes: a conditional feature extraction module, including 4 SDEncoders of different sizes; an intermediate layer module, including an SD intermediate layer module and an intermediate zero convolution layer of the same size; and a zero convolution layer module, including 4 zero convolution layers of different sizes; wherein the conditional feature extraction module and the intermediate layer module use the same structure and parameters as the diffusion model.

[0012] Optionally, generating the auxiliary features includes: copying part of the pre-trained parameters of the diffusion model to train the ControlNet module, inputting the feature prompts and the latent space features of different dimensions into the ControlNet module to guide parameter updating, thereby generating the auxiliary features.

[0013] Optionally, the diffusion model includes: an SD encoder stage, including four SD Encoders; an SD mid-layer, including a mid-layer block; and an SD decoder, including four SD Decoders.

[0014] Optionally, the denoising process includes: setting the number of iteration steps of the denoising process; inputting the auxiliary feature, using the latent space feature to subtract the auxiliary feature to obtain a new latent space feature; using the new latent space feature to subtract the auxiliary feature again; and continuously iterating the above process until the number of iteration steps is completed to obtain the generated image feature.

[0015] A second aspect of an embodiment of the present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute any of the rail transit image generation methods described above in the present application.

[0016] Through the above technical scheme, a method for generating a rail transit image provided by the present invention processes the acquired rail transit image into latent space features, so that the features pay more attention to the deep semantic information of the rail transit image and achieve the purpose of saving computing resources; directly extract the display features and deep features of the rail transit image, and combine the two, so that the combined features are upsampled to the dimension of the rail transit image, and realize the direct extraction of features from the image for fine-tuning training of the diffusion model, solving the problem that the original diffusion model uses text prompts for fine-tuning training and requires a large amount of data and manual annotation. Directly using image features for fine-tuning training not only retains the powerful generation ability of the diffusion model, but also reduces the image data required for training. The auxiliary features are generated by using the feature prompts and the latent space features through the ControlNet module to guide the diffusion model to continuously get closer to the real image in the process of generating image features, so as to ensure the quality of the generated image features. The compression encoder and compression decoder are used to change the feature dimensions of the rail transit image, so as to use feature information of different dimensions to make the generated image have a more realistic generation effect. Therefore, the present invention uses a small amount of acquired rail transit images to fine-tune the diffusion model and generates more diverse rail transit images, thereby enriching the rail transit image data set quickly and efficiently. By using image features instead of text features to fine-tune the diffusion model, the problem that the existing diffusion model requires a large amount of data and manual annotation when using text prompts to generate image features during fine-tuning is solved. On the other hand, compared with the RGB space, the number of features in the latent space is smaller. By using the diffusion model to act on the latent space of the image, the model reduces resource consumption and achieves the purpose of saving computing resources.

[0017] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0019] Figure 1 is an overall flow chart of a method for generating an image of rail transit data disclosed in an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of the structure of a track image generation network disclosed in an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a process for obtaining rail transit images disclosed in an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of the structure of a VAE module disclosed in an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of the structure of a compression encoder module disclosed in an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of the structure of a compression decoder module disclosed in an embodiment of the present invention;

[0025] Figure 7 is a schematic diagram of the structure of a feature extraction module disclosed in an embodiment of the present invention;

[0026] Figure 8 is a schematic diagram of the structure of a ControlNet module disclosed in an embodiment of the present invention;

[0027] Fig. 9 It is a schematic diagram of the denoising and de-noising process of the diffusion model disclosed in the embodiment of the present invention. DETAILED DESCRIPTION

[0028] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0029] Figure 1 FIG. 1 is an overall flow chart of the method for generating rail transit images disclosed in an embodiment of the present invention. Figure 1 As shown, the method includes: acquiring a rail transit image; and inputting the rail transit image into a rail transit image generation network, and performing the following operations in the rail transit image generation network: performing preliminary processing on the rail transit image to generate latent space features, compressing the latent space features, and encoding them into latent space features of different dimensions; directly extracting the explicit features of the rail transit image, extracting the deep features of the preliminary processed image, combining the display features with the deep features, and upsampling them to the dimension of the rail transit image to obtain feature prompts; fusing the feature prompts with the latent space features of different dimensions through a ControlNet module to generate auxiliary features; based on the auxiliary features, guiding the diffusion model to perform a denoising process on the preliminary processed image to obtain generated image features; and decoding the generated image features into an output image.

[0030] Figure 2 Schematic diagram of the structure of the rail transit image generation network disclosed in the embodiment of the present invention. Figure 2As shown, the track image generation network includes a VAE module (Variational Auto-Encoder), a compression encoder and compression decoder module (Generative Adversial Network), a feature extraction module, a ControlNet module, and a pre-trained model parameter LDM diffusion model (Latent Diffusion Model).

[0031] Specifically, the rail transit image is input into the rail transit image generation network, and firstly processed by the VAE module. The encoder part in the VAE module uses fewer linear layers and activation functions to transform the input image from high dimension to low dimension to save computing resources; then the image features in the encoder are input into the implicit distribution representation part to randomly extract features to obtain richer features, and finally the obtained features are decoded into the original image dimensions and input into the subsequent modules as latent space features. The use of this module avoids invalid encoding of input data and enriches the randomness of data. The latent space features generated by the preliminary processing of the VAE module are compressed and encoded to improve the image generation quality of the model, and the latent space features are encoded into different dimensions for subsequent correspondence with the feature dimensions in the feature extraction module. The feature extraction module extracts the display features and deep features of the image and combines them, so that the texture information and semantic information of the image can be better utilized. The ControlNet module copies the structure and some parameters of the LDM diffusion model of the pre-trained model parameters during training, avoiding the problems of high resource consumption and long time caused by retraining. The auxiliary features generated by the ControlNet module are used to guide the generation process of the pre-trained model parameter LDM diffusion model, so that the generated image is closer to the real image.

[0032] Figure 3 FIG. 1 is a schematic diagram of a process for obtaining rail transit images disclosed in an embodiment of the present invention. Figure 3 As shown, the acquisition of rail transit images includes: self-collected real data of train operation scenes, multi-view rail transit image data crawled through the network, and rail transit image data captured through video.

[0033] Specifically, for the collected train vehicle operation data, the key frame data in the video is extracted according to the specified frame interval to obtain a coarse-screened data set, and then the blurred, repeated, and damaged data are screened and eliminated to obtain the self-collected real data of the train operation scene; a large amount of multi-perspective track image data from the network is collected, and multi-perspective rail transit image data is obtained through data enhancement means; a large amount of rail transit videos from the network are collected, and the key frame data in the video is extracted according to the specified frame interval to obtain a coarse-screened data set, and then the blurred, repeated, and damaged data are screened and eliminated to obtain high-quality rail transit data. The track centerline will be marked for the rails in the data through manual labeling, and the rest does not need to be labeled. The above-mentioned method of obtaining rail transit images is only for exemplary description, including but not limited to these three methods of obtaining images. Those skilled in the art can make corresponding changes to the method of obtaining images according to the actual application field.

[0034] In some embodiments, preliminary processing of the rail transit image to generate latent space features includes: using a VAE module to transform the features of the rail transit image from high dimension to low dimension, extracting the most significant feature information of the rail transit image; and randomly extracting features of the most significant feature information and restoring it to the original feature dimension of the rail transit image to obtain the latent space features; wherein the VAE module includes an encoder part, an implicit distribution representation part, and a decoder part.

[0035] Specifically, Figure 4 It is a structural diagram of the VAE module disclosed in the embodiment of the present invention. The VAE module consists of three stages, wherein the first stage is the encoder part, which is composed of "Linear1+Relu+Linear2+Linear3" operation blocks. The second stage is the implicit distribution representation part, which is composed of "randn+Sqrt" operation blocks. The third stage is the decoder part, which is composed of "Linear1+Relu+Linear2+Relu+Linear3" operation blocks. The 3×512×512 image data is input into the VAE module, and the image data is first converted into a one-dimensional linear feature, and then input into the encoder part to gradually convert the image data into a high-dimensional linear feature of 256 length, and the extracted 256-length high-dimensional feature is input into the implicit distribution representation part, and the "randn+Sqrt" operation block is used to combine a Gaussian distribution function similar to the input feature. Finally, similar data of length 256 is sampled from the obtained Gaussian function, and input into the decoder part to restore it to the same 3×512×512 data dimension as the original image, and input into the subsequent module as a latent space feature. It can be understood that the specific values ​​of the image dimensions in this embodiment are only for illustrative purposes.

[0036] In some embodiments, compressing the latent space features and encoding them into latent space features of different dimensions includes: using a compression encoder module to gradually encode the latent space features from large to small, encoding them into latent space features of different dimensions to correspond to the dimensions of the feature prompts; the compression encoder module is cascaded by processing blocks of different dimensions, and the processing blocks include a Conv structure and two PixelNorms, and a Downsample module and a convolution module are used between every two dimensions to ensure dimensional matching.

[0037] Specifically, Figure 5 is a schematic diagram of the structure of a compression encoder module disclosed in an embodiment of the present invention, Figure 6 The structure diagram of the compression decoder module disclosed in the embodiment of the present invention is shown in FIG. The compression encoder structure corresponds to the decoder, and the compression encoder and the compression decoder are introduced to improve the quality of the generated image. Through the control of the codec, the generated image has a richer and more realistic generation effect in terms of image details, texture details, etc.

[0038] In some embodiments, the feature prompt is generated by a feature extraction module; the feature extraction module includes an explicit feature extraction part and a deep feature extraction part; the explicit feature extraction part includes 4 canny functions, which are used to extract image edge features of different sizes, and upsample the edge features of different sizes to the original image resolution to obtain texture detail features at different levels; the deep feature extraction part includes 4 groups of feature extractors for extracting image features of different dimensions, each group of extractors includes three groups of cascaded Conv+Relu+Maxpool operation blocks, which are used to extract image features corresponding to the explicit feature output dimensions, and combine them into feature prompts of different dimensions.

[0039] Specifically, the structure of the feature extraction module is as follows: Figure 7 As shown. The feature extraction module first extracts the edge information feature1 that is consistent with the dimension of the input rail transit image and rich in details, and then obtains feature2, feature3, and feature4 of 256, 128, and 64 dimensions through continuous convolution and downsampling, and then restores the feature dimension of the original image to 512 dimensions. The extracted deep features often pay more attention to the overall structural information in the image, and have a good auxiliary effect on controlling the overall structure of the image generated by the diffusion model. It can be understood that the extraction of features of four dimensions of 512, 256, 128, and 64 in the embodiments of the present invention is exemplary rather than restrictive, and those skilled in the art can extract image features of different dimensions and numbers according to the size of different images and the different requirements for the features to be obtained.

[0040] In some embodiments, the explicit features are directly extracted edge features that are consistent with the dimensions of the rail transit image and of different sizes; the deep features are features of different dimensions obtained by continuously convolving and downsampling the initially processed image.

[0041] Specifically, combining the explicit features of different dimensions with the deep features can help the diffusion model use feature information of different dimensions at different stages of image generation: in the early stage, the general explicit features and the deep features are used to assist the diffusion model in generating structurally determined backbone information; as the generation process of the diffusion model continues to deepen, different dimensions and more detailed feature information are gradually used to assist the diffusion model in generating the texture and detail semantics of the image.

[0042] In some embodiments, the ControlNet module includes: a conditional feature extraction module, including 4 SD Encoders of different sizes; an intermediate layer module, including an SD intermediate layer module and an intermediate zero convolution layer of the same size; and a zero convolution layer module, including 4 zero convolution layers of different sizes; wherein the conditional feature extraction module and the intermediate layer module use the same structure and parameters as the diffusion model.

[0043] In some embodiments, generating the auxiliary features includes: copying part of the pre-trained parameters of the diffusion model to train the ControlNet module, inputting the feature hints and the latent space features of different dimensions into the ControlNet module to guide parameter update, thereby generating the auxiliary features.

[0044] Specifically, Figure 8 It is a structural diagram of the ControlNet module disclosed in an embodiment of the present invention. The structure in the ControlNet module is consistent with the diffusion model and pre-trained parameters are loaded from the diffusion model, which is used to ensure that the training parameters can be directly loaded into the diffusion model, reducing the computing resources and time consumption caused by dimensional conversion. The zero convolution layer module can effectively maintain the original size and position information of the feature map without introducing additional calculations, and enables the network to transfer information between different layers without considering the problem of feature map size mismatch.

[0045] In some embodiments, the diffusion model includes: an SD encoder stage including four SD Encoders; an SD mid-layer including a mid-layer block; and an SD decoder including four SD Decoders.

[0046] Specifically, the diffusion model includes 1000 repeated U-nets, each of which includes three stages, the first stage is the SD encoder stage, the second stage is the SD middle layer, and the third stage is the SD decoder. The diffusion model acts on the latent space of the image. Compared with the RGB space, the number of features in the latent space is smaller, which makes the reasoning speed of the diffusion model faster. And the latent space acted by the diffusion model is a practical visual codebook, so that the diffusion model has a good generation effect on the semantics and details of the image. It is worth noting that the diffusion model used in the embodiment of the present invention is a pre-trained model parameter LDM diffusion model is only used as an example, and those skilled in the art can replace it with other diffusion models according to the needs of actual applications.

[0047] In some embodiments, the denoising process includes: setting the number of iteration steps of the denoising process; inputting the auxiliary feature, using the latent space feature to subtract the auxiliary feature to obtain a new latent space feature; using the new latent space feature to subtract the auxiliary feature again; and continuously iterating the above process until the number of iteration steps is completed to obtain the generated image feature.

[0048] Specifically, Fig. 9 It is a schematic diagram of the diffusion model denoising and denoising process disclosed in an embodiment of the present invention. The diffusion model includes a denoising process and a denoising process. In the training stage, the diffusion model is set to execute 1000 steps. In the training stage, the denoising process uses a denoising function to add gradually enhanced random Gaussian noise step by step, so that the original input data gradually becomes pure noise data; the denoising process is composed of 1000 U-net cascades, corresponding to 1000 steps in the denoising process. When the denoising process occurs, a U-Net predicted noise matrix is ​​introduced for each iteration step, and the predicted noise matrix is ​​subtracted from the picture containing noise in the previous step to produce a picture that is closer to the result and has less noise. After several iterations, the latent space features of the picture without noise are obtained. It should be noted that the 1000 steps given here are only exemplary, and those skilled in the art can set more or fewer steps according to actual needs.

[0049] The embodiment of the present invention inputs the rail transit image into the rail transit image generation network, and after the VAE module performs preprocessing operations on the image, the generated latent space features are input into the pre-trained model parameter LDM diffusion model as image input. The feature prompts output by the feature extraction module and the compressed image features are fused in the ContorlNet module, and the LDM diffusion model is jointly guided to generate image features, and finally the compression decoder module decodes the generated image features into an output image of the same size as the input. The image generation method disclosed in the embodiment of the present invention focuses on extracting features in different dimensions within the image, using feature information of different dimensions, and processing the input image into latent space features, so that the features pay more attention to deep semantic information, and avoid excessive number of feature channels that will cause information redundancy and reduce the amount of calculation. Using the feature extraction module to directly extract image features for model fine-tuning can make full use of image data for direct training, solve the problem of a large amount of data required for fine-tuning the text-image diffusion model, and avoid high-cost manual annotation. At the same time, the diffusion model can be simply and efficiently fine-tuned using a small number of acquired images, and a variety of images can be generated for subsequent tasks.

[0050] A second aspect of an embodiment of the present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute any of the rail transit image generation methods described above in the present application.

[0051] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0052] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0053] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0054] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0055] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0056] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0057] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0058] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0059] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for generating an image of rail transit data, characterized in that: The method comprises: Acquiring rail transit images; and The rail transit image is input into a rail transit image generation network, and the following operations are performed in the rail transit image generation network: Preliminarily processing the rail transit image to generate latent space features, compressing the latent space features, and encoding them into latent space features of different dimensions; Directly extracting explicit features of the rail transit image, extracting deep features of the preliminarily processed image, combining the explicit features with the deep features, and upsampling to the dimension of the rail transit image to obtain feature hints; fusing the feature prompts with the latent space features of different dimensions through a ControlNet module to generate auxiliary features; Instructing the diffusion model to perform a denoising process on the preliminarily processed image according to the auxiliary features to obtain generated image features; and The generated image features are decoded into an output image.

2. The image generation method according to claim 1, characterized in that: The preliminary processing of the rail transit image to generate latent space features comprises: Using a VAE module to transform the features of the rail transit image from high dimension to low dimension, and extract the most significant feature information of the rail transit image; and Randomly extracting features from the most significant feature information and restoring them to the original feature dimensions of the rail transit image to obtain the latent space features; The VAE module includes an encoder part, an implicit distribution representation part, and a decoder part.

3. The image generation method according to claim 1, characterized in that: The step of compressing the latent space features and encoding them into latent space features of different dimensions includes: Using a compression encoder module to gradually encode the latent space features from large to small, the latent space features are encoded into latent space features of different dimensions to correspond to the dimensions of the feature prompts; The compression encoder module is formed by cascading processing blocks of different dimensions. The processing block includes a Conv structure and two PixelNorms. A Downsample module and a convolution module are used between every two dimensions to ensure dimension matching.

4. The image generation method according to claim 1, characterized in that: The feature prompt is generated by a feature extraction module; The feature extraction module includes an explicit feature extraction part and a deep feature extraction part; The explicit feature extraction part includes 4 canny functions, which are used to extract image edge features of different sizes, and upsample the edge features of different sizes to the original image resolution to obtain texture detail features of different levels; The deep feature extraction part includes 4 groups of feature extractors for extracting image features of different dimensions. Each group of extractors contains three groups of cascaded Conv+Relu+Maxpool operation blocks, which are used to extract image features corresponding to the explicit feature output dimensions and combine them into feature prompts of different dimensions.

5. The image generation method according to claim 1, characterized in that: The explicit features are directly extracted edge features that are consistent with the dimensions of the rail transit image and have different sizes; The deep features are features of different dimensions obtained by continuously convolving and downsampling the initially processed image.

6. The image generation method according to claim 1, characterized in that: The ControlNet module includes: Conditional feature extraction module, including 4 SD Encoders of different sizes; An intermediate layer module, including an SD intermediate layer module and an intermediate zero convolutional layer of the same size; and Zero convolution layer module, including 4 zero convolution layers of different sizes; The conditional feature extraction module and the intermediate layer module use the same structure and parameters as the diffusion model.

7. The image generation method according to claim 1, characterized in that: The generating of auxiliary features comprises: Part of the pre-trained parameters of the diffusion model are copied to train the ControlNet module, and the feature prompts and the latent space features of different dimensions are input into the ControlNet module to guide parameter update, thereby generating the auxiliary features.

8. The image generation method according to claim 1, characterized in that: The diffusion model includes: SD encoder stage, including four SD Encoders; SD mid-layer, including a mid-layer block; and SD decoder, including four SD Decoders.

9. The image generation method according to claim 1, characterized in that: The denoising process includes: Setting the number of iteration steps of the denoising process; Input the auxiliary feature, and use the latent space feature to subtract the auxiliary feature to obtain a new latent space feature; Using the new latent space feature to subtract the auxiliary feature again; and The above process is continuously iterated until the number of iteration steps is completed to obtain the generated image features.

10. A machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute any of the rail transit data image generation methods described above in the present application.

Images