A method, device and equipment for generating images of multi-modal railway foreign objects

Abstract

This application discloses a method, apparatus, and device for generating images of multimodal railway foreign objects, aiming to improve the accuracy of identified railway foreign objects. The application acquires a railway track surface image containing the target section and textual description information; inputs the textual description information into a text encoder to obtain a semantic condition vector; inputs the railway track surface image into a track surface detection model to obtain a rail bounding box; obtains a first target region and a second target region based on the rail bounding box; fuses the first and second target regions to obtain a target image; and inputs the target image and semantic condition vector into a multimodal diffusion model to obtain railway foreign object information. By fusing textual description information and the railway track surface image as generation conditions, and employing an adaptive resolution differentiation processing strategy for the first and second target regions, not only is high-precision spatial control of the foreign object generation process achieved, but the physical rationality of the generated foreign object is also significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method, apparatus and equipment for generating images of multimodal railway foreign objects. Background Technology

[0002] With the rapid development of the high-speed railway industry, the importance of ensuring train operation safety has become increasingly prominent. Foreign object intrusion on tracks has become one of the core risk factors threatening high-speed railway safety. Currently, in practical applications, the probability of foreign object intrusion incidents is relatively low, and the forms of intruding foreign objects are diverse. Furthermore, due to limitations in on-site safety management requirements and the complex and ever-changing natural environment, it is difficult to collect sufficient sample data covering different weather conditions, lighting conditions, and different foreign object postures and types. This results in foreign object detection models trained based on existing samples having weak generalization ability and a high false alarm rate, failing to meet the practical application needs of foreign object detection on high-speed railway tracks. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus and equipment for generating images of multimodal railway foreign objects, so as to improve the accuracy of identifying railway foreign objects.

[0004] In a first aspect, embodiments of this application provide an image generation method for multimodal railway foreign objects, the method comprising: Obtain a railway track surface image containing the target section, and text description information based on the railway track surface image; The text description information is input into a pre-trained text encoder to obtain the semantic condition vector output by the text encoder; The railway track surface image is input into a pre-trained track surface detection model to obtain the railway track bounding box output by the track surface detection model; A first target region and a second target region are obtained based on the railway boundary frame; the first target region is obtained based on the railway boundary frame, and the second target region is obtained based on the first target region. The first target region and the second target region are fused together to obtain a target image; The target image and the semantic conditional vector are input into the trained multimodal diffusion model to obtain the railway foreign object information output by the multimodal diffusion model.

[0005] In this embodiment, by fusing textual description information and railway track surface images as core generation conditions, and employing an adaptive resolution differentiation processing strategy for key areas (i.e., the first target area) and background areas (i.e., the second target area), not only is high-precision spatial control of the foreign object generation process achieved, but the physical rationality of the generated foreign object is also significantly improved. At the same time, combined with boundary fusion algorithms, the overall visual coherence and realism of the generated image are effectively guaranteed, fundamentally avoiding problems such as artifacts and seams that affect image quality.

[0006] In some possible embodiments, obtaining the first target area and the second target area based on the railway track boundary includes: The railway boundary box is used as the region of interest. The region of interest is subjected to local diffusion processing to obtain the first target region; Based on the region of interest and the railway track surface image, the region of non-interest is obtained; Image processing is performed on the non-interest region to obtain the second target region.

[0007] In some possible embodiments, the step of performing local diffusion processing on the region of interest to obtain the first target region includes: The width of the buffer area is obtained based on the rail surface height corresponding to the railway rail surface image and the pre-set mapping relationship between the rail surface height and the width of the buffer area; The railway boundary frame is expanded based on the width of the buffer area to obtain the region of interest.

[0008] In some possible embodiments, the image processing of the non-interest region to obtain the second target region includes: The non-interest region is downsampled to obtain the first image; The first image is subjected to bilinear interpolation to obtain the second image; The second image is input into a pre-trained thinning network to obtain the second target region output by the thinning network.

[0009] In some possible embodiments, after obtaining the railway foreign object information output by the multimodal diffusion model, the method further includes: The railway foreign object information is input into a pre-trained image decoder to obtain a railway scene image output by the image decoder; the railway scene image contains the railway foreign object corresponding to the railway foreign object information.

[0010] In some possible embodiments, the fusion processing of the first target region and the second target region to obtain the target image includes: Obtain the image quality value and the current time; The quality efficiency balance coefficient is obtained based on the image quality value and the current time. The compression ratio is determined based on the aforementioned quality efficiency balance coefficient; Determine the boundary pixels between the first target region and the second target region; The boundary pixels are compressed according to the compression ratio to obtain the compressed pixels. The target image is obtained based on the compressed pixels, the first target region, and the second target region.

[0011] In some possible embodiments, after acquiring the railway track surface image containing the target section, the method further includes: The railway track surface image is denoised to obtain a denoised railway track surface image; The step of inputting the railway track surface image into a pre-trained track surface detection model includes: The denoised railway track surface image is input into the pre-trained track surface detection model.

[0012] Secondly, embodiments of this application provide an image generation apparatus for multimodal railway foreign objects, the apparatus comprising: The image acquisition module is used to acquire railway track surface images containing the target section, as well as text description information corresponding to the railway track surface images; The text encoding module is used to input the text description information into a pre-trained text encoder to obtain the semantic condition vector output by the text encoder. The bounding box determination module is used to input the railway track surface image into a pre-trained track surface detection model to obtain the railway track bounding box output by the track surface detection model. The region determination module is used to obtain a first target region and a second target region based on the railway boundary frame; the first target region is obtained based on the railway boundary frame, and the second target region is obtained based on the first target region; The fusion module is used to fuse the first target region and the second target region to obtain a target image; The foreign object identification module is used to input the target image and the semantic condition vector into a trained multimodal diffusion model to obtain the railway foreign object information output by the multimodal diffusion model.

[0013] In some possible embodiments, the region determination module is specifically used for: The railway boundary box is used as the region of interest. The region of interest is subjected to local diffusion processing to obtain the first target region; Based on the region of interest and the railway track surface image, the region of non-interest is obtained; Image processing is performed on the non-interest region to obtain the second target region.

[0014] In some possible embodiments, the region determination module is specifically used for: The width of the buffer area is obtained based on the rail surface height corresponding to the railway rail surface image and the pre-set mapping relationship between the rail surface height and the width of the buffer area; The railway boundary frame is expanded based on the width of the buffer area to obtain the region of interest.

[0015] In some possible embodiments, the region determination module is specifically used for: The non-interest region is downsampled to obtain the first image; The first image is subjected to bilinear interpolation to obtain the second image; The second image is input into a pre-trained thinning network to obtain the second target region output by the thinning network.

[0016] In some possible embodiments, the foreign object detection module is also used for: The railway foreign object information is input into a pre-trained image decoder to obtain a railway scene image output by the image decoder; the railway scene image contains the railway foreign object corresponding to the railway foreign object information.

[0017] In some possible embodiments, the fusion module is specifically used for: Obtain the image quality value and the current time; The quality efficiency balance coefficient is obtained based on the image quality value and the current time. The compression ratio is determined based on the aforementioned quality efficiency balance coefficient; Determine the boundary pixels between the first target region and the second target region; The boundary pixels are compressed according to the compression ratio to obtain the compressed pixels. The target image is obtained based on the compressed pixels, the first target region, and the second target region.

[0018] In some possible embodiments, the image acquisition module is also used for: The railway track surface image is denoised to obtain a denoised railway track surface image; The step of inputting the railway track surface image into a pre-trained track surface detection model includes: The denoised railway track surface image is input into the pre-trained track surface detection model.

[0019] Thirdly, another embodiment of this application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods provided in the first aspect embodiment of this application.

[0020] Fourthly, another embodiment of this application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for causing a computer to perform any of the methods provided in the first aspect of this application.

[0021] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall process of an image generation method for multimodal railway foreign objects provided in an embodiment of this application; Figure 2 A flowchart illustrating the process of obtaining a first target region and a second target region based on the railway track boundary, as provided in an embodiment of this application, for a method of generating images of multimodal railway foreign objects. Figure 3 A flowchart illustrating the process of obtaining the region of interest based on the railway track boundary box in a method for generating images of multimodal railway foreign objects provided in this application embodiment; Figure 4 A schematic diagram of the region of interest and non-region of interest for an image generation method for multimodal railway foreign objects provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the process of image processing of a non-interest region to obtain a second target region in a multimodal railway foreign object image generation method provided in this application embodiment. Figure 6 A schematic diagram illustrating the process of fusing a first target region and a second target region in an image generation method for multimodal railway foreign objects provided in this application embodiment; Figure 7 A schematic diagram of an apparatus for generating images of multimodal railway foreign objects provided in an embodiment of this application; Figure 8 This is a schematic diagram of an electronic device for an image generation method of multimodal railway foreign objects provided in an embodiment of this application. Detailed Implementation

[0024] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0026] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0027] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0028] With the rapid development of the high-speed railway industry, the importance of ensuring train operation safety has become increasingly prominent. Foreign object intrusion on tracks has become one of the core risk factors threatening high-speed railway safety. Currently, in practical applications, the probability of foreign object intrusion incidents is relatively low, and the forms of intruding foreign objects are diverse. Furthermore, due to limitations in on-site safety management requirements and the complex and ever-changing natural environment, it is difficult to collect sufficient sample data covering different weather conditions, lighting conditions, and different foreign object postures and types. This results in foreign object detection models trained based on existing samples having weak generalization ability and a high false alarm rate, failing to meet the practical application needs of foreign object detection on high-speed railway tracks.

[0029] The rise of generative artificial intelligence technology has provided an effective technical path to solve the aforementioned problem of insufficient samples. Diffusion models, represented by Stable Diffusion, have demonstrated excellent performance in general image generation tasks. However, when such general diffusion models are directly applied to the task of generating images of foreign objects on railway tracks, there are still many technical shortcomings that need to be addressed: First, the control precision of the generation process is insufficient, making it difficult to accurately limit the generated foreign object targets to key areas of railway tracks such as rails and ballast. It is easily affected by complex background factors around the track, resulting in the generated foreign object target positions not conforming to the physical spatial constraints of foreign object intrusion in actual scenarios. Second, the rationality and realism of the generated images are poor. Key features such as the contact pattern between the generated foreign object and the rail surface, and the hanging state of foreign objects in the overhead contact line, often contradict actual physical laws, making it impossible to effectively simulate real track foreign object intrusion scenarios. Third, the inference and generation efficiency of the model is low. The original diffusion model has the problems of a large number of parameters and numerous iteration steps, which takes too long to generate high-resolution track foreign object images, making it difficult to meet the actual needs of generating diverse sample data in batches and quickly in railway track foreign object detection tasks.

[0030] To address the aforementioned problems, this application provides a method, apparatus, and device for generating images of multimodal railway foreign objects, thereby resolving these issues. The inventive concept of this application can be summarized as follows: acquiring a railway track surface image containing a target section, and textual description information corresponding to the railway track surface image; inputting the textual description information into a pre-trained text encoder to obtain a semantic condition vector output by the text encoder; inputting the railway track surface image into a pre-trained track surface detection model to obtain a rail bounding box output by the track surface detection model; obtaining a first target region and a second target region based on the rail bounding box; the first target region is obtained based on the rail bounding box, and the second target region is obtained based on the first target region; fusing the first target region and the second target region to obtain a target image; inputting the target image and the semantic condition vector into a trained multimodal diffusion model to obtain railway foreign object information output by the multimodal diffusion model.

[0031] In this embodiment, by fusing textual description information and railway track surface images as core generation conditions, and employing an adaptive resolution differentiation processing strategy for key areas (i.e., the first target area) and background areas (i.e., the second target area), not only is high-precision spatial control of the foreign object generation process achieved, but the physical rationality of the generated foreign object is also significantly improved. At the same time, combined with boundary fusion algorithms, the overall visual coherence and realism of the generated image are effectively guaranteed, fundamentally avoiding problems such as artifacts and seams that affect image quality.

[0032] For ease of understanding, the following detailed description, in conjunction with the accompanying drawings, illustrates a method for generating images of multimodal railway foreign objects according to an embodiment of this application: like Figure 1 The diagram shown is a schematic flowchart of an image generation method for multimodal railway foreign objects provided in an embodiment of this application, wherein: In step 101: Obtain the railway track surface image containing the target section, and the text description information corresponding to the railway track surface image.

[0033] In this embodiment of the application, the target section is the section of the railway where foreign objects need to be detected. The railway track surface image is a digital image generated by taking pictures or scanning the top surface of the rails (i.e. the working surface of the contact between the train wheels and the rails) of the target section using professional imaging equipment (such as line array cameras, area array cameras, laser scanning imaging devices, etc.). The text description information is the conversion of the visual features of the railway track surface image into readable, archiveable, and analyzable text content.

[0034] Specifically, technologies such as machine vision, image processing, or deep learning can be used to identify various visual elements in railway track surface images. These elements include whether the track surface is flat, whether there are scratches, peeling, cracks, or other defects, whether there are foreign objects such as fallen rocks or plastic bags, and the location, shape, and size of the foreign objects or defects, as well as the cleanliness and metallic texture of the track surface. After identifying the visual elements in the railway track surface image, these elements need to be converted into text information. That is, the parsed visual features are converted into standardized or colloquial text descriptions according to preset rules or natural language logic. For example, for the railway track surface image of target section 1, the text description information obtained from the railway track surface image is: There is a fallen rock at the target section KXXX+XXX, with a length of about 15 cm and a width of about 30 cm. The rest of the track surface has no obvious defects.

[0035] In some possible embodiments, in order to further improve the accuracy of the subsequently determined railway foreign object information, after acquiring the railway track surface image containing the target section, the following can be performed: denoising the railway track surface image to obtain the denoised railway track surface image.

[0036] It should be noted that when denoising railway track surface images, Gaussian filtering algorithm or other algorithms can be used, and this application does not limit the specific algorithms used.

[0037] In this embodiment, by denoising the railway track surface image, the recognizability of effective information in the railway track surface image can be improved, and the negative impact of interference factors on subsequent detection and analysis can be eliminated. If the railway track surface image is denoised after step 101, then the images used in subsequent processing will all be the denoised railway track surface images.

[0038] In step 102: The text description information is input into the pre-trained text encoder to obtain the semantic condition vector output by the text encoder.

[0039] In this embodiment, the text encoder takes textual description information as input and outputs a semantic condition vector. The CLIP-ViT-L / 14 model can be trained using a first training sample set. The converged CLIP-ViT-L / 14 model serves as the text encoder, which maps the textual description information into a 768-dimensional semantic condition vector. The first training sample set includes: textual description information corresponding to different railway track surface images, and a semantic condition vector corresponding to each railway track surface image.

[0040] Specifically, the input sequence length of the text encoder can be set to 77; a two-layer fully connected multi-layer perceptron (MLP) is used to encode the physical parameters into a 256-dimensional feature vector. The number of hidden layer neurons in the MLP is 512, and the activation function is the rectified linear unit (ReLU) function. The 768-dimensional semantic vector and the 256-dimensional physical feature vector are concatenated to form a 1024-dimensional fusion vector. The fusion vector is then input into the cross-attention modules of layers 3, 6, and 9 of the U-Net. Through the attention mechanism, the interaction between the conditional constraints and the generated features is realized, thereby ensuring that the generated content conforms to the semantic description and physical laws.

[0041] The training process of the CLIP-ViT-L / 14 model is the same as that of the neural network models in related technologies, and will not be described in detail here.

[0042] In step 103: The railway track surface image is input into the pre-trained track surface detection model to obtain the railway track bounding box output by the track surface detection model.

[0043] In this embodiment of the application, a pre-constructed second training sample set can be used to train the YOLOv7 model, and the YOLOv7 model that has been trained and converged can be used to train the track surface detection model. The second training sample set includes: multiple railway track surface images, and the railway track bounding box corresponding to each railway track surface image.

[0044] In step 104: the first target region and the second target region are obtained based on the railway boundary box; the first target region is obtained based on the railway boundary box, and the second target region is obtained based on the first target region.

[0045] In some possible embodiments, a first target area and a second target area are obtained based on the railway track boundary, which can be specifically implemented as follows: Figure 2 The steps shown are as follows: In step 201: the railway boundary box is designated as the region of interest.

[0046] In this embodiment of the application, in order to focus on key information, reduce computational costs and improve task accuracy and efficiency, after obtaining the railway track bounding box, the railway track bounding box is taken as the region of interest (ROI), and the area outside the ROI in the railway track surface image is taken as the non-interest region.

[0047] In step 202: Local diffusion processing is performed on the region of interest to obtain the first target region.

[0048] In this embodiment of the application, considering that the railway boundary box obtained in step 201 may not be complete during the aforementioned processing, the railway boundary box in the obtained region of interest may be incomplete. In order to ensure that the region of interest can contain a complete railway boundary box, the region of interest can be locally diffused to obtain a first target region containing a complete railway boundary box.

[0049] In some possible embodiments, the region of interest is obtained based on the railway track boundary box, which can be specifically implemented as follows: Figure 3 The steps shown are as follows: In step 301: the width of the buffer area is obtained based on the rail height corresponding to the railway rail surface image and the pre-set mapping relationship between the rail height and the width of the buffer area.

[0050] In this embodiment, the rail surface height refers to the vertical distance between the top surface of the railway rail (i.e., the bearing surface that the train wheels directly contact) and a preset reference datum surface. The preset reference datum surface can be a fixed structure such as the top surface of the sleeper or the surface of the track bed below the rail, or it can be an elevation datum surface during the design of the railway line (e.g., the design rail surface elevation of the line). This application does not limit the selection method of the preset reference datum surface.

[0051] In some possible embodiments, the mapping relationship between the pre-set rail surface height and the buffer area width is shown in Formula 1, where: , (Formula 1) in, The width of the buffer area. The rail height, This is the mapping coefficient between the rail height and the width of the buffer zone.

[0052] For example: when the rail surface height When = 100cm, =30cm.

[0053] In step 302: The railway boundary box is expanded according to the width of the buffer area to obtain the region of interest.

[0054] In this embodiment of the application, after obtaining the width of the buffer area, the railway boundary frame is expanded outward according to the width of the buffer area, and the resulting area containing the railway boundary frame is the region of interest.

[0055] For example: when the rail surface height When = 100cm, =30cm, expand the railway track boundary frame outward by 30cm to obtain the region of interest.

[0056] In step 203: Based on the region of interest and the railway track surface image, the region of non-interest is obtained.

[0057] In this embodiment of the application, after obtaining the region of interest, the remaining part after removing the portion of the region of interest from the railway track surface image is the region of non-interest.

[0058] For example: Figure 4 As shown, the region of interest is the white-filled part in the image. Removing this white part from the railway track surface image results in the diagonal line portion, which is the region of non-interest.

[0059] In step 204: Image processing is performed on the non-interest region to obtain the second target region.

[0060] In this embodiment of the application, in order to obtain the background features in the railway track surface image, the non-interest area can be image processed to obtain a second target area containing high-frequency details.

[0061] In some possible embodiments, image processing is performed on the non-interest region to obtain the second target region, specifically as follows: Figure 5 The steps shown are as follows: In step 501: the non-interest regions are downsampled to obtain the first image.

[0062] In this embodiment, the non-interest regions are downsampled by 8× to reduce the image resolution of the non-interest regions to 240×135, thus obtaining the first image.

[0063] In step 502: the first image is subjected to bilinear interpolation to obtain the second image.

[0064] In this embodiment of the application, after obtaining the first image, the low-resolution first image is upsampled to 1920×1080 by bilinear interpolation to obtain the second image.

[0065] In step 503: the second image is input into a pre-trained thinning network to obtain the second target region output by the thinning network.

[0066] In this embodiment, the thinning network can be a 3-layer convolutional thinning network with a kernel size of 3×3, a stride of 1, and a padding of 1. The second image is input into the pre-trained thinning network so that the thinning network can repair the high-frequency details in the second image, thereby obtaining a second target region containing high-frequency details.

[0067] In step 105: the first target region and the second target region are fused to obtain the target image.

[0068] In this embodiment of the application, in order to make the final target image more reasonable, after obtaining the first target region and the second target region, the edges of the first target region and the second target region are fused to obtain the target image.

[0069] In some possible embodiments, the first target region and the second target region are fused to obtain a target image, which can be specifically implemented as follows: Figure 6 The steps shown are as follows: In step 601: Obtain the image quality value and the current time.

[0070] In this embodiment of the application, the image quality value refers to the score obtained by scoring the railway track surface image based on the Structural Similarity Index (SSIM) metric. The current time is the execution time. Figure 1 The starting time of the steps shown.

[0071] In step 602: the quality efficiency balance coefficient is obtained based on the image quality value and the current time.

[0072] In this embodiment of the application, Formula 2 can be used to determine the mass efficiency balance coefficient, wherein: , (Formula 2) in, This is the quality-efficiency balance coefficient. Image quality value, This refers to the current moment.

[0073] In step 603: the compression ratio is determined based on the mass efficiency balance coefficient.

[0074] In this embodiment of the application, in order to improve the efficiency of image generation while ensuring the quality of generation, the compression ratio is 4× when the quality efficiency balance coefficient is greater than the quality efficiency balance threshold to ensure the quality of generation; and the compression ratio is 8× when the quality efficiency balance coefficient is less than or equal to the quality efficiency balance threshold to ensure the efficiency of generation.

[0075] In step 604: Determine the boundary pixel points between the first target region and the second target region.

[0076] In this embodiment of the application, when determining the boundary pixel, a 50-pixel (px) wide blending transition band can be constructed at the boundary between the first target area and the second target area. The pixel in the blending transition band is taken as the boundary pixel. The Gaussian weighted algorithm is used to fuse the features on both sides. The standard deviation of the Gaussian kernel is set to 5. The texture break and color jump at the boundary are eliminated by weighted averaging, thereby obtaining the pixel value of the boundary pixel.

[0077] In step 605: the boundary pixels are compressed according to the compression ratio to obtain the compressed pixels.

[0078] In this embodiment of the application, after obtaining the compression ratio, the pixels of the boundary pixels are compressed to obtain the compressed pixels.

[0079] In step 606: the target image is obtained based on the compressed pixels, the first target region, and the second target region.

[0080] In this embodiment of the application, the pixel values ​​of the boundary pixels are modulated based on the pixel values ​​of the compressed pixels to obtain the target image.

[0081] In step 106: the target image and semantic condition vector are input into the trained multimodal diffusion model to obtain the railway foreign object information output by the multimodal diffusion model.

[0082] In this embodiment, the master model of the multimodal diffusion model adopts the original Stable Diffusion UNet structure, which contains 128 network layers with a network channel count ranging from 320 to 1280. For the slave model, the number of network layers is reduced to 64 by decreasing the number of network layers, and the network channel count is reduced to a range of 192 to 768, thus achieving a lightweight design for the slave model. To achieve the prediction function of multi-step noise residuals, this model introduces a Long Short-Term Memory (LSTM) network. The hidden layer dimension of this LSTM network is set to 1024, the number of network layers is set to 2, and the dropout probability parameter is configured to 0.1. During the training phase of the model, the noise prediction result output by the master model is used as the supervision signal to calculate the mean squared error (MSE) between the noise prediction result output by the slave model and the noise prediction result output by the master model. The model incorporates a multi-step prediction error constraint term and a temporal consistency constraint term into the loss function to improve prediction accuracy and stability. After 50,000 iterations of training, the resulting model can accurately predict 5-step noise residuals using an LSTM network, thereby reducing the number of iterations in the diffusion process from 50 to 25, effectively improving the model's inference efficiency. During training, a mixed-precision 16-bit floating-point (FP16) training method is used to reduce the memory usage during training. After training, Dynamic Range Quantization (DRQ) is performed on the model to compress the precision of the model weights from 32 bits to 8 bits, while retaining the precision of the model activation values ​​at 16 bits. Tests have verified that this quantization operation can control the model's accuracy loss to within 3%.

[0083] In some possible embodiments, in order to make the identified railway foreign objects more intuitive, after obtaining the railway foreign object information output by the multimodal diffusion model, the following can be implemented: inputting the railway foreign object information into a pre-trained image decoder to obtain a railway scene image output by the image decoder; the railway scene image contains the railway foreign object corresponding to the railway foreign object information.

[0084] In this embodiment, railway foreign object information is input into an image decoder. The image decoder upsamples step by step through a 4-layer transposed convolution structure and finally outputs a railway foreign object sample image with a resolution of 1920×1080. In order to further improve the image quality, the original image of the railway foreign object sample can be further post-processed, that is, adaptive histogram equalization is used to enhance the image contrast, ensure the distinction between foreign objects and background, and finally obtain a high-quality railway scene image.

[0085] Based on the same inventive concept, after introducing the image generation method for multimodal railway foreign objects provided in the embodiments of this application, as follows... Figure 7 As shown, the following describes a multimodal railway foreign object image generation device 700 provided in an embodiment of this application. The device includes: The image acquisition module 7001 is used to acquire a railway track surface image containing the target section, and text description information corresponding to the railway track surface image; The text encoding module 7002 is used to input the text description information into a pre-trained text encoder to obtain the semantic condition vector output by the text encoder. The bounding box determination module 7003 is used to input the railway track surface image into a pre-trained track surface detection model to obtain the railway bounding box output by the track surface detection model; The region determination module 7004 is used to obtain a first target region and a second target region based on the railway boundary frame; the first target region is obtained based on the railway boundary frame, and the second target region is obtained based on the first target region; The fusion module 7005 is used to fuse the first target region and the second target region to obtain a target image; The foreign object determination module 7006 is used to input the target image and the semantic condition vector into a trained multimodal diffusion model to obtain the railway foreign object information output by the multimodal diffusion model.

[0086] In some possible embodiments, the region determination module 7004 is specifically used for: The railway boundary box is used as the region of interest. The region of interest is subjected to local diffusion processing to obtain the first target region; Based on the region of interest and the railway track surface image, the region of non-interest is obtained; Image processing is performed on the non-interest region to obtain the second target region.

[0087] In some possible embodiments, the region determination module 7004 is specifically used for: The width of the buffer area is obtained based on the rail surface height corresponding to the railway rail surface image and the pre-set mapping relationship between the rail surface height and the width of the buffer area; The railway boundary frame is expanded based on the width of the buffer area to obtain the region of interest.

[0088] In some possible embodiments, the region determination module 7004 is specifically used for: The non-interest region is downsampled to obtain the first image; The first image is subjected to bilinear interpolation to obtain the second image; The second image is input into a pre-trained thinning network to obtain the second target region output by the thinning network.

[0089] In some possible embodiments, the foreign object detection module 7006 is further configured to: The railway foreign object information is input into a pre-trained image decoder to obtain a railway scene image output by the image decoder; the railway scene image contains the railway foreign object corresponding to the railway foreign object information.

[0090] In some possible embodiments, the fusion module 7005 is specifically used for: Obtain the image quality value and the current time; The quality efficiency balance coefficient is obtained based on the image quality value and the current time. The compression ratio is determined based on the aforementioned quality efficiency balance coefficient; Determine the boundary pixels between the first target region and the second target region; The boundary pixels are compressed according to the compression ratio to obtain the compressed pixels. The target image is obtained based on the compressed pixels, the first target region, and the second target region.

[0091] In some possible embodiments, the image acquisition module 7001 is further configured to: The railway track surface image is denoised to obtain a denoised railway track surface image; The step of inputting the railway track surface image into a pre-trained track surface detection model includes: The denoised railway track surface image is input into the pre-trained track surface detection model.

[0092] Corresponding to the above embodiments, this application also provides an electronic device. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 800 may include a processor 801, a memory 802, and a communication unit 803. These components communicate through one or more buses. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0093] The communication unit 803 is used to establish a communication channel, enabling the electronic device to communicate with other devices. It receives user data sent by other devices or sends user data to other devices.

[0094] The processor 801 serves as the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 802, and calls data stored in the memory to perform various functions and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 801 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.

[0095] The memory 802 is used to store the execution instructions of the processor 801. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0096] When the execution instructions in memory 802 are executed by processor 801, the electronic device 800 is able to perform operations. Figure 1 Some or all of the steps in the illustrated embodiments.

[0097] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps of the various embodiments of the multimodal railway foreign object image generation method provided by the present invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0098] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0099] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

Claims

1. A method for generating images of multimodal railway foreign objects, characterized in that, The method includes: Obtain a railway track surface image containing the target section, and text description information based on the railway track surface image; The text description information is input into a pre-trained text encoder to obtain the semantic condition vector output by the text encoder; The railway track surface image is input into a pre-trained track surface detection model to obtain the railway track bounding box output by the track surface detection model; A first target region and a second target region are obtained based on the railway boundary frame; the first target region is obtained based on the railway boundary frame, and the second target region is obtained based on the first target region. The first target region and the second target region are fused together to obtain a target image; The target image and the semantic conditional vector are input into the trained multimodal diffusion model to obtain the railway foreign object information output by the multimodal diffusion model.

2. The method according to claim 1, characterized in that, The process of obtaining the first target region and the second target region based on the railway boundary frame includes: The railway boundary box is used as the region of interest. The region of interest is subjected to local diffusion processing to obtain the first target region; Based on the region of interest and the railway track surface image, the region of non-interest is obtained; Image processing is performed on the non-interest region to obtain the second target region.

3. The method according to claim 2, characterized in that, The step of performing local diffusion processing on the region of interest to obtain the first target region includes: The width of the buffer area is obtained based on the rail surface height corresponding to the railway rail surface image and the pre-set mapping relationship between the rail surface height and the width of the buffer area; The railway boundary frame is expanded based on the width of the buffer area to obtain the region of interest.

4. The method according to claim 2, characterized in that, The image processing of the non-interest region to obtain the second target region includes: The non-interest region is downsampled to obtain the first image; The first image is subjected to bilinear interpolation to obtain the second image; The second image is input into a pre-trained thinning network to obtain the second target region output by the thinning network.

5. The method according to claim 1, characterized in that, After obtaining the railway foreign object information output by the multimodal diffusion model, the method further includes: The railway foreign object information is input into a pre-trained image decoder to obtain a railway scene image output by the image decoder; the railway scene image contains the railway foreign object corresponding to the railway foreign object information.

6. The method according to claim 1, characterized in that, The step of fusing the first target region and the second target region to obtain the target image includes: Obtain the image quality value and the current time; The quality efficiency balance coefficient is obtained based on the image quality value and the current time. The compression ratio is determined based on the aforementioned quality efficiency balance coefficient; Determine the boundary pixels between the first target region and the second target region; The boundary pixels are compressed according to the compression ratio to obtain the compressed pixels. The target image is obtained based on the compressed pixels, the first target region, and the second target region.

7. The method according to claim 1, characterized in that, After acquiring the railway track surface image containing the target section, the method further includes: The railway track surface image is denoised to obtain a denoised railway track surface image; The step of inputting the railway track surface image into a pre-trained track surface detection model includes: The denoised railway track surface image is input into the pre-trained track surface detection model.

8. An image generation device for multimodal railway foreign objects, characterized in that, The device includes: The image acquisition module is used to acquire railway track surface images containing the target section, as well as text description information corresponding to the railway track surface images; The text encoding module is used to input the text description information into a pre-trained text encoder to obtain the semantic condition vector output by the text encoder. The bounding box determination module is used to input the railway track surface image into a pre-trained track surface detection model to obtain the railway track bounding box output by the track surface detection model. The region determination module is used to obtain a first target region and a second target region based on the railway boundary frame; the first target region is obtained based on the railway boundary frame, and the second target region is obtained based on the first target region; The fusion module is used to fuse the first target region and the second target region to obtain a target image; The foreign object identification module is used to input the target image and the semantic condition vector into a trained multimodal diffusion model to obtain the railway foreign object information output by the multimodal diffusion model.

9. An electronic device, characterized in that, It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1-7.

Images