A method for generating images of spilled materials on highways from an aerial perspective

Abstract

This invention relates to the field of intelligent transportation technology, and in particular to a method for generating images of spilled materials on highways from an aerial perspective. The method involves the following steps: road region segmentation using a pre-trained semantic segmentation model to generate a binary mask for the road region; vehicle target elimination using a pre-trained instance segmentation model to generate a vehicle contour mask; spilled material mask generation by generating a randomly shaped polygonal mask within the effective road area; local image preprocessing to perform geometric correction and size standardization on the cropped area; conditional image generation using a diffusion model combined with spatial conditions and textual prompts to generate a spilled material image adapted to the background; and image fusion and annotation to fuse the spilled material image back into the original image and automatically generate a standard annotation file. This invention can generate visually consistent and realistic composite images of spilled materials, achieve automatic generation of annotation files, significantly reduce data preparation costs, and improve data generation efficiency and diversity.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a method for generating image data of debris spilled on highways from an aerial perspective. Background Technology

[0002] In the field of intelligent transportation technology, accurate detection of debris on highways is crucial for ensuring road safety. However, training debris detection models relies on a large amount of high-quality aerial image data. Currently, there are significant bottlenecks in acquiring this type of data: Firstly, the occurrence of debris on highways is random and sudden, making it difficult and costly to collect data on debris in real aerial scenarios. Furthermore, the types of debris covered are limited, failing to meet the diverse data requirements for model training. Secondly, traditional methods of manually synthesizing debris images are inefficient. They require manual adjustments to the position and shape of the debris to fit the road scene and are prone to visual inconsistencies between the debris and the background environment, such as debris landing in non-road areas or overlapping with vehicles. They also cannot adapt to the perspective distortion unique to aerial photography, resulting in significant differences between the synthesized data and the real scene. In addition, manually labeling the type and location information of debris is not only labor-intensive but also prone to subjective errors leading to insufficient labeling accuracy, further affecting the training effect of the detection model. These problems collectively restrict the performance improvement of highway spill detection models from aerial perspectives, and there is an urgent need for an efficient and accurate method for generating spill image data. Therefore, this paper proposes a method for generating highway spill image data from aerial perspectives to address the above problems. Summary of the Invention

[0003] The purpose of this invention is to provide a method for generating image data of debris scattered on highways from an aerial perspective, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A method for generating image data of debris spilled on a highway from an aerial perspective includes the following steps:

[0006] Step S1, Road Region Segmentation: The acquired drone aerial images are processed using a pre-trained semantic segmentation model to generate binary masks for road regions.

[0007] Step S2, Vehicle Target Elimination: The original image is processed using a pre-trained instance segmentation model to identify and generate contour masks for all vehicles in the image.

[0008] Step S3, Spray Mask Generation: Based on the binary mask of the road area, and excluding the area covered by the vehicle contour mask, generate one or more random-shaped polygonal masks for placing the spill within the remaining valid road area.

[0009] Step S4, Local Image Preprocessing: Based on the bounding box coordinates of each random-shaped polygonal mask, the corresponding image region is cropped from the original image, and the image region is geometrically corrected and its size is standardized to obtain a standardized local background image.

[0010] Step S5, Conditional Image Generation: The standardized local background image and the randomly shaped polygon mask are used together as spatial conditions, and combined with text prompts describing specific types of projectiles, and input into the pre-trained diffusion model; The diffusion model uses the aforementioned spatial conditions as constraints and the aforementioned text prompts as semantic guidance to generate a projectile image that is in harmony with the surrounding environment in the region of the local background image corresponding to the mask.

[0011] Step S6, Image Fusion and Annotation: The generated projectile image is fused back into the corresponding random-shaped polygon mask position in the original image to form a visually consistent synthetic image; at the same time, based on the coordinate information of the random-shaped polygon mask, an annotation file for training the object detection model is automatically generated.

[0012] As a preferred approach, the specific steps for generating the binary mask for the road area in step S1 include:

[0013] S1-1. Input the original aerial image captured by the drone into the pre-trained semantic segmentation model. The semantic segmentation model outputs a probability map of each pixel belonging to the road category.

[0014] S1-2. Apply a probability threshold to the probability map for binarization. Pixels with probability values ​​higher than the threshold are identified as road areas and assigned a first pixel value. Pixels with probability values ​​lower than the threshold are identified as non-road areas and assigned a second pixel value, thereby generating an initial road area mask.

[0015] S1-3. Perform morphological post-processing on the initial road area mask to eliminate small noise holes and smooth the boundary contour of the road area.

[0016] S1-4. Output the post-processed mask as the final binary mask for the road area.

[0017] As a preferred approach, the original image is processed using a pre-trained instance segmentation model to identify and generate contour masks for all vehicles in the image, including the following steps:

[0018] Step S2-1: Input the original aerial image captured by the drone into the pre-trained instance segmentation model. The instance segmentation model outputs the initial mask and the corresponding class confidence of each detected vehicle instance in the image.

[0019] Step S2-2: For the initial mask of each vehicle instance, apply a confidence threshold to filter based on its category confidence, remove vehicle instance masks with confidence below the preset threshold, and retain vehicle instance masks with high confidence.

[0020] Step S2-3: Perform morphological post-processing on the retained vehicle instance mask to smooth the mask boundaries and eliminate small holes in the mask;

[0021] Step S2-4: Combine the post-processed vehicle instance masks into a contour mask for all vehicles in the image.

[0022] As a preferred embodiment, step S3 includes the following steps:

[0023] Step S3-1: Determine the effective road area: Based on the binary mask of the road area, and excluding the area covered by the vehicle contour mask, obtain the binary mask of the effective road area through logical difference operation;

[0024] Step S3-2, Random Position Selection: Within the binary mask of the effective road area, randomly select one or more pixels as the initial position points of the projectile mask;

[0025] Step S3-3: Random shape generation: Generate a polygon of random shape centered on each initial position point, wherein the number of sides, vertex coordinates and size range of the polygon are randomly determined by preset parameters;

[0026] Step S3-4, Mask Boundary Constraint: Check whether the generated random shape polygon is completely within the valid road area. If the polygon is outside the valid road area, adjust or regenerate the vertex coordinates of the polygon until the polygon is completely within the valid road area.

[0027] Step S3-5, Output of projectile mask: Convert the random shape polygon that meets the boundary constraints into a binary mask, which is used as a random shape polygon mask for placing projectiles.

[0028] As a preferred embodiment, step S4 involves cropping the corresponding image region from the original image based on the bounding box coordinates of each random-shaped polygonal mask, and then performing geometric correction and size standardization on the image region to obtain a standardized local background image. This includes the following steps:

[0029] Step S4-1, Bounding box coordinate calculation: For each random-shaped polygonal mask, calculate its minimum bounding rectangle to obtain the bounding box coordinates of the mask. The bounding box coordinates include the pixel coordinates of the top left and bottom right corners of the rectangle.

[0030] Step S4-2, Image Region Cropping: Based on the bounding box coordinates, crop the corresponding image region from the original drone aerial image to obtain the original cropped image;

[0031] Step S4-3, Geometric Correction Processing: Perform perspective transformation correction on the original cropped image to eliminate perspective distortion caused by the aerial viewpoint, so that the road area in the image presents a planar viewpoint similar to a bird's-eye view, and generate a geometrically corrected image;

[0032] Step S4-4, Size standardization processing: Scale the geometrically corrected image to a preset fixed size to unify the image resolution and obtain a standardized local background image;

[0033] Step S4-5, Local background image output: Use the standardized local background image as input for subsequent conditional image generation.

[0034] As a preferred embodiment, step S5 includes the following steps:

[0035] Step S5-1, Spatial Condition Construction: The standardized local background image and the random-shaped polygonal mask are stitched together by channel dimension to form a multi-channel input tensor containing spatial constraints.

[0036] Step S5-2, Text Prompt Encoding: Input the text prompts describing the specific type of projectile into the pre-trained text encoder, which converts the text prompts into high-dimensional text feature embedding vectors;

[0037] Step S5-3, Condition-guided generation: The multi-channel input tensor and the high-dimensional text feature embedding vector are input into the pre-trained diffusion model. During the iterative denoising process, the diffusion model uses the multi-channel input tensor as a spatial condition constraint to generate the region and the high-dimensional text feature embedding vector as a semantic condition to guide the generation of content. In the region corresponding to the random-shaped polygon mask in the local background image, the projectile image that is coordinated with the surrounding background environment is gradually generated.

[0038] Step S5-4: Output the result: After completing the iterative denoising process, output the final generated image of the projectiles.

[0039] As a preferred embodiment, step S6 includes the following steps:

[0040] Step S6-1, Preprocessing of projectile image: The projectile image is resized and inverse geometrically corrected to make its size and viewpoint consistent with the corresponding random shape polygonal mask area in the original image, so as to obtain a projectile image adapted to the original image.

[0041] Step S6-2, Image Fusion Processing: The projectile image adapted to the original image is fused with the original image at the pixel level at the position of a random-shaped polygonal mask. The image fusion algorithm is used to process the edge transition and illumination consistency between the projectile image and the original image to generate a visually consistent composite image.

[0042] Step S6-3, Annotation information generation: Based on the bounding box coordinates of the random-shaped polygon mask, extract the position information of the projectile in the synthesized image, and generate annotation information containing category labels and bounding box coordinates in combination with the projectile type;

[0043] Step S6-4, Annotation File Output: Convert the annotation information into a standard object detection annotation format and save it as an annotation file for subsequent model training.

[0044] As can be seen from the above technical solution provided by the present invention, the method for generating image data of highway debris from an aerial perspective provided by the present invention has the following beneficial effects:

[0045] Significantly improving the scene realism and adaptability of synthesized images: This invention ensures that the generated debris images fit the actual highway scene from an aerial perspective through precise control of multiple stages. First, a pre-trained semantic segmentation model is used to segment the road area of ​​the original aerial image. Through probabilistic graph binarization and morphological post-processing, a binary mask of the road area with smooth boundaries and no noise holes is generated, achieving accurate differentiation between road and non-road areas. Then, a pre-trained instance segmentation model is used to identify and exclude vehicle areas. After confidence filtering and morphological processing, the accuracy of the vehicle contour mask is ensured. Subsequently, the effective road area without vehicle occlusion is obtained through logical difference operation. Finally, when generating the debris mask within the effective road area, boundary constraints ensure that the mask is completely within the effective area. In the conditional generation stage, the diffusion model uses the color, lighting, and texture of the local background image as a reference, combined with text prompts, to generate the debris image, making the debris visually highly coordinated with the surrounding environment. This effectively avoids the debris appearing in non-road or vehicle areas, perfectly adapting to the perspective characteristics of drone aerial photography and the actual scene features of highways.

[0046] Significantly reducing labor costs and improving data generation efficiency: This invention achieves integrated automatic generation of spill image data and annotation files, completely eliminating the reliance on manual annotation. In the image generation stage, there is no need to manually draw the position of the spill or adjust its shape. In the annotation stage, step S6 directly extracts the position information based on the bounding box coordinates of the spill mask, combines it with the text prompts used to generate the spill to determine the category label, automatically integrates and forms the annotation information containing category and position, and converts it into standard annotation formats in the field of object detection such as PASCALVOC and YOLO, which can be directly adapted to mainstream model training frameworks. The entire process does not require manual participation in coordinate annotation, category division, or format conversion, which not only avoids the subjective error of manual annotation, but also significantly shortens the data generation cycle, significantly improves the generation efficiency of highway spill image data from aerial perspective, and reduces the labor cost of preparing large-scale training data.

[0047] This invention effectively enriches the diversity of training data and solves the problem of scarce real data: It can generate debris data covering various scenarios and forms, perfectly solving the pain point of scarce and monotonous real aerial photography data of debris on highways. On the one hand, by flexibly adjusting the text prompts, the diffusion model can be guided to generate different types of debris such as broken tires, scattered planks, discarded plastic baskets, and metal parts, meeting the diverse needs of common debris on highways. On the other hand, in the debris mask generation stage, the number of sides, vertex coordinates, and size range of the polygon are randomly determined by preset parameters, and the initial position is also randomly selected within the effective road area. This makes the generated debris random in terms of shape (triangles, quadrilaterals, etc.), size (in line with the actual size range of debris), and location (different lanes, different road sections). It can simulate the diverse distribution and morphological characteristics of debris in reality, providing rich and comprehensive training samples for the debris target detection model and avoiding the problem of insufficient generalization ability of the model due to monotonous training data.

[0048] Ensuring high-quality synthetic images to improve model detection accuracy: This invention employs multi-stage image optimization to ensure high visual quality in the generated synthetic images, providing support for accurate training of the subsequent target detection model. In the local image preprocessing stage, step S4 eliminates perspective distortion caused by the aerial view through perspective transformation correction, presenting the road area with a near-bird's-eye view, facilitating the diffusion model's accurate generation of debris details. In the image fusion stage, step S6 first performs inverse geometric correction on the debris image to restore the perspective effect under the aerial view, then processes the edge transition area between the debris and the original image through a pixel-level fusion algorithm to eliminate edge discontinuity, and adjusts the lighting, contrast, and hue of the debris image to match the lighting conditions of the surrounding environment, ultimately generating a synthetic image with no obvious synthetic traces and visual coherence. High-quality synthetic images can realistically reproduce the visual relationship between debris and the road environment under the aerial view, enabling the trained detection model to more accurately identify debris in actual aerial scenes, effectively improving the model's detection accuracy and reliability. Attached Figure Description

[0049] Figure 1 This is a schematic diagram illustrating the steps of a method for generating image data of debris scattered on a highway from an aerial perspective, according to the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific embodiments.

[0052] like Figure 1 As shown, this embodiment of the invention provides a method for generating image data of debris spilled on a highway from an aerial perspective, including the following steps:

[0053] Step S1, Road Region Segmentation: The acquired drone aerial images are processed using a pre-trained semantic segmentation model to generate binary masks for road regions.

[0054] Step S2, Vehicle Target Elimination: The original image is processed using a pre-trained instance segmentation model to identify and generate contour masks for all vehicles in the image.

[0055] Step S3, Spray Mask Generation: Based on the binary mask of the road area, and excluding the area covered by the vehicle contour mask, generate one or more random-shaped polygonal masks for placing the spill within the remaining valid road area.

[0056] Step S4, Local Image Preprocessing: Based on the bounding box coordinates of each random-shaped polygonal mask, the corresponding image region is cropped from the original image, and the image region is geometrically corrected and its size is standardized to obtain a standardized local background image.

[0057] Step S5, Conditional Image Generation: The standardized local background image and the randomly shaped polygon mask are used together as spatial conditions, and combined with text prompts describing specific types of projectiles, and input into the pre-trained diffusion model; The diffusion model uses the aforementioned spatial conditions as constraints and the aforementioned text prompts as semantic guidance to generate a projectile image that is in harmony with the surrounding environment in the region of the local background image corresponding to the mask.

[0058] Step S6, Image Fusion and Annotation: The generated projectile image is fused back into the corresponding random-shaped polygon mask position in the original image to form a visually consistent synthetic image; at the same time, based on the coordinate information of the random-shaped polygon mask, an annotation file for training the object detection model is automatically generated.

[0059] In this embodiment, the core function of step S1 is to use a pre-trained semantic segmentation model to accurately process the original drone aerial images, separate road areas from non-road areas, and finally generate a binary mask of the road area that can be used in subsequent processes, providing an accurate road area benchmark for excluding vehicle areas and determining the placement range of spilled materials; the specific sub-steps are as follows:

[0060] Step S1-1: Original image input and probability map output:

[0061] The raw images captured by the drone are directly input into the pre-trained semantic segmentation model. This semantic segmentation model has been trained on a large amount of aerial image data containing highway scenes and has the ability to identify the category of road pixels in the image. When the model runs, it analyzes each pixel in the raw image, determines the probability that it belongs to the road category, and outputs the corresponding probability map. The size of the probability map is the same as that of the raw image, and the value of each pixel in the map represents the probability that the pixel belongs to the road category.

[0062] Step S1-2: Binarization of the probability map and generation of the initial mask:

[0063] The probability map output in step S1-1 is binarized using a preset probability threshold. The processing rule is as follows: pixels with a probability value higher than the threshold are identified as road areas and assigned a first pixel value; pixels with a probability value lower than the threshold are identified as non-road areas and assigned a second pixel value. Through the above binarization process, an initial road area mask that can initially distinguish between road and non-road areas is generated.

[0064] Step S1-3: Initial mask morphological post-processing:

[0065] Since the initial road area mask may contain small noise holes (caused by image noise or model recognition errors), and the boundary contour of the road area may not be smooth enough, morphological post-processing is required for the initial road area mask. The specific operations include two types of morphological operations: first, filling the small noise holes in the mask through morphological closing operations to ensure the integrity of the road area; second, smoothing the boundary contour of the road area through morphological opening operations to reduce irregular protrusions or depressions at the boundary, making the mask more closely match the shape of the actual road.

[0066] Steps S1-4: Final mask output:

[0067] After completing the morphological post-processing, the processed mask is checked for validity to confirm that there are no obvious noise holes in the mask, the road area boundaries are smooth and consistent with the actual road range in the original image; after passing the check, the mask is output as the final binary mask for the road area, which will be directly used to determine the valid road area in step S3.

[0068] In this embodiment, the core function of step S2 is to accurately identify all vehicle targets in the original drone aerial image using a pre-trained instance segmentation model, generating a complete and accurate vehicle contour mask, which provides a basis for subsequent exclusion of vehicle areas and determination of effective road areas where the scattering material can be placed; the specific sub-steps are as follows:

[0069] Step S2-1: Input of original image and output of vehicle instance information:

[0070] The raw images acquired by drone aerial photography are input into a pre-trained instance segmentation model. This instance segmentation model has been trained on a large amount of aerial image data containing highway vehicle scenes and has the ability to distinguish different individual vehicle instances, rather than just identifying vehicle categories. When the model runs, it performs instance-by-instance detection on vehicle targets in the raw images. For each detected vehicle instance, it outputs two key pieces of information: first, the initial mask of the vehicle instance, which outlines the approximate contour of the vehicle and whose size matches the actual size of the corresponding vehicle in the raw image; and second, the category confidence of the vehicle instance, which characterizes the model's confidence in the current detection result being of the "vehicle" category.

[0071] Step S2-2: Vehicle instance mask confidence filtering:

[0072] A preset confidence threshold is set, which is calibrated through multiple experiments to avoid missing real vehicles due to an excessively high threshold, and to prevent retaining misidentified non-vehicle targets due to an excessively low threshold. For each vehicle instance's initial mask output in step S2-1, it is filtered based on its corresponding category confidence: if the category confidence of a vehicle instance is lower than the preset threshold, the mask is determined to be a suspected misidentification result and is removed; if the category confidence is higher than or equal to the preset threshold, the mask is determined to be a high-confidence valid result and is retained. Through this filtering step, it is initially ensured that the retained vehicle masks are all real vehicle regions that the model is highly confident in.

[0073] Step S2-3: Preserve the morphological post-processing of the mask:

[0074] Since the high-confidence vehicle instance mask retained in step S2-2 may have issues such as uneven boundaries (e.g., jagged contours due to image resolution or model recognition accuracy) or small holes inside the mask (e.g., gaps caused by vehicle window areas or image noise), these issues can affect the accuracy of subsequent vehicle region exclusion. Therefore, morphological post-processing is required on the retained vehicle instance mask: by combining morphological dilation and erosion operations, the small holes inside the mask are first filled, and then the edge contour of the mask is smoothed, so that the processed mask can more accurately fit the actual shape of the vehicle and reduce contour errors.

[0075] Step S2-4: Output of vehicle contour mask combination:

[0076] The masks of all individual vehicle instances after morphological post-processing are integrated; the mask of each vehicle instance is superimposed on the same image coordinate system according to its actual position in the original image to form an overall vehicle contour mask covering all valid vehicle targets in the original image; in this mask, the boundaries between vehicle areas and non-vehicle areas are clear, and it can be directly used to perform calculations with the binary mask of the road area to exclude the areas covered by vehicles, thereby determining the effective road range where the spilled material can be placed.

[0077] In this embodiment, the core function of step S3 is to eliminate invalid areas covered by vehicle contour masks based on the acquired road area binary mask, determine the effective road range where the spilled material can be placed, and generate a random-shaped polygonal mask that conforms to spatial constraints, providing accurate spatial positioning basis for the subsequent generation of spilled material images; the specific sub-steps are as follows:

[0078] Step S3-1: Determining the effective road area:

[0079] The binary mask of the road area output in step S1 and the vehicle contour mask output in step S2 are called and processed by logical difference operation. The specific process of logical difference operation is as follows: retain the pixel areas in the binary mask of the road area that are not covered by the vehicle contour mask, and remove the pixel areas covered by the vehicle contour mask. Through this operation, a valid binary mask of the road area that only contains the road part without vehicle obstruction is finally obtained. This mask clarifies the legal space range in which the spilled material can be placed, ensuring that the subsequent spilled material will not be generated in vehicle or non-road areas.

[0080] Step S3-2: Random location selection:

[0081] Within the pixel space corresponding to the effective road area binary mask generated in step S3-1, a random position selection algorithm is started. This algorithm will traverse all pixels within the effective road area and randomly select one or more pixels as the initial position point of the projectile mask. During the selection process, the pixel value will be judged (there is a difference between the pixel value of the effective road area and the pixel value of the non-effective area) to ensure that the selected initial position point is completely within the effective road area, so as to avoid the initial position falling in the invalid area and causing subsequent mask generation to be abnormal.

[0082] Step S3-3: Random shape generation:

[0083] Using each initial position point selected in step S3-2 as the geometric center, the random polygon generation module is activated. This module randomly determines the key attributes of the polygon based on a preset parameter range, including the number of sides, vertex coordinates, and size range. The number of sides is usually randomly selected within the range of 3 to 8 sides (covering common polygon shapes such as triangles and quadrilaterals to simulate the diverse shapes of actual spilled materials). The vertex coordinates are randomly generated with the initial position point as the center, within a preset range of horizontal and vertical offsets. The size range is controlled by preset maximum and minimum side length parameters to ensure that the size of the generated polygon conforms to the actual size characteristics of highway spilled materials (e.g., avoiding excessively large or small sizes that exceed a reasonable range).

[0084] Step S3-4: Mask boundary constraints:

[0085] For each random polygon generated in step S3-3, a boundary detection algorithm is initiated. This algorithm checks whether the coordinates of each vertex of the polygon are completely within the binary mask of the effective road area generated in step S3-1. If it is detected that some vertices or edges of the polygon exceed the effective road area, a coordinate adjustment mechanism is initiated: based on the boundary coordinates of the effective road area, the coordinates of the vertices exceeding the boundary are shrunk or offset to correct the polygon, so that the polygon is refitted to the effective area. If the adjustment still fails to meet the requirement of being completely within the effective area, a regeneration process is triggered, returning to step S3-3, and a polygon that meets the boundary requirements is regenerated with the original initial position point or a new nearby random position point as the center, until the generated polygon is completely within the effective road area.

[0086] Step S3-5: Output of the projectile mask:

[0087] For the random shape polygon detected by boundary constraints in steps S3-4, a binary mask conversion program is started. This program marks the pixel area covered by the polygon as the target area (assigning a specific pixel value to distinguish it from the non-target area) and marks the area outside the polygon as the background area (assigning another pixel value), thereby converting the polygon into a binary mask consistent with the original image coordinate system. This mask is the random shape polygon mask used for subsequent placement of the projectile, and is directly output to step S4 as the basis for local image cropping.

[0088] In this embodiment, the core function of step S4 is to extract the corresponding local region from the original drone aerial image for each projectile mask generated in step S3, eliminate aerial view distortion through geometric correction, and perform size standardization processing to provide a standardized local background image with uniform specifications and adapted viewpoint for the subsequent generation of projectile images by the diffusion model in step S5; the specific sub-steps are as follows:

[0089] Step S4-1: Calculate the bounding box coordinates:

[0090] For each random-shaped polygon mask output in step S3, a minimum bounding rectangle calculation program is started. This program traverses all vertex coordinates of the mask to determine the minimum rectangle that can completely enclose the polygon; that is, the minimum bounding rectangle. By calculating the pixel coordinates of the top-left vertex and the bottom-right vertex of the rectangle, the bounding box coordinates corresponding to the projectile mask are obtained. These coordinates accurately locate the spatial range of the mask in the original image, providing a clear positional basis for subsequent image cropping.

[0091] Step S4-2: Image region cropping:

[0092] Based on the bounding box coordinates obtained in step S4-1, the image cropping module is called to process the original drone aerial image. The cropping module will take the upper left corner pixel coordinates of the bounding box as the starting point and the lower right corner pixel coordinates of the bounding box as the ending point, and extract the image area within this rectangular range from the original image to generate the original cropped image. The original cropped image completely contains the background area corresponding to the projectile mask, and only retains the local image information related to the generation of the projectile, reducing the amount of data for subsequent processing.

[0093] Step S4-3: Geometric correction processing:

[0094] Because drone aerial images suffer from perspective distortion—that is, road areas appear larger when closer and smaller when farther away—this affects the harmony between the generated debris and the background environment. Therefore, geometric correction processing is required for the original cropped image. A perspective transformation correction algorithm is activated, which constructs a perspective transformation matrix based on the aerial image's viewpoint parameters (such as drone shooting altitude, tilt angle, and other preset reference values) to remap the pixel positions in the original cropped image. This transformation eliminates the visual deviation caused by perspective distortion, giving the road area in the cropped image a near-bird's-eye view, ultimately generating a geometrically corrected image.

[0095] Step S4-4: Size standardization process:

[0096] The image scaling procedure is initiated to scale the geometrically corrected image generated in step S4-3 to a preset fixed size. This fixed size is determined through previous experiments and needs to be adapted to the input specifications of the pre-trained diffusion model in the subsequent step S5. During the scaling process, an interpolation algorithm (such as bilinear interpolation) is used to maintain the clarity of image details and avoid pixel blurring or distortion after scaling. Through this process, all geometrically corrected images are unified to the same resolution, eliminating the influence of differences in the original size of different local areas on the generation effect of the diffusion model, and finally obtaining a standardized local background image.

[0097] Step S4-5: Output local background image:

[0098] The standardized local background image obtained in step S4-4 is subjected to quality verification to confirm that the image has no obvious distortion, the resolution meets the preset requirements, and the road area view is a planar view. After the verification is passed, the standardized local background image is used as the input data output for the subsequent step S5 to ensure that the diffusion model can process the image of the spilled material based on a background of uniform specifications.

[0099] In this embodiment, the core function of step S5 is to construct generation conditions that combine spatial constraints and semantic guidance. A standardized local background image of projectiles and textual prompts indicating projectile types are input into a pre-trained diffusion model to generate projectile images that are visually coordinated with the surrounding environment within a specified area, providing high-quality projectile material for subsequent image fusion. The specific sub-steps are as follows:

[0100] Step S5-1: Spatial Condition Construction:

[0101] The channel-dimensional stitching process is initiated to process the standardized local background image output in step S4 and the corresponding random-shape polygon mask output in step S3. The standardized local background image is usually a three-channel RGB image, and the random-shape polygon mask is a single-channel binary image. The two are superimposed and integrated in the channel dimension so that each pixel position simultaneously contains the background color information and the constraint information of whether it is a region for generating projectiles. Finally, a multi-channel input tensor containing spatial constraints is formed. This tensor directly defines the spatial range of projectiles generated by the diffusion model, ensuring that projectiles are only generated within the preset mask area.

[0102] Step S5-2: Text prompt encoding:

[0103] The process involves identifying text prompts that describe specific types of spilled material. These prompts must accurately represent key features such as the type, shape, and material of the spilled material, for example, broken tires, scattered wooden planks, or discarded plastic crates on a highway. The text prompts are then input into a pre-trained text encoder. This encoder, through semantic parsing and mapping, converts the natural language text prompts into high-dimensional text feature embedding vectors. These vectors transform the semantic information of the spilled material into a numerical form recognizable by the diffusion model, providing a semantic basis for subsequent generation of specific types of spilled material.

[0104] Step S5-3: Conditional Guided Generation:

[0105] The multi-channel input tensor constructed in step S5-1 and the high-dimensional text feature embedding vector generated in step S5-2 are jointly input into the pre-trained diffusion model to initiate the iterative denoising process. During the iteration, the diffusion model uses the multi-channel input tensor as a spatial condition to strictly constrain the generation area of ​​the projectiles, and performs image generation operations only within the pixel area of ​​the mask corresponding to the tensor. At the same time, the high-dimensional text feature embedding vector is used as a semantic condition to guide the type of generated content, ensuring that the generated image conforms to the projectile features described by the text prompt words. In addition, the model will refer to environmental information such as color, lighting, and texture of the standardized local background image in real time, and adjust the parameters to make the generated projectiles visually consistent with the surrounding background, gradually optimizing the image details, and finally forming a preliminary projectile image within the mask area.

[0106] Step S5-4: Generate the output result:

[0107] After the diffusion model completes the preset number of iterations of denoising, the generation result verification program is started to check whether the generated projectile image meets the following conditions: First, the projectile is completely located within the mask area, with no pixels exceeding the boundary; second, there is no obvious discontinuity between the projectile and the background in terms of color, lighting, and texture, and the visual coordination meets the standard; third, the shape details of the projectile match the characteristics described by the text prompt words. After the verification is passed, the projectile image is output as the final generation result for use in the subsequent image fusion processing step S6.

[0108] In this embodiment, the core function of step S6 is to visually coordinate and fuse the projectile image generated in step S5 with the original drone aerial image, and simultaneously automatically generate a standard-compliant annotation file based on the projectile mask coordinates, providing complete synthetic image data and annotation information for subsequent target detection model training; the specific sub-steps are as follows:

[0109] Step S6-1: Image preprocessing of spilled material:

[0110] The projectile image adaptation processing procedure is initiated, and two key processes are performed on the projectile image output in step S5. The first is size adjustment. Referring to the actual size of the corresponding random-shaped polygonal mask in the original image in step S3, the projectile image is scaled to perfectly match the size of the mask area, ensuring that the image size is adapted to the target position in the original image. The second is inverse geometric correction. Since perspective transformation correction was performed on the local image in step S4 to eliminate aerial distortion, the perspective transformation matrix opposite to that in step S4-3 is used here to perform an inverse transformation on the projectile image, restoring the perspective distortion effect under the aerial viewpoint, so that the viewpoint of the projectile image is consistent with the viewpoint of the corresponding mask area in the original image. After these two processes, a projectile image that is perfectly adapted to the original image is obtained.

[0111] Step S6-2: Image fusion processing:

[0112] The pixel-level image fusion module is invoked to fuse the adapted projectile image obtained in step S6-1 with the original drone aerial image. The fusion process uses the randomly shaped polygonal mask generated in step S3 as the positioning basis, and only merges the pixels of the projectile image with the pixels of the original image within the pixel area covered by the mask. To ensure visual consistency, the fusion algorithm focuses on processing the edge transition area between the projectile image and the original image, and eliminates the edge discontinuity through gradient smoothing. At the same time, the illumination intensity and color distribution characteristics around the mask area in the original image are analyzed, and the brightness, contrast and hue of the projectile image are adjusted to make it consistent with the lighting conditions of the surrounding environment. After the fusion is completed, a composite image with overall visual coherence and no obvious compositing traces is generated.

[0113] Step S6-3: Generate annotation information:

[0114] The annotation information extraction program is started. First, the bounding box coordinates of the random-shaped polygonal mask in step S3 are read. These coordinates correspond to the actual position of the spilled object in the composite image and are directly used as the position information of the spilled object. Then, the text prompts used to generate the spilled object in step S5 are associated with the text prompts to extract the category label of the spilled object, such as broken tires, wooden boards, plastic buckets, etc. The category label and the bounding box coordinates are associated and integrated to form complete annotation data for each annotation information, which includes the category label of the spilled object and the corresponding bounding box coordinates, to ensure that the annotation information can accurately correspond to the spilled object target in the composite image.

[0115] Step S6-4: Output the annotation file:

[0116] Start the annotation format conversion program to convert the annotation information generated in step S6-3 into a standard annotation format commonly used in the field of object detection, such as PASCALVOC or YOLO, to ensure that the annotation file can be recognized by mainstream object detection model training frameworks. After the format conversion is completed, associate the annotation file with the corresponding synthetic image filename to ensure that the two correspond one-to-one, and then save the annotation file to the specified storage path. The annotation file and the synthetic image together constitute complete training data, which can be directly used in the subsequent training process of the projectile object detection model.

[0117] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating image data of a highway litter under an aerial view, the method comprising: Includes the following steps: ​ Step S1, Road Region Segmentation: The acquired drone aerial images are processed using a pre-trained semantic segmentation model to generate binary masks for road regions. Step S2, Vehicle Target Elimination: The original image is processed using a pre-trained instance segmentation model to identify and generate contour masks for all vehicles in the image. Step S3, Spray Mask Generation: Based on the binary mask of the road area, and excluding the area covered by the vehicle contour mask, generate one or more random-shaped polygonal masks for placing the spill within the remaining valid road area. Step S4, Local Image Preprocessing: Based on the bounding box coordinates of each random-shaped polygonal mask, the corresponding image region is cropped from the original image, and the image region is geometrically corrected and its size is standardized to obtain a standardized local background image. Step S5, Conditional Image Generation: The standardized local background image and the randomly shaped polygon mask are used together as spatial conditions, and combined with text prompts describing specific types of projectiles, and input into the pre-trained diffusion model; The diffusion model uses the aforementioned spatial conditions as constraints and the aforementioned text prompts as semantic guidance to generate a projectile image that is in harmony with the surrounding environment in the region of the local background image corresponding to the mask. Step S6, Image Fusion and Annotation: The generated projectile image is fused back into the corresponding random-shaped polygon mask position in the original image to form a visually consistent synthetic image; at the same time, based on the coordinate information of the random-shaped polygon mask, an annotation file for training the object detection model is automatically generated.

2. The method of claim 1, wherein the method further comprises: determining a height of the aerial vehicle; and determining a height of the object; and wherein the determining the aerial vehicle's position and orientation is based on the height of the aerial vehicle and the height of the object. The specific steps for generating the binary mask for the road area in step S1 include: S1-1. Input the original aerial image captured by the drone into the pre-trained semantic segmentation model. The semantic segmentation model outputs a probability map of each pixel belonging to the road category. S1-2. Apply a probability threshold to the probability map for binarization. Pixels with probability values ​​higher than the threshold are identified as road areas and assigned a first pixel value. Pixels with probability values ​​lower than the threshold are identified as non-road areas and assigned a second pixel value, thereby generating an initial road area mask. S1-3. Perform morphological post-processing on the initial road area mask to eliminate small noise holes and smooth the boundary contour of the road area. S1-4. Output the post-processed mask as the final binary mask for the road area. 3.The aerial-view high-way-spraying-image-data generating method according to claim 1, wherein: The original image is processed using a pre-trained instance segmentation model to identify and generate contour masks for all vehicles in the image, including the following steps: Step S2-1: Input the original aerial image captured by the drone into the pre-trained instance segmentation model. The instance segmentation model outputs the initial mask and the corresponding class confidence of each detected vehicle instance in the image. Step S2-2: For the initial mask of each vehicle instance, apply a confidence threshold to filter based on its category confidence, remove vehicle instance masks with confidence below the preset threshold, and retain vehicle instance masks with high confidence. Step S2-3: Perform morphological post-processing on the retained vehicle instance mask to smooth the mask boundaries and eliminate small holes in the mask; Step S2-4: Combine the post-processed vehicle instance masks into a contour mask for all vehicles in the image.

4. The method for generating highway spill image data from an aerial perspective according to claim 1, characterized in that: Step S3 includes the following steps: Step S3-1: Determine the effective road area: Based on the binary mask of the road area, and excluding the area covered by the vehicle contour mask, obtain the binary mask of the effective road area through logical difference operation; Step S3-2, Random Position Selection: Within the binary mask of the effective road area, randomly select one or more pixels as the initial position points of the projectile mask; Step S3-3: Random shape generation: Generate a polygon of random shape centered on each initial position point, wherein the number of sides, vertex coordinates and size range of the polygon are randomly determined by preset parameters; Step S3-4, Mask Boundary Constraint: Check whether the generated random shape polygon is completely within the valid road area. If the polygon is outside the valid road area, adjust or regenerate the vertex coordinates of the polygon until the polygon is completely within the valid road area. Step S3-5, Output of projectile mask: Convert the random shape polygon that meets the boundary constraints into a binary mask, which is used as a random shape polygon mask for placing projectiles.

5. The method for generating highway spill image data from an aerial perspective according to claim 1, characterized in that: In step S4, based on the bounding box coordinates of each random-shaped polygonal mask, the corresponding image region is cropped from the original image, and geometric correction and size standardization are performed on the image region to obtain a standardized local background image, including the following steps: Step S4-1: Calculate bounding box coordinates: For each random-shaped polygonal mask, calculate its minimum bounding rectangle to obtain the bounding box coordinates of the mask. The bounding box coordinates include the pixel coordinates of the upper left and lower right corners of the rectangle. Step S4-2, Image Region Cropping: Based on the bounding box coordinates, crop the corresponding image region from the original drone aerial image to obtain the original cropped image; Step S4-3, Geometric Correction Processing: Perform perspective transformation correction on the original cropped image to eliminate perspective distortion caused by the aerial viewpoint, so that the road area in the image presents a planar viewpoint similar to a bird's-eye view, and generate a geometrically corrected image; Step S4-4, Size standardization processing: Scale the geometrically corrected image to a preset fixed size to unify the image resolution and obtain a standardized local background image; Step S4-5, Local background image output: Use the standardized local background image as input for subsequent conditional image generation.

6. The method for generating highway spill image data from an aerial perspective according to claim 1, characterized in that: Step S5 includes the following steps: Step S5-1, Spatial Condition Construction: The standardized local background image and the random-shaped polygonal mask are stitched together by channel dimension to form a multi-channel input tensor containing spatial constraints. Step S5-2, Text Prompt Encoding: Input the text prompts describing the specific type of projectile into the pre-trained text encoder, which converts the text prompts into high-dimensional text feature embedding vectors; Step S5-3, Condition-guided generation: The multi-channel input tensor and the high-dimensional text feature embedding vector are input into the pre-trained diffusion model. During the iterative denoising process, the diffusion model uses the multi-channel input tensor as a spatial condition constraint to generate the region and the high-dimensional text feature embedding vector as a semantic condition to guide the generation of content. In the region corresponding to the random-shaped polygon mask in the local background image, the projectile image that is coordinated with the surrounding background environment is gradually generated. Step S5-4: Output the result: After completing the iterative denoising process, output the final generated image of the projectiles.

7. The method for generating highway spill image data from an aerial perspective according to claim 1, characterized in that: Step S6 includes the following steps: Step S6-1, Preprocessing of projectile image: The projectile image is resized and inverse geometrically corrected to make its size and viewpoint consistent with the corresponding random shape polygonal mask area in the original image, so as to obtain a projectile image adapted to the original image. Step S6-2, Image Fusion Processing: The projectile image adapted to the original image is fused with the original image at the pixel level at the position of a random-shaped polygonal mask. The image fusion algorithm is used to process the edge transition and illumination consistency between the projectile image and the original image to generate a visually consistent composite image. Step S6-3, Annotation information generation: Based on the bounding box coordinates of the random-shaped polygon mask, extract the position information of the projectile in the synthesized image, and generate annotation information containing category labels and bounding box coordinates in combination with the projectile type; Step S6-4, Annotation File Output: Convert the annotation information into a standard object detection annotation format and save it as an annotation file for subsequent model training.

Images