Method, apparatus, and computer program for augmenting soiled data of autonomous driving camera sensors

Abstract

The present invention is characterized in that a method for augmenting foreign matter data from an autonomous driving camera sensor comprises: receiving a clean image and a foreign matter area designation image captured by an autonomous driving camera sensor; generating a conditioning image by performing a synthesis operation on the clean image and the foreign matter area designation image; and generating a first sensor contamination image by inputting a text prompt specifying the foreign matter and the conditioning image into a diffusion model.

Description

Technology Field

[0001] The present invention relates to a method, apparatus, and computer program for augmenting foreign matter data of an autonomous driving camera sensor, and more particularly to a method, apparatus, and computer program for augmenting foreign matter data of an autonomous driving camera sensor for training an artificial intelligence model for determining the contamination level of an autonomous driving camera sensor. Background Technology

[0002] Autonomous driving systems must overcome various challenging tasks to enter the stage of full automation (SAE Level 5). Among these, ensuring the reliability of sensors that directly interact with the external environment is considered a particularly critical challenge. Wide-angle camera sensors used in autonomous driving systems are installed on the front, sides, and rear of the vehicle and play a key role in recognizing the surrounding environment; however, because these sensors are exposed to the external environment, they face the problem of being easily contaminated by various foreign substances such as rain, snow, dust, and mud.

[0003] Contaminated camera sensors can be a serious obstacle to the normal operation of autonomous driving systems. If a sensor is contaminated with foreign matter, its ability to perceive the surrounding environment deteriorates, potentially leading to a failure to properly detect potential hazardous situations, which increases the likelihood of accidents. Therefore, to enter the stage of full automation, technology capable of accurately detecting, removing, or correcting camera sensor contamination is essential.

[0004] While numerous studies exist on detecting contaminants in camera sensors, most struggle to improve performance due to a lack of data. Very little publicly available camera image data regarding foreign matter contamination in autonomous driving environments exists, and even the available datasets often lack sufficient environmental diversity (weather, time of day, road conditions, etc.). Furthermore, collecting pixel-level ground truth labels for tasks such as foreign matter segmentation is a cumbersome and time-consuming process.

[0005] In particular, classifying contamination at the pixel level is difficult and requires significant human resources due to unclear boundaries between contamination categories and ambiguous definitions of contamination classes based on the data. Consistent labeling is challenging because foreign objects exhibit varying transparency and irregular patterns, often making it difficult to distinguish them from the background depending on the extent of adhesion. These difficulties in data collection lead to a shortage of training data, posing a major obstacle to optimizing the performance of autonomous driving systems.

[0006] Accordingly, the use of generative models that generate various realistic images to solve the problem of data scarcity is gaining attention. With the recent advancement of deep learning-based generative models, it has become possible to synthesize high-quality images, and through this, data augmentation techniques are being effectively utilized in various computer vision tasks.

[0007] In particular, diffusion models have the advantage of generating high-quality images through a stepwise noise addition and removal process, as well as enabling fine-grained control over specific regions. Unlike conventional Generative Adversarial Networks (GANs), diffusion models are attracting attention in the field of image generation due to their ability to perform stable training and generate diverse outputs. Conditional diffusion models, such as ControlNet, are particularly suitable for generating data that reflects various foreign object patterns, transparency, and environmental characteristics required in autonomous driving environments, as they can generate images based on user-specified conditions (e.g., areas where foreign objects will be buried).

[0008] Therefore, there is a need for a method, device, and computer program for augmenting foreign matter data from autonomous driving camera sensors that can efficiently generate foreign matter contamination data required for autonomous driving systems using a diffusion model. Prior art literature

[0009] U.S. Patent Publication No. 2024-0211797 The problem to be solved

[0010] The present invention aims to provide a method, apparatus, and computer program for augmenting autonomous driving camera sensor foreign object data that effectively augments camera sensor foreign object data having various patterns and characteristics in an actual driving environment. means of solving the problem

[0011] To achieve the above objective, the present invention is characterized in that, in a method for augmenting foreign object data from an autonomous driving camera sensor performed by a computing device, the method comprises: receiving a clean image and a foreign object area designation image captured by an autonomous driving camera sensor; generating a conditioning image by performing a composite operation on the clean image and the foreign object area designation image; and generating a first sensor contamination image by inputting a text prompt specifying the foreign object and the conditioning image into a diffusion model.

[0012] Preferably, the input receiving step may receive a foreign object region designation image generated by a Variational Auto Encoder (VAE).

[0013] Preferably, the step of generating the first sensor contamination image may use a ControlNet structure as the diffusion model.

[0014] Preferably, the method may further include the step of generating a second sensor contamination image by applying transparency to the foreign matter area of ​​the first sensor contamination image.

[0015] Preferably, the step of generating the second sensor contamination image may include the step of separating the foreign matter generating area image from the first sensor contamination image; and the step of separating the foreign matter generating area image into a foreign matter area image and a clean area image.

[0016] Preferably, the step of generating the second sensor contamination image may further include the step of classifying the foreign matter area image into one or more class areas according to the brightness value of the foreign matter.

[0017] Preferably, the step of generating the second sensor contamination image may further include the step of performing an alpha blending operation by applying different transparency to each classified class area, and synthesizing the second sensor contamination image by performing an inpainting operation with the result of the alpha blending operation and the clean area image.

[0018] Preferably, the step of classifying into one or more classes comprises a brightness value threshold The step of classifying the foreign matter area image into n class areas based on the above and synthesizing the second sensor contamination image can perform an alpha blending operation by applying the following mathematical formula 1 to each classified class area.

[0019] [Mathematical Formula 1]

[0020]

[0021] (Here, is the second sensor contamination image, I is the clean image, is the image of the foreign object generation area, r(·,·) is a class area classification function based on brightness values, is the brightness value threshold, represents the alpha value of the i-th class)

[0022] In addition, the present invention is further characterized in that the device for augmenting foreign matter data from an autonomous driving camera sensor comprises: a processor including one or more cores; and a memory; wherein the processor receives a clean image and a foreign matter area designation image captured by an autonomous driving camera sensor, performs a composite operation on the clean image and the foreign matter area designation image to generate a conditioning image, inputs a text prompt specifying the foreign matter and the conditioning image into a diffusion model to generate a first sensor contamination image, and applies transparency to the foreign matter area of ​​the first sensor contamination image to generate a second sensor contamination image.

[0023] In addition, the present invention is further characterized by a computer program that includes instructions stored in a computer-readable storage medium to cause a computer to perform the following operations, wherein the operations include: receiving a clean image captured by an autonomous driving camera sensor and a foreign object region designation image generated by a variation autoencoder; generating a conditioning image by performing a composite operation on the clean image and the foreign object region designation image; generating a first sensor contamination image by inputting a text prompt specifying foreign objects and the conditioning image into a diffusion model; and generating a second sensor contamination image by applying transparency to the foreign object region of the first sensor contamination image. Effects of the invention

[0024] The present invention has the advantage of enhancing the safety and reliability of an autonomous driving system by improving the accuracy, precision, and mIoU performance of a camera sensor foreign object detection model through the combination of a diffusion model and a mutation autoencoder and the application of an alpha blending technique that considers the transparency characteristics of foreign objects, thereby effectively augmenting high-quality foreign object data in various autonomous driving environments and camera views. Brief explanation of the drawing

[0025] Figure 1 shows a flowchart of a method for augmenting foreign object data from an autonomous driving camera sensor according to an embodiment of the present invention. FIG. 2 shows a flowchart of a method for augmenting foreign object data from an autonomous driving camera sensor according to an embodiment of the present invention. FIG. 3 shows a flowchart of the step of generating a second sensor contamination image according to an embodiment of the present invention. Figure 4 shows a post-processing method for generating a second sensor contamination image according to an embodiment of the present invention. FIG. 5 shows a configuration diagram of a device for augmenting foreign object data from an autonomous driving camera sensor according to an embodiment of the present invention. FIG. 6 shows a schematic diagram of a computing environment according to an embodiment of the present invention. Specific details for implementing the invention

[0026] The present invention will be described in detail below with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by exemplary embodiments. Identical reference numerals in each drawing indicate components that perform substantially the same function.

[0027] The purpose and effects of the present invention may be naturally understood or become clearer through the following description, and the purpose and effects of the present invention are not limited solely to the description below. Furthermore, in describing the present invention, if it is determined that a detailed description of known technology related to the present invention may unnecessarily obscure the essence of the present invention, such detailed description will be omitted.

[0028] The terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the description of the invention, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0029] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this invention.

[0031] In interpreting the components, they are interpreted to include a margin of error even without a separate explicit indication. In the case of descriptions regarding temporal relationships, for example, where the temporal sequence is described using 'after,' 'following,' 'next,' 'before,' etc., cases that are not continuous are included unless 'immediately' or 'directly' is used.

[0032] Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.

[0033] FIG. 1 shows a flowchart of a method for augmenting foreign object data of an autonomous driving camera sensor according to an embodiment of the present invention. Referring to FIG. 1, the method for augmenting foreign object data of an autonomous driving camera sensor may include a step of receiving input (S100), a step of generating a conditioning image (S300), a step of generating a first sensor contamination image (S500), and a step of generating a second sensor contamination image (S700).

[0034] FIG. 2 shows a flowchart of a method for augmenting foreign object data of an autonomous driving camera sensor according to an embodiment of the present invention. Referring to FIG. 2, the method for augmenting foreign object data of an autonomous driving camera sensor receives a foreign object region designation image and a clean image generated by a variational autoencoder (VAE) as inputs, generates a conditioning image through an inpainting operation, and generates a first sensor contamination image by inputting the generated conditioning image and a text prompt such as "mud" into a diffusion model. Subsequently, the method for augmenting foreign object data of an autonomous driving camera sensor can generate a final second sensor contamination image by separating the foreign object generation region from the first sensor contamination image and applying alpha blending that reflects the transparency characteristics of the foreign object through class classification based on brightness values. Each step is described in detail below.

[0035] The method of augmenting foreign object data from autonomous driving camera sensors can effectively generate high-quality data with realistic foreign object patterns and transparency from various camera views (front, left / right mirrors, rear, etc.), thereby significantly improving the performance of camera sensor foreign object detection models in real-world autonomous driving environments. In particular, this method can greatly enhance the safety and reliability of autonomous driving systems by improving key metrics such as accuracy, precision, and mIoU.

[0036] The input receiving step (S100) can receive a clean image and a foreign object area designation image captured by the autonomous driving camera sensor.

[0037] The input receiving step (S100) may receive a clean image free of foreign matter captured from various camera sensors (front camera, side mirror camera, rear camera, etc.) mounted on an autonomous vehicle. The clean image may include various lighting conditions, weather conditions, road environments, etc. acquired in an autonomous driving environment, and may have various resolutions ranging from high-resolution images to low-resolution images. Additionally, the foreign matter area designation image is an image that designates the location where foreign matter is to be buried in a specific area of ​​the clean image, and may include information indicating the distribution area, pattern, shape, etc. of the foreign matter. The foreign matter area designation image may be provided in various formats, such as a binary mask, a grayscale map, or a form including multi-channel information, and may perform the role of specifying the area to which the foreign matter is applied during the subsequent augmentation process.

[0038] The input receiving step (S100) can receive an image designating a foreign object region generated by a variational autoencoder (VAE).

[0039] The input receiving step (S100) may utilize a Variation Autoencoder (VAE) to enhance the diversity and realism of the foreign object region designation images. The Variation Autoencoder is a model trained using only ground truth label data for actual foreign object region designations, enabling it to effectively learn the distribution of foreign object patterns. During the training phase, the Variation Autoencoder can learn the core characteristics of foreign object patterns by converting the ground truth label images into a latent space through an encoder and then restoring them to a form similar to the original through a decoder. During the inference phase, the Variation Autoencoder can generate various foreign object patterns by randomly sampling a latent vector z from a standard normal distribution N(0,1) and using only the trained decoder. This method can generate new foreign object patterns that were not present in the training data, thereby significantly expanding data diversity.

[0040] The foreign object region designation images generated by the variation autoencoder may have the following characteristics: 1) they can be generated from vectors randomly sampled from a standard normal distribution to represent various foreign object patterns; 2) they can mimic the characteristics of actual foreign objects, such as irregular boundaries, various densities, and gradients; and 3) they can reflect pattern characteristics according to the type of foreign object (mud, water droplets, dust, etc.). The foreign object region designation images generated by the variation autoencoder can be generated in various forms, ranging from simple binary masks to images containing complex patterns and gradients, which enables more realistic representation of foreign objects during the subsequent augmentation process.

[0041] The foreign object region designation images generated by the variation autoencoder go beyond simply designating foreign object regions; they can be utilized as conditioning inputs for the diffusion model to induce the generation of realistic foreign object textures that match the corresponding patterns. In this way, the combination of the variation autoencoder and the diffusion model enables the generation of infinitely diverse foreign object patterns with only a small amount of label data, which can effectively resolve the problem of data scarcity required for training foreign object detection models in autonomous driving sensors.

[0042] The step of generating a conditioning image (S300) can generate a conditioning image by performing a synthesis operation on a clean image and a foreign matter area designation image. The step of generating a conditioning image (S300) can generate a conditioning image for controlling a diffusion model by combining the input clean image and the foreign matter area designation image through a special synthesis operation.

[0043] In the present disclosure, a model structure is implemented to generate a first sensor contamination image in which foreign matter is attached to a specific area from a clean image by utilizing a diffusion model trained on a large dataset. Among the diffusion models, the ControlNet model basically uses an approach that receives a conditioning image, such as a depth map or a Canny edge image, along with text and modifies the original image. However, considering the limited characteristics of the foreign matter data, it is practically difficult to obtain a pair of clean images and sensor contamination images taken in the same environment. Therefore, the step of generating a conditioning image (S300) may utilize a foreign matter area designation image containing information about a designated foreign matter area for generating the conditioning image.

[0044] The step of generating a conditioning image (S300) may utilize an inpainting operation as a composite operation. The step of generating a conditioning image (S300) can generate a conditioning image that clearly distinguishes between foreign matter areas and non-foreign matter areas through the inpainting operation.

[0045] The step of generating a first sensor contamination image (S500) can generate the first sensor contamination image by inputting a text prompt specifying foreign matter and the conditioning image into a diffusion model. When generating the first sensor contamination image, which is an image with foreign matter attached, the step of generating the first sensor contamination image (S500) can generate the foreign matter by limiting a specific area without affecting all pixels.

[0046] The step of generating the first sensor contamination image (S500) can generate an image of foreign matter adhering to a specific area by inputting the conditioning image and a text prompt specifying foreign matter together into a diffusion model. As the step of generating the first sensor contamination image (S500) is a core process of conditional image generation, a multi-conditional generation method combining a text prompt and visual conditioning can be used.

[0047] Text prompts can specify the types and characteristics of various foreign substances, such as "mud," "water droplets," "dust," and "snow," serving as a semantic guide for the diffusion model to determine which type of foreign substance to generate. The text prompts are converted into embedding vectors via a text encoder and can influence the diffusion model's process of generating the first sensor contamination image.

[0048] In the step (S500) of generating the first sensor contamination image, the conditioning image can specify the spatial location and distribution pattern to which the foreign substance is applied. The conditioning image is converted into a feature embedding through an encoder, which can act as a spatial control signal guiding the noise prediction process at each step of the diffusion model.

[0049] A diffusion model may include a forward process and a reverse process. In the forward process, noise is gradually added to the original image, while in the reverse process, a meaningful image can be generated by starting from a pure noise state and gradually removing the noise. In this process, the text prompt and the embeddings of the conditioning image are provided as conditional inputs to the noise prediction network to guide noise prediction at each step of the reverse process.

[0050] The formula of the diffusion model applied in the step (S500) of generating the first sensor contamination image is defined as Equation 2 and Equation 3 below.

[0051] [Mathematical Formula 2]

[0052]

[0053] [Mathematical Formula 3]

[0054]

[0055] In mathematical formula 2 is the conditioning image, E is the image encoder, I is the input image, The image for specifying the area of ​​the foreign object, The function represents a binarized image for two regions: the foreign object region and the non-foreign object region. In Equation 3 is normal distribution noise, represents the noise predicted by the diffusion model. Here, the input t is the timestep of the diffusion model, means text prompt.

[0056] The step of generating the first sensor contamination image (S500) may use a ControlNet structure as a diffusion model. ControlNet is a model structure that introduces an additional control mechanism to an existing diffusion model, enabling fine control of the structure and content of the output image according to the input conditioning image.

[0057] The core architecture of ControlNet involves freezing the weights of the basic diffusion model and combining this with additional network paths (including zero convolution layers) that process the conditioning image. By appropriately injecting the conditioning signal into various layers of the U-Net structure, these additional paths can maintain the influence of the conditioning image throughout the generation process.

[0058] The ControlNet structure can provide features and advantages such as: 1) fine control of the structure of the generated image by utilizing spatial information of the conditioning image; 2) control of features of various scales, ranging from fine textures of foreign objects to overall distribution patterns, by injecting conditioning signals into various resolution layers of the U-Net structure; 3) effective conditioning ability with relatively few computational resources by training only additional control modules while reusing the weights of the existing diffusion model; 4) the conditioning signal can be gradually integrated without compromising the performance of the base model through zero convolution layers that are gradually trained starting with zero weights that have no effect initially; and 5) processing various forms of conditioning images such as foreign object region masks, edge maps, and depth maps.

[0059] A diffusion model using a ControlNet structure can generate a first sensor contamination image in which foreign matter is naturally applied only to specific areas by combining the types of foreign matter represented by text prompts with distribution information represented by conditioning images. This approach is ideal for autonomous driving sensor data augmentation as it ensures both diversity of foreign matter types and accuracy of distribution patterns.

[0060] In the step of generating the first sensor contamination image (S500), it is important that after generating the first sensor contamination image, foreign matter is applied only to specific areas, while the remaining areas maintain the characteristics of the original clean image. Therefore, the step of generating the first sensor contamination image (S500) may perform additional processing to ensure that foreign matter is not generated in pixels other than specific areas within the first sensor contamination image generated through the diffusion model in the inference step. Specifically, the step of generating the first sensor contamination image (S500) may apply an inpainting technique that accurately separates the foreign matter region from the first sensor contamination image passed through ControlNet and appropriately composites it with the original clean image. This process can be mathematically expressed by the following Equation 4.

[0061] [Mathematical Formula 4]

[0062]

[0063] Here, is the second sensor contamination image, I is the clean image, represents the image of the foreign substance generation area.

[0064] FIG. 3 shows a flowchart of the step (S700) of generating a second sensor contamination image according to an embodiment of the present invention. Referring to FIG. 3, the step (S700) of generating a second sensor contamination image may include the step of separating a foreign matter generating area image (S710), the step of separating into a foreign matter area image and a clean area image (S730), the step of classifying into one or more class areas (S750), and the step of synthesizing the second sensor contamination image (S770).

[0065] The step of generating a second sensor contamination image (S700) can generate a second sensor contamination image by applying transparency to the foreign matter area of ​​the first sensor contamination image. The step of generating a second sensor contamination image (S700) is a process of generating a more realistic image of foreign matter by applying transparency, which is an important characteristic of actual foreign matter, to the first sensor contamination image. In an actual autonomous driving environment, foreign matter adhering to a camera sensor is not simply an opaque cover, but has characteristics of having various degrees of transparency depending on its thickness and density.

[0066] The first sensor contamination image generated in the step (S500) of generating the first sensor contamination image succeeded in generating a foreign matter region, but this is a form that is not entirely suitable for actual foreign matter data augmentation because it did not consider transparency, which is one of the characteristics present in the foreign matter pattern. Therefore, it is essential to apply an image post-processing process based on transparency to the foreign matter region image separated from the first sensor contamination image by mimicking the characteristics of the foreign matter.

[0067] Considering the physical characteristics of actual foreign substances, areas where the foreign substances are densely concentrated have dark brightness values ​​and low transparency due to low light transmittance. Conversely, areas where the foreign substances are thinly spread have relatively high transparency. Therefore, it is reasonable to divide the class area segmentation in the foreign substance region based on the brightness values ​​of the image, and accordingly, a post-processing technique that emphasizes transparency is applied. In the step of generating the first sensor contamination image (S500), the foreign substance region of the first sensor contamination image is extracted, and transparency attributes are assigned based on the brightness value distribution of the corresponding region, thereby simulating the characteristics of actual foreign substances with various thicknesses and densities. The second sensor contamination image generated through this process can provide data more suitable for autonomous driving sensor foreign substance detection training by expressing a continuum of foreign substance transparency, going beyond simply determining whether the foreign substance is 'present or absent'.

[0068] FIG. 4 illustrates a post-processing method for generating a second sensor contamination image according to an embodiment of the present invention. Hereinafter, with reference to FIG. 4, the post-processing process for generating a second sensor contamination image will be described in detail.

[0069] The step of separating the image of the foreign matter generation area (S710) can separate the image of the foreign matter generation area from the first sensor contamination image. Here, the image of the foreign matter generation area may refer to an image in which only the area contaminated with foreign matter is separated from the first sensor contamination image.

[0070] The step of separating the image of the foreign substance generation area (S710) is a process of effectively separating only the area where foreign substances are attached from the first sensor contamination image. The first sensor contamination image is an image generated through a diffusion model, in which foreign substances are applied to a specific area. In the step of separating the image of the foreign substance generation area (S710), the previously used image of the foreign substance area designation can be utilized as reference information to accurately identify the foreign substance area.

[0071] The step of separating the image of the foreign substance generation area (S710) can be implemented through various image processing techniques, such as 1) area extraction using a foreign substance area designation image as a mask, 2) area segmentation based on image characteristics such as color, texture, and brightness values, 3) identification of the foreign substance area using a deep learning-based segmentation model, and 4) analysis of the difference value between the original clean image and the first sensor contamination image.

[0072] The image of the foreign substance generation area separated in the step (S710) of separating the image of the foreign substance generation area ( ) can serve as basic data for applying transparency characteristics in a subsequent step. The step of separating the image of the foreign substance generation area (S710) is an important step that ensures the accuracy of subsequent transparency processing by accurately identifying the application area of ​​the foreign substance and clearly distinguishing it from the background area.

[0073] The step of separating into a foreign matter area image and a clean area image (S730) can separate the foreign matter generated area image into a foreign matter area image and a clean area image. The foreign matter generated area image can be defined as having foreign matter attached to the corresponding area of ​​the clean image. Therefore, the foreign matter area image may mean an image in which only foreign matter is separated from the corresponding area, and the clean area image may mean a clean image corresponding to the corresponding area.

[0074] The step of separating into a foreign matter area image and a clean area image (S730) can subdivide the foreign matter generated area image into two components: a foreign matter area image and a clean area image. The step of separating into a foreign matter area image and a clean area image (S730) is a separation method that reflects the essential structure of the image with actual foreign matter attached, and the foreign matter generated area can be viewed as a composite structure in which a foreign matter layer is placed on top of the original clean image.

[0075] The foreign object area image is a layer containing only the characteristics (texture, color, pattern, etc.) of the foreign object and can be subjected to transparency processing later. On the other hand, the clean area image is a layer representing the original background of that location and can be used as the background visible beneath the foreign object during the final compositing process.

[0076] The step of separating into a foreign object area image and a clean area image (S730) can be implemented by processing methods such as 1) difference operation between the original clean image and the foreign object generated area image, 2) separation of the foreground (foreign object) and background (clean area) through a layer decomposition algorithm, 3) application of an image decomposition technique considering the optical characteristics of the foreign object, and 4) layer extraction using a reference clean image.

[0077] The foreign area image and the clean area image obtained through the step (S730) of separating into a foreign area image and a clean area image can be used as core input data for subsequent transparency processing and final synthesis processes.

[0078] The step of classifying into one or more class regions (S750) can classify the foreign matter area image into one or more class regions according to the brightness value of the foreign matter. The step of classifying into one or more class regions (S750) is a process of subdividing the foreign matter area image into multiple classes according to the brightness value (luminance) of the foreign matter. The step of classifying into one or more class regions (S750) is based on the physical characteristic that transparency, which is an important characteristic of the actual foreign matter, has a correlation with the brightness value.

[0079] Generally, areas where foreign matter is thick and dense have low light transmittance, resulting in dark brightness values ​​and low transparency. Conversely, areas where foreign matter is thinly distributed have relatively bright values ​​and exhibit high transparency. Reflecting these characteristics, changes in transparency according to the thickness and density of the foreign matter can be modeled by classifying the pixels of the foreign matter area image into various classes based on brightness values.

[0080] In one embodiment, the step (S750) of classifying into one or more class regions includes a brightness value (y) and a threshold value It can be based on, and the foreign matter area is divided into four classes It can be classified as. Each class is (Region where y > σ₃, brightest region, close to clean), ( (region, medium brightness area), ( (inner area, intermediate dark area), ( It can be defined as the darkest region, the darkest region, and the part where the foreign matter is thickest. This classification method is effective for modeling the continuous transparency changes of foreign matter in stages and can represent various foreign matter patterns and characteristics.

[0081] The step of classifying into one or more class regions (S750) is a brightness value threshold Based on [this], the above foreign matter area image can be classified into n class areas. The step of classifying into one or more class areas (S750) is an approach to quantitatively and precisely model the transparency characteristics of the foreign matter.

[0082] threshold ( The threshold can be set by considering the distribution of brightness values ​​of pixels within the foreign substance region, and this can be predefined as a fixed value or dynamically determined through histogram analysis of the image. Various implementations are possible depending on the threshold setting method, and methods such as 1) dividing the brightness value range (0~255) into n intervals evenly, 2) using a specific quantile in the brightness value distribution of pixels in the foreign substance region as a threshold, 3) using natural cluster boundaries as a threshold through algorithms such as K-means, and 4) setting a threshold based on the relationship between the optical characteristics of the foreign substance and transparency may be included.

[0083] As shown in FIG. 4, in one embodiment, three thresholds It can be classified into four classes using, and each threshold can represent a specific transparency transition point. This classification is This is performed through a function that determines the class to which a pixel belongs by comparing its brightness value with a threshold. This classification method based on brightness thresholds enables fine-grained control of transparency processing and allows for the effective simulation of various physical characteristics of foreign substances (thickness, density, distribution patterns, etc.).

[0084] The step of synthesizing the second sensor contamination image (S770) can perform an alpha blending operation by applying different transparency to each classified class area, and synthesize the second sensor contamination image by performing an inpainting operation with the result of the alpha blending operation and the clean area image.

[0085] The step of synthesizing the second sensor contamination image (S770) is a final synthesis process that applies differentiated transparency to the previously classified foreign substance class regions and naturally integrates them with the clean region image. The step of synthesizing the second sensor contamination image (S770) is a core process that generates a final sensor contamination image that realistically reflects the transparency characteristics of the foreign substance.

[0086] Alpha blending operations that apply different transparency values ​​(α) to each class area are essential for simulating continuous transparency changes of foreign matter patterns. In this case, the transparency values ​​are assigned by reflecting the physical characteristics of each class, and generally, low transparency (low α value) can be applied to classes with low brightness values ​​(areas where foreign matter is thickly distributed), and high transparency (high α value) can be applied to classes with high brightness values.

[0087] After the alpha blending operation, the result is integrated with the clean area image through an inpainting operation. This inpainting process ensures a natural transition between foreign and non-foreign areas and minimizes artifacts near the boundaries.

[0088] The step of synthesizing the second sensor contamination image (S770) is for each class area Intermediate result by applying alpha blending to Generate and sum them to create the final image ( It can generate a ). In this process, a clean area The corresponding part of the original image can be used as is.

[0089] The step of synthesizing the second sensor contamination image (S770) can perform an alpha blending operation by applying the following mathematical formula 1 to each classified class region.

[0090] [Mathematical Formula 1]

[0091]

[0092] (Here, is the second sensor contamination image, I is the clean image, is the image of the foreign object generation area, r(·,·) is a class area classification function based on brightness values, is the brightness value threshold, represents the alpha value of the i-th class)

[0093] Mathematical Equation 1 can accurately model the transparency characteristics of foreign substances, and the number of classes (n) and the alpha value of each class ( By adjusting ), flexibility can be provided to express various foreign matter patterns and characteristics. In addition, Equation 1 is applied at the pixel level, allowing even minute changes within the foreign matter pattern to be expressed.

[0094] When the number of classes was classified into 2, 3, and 4 based on transparency including a clean area, the result was best when classifying into 4 classes and considering transparency.

[0095] Table 1 below compares the results of training a segmentation model using the Front view class, which collects only data captured by installing a camera in front of a vehicle from the publicly available foreign object data, with the results of using images generated by applying the invention to the same model as additional training data. The comparison was performed across all camera view classes of the existing foreign object data, and it can be confirmed that the accuracy, precision, and mIoU performance are superior.

[0096] [Table 1]

[0097]

[0098] FIG. 5 shows a configuration diagram of a device (100) for augmenting foreign object data of an autonomous driving camera sensor according to an embodiment of the present invention. Referring to FIG. 5, the configuration of the illustrated device (100) for augmenting foreign object data of an autonomous driving camera sensor is merely a simplified example. In one embodiment of the present invention, the device (100) for augmenting foreign object data of an autonomous driving camera sensor may include other configurations for performing the computing environment of the device (100), and only some of the disclosed configurations may constitute the device (100).

[0099] A device (100) for augmenting foreign object data from an autonomous driving camera sensor may include a processor (110) including one or more cores, a memory (120), and a network (130).

[0100] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in memory (120) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the learning of a network function. For example, a CPU and a GPGPU can work together to process the learning of a network function and data classification using the network function. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the learning of a network function and data classification using the network function. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0101] The processor (110) can receive a clean image and a foreign object area designation image captured by an autonomous driving camera sensor. The processor (110) can perform the aforementioned input receiving step (S100).

[0102] The processor (110) can generate a conditioning image by performing a composite operation on a clean image and a foreign matter area designation image. The processor (110) can perform the step (S300) of generating the aforementioned conditioning image.

[0103] The processor (110) can generate a first sensor contamination image by inputting a text prompt specifying foreign matter and the conditioning image into a diffusion model. The processor (110) can perform the step (S500) of generating the aforementioned first sensor contamination image.

[0104] The processor (110) can generate a second sensor contamination image by applying transparency to the foreign matter area of ​​the first sensor contamination image. The processor (110) can perform the step (S700) of generating the aforementioned second sensor contamination image.

[0105] The memory (120) can store any form of information generated or determined by the processor (110) and any form of information received by the network (130).

[0106] The memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (120) on the internet. The description of the memory described above is merely an example, and the present disclosure is not limited thereto.

[0107] The network (130) may use any type of known wired or wireless communication system. The network (130) may receive clean images, etc. from a related device or related system.

[0108] The network (130) can transmit and receive information, user interfaces, etc., processed by the processor (110) through communication with other terminals. For example, the network (130) can provide a user interface generated by the processor (100) to a client (e.g., a user terminal). In addition, the network (130) can receive external input from a user authorized as a client and transmit it to the processor (110). At this time, the processor (110) can process operations such as outputting, modifying, changing, or adding information provided through the user interface based on the external input from the user received from the network (130).

[0109] Meanwhile, the device (100) for augmenting autonomous driving camera sensor foreign object data according to one embodiment of the present disclosure may include a server as a computing system that transmits and receives information through communication with a client. At this time, the client may be any type of terminal capable of accessing the server.

[0110] In an additional embodiment, the device (100) for augmenting autonomous driving camera sensor foreign object data may include any type of terminal that receives data resources generated from any server and performs additional information processing.

[0111] FIG. 6 shows a schematic diagram of a computing environment according to an embodiment of the present invention.

[0112] Although the present disclosure has been described as generally being implementable by a computing device, those skilled in the art will understand that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers, and / or as a combination of hardware and software.

[0113] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, those skilled in the art will be well aware that the method of the present disclosure may be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0114] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0115] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0116] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0117] An exemplary environment for implementing various aspects of the present disclosure, including a computer (1000), is shown, wherein the computer (1000) includes a processing unit (1020), a system memory (1030), and a system bus (1010). The system bus (1010) connects system components, including the system memory (1030) (but not limited thereto), to the processing unit (1020). The processing unit (1020) may be any processor among various commercial processors. Dual processors and other multiprocessor architectures may also be used as the processing unit (1020).

[0118] The system bus (1010) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. The system memory (1030) includes read-only memory (ROM) (1034) and random access memory (RAM) (1032). The basic input / output system (BIOS) is stored in non-volatile memory (1034), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1000) at times such as during startup. The RAM (1032) may also include high-speed RAM, such as static RAM, for caching data.

[0119] The computer (1000) also includes an internal hard disk drive (HDD) (1050) (e.g., EIDE, SATA)—this internal hard disk drive (1050) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1060) (e.g., for reading from or writing to a removable diskette), and an optical disk drive (1070) (e.g., for reading from a CD-ROM disk or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1050), the magnetic disk drive (1060), and the optical disk drive (1070) may each be connected to the system bus (1010) via a hard disk drive interface, a magnetic disk drive interface, and an optical drive interface. Interfaces for implementing external drives include at least one or both of the Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0120] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1000), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0121] A number of program modules, including an operating system (1092), one or more application programs (1094), other program modules (1096), and a database (1098), may be stored in the drive and RAM (1032). All or part of the operating system, applications, modules, and / or data may also be cached in RAM (1032). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0122] The user can input commands and information into the computer (1000) through one or more wired / wireless input devices (1042), such as pointing devices like a keyboard and a mouse. Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1020) via an input / output interface (1040) connected to the system bus (1010), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0123] A monitor or other type of display device is also connected to the system bus (1010) via an interface such as a video adapter. In addition to the monitor, the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0124] A computer (1000) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1082), via wired and / or wireless communication. The remote computer(s) (1082) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and may include many or all of the components generally described for the computer (1000). The logical connection includes a wired / wireless connection to a local area network (LAN) and / or a larger network, e.g., a wide area network (WAN). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a global computer network, e.g., the Internet.

[0125] When used in a LAN networking environment, the computer (1000) is connected to a local network (not shown) via a wired and / or wireless communication network interface or adapter (not shown). The adapter (not shown) may facilitate wired or wireless communication to the LAN (not shown), and the LAN (not shown) may also include a wireless access point installed therein to communicate with the wireless adapter (not shown). When used in a WAN networking environment, the computer (1000) may include a modem (not shown), be connected to a communication computing device on the WAN (not shown), or have other means to establish communication over the WAN (not shown), such as through the Internet. The modem (not shown), which may be internal or external and wired or wireless, is connected to the system bus (1010) via a serial port interface (not shown). In a networked environment, the program modules described for the computer (1000) or parts thereof may be stored in a remote memory / storage device (not shown). You will be well aware that the illustrated network connection is exemplary and that other means of establishing communication links between computers can be used.

[0126] The computer (1000) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0127] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0128] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0129] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and model steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of the present disclosure.

[0130] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0131] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0132] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0133] The embodiments of the present invention described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present invention, or a recording medium on which such a program is recorded. Such a recording medium may be executed not only on a server but also on a user terminal.

[0134] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

Claim 1 A method for augmenting foreign matter data from an autonomous driving camera sensor performed by a computing device, comprising: receiving a clean image and a foreign matter area designation image captured by an autonomous driving camera sensor; generating a conditioning image by performing a synthesis operation on the clean image and the foreign matter area designation image; generating a first sensor contamination image by inputting a text prompt specifying the foreign matter and the conditioning image into a diffusion model; and generating a second sensor contamination image by applying transparency to the foreign matter area of ​​the first sensor contamination image; wherein the step of generating the second sensor contamination image comprises: separating a foreign matter generation area image from the first sensor contamination image; separating the foreign matter generation area image into a foreign matter area image and a clean area image; classifying the foreign matter area image into one or more class areas according to the brightness value of the foreign matter; and performing an alpha blending operation by applying different transparency to each classified class area, and synthesizing the second sensor contamination image by performing an inpainting operation on the result of the alpha blending operation and the clean area image. Claim 2 A method according to claim 1, wherein the input receiving step is to receive an image designating a foreign object region generated by a Variational Auto Encoder (VAE). Claim 3 A method according to claim 1, wherein the step of generating the first sensor contamination image is to use a ControlNet structure as the diffusion model. Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 In claim 1, the step of classifying into one or more class regions is a brightness value threshold A method comprising the step of classifying the foreign matter region image into n class regions based on [ ] and synthesizing the second sensor contamination image, wherein the method comprises applying the following Equation 1 to each classified class region to perform an alpha blending operation. [Equation 1] (Here, is the second sensor contamination image, I is the clean image, is the image of the foreign object generation area, r(·,·) is a class area classification function based on brightness values, is the brightness value threshold, represents the alpha value of the i-th class) Claim 9 A device for augmenting foreign matter data from an autonomous driving camera sensor, comprising: a processor including one or more cores; and a memory; wherein the processor receives a clean image and a foreign matter area designation image captured by an autonomous driving camera sensor, generates a conditioning image by performing a synthesis operation on the clean image and the foreign matter area designation image, generates a first sensor contamination image by inputting a text prompt specifying foreign matter and the conditioning image into a diffusion model, and generates a second sensor contamination image by applying transparency to the foreign matter area of ​​the first sensor contamination image, wherein generating the second sensor contamination image involves separating a foreign matter generation area image from the first sensor contamination image, separating the foreign matter generation area image into a foreign matter area image and a clean area image, classifying the foreign matter area image into one or more class areas according to the brightness value of the foreign matter, performing an alpha blending operation by applying different transparency to each classified class area, and synthesizing the second sensor contamination image by performing an inpainting operation on the result of the alpha blending operation and the clean area image. Claim 10 A computer program stored on a computer-readable storage medium and comprising instructions that cause a computer to perform the following operations, wherein the operations include: receiving a clean image captured by an autonomous driving camera sensor and a foreign matter area designation image generated by a variant autoencoder; generating a conditioning image by performing a composite operation on the clean image and the foreign matter area designation image; generating a first sensor contamination image by inputting a text prompt specifying foreign matter and the conditioning image into a diffusion model; and generating a second sensor contamination image by applying transparency to the foreign matter area of ​​the first sensor contamination image; wherein the operation of generating the second sensor contamination image includes: separating a foreign matter generation area image from the first sensor contamination image; separating the foreign matter generation area image into a foreign matter area image and a clean area image; and classifying the foreign matter area image into one or more class areas according to the brightness value of the foreign matter. A computer program stored on a computer-readable storage medium, comprising: performing an alpha blending operation by applying different transparency to each classified class area, and synthesizing a second sensor contamination image by performing an inpainting operation on the result of the alpha blending operation and the clean area image.

Images